SlideShare a Scribd company logo
1 of 22
Download to read offline
LearningCheap and
Novel Flight Itineraries
Dima Karamshuk, David Matthews
Skyscanner
Applied Data
ScienceTrack
Planning aTrip
How much time you spend to
choose a flight?
Planning aTrip
–3.5h European travellers spend on average to find a perfect
flight, often longer than the flight itself https://goo.gl/74CivT
Planning aTrip
How much of you choose airline
by price?
–3.5h European travellers spend on average to find a perfect
flight, often longer than the flight itself https://goo.gl/74CivT
Planning aTrip
–37%of users choose airlines by competitive price, more want to
see cheapest price for comparison https://goo.gl/8UX3vx
–3.5h European travellers spend on average to find a perfect
flight, often longer than the flight itself https://goo.gl/74CivT
Combination
Itineraries
Potentially cheaper itineraries in over half of all search results
Virgin
Delta
Dublin Barcelona
Ryanair
Aer Lingus
Barcelona NewYork
Ryanair American
Competitive
Combinations
Tips for booking your next flight
– good for last minute booking
– average savings of 9% on return ticket
– 90% of competitive combinations are
from top-30% airlines
– good deals when flying from US, UK,
Spain, Germany, Italy and other origins
Problem
Global Distribution
Systems
OnlineTravel
Agencies
Airlines
Traveler
• Combinations require more queries to ticket providers
• Most of variants are not competitive
Solution: Only choose combinations which are likely to be competitive
Supervised
Learning
Classify whether for a query Q a combination of partners (X andY) is
going to be in Top-10 search results
Coverage: How many of all possible cheap itineraries we recall
Cost: How much queries for flight quotes are required
Metrics
Dataset
• sample all possible combinations for a share of searches
• collect examples of competitive and non-competitive combinations
CostCoverage
Supervised
Learning
Use your favorite classifier
perfect predictor
heuristic-based baseline
CostCoverage
Supervised
Learning
Tree ensembles (Random Forest) achieve good performs
5%
45%
In practice
Feature
Engineering
Can we improve performance with smart feature engineering?
London Gatwick [1 0 0 … 0 ]
London Stansted [0 1 0 … 0]
One-hot encoding Better encoding
London Gatwick
London Stansted
[1.0 0.9 0.9 ...]
[1.0 0.9 0.1 …]
Barcelona [0 0 1 … 0] Barcelona [0.0 1.0 0.5 …]
London
European
Trans-Atlantic
Location
Embeddings Perozzi et al., KDD, 2014.
[London, Barcelona, Frankfurt am Main, NewYork, ….]
word
sentence
• Option N1: Every user’s history is a sentence (think ofWord2Vec)
• Option N2: Learn embeddings on graphs of locations
competitive
or not
origin
destination
…
…
• Option N3:Train embeddings for target
problem
Location
Embeddings
• Capture geographical proximity (Europe vs. Asia)
• Learn function of the airport (Heathrow and Gatwick vs. Stansted)
• Produce a slight improvement in prediction performance
Model
Staleness
Performance of the model stales, hence needs to be updated regularly
everyday re-training
one-off training
Production
Pipeline
Data Querying
AWS Athena
Data Archive
AWS S3
Data Collection Current Model
Model Training
scikit-learn
Training Data
7 recent days
Validation Data
5% of the last day
Model Validation
Passed?
Skyscanner
Traffic
Pre-processing
Experiments with
Challenger Model
5%
5%
90%
Training Component (AWS CF + AWS Data Pipeline)
Report Failure
Update ModelApache Kafka
Serving Component
Production
Pipeline
Data Querying
AWS Athena
Data Archive
AWS S3
Data Collection Current Model
Model Training
scikit-learn
Training Data
7 recent days
Validation Data
5% of the last day
Model Validation
Passed?
Skyscanner
Traffic
Pre-processing
Experiments with
Challenger Model
5%
5%
90%
Training Component (AWS CF + AWS Data Pipeline)
Report Failure
Update ModelApache Kafka
Serving Component
• re-train the model everyday against model drift
• run on a single large machine vs. distributed cluster
Production
Pipeline
Data Querying
AWS Athena
Data Archive
AWS S3
Data Collection Current Model
Model Training
scikit-learn
Training Data
7 recent days
Validation Data
5% of the last day
Model Validation
Passed?
Skyscanner
Traffic
Pre-processing
Experiments with
Challenger Model
5%
5%
90%
Training Component (AWS CF + AWS Data Pipeline)
Report Failure
Update ModelApache Kafka
Serving Component
• sample all possible combinations on 5% of users’ traffic
Production
Pipeline
Data Querying
AWS Athena
Data Archive
AWS S3
Data Collection Current Model
Model Training
scikit-learn
Training Data
7 recent days
Validation Data
5% of the last day
Model Validation
Passed?
Skyscanner
Traffic
Pre-processing
Experiments with
Challenger Model
5%
5%
90%
Training Component (AWS CF + AWS Data Pipeline)
Report Failure
Update ModelApache Kafka
Serving Component
• update the model if it passes the tests and serve it to 90% of the users
• leave 5% for A/B experiments with better models
ModelStability
(Origin, Destination, Provider) rules
• combine the models trained on consecutive days to control for stability
Lessons From
theTrenches – bootstrapping ML projects requires 20% of modelingand 80% of
engineering– in the long run should be vice versa
– many interesting ML problems arise in production(e.g., model
staleness ad stability)
– simple solutions are often good enough
Join our Team!
@SkyscannerEng

More Related Content

Recently uploaded

SCM Symposium PPT Format Customer loyalty is predi
SCM Symposium PPT Format Customer loyalty is prediSCM Symposium PPT Format Customer loyalty is predi
SCM Symposium PPT Format Customer loyalty is predieusebiomeyer
 
『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书
『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书
『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书rnrncn29
 
TRENDS Enabling and inhibiting dimensions.pptx
TRENDS Enabling and inhibiting dimensions.pptxTRENDS Enabling and inhibiting dimensions.pptx
TRENDS Enabling and inhibiting dimensions.pptxAndrieCagasanAkio
 
『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书
『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书
『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书rnrncn29
 
ETHICAL HACKING dddddddddddddddfnandni.pptx
ETHICAL HACKING dddddddddddddddfnandni.pptxETHICAL HACKING dddddddddddddddfnandni.pptx
ETHICAL HACKING dddddddddddddddfnandni.pptxNIMMANAGANTI RAMAKRISHNA
 
Top 10 Interactive Website Design Trends in 2024.pptx
Top 10 Interactive Website Design Trends in 2024.pptxTop 10 Interactive Website Design Trends in 2024.pptx
Top 10 Interactive Website Design Trends in 2024.pptxDyna Gilbert
 
Company Snapshot Theme for Business by Slidesgo.pptx
Company Snapshot Theme for Business by Slidesgo.pptxCompany Snapshot Theme for Business by Slidesgo.pptx
Company Snapshot Theme for Business by Slidesgo.pptxMario
 
Unidad 4 – Redes de ordenadores (en inglés).pptx
Unidad 4 – Redes de ordenadores (en inglés).pptxUnidad 4 – Redes de ordenadores (en inglés).pptx
Unidad 4 – Redes de ordenadores (en inglés).pptxmibuzondetrabajo
 
Film cover research (1).pptxsdasdasdasdasdasa
Film cover research (1).pptxsdasdasdasdasdasaFilm cover research (1).pptxsdasdasdasdasdasa
Film cover research (1).pptxsdasdasdasdasdasa494f574xmv
 
办理多伦多大学毕业证成绩单|购买加拿大UTSG文凭证书
办理多伦多大学毕业证成绩单|购买加拿大UTSG文凭证书办理多伦多大学毕业证成绩单|购买加拿大UTSG文凭证书
办理多伦多大学毕业证成绩单|购买加拿大UTSG文凭证书zdzoqco
 
IP addressing and IPv6, presented by Paul Wilson at IETF 119
IP addressing and IPv6, presented by Paul Wilson at IETF 119IP addressing and IPv6, presented by Paul Wilson at IETF 119
IP addressing and IPv6, presented by Paul Wilson at IETF 119APNIC
 

Recently uploaded (11)

SCM Symposium PPT Format Customer loyalty is predi
SCM Symposium PPT Format Customer loyalty is prediSCM Symposium PPT Format Customer loyalty is predi
SCM Symposium PPT Format Customer loyalty is predi
 
『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书
『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书
『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书
 
TRENDS Enabling and inhibiting dimensions.pptx
TRENDS Enabling and inhibiting dimensions.pptxTRENDS Enabling and inhibiting dimensions.pptx
TRENDS Enabling and inhibiting dimensions.pptx
 
『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书
『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书
『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书
 
ETHICAL HACKING dddddddddddddddfnandni.pptx
ETHICAL HACKING dddddddddddddddfnandni.pptxETHICAL HACKING dddddddddddddddfnandni.pptx
ETHICAL HACKING dddddddddddddddfnandni.pptx
 
Top 10 Interactive Website Design Trends in 2024.pptx
Top 10 Interactive Website Design Trends in 2024.pptxTop 10 Interactive Website Design Trends in 2024.pptx
Top 10 Interactive Website Design Trends in 2024.pptx
 
Company Snapshot Theme for Business by Slidesgo.pptx
Company Snapshot Theme for Business by Slidesgo.pptxCompany Snapshot Theme for Business by Slidesgo.pptx
Company Snapshot Theme for Business by Slidesgo.pptx
 
Unidad 4 – Redes de ordenadores (en inglés).pptx
Unidad 4 – Redes de ordenadores (en inglés).pptxUnidad 4 – Redes de ordenadores (en inglés).pptx
Unidad 4 – Redes de ordenadores (en inglés).pptx
 
Film cover research (1).pptxsdasdasdasdasdasa
Film cover research (1).pptxsdasdasdasdasdasaFilm cover research (1).pptxsdasdasdasdasdasa
Film cover research (1).pptxsdasdasdasdasdasa
 
办理多伦多大学毕业证成绩单|购买加拿大UTSG文凭证书
办理多伦多大学毕业证成绩单|购买加拿大UTSG文凭证书办理多伦多大学毕业证成绩单|购买加拿大UTSG文凭证书
办理多伦多大学毕业证成绩单|购买加拿大UTSG文凭证书
 
IP addressing and IPv6, presented by Paul Wilson at IETF 119
IP addressing and IPv6, presented by Paul Wilson at IETF 119IP addressing and IPv6, presented by Paul Wilson at IETF 119
IP addressing and IPv6, presented by Paul Wilson at IETF 119
 

Featured

2024 State of Marketing Report – by Hubspot
2024 State of Marketing Report – by Hubspot2024 State of Marketing Report – by Hubspot
2024 State of Marketing Report – by HubspotMarius Sescu
 
Everything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPTEverything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPTExpeed Software
 
Product Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage EngineeringsProduct Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage EngineeringsPixeldarts
 
How Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental HealthHow Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental HealthThinkNow
 
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfAI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfmarketingartwork
 
PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024Neil Kimberley
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)contently
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024Albert Qian
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsKurio // The Social Media Age(ncy)
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Search Engine Journal
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summarySpeakerHub
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next Tessa Mero
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentLily Ray
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best PracticesVit Horky
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project managementMindGenius
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...RachelPearson36
 

Featured (20)

2024 State of Marketing Report – by Hubspot
2024 State of Marketing Report – by Hubspot2024 State of Marketing Report – by Hubspot
2024 State of Marketing Report – by Hubspot
 
Everything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPTEverything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPT
 
Product Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage EngineeringsProduct Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage Engineerings
 
How Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental HealthHow Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental Health
 
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfAI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
 
Skeleton Culture Code
Skeleton Culture CodeSkeleton Culture Code
Skeleton Culture Code
 
PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search Intent
 
How to have difficult conversations
How to have difficult conversations How to have difficult conversations
How to have difficult conversations
 
Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data Science
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best Practices
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project management
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
 

Learning Cheap and Novel Flight Itineraries

  • 1. LearningCheap and Novel Flight Itineraries Dima Karamshuk, David Matthews Skyscanner Applied Data ScienceTrack
  • 2. Planning aTrip How much time you spend to choose a flight?
  • 3. Planning aTrip –3.5h European travellers spend on average to find a perfect flight, often longer than the flight itself https://goo.gl/74CivT
  • 4. Planning aTrip How much of you choose airline by price? –3.5h European travellers spend on average to find a perfect flight, often longer than the flight itself https://goo.gl/74CivT
  • 5. Planning aTrip –37%of users choose airlines by competitive price, more want to see cheapest price for comparison https://goo.gl/8UX3vx –3.5h European travellers spend on average to find a perfect flight, often longer than the flight itself https://goo.gl/74CivT
  • 6. Combination Itineraries Potentially cheaper itineraries in over half of all search results Virgin Delta Dublin Barcelona Ryanair Aer Lingus Barcelona NewYork Ryanair American
  • 7. Competitive Combinations Tips for booking your next flight – good for last minute booking – average savings of 9% on return ticket – 90% of competitive combinations are from top-30% airlines – good deals when flying from US, UK, Spain, Germany, Italy and other origins
  • 8. Problem Global Distribution Systems OnlineTravel Agencies Airlines Traveler • Combinations require more queries to ticket providers • Most of variants are not competitive Solution: Only choose combinations which are likely to be competitive
  • 9. Supervised Learning Classify whether for a query Q a combination of partners (X andY) is going to be in Top-10 search results Coverage: How many of all possible cheap itineraries we recall Cost: How much queries for flight quotes are required Metrics Dataset • sample all possible combinations for a share of searches • collect examples of competitive and non-competitive combinations
  • 10. CostCoverage Supervised Learning Use your favorite classifier perfect predictor heuristic-based baseline
  • 11. CostCoverage Supervised Learning Tree ensembles (Random Forest) achieve good performs 5% 45% In practice
  • 12. Feature Engineering Can we improve performance with smart feature engineering? London Gatwick [1 0 0 … 0 ] London Stansted [0 1 0 … 0] One-hot encoding Better encoding London Gatwick London Stansted [1.0 0.9 0.9 ...] [1.0 0.9 0.1 …] Barcelona [0 0 1 … 0] Barcelona [0.0 1.0 0.5 …] London European Trans-Atlantic
  • 13. Location Embeddings Perozzi et al., KDD, 2014. [London, Barcelona, Frankfurt am Main, NewYork, ….] word sentence • Option N1: Every user’s history is a sentence (think ofWord2Vec) • Option N2: Learn embeddings on graphs of locations competitive or not origin destination … … • Option N3:Train embeddings for target problem
  • 14. Location Embeddings • Capture geographical proximity (Europe vs. Asia) • Learn function of the airport (Heathrow and Gatwick vs. Stansted) • Produce a slight improvement in prediction performance
  • 15. Model Staleness Performance of the model stales, hence needs to be updated regularly everyday re-training one-off training
  • 16. Production Pipeline Data Querying AWS Athena Data Archive AWS S3 Data Collection Current Model Model Training scikit-learn Training Data 7 recent days Validation Data 5% of the last day Model Validation Passed? Skyscanner Traffic Pre-processing Experiments with Challenger Model 5% 5% 90% Training Component (AWS CF + AWS Data Pipeline) Report Failure Update ModelApache Kafka Serving Component
  • 17. Production Pipeline Data Querying AWS Athena Data Archive AWS S3 Data Collection Current Model Model Training scikit-learn Training Data 7 recent days Validation Data 5% of the last day Model Validation Passed? Skyscanner Traffic Pre-processing Experiments with Challenger Model 5% 5% 90% Training Component (AWS CF + AWS Data Pipeline) Report Failure Update ModelApache Kafka Serving Component • re-train the model everyday against model drift • run on a single large machine vs. distributed cluster
  • 18. Production Pipeline Data Querying AWS Athena Data Archive AWS S3 Data Collection Current Model Model Training scikit-learn Training Data 7 recent days Validation Data 5% of the last day Model Validation Passed? Skyscanner Traffic Pre-processing Experiments with Challenger Model 5% 5% 90% Training Component (AWS CF + AWS Data Pipeline) Report Failure Update ModelApache Kafka Serving Component • sample all possible combinations on 5% of users’ traffic
  • 19. Production Pipeline Data Querying AWS Athena Data Archive AWS S3 Data Collection Current Model Model Training scikit-learn Training Data 7 recent days Validation Data 5% of the last day Model Validation Passed? Skyscanner Traffic Pre-processing Experiments with Challenger Model 5% 5% 90% Training Component (AWS CF + AWS Data Pipeline) Report Failure Update ModelApache Kafka Serving Component • update the model if it passes the tests and serve it to 90% of the users • leave 5% for A/B experiments with better models
  • 20. ModelStability (Origin, Destination, Provider) rules • combine the models trained on consecutive days to control for stability
  • 21. Lessons From theTrenches – bootstrapping ML projects requires 20% of modelingand 80% of engineering– in the long run should be vice versa – many interesting ML problems arise in production(e.g., model staleness ad stability) – simple solutions are often good enough