The machine learning team at Orbitz, established in 2009, aims to enhance the customer experience through data-driven methods in hotel search optimizations and cache management. They encountered significant data access challenges but implemented Hadoop and Hive to improve data processing and analysis capabilities. Their work focuses on optimizing cache strategies and traffic management to improve search performance while controlling the impact on suppliers.

1. Machine Learning at Orbitz
Robert Lancaster and Jonathan Seidman
Strata 2011
February 02 | 2011

2. Launched: 2001, Chicago, IL
page 2

3. Why Start the Machine Learning Team at Orbitz?
• Team was created in 2009 with the goal to apply machine
learning techniques to improve the customer experience.
• For example:
– Hotel sort optimization: How can we improve the ranking of
hotel search results in order to show consumers hotels that
more closely match their preferences?
– Cache optimization: can we intelligently cache hotel rates in
order to optimize the performance of hotel searches?
– Personalization/segmentation: can we show targeted search
results to specific consumer segments?
page 3

4. Data Challenges
• The team immediately faced challenges getting access to data:
– Performing required analysis requires access to large
amounts of data on user interaction with the site.
– This data is available in web analytics logs, but required
fields were not available in our data warehouse because of
size considerations.
– Even worse, we had no archive of the data beyond several
days.
– Size constraints aside, there’s considerable time and effort
to get new data added to the data warehouse.
page 4

5. New Data Infrastructure to Address These Challenges
• Hadoop provides a solution to these challenges by:
– Providing long-term storage of entire raw dataset without
placing constraints on how that data is processed.
– Allowing us to immediately take advantage of new web
analytics data added to the site.
– Providing a platform for efficient analysis of data, as well as
preparation of data for input to external processes for further
analysis.
• Hive was added to the infrastructure to provide structure over
the prepared data, facilitating ad-hoc queries and selection of
specific data sets for analysis.
• Data stored in Hive not only supports machine learning efforts,
but also provides metrics to analysts not available through
other sources.
page 5

6. New Data Infrastructure – Cont’d
• Hadoop and Hive are now being used by the machine learning
team to:
– Extract data from logs for hotel sort and cache optimization
analyses.
– Distribute complex cross-validation and performance
evaluation operations.
– Extracting data for clustering.
• Hadoop and Hive have also gained rapid adoption in the
organization beyond the machine learning team: evaluating
page download performance, searching production logs,
keyword analysis, etc.
page 6

7. Use Case – Hotel Cache Optimization
Overview:
Search methodology:
• Subset of total properties in a location (1 page at a time).
• Get “just enough” information to present to consumers.
Caching:
• Reduces impact to suppliers (maintain “look-to-book” ratio).
• Reduces latency.
• Increases “coverage.”
Optimization Goal:
Improve the customer experience (reduce latency, increase
coverage) when searching for hotel rates while controlling impact
on suppliers (maintain look-to-book).
page 7

8. Hotel Cache Optimization – Early Attempts
Early approaches were well intended, but were not driven by analysis of
the available data. For example:
Theory: High amount of thrashing leads to eviction of more useful cache entries.
Attempted Solution: Increase cache size.
Result: No increase in measured coverage.
Problem: No actual analysis on required cache size.
Theory: Locally managed inventory represents “free” information and can be
requested without limit to improve coverage.
Attempted Solution: Don’t cache locally managed inventory. Increase the amount
of local inventory requested with each user search.
Result: No increase in measured coverage.
Problem: Locally managed inventory doesn’t represent a large percentage of total
inventory and is already highly preferenced.
page 8

9. Hotel Cache Optimization – Data Driven Approaches
Data Driven Approaches:
Traffic Partitioning: Identify the subset of traffic that is most
efficient and optimize that subset through prefetching and
increased bursting.
TTL Optimization: Use historic logs of availability and rate
change information to predict volatility of hotel rates and
optimize cache TTL.
page 9

10. Hotel Cache Optimization– Traffic Distribution
100.00%
72% of queries are Queries
singletons and make up
90.00% Searches
nearly a third of total
search volume.
80.00% Reverse Running Total
(Searches)
71.67%
Reverse Running Total
70.00% (Queries)
60.00%
A small number of
queries (3%) make
50.00% up more than a third
of search volume.
40.00%
34.30%
31.87%
30.00%
20.00%
10.00%
2.78%
0.00%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
page 10

11. Optimize Hotel Cache – Traffic Partitioning
Evaluate possible mechanisms for determining most
frequent queries.
Favor mechanisms that gives high search/query ratio for
the greatest percentage of search volume.
Test for stability of mechanism across multiple time periods.
Par$on
Strategy
Descrip$on
Pct
Queries
Pct
Searches
Searches/Query
Baseline
All
traffic
100.00%
100.00%
2.19
Top
50
Top
50
searched
markets
14.88%
26.76%
3.94
Top
50
searched
markets,
weekend
stay
HeurisCc
within
1
month.
0.87%
8.52%
21.4
EnumeraCon
Queries
repeated
5
or
more
Cmes.
3.45%
28.80%
18.29
PredicCon
TBD
TBD
TBD
TBD
page 11

12. Conclusions and Lessons Learned
• Start with a manageable problem (ease of measuring success,
availability of data, etc.)
• Avoid thinking of machine learning team as an R&D
organization.
• Instead, foster machine learning approaches throughout the
organization:
– Embed resources on actual feature teams.
– Machine learning study groups, etc.
page 12