Hadoop and Hive at Orbitz, Hadoop World 2010Presentation Transcript
Hadoop and Hive at Orbitz
Jonathan Seidman and Ramesh Venkataramaiah
Hadoop World 2010
• Orbitz Worldwide
• The challenge of big data at Orbitz
• Hadoop as a solution to the data challenge
• Applications of Hadoop and Hive at Orbitz – improving hotel
• Sample analysis and data trends
• Other uses of Hadoop and Hive at Orbitz
• Lessons learned and conclusion
Launched: 2001, Chicago, IL
…poster children for Hadoop
Data Challenges at Orbitz
On Orbitz alone we do millions of searches and transactions daily,
which leads to hundreds of gigabytes of log data every day.
So how do we store and process all of this data?
Utterly redonkulous amounts of money
$ per managed TB
Utterly redonkulous amounts of money
More reasonable amounts of money
$ per managed TB
• Adding data to our data warehouse also requires a lengthy
• Because of the expense and time our data teams need to be
very judicious about which data gets added. This means that
potentially valuable data may not be saved.
• We needed a solution that would allow us to economically store
and process the growing volumes of data we collect.
Hadoop brings our cost per TB down to $1500 (or even less)
• Important to note that Hadoop is not a replacement to a data
warehouse, but rather is a complement to it.
• On the other hand, Hadoop offers benefits other than just cost.
How can we improve hotel ranking?
Hey! Let’s use machine learning!
All the cool kids are doing it!
Requires data – lots of data
• Web analytics software providing session data about user
• Unfortunately specific data fields we needed weren’t loaded
into our data warehouse, and just to make things worse the
only archive of raw logs available only went back a few days.
• We decided to turn to Hadoop to provide a long-term archive
for these logs.
• Storing raw data in HDFS provides access to data not available
elsewhere, for example “hotel impression” data:
Now we need to process the data…
• Extract data from raw Webtrends logs for input to a trained
• Logs provide input to MapReduce processing which extracts
• Previous process used a series of Perl and Bash scripts to
extract data serially.
• Comparison of performance
– Months worth of data
– Manual process took 109m14s
– MapReduce process took 25m58s
Processing Flow – Step 1
Processing Flow – Step 2
Processing Flow – Step 3
Processing Flow – Step 4
Processing Flow – Step 5
Processing Flow – Step 6
Once data is in hive…
• Provides input data to machine learning processes.
• Used to create data exports for further analysis with R scripts,
allowing us to derive more complex statistics and visualizations
of our data.
• Provides useful metrics, many of which were unavailable with
our existing data stores.
• Used for aggregating data for import into our data warehouse
for creation of new data cubes, providing analysts access to
data unavailable in existing data cubes.
Statistical Analysis: Infrastructure and Dataset
• Hive + R platform for query processing and statistical analysis.
• R - Open-source stat package with visualization.
• Hive Dataset:
– Customer hotel booking on our sites and User rating of hotels.
– Are there built-in data bias? Any Lurking variables?
– What approximations and biases exist?
– Are variables pair-wise correlated?
– Are there macro patterns?
Statistical Analysis - Positional Bias
• Lurking variable is…
• Top positions invariably
picked the most.
• Aim to position Best Ranked
Hotels at the top based on
customer search criteria and
Statistical Analysis - Kernel Density
• User Ratings of Hotels
• Strongly affected by the number
of bins used.
• Kernel density plots are usually
a much more effective way to
overcome the limitations of
Statistical Analysis - More seasonal variations
• Customer hotel stay gets longer during summer months
• Could help in designing search based on seasons.
• Outliers removed.
Analysis: take away’s…
• Costs of cleaning and processing data is significant.
• Tendency to create stories out of noise.
• “Median is not the message”; Find macro patterns first.
• If website originated data, watch for hidden bias in data collection.
• Make sure you’re using the appropriate tool – avoid the temptation to
start throwing all of your data in Hadoop when a relational store may be
a better choice.
• Expect the unexpected in your data. When processing billions of records
of data it’s inevitable that you’ll encounter at least one bad record which
will blow up your processing.
• To get buy-in from upper management,
present a long-term, unstructured
data growth story and explain how this
will help harness long-tail opportunities.
Lessons Learned (continued)
• Hadoop’s limited security model creates challenges when
trying to deploy Hadoop in the enterprise.
• Configuration currently seems to be a black art. It can be
difficult to understand which parameters to set and how to
determine an optimal configuration.
• Watch your memory use. Sloppy programming practices will
bite you when your code needs to process large volumes of
Hadoop is a virus…
Just a few more examples of how Hadoop is being used at Orbitz…
• Measuring page download performance: using web analytics logs as
input, a set of MapReduce scripts are used to derive detailed client
side performance metrics which allow us to track trends in page
• Searching production logs: an effort is underway to utilize Hadoop to
store and process our large volume of production logs, allowing
developers and analysts to perform tasks such as troubleshooting
• Cache analysis: extraction and aggregation of data to provide input to
analyses intended to improve the performance of data caches utilized
by our web sites.
Applications of Hadoop at orbitz are just beginning…
• We’re in the process of quadrupling the capacity of our
• Multiple teams are working on new applications of Hadoop
• We continue to explore the use of associated tools – Hbase,
Pig, Flume, etc.
• Hadoop project: http://hadoop.apache.org/
• Hive project: http://hadoop.apache.org/hive/
• Hive – A Petabyte Scale Data Warehouse Using Hadoop:
• Hadoop The Definitive Guide, Tom White, O’Reilly Press, 2009
• Why Model, J. Epstein, 2008
• Beautiful Data, T. Segaran & J. Hammerbacher, 2009
• Karmasphere Developer Study: http://www.karmasphere.com/
• Jonathan Seidman:
– Chicago area Hadoop User Group: http://www.meetup.com/
• Ramesh Venkataramaiah: