Loading…

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

Like this presentation? Why not share!

Hadoop and Hive at Orbitz, Hadoop World 2010

on

  • 4,808 views

 

Statistics

Views

Total Views
4,808
Views on SlideShare
4,133
Embed Views
675

Actions

Likes
2
Downloads
143
Comments
0

9 Embeds 675

http://www.pamil-visions.net 392
http://www.analyticsteam.com 246
http://everything-pr.com 25
http://feeds.feedburner.com 5
http://webcache.googleusercontent.com 2
http://www.linkedin.com 2
http://translate.googleusercontent.com 1
http://hiderefer.org 1
https://www.linkedin.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Hadoop and Hive at Orbitz, Hadoop World 2010 Hadoop and Hive at Orbitz, Hadoop World 2010 Presentation Transcript

    • Hadoop and Hive at Orbitz Jonathan Seidman and Ramesh Venkataramaiah Hadoop World 2010
    • Agenda •  Orbitz Worldwide •  The challenge of big data at Orbitz •  Hadoop as a solution to the data challenge •  Applications of Hadoop and Hive at Orbitz – improving hotel sort •  Sample analysis and data trends •  Other uses of Hadoop and Hive at Orbitz •  Lessons learned and conclusion page 2
    • Launched: 2001, Chicago, IL page 3
    • Orbitz… …poster children for Hadoop page 4
    • Data Challenges at Orbitz On Orbitz alone we do millions of searches and transactions daily, which leads to hundreds of gigabytes of log data every day. So how do we store and process all of this data? page 5
    • Utterly redonkulous amounts of money $ per managed TB page 6
    • Utterly redonkulous amounts of money More reasonable amounts of money $ per managed TB page 7
    • •  Adding data to our data warehouse also requires a lengthy plan/implement/deploy cycle. •  Because of the expense and time our data teams need to be very judicious about which data gets added. This means that potentially valuable data may not be saved. •  We needed a solution that would allow us to economically store and process the growing volumes of data we collect. page 8
    • Hadoop brings our cost per TB down to $1500 (or even less) page 9
    • •  Important to note that Hadoop is not a replacement to a data warehouse, but rather is a complement to it. •  On the other hand, Hadoop offers benefits other than just cost. page 10
    • page 11
    • How can we improve hotel ranking? Hey! Let’s use machine learning! All the cool kids are doing it! page 12
    • Requires data – lots of data •  Web analytics software providing session data about user behavior. •  Unfortunately specific data fields we needed weren’t loaded into our data warehouse, and just to make things worse the only archive of raw logs available only went back a few days. •  We decided to turn to Hadoop to provide a long-term archive for these logs. •  Storing raw data in HDFS provides access to data not available elsewhere, for example “hotel impression” data: –  115004,1,70.00;35217,2,129.00;239756,3,99.00;83389,4,99.00! page 13
    • Now we need to process the data… •  Extract data from raw Webtrends logs for input to a trained classification process. •  Logs provide input to MapReduce processing which extracts required fields. •  Previous process used a series of Perl and Bash scripts to extract data serially. •  Comparison of performance –  Months worth of data –  Manual process took 109m14s –  MapReduce process took 25m58s page 14
    • Processing Flow – Step 1 page 15
    • Processing Flow – Step 2 page 16
    • Processing Flow – Step 3 page 17
    • Processing Flow – Step 4 page 18
    • Processing Flow – Step 5 page 19
    • Processing Flow – Step 6 page 20
    • Once data is in hive… •  Provides input data to machine learning processes. •  Used to create data exports for further analysis with R scripts, allowing us to derive more complex statistics and visualizations of our data. •  Provides useful metrics, many of which were unavailable with our existing data stores. •  Used for aggregating data for import into our data warehouse for creation of new data cubes, providing analysts access to data unavailable in existing data cubes. page 21
    • Statistical Analysis: Infrastructure and Dataset •  Hive + R platform for query processing and statistical analysis. •  R - Open-source stat package with visualization. •  Hive Dataset: –  Customer hotel booking on our sites and User rating of hotels. •  Investigation: –  Are there built-in data bias? Any Lurking variables? –  What approximations and biases exist? –  Are variables pair-wise correlated? –  Are there macro patterns? page 22
    • Statistical Analysis - Positional Bias •  Lurking variable is… Positional Bias. •  Top positions invariably picked the most. •  Aim to position Best Ranked Hotels at the top based on customer search criteria and user ratings. page 23
    • Statistical Analysis - Kernel Density •  User Ratings of Hotels •  Strongly affected by the number of bins used. •  Kernel density plots are usually a much more effective way to overcome the limitations of histograms. page 24
    • Statistical Analysis - Exploratory correlation page 25
    • Statistical Analysis - More seasonal variations •  Customer hotel stay gets longer during summer months •  Could help in designing search based on seasons. •  Outliers removed. page 26
    • Analysis: take away’s… •  Costs of cleaning and processing data is significant. •  Tendency to create stories out of noise. •  “Median is not the message”; Find macro patterns first. •  If website originated data, watch for hidden bias in data collection. page 27
    • Lessons Learned •  Make sure you’re using the appropriate tool – avoid the temptation to start throwing all of your data in Hadoop when a relational store may be a better choice. •  Expect the unexpected in your data. When processing billions of records of data it’s inevitable that you’ll encounter at least one bad record which will blow up your processing. •  To get buy-in from upper management, present a long-term, unstructured data growth story and explain how this will help harness long-tail opportunities. page 28
    • Lessons Learned (continued) •  Hadoop’s limited security model creates challenges when trying to deploy Hadoop in the enterprise. •  Configuration currently seems to be a black art. It can be difficult to understand which parameters to set and how to determine an optimal configuration. •  Watch your memory use. Sloppy programming practices will bite you when your code needs to process large volumes of data. page 29
    • Hadoop is a virus… page 30
    • Just a few more examples of how Hadoop is being used at Orbitz… •  Measuring page download performance: using web analytics logs as input, a set of MapReduce scripts are used to derive detailed client side performance metrics which allow us to track trends in page download times. •  Searching production logs: an effort is underway to utilize Hadoop to store and process our large volume of production logs, allowing developers and analysts to perform tasks such as troubleshooting production issues. •  Cache analysis: extraction and aggregation of data to provide input to analyses intended to improve the performance of data caches utilized by our web sites. page 31
    • Applications of Hadoop at orbitz are just beginning… •  We’re in the process of quadrupling the capacity of our production cluster. •  Multiple teams are working on new applications of Hadoop •  We continue to explore the use of associated tools – Hbase, Pig, Flume, etc. page 32
    • References •  Hadoop project: http://hadoop.apache.org/ •  Hive project: http://hadoop.apache.org/hive/ •  Hive – A Petabyte Scale Data Warehouse Using Hadoop: http://i.stanford.edu/~ragho/hive-icde2010.pdf •  Hadoop The Definitive Guide, Tom White, O’Reilly Press, 2009 •  Why Model, J. Epstein, 2008 •  Beautiful Data, T. Segaran & J. Hammerbacher, 2009 •  Karmasphere Developer Study: http://www.karmasphere.com/ images/documents/Karmasphere-HadoopDeveloperResearch.pdf page 33
    • Contact •  Jonathan Seidman: –  jseidman@orbitz.com –  @jseidman –  Chicago area Hadoop User Group: http://www.meetup.com/ Chicago-area-Hadoop-User-Group-CHUG/ •  Ramesh Venkataramaiah: –  rvenkataramaiah@orbitz.com page 34