Hive user group presentation from Netflix (3/18/2010)

Hive user group presentation from Netflix (3/18/2010)

• 1.
  Use case studyof Hive/HadoopEva Tse,Jerome Boulon
• 2.
  What are wetrying to achieve?Scalable log analysis to gain business insights:Logs for website streaming (phase 1)All logs from web (phase 2)Output required:Engineers access:Ad-hoc query and reportingBI access:Flat files to be loaded into BI system for cross-functional reporting.
• 3.
  Some Metrics Parsing 0.6TB logs per dayRunning 50+ persistent nodes
• 4.
  Architecture Overview WebAppPhase 1Phase 2Phase 2Chukwa CollectorLog copy deamonHive & Hadoop (for query)Hive MetaStore S3 HDFS / S3Hive & Hadoop running on the cloud
• 5.
  Chukwa StreamingMyAppCollectorNo datawritten to disk on the application side
• 6.
  Data sent toa remote collector using Thrift
• 7.
  Collector write tolocalFS/S3n/HDFS compressed
• 8.
  http://wiki.github.com/jboulon/Honu/ (stay tuned)Workflowto Hive (phase 1)Streaming Session reconstructionEach hour: Run hadoop job to parse last hour log and reconstruct sessions merge small files (from each reducer) load to hiveSession expiration after 24 hours.Will have sessions for each of past 24 hours.After 24 hours, will need to merge again by: insert overwrite …. select <column list> from table
• 9.
  Workflow to Hive(phase 2)Continuous log collection via ChukwaGeneric and continuous parse/merge/load to ‘real-time’ Hive warehousemerge at hourly boundary and load to public Hive warehouse. SLA is 2 Hr on merged data.Daily/Hourly job: For summary.For publishing data to BI for reporting.