Low Latency OLAP with Hadoop and HBase

26,715 views
26,260 views

Published on

We use "SaasBase Analytics" to incrementally process large heterogeneous data sets into pre-aggregated, indexed views, stored in HBase to be queried in realtime. The requirement we started from was to get large amounts of data available in near realtime (minutes) to large amounts of users for large amounts of (different) queries that take milliseconds to execute. This set our problem apart from classical solutions such as Hive and PIG. In this talk I`ll go through the design of the solution and the strategies (and hacks) to achieve low latency and scalability from theoretical model to the entire process of ETL to warehousing and queries.

Published in: Technology
0 Comments
49 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
26,715
On SlideShare
0
From Embeds
0
Number of Embeds
226
Actions
Shares
0
Downloads
0
Comments
0
Likes
49
Embeds 0
No embeds

No notes for slide

Low Latency OLAP with Hadoop and HBase

  1. 1. Low-Latency “OLAP” with Hadoop and HBase Andrei Dragomir | Software Engineer© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
  2. 2. Synopsis §  What are we trying to solve §  Description of our system §  How it works §  Minimizing Latency© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 2
  3. 3. In a nutshell Low-latency OLAP system Hadoop DFS to store input data (ie log files, or HBase tables) The processing loop of the system takes a cube description and processes it (pre-aggregations) using Hadoop Map/Reduce. The output is written to a statistics HBase table. To get the data, users query a server, which scans the HBase table, applying the filters, roll-ups or drill-downs, and returning the result.© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 3
  4. 4. In a nutshell Low-latency OLAP system Hadoop DFS to store input data (ie log files, or HBase tables) The processing loop of the system takes a cube description and processes it (pre-aggregations) using Hadoop Map/Reduce. The output is written to a statistics HBase table. To get the data, users query a server, which scans the HBase table, applying the filters, roll-ups or drill-downs, and returning the result.© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 4
  5. 5. In a nutshell Low-latency OLAP system Hadoop DFS to store input data (ie log files, or HBase tables) The processing loop of the system takes a cube description and processes it (pre-aggregations) using Hadoop Map/Reduce. The output is written to a statistics HBase table. To get the data, users query a server, which scans the HBase table, applying the filters, roll-ups or drill-downs, and returning the result.© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 5
  6. 6. In a nutshell Low-latency OLAP system Hadoop DFS to store input data (ie log files, or HBase tables) The processing loop of the system takes a cube description and processes it (pre-aggregations) using Hadoop Map/Reduce. The output is written to a statistics HBase table. To get the data, users query a server, which scans the HBase table, applying the filters, roll-ups or drill-downs, and returning the result.© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 6
  7. 7. In a nutshell Low-latency OLAP system Hadoop DFS to store input data (ie log files, or HBase tables) The processing loop of the system takes a cube description and processes it (pre-aggregations) using Hadoop Map/Reduce. The output is written to a statistics HBase table. To get the data, users query a server, which scans the HBase table, applying the filters, roll-ups or drill-downs, and returning the result.© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 7
  8. 8. In a nutshell Low-latency OLAP system Hadoop DFS to store input data (ie log files, or HBase tables) The processing loop of the system takes a cube description and processes it (pre-aggregations) using Hadoop Map/Reduce. The output is written to a statistics HBase table. To get the data, users query a server, which scans the HBase table, applying the filters, roll-ups or drill-downs, and returning the result.© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 8
  9. 9. Vocabulary Date Country City OS Browser Sales 2012-05-12 USA NY Win FF $ 0.0 2012-05-12 USA NY Win FF $ 10.0 2012-05-13 USA SF OSX Chrome $ 25.0 2012-05-13 Canada Ontario Linux Chrome $ 0.0 2012-05-14 USA Chicago OSX Safari $ 15.0 ... ... ... ... ... ... 5 Visits 2 Countries 4 Cities: 3 OS: 3 Browser: $50.0 3 Days USA: 4 NY: 2 Win: 2 FF: 2 3 sales Canada: 1 SF: 1 OSX: 2 Chrome: 2© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 9
  10. 10. Vocabulary Date Country City OS Browser Sales 2012-05-12 USA NY Win FF $ 0.0 2012-05-12 USA NY Win FF $ 10.0 2012-05-13 USA SF OSX Chrome $ 25.0 2012-05-13 Canada Ontario Linux Chrome $ 0.0 2012-05-14 USA Chicago OSX Safari $ 15.0 ... ... ... ... ... ... 5 Visits 2 Countries 4 Cities: 3 OS: 3 Browser: $50.0 3 Days USA: 4 NY: 2 Win: 2 FF: 2 3 sales Canada: 1 SF: 1 OSX: 2 Chrome: 2 §  We want to get (mostly) numeric data: metrics §  These metrics have a set of labels (dimensions) §  We want to view the metrics by any combination of dimensions© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 10
  11. 11. Vocabulary Date Country City OS Browser Sales 2012-05-12 USA NY Win FF $ 0.0 2012-05-12 USA NY Win FF $ 10.0 2012-05-13 USA SF OSX Chrome $ 25.0 2012-05-13 Canada Ontario Linux Chrome $ 0.0 2012-05-14 USA Chicago OSX Safari $ 15.0 ... ... ... ... ... ... 5 Visits 2 Countries 4 Cities: 3 OS: 3 Browser: $50.0 3 Days USA: 4 NY: 2 Win: 2 FF: 2 3 sales Canada: 1 SF: 1 OSX: 2 Chrome: 2 §  We want to get (mostly) numeric data: metrics §  These metrics have a set of labels (dimensions) §  We want to view the metrics by any combination of dimensions© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 11
  12. 12. Vocabulary Date Country City OS Browser Sales 2012-05-12 USA NY Win FF $ 0.0 2012-05-12 USA NY Win FF $ 10.0 2012-05-13 USA SF OSX Chrome $ 25.0 2012-05-13 Canada Ontario Linux Chrome $ 0.0 2012-05-14 USA Chicago OSX Safari $ 15.0 ... ... ... ... ... ... 5 Visits 2 Countries 4 Cities: 3 OS: 3 Browser: $50.0 3 Days USA: 4 NY: 2 Win: 2 FF: 2 3 sales Canada: 1 SF: 1 OSX: 2 Chrome: 2 §  We want to get (mostly) numeric data: metrics §  These metrics have a set of labels (dimensions) §  We want to view the metrics by any combination of dimensions© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 12
  13. 13. OLAP Queries §  Rolling up to country level Country visits sales SELECT  COUNT(visits),  SUM(sales)   USA 4 $50 GROUP  BY  country   Canada 1 0 §  “Slicing” by browser Country visits sales SELECT  COUNT(visits),  SUM(sales)   USA 2 $10 GROUP  BY  country   Canada 0 0 HAVING  browser  =  “FF”   Browser sales visits §  Top browsers by sales Chrome $25 2 SELECT  SUM(sales),  COUNT(visits)     GROUP  BY  browser     Safari $15 1 ORDER  BY  sales   FF $10 2© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 13
  14. 14. Looking inside – physical diagram© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
  15. 15. Looking inside – logical diagram© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
  16. 16. Simplifying assumptions: pre-aggregation §  In most cases... §  Data needs to be summarized – hard to draw 1B data points §  You don’t need to look at all dimensions at the same time – hard to correlate §  Not all queries are used with the same frequency© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 16
  17. 17. A timeless CS problem: Optimize... Time Space §  Pre-aggregation §  Runtime §  Fast aggregation §  Flexible §  Efficient reads – O(1) §  Inflexible §  I/O, CPU intensive §  Processing latency §  Slow– always need §  Combinatorial to look at all the Explosion data §  Low throughput© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 17
  18. 18. Solution ? §  Just do both ! §  Can tune: pre-aggregate more, or rely on runtime aggregation §  Ingestion + process speed vs Query speed §  Works just like normal queries + materialized views© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 18
  19. 19. Solution ? §  Process: pre-aggregate all the report definitions, create an indexed HBase table. §  Query: use the indexes to get the data fast. Perform extra aggregation, filtering if needed at runtime. §  Platform strengths §  Parallelism in M/R §  Fast access and natural key ordering in HBase© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 19
  20. 20. Minimal HBase details Row   Columns...   §  Data is stored in tables Key   u1   v1   v2   v3   §  Each row has a key, u2   v   X   ...   and any number of columns (long & wide) u3   v   x   ...   u4   x   v2   ...   §  Ordered by row keys: u5   ...   v3   ...   clustered indexes u6   ...   v5   ...   built-in u7   ...   ...   ...   §  Sparse tables. NULLs u8   ...   ...   ...   are free.© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 20
  21. 21. Minimal HBase details Row   Column §  Operations use row key   ...   key: get(), put()   aaa   v1   aab   v2   §  Can scan a range of ←   rows:[start,  end)   aac   v3   ←   aad   v4   §  We can use the row ←   aae   v5   key as a built-in ←   aaf   v6   indexing aba   ...   mechanism abb   ...  © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 21
  22. 22. SaasBase vs. SQL Views Comparison© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 22
  23. 23. Reports configuration §  List of Dimensions (with custom classes, arguments, etc) §  List of Metrics (with custom classes, arguments, etc) §  List of Reports, each containing §  Dimensions (subset) §  Metrics (subset) §  Sorting, etc §  The reports configuration is used in the entire system: import, process, query© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 23
  24. 24. Solution ? Date Countr Cit Sale y y s 2012-05-1 USA NY 3 2 2012-05-1 USA NY 10 2 2012-05-1 USA SF 25 3 2012-05-1 CAN ON 0 3 2012-05-1 USA CH 15 4© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 24
  25. 25. Solution ? Date Countr Cit Sale y y s 2012-05-1 USA NY 3 2 2012-05-1 USA NY 10 2 2012-05-1 USA SF 25 3 2012-05-1 CAN ON 0 3 2012-05-1 USA visits_by_city:  {   CH 15    dimensions:  [country,  city],     4    metrics:  [visits]   },     daily_sales:  {      dimensions:  [year,  month,  day,   country],        metrics:  [sales]   }  © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 25
  26. 26. Solution ? Date Countr Cit Sale y y s 2012-05-1 USA NY 3 2 2012-05-1 USA NY 10 2      Statistics  HBASE  Output  Table            ROWKEY                        VALUE   2012-05-1 USA SF 25 3 daily_sales/2012+05+12+USA        $13     daily_sales/2012+05+13+CAN        $0   2012-05-1 CAN ON 0 daily_sales/2012+05+13+USA        $25   3 daily_sales/2012+05+14+USA        $15   2012-05-1 USA visits_by_city:  {   CH 15 visits_by_city/CAN+ON                  1      dimensions:  [country,  city],     4    metrics:  [visits]   visits_by_city/USA+CH                  1   },     daily_sales:  {   visits_by_city/USA+NY                  2      dimensions:  [year,  month,  day,   visits_by_city/USA+SF                  1   country],        metrics:  [sales]   }  © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 26
  27. 27. HBase natural order: hierarchical filtering© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 27
  28. 28. Sorting §  Add the metrics that you want to sort by to the row key... §  In a way that preserves the ordering© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 28
  29. 29. Sorting §  Add the metrics that you want to sort by to the row key... §  In a way that preserves the ordering §  ORDER  BY  metric  DESC  ==  Long.MAX_VALUE  –  metric   2012+05+USA+0000000000+   2012+05+USA+4294961296+SF  =  1000  visits   2012+05+USA+4294961396+NY  =  900  visits   .  .  .       2012+05+USA+9999999999+  © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 29
  30. 30. Minimizing Latency© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
  31. 31. Minimizing Import Latency §  Only import the minimal set of changes §  Map/Reduce input filters: §  c.a.s.a.i.FileCache – checks if file already processed §  c.a.s.a.i.FileDateFilter – checks if a date in the file path is against a specified interval §  process files from 3 days ago up until now, once §  HBase scan (from import table) start and stop row §  Minimize map-task overhead – stitch input splits© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 31
  32. 32. Minimizing Import Latency §  Minimize map-task overhead – stitch input splits §  for 400000 files -> 400000 Map Tasks, slow reduce-copy phase §  o.a.h.m.i.CombineFileInputFormat – make 2GB splits §  c.a.s.a.m.i.FixedMappersTableInputFormat – stitches multiple HBase regions in the same map task© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 32
  33. 33. Minimizing Import Latency §  If warehousing in HBase, use o.a.h.h.m.HFileOutputFormat   §  ~ 100 times faster than using the API §  No shuffle step! you must use a global order partitioner §  Problem: data grows over time §  Solution: estimate output partitions based on input data size, and make partitions (regions) using this heuristic §  c.a.s.a.m.FileSizeDatePartitioner – inject input files size and dates and rebalance regions based on these, and a fixed size (2GB)© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 33
  34. 34. Minimizing Processing Latency §  Processing involves reading the input (files, tables, events), pre-aggregating it (reducing cardinality) and generating tables that can be queried in real-time §  Processing does GROUP BY, COUNT/SUM/AVG, ORDER BY §  Minimize each M/R step: read, map, partition, combine, copy, sort, reduce, write §  Read §  Filter input data (incremental processing) – differentiate between OPEN and CLOSED data §  HBase Scan options: caching, batching, etc §  Ensure HBase table regions are distributed in the cluster© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 34
  35. 35. Minimizing Processing Latency §  c.a.s.a.m.j.SuperProcessor   §  One shot M/R job: for all data, for all reports, emit the pre-aggregated values in 1 map() call §  no allocations §  Simple and tight §  no system calls (avoid context switches) §  no String <> byte[] transformations §  minimize Map > Combine > Reduce I/O §  NO ALLOCATIONS© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 35
  36. 36. Minimizing Query Latency §  c.a.s.a.m.t.ReportHandler   §  Simple Thrift server §  Data is already processed and pre-aggregated §  Query time does HAVING/WHERE (filters), extra GROUP BY (roll-ups) §  Calculate an optimal set of HBase scan()s   §  single / multiple scans §  start / stop rows (prefixes, index positions) §  Perform extra roll-ups / sorting §  Assorted sundries: paging, display-time ser/des, etc© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 36
  37. 37. Flexible §  Report configuration – the core of the system §  c.a.s.a.e.Dimension, c.a.s.a.e.Metric   §  Can override ser/des, aggregate functions (for metrics) §  Can override behavior (only add 1 if X...) §  Emergent patterns are rolled-up in the reporting core §  The entire processing loop can be written outside of M/R for realtime §  Storm ? §  Applied in 4 use-cases right now, easy to extend §  Some programming required© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 37
  38. 38. Thank you adragomi@adobe.com / @adragomir http://hstack.org Our team: Adrian Muraru, Andrei Dulvac, Bogdan Dragu, Bogdan Drutu, Cosmin Lehene, Raluca Podiuc, Tudor Scurtu© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
  39. 39. © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
  40. 40. Break!Break takes place in the Community Showcase (Hall 2)Sessions will resume at 3:35pm Page 40

×