HBaseCon 2012 | Low Latency OLAP with HBase - Cosmin Lehene, Adobe
 

HBaseCon 2012 | Low Latency OLAP with HBase - Cosmin Lehene, Adobe

on

  • 4,322 views

Adobe Systems uses “SaasBase Analytics” to incrementally process large heterogeneous data sets into pre-aggregated, indexed views, stored in HBase to be queried in real- time. Our goal was to ...

Adobe Systems uses “SaasBase Analytics” to incrementally process large heterogeneous data sets into pre-aggregated, indexed views, stored in HBase to be queried in real- time. Our goal was to process new data in real- time (currently minutes) and have it ready for a large number of concurrent queries that execute in milliseconds. This set our problem apart from what is traditionally solved with Hive or PIG. In this talk I’ll describe the design and the strategies (and hacks) we used to achieve low latency and scalability, from theoretical model to the entire process of ETL to warehousing and queries.

Statistics

Views

Total Views
4,322
Views on SlideShare
3,689
Embed Views
633

Actions

Likes
9
Downloads
212
Comments
0

7 Embeds 633

http://www.cloudera.com 578
http://blog.cloudera.com 30
https://twitter.com 12
http://marilson.pbworks.com 10
http://us-w1.rockmelt.com 1
http://test.cloudera.com 1
https://si0.twimg.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • How many HBase users?
  • Data as first class citizen
  • Check contrast on projector
  • Just like speedvs space in general CS/algoQueries always hit indexes
  • Dimensions – readtransformserializedeserialize data attributesMetrics – read/transform/aggregate/serializeConstraints: ingestion filteringReport: instrument dimensions groups + metrics with aggregations, sorting
  • QUERY ENGINE -> INDEX(always realtime)
  • Initial import/process and NEW reports (not covered) on historical data
  • 18K regions, upgrade to 0.92
  • DiagramHARD TO DIGEST (TOO MUCH INFO, TOO CONDENSED)
  • Process = aggregate,generate indexes (natural)Query = uses indexes, can do extra aggregation
  • LEFT: report definition, NOT a QUERYLIKE A VIEW - CREATED - THEN QUERIED
  • Inconsistent
  • Rowkey =dimensions group -> metrics (right)
  • GO BACK to EXPLAIN
  • >100K/sec/threadREALTIME
  • Data analysts work with familiar concepts

HBaseCon 2012 | Low Latency OLAP with HBase - Cosmin Lehene, Adobe HBaseCon 2012 | Low Latency OLAP with HBase - Cosmin Lehene, Adobe Presentation Transcript

  • Low Latency “OLAP” with HBase Cosmin Lehene | Adobe© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
  • What we needed … and built  OLAP Semantics  Low Latency Ingestion  High Throughput  Real-time Query API  Not hardcoded to web analytics or x-, y-, z- analytics, but extensible© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 2
  • Building Blocks  Dimensions, Metrics  Aggregations  Roll-up, drill-down, slicing and dicing, sorting© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 3
  • OLAP 101 – Queries example Date Countr City OS Browser Sale y 2012-05-21 USA NY Windows FF 0.0 2012-05-21 USA NY Windows FF 10.0 2012-05-22 USA SF OSX Chrome 25.0 2012-05-22 Canada Ontario Linux Chrome 0.0 2012-05-23 USA Chicago OSX Safari 15.0 5 visits, 2 4 cities: 3 OS-es 3 browsers 50.0 3 days countries NY: 2 Win: 2 FF: 2 3 sales USA: 4 SF: 1 OSX: 2 Chrome:2 Canada: 1© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 4
  • OLAP 101 – Queries example  Rolling up to country level: Country visits sales SELECT COUNT(visits), SUM(sales) USA 4 $50 GROUP BY country Canada 1 0  “Slicing” by browser Country visits sales SELECT COUNT(visits), SUM(sales) USA 2 $10 GROUP BY country Canada 0 0 HAVING browser = “FF”  Top browsers by sales Browser sales visits SELECT SUM(sales), COUNT(visits) Chrome $25 2 GROUP BY browser Safari $15 1 ORDER BY sales FF $10 2© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 5
  • OLAP – Runtime Aggregation vs. Pre-aggregation  Aggregate at runtime  Pre-aggregate  Most flexible  Fast  Fast – scatter gather  Efficient – O(1)  Space efficient  High throughput  But  But  I/O, CPU intensive  More effort to process (latency)  slow for larger data  Combinatorial explosion (space)  low throughput  No flexibility© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 6
  • Pre-aggregation  Data needs to be summarized  Can’t visualize 1B data points (no, not even with Retina display)  Difficult to comprehend correlations among more than 3 dimensions  Not all dimension groups are relevant  Index on a needed basis (view selection problem)  Runtime aggregation == TeraSort for every query?  Pre-aggregate to reduce cardinality© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 7
  • SaasBase  We tune both  pre-aggregation level vs. runtime post-aggregation  (ingestion speed + space ) vs. (query speed)  Think materialized views from RDBMS© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 8
  • SaasBase Domain Model Mapping© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 9
  • SaasBase - Domain Model Mapping© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 10
  • SaasBase - Ingestion, Processing, Indexing, Querying© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 11
  • SaasBase - Ingestion, Processing, Indexing, Querying© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 12
  • Ingestion© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 13
  • Ingestion throughput vs. latency  Historical data (large batches)  Optimize for throughput  Increments (latest data, smaller)  Optimize for latency© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 14
  • Large, granular input strategies  Slow listing in HDFS  Archive processed files  Filtering input  FileDateFilter (log name patterns: log-YYYY-MM-dd-HH.log)  TableInputFormat start/stop row  File Index in HBase (track processed/new files)  Map tasks overhead - stitching input splits  400K files => 400K map tasks => overhead, slow reduce copy  CombineFileInputFormat – 2GB-splits => 500 splits for 1TB  FixedMappersTableInputFormat (e.g. 5-region splits)© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 15
  • Ingestion – Bulk Import  HFileOutputFormat (HFOF)  100s X faster than HBase API  No need to recover from failed jobs  No unnecessary load on machines * No shuffle - global reduce order required!  e.g. first reduce key needs to be in the first region, last one in the last region  Watch for uneven partitions© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 16
  • HFOF – FileSizeDatePartitioner  1 partition(reduce) / day for initial import  Uneven reduce (partitions) due to data growth over time  Reduce k: 2010-12-04 = 500MB  Reduce n: 2012-05-22 = 5GB => slow and will result in a 5GB region  Balance reduce buckets based on input file sizes and the reduce key  Generate sub-partitions based on predefined size (e.g. 1GB)© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 17
  • Processing© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 18
  • Processing  Processing involves reading the Input (files, tables, events), pre- aggregating it (reducing cardinality) and generating tables that can be queried in real-time  1 year: 1B events => 100B data points indexed  Query => scan 365 data points (e.g. daily page views)  Processing could be either MR or real-time (e.g. Storm)© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 19
  • Processing for OLAP semantics  GROUP BY (process, query)  COUNT, SUM, AVG, etc. (process, query)  SORT (process, query)  HAVING (mostly query, can define pre-process constraints)© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 20
  • SaasBase vs. SQL Views Comparison© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 21
  • reports.json entities definition© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 22
  • Processing Performance  read, map, partition, combine, copy, sort, reduce, write  Read:  Scan.setCaching() (I/O ~ buffer)  Scan.setBatching() (avoid timeouts for abnormal input, e.g. 1M hits/visit)  Even region distribution across cluster (distributes CPU, I/O)  Map:  No unnecessary transformations: Bytes.toString(bytes) + Bytes.toBytes(string) (CPU)  Avoid GC : new X() (CPU, Memory)  Avoid system calls (context switching)  Stripping unnecessary data (I/O)© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 23
  • Processing Performance  Hot (in memory) vs. Cold (on disk, on network) data  Minimize I/O from disk/network  Single shot MR job: SuperProcessor  Emit all groups from one map() call  Incremental processing  Data format YYYY-MM-DD prefixed rowkey (HH:mm for more granularity)© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 24
  • Indexing© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 25
  • HBase natural order: hierarchical representation© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 26
  • Indexing - Why  Example: top 10 cities  ~50K [country, city] combinations per day  Top 10 cities for 1 year =>  365 (days) X 50K ~=15M data points scanned  If you add gender => 30M  If you add Device, OS, Browser …  Might compress well, but think about the environment  How much energy would you spend for just top 10 cities? * Image from: http://my.neutralexistence.com/images/Green-Earth.jpg© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 27
  • Indexing with HBase “10” < “2” GROUP BY year, month, country, city ORDER BY visits DESC LIMIT 10  Lexicographic sorting 2012/05/USA/0000000000/ 2012/05/USA/4294961296/San Francisco = 1000 visits* 2012/05/USA/4294961396/New York = 900 visits* . . . 2012/05/USA/9999999999/  scan “t” startrow => “2012/05/USA/”, limit => 10 * Padding numbers for lexicographic sorting: 1000 -> Long.MAX_VALUE – 1000 = 4294961296© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 28
  • Query Engine  Always reads indexed, compact data  Query parsing  Scan strategy  Single vs. multiple scans  Start/stop rows (prefixes, index positions, etc.)  Index selection (volatile indexes with incremental processing)  Deserialization  Post-aggregation, sorting, fuzzy-sorting etc.  Paging  Custom dimension/metric class loading© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 29
  • Conclusions  OLAP semantics on a simple data model  Data as first class citizen  Domain Specific “Language” for Dimensions, Metrics, Aggregations  Tunable performance, resource allocation  Framework for vertical analytics systems© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 30
  • Thank you! Cosmin Lehene @clehene http://hstack.org Credits: Andrei Dragomir Adrian Muraru Andrei Dulvac Raluca Podiuc Tudor Scurtu Bogdan Dragu Bogdan Drutu© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 31
  • © 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
  • OLAP 101 - Rollup Countr Visits Sale y USA 4 $50 Canada 1 $0  Rollup: SELECT COUNT(visits), SUM(sales) GROUP BY country© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 33
  • OLAP 101 - Slicing Date Countr City OS Browser Sale y 2012-03-02 USA NY Windows FF 0.0 2012-03-02 USA NY Windows FF 10.0 2012-03-03 USA S OSX Chrome 25.0 2012-03-03 Canada Ontario Linux Chrome 0.0 2012-03-04 USA Chicago OSX Safari 15.0 5 visits, 2 4 cities: 3 OS-es 3 browsers 50.0 3 days countries NY: 2 Win: 2 FF: 2 3 sales USA: 4 SF: 1 OSX: 2 Chrome:2 Canada: 1  Filter or Segment or Slice (WHERE or HAVING)© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 34
  • OLAP 101 – Sorting, TOP n Date Countr City OS Browser Sale y Chrome $25 Safari $15 Firefox $10  SELECT SUM(sales) as total GROUP BY browser ORDER BY total© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 35