Low Latency “OLAP” with HBase - HBaseCon 2012

Low Latency “OLAP” with HBase
Cosmin Lehene | Adobe

© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

What we needed … and built

 OLAP Semantics
 Low Latency Ingestion
 High Throughput
 Real-time Query API

 Not hardcoded to web analytics or x-, y-, z-
analytics, but extensible
© 2012 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 2

Building Blocks

 Dimensions, Metrics
 Aggregations
 Roll-up, drill-down, slicing and dicing, sorting


OLAP 101 – Queries example

Date Countr City OS Browser Sale
y
2012-05-21 USA NY Windows FF 0.0


2012-05-22 USA SF OSX Chrome 25.0

2012-05-22 Canada Ontario Linux Chrome 0.0

2012-05-23 USA Chicago OSX Safari 15.0

5 visits, 2 4 cities: 3 OS-es 3 browsers 50.0
3 days countries NY: 2 Win: 2 FF: 2 3 sales
USA: 4 SF: 1 OSX: 2 Chrome:2
Canada: 1


OLAP 101 – Queries example

 Rolling up to country level: Country visits sales
SELECT COUNT(visits), SUM(sales)
USA 4 $50
GROUP BY country
Canada 1 0

 “Slicing” by browser Country visits sales
SELECT COUNT(visits), SUM(sales) USA 2 $10
GROUP BY country
Canada 0 0
HAVING browser = “FF”

 Top browsers by sales Browser sales visits
SELECT SUM(sales), COUNT(visits) Chrome $25 2
GROUP BY browser
Safari $15 1
ORDER BY sales
FF $10 2


OLAP – Runtime Aggregation vs. Pre-aggregation

 Aggregate at runtime  Pre-aggregate
 Most flexible  Fast
 Fast – scatter gather  Efficient – O(1)
 Space efficient  High throughput
 But  But
 I/O, CPU intensive  More effort to process (latency)
 slow for larger data  Combinatorial explosion (space)
 low throughput  No flexibility


Pre-aggregation

 Data needs to be summarized
 Can’t visualize 1B data points (no, not even with Retina display)
 Difficult to comprehend correlations among more than 3 dimensions

 Not all dimension groups are relevant
 Index on a needed basis (view selection problem)

 Runtime aggregation == TeraSort for every query?
 Pre-aggregate to reduce cardinality


SaasBase

 We tune both
 pre-aggregation level vs. runtime post-aggregation
 (ingestion speed + space ) vs. (query speed)

 Think materialized views from RDBMS


SaasBase Domain Model Mapping


SaasBase - Domain Model Mapping


SaasBase - Ingestion, Processing, Indexing, Querying


Ingestion


Ingestion throughput vs. latency

 Historical data (large batches)
 Optimize for throughput
 Increments (latest data, smaller)
 Optimize for latency


Large, granular input strategies

 Slow listing in HDFS
 Archive processed files

 Filtering input
 FileDateFilter (log name patterns: log-YYYY-MM-dd-HH.log)
 TableInputFormat start/stop row
 File Index in HBase (track processed/new files)

 Map tasks overhead - stitching input splits
 400K files => 400K map tasks => overhead, slow reduce copy
 CombineFileInputFormat – 2GB-splits => 500 splits for 1TB
 FixedMappersTableInputFormat (e.g. 5-region splits)

Ingestion – Bulk Import

 HFileOutputFormat (HFOF)
 100s X faster than HBase API
 No need to recover from failed jobs
 No unnecessary load on machines

* No shuffle - global reduce order
required!
 e.g. first reduce key needs to be in the
first region, last one in the last region
 Watch for uneven partitions


HFOF – FileSizeDatePartitioner

 1 partition(reduce) / day for initial import
 Uneven reduce (partitions) due to data growth over time
 Reduce k: 2010-12-04 = 500MB
 Reduce n: 2012-05-22 = 5GB => slow and will result in a 5GB region

 Balance reduce buckets based on input file sizes and the reduce key
 Generate sub-partitions based on predefined size (e.g. 1GB)


Processing


Processing

 Processing involves reading the Input (files, tables, events), pre-
aggregating it (reducing cardinality) and generating tables that can be
queried in real-time
 1 year: 1B events => 100B data points indexed
 Query => scan 365 data points (e.g. daily page views)

 Processing could be either MR or real-time (e.g. Storm)


Processing for OLAP semantics

 GROUP BY (process, query)
 COUNT, SUM, AVG, etc. (process, query)
 SORT (process, query)
 HAVING (mostly query, can define pre-process constraints)


SaasBase vs. SQL Views Comparison


reports.json entities definition


Processing Performance

 read, map, partition, combine, copy, sort, reduce, write

 Read:
 Scan.setCaching() (I/O ~ buffer)
 Scan.setBatching() (avoid timeouts for abnormal input, e.g. 1M hits/visit)
 Even region distribution across cluster (distributes CPU, I/O)
 Map:
 No unnecessary transformations: Bytes.toString(bytes) + Bytes.toBytes(string)
(CPU)
 Avoid GC : new X() (CPU, Memory)
 Avoid system calls (context switching)
 Stripping unnecessary data (I/O)


Processing Performance

 Hot (in memory) vs. Cold (on disk, on network) data
 Minimize I/O from disk/network

 Single shot MR job: SuperProcessor
 Emit all groups from one map() call

 Incremental processing
 Data format YYYY-MM-DD prefixed rowkey (HH:mm for more granularity)


Indexing


HBase natural order: hierarchical representation


Indexing - Why

 Example: top 10 cities
 ~50K [country, city] combinations per day
 Top 10 cities for 1 year =>
 365 (days) X 50K ~=15M data points scanned
 If you add gender => 30M
 If you add Device, OS, Browser …

 Might compress well, but think about the environment
 How much energy would you spend for just top 10 cities?

* Image from: http://my.neutralexistence.com/images/Green-Earth.jpg


Indexing with HBase “10” < “2”

GROUP BY year, month, country, city ORDER BY visits DESC LIMIT 10

 Lexicographic sorting

2012/05/USA/0000000000/
2012/05/USA/4294961296/San Francisco = 1000 visits*
2012/05/USA/4294961396/New York = 900 visits*
. . .
2012/05/USA/9999999999/

 scan “t” startrow => “2012/05/USA/”, limit => 10

* Padding numbers for lexicographic sorting:
1000 -> Long.MAX_VALUE – 1000 = 4294961296


Query Engine

 Always reads indexed, compact data
 Query parsing
 Scan strategy
 Single vs. multiple scans
 Start/stop rows (prefixes, index positions, etc.)
 Index selection (volatile indexes with incremental processing)
 Deserialization
 Post-aggregation, sorting, fuzzy-sorting etc.
 Paging
 Custom dimension/metric class loading


Conclusions

 OLAP semantics on a simple data model
 Data as first class citizen
 Domain Specific “Language” for Dimensions, Metrics, Aggregations
 Tunable performance, resource allocation
 Framework for vertical analytics systems


Thank you!
Cosmin Lehene @clehene

http://hstack.org
Credits:
Andrei Dragomir
Adrian Muraru
Andrei Dulvac
Raluca Podiuc
Tudor Scurtu
Bogdan Dragu
Bogdan Drutu


OLAP 101 - Rollup

Countr Visits Sale
y
USA 4 $50

Canada 1 $0

 Rollup: SELECT COUNT(visits), SUM(sales) GROUP BY country


OLAP 101 - Slicing

y


2012-03-03 USA S OSX Chrome 25.0

2012-03-03 Canada Ontario Linux Chrome 0.0

2012-03-04 USA Chicago OSX Safari 15.0

5 visits, 2 4 cities: 3 OS-es 3 browsers 50.0
3 days countries NY: 2 Win: 2 FF: 2 3 sales
USA: 4 SF: 1 OSX: 2 Chrome:2
Canada: 1
 Filter or Segment or Slice (WHERE or HAVING)


OLAP 101 – Sorting, TOP n

y
Chrome $25

Safari $15

Firefox $10

 SELECT SUM(sales) as total GROUP BY browser ORDER BY total


Low Latency “OLAP” with HBase - HBaseCon 2012

Recommended

Recommended

More Related Content

What's hot

What's hot (13)

Viewers also liked

Viewers also liked (20)

Similar to Low Latency “OLAP” with HBase - HBaseCon 2012

Similar to Low Latency “OLAP” with HBase - HBaseCon 2012 (20)

Recently uploaded

Recently uploaded (20)

Low Latency “OLAP” with HBase - HBaseCon 2012

Editor's Notes