• Save
HBaseCon 2013: ETL for Apache HBase
 

HBaseCon 2013: ETL for Apache HBase

on

  • 2,363 views

Presented by: Manoj Khanwalkar (Experian) and Govind Asawa (Experian)

Presented by: Manoj Khanwalkar (Experian) and Govind Asawa (Experian)

Statistics

Views

Total Views
2,363
Views on SlideShare
2,339
Embed Views
24

Actions

Likes
4
Downloads
0
Comments
0

3 Embeds 24

http://www.linkedin.com 19
https://www.linkedin.com 4
http://dschool.co 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

HBaseCon 2013: ETL for Apache HBase HBaseCon 2013: ETL for Apache HBase Presentation Transcript

  • © 2013 Experian Limited. All rights reserved. HBaseCon 2013 Application Track – Case Study Experian Marketing Services ETL for HBase
  • © 2013 Experian Limited. All rights reserved. Manoj Khanwalkar Chief Architect Experian Marketing Services, New York Govind Asawa Big Data Architect Experian Marketing Services, New York Who We Are
  • © 2013 Experian Limited. All rights reserved. 1. About Experian Marketing Services 2. Why HBase 3. Why custom ETL 4. ETL solution features 5. Performance 6. Case Study 7. Conclusion Agenda
  • © 2013 Experian Limited. All rights reserved. Experian Marketing Services 1 billion+ messages daily 2000+ Institutional clients 9 regions, 24/ 7 500+ Tetabytes of data 200+ big queries 2000+ data export jobs Email and social digital marketing messages. 100% surge in volume during peak season Across all verticals Platforms operating globally Client needs 1 to 7 years of marketing data depending on verticals Complicated queries on 200+ million records 400+ columns for segmentation Client needs daily incremental activity data
  • © 2013 Experian Limited. All rights reserved. • Traditional RDBMS based solution is very challenging and cost prohibitive for the scale of operations • In SaaS based multi-tenancy model we require schema flexibility to support thousands of clients with their individual requirements • In majority of cases key based lookups can satisfy data extraction requirements (including range scans and filters) which is well supported by HBase • Automatic sharding and horizontally scalable • HBase provides a Java API which can be integrated with Experian’s other systems. 5 Why HBase
  • © 2013 Experian Limited. All rights reserved. 6 Why develop an Integrator toolkit? Connectivity Environment Cost • Ability to ingest and read data from HBase and MongoDB • Connectors for cloud computing • Support for REST and other industry standard API’s • Supports SaaS Model • Dynamically handles data input changes (# of fields & new fields) • Licensing • Integrate with other systems seamlessly thus improving time to market • Resources required to develop, administer and maintain solution • Major ETL vendors do not support HBase • ETL solution needs extensive development if data structure changes which negates advantages offered by No-SQL solution
  • © 2013 Experian Limited. All rights reserved. 7 Integrator Architecture DataIngester TargetSystems Third Party JMS Database SourceSystems Connectors CSV Reader Processor Event Listener Message Broker File Watcher Parser Factory Key Generator Parser Loader RDBMS Loader HBase Loader Container Metadata Analyzer Loader Aggregator RDBMS HBase Extractor Query Output Aggregate Aware Stamping Transform SaaS JMS Files RDBMS HBase RDBMS HBase MongoDB
  • © 2013 Experian Limited. All rights reserved. 8 Extractor Architecture HDFS Integrator Send Data Click Data Bounce Data TXN Data Metadata Detailed data Aggregates HBase Web Server Reporting Analytics Extractor Query Optimizer
  • © 2013 Experian Limited. All rights reserved. Data ingestion from multiple sources • Flat files • NO-SQL • RDBMS (through JDBC) • SaaS (Salesforce etc.) • Messaging and any system providing events streaming Ability to de-normalize fact table while ingesting data • # of lookup tables can be configured Near real time generation of aggregate table • # of aggregate tables can be configured • HBase counters are used to keep aggregated sum/count • Concurrently aggregates can be populated in RDBMS of choice 9 Integrator & Extractor
  • © 2013 Experian Limited. All rights reserved. Transformation of column value to another value • Add column by transformation • Drop columns from input stream if no persistence is required Data filter capability • Drop record while ingesting base table • Drop record while aggregation Aggregate aware optimized query execution • Query Performance: Analyze column requested by user in query and determine based on count table with minimum record which can satisfy this requirement. • Transparent: No user intervention or knowledge of schema is required • Optimizer: Conceptually similar to RDBMS query plan optimizer. Concept extended to No-SQL databases • Metadata Management: Integrated metadata with ETL process can be used by variety of applications. 10 Integrator & Extractor
  • © 2013 Experian Limited. All rights reserved. Framework • Solution based on Spring as a light weight container and built a framework around it to standardize on the lifecycle of the process and to enable any arbitrary functionality to reside in the container by implementing a Service interface. • The container runs in a batch processing or daemon mode. • In the daemon mode , it uses the Java 7 File Watcher API to react to files placed in the specified directory for processing. Metadata catalogue • Metadata about all HBase table in which data ingested is stored • For each table primary key, columns and record counter is stored • HBase count is brute force scan and expensive API call. This can be avoided if metadata is published at the time of data ingestion • Avoid expensive queries which can bring cluster to its knees • Provide faster query performance 11 Integrator
  • © 2013 Experian Limited. All rights reserved. • We used a 20 node cluster in production; each node had 24 cores with a 10GigE network backbone. • We observed a throughput of 1.3 million records inserted in HBase per minute per node. • Framework allowed us to run ETL process on multiple machines thus providing horizontal scalability. • Most of our queries returned back in at most a few seconds. 12 Integrator – System Performance
  • © 2013 Experian Limited. All rights reserved. • Our experience shows that HBase offers a cost effective and performance solution for managing our data explosion while meeting the increasingly sophisticated analytical and reporting requirements of clients. • ETL framework allows us to leverage HBase and its features while improving developer productivity. • Framework gives us ability to roll out new functionality with minimum time to market. • Metadata catalogue optimizes query and improves cluster performance • Select count() on big HBase table take minutes/hours and can bring cluster to knees. Metadata of Integrator will give counts along with PrimaryKey, Columns in milliseconds 13 Conclusion
  • © 2013 Experian Limited. All rights reserved. • Case Study 14 Appendix
  • © 2013 Experian Limited. All rights reserved. 15 HBase Schema & Record Client ID Campaign ID Time logged User ID Orig domain Rcpt domain DS status Bounce cat IP Time queued 1 11 01/01/13 21 abc.com gmail.com success 192.168. 6.23 01/01/ 2013 2 12 01/02/13 31 xyz.com yahoo.com success bad- mailbox 112.168. 6.23 01/01/ 2013 Fact Table  send Send Record client_id,campaign_id,time_logged,user_id,orig_domain,rcpt_domain,dsn_status,bounce_cat,ip,Time_queued 1,11,01/01/2013,21,abc.com,gmail.com,success,192.168.6.23,01/01/2013
  • © 2013 Experian Limited. All rights reserved. 16 HBase Schema & Record Fact Table  activity Activity Record client_id,campaign_id,event_time,user_id,event_type 1,11,01/01/2013,21,open Client ID Campaign ID Time logged User ID Orig domain Rcpt domain IP city Event type IP Send time 1 11 01/01/13 21 abc.com gmail.com SFO Open 192.168. 6.23 01/01/ 2013 2 12 01/04/13 31 xyz.com yahoo.com LA Click 112.168. 6.23 01/01/ 2013
  • © 2013 Experian Limited. All rights reserved. 17 HBase Schema & Record Dimension Table  demographics Dimension Table  ip Client ID User ID Date Age Gender State City Zip Country Flag 1 11 01/01/13 21 M CA SFO 94087 USA Y 2 12 01/02/13 31 M CA SFO 94087 USA N IP Date Domain State Country City 192.168.6.23 01/01/2013 gmail.com CA USA SFO 112.168.6.23 01/02/2013 abc.edu NJ USA Newark
  • © 2013 Experian Limited. All rights reserved. 18 HBase Schema & Record Aggregate Table  A1 Aggregate Table  A2 Campaign ID Date Gender State Country Count 11 01/01/13 M CA USA 5023 12 01/02/13 M CA USA 74890 Client ID Date Gender State Country Count 1 01/01/13 M CA USA 742345 2 01/02/13 M CA USA 1023456
  • © 2013 Experian Limited. All rights reserved. 19 Metadata Metadata Table Table Name Primary Key Columns Count demographics Client_id,Campaig n_id,Date Client_id, Campaign_id, Date, Age, Gender,State,City,Country,Flag 10,000,000 A1 Campaign_id,Date Campaign_id,Date,Gender,State,Country,Count 1,000,000 A2 Client_id,Date Client_id,Date,Gender,State,Country,Count 500,000
  • © 2013 Experian Limited. All rights reserved. User Query without Extractor Aggregate Awareness • Select client_id,state,count from demographics • Query Execution: Query will be executed on demographics table which has 300,000,000 rows User Query with Extractor Aggregate Awareness • Select client_id,state,count from demographics • Query Execution: – Step 1: Extractor will parse list of columns from query – Step 2: Extractor will find list of tables which has these columns. In this example extractor will get 2 tables demographics and A1 which can satisfy this query request – Step 3: Extractor will decide which is best table to satisfy this query. This decision will be based on # of rows in table. In this example table A1 has less # of rows compared to table demographics so table A1 will be selected – Step 4: Query will be executed against table A1 with appropriate where clause specified by user 20 Query Execution in Action
  • © 2013 Experian Limited. All rights reserved. • Bloom filters were enabled at the row level to enable HBase to skip files efficiently. • We used HBase filters extensively in the Scans to filter out as much data as possible on the server side. • Defined Aggregates judiciously to be able to respond to queries without requiring HBase to resort to large file scans.. • We used a key concatenation that aligned to expected search patterns to enable HBase to provide an exact match or do efficient key range scans when a partial key was provided. 21 HBase Design Considerations
  • © 2013 Experian Limited. All rights reserved. • We didn’t use MapReduce in our ETL framework for following considerations – Overhead of MapReduce based processes. – Real-time access to data – Every file had different header metadata , in MapReduce we had difficulty in passing header metadata to each Map process – Avoid intermediate reads and writes to the HDFS file system. 22 HBase Design Considerations
  • © 2013 Experian Limited. All rights reserved. • We broke the Input and Output processing into separate threads and allocated a lot more threads for output processing to compensate for the relative processing speeds. • Batched the Writes to HBase to reduce number of calls to the server • Turned off the WAL in HBase , since we could always reprocess the file in case of a rare failure • Used primitives and Arrays in the code where feasible instead of Java Objects and Collections, to reduce the memory footprint and the pressure on the Garbage collector. 23 HBase Tuning
  • © 2013 Experian Limited. All rights reserved. • Increased the Client Write Buffer size to several megabytes. • To avoid hotspots and best data retrieval we designed composite primary key. Key design allowed us to access data by providing exact key or range scan by leading portion of key. • We found that too many filters for scan provides diminishing returns and after some point it degrades the overall scan performance 24 HBase Tuning
  • © 2013 Experian Limited. All rights reserved. Thank you For more information, please contact Manoj Khanwalkar Chief Architect manoj.khanwalkar@experian.com Govind Asawa Big Data Architect govind.asawa@experian.com