Hadoop Summit 2012 | HBase Consistency and Performance Improvements

June 13, 2012HBase Consistency andPerformance ImprovementsEsteban Gutierrez, Gregory Chanan{esteban, gchanan}@cloudera.com

Who We Are • Esteban Gutierrez – Customer Operations Engineer - Focused on HBase operations • Gregory Chanan – HBase developer – Currently focused on wire compatibility2 ©2012 Cloudera, Inc. All Rights Reserved.

Apache HBase Apache HBase is a distributed, scalable column-oriented data store that runs on top of HDFS. It provides consistent, low latency, random read/write access.3 ©2012 Cloudera, Inc. All Rights Reserved.

HBase Data FormatColumn names are family:qualifierRowKey header:from header:subject body:textgreg_email1 sister@gmail.com Father’s day card <…>greg_email2 friend@gmail.com Taco night <…>5 ©2012 Cloudera, Inc. All Rights Reserved.

HBase Data FormatColumn names are family:qualifierRowKey header:from header:subject body:textgreg_email1 sister@gmail.com Father’s day card <…>greg_email2 friend@gmail.com Taco night <…>Column Families are a set of related columnsthat are physically stored together on disk6 ©2012 Cloudera, Inc. All Rights Reserved.

HBase Write Path HBase Put Client Put HBase Server HLog1. Write to HLog for disaster recovery Put Put MemStore MemStore2. Write to MemStore (in memory map) Put Put 13 ©2012 Cloudera, Inc. All Rights Reserved.

HBase Write Path HBase Put Client Put HBase Server HLog1. Write to HLog for disaster recovery Put Put MemStore MemStore2. Write to MemStore (in memory map) Put Put HFile3. Flush MemStore to disk as HFile Put 14 ©2012 Cloudera, Inc. All Rights Reserved.

What are ACID Transactions? • Atomicity – All parts of transaction complete or none complete • Consistency – Only valid data written to database • Isolation – Parallel transactions do not impact each other’s execution • Durability – Once transaction committed, it remains18 ©2012 Cloudera, Inc. All Rights Reserved.

HBase ACID in 0.92 • ―Any row returned by [a] scan will be a consistent view (i.e. that version of the complete row existed at some point in time)‖[1] [1] http://hbase.apache.org/acid-semantics.html19 ©2012 Cloudera, Inc. All Rights Reserved.

Atomic Bulk Upload • A common pattern of use in HBase is to upload data as fast as possible from external sources • HRegion.bulkLoadHFile() makes that possible21 ©2011 Cloudera, Inc. All Rights Reserved.

Atomic Bulk Upload Row 1 HRegion.bulkLoadHFile() ≤ HBase 0.90.5 HFile1: HFile2: HFile3: HFile4: header:to meta:labels body:text attach:file T1 sister@... T2 sister@... family Scan T3 sister@... family Hi… T4 sister@... family Hi… image/jpeg24 ©2012 Cloudera, Inc. All Rights Reserved.

Atomic Bulk Upload Row 1 HRegion.bulkLoadHFiles() ≥ HBase 0.92 HFile1: HFile2: HFile3: HFile4: header:to meta:labels body:text attach:file T1 T2 Scan T3 T4 sister@... family Hi… image/jpeg …27 ©2012 Cloudera, Inc. All Rights Reserved.

Read ACID Compliance Symptoms Run 1 Run 2 Run 3 … … … … … SPLIT_RAW_FILES … … … Map-Reduce Framework Map output records 500,000 499,997 500,001 header:to header:from body:text greg_email1 sister@... greg@... Hi… greg_email2 sister@... esteban_email3 esteban@... Good news!.. esteban_email3 brother@...32 ©2011 Cloudera, Inc. All Rights Reserved.

Read ACID Compliance Symptoms Scale testing shows between 0.5% to 2% of inconsistent results between runs Run 1 Run 2 Run 3 … … … … … SPLIT_RAW_FILES … … … Map-Reduce Framework Map output records 500,000 499,997 500,001 header:to header:from body:text greg_email1 sister@... greg@... Hi… greg_email2 sister@... esteban_email3 esteban@... Good news!.. esteban_email3 brother@...33 ©2011 Cloudera, Inc. All Rights Reserved.

MVCC • HBase maintains ACID semantics using Multiversion Concurrency Control • Instead of overwriting state, create a new version of object with timestamp memStoreTs RowKey fam1:col1 fam2:col2 t1 row1 val1 val137 ©2012 Cloudera, Inc. All Rights Reserved.

Multi Version Concurrency Control • HBase maintains ACID semantics using Multiversion Concurrency Control • Instead of overwriting state, create a new version of object with timestamp (―memStoreTs‖) memstoreTs RowKey fam1:col1 fam2:col2 t2 row1 val2 val2 t1 row1 val1 val1 • Reads never have to block • ―memStoreTs‖ is not externally visible! Different from external timestamp38 ©2012 Cloudera, Inc. All Rights Reserved.

Review: HBase Write Path HBase Put Client Put HBase Server HLog1. Write to Hlog for disaster recovery Put Put MemStore MemStore2. Write to MemStore (in memory map) Put Put HFile3. Flush MemStore to disk as HFile Put 39 ©2012 Cloudera, Inc. All Rights Reserved.

Putting it together Let’s go back to the beginning… MemStore memstoreTs RowKey hdr:from body:text t2 greg_email coworker bug report t1 greg_email wife pick up kids And start a scan. And concurrently put.42 ©2012 Cloudera, Inc. All Rights Reserved.

Putting it together Let’s go back to the beginning… MemStore memstoreTs RowKey hdr:from body:text t2 greg_email coworker bug report t1 greg_email wife pick up kids And start a scan. HFile And concurrently put. RowKey body:text Which causes a flush. greg_email bug report greg_email pick up kids43 ©2012 Cloudera, Inc. All Rights Reserved.

Putting it together Now, scan needs to make sense of this… MemStore memstoreTs RowKey hdr:from t2 greg_email coworker t1 greg_email wife HFile RowKey body:text greg_email bug report greg_email pick up kids44 ©2012 Cloudera, Inc. All Rights Reserved.

Putting it together Now, scan needs to make sense of this… MemStore memstoreTs RowKey hdr:from t2 greg_email coworker t1 greg_email wife HFile RowKey body:text greg_email bug report greg_email pick up kids But HFile has no timestamp!45 ©2012 Cloudera, Inc. All Rights Reserved.

Putting it together Now, scan needs to make sense of this… MemStore memstoreTs RowKey hdr:from t2 greg_email coworker t1 greg_email wife HFile Inconsistent Result RowKey body:text RowKey hdr:from body:text greg_email bug report greg_email wife wife bug report bug report greg_email pick up kids But HFile has no timestamp!46 ©2012 Cloudera, Inc. All Rights Reserved.

Putting it together Now, scan needs to make sense of this… MemStore memstoreTs RowKey hdr:from t2 greg_email coworker t1 greg_email wife HFile Inconsistent Result RowKey body:text RowKey hdr:from body:text greg_email bug report greg_email wife wife bug report bug report greg_email pick up kids But HFile has no timestamp!47 ©2012 Cloudera, Inc. All Rights Reserved.

Solution Store the timestamp in the Hfile MemStore HFilememstoreTs RowKey hdr:from memStoreTs RowKey body:text t2 greg_email bug reportt2 greg_email coworkert1 greg_email wife t1 greg_email pick up kids Correct Result RowKey hdr:from body:text greg_email val1 wife val1 up kids pick Now we have all the information we need 48 ©2012 Cloudera, Inc. All Rights Reserved.

Consistency to Performance • Initial community focus on correctness and consistency • HBase adoption growing – Number of customers – Size of deployment • Newer focus on performance50 ©2012 Cloudera, Inc. All Rights Reserved.

Performance • Initial community focus on correctness and consistency • HBase adoption growing – Number of customers – Size of deployment • Newer focus on performance – 0.94 dubbed the ―performance release‖51 ©2012 Cloudera, Inc. All Rights Reserved.

Performance Areas for Improvement • Read Path – Support checksums in HFile format (HBASE-5047) • Compactions – Delete out of TTL store files before compactions (HBASE-5199) • Write Path – HLog Compression (HBASE-4608) • HDFS level – Works with hadoop 2.0 – See HBase and HDFS: Past, Present and Future • And much more!53 ©2012 Cloudera, Inc. All Rights Reserved.

Read Path Performance: Checksums • HDFS stores checksum in separate file HFile Checksum • So each file read actually requires two disk iops • HBase often bottlenecked by random disk iops55 ©2012 Cloudera, Inc. All Rights Reserved.

Read Path Performance: Checksums • Solution: Store checksum in HFile block • Turn off HDFS-level checksum HFile HFile Block Chksum Data • On by default (―hbase.regionserver.checksum.verify‖) • Bytes per checksum (―hbase.hstore.bytes.per.checksum‖) – default is 16K56 ©2012 Cloudera, Inc. All Rights Reserved.

HBase Performance Comparison Test Setup: • Compare CDH4 to CDH3u4 • 5 node cluster running Yahoo Cloud Serving Benchmark (YCSB) • 5 million records • Two distributions of operations: – 100% write – 50% read, 50% write61 ©2012 Cloudera, Inc. All Rights Reserved.

HBase Performance Results • 100% write workload: – 49% throughput improvement – 28% latency improvement • 50% write, 50% read workload: – 14% throughput improvement – 14% latency improvement62 ©2012 Cloudera, Inc. All Rights Reserved.

Conclusion • Many consistency improvements in 0.92 / CDH4 • Performance improvements in 0.94 • 0.94 is wire compatible with 0.92, so will be in a CDH4 update64 ©2012 Cloudera, Inc. All Rights Reserved.

References • HBase Acid Semantics, http://hbase.apache.org/acid- semantics.html • Apache HBase Meetup @ SU; Michael Stack. http://files.meetup.com/1350427/20120327hbase_meetu p.pdf • HBase Internals; Lars Hofhansl. http://www.cloudera.com/resource/hbasecon-2012- learning-hbase-internals/ • Hbase and HDFS: Past, Present, and Future; Todd Lipcon http://www.cloudera.com/resource/hbasecon- 2012-hbase-and-hdfs-past-present-future/65 ©2012 Cloudera, Inc. All Rights Reserved.

http://www.scoop.it	3749
http://www.cloudera.com	184
http://blog.cloudera.com	9
http://webcache.googleusercontent.com	7
https://twitter.com	5
http://us-w1.rockmelt.com	1
https://si0.twimg.com	1
http://author.cloudera.solutionset.com	1

Hadoop Summit 2012 | HBase Consistency and Performance Improvements

by Cloudera, Inc. on Jun 18, 2012

Accessibility

Categories

Upload Details

Usage Rights

8 Embeds 3,957

Statistics

Hadoop Summit 2012 | HBase Consistency and Performance Improvements Presentation Transcript