2. 2
Bio
Govind Kamat
• Member of the Performance Engineering Team at Cloudera
• Focuses on Hadoop and HBase performance and scalability
• Experience includes the development of large-scale software systems,
microprocessor architecture, compilers and electronic design
Yanpei Chen
• Member of the Performance Engineering Team at Cloudera
• Works on cross-component performance - Hadoop, HBase, Search and Impala
• Ph.D. from UC Berkeley, focus on performance measurement method and theory
3. 3
Outline
• Apache HBase overview
• Measuring performance + YCSB basics
• Cluster setup best practices
• Techniques for rigorous measurement
• HBase in a multi-tenant environment
4. 4
HBase Overview
• Distributed, "NoSQL" key-value store
• Column-oriented, sorted map
• Keys are lexicographically sorted
• Multiple regions across “regionservers”
• Built on HDFS, MapReduce not required
5. 5
Measuring HBase Performance is Hard!
• Numbers not reproducible
• Large run-to-run variation
• Testbeds not clearly defined/properly setup
• Various workloads have been used
• Configuration parameters not specified
• State of regionservers not taken into account
• Reported numbers not comparable
7. 7
Workloads for Performance Measurement
• Set of transactions to be imposed against it
• read, update, insert, scan and mixes thereof
• Initial data to be loaded into the DB
• Insert
• Transaction load intensity variation over time
• Possible HBase workloads:
• Actual customer/production workloads (best)
• PerformanceEvaluation (not really a workload )
• YCSB (Yahoo! Cloud Serving Benchmark, commonly used)
8. 8
Yahoo! Cloud Serving Benchmark (YCSB) Basics
• Performance evaluation framework for key-value
databases, such as:
• HBase, Cassandra, Sherpa, Accumulo, Voldemort
• Abstracts out the client from the DB
• Flexible and configurable
• Comes with a standard “core” workload
• Reports throughput and latency metrics
13. 13
Cluster Setup Best Practices
• Setting up the cluster
• Configuring HBase
• Creating tables
• Pre-splitting tables
• Loading data
14. 14
HBase Cluster Configuration Best Practices
• Use the appropriate hardware, correctly sized: memory, disk
• Dedicate separate nodes for master services and worker roles
• No Task Trackers and Node Managers on regionserver nodes
• Segregate clients from the regionservers
• Configure HBase properly:
• Block cache (read), memstore (write)
• Bloom filters, compression, compaction, short-circuit reads, etc.
• Use the appropriate data set size, number of regions, etc.
• Monitor the cluster constantly
16. 16
Data Loading – Several Options
• Real, actual, production (hot) data
• Custom loader
• PerformanceEvaluation
• Loading using YCSB
• HFileGenerator followed by bulk-load
17. 17
Data Loading - Pre-split the Table
• Auto-splitting has significant overhead
• RegionSplitter utility
• UniformSplit
• HexStringSplit
• YCSB: user100000 .. user999999
hbase(main):1:0> create 'usertable', 'cf’,
{ SPLITS=> (1..(50-1)).map {|i| "user#{1000 +
i*9000/50}" } } #50 splits
• Set maximum region file size to a large value
18. 18
Techniques for Rigorous Measurement
• Keep the input data set fixed
• Warm up the cache
• Set the target throughput
• Use the correct workload distribution
20. 20
Keep the Input Data Set Fixed!
A beginning is the time for taking the most
delicate care that the balances are correct.
The manual of Muad’Dib
From “Dune” by Frank Herbert
22. 22
Warm Up the Cache
• Performance depends significantly on memory
• HBase block cache and OS page cache for reads
• Memstore and WAL for writes
• Load all the rows in the table
• Write until data starts getting flushed
• Compaction can affect performance significantly
• Carry out long-running tests
• Repeat till steady-state
• Otherwise, performance can vary a lot
24. 24
Set the Target Throughput
• Two parameters to set desired throughput
• -threads
• -target
• Actual throughput will match target throughput ...
• ... until the DB hits its limit
• Performance may then begin to degrade
• This throughput defines maximum cluster performance
• Can be used to evaluate different HBase releases
• Otherwise, HBase is never stressed beyond saturation
26. 26
Use the Appropriate Workload Distribution
• Various types possible
• Uniform (default, but unrealistic)
• Latest
• Hotspot
• Zipfian
27. 27
Rigorous Measurement Techniques
• Set the cluster up properly
• Keep the input data set fixed
• Pre-split the key space
• Warm up the cache properly
• Set the target throughput
• Use the correct workload distribution
• Monitor cluster statistics continually