![HBase
Overview
• Key
value
store
• Column
family
oriented
• Data
stored
as
byte[]
• Data
indexed
by
key
value
• Data
stored
in
sorted
order
by
key
• Data
model
doesn’t
have
to
be
pre-‐defined
• Scales
horizontally
2](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (15)
Similar to HBase Application Performance Improvement
Similar to HBase Application Performance Improvement (20)
More from Biju Nair
Recently uploaded
Recently uploaded (20)
HBase Application Performance Improvement
- 1. HBase Cache & Performance Biju Nair Boston Hadoop User Group Meet-‐up 28 May 2015
- 2. HBase Overview • Key value store • Column family oriented • Data stored as byte[] • Data indexed by key value • Data stored in sorted order by key • Data model doesn’t have to be pre-‐defined • Scales horizontally 2
- 3. HBase Overview create ‘stock’, ‘company’, ‘financials’ 3 … msft,company,loc,ts1,Seattle msft,company,name,ts1,Microsoft … orcl,company,loc,ts1,Redwood orcl,company,name,ts1,Oracle … … msft,financials,cap,ts1,379B msft,financials,pe,ts1,20 … orcl,financials,cap,ts1,190B orcl,financials,pe,ts1,18 … Physical Storage put ‘stock’, ’ms9’, ‘company:name’, ‘microso9’ get ‘stock’, ’ms9’ company:loc,ts1,Seattle company:name,ts1,Microsoft financials:cap,ts1,379B financials:PE,ts1,20
- 4. HBase Overview 4 … appl,company… … ge,company… … ibm,company… … msft,company… msft,company… … orcl,company… orcl,company… … … appl,financials… … ge,financials… … ibm,financials… … msft,financials… msft,financials… … orcl,financials… orcl,financials… … … appl,company… … … ge,company… … … ibm,company… … … msft,company… … … orcl,company… … … appl,financials… … … ge,financials… … … ibm,financials… … … msft,financials… … … orcl,financials… … Regions
- 5. HBase Overview 5 Region Server Region Server Region Server Region Server Region Server … appl,company… … … ge,company… … … ibm,company… … … msft,company… … … orcl,company… … HBase Master ZooKeeper Client
- 6. Use Case: Data and Query • Time series data – Tickers and aYributes – Monthly data stored in a column; 256 bytes – Up to 20 years worth of data • Queries – “get”s for up to 1 year data; 3072 bytes 6
- 7. Use Case: Requirements • Meet “get” query performance requirements – Under 10 ms for 99% of queries – Median latency 2 to 3 ms – 99.99% latency under 50 ms • Efficient HBase cluster capacity uelizaeon – 32 cores per node – 128 GB of memory per node – SSD storage in all nodes 7
- 8. Baseline Test Observaeons • Spikes in read response emes • Less than 10% uelizaeon of RS node CPUs • Less than 15% uelizaeon of RS node memory • Block cache uelizaeon was inefficient – Low hit raeo and high eviceon rates 8
- 9. HBase Internals (Simplified) HBase Memory (RS) Mem Store Block cache HBase Storage WAL HFiles 9
- 10. HBase Write Path (Simplified) HBase Memory (RS) Mem Store Block cache HBase Storage WAL HFiles 1 10 3 2
- 11. HBase Read Path (Simplified) HBase Memory (RS) Mem Store Block cache HBase Storage WAL HFiles 1 2 11
- 12. Cache Uelizaeon • Low hit raeo and high eviceon rates • Frequently read data size – ~ 3 K • Table block size – 65 K • Proposed change – Reduce block size 12
- 13. Impact of Table Blk Size Change Avg 3.002 5.362 5.361 5.357 6.419 6.369 6.405 6.383 6.188 6.196 6.182 6.174 6.246 6.264 6.268 6.253 5.194 5.207 5.219 3.031 Median 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 95% 10 15 15 15 18 18 18 18 18 18 17 17 18 18 18 18 15 15 15 10 99% 15 26 26 26 30 30 30 30 28 28 28 28 29 29 29 29 25 24 25 15 99.90% 26 41 41 41 45 45 45 45 43 43 43 43 44 44 44 44 41 41 41 26 Max 2261 127 185 102 90 106 92 102 93 106 119 114 89 140 132 82 81 150 93 1910 BAvg 16.731 16.728 16.761 16.763 16.418 16.371 16.37 16.431 16.152 16.14 16.169 16.158 16.308 16.29 16.325 16.307 16.34 16.381 16.391 16.352 BMedian 14 14 14 14 13 13 13 13 15 15 15 15 13 13 13 13 13 13 13 13 B95% 41 41 41 41 41 41 41 41 43 43 43 43 40 40 40 40 41 41 41 41 B99% 55 55 55 55 54 54 54 54 55 55 55 55 54 54 54 54 54 54 55 54 B99.9% 71 71 71 71 70 70 70 70 67 67 67 67 71 70 70 71 71 71 71 70 BMax 545 1062 559 567 1075 1027 561 567 564 541 558 1062 1062 561 1075 1072 1067 563 1035 1032 Get Performance (ms) – 64 K Blk Get Performance (ms) – 16 K Blk Note: Smaller block size increases the overhead of increased index blocks 13
- 14. Memory uelizaeon/Latency Spikes • JVM GC contributed to latency spikes • Increase in heap size increased GC eme – Prevented using all the available memory • Proposed change: Use off-‐heap caching – Minimize spikes in response eme due to GC – Increased uelizaeon of node memory 14
- 15. HBase Off-‐Heap Caching HBase Memory (RS) Mem Store Block cache (L1) Idx & BF data HBase Storage WAL HFiles Off-‐heap cache (L2) Tbl Data (Bucket Cache) 15
- 16. HBase Read Path (Simplified) HBase Memory (RS) Mem Store Block cache HBase Storage WAL HFile 1 2 L2 Cache 3 4 16
- 17. Bucket Cache Configuraeon • Hbase env.sh HBASE_REGIONSERVER_OPTS parameters – Xmx – XX:MaxDirectMemorySize • Hbase site.xml properties – hbase.regionserver.global.memstore.upperLimit – hfile.block.cache.size – hbase.bucketcache.size – hbase.bucketcache.ioengine – hbase.bucketcache.percentage.in.combinedcache 17
- 18. Bucket Cache Configuraeon Item id Values Total RS memory Tot Memstore size MSz L1 (LRU) Cache L1Sz Heap for JVM JHSz XX:MaxDirectMemorySize DMem Tot-‐MSz-‐L1Sz-‐JHSz Xmx Xmx MSz+L1Sz+JHSz hbase.regionserver.global.memstore.upperLimit ULim MSz/Xmx hfile.block.cache.size blksz 0.8-‐ULim hbase.bucketcache.size bucsz Dmem+(blksz*Xmx) hbase.bucketcache.percentage.in.combinedcache ccsz 1-‐((blksz*Xmx)/bucsz)) hbase.bucketcache.ioengine Oueap/”file:/localfile” 18
- 19. Bucket Cache Configuraeon Item id Values Total RS memory Tot 96000 Memstore size MSz 2000 L1 (LRU) Cache L1Sz 2000 Heap for JVM JHSz 1000 XX:MaxDirectMemorySize DMem 91000 Xmx Xmx 5000 hbase.regionserver.global.memstore.upperLimit ULim 0.4 hfile.block.cache.size blksz 0.4 hbase.bucketcache.size bucsz 93000 hbase.bucketcache.percentage.in.combinedcache ccsz 0.97849 hbase.bucketcache.ioengine ”file:/localfile” 19
- 20. Impact of Using Off-‐Heap Cache Get Performance with L1 cache Get Performance with L1 & L2 cache Note: L1 cache test used 38 GB of data, L1+L2 test used 3 TB of data Avg 3.872 3.995 3.936 4.007 4.052 Median 1 1 1 1 1 95% 14 14 14 15 15 99% 20 20 20 20 20 99.90% 27 27 27 28 28 99.99% 36 36 36 37 37 99.999% 208 310 332 207 232 Max 1360 1906 1736 1359 1363 807Mil797107Mil7Requests BAvg 3.429 2.552 3.447 3.502 3.554 BMedian 2 2 2 2 2 B95% 10 8 10 10 10 B99% 18 14 18 18 18 B99.9% 30 23 30 30 31 BMax 78 1135 58 77 67 18Mil8Rows8>818Mil8Requests 20
- 21. Maximize CPU & Memory Uelizaeon • Run addieonal RS per node • Throughput increased 50% when RS increased to 2 – Through put reduced on AWS cluster – There was no degradaeon on the response eme – Through put increase tapered awer 3 RS per node • Note: Maintenance over head using mule-‐RS 21
- 22. Known Issues • Using “oueap” opeon of BucketCache prevents RS start – [HBASE-‐10643] – Can be miegated using tempfs • LoadIncrementalHFiles doesn’t work with BucketCache – [HBase-‐10500] • BucketCache for different block sizes is not configurable – [HBASE-‐10641] Fixed 22
- 23. Key Takeaways • Store what is really required – Understand the query paYern – Leverage column family (CF) to group data • Choose appropriate block size for table/CF • Use off heap cache to minimize latency spikes • Test all assumpeons 23
- 24. Further Reading • hYp://blog.asquareb.com/blog/2014/11/21/leverage-‐hbase-‐cache-‐and-‐ improve-‐read-‐performance • hYp://blog.asquareb.com/blog/2014/11/24/how-‐to-‐leverage-‐large-‐ physical-‐memory-‐to-‐improve-‐hbase-‐read-‐performance • hYps://issues.apache.org/jira/browse/HBASE-‐7404 • hYp://www.n10k.com/blog/blockcache-‐101/ • hYp://www.n10k.com/blog/blockcache-‐showdown/ 24