HBase Application Performance Improvement

10,126 views

Published on

Changes which can be made to get the best read performance out of HBase.

Published in: Technology
0 Comments
9 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
10,126
On SlideShare
0
From Embeds
0
Number of Embeds
5,329
Actions
Shares
0
Downloads
166
Comments
0
Likes
9
Embeds 0
No embeds

No notes for slide

HBase Application Performance Improvement

  1. 1. HBase  Cache  &  Performance   Biju  Nair   Boston  Hadoop  User  Group  Meet-­‐up   28  May  2015  
  2. 2. HBase  Overview   •  Key  value  store   •  Column  family  oriented   •  Data  stored  as  byte[]   •  Data  indexed  by  key  value   •  Data  stored  in  sorted  order  by  key   •  Data  model  doesn’t  have  to  be  pre-­‐defined   •  Scales  horizontally   2  
  3. 3. HBase  Overview   create  ‘stock’,  ‘company’,  ‘financials’   3   … msft,company,loc,ts1,Seattle msft,company,name,ts1,Microsoft … orcl,company,loc,ts1,Redwood orcl,company,name,ts1,Oracle … … msft,financials,cap,ts1,379B msft,financials,pe,ts1,20 … orcl,financials,cap,ts1,190B orcl,financials,pe,ts1,18 … Physical  Storage   put  ‘stock’,  ’ms9’,  ‘company:name’,  ‘microso9’   get  ‘stock’,  ’ms9’   company:loc,ts1,Seattle company:name,ts1,Microsoft financials:cap,ts1,379B financials:PE,ts1,20
  4. 4. HBase  Overview   4   … appl,company… … ge,company… … ibm,company… … msft,company… msft,company… … orcl,company… orcl,company… … … appl,financials… … ge,financials… … ibm,financials… … msft,financials… msft,financials… … orcl,financials… orcl,financials… … … appl,company… … … ge,company… … … ibm,company… … … msft,company… … … orcl,company… … … appl,financials… … … ge,financials… … … ibm,financials… … … msft,financials… … … orcl,financials… … Regions  
  5. 5. HBase  Overview   5   Region  Server  Region  Server  Region  Server  Region  Server  Region  Server   … appl,company… … … ge,company… … … ibm,company… … … msft,company… … … orcl,company… … HBase  Master   ZooKeeper   Client  
  6. 6. Use  Case:  Data  and  Query   •  Time  series  data   – Tickers  and  aYributes   – Monthly  data  stored  in  a  column;  256  bytes   – Up  to  20  years  worth  of  data   •  Queries   – “get”s  for  up  to  1  year  data;  3072  bytes   6  
  7. 7. Use  Case:  Requirements   •  Meet  “get”  query  performance  requirements   – Under  10  ms  for  99%  of  queries   – Median  latency  2  to  3  ms   – 99.99%  latency  under  50  ms   •  Efficient  HBase  cluster  capacity  uelizaeon   – 32  cores  per  node   – 128  GB  of  memory  per  node   – SSD  storage  in  all  nodes   7  
  8. 8. Baseline  Test  Observaeons   •  Spikes  in  read  response  emes   •  Less  than  10%  uelizaeon  of  RS  node  CPUs   •  Less  than  15%  uelizaeon  of  RS  node  memory   •  Block  cache  uelizaeon  was  inefficient   – Low  hit  raeo  and  high  eviceon  rates   8  
  9. 9. HBase  Internals  (Simplified)   HBase  Memory  (RS)   Mem  Store   Block  cache   HBase  Storage   WAL   HFiles   9  
  10. 10. HBase  Write  Path  (Simplified)   HBase  Memory  (RS)   Mem  Store   Block  cache   HBase  Storage   WAL   HFiles   1 10   3   2
  11. 11. HBase  Read  Path  (Simplified)   HBase  Memory  (RS)   Mem  Store   Block  cache   HBase  Storage   WAL   HFiles   1 2 11  
  12. 12. Cache  Uelizaeon   •  Low  hit  raeo  and  high  eviceon  rates   •  Frequently  read  data  size   – ~  3  K   •  Table  block  size   – 65  K   •  Proposed  change   – Reduce  block  size   12  
  13. 13. Impact  of  Table  Blk  Size  Change   Avg 3.002 5.362 5.361 5.357 6.419 6.369 6.405 6.383 6.188 6.196 6.182 6.174 6.246 6.264 6.268 6.253 5.194 5.207 5.219 3.031 Median 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 95% 10 15 15 15 18 18 18 18 18 18 17 17 18 18 18 18 15 15 15 10 99% 15 26 26 26 30 30 30 30 28 28 28 28 29 29 29 29 25 24 25 15 99.90% 26 41 41 41 45 45 45 45 43 43 43 43 44 44 44 44 41 41 41 26 Max 2261 127 185 102 90 106 92 102 93 106 119 114 89 140 132 82 81 150 93 1910 BAvg 16.731 16.728 16.761 16.763 16.418 16.371 16.37 16.431 16.152 16.14 16.169 16.158 16.308 16.29 16.325 16.307 16.34 16.381 16.391 16.352 BMedian 14 14 14 14 13 13 13 13 15 15 15 15 13 13 13 13 13 13 13 13 B95% 41 41 41 41 41 41 41 41 43 43 43 43 40 40 40 40 41 41 41 41 B99% 55 55 55 55 54 54 54 54 55 55 55 55 54 54 54 54 54 54 55 54 B99.9% 71 71 71 71 70 70 70 70 67 67 67 67 71 70 70 71 71 71 71 70 BMax 545 1062 559 567 1075 1027 561 567 564 541 558 1062 1062 561 1075 1072 1067 563 1035 1032 Get  Performance  (ms)  –  64  K  Blk   Get  Performance  (ms)  –  16  K  Blk   Note:  Smaller  block  size  increases  the  overhead  of  increased  index  blocks     13  
  14. 14. Memory  uelizaeon/Latency  Spikes   •  JVM  GC  contributed  to  latency  spikes   •  Increase  in  heap  size  increased  GC  eme   – Prevented  using  all  the  available  memory       •  Proposed  change:  Use  off-­‐heap  caching   – Minimize  spikes  in  response  eme  due  to  GC   – Increased  uelizaeon  of  node  memory   14  
  15. 15. HBase  Off-­‐Heap  Caching   HBase  Memory  (RS)   Mem  Store   Block  cache  (L1)  Idx  &  BF  data   HBase  Storage   WAL   HFiles   Off-­‐heap  cache  (L2)  Tbl  Data   (Bucket  Cache)   15  
  16. 16. HBase  Read  Path  (Simplified)   HBase  Memory  (RS)   Mem  Store   Block  cache   HBase  Storage   WAL   HFile   1 2 L2  Cache   3 4 16  
  17. 17. Bucket  Cache  Configuraeon   •  Hbase env.sh HBASE_REGIONSERVER_OPTS parameters –  Xmx –  XX:MaxDirectMemorySize •  Hbase site.xml properties –  hbase.regionserver.global.memstore.upperLimit –  hfile.block.cache.size –  hbase.bucketcache.size –  hbase.bucketcache.ioengine –  hbase.bucketcache.percentage.in.combinedcache   17  
  18. 18. Bucket  Cache  Configuraeon   Item   id   Values   Total  RS  memory   Tot   Memstore  size   MSz   L1  (LRU)  Cache   L1Sz   Heap  for  JVM   JHSz   XX:MaxDirectMemorySize   DMem   Tot-­‐MSz-­‐L1Sz-­‐JHSz   Xmx   Xmx   MSz+L1Sz+JHSz   hbase.regionserver.global.memstore.upperLimit   ULim   MSz/Xmx   hfile.block.cache.size   blksz   0.8-­‐ULim   hbase.bucketcache.size   bucsz   Dmem+(blksz*Xmx)   hbase.bucketcache.percentage.in.combinedcache   ccsz   1-­‐((blksz*Xmx)/bucsz))   hbase.bucketcache.ioengine   Oueap/”file:/localfile”   18  
  19. 19. Bucket  Cache  Configuraeon   Item   id   Values   Total  RS  memory   Tot   96000   Memstore  size   MSz   2000   L1  (LRU)  Cache   L1Sz   2000   Heap  for  JVM   JHSz   1000   XX:MaxDirectMemorySize   DMem   91000   Xmx   Xmx   5000   hbase.regionserver.global.memstore.upperLimit   ULim   0.4   hfile.block.cache.size   blksz   0.4   hbase.bucketcache.size   bucsz   93000   hbase.bucketcache.percentage.in.combinedcache   ccsz   0.97849   hbase.bucketcache.ioengine   ”file:/localfile”   19  
  20. 20. Impact  of  Using  Off-­‐Heap  Cache   Get  Performance  with  L1  cache     Get  Performance  with  L1  &  L2  cache   Note:  L1  cache  test  used  38  GB  of  data,  L1+L2  test  used  3  TB  of  data     Avg 3.872 3.995 3.936 4.007 4.052 Median 1 1 1 1 1 95% 14 14 14 15 15 99% 20 20 20 20 20 99.90% 27 27 27 28 28 99.99% 36 36 36 37 37 99.999% 208 310 332 207 232 Max 1360 1906 1736 1359 1363 807Mil797107Mil7Requests BAvg 3.429 2.552 3.447 3.502 3.554 BMedian 2 2 2 2 2 B95% 10 8 10 10 10 B99% 18 14 18 18 18 B99.9% 30 23 30 30 31 BMax 78 1135 58 77 67 18Mil8Rows8>818Mil8Requests 20  
  21. 21. Maximize  CPU  &  Memory  Uelizaeon   •  Run  addieonal  RS  per  node   •  Throughput  increased  50%  when  RS  increased   to  2   – Through  put  reduced  on  AWS  cluster   – There  was  no  degradaeon  on  the  response  eme   – Through  put  increase  tapered  awer  3  RS  per  node   •  Note:  Maintenance  over  head  using  mule-­‐RS   21  
  22. 22. Known  Issues   •  Using  “oueap”  opeon  of  BucketCache  prevents   RS  start   –  [HBASE-­‐10643]   –  Can  be  miegated  using  tempfs   •  LoadIncrementalHFiles  doesn’t  work  with   BucketCache     –  [HBase-­‐10500]   •  BucketCache  for  different  block  sizes  is  not   configurable   –  [HBASE-­‐10641]  Fixed   22  
  23. 23. Key  Takeaways   •  Store  what  is  really  required   – Understand  the  query  paYern   – Leverage  column  family  (CF)  to  group  data   •  Choose  appropriate  block  size  for  table/CF   •  Use  off  heap  cache  to  minimize  latency  spikes   •  Test  all  assumpeons   23  
  24. 24. Further  Reading   •  hYp://blog.asquareb.com/blog/2014/11/21/leverage-­‐hbase-­‐cache-­‐and-­‐ improve-­‐read-­‐performance   •  hYp://blog.asquareb.com/blog/2014/11/24/how-­‐to-­‐leverage-­‐large-­‐ physical-­‐memory-­‐to-­‐improve-­‐hbase-­‐read-­‐performance   •  hYps://issues.apache.org/jira/browse/HBASE-­‐7404   •  hYp://www.n10k.com/blog/blockcache-­‐101/     •  hYp://www.n10k.com/blog/blockcache-­‐showdown/   24  
  25. 25. 25   bnair@asquareb.com blog.asquareb.com https://github.com/bijugs @gsbiju

×