• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
HBase Blockcache 101
 

HBase Blockcache 101

on

  • 377 views

An introduction to the HBase BlockCache, the internal system for caching data and speeding reads.

An introduction to the HBase BlockCache, the internal system for caching data and speeding reads.

Statistics

Views

Total Views
377
Views on SlideShare
348
Embed Views
29

Actions

Likes
2
Downloads
4
Comments
0

2 Embeds 29

https://twitter.com 27
http://www.slideee.com 2

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution-NonCommercial LicenseCC Attribution-NonCommercial License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    HBase Blockcache 101 HBase Blockcache 101 Presentation Transcript

    • HBase  BlockCache  101   Nick  Dimiduk,  Hortonworks              @xefyr                n10k.com  
    • Agenda   •  What  is  a  BlockCache?   •  Why  cache  blocks?   •  Available  BlockCaches   •  ConfiguraFons  
    • RegionServer (HBase) DataNode (Hadoop DFS) HLog (WAL) HRegion HStore StoreFile HFile StoreFile HFile MemStore ... ... HStore BlockCache HRegion ... HStoreHStore ... 1 5 2 3 3 2 4 Read  Path  Refresher   BlockCache  
    • What  is  a  block?   HFile  v2  Format   HFile  v2  format  figure  reproduced  from  MaMeo  Bertozzi,  "Apache   HBase  I/O  –  HFile”,  hMp://blog.cloudera.com/blog/2012/06/hbase-­‐io-­‐ hfile-­‐input-­‐output/  
    • Why  cache  blocks?  Storage hierarchy: a different view Dean/2009   Memory  is  100000x   faster  than  disk!   Disk  seek  =  10ms  
    • LruBlockCache! •  On-­‐heap  ConcurrentHashMap! •  3  block  prioriFes,  based  on  access  frequency   –  Single   –  MulF   –  In-­‐memory   •  Background  evicFon  thread   •  HBASE-­‐1460,  0.20  
    • LruBlockCache  ConfiguraFon   •  Size  as  pct  of  heap   –  hfile.block.cache.size   –  default:  0.4   •  Single  access  Priority  size   –  hbase.lru.blockcache.single.percentage   –  default:  0.25   •  MulF-­‐access  priority  size   –  hbase.lru.blockcache.multi.percentage   –  default:  0.5   •  In-­‐memory  priority  size   –  hbase.lru.blockcache.memory.percentage   –  default:  0.25  
    • SlabCache! •  L2  cache;  fronted  by  LruBlockCache  (L1)   •  DoubleBlockCache  caches  blocks  in  both  L1  and  L2   •  Allocates  porFons  of  memory  with  target  block  size   •  DirectByteBuffers  manage  off-­‐heap  memory   •  Blocks  copied  on  and  off  heap   •  HBASE-­‐4027,  0.92  
    • SlabCache  ConfiguraFon   •  SlabCache  size   –  hbase.offheapcache.percentage! •  Target  Block  size   –  hbase.offheapcache.minblocksize! –  default:  64k   •  Slab  proporFons   –  hbase.offheapcache.slab.proportions! –  default:  0.8,  0.2   •  Slab  target  block  sizes   –  hbase.offheapcache.slab.sizes! –  default:  64k,  128k  
    • BucketCache! •  L2  cache;  fronted  by  LruBlockCache  (L1)   •  CombinedBlockCache  caches  data  blocks  in  L2,  everything  else  in  L1   •  Blocks  evicted  from  L1  are  promoted  to  L2   •  Allocates  porFons  of  space  with  target  block  size   •  Three  operaFng  modes:   –  onheap  –  HeapByteBuffers  manage  on-­‐heap  memory  space   –  offheap  –  DirectByteBuffers  manage  off-­‐heap  memory  space   –  file  –  RandomAccessFile  manages  file-­‐backed  space   •  Blocks  copied  on  and  off  heap   •  HBASE-­‐7404,  0.96  
    • BucketCache  ConfiguraFons   •  OperaFng  mode   –  hbase.bucketcache.ioengine! •  Total  cache  size  as  pct  of  heap  OR  size  in  MB:   –  hbase.bucketcache.size! •  Size  of  L1  as  pct  of  total  cache  size:   –  hbase.bucketcache.percentage.in.combinedcache! –  default:  0.9   •  Target  Block  size   –  hbase.offheapcache.minblocksize! –  default:  64k   •  Bucket  target  block  sizes   –  hbase.bucketcache.bucket.sizes! –  default:  14  sizes,  clustered  around  64k  
    • BlockCache  Showdown   •  Compare  performance:   –  Total  cache  size   –  Cache  size  vs.  Data  size     n10k.com/blog/ blockcache-­‐showdown/   L2  off-­‐heap  BucketCache   makes  a  strong  showing  
    • Thanks!   M A N N I N G Nick Dimiduk Amandeep Khurana FOREWORD BY Michael Stack hbaseinacFon.com   Nick  Dimiduk              github.com/ndimiduk              @xefyr              n10k.com   n10k.com/blog/blockcache-­‐101