HBase Blockcache 101

Nick Dimiduk
Nick DimidukEngineer, Hacker, Author
HBase	
  BlockCache	
  101	
  
Nick	
  Dimiduk,	
  Hortonworks	
  
	
  	
  	
  	
  	
  	
  @xefyr	
  	
  	
  	
  	
  	
  	
  	
  n10k.com	
  
Agenda	
  
•  What	
  is	
  a	
  BlockCache?	
  
•  Why	
  cache	
  blocks?	
  
•  Available	
  BlockCaches	
  
•  ConfiguraFons	
  
RegionServer (HBase)
DataNode (Hadoop DFS)
HLog
(WAL)
HRegion
HStore
StoreFile
HFile
StoreFile
HFile
MemStore
...
...
HStore
BlockCache
HRegion
...
HStoreHStore
...
1 5
2
3
3
2
4
Read	
  Path	
  Refresher	
  
BlockCache	
  
What	
  is	
  a	
  block?	
  
HFile	
  v2	
  Format	
  
HFile	
  v2	
  format	
  figure	
  reproduced	
  from	
  MaMeo	
  Bertozzi,	
  "Apache	
  
HBase	
  I/O	
  –	
  HFile”,	
  hMp://blog.cloudera.com/blog/2012/06/hbase-­‐io-­‐
hfile-­‐input-­‐output/	
  
Why	
  cache	
  blocks?	
  Storage hierarchy: a different view
Dean/2009	
  
Memory	
  is	
  100000x	
  
faster	
  than	
  disk!	
  
Disk	
  seek	
  =	
  10ms	
  
LruBlockCache!
•  On-­‐heap	
  ConcurrentHashMap!
•  3	
  block	
  prioriFes,	
  based	
  on	
  access	
  frequency	
  
–  Single	
  
–  MulF	
  
–  In-­‐memory	
  
•  Background	
  evicFon	
  thread	
  
•  HBASE-­‐1460,	
  0.20	
  
LruBlockCache	
  ConfiguraFon	
  
•  Size	
  as	
  pct	
  of	
  heap	
  
–  hfile.block.cache.size	
  
–  default:	
  0.4	
  
•  Single	
  access	
  Priority	
  size	
  
–  hbase.lru.blockcache.single.percentage	
  
–  default:	
  0.25	
  
•  MulF-­‐access	
  priority	
  size	
  
–  hbase.lru.blockcache.multi.percentage	
  
–  default:	
  0.5	
  
•  In-­‐memory	
  priority	
  size	
  
–  hbase.lru.blockcache.memory.percentage	
  
–  default:	
  0.25	
  
SlabCache!
•  L2	
  cache;	
  fronted	
  by	
  LruBlockCache	
  (L1)	
  
•  DoubleBlockCache	
  caches	
  blocks	
  in	
  both	
  L1	
  and	
  L2	
  
•  Allocates	
  porFons	
  of	
  memory	
  with	
  target	
  block	
  size	
  
•  DirectByteBuffers	
  manage	
  off-­‐heap	
  memory	
  
•  Blocks	
  copied	
  on	
  and	
  off	
  heap	
  
•  HBASE-­‐4027,	
  0.92	
  
SlabCache	
  ConfiguraFon	
  
•  SlabCache	
  size	
  
–  hbase.offheapcache.percentage!
•  Target	
  Block	
  size	
  
–  hbase.offheapcache.minblocksize!
–  default:	
  64k	
  
•  Slab	
  proporFons	
  
–  hbase.offheapcache.slab.proportions!
–  default:	
  0.8,	
  0.2	
  
•  Slab	
  target	
  block	
  sizes	
  
–  hbase.offheapcache.slab.sizes!
–  default:	
  64k,	
  128k	
  
BucketCache!
•  L2	
  cache;	
  fronted	
  by	
  LruBlockCache	
  (L1)	
  
•  CombinedBlockCache	
  caches	
  data	
  blocks	
  in	
  L2,	
  everything	
  else	
  in	
  L1	
  
•  Blocks	
  evicted	
  from	
  L1	
  are	
  promoted	
  to	
  L2	
  
•  Allocates	
  porFons	
  of	
  space	
  with	
  target	
  block	
  size	
  
•  Three	
  operaFng	
  modes:	
  
–  onheap	
  –	
  HeapByteBuffers	
  manage	
  on-­‐heap	
  memory	
  space	
  
–  offheap	
  –	
  DirectByteBuffers	
  manage	
  off-­‐heap	
  memory	
  space	
  
–  file	
  –	
  RandomAccessFile	
  manages	
  file-­‐backed	
  space	
  
•  Blocks	
  copied	
  on	
  and	
  off	
  heap	
  
•  HBASE-­‐7404,	
  0.96	
  
BucketCache	
  ConfiguraFons	
  
•  OperaFng	
  mode	
  
–  hbase.bucketcache.ioengine!
•  Total	
  cache	
  size	
  as	
  pct	
  of	
  heap	
  OR	
  size	
  in	
  MB:	
  
–  hbase.bucketcache.size!
•  Size	
  of	
  L1	
  as	
  pct	
  of	
  total	
  cache	
  size:	
  
–  hbase.bucketcache.percentage.in.combinedcache!
–  default:	
  0.9	
  
•  Target	
  Block	
  size	
  
–  hbase.offheapcache.minblocksize!
–  default:	
  64k	
  
•  Bucket	
  target	
  block	
  sizes	
  
–  hbase.bucketcache.bucket.sizes!
–  default:	
  14	
  sizes,	
  clustered	
  around	
  64k	
  
BlockCache	
  Showdown	
  
•  Compare	
  performance:	
  
–  Total	
  cache	
  size	
  
–  Cache	
  size	
  vs.	
  Data	
  size	
  
	
  
n10k.com/blog/
blockcache-­‐showdown/	
  
L2	
  off-­‐heap	
  BucketCache	
  
makes	
  a	
  strong	
  showing	
  
Thanks!	
  
M A N N I N G
Nick Dimiduk
Amandeep Khurana
FOREWORD BY
Michael Stack
hbaseinacFon.com	
  
Nick	
  Dimiduk	
  
	
   	
  	
  	
  	
  	
  github.com/ndimiduk	
  
	
   	
  	
  	
  	
  	
  @xefyr	
  
	
   	
  	
  	
  	
  	
  n10k.com	
  
n10k.com/blog/blockcache-­‐101	
  
1 of 13

Recommended

Off-heaping the Apache HBase Read Path by
Off-heaping the Apache HBase Read Path Off-heaping the Apache HBase Read Path
Off-heaping the Apache HBase Read Path HBaseCon
4.2K views19 slides
Apache HBase Performance Tuning by
Apache HBase Performance TuningApache HBase Performance Tuning
Apache HBase Performance TuningLars Hofhansl
39.6K views54 slides
hbaseconasia2019 HBCK2: Concepts, trends, and recipes for fixing issues in HB... by
hbaseconasia2019 HBCK2: Concepts, trends, and recipes for fixing issues in HB...hbaseconasia2019 HBCK2: Concepts, trends, and recipes for fixing issues in HB...
hbaseconasia2019 HBCK2: Concepts, trends, and recipes for fixing issues in HB...Michael Stack
582 views16 slides
ORC File - Optimizing Your Big Data by
ORC File - Optimizing Your Big DataORC File - Optimizing Your Big Data
ORC File - Optimizing Your Big DataDataWorks Summit
11.6K views26 slides
Efficient Data Storage for Analytics with Apache Parquet 2.0 by
Efficient Data Storage for Analytics with Apache Parquet 2.0Efficient Data Storage for Analytics with Apache Parquet 2.0
Efficient Data Storage for Analytics with Apache Parquet 2.0Cloudera, Inc.
158.6K views40 slides
Getting Started with HBase by
Getting Started with HBaseGetting Started with HBase
Getting Started with HBaseCarol McDonald
4.8K views102 slides

More Related Content

What's hot

HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon by
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUponHBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUponCloudera, Inc.
13.1K views25 slides
Hive: Loading Data by
Hive: Loading DataHive: Loading Data
Hive: Loading DataBenjamin Leonhardi
34K views28 slides
HBaseCon 2015: Taming GC Pauses for Large Java Heap in HBase by
HBaseCon 2015: Taming GC Pauses for Large Java Heap in HBaseHBaseCon 2015: Taming GC Pauses for Large Java Heap in HBase
HBaseCon 2015: Taming GC Pauses for Large Java Heap in HBaseHBaseCon
14.6K views39 slides
Your 1st Ceph cluster by
Your 1st Ceph clusterYour 1st Ceph cluster
Your 1st Ceph clusterMirantis
22.5K views26 slides
HBaseCon 2015: HBase Performance Tuning @ Salesforce by
HBaseCon 2015: HBase Performance Tuning @ SalesforceHBaseCon 2015: HBase Performance Tuning @ Salesforce
HBaseCon 2015: HBase Performance Tuning @ SalesforceHBaseCon
6.1K views54 slides
HBase Low Latency by
HBase Low LatencyHBase Low Latency
HBase Low LatencyDataWorks Summit
5.1K views38 slides

What's hot(20)

HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon by Cloudera, Inc.
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUponHBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
Cloudera, Inc.13.1K views
HBaseCon 2015: Taming GC Pauses for Large Java Heap in HBase by HBaseCon
HBaseCon 2015: Taming GC Pauses for Large Java Heap in HBaseHBaseCon 2015: Taming GC Pauses for Large Java Heap in HBase
HBaseCon 2015: Taming GC Pauses for Large Java Heap in HBase
HBaseCon14.6K views
Your 1st Ceph cluster by Mirantis
Your 1st Ceph clusterYour 1st Ceph cluster
Your 1st Ceph cluster
Mirantis22.5K views
HBaseCon 2015: HBase Performance Tuning @ Salesforce by HBaseCon
HBaseCon 2015: HBase Performance Tuning @ SalesforceHBaseCon 2015: HBase Performance Tuning @ Salesforce
HBaseCon 2015: HBase Performance Tuning @ Salesforce
HBaseCon6.1K views
Scalability, Availability & Stability Patterns by Jonas Bonér
Scalability, Availability & Stability PatternsScalability, Availability & Stability Patterns
Scalability, Availability & Stability Patterns
Jonas Bonér515.7K views
Apache phoenix: Past, Present and Future of SQL over HBAse by enissoz
Apache phoenix: Past, Present and Future of SQL over HBAseApache phoenix: Past, Present and Future of SQL over HBAse
Apache phoenix: Past, Present and Future of SQL over HBAse
enissoz6.3K views
Facebook Messages & HBase by 强 王
Facebook Messages & HBaseFacebook Messages & HBase
Facebook Messages & HBase
强 王39.2K views
Tez Shuffle Handler: Shuffling at Scale with Apache Hadoop by DataWorks Summit
Tez Shuffle Handler: Shuffling at Scale with Apache HadoopTez Shuffle Handler: Shuffling at Scale with Apache Hadoop
Tez Shuffle Handler: Shuffling at Scale with Apache Hadoop
DataWorks Summit1.2K views
Webinar: 99 Ways to Enrich Streaming Data with Apache Flink - Konstantin Knauf by Ververica
Webinar: 99 Ways to Enrich Streaming Data with Apache Flink - Konstantin KnaufWebinar: 99 Ways to Enrich Streaming Data with Apache Flink - Konstantin Knauf
Webinar: 99 Ways to Enrich Streaming Data with Apache Flink - Konstantin Knauf
Ververica 2.2K views
[211] HBase 기반 검색 데이터 저장소 (공개용) by NAVER D2
[211] HBase 기반 검색 데이터 저장소 (공개용)[211] HBase 기반 검색 데이터 저장소 (공개용)
[211] HBase 기반 검색 데이터 저장소 (공개용)
NAVER D28.5K views
Windows 10 Nt Heap Exploitation (English version) by Angel Boy
Windows 10 Nt Heap Exploitation (English version)Windows 10 Nt Heap Exploitation (English version)
Windows 10 Nt Heap Exploitation (English version)
Angel Boy44.1K views
Distributed computing with Ray. Find your hyper-parameters, speed up your Pan... by Jan Margeta
Distributed computing with Ray. Find your hyper-parameters, speed up your Pan...Distributed computing with Ray. Find your hyper-parameters, speed up your Pan...
Distributed computing with Ray. Find your hyper-parameters, speed up your Pan...
Jan Margeta730 views
HBaseCon 2013: Compaction Improvements in Apache HBase by Cloudera, Inc.
HBaseCon 2013: Compaction Improvements in Apache HBaseHBaseCon 2013: Compaction Improvements in Apache HBase
HBaseCon 2013: Compaction Improvements in Apache HBase
Cloudera, Inc.19.1K views

Viewers also liked

HBase Application Performance Improvement by
HBase Application Performance ImprovementHBase Application Performance Improvement
HBase Application Performance ImprovementBiju Nair
23.5K views25 slides
HBase for Architects by
HBase for ArchitectsHBase for Architects
HBase for ArchitectsNick Dimiduk
33.7K views21 slides
HBase Advanced - Lars George by
HBase Advanced - Lars GeorgeHBase Advanced - Lars George
HBase Advanced - Lars GeorgeJAX London
9.9K views45 slides
Intro to HBase Internals & Schema Design (for HBase users) by
Intro to HBase Internals & Schema Design (for HBase users)Intro to HBase Internals & Schema Design (for HBase users)
Intro to HBase Internals & Schema Design (for HBase users)alexbaranau
28.1K views31 slides
HBase: Extreme Makeover by
HBase: Extreme MakeoverHBase: Extreme Makeover
HBase: Extreme MakeoverHBaseCon
3.3K views67 slides
Apache HBase Low Latency by
Apache HBase Low LatencyApache HBase Low Latency
Apache HBase Low LatencyNick Dimiduk
12K views38 slides

Viewers also liked(20)

HBase Application Performance Improvement by Biju Nair
HBase Application Performance ImprovementHBase Application Performance Improvement
HBase Application Performance Improvement
Biju Nair23.5K views
HBase for Architects by Nick Dimiduk
HBase for ArchitectsHBase for Architects
HBase for Architects
Nick Dimiduk33.7K views
HBase Advanced - Lars George by JAX London
HBase Advanced - Lars GeorgeHBase Advanced - Lars George
HBase Advanced - Lars George
JAX London9.9K views
Intro to HBase Internals & Schema Design (for HBase users) by alexbaranau
Intro to HBase Internals & Schema Design (for HBase users)Intro to HBase Internals & Schema Design (for HBase users)
Intro to HBase Internals & Schema Design (for HBase users)
alexbaranau28.1K views
HBase: Extreme Makeover by HBaseCon
HBase: Extreme MakeoverHBase: Extreme Makeover
HBase: Extreme Makeover
HBaseCon3.3K views
Apache HBase Low Latency by Nick Dimiduk
Apache HBase Low LatencyApache HBase Low Latency
Apache HBase Low Latency
Nick Dimiduk12K views
HBase HUG Presentation: Avoiding Full GCs with MemStore-Local Allocation Buffers by Cloudera, Inc.
HBase HUG Presentation: Avoiding Full GCs with MemStore-Local Allocation BuffersHBase HUG Presentation: Avoiding Full GCs with MemStore-Local Allocation Buffers
HBase HUG Presentation: Avoiding Full GCs with MemStore-Local Allocation Buffers
Cloudera, Inc.15.6K views
Hands-on-Lab: Adding Value to HBase with IBM InfoSphere BigInsights and BigSQL by Piotr Pruski
Hands-on-Lab: Adding Value to HBase with IBM InfoSphere BigInsights and BigSQLHands-on-Lab: Adding Value to HBase with IBM InfoSphere BigInsights and BigSQL
Hands-on-Lab: Adding Value to HBase with IBM InfoSphere BigInsights and BigSQL
Piotr Pruski4K views
Bring Cartography to the Cloud by Nick Dimiduk
Bring Cartography to the CloudBring Cartography to the Cloud
Bring Cartography to the Cloud
Nick Dimiduk2.7K views
HBase Low Latency, StrataNYC 2014 by Nick Dimiduk
HBase Low Latency, StrataNYC 2014HBase Low Latency, StrataNYC 2014
HBase Low Latency, StrataNYC 2014
Nick Dimiduk1.8K views
Apache Big Data EU 2015 - HBase by Nick Dimiduk
Apache Big Data EU 2015 - HBaseApache Big Data EU 2015 - HBase
Apache Big Data EU 2015 - HBase
Nick Dimiduk3.3K views
Introduction to Hadoop, HBase, and NoSQL by Nick Dimiduk
Introduction to Hadoop, HBase, and NoSQLIntroduction to Hadoop, HBase, and NoSQL
Introduction to Hadoop, HBase, and NoSQL
Nick Dimiduk7.5K views
HBase Client APIs (for webapps?) by Nick Dimiduk
HBase Client APIs (for webapps?)HBase Client APIs (for webapps?)
HBase Client APIs (for webapps?)
Nick Dimiduk9K views
Apache Big Data EU 2015 - Phoenix by Nick Dimiduk
Apache Big Data EU 2015 - PhoenixApache Big Data EU 2015 - Phoenix
Apache Big Data EU 2015 - Phoenix
Nick Dimiduk4.7K views
Hadoop World 2011: Advanced HBase Schema Design by Cloudera, Inc.
Hadoop World 2011: Advanced HBase Schema DesignHadoop World 2011: Advanced HBase Schema Design
Hadoop World 2011: Advanced HBase Schema Design
Cloudera, Inc.17.9K views
Apache HBase 1.0 Release by Nick Dimiduk
Apache HBase 1.0 ReleaseApache HBase 1.0 Release
Apache HBase 1.0 Release
Nick Dimiduk25.6K views
HBase: Where Online Meets Low Latency by HBaseCon
HBase: Where Online Meets Low LatencyHBase: Where Online Meets Low Latency
HBase: Where Online Meets Low Latency
HBaseCon4.7K views

Similar to HBase Blockcache 101

Large-scale Web Apps @ Pinterest by
Large-scale Web Apps @ PinterestLarge-scale Web Apps @ Pinterest
Large-scale Web Apps @ PinterestHBaseCon
4.1K views26 slides
004 architecture andadvanceduse by
004 architecture andadvanceduse004 architecture andadvanceduse
004 architecture andadvanceduseScott Miao
2.9K views78 slides
Kfs presentation by
Kfs presentationKfs presentation
Kfs presentationPetrovici Florin
459 views23 slides
HBaseConAsia2018 Track1-2: WALLess HBase with persistent memory devices by
HBaseConAsia2018 Track1-2: WALLess HBase with persistent memory devicesHBaseConAsia2018 Track1-2: WALLess HBase with persistent memory devices
HBaseConAsia2018 Track1-2: WALLess HBase with persistent memory devicesMichael Stack
1.5K views11 slides
Hadoop 24/7 by
Hadoop 24/7Hadoop 24/7
Hadoop 24/7Allen Wittenauer
2.6K views33 slides
HBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBase by
HBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBaseHBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBase
HBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBaseCloudera, Inc.
10.9K views33 slides

Similar to HBase Blockcache 101(20)

Large-scale Web Apps @ Pinterest by HBaseCon
Large-scale Web Apps @ PinterestLarge-scale Web Apps @ Pinterest
Large-scale Web Apps @ Pinterest
HBaseCon4.1K views
004 architecture andadvanceduse by Scott Miao
004 architecture andadvanceduse004 architecture andadvanceduse
004 architecture andadvanceduse
Scott Miao2.9K views
HBaseConAsia2018 Track1-2: WALLess HBase with persistent memory devices by Michael Stack
HBaseConAsia2018 Track1-2: WALLess HBase with persistent memory devicesHBaseConAsia2018 Track1-2: WALLess HBase with persistent memory devices
HBaseConAsia2018 Track1-2: WALLess HBase with persistent memory devices
Michael Stack1.5K views
HBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBase by Cloudera, Inc.
HBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBaseHBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBase
HBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBase
Cloudera, Inc.10.9K views
Digital Library Collection Management using HBase by HBaseCon
Digital Library Collection Management using HBaseDigital Library Collection Management using HBase
Digital Library Collection Management using HBase
HBaseCon3.1K views
Heapoff memory wtf by Olivier Lamy
Heapoff memory wtfHeapoff memory wtf
Heapoff memory wtf
Olivier Lamy2.1K views
Fabian Hueske – Juggling with Bits and Bytes by Flink Forward
Fabian Hueske – Juggling with Bits and BytesFabian Hueske – Juggling with Bits and Bytes
Fabian Hueske – Juggling with Bits and Bytes
Flink Forward7.4K views
Big Data and Hadoop - History, Technical Deep Dive, and Industry Trends by Esther Kundin
Big Data and Hadoop - History, Technical Deep Dive, and Industry TrendsBig Data and Hadoop - History, Technical Deep Dive, and Industry Trends
Big Data and Hadoop - History, Technical Deep Dive, and Industry Trends
Esther Kundin1.1K views
Elements of cache design by Rohail Butt
Elements of cache designElements of cache design
Elements of cache design
Rohail Butt5.9K views
AOS Lab 9: File system -- Of buffers, logs, and blocks by Zubair Nabi
AOS Lab 9: File system -- Of buffers, logs, and blocksAOS Lab 9: File system -- Of buffers, logs, and blocks
AOS Lab 9: File system -- Of buffers, logs, and blocks
Zubair Nabi1.5K views
Optimizing Latency-sensitive queries for Presto at Facebook: A Collaboration ... by Alluxio, Inc.
Optimizing Latency-sensitive queries for Presto at Facebook: A Collaboration ...Optimizing Latency-sensitive queries for Presto at Facebook: A Collaboration ...
Optimizing Latency-sensitive queries for Presto at Facebook: A Collaboration ...
Alluxio, Inc.120 views
CBlocks - Posix compliant files systems for HDFS by DataWorks Summit
CBlocks - Posix compliant files systems for HDFSCBlocks - Posix compliant files systems for HDFS
CBlocks - Posix compliant files systems for HDFS
DataWorks Summit682 views
HBase Status Report - Hadoop Summit Europe 2014 by larsgeorge
HBase Status Report - Hadoop Summit Europe 2014HBase Status Report - Hadoop Summit Europe 2014
HBase Status Report - Hadoop Summit Europe 2014
larsgeorge1.1K views
Ehcache3 — JSR-107 on steroids by Alex Snaps
Ehcache3 — JSR-107 on steroidsEhcache3 — JSR-107 on steroids
Ehcache3 — JSR-107 on steroids
Alex Snaps510 views
cache memory by NAHID HASAN
 cache memory cache memory
cache memory
NAHID HASAN1.8K views

Recently uploaded

Special_edition_innovator_2023.pdf by
Special_edition_innovator_2023.pdfSpecial_edition_innovator_2023.pdf
Special_edition_innovator_2023.pdfWillDavies22
14 views6 slides
Five Things You SHOULD Know About Postman by
Five Things You SHOULD Know About PostmanFive Things You SHOULD Know About Postman
Five Things You SHOULD Know About PostmanPostman
25 views43 slides
Throughput by
ThroughputThroughput
ThroughputMoisés Armani Ramírez
32 views11 slides
Voice Logger - Telephony Integration Solution at Aegis by
Voice Logger - Telephony Integration Solution at AegisVoice Logger - Telephony Integration Solution at Aegis
Voice Logger - Telephony Integration Solution at AegisNirmal Sharma
17 views1 slide
TouchLog: Finger Micro Gesture Recognition Using Photo-Reflective Sensors by
TouchLog: Finger Micro Gesture Recognition  Using Photo-Reflective SensorsTouchLog: Finger Micro Gesture Recognition  Using Photo-Reflective Sensors
TouchLog: Finger Micro Gesture Recognition Using Photo-Reflective Sensorssugiuralab
11 views15 slides
Data-centric AI and the convergence of data and model engineering: opportunit... by
Data-centric AI and the convergence of data and model engineering:opportunit...Data-centric AI and the convergence of data and model engineering:opportunit...
Data-centric AI and the convergence of data and model engineering: opportunit...Paolo Missier
29 views40 slides

Recently uploaded(20)

Special_edition_innovator_2023.pdf by WillDavies22
Special_edition_innovator_2023.pdfSpecial_edition_innovator_2023.pdf
Special_edition_innovator_2023.pdf
WillDavies2214 views
Five Things You SHOULD Know About Postman by Postman
Five Things You SHOULD Know About PostmanFive Things You SHOULD Know About Postman
Five Things You SHOULD Know About Postman
Postman25 views
Voice Logger - Telephony Integration Solution at Aegis by Nirmal Sharma
Voice Logger - Telephony Integration Solution at AegisVoice Logger - Telephony Integration Solution at Aegis
Voice Logger - Telephony Integration Solution at Aegis
Nirmal Sharma17 views
TouchLog: Finger Micro Gesture Recognition Using Photo-Reflective Sensors by sugiuralab
TouchLog: Finger Micro Gesture Recognition  Using Photo-Reflective SensorsTouchLog: Finger Micro Gesture Recognition  Using Photo-Reflective Sensors
TouchLog: Finger Micro Gesture Recognition Using Photo-Reflective Sensors
sugiuralab11 views
Data-centric AI and the convergence of data and model engineering: opportunit... by Paolo Missier
Data-centric AI and the convergence of data and model engineering:opportunit...Data-centric AI and the convergence of data and model engineering:opportunit...
Data-centric AI and the convergence of data and model engineering: opportunit...
Paolo Missier29 views
Black and White Modern Science Presentation.pptx by maryamkhalid2916
Black and White Modern Science Presentation.pptxBlack and White Modern Science Presentation.pptx
Black and White Modern Science Presentation.pptx
maryamkhalid291614 views
Future of Learning - Khoong Chan Meng by NUS-ISS
Future of Learning - Khoong Chan MengFuture of Learning - Khoong Chan Meng
Future of Learning - Khoong Chan Meng
NUS-ISS31 views
Understanding GenAI/LLM and What is Google Offering - Felix Goh by NUS-ISS
Understanding GenAI/LLM and What is Google Offering - Felix GohUnderstanding GenAI/LLM and What is Google Offering - Felix Goh
Understanding GenAI/LLM and What is Google Offering - Felix Goh
NUS-ISS39 views
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV by Splunk
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV
Splunk86 views
.conf Go 2023 - Data analysis as a routine by Splunk
.conf Go 2023 - Data analysis as a routine.conf Go 2023 - Data analysis as a routine
.conf Go 2023 - Data analysis as a routine
Splunk90 views
Beyond the Hype: What Generative AI Means for the Future of Work - Damien Cum... by NUS-ISS
Beyond the Hype: What Generative AI Means for the Future of Work - Damien Cum...Beyond the Hype: What Generative AI Means for the Future of Work - Damien Cum...
Beyond the Hype: What Generative AI Means for the Future of Work - Damien Cum...
NUS-ISS28 views
Business Analyst Series 2023 - Week 3 Session 5 by DianaGray10
Business Analyst Series 2023 -  Week 3 Session 5Business Analyst Series 2023 -  Week 3 Session 5
Business Analyst Series 2023 - Week 3 Session 5
DianaGray10165 views
Perth MeetUp November 2023 by Michael Price
Perth MeetUp November 2023 Perth MeetUp November 2023
Perth MeetUp November 2023
Michael Price12 views
How the World's Leading Independent Automotive Distributor is Reinventing Its... by NUS-ISS
How the World's Leading Independent Automotive Distributor is Reinventing Its...How the World's Leading Independent Automotive Distributor is Reinventing Its...
How the World's Leading Independent Automotive Distributor is Reinventing Its...
NUS-ISS15 views
Web Dev - 1 PPT.pdf by gdsczhcet
Web Dev - 1 PPT.pdfWeb Dev - 1 PPT.pdf
Web Dev - 1 PPT.pdf
gdsczhcet52 views

HBase Blockcache 101

  • 1. HBase  BlockCache  101   Nick  Dimiduk,  Hortonworks              @xefyr                n10k.com  
  • 2. Agenda   •  What  is  a  BlockCache?   •  Why  cache  blocks?   •  Available  BlockCaches   •  ConfiguraFons  
  • 3. RegionServer (HBase) DataNode (Hadoop DFS) HLog (WAL) HRegion HStore StoreFile HFile StoreFile HFile MemStore ... ... HStore BlockCache HRegion ... HStoreHStore ... 1 5 2 3 3 2 4 Read  Path  Refresher   BlockCache  
  • 4. What  is  a  block?   HFile  v2  Format   HFile  v2  format  figure  reproduced  from  MaMeo  Bertozzi,  "Apache   HBase  I/O  –  HFile”,  hMp://blog.cloudera.com/blog/2012/06/hbase-­‐io-­‐ hfile-­‐input-­‐output/  
  • 5. Why  cache  blocks?  Storage hierarchy: a different view Dean/2009   Memory  is  100000x   faster  than  disk!   Disk  seek  =  10ms  
  • 6. LruBlockCache! •  On-­‐heap  ConcurrentHashMap! •  3  block  prioriFes,  based  on  access  frequency   –  Single   –  MulF   –  In-­‐memory   •  Background  evicFon  thread   •  HBASE-­‐1460,  0.20  
  • 7. LruBlockCache  ConfiguraFon   •  Size  as  pct  of  heap   –  hfile.block.cache.size   –  default:  0.4   •  Single  access  Priority  size   –  hbase.lru.blockcache.single.percentage   –  default:  0.25   •  MulF-­‐access  priority  size   –  hbase.lru.blockcache.multi.percentage   –  default:  0.5   •  In-­‐memory  priority  size   –  hbase.lru.blockcache.memory.percentage   –  default:  0.25  
  • 8. SlabCache! •  L2  cache;  fronted  by  LruBlockCache  (L1)   •  DoubleBlockCache  caches  blocks  in  both  L1  and  L2   •  Allocates  porFons  of  memory  with  target  block  size   •  DirectByteBuffers  manage  off-­‐heap  memory   •  Blocks  copied  on  and  off  heap   •  HBASE-­‐4027,  0.92  
  • 9. SlabCache  ConfiguraFon   •  SlabCache  size   –  hbase.offheapcache.percentage! •  Target  Block  size   –  hbase.offheapcache.minblocksize! –  default:  64k   •  Slab  proporFons   –  hbase.offheapcache.slab.proportions! –  default:  0.8,  0.2   •  Slab  target  block  sizes   –  hbase.offheapcache.slab.sizes! –  default:  64k,  128k  
  • 10. BucketCache! •  L2  cache;  fronted  by  LruBlockCache  (L1)   •  CombinedBlockCache  caches  data  blocks  in  L2,  everything  else  in  L1   •  Blocks  evicted  from  L1  are  promoted  to  L2   •  Allocates  porFons  of  space  with  target  block  size   •  Three  operaFng  modes:   –  onheap  –  HeapByteBuffers  manage  on-­‐heap  memory  space   –  offheap  –  DirectByteBuffers  manage  off-­‐heap  memory  space   –  file  –  RandomAccessFile  manages  file-­‐backed  space   •  Blocks  copied  on  and  off  heap   •  HBASE-­‐7404,  0.96  
  • 11. BucketCache  ConfiguraFons   •  OperaFng  mode   –  hbase.bucketcache.ioengine! •  Total  cache  size  as  pct  of  heap  OR  size  in  MB:   –  hbase.bucketcache.size! •  Size  of  L1  as  pct  of  total  cache  size:   –  hbase.bucketcache.percentage.in.combinedcache! –  default:  0.9   •  Target  Block  size   –  hbase.offheapcache.minblocksize! –  default:  64k   •  Bucket  target  block  sizes   –  hbase.bucketcache.bucket.sizes! –  default:  14  sizes,  clustered  around  64k  
  • 12. BlockCache  Showdown   •  Compare  performance:   –  Total  cache  size   –  Cache  size  vs.  Data  size     n10k.com/blog/ blockcache-­‐showdown/   L2  off-­‐heap  BucketCache   makes  a  strong  showing  
  • 13. Thanks!   M A N N I N G Nick Dimiduk Amandeep Khurana FOREWORD BY Michael Stack hbaseinacFon.com   Nick  Dimiduk              github.com/ndimiduk              @xefyr              n10k.com   n10k.com/blog/blockcache-­‐101