SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.
SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.
Successfully reported this slideshow.
Activate your 14 day free trial to unlock unlimited reading.
Apache HBase, Accelerated: In-Memory Flush and Compaction
Real-time HBase application performance depends critically on the amount of I/O in the datapath. Here we’ll describe an optimization of HBase for high-churn applications that frequently insert/update/delete the same keys, such as for high-speed queuing and e-commerce.
Real-time HBase application performance depends critically on the amount of I/O in the datapath. Here we’ll describe an optimization of HBase for high-churn applications that frequently insert/update/delete the same keys, such as for high-speed queuing and e-commerce.
Apache HBase, Accelerated: In-Memory Flush and Compaction
1.
HBase Accelerated:
In-Memory Flush and Compaction
E s h c a r H i l l e l , A n a s t a s i a B r a g i n s k y , E d w a r d B o r t n i k o v ⎪ H B a s e C o n , S a n
F r a n c i s c o , M a y 2 4 , 2 0 1 6
3.
Motivation: Dynamic Content Processing on Top of HBase
3
Real-time content processing pipelines
› Store intermediate results in persistent map
› Notification mechanism is prevalent
Storage and notifications on the same platform
Sieve – Yahoo’s real-time content management platform
Crawl
Docpro
c
Link
Analysis Queue
Crawl
schedule
Content
Queue
Links
Serving
Apache Storm
Apache HBase
4.
Notification Mechanism is Like a Sliding Window
4
Small working set but not necessarily FIFO queue
Short life-cycle delete message after processing it
High-churn workload message state can be updated
Frequent scans to consume message
5.
HBase Accelerated: Mission Definition
5
Goal:
Real-time performance in persistent KV-stores
How:
Use less in-memory space less I/O
6.
HBase Accelerated: Two Base Ideas
6
In-Memory Compaction
Exploit redundancies in the workload to eliminate duplicates in memory
Gain is proportional to the duplicate ratio
In-Memory Index Reduction
Reduce the index memory footprint, less overhead per cell
Gain is proportional to the cell size
Prolong in-memory lifetime, before flushing to disk
Reduce amount of new files
Reduce write amplification effect (overall I/O)
Reduce retrieval latencies
8.
Random writes absorbed in active segment (Cm)
When active segment is full
› Becomes immutable segment (snapshot)
› A new mutable (active) segment serves writes
› Flushed to disk, truncate WAL
On-Disk compaction reads a few files, merge-sorts them, writes back new files
9
HBase Writes
C’m
flush
memory HDFS
Cm
prepare-for-flush
Cd
WAL
write
9.
10
HBase Reads
C’m
memory HDFS
Cm
Cd
Read
Random reads from Cm or C’m or Cd (Block Cache)
When data piles-up on disk
› Hit ratio drops
› Retrieval latency up
Compaction re-writes small files into fewer bigger files
› Causes replication-related network and disk IO
Block cache
10.
12
Hbase In-Memory Compaction
C’m
flush
memory HDFS
Cm
in-memory-flush
Cd
WAL
Block cache
Compaction
pipeline
11.
New compaction pipeline
› Active segment flushed to pipeline
› Pipeline segments compacted in memory
› Flush to disk only when needed
13
New Design: In-Memory Flush and Compaction
C’m
flush-to-disk
memory HDFS
Cm
in-memory-flush
Cd
prepare-for-flush
Compaction
pipeline
WAL
Block
cache
memory
12.
Trade read cache (BlockCache) for write cache (compaction pipeline)
14
New Design: In-Memory Flush and Compaction
write read
cache cache
C’m
flush-to-disk
memory HDFS
Cm
in-memory-flush
Cd
prepare-for-flush
Compaction
pipeline
WAL
Block
cache
memory
13.
15
New Design: In-Memory Flush and Compaction
CPU
IO
C’m
flush-to-disk
memory HDFS
Cm
in-memory-flush
Cd
prepare-for-flush
Compaction
pipeline
WAL
Block
cache
memory
Trade read cache (BlockCache) for write cache (compaction pipeline)
More CPU cycles for less I/O
24.
26
New Design: Effective in-memory representation
C’m
flush-to-disk
memory HDFS
Cm
in-memory-flush
Cd
prepare-for-flush
Compaction
pipeline
WAL
Block
cache
memory
25.
27
New Design: Effective in-memory representation
C’m
flush-to-disk
memory HDFS
Cm
in-memory-flush
Cd
prepare-for-flush
Compaction
pipeline
WAL
Block
cache
memory
26.
28
Segment for dynamic updates
Cell
A
Cell G
Cell F
Cell
BCell D Cell E
MSLAB
27.
Exploit the Immutability of Segment after Compaction
29
Current design
› Data stored in flat buffers, index is a skip-list
› All memory allocated on-heap
New Design: Flat layout for immutable segments index
› Less overhead per cell
› Manage (allocate, store, release) data buffers off-heap
Pros
› Locality in access to index
› Reduce memory fragmentation
› Significantly reduce GC work
› Better utilization of memory and CPU
28.
30
Read-Only Segment
Cell
A
Cell G
Cell F
Cell
BCell D Cell E
MSLAB
29.
31
New Design: Effective in-memory representation
C’m
flush-to-disk
memory HDFS
Cm
in-memory-flush
Cd
prepare-for-flush
Compaction
pipeline
WAL
memory
Are there redundancies
to compact?
yesno
Flatten the index –
less overhead per
cell
Flatten the index &
compact – less cells
& less overhead per
cell
34.
Summary
36
Feature intended for HBase 2.0.0
New design pros over default implementation
› Predictable retrieval latency by serving (mainly) from memory
› Less compaction on disk reduces write amplification effect
› Less disk I/O and network traffic reduces load on HDFS
› New space efficient index representation
We would like to thank the reviewers
› Michael Stack, Anoop Sam John, Ramkrishna s. Vasudevan, Ted Yu