Apache HBase, Accelerated: In-Memory Flush and Compaction
•
4 likes•2,505 views
Eshcar Hillel and Anastasia Braginsky (Yahoo!) Real-time HBase application performance depends critically on the amount of I/O in the datapath. Here we’ll describe an optimization of HBase for high-churn applications that frequently insert/update/delete the same keys, such as for high-speed queuing and e-commerce.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Apache HBase, Accelerated: In-Memory Flush and Compaction
Similar to Apache HBase, Accelerated: In-Memory Flush and Compaction (20)
More from HBaseCon
More from HBaseCon (20)
Recently uploaded
Recently uploaded (20)
Apache HBase, Accelerated: In-Memory Flush and Compaction
- 1. HBase Accelerated: In-Memory Flush and Compaction E s h c a r H i l l e l , A n a s t a s i a B r a g i n s k y , E d w a r d B o r t n i k o v ⎪ H B a s e C o n , S a n F r a n c i s c o , M a y 2 4 , 2 0 1 6
- 2. Outline 2 Background In-Memory Compaction › Design & Evaluation In-Memory Index Reduction › Design & Evaluation
- 3. Motivation: Dynamic Content Processing on Top of HBase 3 Real-time content processing pipelines › Store intermediate results in persistent map › Notification mechanism is prevalent Storage and notifications on the same platform Sieve – Yahoo’s real-time content management platform Crawl Docpro c Link Analysis Queue Crawl schedule Content Queue Links Serving Apache Storm Apache HBase
- 4. Notification Mechanism is Like a Sliding Window 4 Small working set but not necessarily FIFO queue Short life-cycle delete message after processing it High-churn workload message state can be updated Frequent scans to consume message
- 5. HBase Accelerated: Mission Definition 5 Goal: Real-time performance in persistent KV-stores How: Use less in-memory space less I/O
- 6. HBase Accelerated: Two Base Ideas 6 In-Memory Compaction Exploit redundancies in the workload to eliminate duplicates in memory Gain is proportional to the duplicate ratio In-Memory Index Reduction Reduce the index memory footprint, less overhead per cell Gain is proportional to the cell size Prolong in-memory lifetime, before flushing to disk Reduce amount of new files Reduce write amplification effect (overall I/O) Reduce retrieval latencies
- 7. Outline 7 Background In-Memory Compaction › Design & Evaluation In-Memory Index Reduction › Design & Evaluation In-Memory Compaction Design
- 8. Random writes absorbed in active segment (Cm) When active segment is full › Becomes immutable segment (snapshot) › A new mutable (active) segment serves writes › Flushed to disk, truncate WAL On-Disk compaction reads a few files, merge-sorts them, writes back new files 9 HBase Writes C’m flush memory HDFS Cm prepare-for-flush Cd WAL write
- 9. 10 HBase Reads C’m memory HDFS Cm Cd Read Random reads from Cm or C’m or Cd (Block Cache) When data piles-up on disk › Hit ratio drops › Retrieval latency up Compaction re-writes small files into fewer bigger files › Causes replication-related network and disk IO Block cache
- 10. 12 Hbase In-Memory Compaction C’m flush memory HDFS Cm in-memory-flush Cd WAL Block cache Compaction pipeline
- 11. New compaction pipeline › Active segment flushed to pipeline › Pipeline segments compacted in memory › Flush to disk only when needed 13 New Design: In-Memory Flush and Compaction C’m flush-to-disk memory HDFS Cm in-memory-flush Cd prepare-for-flush Compaction pipeline WAL Block cache memory
- 12. Trade read cache (BlockCache) for write cache (compaction pipeline) 14 New Design: In-Memory Flush and Compaction write read cache cache C’m flush-to-disk memory HDFS Cm in-memory-flush Cd prepare-for-flush Compaction pipeline WAL Block cache memory
- 13. 15 New Design: In-Memory Flush and Compaction CPU IO C’m flush-to-disk memory HDFS Cm in-memory-flush Cd prepare-for-flush Compaction pipeline WAL Block cache memory Trade read cache (BlockCache) for write cache (compaction pipeline) More CPU cycles for less I/O
- 14. Outline 16 Background In-Memory Compaction › Design & Evaluation In-Memory Index Reduction › Design & Evaluation
- 15. Evaluation Settings: In-Memory Working Set 17 YCSB: compares compacting vs. default memstore Small cluster: 3 HDFS nodes on a single rack, 1 RS › 1GB heap space, MSLAB enabled (2MB chunks) › Default: 128MB flush size, 100MB block-cache › Compacting: 192MB flush size, 36MB block-cache High-churn workload, small working set › 128,000 records, 1KB value field › 10 threads running 5 millions operations, various key distributions › 50% reads 50% updates, target 1000ops › 1% (long) scans 99% updates, target 500ops Measure average latency over time › Latencies accumulated over intervals of 10 seconds
- 16. Evaluation Results: Read Latency (Zipfian Distribution) 18 Flush to disk Compaction Data fits into cache
- 17. Evaluation Results: Read Latency (Uniform Distribution) 19 Region split
- 18. Evaluation Results: Scan Latency (Uniform Distribution) 20
- 19. Evaluation Settings: Handling Tombstones 21 YCSB: compares compacting vs. default memstore Small cluster: 3 HDFS nodes on a single rack, 1 RS › 1GB heap space, MSLAB enabled (2MB chunks), 128MB flush size, 64MB block-cache › Default: Minimum 4 files for compaction › Compaction: Minimum 2 files for compaction High-churn workload, small working set with deletes › 128,000 records, 1KB value field › 10 threads running 5 millions operations, various key distributions › 40% reads 40% updates 20% deletes (with 50,000 updates head start), target 1000ops › 1% (long) scans 66% updates 33% deletes (with head start), target 500ops Measure average latency over time › Latencies accumulated over intervals of 10 seconds
- 20. Evaluation Results: Read Latency (Zipfian Distribution) 22 (total 2 flushes and 1 disk compactions) (total 15 flushes and 4 disk compactions)
- 21. Evaluation Results: Read Latency (Uniform Distribution) 23 (total 3 flushes and 2 disk compactions) (total 15 flushes and 4 disk compactions)
- 22. Evaluation Results: Scan Latency (Zipfian Distribution) 24
- 23. Outline 25 Background In-Memory Compaction › Design & Evaluation In-Memory Index Reduction › Design & Evaluation In-Memory Index Reduction Design
- 24. 26 New Design: Effective in-memory representation C’m flush-to-disk memory HDFS Cm in-memory-flush Cd prepare-for-flush Compaction pipeline WAL Block cache memory
- 25. 27 New Design: Effective in-memory representation C’m flush-to-disk memory HDFS Cm in-memory-flush Cd prepare-for-flush Compaction pipeline WAL Block cache memory
- 26. 28 Segment for dynamic updates Cell A Cell G Cell F Cell BCell D Cell E MSLAB
- 27. Exploit the Immutability of Segment after Compaction 29 Current design › Data stored in flat buffers, index is a skip-list › All memory allocated on-heap New Design: Flat layout for immutable segments index › Less overhead per cell › Manage (allocate, store, release) data buffers off-heap Pros › Locality in access to index › Reduce memory fragmentation › Significantly reduce GC work › Better utilization of memory and CPU
- 28. 30 Read-Only Segment Cell A Cell G Cell F Cell BCell D Cell E MSLAB
- 29. 31 New Design: Effective in-memory representation C’m flush-to-disk memory HDFS Cm in-memory-flush Cd prepare-for-flush Compaction pipeline WAL memory Are there redundancies to compact? yesno Flatten the index – less overhead per cell Flatten the index & compact – less cells & less overhead per cell
- 30. Evaluation Results: Read Latency 1K Cell (Small Cache) 32 0 500 1000 1500 2000 2500 10 110 210 310 410 510 610 710 810 910 1010 1110 1210 1310 1410 1510 1610 1710 1810 1910 2010 2110 2210 2310 2410 2510 2610 2710 2810 2910 3010 3110 3210 3310 3410 3510 3610 3710 3810 3910 4010 4110 4210 4310 4410 Latency(us) Timeline (seconds) Uniform (Reads 50% - Writes 50%) - Read Latency skip-list based compaction cell-array based compaction
- 31. Evaluation Results: Read Latency 100Byte Cell (Small Cache) 33 0 200 400 600 800 1000 1200 10 140 270 400 530 660 790 920 1050 1180 1310 1440 1570 1700 1830 1960 2090 2220 2350 2480 2610 2740 2870 3000 3130 3260 3390 3520 3650 3780 3910 4040 4170 4300 4430 4560 4690 4820 4950 5080 5210 5340 5470 5600 Latency(us) Timeline (seconds) Zipfian (Reads 50% - Writes 50%) - Read Latency skip-list based compaction cell-array based compaction
- 32. Evaluation Results: Scan Latency (Uniform Distribution) 34 0 50000 100000 150000 200000 250000 10 250 490 730 970 1210 1450 1690 1930 2170 2410 2650 2890 3130 3370 3610 3850 4090 4330 4570 4810 5050 5290 5530 5770 6010 6250 6490 6730 6970 7210 7450 7690 7930 8170 8410 8650 8890 9130 9370 9610 9850 10090 10330 10570 10810 11050 11290 11530 11770 12010 12250 12490 12730 12970 13210 13450 13690 13930 Latency(us) Timeline (seconds) skip-list based compaction cell-array based compaction
- 33. Status Umbrella Jira HBASE-14918 35 HBASE-14919 HBASE-15016 HBASE-15359 Infrastructure refactoring › Status: committed HBASE-14920 new compacting memstore › Status: pre-commit HBASE-14921 memory optimizations (memory layout, off-heaping) › Status: under code review
- 34. Summary 36 Feature intended for HBase 2.0.0 New design pros over default implementation › Predictable retrieval latency by serving (mainly) from memory › Less compaction on disk reduces write amplification effect › Less disk I/O and network traffic reduces load on HDFS › New space efficient index representation We would like to thank the reviewers › Michael Stack, Anoop Sam John, Ramkrishna s. Vasudevan, Ted Yu
- 36. 38 Evaluation Results: Write Latency