CCSMap is a new data structure introduced by Alibaba to improve the performance of HBase. It aims to reduce the overhead of the default Java ConcurrentSkipListMap (CSLM) data structure and improve young garbage collection times. CCSMap chunks data into fixed size blocks for better memory management and uses direct pointers between nodes for faster access. It also provides various configuration options. Alibaba has achieved significant performance gains using CCSMap in HBase, including reduced young GC times, and it continues working to integrate CCSMap further and add new features.

1. hosted by
Use CCSMap to Improve HBase YGC Time &
Efforts on SLA improvements from Alibaba search in 2018
Chance Li, Xiang Wang
Lijin Bin

2. hosted by
Agenda 01
02
04
03
Why we need CCSMap
CCSMap structure
How to use CCSMap
Future work
Use CCSMap to Improve HBase YGC Time

3. hosted by
Why we need CCSMap
1
2
3
4
Overhead of JDK CSLM
Needs 3 objects to store one key-
value: index, Node and Cell, plenty of
objects required than those for real
data.
Memory fragment
Need SLAB to prevent CSLM from
fragment.
Heap is huge
Big memory preserved for HBase,
BucketCache for read path but
what about write path
CSLM not GC-friendly
Thousands of millions objects,
long time to scan card table/old
space leads to slow YGC
Overview

4. hosted by
Why we need CCSMap
Anatomy of JDK CSLM
title
Index: Almost
¼ of Nodes"
Nodes"
Extra Objects
Overhead 40B
per Index"
Overhead 40B
per Node"
Memory overhead
Extra 6.25M
objects"
250M memory
overhead"
Example of 5M cell
text" text" text" text" text"

5. hosted by
CCSMap Structure
Anatomy of CCSMap
title
Thousands of
Chunks(4M)"
Multi nodes in a
chunk"
Self-managed memory
Overhead 8B
each Index"
Overhead 16B
per Node Meta"
Memory overhead
No Extra
objects"
90M memory
overhead"
Example of 5M cell
text" text" text" text" text"

6. hosted by
CCSMap Structure
Overall look
Maps
BaseCCSMap<K,V
>"
BaseTwinCCSMap<K
>"
CellCCSMap"
ChunkPool
CCSMapCellCom
paratorDefault"
CCSMapCellComp
artorDirectly"
INodeComparator
CellSerde"
ISerde
Reuse " individual"
Configuartion
Global "
On-heap/off-heap
pooled chunk"
Special for big
object"
Space efficiency" Big chunk(4M)"
Meta" Data"
Direct
Pointer"
Two Type
of data"
align"
Data Structure

7. hosted by
CCSMap Structure
4
Data structure of Node
For Hbase, key and value are all
Cell, so data is byte[] of Cell, and
we can deserialize the data to
Cell(ByteBufferKeyValue) directly

8. hosted by
CCSMap Structure
1. CCSMapMemstore
2. ImmutableSegmentOnCCSMap
3. MutableSegmentOnCCSMap
4. MemstoreLABProxyForCCSMap
4
Classes for HBase internal

9. hosted by
How to use CCSMap
To enable CCSMap
CCSMap will be enable by default
after turning off CompactingMemstore
on-heap or off-heap
CCSMap individual configuration, or
original configuration used by
DefaultMemstore
Other configurations
Chunk pool capacity, Chunk size,
count of pooled chunk to be initialized

10. hosted by
Before using CCSMap

11. hosted by
After using CCSMap

12. hosted by
Future work
To support in-memory flush/compaction
Combine CompactingMemstore
To improve performance of put with dense columns
To improve performance of get with dense columns,
especially hot key
To support on Persistent Memory
To further improve ability of Memstore
4
01
02

13. hosted by
Agenda 01
02
05
03
Client Connection Flood
HDFS Failure Affection
Disaster Recovery
Dense Columns
Efforts on SLA improvements
06 Client Meta Acess Flood
04 Stall Server

14. hosted by
Client Connection Flood
² Separate client and server ZK quorum to avoid client connection boost exhausted zookeeper and
cause RegionServer ZK session loss(HBASE-20159)
ª Configurations to enable this feature
• hbase.client.zookeeper.quorum
• hbase.client.zookeeper.property.clientPort
• hbase.client.zookeeper.observer.mode

15. hosted by
HDFS Failure Affection
² Allow RegionServer to live during HDFS failure(HBASE-20156)
ª Not abort when HDFS failure
• Check FS and retry if flush fail
• Postpone WAL rolling when FileSystem not available
ª RegionServer service will recover after HDFS comes back
ª Upstreaming in progress

16. hosted by
Disaster Recovery
² Reduce ZK request to prevent ZK suspend during disaster recovery (HBASE-19290)
ª Before: request flood to ZK during disaster recovery
• Get all SplitTasks from splitLogZNode
• For each SplitTask
• getChildren of rsZNode’s to get number of available RSs
• try grab SplitTask’s lock break if no splitter available
• Try to grab task if no splitter available
ª After
• getChildren of rsZNode’s to get number of available RSs
• For each SplitTask: try grab SplitTask’s lock break if no splitter available
• Won’t try grab task if no splitter available

17. hosted by
Stall Server
² Enhance RegionServer self health check to avoid stall server (HBASE-20158)
ª Before
• Heartbeat still works when resource exhausted, RS regarded as alive but inaccessible
• Request hang on Disk failure cannot be captured by existing metrics
ª After (w/ DirectHealthChecker)
• Pre-launched chore thread, not affected if physical resource exhausted
• Periodically pick some regions on the RS and send request to itself
• Shortcut requests, don’t need to access zookeeper/master/meta
ª Upstreaming in progress

18. hosted by
Dense Columns
² Limit concurrency of put with dense columns to prevent write handler
exhausted(HBASE-19389)
ª MemStore insertion is at KV granularity, dense columns will cause CPU bottleneck
ª We must limit the concurrency of such puts
ª Configurations
• hbase.region.store.parallel.put.limit.min.column.count
• hbase.region.store.parallel.put.limit

19. hosted by
Client Meta Acess Flood
² Separate client and server request of meta (to be upstreamed)

20. hosted by
Thanks
Personal WechatHBase Dingding Group