This document proposes a read-write split near-line data loading method and architecture to: - Increase data loading performance by separating write operations from read operations. A WriteServer handles write requests and loads data to HDFS to be read from by RegionServers. - Control resources used by write operations to ensure read operations are not starved of resources like CPU, network, disk I/O, and handlers. - Provide an architecture corresponding to Kafka and HDFS for streaming data from Kafka to HDFS to be loaded into HBase in a delayed manner. - Include optimizations like task balancing across WriteServer slaves, prioritized compaction of small files, and customizable storage engines. - Report test results showing one Write

1. Large scale data near-line loading
method and architecture
FiberHome Telecommunication
2017-7-19

2. /usr/bin/whoami
Shuaifeng Zhou(周帅锋):
• Big data research and development director ( Fiberhome 2013-)
• Software engineer (Huawei 2007-2013)
• Use and contribute to HBase since 2009
• sfzhou1791@fiberhome.com

3. Motivation1
Contents
Solution2
Optimization3
Tests4
Summarize5

4. HBase Realtime Data Loading
 WAL/Flush/Compact
Triple IO pressure
 Read/Write operations
share resource:
 Cpu
 Network
 Disk IO
 Handler
 Read performance
decrease too much when
write load is heavy

5. Why near-line data loading?
DelayScale
ReliableResource
Large scale data loading reliably with acceptable time delay and resource occupation
Billions write ops per
region server one day
Usually, several minutes
delay is acceptable for
customers
Resource occupied can
be limited under an
acceptable level
Write op can be
repeated
Optimistic failure
handling
HBase

6. Motivation1
Contents
Solution2
Optimization3
Tests4
Summarize5

7. Read-Write split data loading
 Independent WriterServer to
handle put request
 RegionServer only handle
read request
 WriteServer write HFile on
HDFS, send do-bulkload
operation.
 Several minutes delay
between put and data
readable.

8. Architecture
Kafka
WriteServer Master HMaster
HDFS
WriteServer
Slave
WriteServer
Slave
RegionServerRegionServer
Data
Stream
Topic
Discovery
Contral Message Contral Message
Read Write Read/Write

9. WriteServer Master
Task Management
• Create new loading tasks
every five minutes or every
10,000 records
• Find a slave to load the task
• Task status control
Topic Management
• Discover new kafka topics
• Receive loading request
• Loading records statistic
Slaves Management
• Slave status report to
master
• Balance
• failover

10. WriteServer Slave

11. Failure Handling
Meta Data
based
Failure
Handling
Recover: Redo failed tasks when slave
down or master restart.
Task Meta Data is constructed when a
task is created by master, and change
status to succeed when slave finish the
task.
Task Meta Data is the descricption info of
a task, include the topic, partitions, start
and end offset, status. Stored on disk.

12. Motivation1
Contents
Solution2
Optimization3
Tests4
Summarize5

13. Balance
Load balance according tasks:
• Send new tasks to slaves with less tasks on handling
• Try to send tasks of one topic to a few fixed slaves
−avoid one region open everywhere
−Less region open, less small files
• Keep region opened for a while, even there are no tasks
− avoid region open/close too frequently

14. Compact
2
1
3 4 6
5
7 8
9
• Small files with higher priority
• Avoid one large file together with many small
files compact again and again
compact compact

15. StoreEngine
Customized store engine:
• organize store files in two queues
−one can be read and compact
−The other can only be compact
−If there are too many files, new file will not be readable
until they are compact
• Some new files discovered later better than all files can not
be read before time out
− Occasionally data explosion can be handled
− Region need split
− “Hot key” should be handled

16. HDFS Heterogeneous Storage Usage
• Use SSD storage as WriteServer tmp dir
• Use SATA as HBase data dir storage
− WriteServer write HFile on SSD
− Load HFile to HBase(Only move)
− Change to SATA storage after compact by regionServer
HDFS
SSD Storage SATA Storagecompact
WriteServer RegionServer

17. Resource Control
Resource used by WriteServer should be
controllable:
• Memory:
−JVM parameters 30~50GB memory
−Large Memory Store will avoid small files
−Too Large memory store will cause gc problems
• CPU:
− Slave can use 80% cpu cores at most
− Compare to real-time data load, a big optimize is we can control
the cpu occupation by write operations.

18. Motivation1
Contents
Solution2
Optimization3
Tests4
Summarize5

19. Loading Performance
WriterServer Slave
CPU Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
Memory 128G
Disk 1TB SSD * 4
Network 10GE
Record size 1KB
Compress Snappy
Performance 300,000 records/s
One WriteServer slave can match 5 RegionServer’s loading requirements before
RegionServer reach compact limitation.

20. Resource Performance
0
20
40
60
80
100
13:16
13:22
13:29
13:36
13:42
13:49
13:55
14:02
14:08
14:15
14:21
14:28
14:34
14:41
14:47
14:54
15:00
15:07
15:13
15:20
15:26
15:33
15:39
15:46
15:52
15:59
16:05
16:12
16:18
16:25
16:31
CPU Total WS-Slave5 – 2017/2/17
User% Sys% Wait%
-400
-200
0
200
400
600
13:19
13:25
13:32
13:38
13:45
13:51
13:58
14:04
14:11
14:17
14:24
14:30
14:37
14:43
14:50
14:56
15:03
15:09
15:16
15:22
15:29
15:35
15:42
15:48
15:55
16:01
16:08
16:14
16:21
16:27
16:34
Network I/O WS-Slave5 (MB/s) - 2017/2/17
Total-Read Total-Write (-ve)
0
500
1000
1500
2000
0
100
200
300
400
13:19
13:27
13:34
13:42
13:50
13:57
14:05
14:13
14:20
14:28
14:36
14:43
14:51
14:59
15:06
15:14
15:22
15:29
15:37
15:45
15:52
16:00
16:08
16:15
16:23
16:31
IO/sec
MB/sec
Disk total MB/s WS-Slave5- 2017/2/17
Disk Read KB/s Disk Write KB/s IO/sec
Memory
JVM: aways use memory as much as
assigned
GC: config gc policy to avoid full gc.

21. Motivation1
Contents
Solution2
Optimization3
Tests4
Summarize5

22. Summarize
We proposed an read-write split near-line
loading method and architecture:
• Increase loading performance
• Control resource used by write operation, make sure read
operation can not be starved
• Provide an architecture corresponding with kafka and hdfs
• Provide some optimize method, eg: compact, balance, etc.
• Provide test result

23. FiberHome
Questions ?

24. Thanks