Downloaded 35 times
Uploaded byHBaseCon
hbaseconasia2017: Large scale data near-line loading method and architecture
This document proposes a read-write split near-line data loading method and architecture to: - Increase data loading performance by separating write operations from read operations. A WriteServer handles write requests and loads data to HDFS to be read from by RegionServers. - Control resources used by write operations to ensure read operations are not starved of resources like CPU, network, disk I/O, and handlers. - Provide an architecture corresponding to Kafka and HDFS for streaming data from Kafka to HDFS to be loaded into HBase in a delayed manner. - Include optimizations like task balancing across WriteServer slaves, prioritized compaction of small files, and customizable storage engines. - Report test results showing one Write
More Related Content
Similar to hbaseconasia2017: Large scale data near-line loading method and architecture
hbaseconasia2017: Large scale data near-line loading method and architecture
- 1. Large scale datanear-line loading method and architecture FiberHome Telecommunication 2017-7-19
- 2. /usr/bin/whoami Shuaifeng Zhou(周帅锋): • Bigdata research and development director ( Fiberhome 2013-) • Software engineer (Huawei 2007-2013) • Use and contribute to HBase since 2009 • sfzhou1791@fiberhome.com
- 3.
- 4. HBase Realtime DataLoading WAL/Flush/Compact Triple IO pressure Read/Write operations share resource: Cpu Network Disk IO Handler Read performance decrease too much when write load is heavy
- 5. Why near-line dataloading? DelayScale ReliableResource Large scale data loading reliably with acceptable time delay and resource occupation Billions write ops per region server one day Usually, several minutes delay is acceptable for customers Resource occupied can be limited under an acceptable level Write op can be repeated Optimistic failure handling HBase
- 6.
- 7. Read-Write split dataloading Independent WriterServer to handle put request RegionServer only handle read request WriteServer write HFile on HDFS, send do-bulkload operation. Several minutes delay between put and data readable.
- 8. Architecture Kafka WriteServer Master HMaster HDFS WriteServer Slave WriteServer Slave RegionServerRegionServer Data Stream Topic Discovery ContralMessage Contral Message Read Write Read/Write
- 9. WriteServer Master Task Management •Create new loading tasks every five minutes or every 10,000 records • Find a slave to load the task • Task status control Topic Management • Discover new kafka topics • Receive loading request • Loading records statistic Slaves Management • Slave status report to master • Balance • failover
- 10.
- 11. Failure Handling Meta Data based Failure Handling Recover:Redo failed tasks when slave down or master restart. Task Meta Data is constructed when a task is created by master, and change status to succeed when slave finish the task. Task Meta Data is the descricption info of a task, include the topic, partitions, start and end offset, status. Stored on disk.
- 12.
- 13. Balance Load balance accordingtasks: • Send new tasks to slaves with less tasks on handling • Try to send tasks of one topic to a few fixed slaves −avoid one region open everywhere −Less region open, less small files • Keep region opened for a while, even there are no tasks − avoid region open/close too frequently
- 14. Compact 2 1 3 4 6 5 78 9 • Small files with higher priority • Avoid one large file together with many small files compact again and again compact compact
- 15. StoreEngine Customized store engine: •organize store files in two queues −one can be read and compact −The other can only be compact −If there are too many files, new file will not be readable until they are compact • Some new files discovered later better than all files can not be read before time out − Occasionally data explosion can be handled − Region need split − “Hot key” should be handled
- 16. HDFS Heterogeneous StorageUsage • Use SSD storage as WriteServer tmp dir • Use SATA as HBase data dir storage − WriteServer write HFile on SSD − Load HFile to HBase(Only move) − Change to SATA storage after compact by regionServer HDFS SSD Storage SATA Storagecompact WriteServer RegionServer
- 17. Resource Control Resource usedby WriteServer should be controllable: • Memory: −JVM parameters 30~50GB memory −Large Memory Store will avoid small files −Too Large memory store will cause gc problems • CPU: − Slave can use 80% cpu cores at most − Compare to real-time data load, a big optimize is we can control the cpu occupation by write operations.
- 18.
- 19. Loading Performance WriterServer Slave CPUIntel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz Memory 128G Disk 1TB SSD * 4 Network 10GE Record size 1KB Compress Snappy Performance 300,000 records/s One WriteServer slave can match 5 RegionServer’s loading requirements before RegionServer reach compact limitation.
- 20. Resource Performance 0 20 40 60 80 100 13:16 13:22 13:29 13:36 13:42 13:49 13:55 14:02 14:08 14:15 14:21 14:28 14:34 14:41 14:47 14:54 15:00 15:07 15:13 15:20 15:26 15:33 15:39 15:46 15:52 15:59 16:05 16:12 16:18 16:25 16:31 CPU TotalWS-Slave5 – 2017/2/17 User% Sys% Wait% -400 -200 0 200 400 600 13:19 13:25 13:32 13:38 13:45 13:51 13:58 14:04 14:11 14:17 14:24 14:30 14:37 14:43 14:50 14:56 15:03 15:09 15:16 15:22 15:29 15:35 15:42 15:48 15:55 16:01 16:08 16:14 16:21 16:27 16:34 Network I/O WS-Slave5 (MB/s) - 2017/2/17 Total-Read Total-Write (-ve) 0 500 1000 1500 2000 0 100 200 300 400 13:19 13:27 13:34 13:42 13:50 13:57 14:05 14:13 14:20 14:28 14:36 14:43 14:51 14:59 15:06 15:14 15:22 15:29 15:37 15:45 15:52 16:00 16:08 16:15 16:23 16:31 IO/sec MB/sec Disk total MB/s WS-Slave5- 2017/2/17 Disk Read KB/s Disk Write KB/s IO/sec Memory JVM: aways use memory as much as assigned GC: config gc policy to avoid full gc.
- 21.
- 22. Summarize We proposed anread-write split near-line loading method and architecture: • Increase loading performance • Control resource used by write operation, make sure read operation can not be starved • Provide an architecture corresponding with kafka and hdfs • Provide some optimize method, eg: compact, balance, etc. • Provide test result
- 23.
- 24.