Qianxi Zhang 1. Hulu是美国最受欢迎的在线视频网站之一，Hulu Beijing是Hulu第二大研发中心。北京大数据基础架构团队负责整个公司的大数据基础架构的研发和运维。 2. HBase在Hulu的概况 3. HBase在Hulu的使用 4. 用户画像系统，存放所有用户的基本信息，用户行为，第三方DMP数据和机器学习结果标签(几十万个Qualifier)，Spark和Spark Streaming读写HBase数据，运行各种机器学习模型，为公司的视频推荐，精准广告和Marketing团队服务 5. HBase在Hulu的优化 hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#

1. HBase在hulu的使用和实践
张虔熙 @ hulu
qianxi.zhang@hulu.com

2. About hulu

3. About me
• 张虔熙
ü软件工程师@Hulu大数据平台组
ü专注于分布式计算和存储技术
ü热衷于参与开源社区贡献代码
üqianxi.zhang@hulu.com

4. Agenda
• Overview
• Audience Platform(用户画像系统)
• Auto Balance InputFormat and Snapshot
• Online Bill Storage(订单信息存储系统)
• Replication, RPC Queue and Replica

5. Agenda
• Overview
• Audience Platform(用户画像系统)
• Auto Balance InputFormat and Snapshot
• Online Bill Storage(订单信息存储系统)
• Replication, RPC Queue and Replica

6. Overview
• HBase version ： 1.2.0
• Hadoop nodes ：1000+
• HBase nodes：200+
• HBase table： 200+
• HBase data size：700TB
• Cluster：4

7. Scenario
• Audience Platform(用户画像系统)
• Log Storage(日志存储系统)
• Online Bill Storage(订单信息存储系统)
• OpenTSDB

8. Agenda
• Overview
• Audience Platform(用户画像系统)
• Auto Balance InputFormat and Snapshot
• Online Bill Storage(订单信息存储系统)
• Replication, RPC Queue and Replica

9. Audience Platform(用户画像系统)
• 用户画像：根据用户行为抽象出的一个标签化的用户模型
• Data
üProfile(基本属性)
üUser behavior(用户行为)
üThird party data(第三方数据)
üLabel(标签)

10. Audience Platform(用户画像系统)
• Data characteristic
üSparse(10^6 qualifier)
üMulti-version(User behavior)
• Purpose
üMarketing decision
üPersonalized recommendation
üAdvertisement

11. Audience Platform(用户画像系统)
Kafka
HDFS
DB
HBase
Service
Cache
HDFS
Spark
Streaming
Bulk Load
Spark
MapReduce

12. Audience Platform(用户画像系统)
• Key technology
üAuto balance InputFormat
üSnapshot

13. Agenda
• Overview
• Audience Platform(用户画像系统)
• Auto Balance InputFormat and Snapshot
• Online Bill Storage(订单信息存储系统)
• Replication, RPC Queue and Replica

14. Region Size Distribution

15. Application Performance
• Problem
üTask execution time in MapReduce and Spark is positive correlationwithRegion Size
üTask execution time varies wildly
• Resolve
üEnable TableInputFormat autobalance(hbase.mapreduce.input.autobalance)
üSplit large Region and merge small Region for InputFormat
• Bug
üHBASE-15357(Wrong split/middle key)

16. Snapshot
• Snapshot
üTable Meta
üHFile Link
• Why Snapshot?
üPerformance
üThe view of data at specific time

17. Snapshot
• Problem
üCreate one snapshot per application？
üHow to share snapshot between application？
• Snapshot Service
üManage snapshot lifecycle
üAssign the reasonable snapshot to the application

18. Agenda
• Overview
• Audience Platform(用户画像系统)
• Auto Balance InputFormat and Snapshot
• Online Bill Storage(订单信息存储系统)
• Replication, RPC Queue and Replica

19. Online Bill Storage(订单信息存储系统)
• Characteristic
üBill information
üOnline service
üWrite more, read less
üRead delay < 1s

20. Online Bill Storage(订单信息存储系统)
• Key technology
üReplication
üRPC Queue
üReplica

21. Agenda
• Overview
• Audience Platform(用户画像系统)
• Auto Balance InputFormat and Snapshot
• Online Bill Storage(订单信息存储系统)
• Replication, RPC Queue and Replica

22. Replication
• Two datacenter，Master-Master Replication
Cluster A Cluster B
Write Read Write Read
Replication
Replication

23. Replication
• Problem
üReplicationTable and CF configurationwill be wrong if the table name includes namespace
üPrevious design did not consider namespace
üUse “:” when parsing tables and family,such as “userTable:family1”
üBut Namespace and table segmentationis also “:”, such as “namespace1:userTable:faimly1”
• Resolve
üHBASE-11386, HBASE-11393(Use Protobuf insteadof string)

24. Replication
• Problem
üSome data couldn’t be replicated
üPeerClusterZnode under regionserver of removed peer may never be deleted
üIf some regionserver crash, other regionserver couldn’t take over the rest
replication work since the method “copyQueuesFromRSUsingMulti” fails
• Resolve
üHBASE-16135, HBASE-14476

25. RPC Queue
• Improve Performance
üMulti RPC Queue
üHBASE-11355
• More
üControlling Queue Delay（CoDel）
üHBASE-15136
Write Queue
Get Queue
Scan Queue

26. Replica
• Problem
üWhen a RegionServer crash, the region on it is unavailable for a period
• Resolve
üRegion replicas
üThere could be more than one replica for one region
üOne primary replica could accept write and read operation
üMulti secondary replica only accepts read operation
üHBASE-10070

27. Replica
WAL HFile-1 HFile-2HDFS
RegionServer
Region
(Primary)
RegionServer
Region
(Secondary)
HBase
Client
Read and Write Read Only

28. Replica
• Client strategy
üQuery primaryregion first
üIf don’t get the result in 10ms, add a query to the secondary replicas
üTake the first answer and cancel others
• Problem
üThe data in secondary replica may be stale.
• More
üHBASE-11568(Async WAL to secondary replica)

29. Future
• Multi-Tenancy(HBASE-10994)
• Strong schema
• High availability

30. Reference
• https://issues.apache.org/jira/browse/HBASE-15357
• https://issues.apache.org/jira/browse/HBASE-11386
• https://issues.apache.org/jira/browse/HBASE-11393
• https://issues.apache.org/jira/browse/HBASE-16135
• https://issues.apache.org/jira/browse/HBASE-14476
• https://issues.apache.org/jira/browse/HBASE-15136
• https://issues.apache.org/jira/browse/HBASE-10070
• https://issues.apache.org/jira/browse/HBASE-11568
• https://issues.apache.org/jira/browse/HBASE-10994

31. Thank you
qianxi.zhang@hulu.com