Download as PDF, PPTX
Uploaded byHBaseCon
hbaseconasia2017: HBase在Hulu的使用和实践
The document discusses the use and implementation of HBase at Hulu, focusing on key services such as the audience platform, online bill storage, and replication strategies. It highlights operational challenges, performance issues, and solutions related to task execution, snapshot management, and data replication across clusters. The presentation concludes with future directions in multi-tenancy and schema improvements, supported by references to relevant issues on the Apache JIRA platform.
More Related Content
Similar to hbaseconasia2017: HBase在Hulu的使用和实践
hbaseconasia2017: HBase在Hulu的使用和实践
- 1.
- 2.
- 3.
- 4. Agenda • Overview • AudiencePlatform(用户画像系统) • Auto Balance InputFormat and Snapshot • Online Bill Storage(订单信息存储系统) • Replication, RPC Queue and Replica
- 5. Agenda • Overview • AudiencePlatform(用户画像系统) • Auto Balance InputFormat and Snapshot • Online Bill Storage(订单信息存储系统) • Replication, RPC Queue and Replica
- 6. Overview • HBase version： 1.2.0 • Hadoop nodes ：1000+ • HBase nodes：200+ • HBase table： 200+ • HBase data size：700TB • Cluster：4
- 7. Scenario • Audience Platform(用户画像系统) •Log Storage(日志存储系统) • Online Bill Storage(订单信息存储系统) • OpenTSDB
- 8. Agenda • Overview • AudiencePlatform(用户画像系统) • Auto Balance InputFormat and Snapshot • Online Bill Storage(订单信息存储系统) • Replication, RPC Queue and Replica
- 9. Audience Platform(用户画像系统) • 用户画像：根据用户行为抽象出的一个标签化的用户模型 •Data üProfile(基本属性) üUser behavior(用户行为) üThird party data(第三方数据) üLabel(标签)
- 10. Audience Platform(用户画像系统) • Datacharacteristic üSparse(10^6 qualifier) üMulti-version(User behavior) • Purpose üMarketing decision üPersonalized recommendation üAdvertisement
- 11.
- 12. Audience Platform(用户画像系统) • Keytechnology üAuto balance InputFormat üSnapshot
- 13. Agenda • Overview • AudiencePlatform(用户画像系统) • Auto Balance InputFormat and Snapshot • Online Bill Storage(订单信息存储系统) • Replication, RPC Queue and Replica
- 14.
- 15. Application Performance • Problem üTaskexecution time in MapReduce and Spark is positive correlationwithRegion Size üTask execution time varies wildly • Resolve üEnable TableInputFormat autobalance(hbase.mapreduce.input.autobalance) üSplit large Region and merge small Region for InputFormat • Bug üHBASE-15357(Wrong split/middle key)
- 16. Snapshot • Snapshot üTable Meta üHFileLink • Why Snapshot? üPerformance üThe view of data at specific time
- 17. Snapshot • Problem üCreate onesnapshot per application？ üHow to share snapshot between application？ • Snapshot Service üManage snapshot lifecycle üAssign the reasonable snapshot to the application
- 18. Agenda • Overview • AudiencePlatform(用户画像系统) • Auto Balance InputFormat and Snapshot • Online Bill Storage(订单信息存储系统) • Replication, RPC Queue and Replica
- 19. Online Bill Storage(订单信息存储系统) •Characteristic üBill information üOnline service üWrite more, read less üRead delay < 1s
- 20. Online Bill Storage(订单信息存储系统) •Key technology üReplication üRPC Queue üReplica
- 21. Agenda • Overview • AudiencePlatform(用户画像系统) • Auto Balance InputFormat and Snapshot • Online Bill Storage(订单信息存储系统) • Replication, RPC Queue and Replica
- 22. Replication • Two datacenter，Master-MasterReplication Cluster A Cluster B Write Read Write Read Replication Replication
- 23. Replication • Problem üReplicationTable andCF configurationwill be wrong if the table name includes namespace üPrevious design did not consider namespace üUse “:” when parsing tables and family,such as “userTable:family1” üBut Namespace and table segmentationis also “:”, such as “namespace1:userTable:faimly1” • Resolve üHBASE-11386, HBASE-11393(Use Protobuf insteadof string)
- 24. Replication • Problem üSome datacouldn’t be replicated üPeerClusterZnode under regionserver of removed peer may never be deleted üIf some regionserver crash, other regionserver couldn’t take over the rest replication work since the method “copyQueuesFromRSUsingMulti” fails • Resolve üHBASE-16135, HBASE-14476
- 25. RPC Queue • ImprovePerformance üMulti RPC Queue üHBASE-11355 • More üControlling Queue Delay（CoDel） üHBASE-15136 Write Queue Get Queue Scan Queue
- 26. Replica • Problem üWhen aRegionServer crash, the region on it is unavailable for a period • Resolve üRegion replicas üThere could be more than one replica for one region üOne primary replica could accept write and read operation üMulti secondary replica only accepts read operation üHBASE-10070
- 27.
- 28. Replica • Client strategy üQueryprimaryregion first üIf don’t get the result in 10ms, add a query to the secondary replicas üTake the first answer and cancel others • Problem üThe data in secondary replica may be stale. • More üHBASE-11568(Async WAL to secondary replica)
- 29. Future • Multi-Tenancy(HBASE-10994) • Strongschema • High availability
- 30. Reference • https://issues.apache.org/jira/browse/HBASE-15357 • https://issues.apache.org/jira/browse/HBASE-11386 •https://issues.apache.org/jira/browse/HBASE-11393 • https://issues.apache.org/jira/browse/HBASE-16135 • https://issues.apache.org/jira/browse/HBASE-14476 • https://issues.apache.org/jira/browse/HBASE-15136 • https://issues.apache.org/jira/browse/HBASE-10070 • https://issues.apache.org/jira/browse/HBASE-11568 • https://issues.apache.org/jira/browse/HBASE-10994
- 31.