1. Apache HBase at Airbnb
JINGWEI LU, LIYIN TANG, AND JASON ZHANG
1

2. Data Infrastructure at Airbnb

3. Event
Logs
MySQL
Dumps
Gold Cluster
HDFS
Hive
Kafk
a
Sqoo
p
Silver Cluster Spark Cluster
Spark
ReAi
r
Airflow Scheduling
S3
Presto Cluster
AirPal
Caravel
Tableau
Batch Infrastructure
Yarn HDFS
Hive
Yarn
Jingwei Lu, Liyin Tang, Jason Zhang
3

4. Streaming at Airbnb
Event
Logging
MySQL
BINLOG
Cluster
HDFS
Hive
Spinal tap
Presto Cluster
Yarn
Kafk
a
HBase
Spark Streaming
Datadog
Druid
Kafka
Jingwei Lu, Liyin Tang, Jason Zhang
4

5. Growing Pain

6. Stateless
Jingwei Lu, Liyin Tang, Jason Zhang
Computation SinkSource
DStream DF DF

7. Stateful
Jingwei Lu, Liyin Tang, Jason Zhang
Computatio
n
Source
DStream DF DF
Sink1
Sink2
Sink N
State Storage
RDD

8. Multiple Streams
Jingwei Lu, Liyin Tang, Jason Zhang
DataFrame
Sink1
Process
A
Sink2
Sink3
SinkN
…
DataFrame
Sink1
Process
N
Sink2
Sink3
SinkN
…
Source
DStream
Align by Time
DataFram
e
DataFram
e
State
Storage
Source
DStream
…

9. Streaming + Batch
Jingwei Lu, Liyin Tang, Jason Zhang
DataFrame
Sink1
Process
A
Sink2
Sink3
SinkN
…
DataFram
e
State
Storage
Sourc
e
DStream
Sourc
e
…
Align by Time
…
DataFrame
Sink1
Process
A
Sink2
Sink3
SinkN
…

10. Simplify and Unify

11. AirStream Architecture
Jingwei Lu, Liyin Tang, Jason Zhang
Sources
Stream #1 Stream #N
Hive Tables
HBase
Tables
Virtual Table Views for Computation
Sinks
…
Customized ComputationSpark SQL
Simple Config
HBase Services
Streaming
Sources
Druid

12. AirStream Architecture
Jingwei Lu, Liyin Tang, Jason Zhang
Sources
Stream #1 Stream #N
Hive Tables
HBase
Tables
Virtual Table Views for Computation
Sinks
…
Customized ComputationSpark SQL
HBase Services
Streaming
Sources
Druid
Same Computation
for Batch
processing

13. Stateful

14. Jingwei Lu, Liyin Tang, Jason Zhang
State Store
• Merge changes
• Provide fast lookup
• Fast persistent storage across
streaming and batch jobs
14

15. Why HBase
Jingwei Lu, Liyin Tang, Jason Zhang
Rich Functionalities
Rich Integration with Hadoop EcoSystem
Easy Management
Strong Community
Reliable and Scalable

16. HBase State Store
Operators in Airstream
Jingwei Lu, Liyin Tang, Jason Zhang
16
Full Table Scan
Simple Aggregation
Bulk Upload
Key/Prefix Lookup
Update

17. Jingwei Lu, Liyin Tang, Jason Zhang
Computation DAG
17
Input Data
Left Outer Join Result
Key Lookup

18. Jingwei Lu, Liyin Tang, Jason Zhang
Key Space Design
• Hash partition key space
for load balance
• Composite key for K -> V
• Support full key lookup
• Prefix lookup supported for
all keys used in hash
function
Hash key1 key2 key3
Hash based on key prefix
Hash key1 key2
Lookup based on key prefix
key1 = ‘value1’ and key2 = ‘value2’
18

19. • Partition based on key before write
• Use bulk upload for large volume update
Write Performance
Jingwei Lu, Liyin Tang, Jason Zhang
19

20. Case Study
Jingwei Lu, Liyin Tang, Jason Zhang
Experiment realtime feedback
Update
Experiment Assignment
Event
Lookup
HBase
with TTL
Booking Event
Druid
Datado
g
20
one airstream job

21. Realtime Data Ingestion

22. Realtime Ingestion on HBase
Data Infrastructure
MySQL
Analytica
l Events
Kafka
Spark
Streamin
g
HBase
HDFS
Presto/Hive/Spar
k
Source
Inges
t
Realtime
Query
Snapsh
ot
Batch
Query
Jingwei Lu, Liyin Tang, Jason Zhang
22

23. Access Data in HBase
Jingwei Lu, Liyin Tang, Jason Zhang
HBase
Hive Presto
Spark
SQL
Spark
Streaming
Batch Jobs Interactive Query Streaming
HDFS
Snapshot
Table Mapping/Unifed View on realtime data
23

24. Snapshot & Reseed
Jingwei Lu, Liyin Tang, Jason Zhang
HBase HDFS
Snapshot（HFile Links)
Bulk Upload
24

25. Case Study 1: Events Ingestion
Jingwei Lu, Liyin Tang, Jason Zhang
Kafka
topic
…
topic
topic
Spark
Executor
1
…
Executor
2
Executor
K
HBase
DeDup
HDFS
Region1…Region2R
egion M
Daily
Snapshot
Realtime
Query
Hive
Presto
Events
Partition
25

26. Case Study 2: Streaming DB Export
KafkaRDS
Table1
…
Spinalta
p.
Table1
…
Table2
TableN
Spinaltap.
Table2
Spinaltap.
TableN
Spark
Executor
1
…
Executor2
Executor K
HBase
Region1
…
Region2
Region M
HDFS
Region1…Region2R
egion M
Daily Snapshot
Realtime Query
Jingwei Lu, Liyin Tang, Jason Zhang
26

27. Case Study: Streaming DB Export
Rows CF: Colums Version Value
<ShardKey><DB_TABLE_#1><PK_a=A> id Fri May 19 00:33:19 2016 101
<ShardKey><DB_TABLE_#1><PK_a=A> city Fri May 19 00:33:19 2016 San Francisco
<ShardKey><DB_TABLE_#1><PK_a=A> city Fri May 10 00:34:19 2016 New York
<ShardKey><DB_TABLE_#2><PK_a=A’> id Fri May 19 00:33:19 2016 1
Jingwei Lu, Liyin Tang, Jason Zhang
27

28. Case Study: Streaming DB Export
TXN 1
Commit_TS:
101
…
TXN 2
Commit_TS:
102
TXN 3
Commit_TS:
103
TXN N
Commit_TS:
N’
Binlog Order
Jingwei Lu, Liyin Tang, Jason Zhang
28

29. Case Study: Streaming DB Export
TXN 1
Commit_TS:
101
…
TXN 2
Commit_TS:
103
TXN 3
Commit_TS:
102
TXN N
Commit_TS:
N’
NTP
Binlog Order
Jingwei Lu, Liyin Tang, Jason Zhang
29

30. Case Study: Streaming DB Export
TXN 1
Commit_TS:
101
…
Binlog Order
TXN 2
Commit_TS:
103
TXN 3
Commit_TS:
102
TXN N
Commit_TS:
N’
Point-in-Time Restore on TS 102
Jingwei Lu, Liyin Tang, Jason Zhang
30

31. Case Study: Streaming DB Export
Rows CF: Colums Version Value
<ShardKey><DB_TABLE_#1><PK_a=A> id bin100 101
<ShardKey><DB_TABLE_#1><PK_a=A> city bin101 San Francisco
<ShardKey><DB_TABLE_#1><PK_a=A> city bin102 New York
<ShardKey><DB_TABLE_#2><PK_a=A’> id bin100 1
Jingwei Lu, Liyin Tang, Jason Zhang
31

32. Case Study: Streaming DB Export
Rows Version (Logical Offset) Value
<ShardKey><DB_TABLE_#1><2016-05-23 23><100> 100 mysql-bin.00000:100
<ShardKey><DB_TABLE_#1><2016-05-23 23><101> 101 mysql-bin.00000:101
<ShardKey><DB_TABLE_#1><2016-05-23 23><103> 103 mysql-bin.00000:103
<ShardKey><DB_TABLE_#1><2016-05-24 00><102> 102 mysql-bin.00000:102
Jingwei Lu, Liyin Tang, Jason Zhang
32

33. Case Study: Streaming DB Export
Rows Version (Logical Offset) Value
<ShardKey><DB_TABLE_#1><2016-05-23 23><100> 100 mysql-bin.00000:100
<ShardKey><DB_TABLE_#1><2016-05-23 23><101> 101 mysql-bin.00000:101
<ShardKey><DB_TABLE_#1><2016-05-23 23><103> 103 mysql-bin.00000:103
<ShardKey><DB_TABLE_#1><2016-05-24 00><102> 102 mysql-bin.00000:102
Jingwei Lu, Liyin Tang, Jason Zhang
33

34. Summary
Jingwei Lu, Liyin Tang, Jason Zhang
Scalable and Reliable
Rich Stateful Computation
Rich Integration with Hadoop EcoSystem
Easy Operation

35. 35