Building Data Product Based on Apache Spark at Airbnb with Jingwei Lu and Liyin Tang
•
11 likes•4,282 views
Building data product requires having Lambda Architecture to bridge the batch and streaming processing. AirStream is a framework built on top of Apache Spark to allow users to easily build data products at Airbnb. It proved Spark is impactful and useful in the production for mission-critical data products. On the streaming side, hear how AirStream integrates multiple ecosystems with Spark Streaming, such as HBase, Elasticsearch, MySQL, DynamoDB, Memcache and Redis. On the batch side, learn how to apply the same computation logic in Spark over large data sets from Hive and S3. The speakers will also go through a few production use cases, and share several best practices on how to manage Spark jobs in production.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Building Data Product Based on Apache Spark at Airbnb with Jingwei Lu and Liyin Tang
Similar to Building Data Product Based on Apache Spark at Airbnb with Jingwei Lu and Liyin Tang (20)
More from Databricks
More from Databricks (20)
Recently uploaded
Recently uploaded (20)
Building Data Product Based on Apache Spark at Airbnb with Jingwei Lu and Liyin Tang
- 1. Building Data Products on Spark at Airbnb LIYIN TANG & JINGWEI LU
- 2. Data Infrastructure at Airbnb
- 3. Event Logs MySQL Dumps Gold Cluster HDFS Hive Kafka Sqoop Silver Cluster Spark Cluster Spark ReAir Airflow Scheduling S3 Presto Cluster AirPal SuperSet Tableau Batch Infrastructure Yarn HDFS Hive Yarn Liyin Tang and Jingwei Lu 3
- 4. Streaming at Airbnb Liyin Tang and Jingwei Lu 4 Cluster Spark Streaming Airflow Scheduling HBase HDFS Sources Kafka S3 HDFS … Sinks Datadog Kafka Dynamo DB Elastic Search …
- 6. Batch AirStream Hive Spark SQL Lambda Architecture Liyin Tang and Jingwei Lu 6 Streaming Kafka Spark Streaming State Storage
- 7. Combine Streaming and Batch Processing
- 8. Sources Liyin Tang and Jingwei Lu 8 Streaming source: [ { name: source_example, type: kafka, config: { topic: "example_topic", } } ] Batch source: [ { name: source_example, type: hive, sql: { select * from db.table where ds=‘2017-06-05’; } } ]
- 9. Computation Liyin Tang and Jingwei Lu 9 Streaming/Batch process: [{ name = process_example, type = sql, sql = """ SELECT listing_id, checkin_date, context.source as source FROM source_example WHERE user_id IS NOT NULL """ }]
- 10. Sinks Liyin Tang and Jingwei Lu 10 Streaming sink: [ { name = sink_example input = process_example type = hbase_update hbase_table_name = test_table bulk_upload = false } ] Batch sink: [ { name = sink_example input = process_example type = hbase_update hbase_table_name = test_table bulk_upload = true } ]
- 11. Streaming Computation Flow Liyin Tang and Jingwei Lu 11 Source Process_A Process_B Process_A1 Sink_A2 Sink_B2 Batch Source Process_A Process_B Process_A1 Sink_A2 Sink_B2
- 12. Liyin Tang and Jingwei Lu Unified API through AirStream • Declarative job configuration • Streaming source vs static source • Computation operator or sink can be shared by streaming and batch job. • Computation flow is shared by streaming and batch • Single driver executes in both streaming and batch mode job 12
- 14. AirStream Shared Global State Store Liyin Tang and Jingwei Lu 14 HBase Tables Spark StreamingSpark StreamingSpark StreamingSpark Streaming Spark BatchSpark BatchSpark BatchSpark Batch
- 15. •Well integrated with Hadoop eco system •Efficient API for streaming writes and bulk uploads •Rich API for sequential scan and point-lookups • Merged view based on version 15 Why HBase
- 16. Unified Write API Liyin Tang and Jingwei Lu 16 DataFrame HBase Region 1 Region 2 Region N Re-partition <Region 1, [RowKey, Value]> <Region 2, [RowKey, Value]> <Region N, [RowKey, Value]> … … Puts HFile BulkLoad
- 17. Rich Read API Liyin Tang and Jingwei Lu 17 HBase Tables Spark Streaming/Batch Jobs Multi-Gets Prefix Scan Time Range Scan
- 18. Merged Views Liyin Tang and Jingwei Lu 18 Row Key R1 V200 TS200 R1 V150 TS150 R1 V01 TS01 … … … … Time Streaming Writes Streaming Writes Streaming Writes
- 19. Merged Views Liyin Tang and Jingwei Lu 19 Row Key R1 V200 TS200 R1 V150 TS150 R1 V01 TS01 Time Streaming Writes Streaming Writes Streaming Writes R1 V100 TS100Batch Bulk Upload
- 20. Liyin Tang and Jingwei Lu Our Foundations •Unify streaming with batch process •Shared global state store 20
- 21. Use Cases
- 22. MySQL DB Snapshot Using Binlog Replay
- 23. • Large amount of data: Multiple large mysql DBs • Realtime-ness: minutes delay/ hours delay • Transaction : Need to keep transaction across different tables • Schema change: Table schema evolves Database Snapshot 23 Move Elephant
- 24. 24 Binlog Replay on Spark 20+ hr 4+ hr AirStream Job 5 mins 15 mins 1 hr spinal tap seed
- 25. • Streaming and Batch shares Logic: Binlog file reader, DDL processor, transaction processor, DML processor. • Merged by binlog position: <filenum, offset> • Idempotent: Log can be replayed multiple times. • Schema changes: Full schema change history. 25 Log Parser Transaction Processor Change Processor Schema Processor HBASE Lambda Architecture Binlog(realtime/history) DML DDL XVID Mysql Instance
- 27. Hive Realtime Indexing Liyin Tang and Jingwei Lu 27 Elastic Search es_version = mutation id AirStream Spark Streaming Spark Batch Table A Event Event Event… … Kafka Table B Table C
- 29. Druid Ingestion Liyin Tang and Jingwei Lu 29 Druid AirStream Spark Streaming Kafka Dimension Metrics Druid Beam
- 30. Superset Powered by Druid 30
- 31. Tips
- 33. Long Window Computation 33 What if window is weeks, months, or even years?
- 34. Distinct in a Large Window 34 I don’t want approximation. What should I do?
- 35. Distinct Count Liyin Tang and Jingwei Lu 35 Row Key Listing 1 Visitor 01 TS100 Listing 1 Visitor 02 TS100 Listing 1 Visitor 04 TS98 Listing 1 Visitor 03 TS99 Prefix Scan with TimeRange Prefix Scan with TimeRange Time
- 36. Moving Average Liyin Tang and Jingwei Lu 36 Row Key Listing 1 Total Review Cnt: 100 TS100 Listing 1 Total Review Cnt: 98 TS99 Listing 1 Total Review Cnt: 01 TS01 Listing 1 Total Review Cnt: 50 TS50 Count Difference/ Time Elapsed Count Difference/ Time Elapsed Time … … … … … … Window 1 Window 2
- 38. Realtime Ingestion and Interactive Query Liyin Tang and Jingwei Lu 38 HBase AirStream Spark Streaming Kafka Query Engine Data Portal Spark SQL Hive SQL Presto SQL
- 39. Interactive Query in SqlLab 39
- 41. Thrift-> DataFrame Liyin Tang and Jingwei Lu 41 Thrift Event https://github.com/airbnb/airbnb-spark-thrift Thrift Class Thrift Object Field Meta Data Struct Type Field Value Row DataFrame
- 42. Summary
- 43. Unify Batch and Streaming Computation 43
- 44. Global State Store Using HBase 44
- 45. 45 We are hiring Happy Hour: 6pm, B Restaurant&Bar, 720 Howard St, SF