Download as PDF, PPTX
Uploaded byDatabricks
Zipline: Airbnb’s Machine Learning Data Management Platform with Nikhil Simha and Andrew Hoh
Zipline is Airbnb's machine learning data management framework. It handles feature engineering, discovering and accessing data sources, generating training sets, and monitoring data quality. Zipline includes a feature store, training set generation, and clients to access features and training data. It uses various data sources like Hive tables and streams data in and handles backfilling and mutations to training data. Zipline aims to make machine learning processes more scalable, robust, and transparent at Airbnb.
More Related Content
Similar to Zipline: Airbnb’s Machine Learning Data Management Platform with Nikhil Simha and Andrew Hoh
Zipline: Airbnb’s Machine Learning Data Management Platform with Nikhil Simha and Andrew Hoh
- 1. Nikhil Simha, Airbnb AndrewHoh, Airbnb Zipline – Airbnb’s ML Data Management Framework #ML3SAIS
- 2.
- 3. Discovering data 3#ML3SAIS • Whereis it? • Is it good? • Will it continue to be good? • Has it already been done before?
- 4. Training set 4#ML3SAIS • Whatprocessing engine to use? • Backfilling • Point-in-time correct windows
- 5. Production 5#ML3SAIS Training Set Generation Model Training Push model toprediction service Online Features Model monitoring Data quality monitoring
- 6. What is Zipline? •Part of Bighead – Airbnb’s e2e ml platform • Training Sets • Feature Store • Data quality 6#ML3SAIS
- 7. Concepts • Data Source •Feature-set • Training-set • Client 7#ML3SAIS
- 8. Feature-Set • Source queries –Hive tables – Event streams – Mutation streams • Primary Keys • Time Stamp 8#ML3SAIS
- 9. Feature-Set – Sources •Multiple sources – Backfills – Migrations 9#ML3SAIS
- 10.
- 11. 11#ML3SAIS The picture can'tbe displayed. Data Quality
- 12. Training set • Driverquery – primary keys – timestamps – Point-in-time correct 12#ML3SAIS
- 13. Training set –example 13#ML3SAIS
- 14. Training set –example 14#ML3SAIS
- 15.
- 16.
- 17. Online scoring • Backfills •Low latency • DB Mutations – Batch correction 17#ML3SAIS
- 18. Lambda • Batch –Spark • Streaming – Flink • GDPR 18#ML3SAIS User Conf Batch Streaming Zipline Client User App *Daily *Continuous KV Store
- 19.
- 20.
- 21. • Across batchand streaming • Fixed length sliding windows • Raw Events at the tail and head Aggregations 21#ML3SAIS 7 day sliding window
- 22.
- 23. Client 23#ML3SAIS • Aggregation • Logicto handle availability • Java – The API is (List of Feature Names) => Map of feature values – as objects.
- 24. Mutations • Organizationally easy •Consistent • Not flexible • More complex – Inverse 24#ML3SAIS
- 25. Mutations - Aggregation 25#ML3SAIS •Monoids – Associative: (a + b) + c = a + (b + c) – Distribute aggregation – Sum, Avg, Count etc.. • Group – Invertible – Min, Max, Median, Ntile – Not possible without memory – Compromise
- 26. Mutations - Aggregation 26#ML3SAIS •Deletes = Inverse(before) • Updates = Inverse(before) + after
- 27.
- 28. When? • Plan toopen source in Q3 2018. • Will be part of Bighead – Talk tomorrow. 28#ML3SAIS
- 29.