Recommended
More Related Content
What's hot
What's hot (20)
Similar to Dropbox Talk at Netflix ML Platform Meetup Spe 2019
Similar to Dropbox Talk at Netflix ML Platform Meetup Spe 2019 (20)
Recently uploaded
Recently uploaded (20)
Dropbox Talk at Netflix ML Platform Meetup Spe 2019
- 1. Sep 12, 2019 ML Infra @ Dropbox Overview
- 2. ML @ dropbox - context Our data sources: ● Files - multi-exabyte of user content ● File metadata - File names and directory trees ● User activity - Billions of file access and sharing events per day
- 3. ML @ dropbox - context ML Impact at Dropbox: ● Search Ranking ● Content Suggestions ● File naming Suggestions ● Smart sync ● Spam detection ● Payments ● OCR ● Prompt campaign ranking
- 5. Realtime User Activity Signals - Antenna & UPS
- 6. Data collection - ETL HDFS Signal store Hive Dropbox data lake Antenna User activity logs Predict Logs User/File Metadata User/File Activities Airflow Workflow orchestration Spark Jobs Compute signals and use-case specific datasets HDFS Training data store Online data Offline dataOffline ETL
- 7. Workbench and Notebooks HDFS Signal and training data store Spark Zeppelin Notebooks Multi-user notebook environment Workbench 40 cores, 400GB ram dbxlearn Elastic ML training and hyperparameter tuning
- 8. dbxlearn dbxlearn Datasets Training script bazelized binary Dropbox Data Center Public Cloud (AWS) S3 Data and code store Sagemaker Training Instances Training Instances S3 Model store deploy train/tuneexport
- 9. dbxlearn workflow $ dbxlearn train --py-binary <script> --train_uri <...> --validation_uri <...> [--local] $ dbxlearn tune --py-binary <script> --train_uri <...> --validation_uri <...> $ dbxlearn query --tuning_job_id <id> print_top_summary $ dbxlearn deploy-model --tuning-job_id <id> <experiment-group>
- 10. Predict logger - Converting raw logs into labeled datasets - Logging partial information from different services at different times - Reward windows
- 11. Predict service
- 12. Inference configuration - Defines signal sources - Enable complex inference graphs - Allow shadow and A/B experimentation
- 13. RL Example - Multi Arm Bandits
- 14. RL example - Multi-Arm Bandits
- 17. Summary ● End-to-End platform to support all steps on ML development @ dropbox ● Deep integration with Dropbox most valuable data sources ● Built for easy integration with cloud services for model training and inference ● Built using open source technologies: Hadoop, Spark, TF, Scikit- learn, … ● Support wide variety of use cases
- 18. Questions?
Editor's Notes
- Dropbox huge data sources provide a lot of potential for ML It is also very challenging to handle and ML developer need high infrastructure expertise to use those sources Making those sources accessible for ML developers is crustal for accelerating ml development
- Go through some of the use cases Wide variety of use case, with variety of needs Basic common workflow, but need flexible tools and api
- Existing open source solutions didn’t work for this scale and SLA. Provide access to user actions to various ui surfaces. It is also used for online suggestion models to generate candidate list for suggestions: recent files that were accessed, recent users that were shared with, ... It is often difficult to compute features in a performant manner directly from action events. Antenna provides aggregations on top of user actions: counters, histograms. Offline builds are generated daily. Building indexes and aggregations. Fixing accumulated errors and easy backfill of new aggregations/indexes.
- Add labels to the arrows
- Many competing campaigns. Each experiment wants to get priority Can appear in different pages with different context Implemented as a contextual multi-arm bandits problem
- Content ingestion pipeline: File update -> content extractors from file blocks-> ML models -> Doc store Backfills: Huge time|computation cost. Scoped by user populations, like user subscription type. Content plugins are isolated into jails to address security exploits concerns
- Plugin examples: Text extractor OCR Image labeling Metadata extractor