Design Patterns for Machine Learning in Production - Sergei Izrailev, Chief Data Scientist, BeeswaxIO
•Download as PPTX, PDF•
13 likes•4,141 views
Presented at #H2OWorld 2017 in Mountain View, CA. Enjoy the video: https://youtu.be/-rGRHrED94Y. Learn more about H2O.ai: https://www.h2o.ai/. Follow @h2oai: https://twitter.com/h2oai. - - - Abstract: Most machine learning systems enable two essential processes: creating a model and applying the model in a repeatable and controlled fashion. These two processes are interrelated and pose technological and organizational challenges as they evolve from research to prototype to production. This presentation outlines common design patterns for tackling such challenges while implementing machine learning in a production environment. Sergei's Bio: Dr. Sergei Izrailev is Chief Data Scientist at BeeswaxIO, where he is responsible for data strategy and building AI applications powering the next generation of real-time bidding technology. Before Beeswax, Sergei led data science teams at Integral Ad Science and Collective, where he focused on architecture, development and scaling of data science based advertising technology products. Prior to advertising, Sergei was a quant/trader and developed trading strategies and portfolio optimization methodologies. Previously, he worked as a senior scientist at Johnson & Johnson, where he developed intelligent tools for structure-based drug discovery. Sergei holds a Ph.D. in Physics and Master of Computer Science degrees from the University of Illinois at Urbana-Champaign.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Design Patterns for Machine Learning in Production - Sergei Izrailev, Chief Data Scientist, BeeswaxIO
Similar to Design Patterns for Machine Learning in Production - Sergei Izrailev, Chief Data Scientist, BeeswaxIO (20)
More from Sri Ambati
More from Sri Ambati (20)
Recently uploaded
Recently uploaded (20)
Design Patterns for Machine Learning in Production - Sergei Izrailev, Chief Data Scientist, BeeswaxIO
- 2. Chief Data Scientist @ Beeswax @sizrailev / bit.ly/MLatScale / sergei@beeswax.com / www.beeswax.com Sergei Izrailev
- 3. Design Patterns for Machine Learning in Production
- 4. Motivation • A widespread need to leverage in-house data • Data science expertise is available • And yet, it seems to be too hard to extract value from ML
- 5. • Beeswax • Beeswax is an ad tech startup; 40 employees in NYC and London • Founded by three ex-Googlers • Real-time bidding (RTB) platform for buying online ads (1M+ QPS) • Platform tailored for customers to leverage in-house data science • Myself • Production AI systems in Pharma, Finance and Ad Tech • Interested in both technology and organizations • bit.ly/MLatScale About
- 6. Overall process Discovery Research Prototype Production Value Cost Problem Statement Constraints
- 7. Start with defining the problem
- 8. Problem statement • Is this the right problem to solve? • Suppose, we’ve solved the stated problem - what’s the value? • Is ML the right tool to solve the problem? • What are the constraints?
- 9. Define constraints • Existing production environment architecture • Technology stack • Available people and their skills • Requirements for scale
- 10. Dimensions of ML system scalability • Volume: how much data do we need to process? • Velocity: how quickly does the data change? • Variety: what are the types of data, models, and applications? • Veracity: how accurate are our models? • Value: how does it matter to the ML consumer? • Viability: do the benefits outweigh the costs?
- 11. Technical Design of ML Systems
- 12. Machine learning systems ApplyBuild Model Data Sources Labeled data Unlabeled data Consumer
- 13. ML system design Build Apply Model Data Sources Labeled data Unlabeled data Consumer ? ? ?? ? ? ?
- 14. Model deployment Data Sources Labeled data Unlabeled data ? ? Consumer ? ? Build Model ? ? Apply ?
- 15. Model deployment Build Trans- form Train Code + Data Apply Predict Trans- form • Data transformations must be the same in training and scoring • Some transformations are “models” (PCA, top N, TF-IDF) • Hence, most ML pipelines are DAGs • These DAGs must be reproduced in production scoring
- 16. Interface between building and scoring • In-memory - model is never persisted, train then score • single application, also streaming • Data only - linear coefficients, PMML, etc • code is independent • Serialized objects - Pickle, R, Spark, custom • reuse code • Code + Data - e.g., H2O’s POJO • code is generated
- 17. Scoring systems Build Model Data Sources Labeled data Unlabeled data ? ? ? ? Apply Consumer ?? ?
- 18. Batch processing ConsumerApply Predictions Predictions New data Trans- form Predict
- 19. Batch features; consumer predicts New data ConsumerApply Model Features Predict Features Trans- form
- 21. A service for a single row consumer ConsumerApply New data Prediction New data Trans- form Predict
- 22. A service for a single row consumer ConsumerApply New data Prediction New data Trans- form Predict Cache
- 23. Cached predictions ConsumerApply Cache Predictions PredictionKey Default New data Trans- form Predict New data Predictions
- 25. Cached features ConsumerApply Cache Features FeaturesKey Model Features Predict New data Trans- form New data
- 26. Evolution of ML systems
- 28. Prototype Build Sample Transform generate features Collect labels Train & Test Extract & Clean Apply Transform Generate features Predict Evaluate Extract & Clean Serialized Model
- 29. Production Build Apply Serialized Model Visualization CI/CD Monitoring & Alerts Model Evaluation Change Mgmt Error Handling Data Validation Versioning Security & Access Reporting Testing & QA Logging
- 30. Production Fault-tolerant systems • We should expect problems • models don’t converge; input data changes; bugs • Produce acceptable results - even when something fails • Handle predictable error conditions automatically • Minimal human intervention • Easy diagnostics and recovery in case of fatal errors
- 31. Conway's law: "Any piece of software reflects the organizational structure that produced it."
- 32. People Questions • Who is developing training? • Who is developing scoring? • Who is responsible for training in production? • Who is responsible for scoring in production? • Who deploys new models and model updates? • Who is responsible for quality control?
- 33. Product management Data science Data engineering Application engineering (RT server-side applications, client- side applications) UX People’s Functions • Front-end development • Data collection (e.g., logging) • Code deployment • Testing and QA • Infrastructure provisioning • Support
- 34. Data scientists Data scientists as consumers Apply ConsumerBuild Serialized Model
- 35. EngineersData scientists Over the wall Build Apply Serialized Model Build Apply Serialized Model Consumer
- 36. EngineersData scientists Deliver predictions only (aka “black box”) Build Apply Serialized Model Consumer Predictions
- 37. EngineersData scientists Deliver data (serialized model) Apply ConsumerBuild Serialized Model Scoring Code
- 38. EngineersData scientists Deliver code and data Apply ConsumerBuild Serialized Model Scoring Code
- 39. Team: Data Scientists + Engineers Cross-functional team Apply ConsumerBuild Serialized Model Scoring Code
- 40. EngineersData scientists ML platform Build Apply Serialized Model Consumer Apply Build
- 41. ML platform Serialized Model Visualization CI/CD Monitoring & Alerts Model Evaluation Change Mgmt Error Handling Data Validation Versioning Security & Access Reporting Testing & QA Logging ApplyBuild
- 42. Conclusion • Find the right problem • Define constraints • Design components and interfaces • Take into account organizational constraints • Production can’t be an afterthought • The process is a lot of work, but it’s not rocket science
- 43. Questions? Yes, we are hiring… sergei@beeswax.com