ML Playbook
•
2 likes•363 views
ML Products have become a prolific and integral part of taking the insights of Data Science from theory to reality. Oddly though, the path from conception to implementation is often unclear with seemingly few similar examples to work from. The result is often a sea of agony between sliding deadlines, heroic efforts of people working though unforeseen challenges and haphazard innovation. Each time a beautiful model makes its impact on the business bottom line, something worked. In this talk we present the ML Playbook. It pulls together the best aspects from a variety of successful ML Product launches into a cohesive strategy to Plan, Build, Test, Learn, and Release ML Products. We'll demonstrate the ML Playbook in action with the story of launching an alert monitoring product for the world's most powerful jet engines, the GE90-115B.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to ML Playbook
Similar to ML Playbook (20)
More from June Andrews
More from June Andrews (12)
Recently uploaded
Recently uploaded (20)
ML Playbook
- 1. Dr June Andrews ML Playbook
- 2. Getting Machine Learning Products into Production … … bit more of an art than a science. “For every idea in production, there are ~7 ideas that did not make the cut.” Ellsworth Kelly
- 3. Optimize and refine products by cycling multiple times with improvements. Machine Learning Product Development Cycle - Phases Extending Boehm Spiral Model from Software Engineering Plan BuildTest Learn Release
- 4. Works for Social Networks, Consumer Web and Industrial Internet of Things. Diverse Experiences over Years Incapsulated in the ML Playbook Extending Boehm Spiral Model from Software Engineering Plan BuildTest Learn Release
- 5. Today - Demonstrate the ML Playbook in Action 3 Times Example Playbook cycles: Build Improve Prototype
- 6. Example of a Build Product - Augmenting Aviation Monitoring Ellsworth Kelly … you can tell how challenging a project is - based on the graveyard of previous attempts.
- 7. Aviation: History of Innovation & Excellence Best Practices & Decision Design has Evolved over 15+ Years of Monitoring Fleets
- 8. Plan Source Ideas Learned from previous work. Immediately prioritized to build without prototype. Model Performance Success Metrics: ๏ Coverage x % ๏ Accuracy y % ๏ Low Latency < z seconds Comprehensive Success Metrics: ๏ Usability is intuitive & transparent ๏ Delivered on time ๏ Maintains current history of excellence ๏ Collaborators enjoy working with Wise Prioritize Define Success Design Roadmap
- 9. Plan Source Ideas Make it easy on yourself & proportional to the success metrics. Minor variations in how the ML product is designed lead to drastically different difficulties of implementation. Explore design decisions around: ๏ Critical Paths of existing systems ๏ Augment v Automate ๏ UI - number of results, placement, text length, … Prioritize Define Success Design Roadmap
- 10. New Workflow Preserves Baseline Reliability with Increasing Speed & Accuracy Delivered an Augmented Human Interpretable Model
- 11. Increased the Power of the ML Models by Demanding More Handling All Alerts was Key Design Choice
- 12. Competitive Machine Learning Requires Strategic Talent Contributions Augmenting Fleet Monitor was an Orchestration of Aligning Talents with Needs … the Roadmap was detailed.
- 13. Plan Build Source Ideas We love this stage! Everything else is very linear - Build is where it can get messy and enthralling. There are plenty of great explanations and ‘wheels’ with more detail. Remember - building is not the end goal. Proactively communicate complications & delays to project stake holders. Prioritize Define Success Design Roadmap Find & ETL Data Feature Engineer Build & Train Optimize
- 14. Plan Build Source Ideas Lessons Learned working with Time Series: ๏ Time is not the same for all engines. Normalize by Life Cycle. ๏ Inferring missing data from nearby sensor readings for use with simpler models assuming even time steps is a bad idea. ๏ Create features resilient to missing data. ๏ Take the time to be clever. More data is not an option. (Deep Learning?) There have been more flights in a year than websites about Dogs on Google - but far fewer maintenance alerts than there are T-Rex Dogs. Prioritize Define Success Design Roadmap Find & ETL Data Feature Engineer Build & Train Optimize
- 15. Machine Learning Experts Transform Domain Knowledge into Model Inputs Aviation Experts are a Strategic Advantage in IIoT
- 16. Plan BuildTest Source IdeasThird party support with MTurk, Crowdflower …. Largest companies tend to use in-house solutions, Pinterest uses Sofia, Stitchfix has Stylists, Facebook and Google famously have in house teams. We worked with in house Aviation Experts. Prioritize Define Success Design Roadmap Find & ETL Data Feature Engineer Productionize A/B Test Build & Train OptimizeEvaluateDog Food
- 17. Plan BuildTest Source Ideas Productionizing is becoming much easier! Airbnb has Airflow Netflix has Spinnaker+ Wise has Wise Factory 2.0 Clouds are now starting to offer Google Cloud with SavedModel AWS with Sagemaker Lowering hiring requirements. Prioritize Define Success Design Roadmap Find & ETL Data Feature Engineer Productionize A/B Test Build & Train OptimizeEvaluateDog Food
- 18. Plan BuildTest Learn Source Ideas Overall learning for GE - break from the idea of one task one model, one alert one model. There are cross learnings from common alerts that are pertinent for rare alerts. Learnings from Aviation are being used in Power. Tactical learning - found there was a difference in performance between offline and online systems due to data delays. Prioritize Define Success Design Roadmap Find & ETL Data Feature Engineer Productionize A/B Test Analyze Product Review Build & Train OptimizeEvaluateDog Food Evangelize
- 19. LearnAnalyze Product Review Evangelize Resist letting the Data Used and Scope of the model reflect your organization. bonkersworld.net
- 20. Plan BuildTest Learn Release Source Ideas Non trivial - expect this ML Product to be in production 2 to 25 years. Plan accordingly with documentation, multiple teams supporting & Black Swan simulations. Prioritize Define Success Design Roadmap Find & ETL Data Feature Engineer Productionize A/B Test Analyze Product Review Build & Train OptimizeEvaluateDog Food Evangelize Release 100% Retrain & Maintain
- 21. Example of an Improve Product - Increase Coverage by 20% Ellsworth Kelly … there’s always an improvement to be made. Is it worth the effort is another question.
- 22. Plan Source Ideas Project Prioritization - books have been written. Let’s distill: ๏ Gather all ideas. ๏ Measure each idea by {Impact, Cost, Strategy} ๏ Use physics and math principles for estimating cost and error propagation for upper and lower bounds. ๏ Keep records of assumptions and results, update assumptions as more is learned. Prioritize Define Success Design Roadmap
- 23. Plan Source Ideas Prioritize Define Success Design Roadmap Estimated Impact Cost Strategic Tuning of Window Parameters z% 3 days No Debugging of missing data. z% 2 weeks Yes Train new model on uncovered alerts z% 1 month+ No Add more Time Series Features z% 2 days per feature Yes
- 24. Plan Build Source Ideas Building Improvements is very quick for scalable models. Scalable engineering allows arbitrary number of features. Scalable math allows inclusion of additional features with predictable or safe outcomes (don’t mix logs and multiplications, don’t mix embeddings with clustering.) Improvement features are often bucketed together. Balance ‘fix it while I’m here’ with feature creep. There will be another cycle, can always add ideas to be prioritized in the next cycle. Prioritize Define Success Design Roadmap Find & ETL Data Feature Engineer Build & Train Optimize
- 25. Plan BuildTest Source Ideas Test stage should be straight forward. Prioritize Define Success Design Roadmap Find & ETL Data Feature Engineer Productionize A/B Test Build & Train OptimizeEvaluateDog Food
- 26. Plan BuildTest Learn Source Ideas Even straight forward feature additions offer a chance to learn: New Feature Family? Suspicious behavior? Bugs? Improvements for faster iteration? Prioritize Define Success Design Roadmap Find & ETL Data Feature Engineer Productionize A/B Test Analyze Product Review Build & Train OptimizeEvaluateDog Food Evangelize
- 27. Plan BuildTest Learn Release Source Ideas Should be straight forward. Update documentation, monitoring, dependancies … Prioritize Define Success Design Roadmap Find & ETL Data Feature Engineer Productionize A/B Test Analyze Product Review Build & Train OptimizeEvaluateDog Food Evangelize Release 100% Retrain & Maintain
- 28. Example of Prototype Product - Human in the Loop Clustering Ellsworth Kelly …no recommendation here is domain specific. Let’s test that.
- 29. Pinterest: History of Investing in Tools for Scalable Data Science Project - Understand how users are engaging with Link Domains. Make tool reusable.
- 30. Plan Source Ideas How are Users Engaging with Link Domains? Success Metrics: ๏ Human Interpretable Answer ๏ Representative of Entire Domain ๏ Easily Reused ๏ Done in 2 weeks Prioritize Define Success Design Roadmap
- 31. Plan Source Ideas Prioritize Define Success Design Roadmap ๏ Human Interpretable Answer ๏ Representative of Entire Domain ๏ Easily Reused ๏ Done in 2 weeks ๏ Reuse existing code and features ๏ Use Python Notebook ๏ Extend a Human in the Loop Algorithm ๏ Ask some work of the user
- 32. Plan Build Source Ideas Prioritize Define Success Design Roadmap Find & ETL Data Feature Engineer Build & Train Optimize Another time series! Focused time on visualization and human interaction components. ( reused ETL jobs, basic features, used K-means )
- 33. Plan BuildTest Source Ideas Prioritize Define Success Design Roadmap Find & ETL Data Feature Engineer Productionize A/B Test Build & Train OptimizeEvaluateDog Food
- 34. Plan BuildTest Learn Source Ideas ๏ Identifiable skill gaps: ๏ Cherry Picking ๏ Biased Exploration ๏ Replication Crisis of Science extends to Data Science Prioritize Define Success Design Roadmap Find & ETL Data Feature Engineer Productionize A/B Test Analyze Product Review Build & Train OptimizeEvaluateDog Food Evangelize
- 35. Plan BuildTest Learn Release Source Ideas Code was documented & committed. Prioritize Define Success Design Roadmap Find & ETL Data Feature Engineer Productionize A/B Test Analyze Product Review Build & Train OptimizeEvaluateDog Food Evangelize Release 100% Retrain & Maintain
- 36. Conclusions Ellsworth Kelly … all together now.
- 37. Plan BuildTest Learn Release Works for prototyping, building & improving ML Products. Use cycles to focus efforts, avoid feature creep & set expectations.
- 38. Plan Source Ideas Good companies make good decisions. Great companies forgo good decisions to make great ones. Proper planning leads to efficiently executing great ideas. Prioritize Define Success Design Roadmap
- 39. Plan Build Source Ideas Building & Optimizing models is largely a solved problem. Identify & prioritize unsolved problems early on. Prioritize Define Success Design Roadmap Find & ETL Data Feature Engineer Build & Train Optimize
- 40. Plan BuildTest Source Ideas Testing should match the diversity of the user base & the gravity of the ML Product’s responsibility. Prioritize Define Success Design Roadmap Find & ETL Data Feature Engineer Productionize A/B Test Build & Train OptimizeEvaluateDog Food
- 41. Plan BuildTest Learn Source Ideas Take the time to learn. Small changes in complex environments often reveal cascading effects. Prioritize Define Success Design Roadmap Find & ETL Data Feature Engineer Productionize A/B Test Analyze Product Review Build & Train OptimizeEvaluateDog Food Evangelize
- 42. Plan BuildTest Learn Release Source Ideas Model life expectancy is increasing. Model Robustness should follow. Prioritize Define Success Design Roadmap Find & ETL Data Feature Engineer Productionize A/B Test Analyze Product Review Build & Train OptimizeEvaluateDog Food Evangelize Release 100% Retrain & Maintain
- 43. Plan BuildTest Learn Release Source Ideas All stages should be addressed, even if they are skipped. A modification to an upstream stage triggers changes to all downstream stages. Prioritize Define Success Design Roadmap Find & ETL Data Feature Engineer Productionize A/B Test Analyze Product Review Release 100% Retrain & Maintain Build & Train OptimizeEvaluateDog Food Evangelize
- 44. Source Ideas Each stage may involve input from many roles including users & customers, but each stage should have an Owner. Who owns Prioritize is a reflection of {Product, Data, Engineering, Design} - Driven Companies. Prioritize Define Success Design Roadmap Find & ETL Data Feature Engineer Productionize A/B Test Analyze Product Review Release 100% Retrain & Maintain Build & Train OptimizeEvaluateDog Food Evangelize Product Data Science Data Engineering ML Engineering Software Engineering Per Company
- 46. Plan BuildTest Learn Release Source Ideas Platforms will automate many stages. ML teams will focus efforts on stages that yield a competitive advantage. Prioritize Define Success Design Roadmap Find & ETL Data Feature Engineer Productionize A/B Test Analyze Product Review Release 100% Retrain & Maintain Build & Train OptimizeEvaluateDog Food Evangelize
- 47. Turing Test, tests a mastery of communication. Bob Test, tests a mastery of communication, data synthesis & problem solving. Challenge to AI - The Bob Test Bob gathers input from the monitoring team, synthesizes the data & calls the airline. Bob works with experts from the airline to determine root cause & design safe plans of action for quick resolution.
- 48. ML + IIoT Involves an Ongoing Negotiation to Figure Out What is Possible 1/3 of the World’s Power is Generated by GE. 60% of Airplane Engines are made by GE. … Preventative Maintenance Failure/ Anomaly Detection Assistive Diagnosis & Treatment Systems Optimization
- 49. 5 Months to Augment Aviation — 2 Months to Augment Power First Order Effects are When Innovations for Planes Help Planes. Second Order Effects are When Innovations for Planes Help Power. Third Order Effects are …