Download as PDF, PPTX
Uploaded byDatabricks
MLflow: Infrastructure for a Complete Machine Learning Life Cycle
The document discusses the challenges of machine learning (ML) development and introduces MLflow as an open platform that simplifies the ML lifecycle by supporting various libraries and deployment methods. It emphasizes the need for a standardized process across teams to enhance collaboration and model reproducibility. Key features of MLflow include experiment tracking, packaging for reproducible runs, a general model format, and a modular design allowing integration into existing workflows.
More Related Content
Similar to MLflow: Infrastructure for a Complete Machine Learning Life Cycle
![[DSC DACH 23] Go with the flow – Track your machine learning lifecycle using ...](https://cdn.slidesharecdn.com/ss_thumbnails/pspddscdachmlflow-230424084504-4f833f21-thumbnail.jpg?width=640&height=640&fit=bounds)
MLflow: Infrastructure for a Complete Machine Learning Life Cycle
- 1. : Infrastructure forComplete Machine Learning Lifecycle Andrew Chen & Mani Parkhe Sept 6th, 2018
- 2. Outline ML development challenges HowMLflow tackles these Demo How to get started
- 3.
- 4. ML Lifecycle 4 Delta Data Prep Training Deploy RawData μ λ θ Tuning Scale μ λ θ Tuning Scale Scale Scale Model Exchange Governance
- 5. “I build 100sof models/day to lift revenue, using any library: MLlib, PyTorch, R, etc. There’s no easy way to see what data went in a model from a week ago, tune it and rebuild it.” -- Chief scientist at ad tech firm Example
- 6. Example “Our company has100 teams using ML worldwide. We can’t share work across them: when a new team tries to run some code, it often doesn’t even give the same result.” -- Large consumer electronics firm
- 7. Custom ML Platforms FacebookFBLearner, Uber Michelangelo, Google TFX +Standardize the data prep / training / deploy loop: if you work with the platform, you get these! – Limited to a few algorithms or frameworks – Tied to one company’s infrastructure Can we provide similar benefits in an open manner?
- 8. Introducing Open machine learningplatform •Works with any ML library& language • Runs the same wayanywhere (e.g. any cloud) • Designed to be useful for 1 or 100,000 person orgs
- 9. MLflow Components 9 Tracking Record andquery experiments: code, configs, results,…etc Projects Packaging format for reproducible runs on any platform Models General model format that supports diverse deployment tools
- 10.
- 11. bathrooms: 1 bedrooms: 2 accommodates:4 total_reviews: 45 cleanliness_rating:9 location_rating:10 checkin_rating: 10 zip_code: 94105 price: 150 f (x) Model listing attributes based on data from insideairbnb.com Goal: Predict Price of Airbnb Listings
- 12.
- 13. Project Spec Code DataConfig LocalExecution RemoteExecution MLflow Projects
- 14. Model Format Flavor 2Flavor1 Run Sources InferenceCode Batch & Stream Scoring Cloud ServingTools MLflow Models Simple model flavors usableby many tools
- 15. Getting Started withMLflow Install with pip install mlflow Find detailed tutorials at mlflow.org Sign up at databricks.com/mlflow for futureupdates
- 16. Ongoing MLflow Roadmap •TensorFlow, Keras, PyTorch, H2O, MLlib integrations ✔ • Java and R language APIs (both in review!) • Multi-step workflows • Hyperparameter tuning • Data source API based on Spark data sources • Model metadata & management
- 17. Conclusion Workflow tools cangreatly simplifythe ML lifecycle • Improve usability for both data scientists and engineers • Same way software dev lifecycle tools simplify development Learn more about MLflow at mlflow.org
- 18.
- 19.
- 20. MLflow Design Philosophy 1.“API-first”, open platform • Allow submittingruns,models,etc from anylibrary & language • Example: a “model” can justbe a lambdafunction thatMLflow can thendeploy in many places (Docker, AzureML, Spark UDF, …) Key enabler: built aroundREST APIs and CLI
- 21. MLflow Design Philosophy 2.Modular design • Let people use different components individually(e.g.,use MLflow’s project format but not its deployment tools) • Easy to integrateinto existing ML platforms & workflows Key enabler: distinct components (Tracking/Projects/Models)