Tuning ML Models: Scaling, Workflows, and Architecture
•
0 likes•443 views
This document discusses best practices for tuning machine learning models. It covers architectural patterns like single-machine versus distributed training and training one model per group. It also discusses workflows for hyperparameter tuning including setting up full pipelines before tuning, evaluating metrics on validation data, and tracking results for reproducibility. Finally it provides tips for handling code, data, and cluster configurations for distributed hyperparameter tuning and recommends tools to use.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Tuning ML Models: Scaling, Workflows, and Architecture
Similar to Tuning ML Models: Scaling, Workflows, and Architecture (20)
More from Databricks
More from Databricks (20)
Recently uploaded
Recently uploaded (20)
Tuning ML Models: Scaling, Workflows, and Architecture
- 1. Tuning ML Models: Scaling, Workflows, and Architecture Joseph Bradley Solutions Architect
- 2. About me ▪ Solutions Architect at Databricks (½ a year) ▪ Software Engineer at Databricks (5 ½ years) ▪ Apache Spark committer and PMC member
- 3. Global company with over 5,000 customers and 450+ partners Original creators of popular data and machine learning open source projects A unified data analytics platform for accelerating innovation across data engineering, data science, and analytics
- 4. Hyperparameter Model 1: “Dress” Model 2: “Sneaker” Why the difference? Model 2 had better hyperparameter settings: • Learning rate • Model structure • ... What’s a hyperparameter? • Statistical: assumptions about your model/data • Practical: inputs your ML library does not learn from data • Algorithmic: problem-dependent configs
- 5. Hyperparameter tuning Expert knowledge Not in this talk: • Statistical best practices • Overview of methods for tuning In this talk: • Data Science workflow best practices • Tips for the big data and cloud computing space Black-box tuning Ignore until needed à See references at end!
- 6. Hyperparameter tuning is tough 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1 2 3 4 5 6 7 8 9 10 # hyperparameters (7 possible values each) Fractioncoverage Non-convex optimization Curse of dimensionality è Computational cost Unintuitive hyperparameters • Regularization • Neural net structure • and many more...
- 7. In this talk Architectural patterns Workflows Tips and details Getting started
- 8. In this talk Architectural patterns Workflows Tips and details Getting started
- 9. Single-machine vs. distributed training • Single-machine training • Distributed training • Training 1 model per group (per customer, product, etc.)
- 10. Single-machine training Tuning Scale out via distributed hyperparameter tuning à Train 1 model per Spark task Driver Worker WorkerWorker Tuning Tools for this: • Hyperopt + SparkTrials • sklearn + joblibspark • Pandas UDFs
- 11. Distributed training Driver Worker WorkerWorker Training Tuning ML ML Scale out via parallel tuning à Train 1 model at a time, or train 2+ models in parallel Tools for this: • Apache Spark ML • Hyperopt
- 12. Training one model per group Driver Worker WorkerWorker Distribute over groups Tuning Tuning Tuning Scale out by distributing over groups à Train 1 group’s model per Spark task Tools for this: • Apache Spark, Pandas UDFs • Hyperopt
- 13. In this talk Architectural patterns Workflows Tips and details Getting started
- 14. Getting started Start small • Bias towards smaller models, fewer iterations, etc. • Small is cheaper, and it may suffice. • Regardless, it gives a baseline. Think before you tune • Separate train/val/test sets. • Use early stopping or smart tuning wherever possible. • Pick hyperparameters carefully. • Pick ranges carefully.
- 15. Models vs. pipelines Best practice: Set up full pipeline before tuning. • At what point does the pipeline compute the metric you care about? Tuning models vs. tuning pipelines • Tuning featurization Optimizing tuning for pipelines • Cache intermediate results
- 16. Evaluating and iterating Validation data and metrics • Record many metrics on both training and validation data. Tuning hyperparameters independently vs. jointly • Using smarter hyperparameter search algorithms Tracking and reproducibility • Data, code, params, metrics, models, metadata • Tip: Parametrize code to facilitate tracking
- 17. (Demo)
- 18. In this talk Architectural patterns Workflows Tips and details Getting started
- 19. Handling code Getting code to workers • Generally simple: Pandas UDFs or integrations (Hyperopt, etc.) • Debugging code serialization • Errors are often in worker logs and look like ML library bugs (e.g., “no module named X...”) • Tip: For Python, import libraries within closures Passing configs and credentials • E.g., MLflow active runs and credentials Helpful resource: Distributed Hyperopt best practices and troubleshooting
- 20. Moving data Single-machine ML • Broadcast • Load from blob storage • Caching data Distributed ML • Caching data Blob storage data prep • Delta Lake format • Petastorm and TFRecords Helpful resources: • Distributed Hyperopt best practices and troubleshooting • Prepare data for distributed training
- 21. Configuring clusters Single-machine ML • Sharing machine resources • Selecting machine types Distributed ML • Right-sizing clusters • Sharing a cluster Helpful resource: Scaling Hyperopt to Tune Machine Learning Models in Python
- 22. In this talk Architectural patterns Workflows Tips and details Getting started
- 23. Tools to know about Apache Spark CrossValidator, TrainValidationSplit, Pandas UDFs MLflow Tracking, Auto-logging ML in Python Hyperopt SparkTrials & more (See last year’s SAIS talk) Scikit-learn sklearn.model_selection, skopt, joblib+Spark TensorFlow HParams, Keras Tuner, Model Optimization Toolkit PyTorch Ax
- 24. Resources Blog posts + example notebooks • Hyperparameter Tuning with MLflow, Apache Spark MLlib and Hyperopt Talks • Best Practices for Hyperparameter Tuning with MLflow (SAIS 2019) • Advanced Hyperparameter Optimization for Deep Learning with MLflow (SAIS 2019) Project pages + docs • Hyperopt docs and Github • MLflow homepage and Github Slides: tinyurl.com/sais2020-joseph Notebook: tinyurl.com/sais2020-joseph-demo
- 25. Thanks! Architectural patterns Workflows Tips and details Getting started Any questions?