Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Best Practices for Hyperparameter Optimization Alexandra Johnson @alexandraj777
Example: Beating Vegas Scott Clark. Using Model Tuning to Beat Vegas.
Terminology ● Optimization ● Hyperparameter Optimization ● Hyperparameter tuning ● Model tuning
Tune the Whole Pipeline
Optimize all Parameters at Once TensorFlow Playground
Include Feature Parameters
Include Feature Parameters
Choosing a Metric ● Balance long-term and short-term goals ● Question underlying assumptions ● Example from Microsoft
Composite Metric Example: Lifetime Value clicks*wclicks + likes*wlikes + views*wviews
Choose Multiple Metrics ● Balance competing metrics ● Explore entire result space Image from PhD Comics
Avoiding Overfitting
Get A Suggestion
Shuffle and Split Data
Train the Model
Test the Performance
Repeat Shuffle Train Evaluate
Report An Observation
Repeat the Entire Process Shuffle Train Evaluate
Optimization Methods
Hand Tuning ● Hand tuning is time consuming and expensive ● Algorithms can quickly and cheaply beat expert tuning
Grid Search Random Search Bayesian Optimization Alternatives to Hand Tuning
No Grid Search Hyper- parameters Model Evaluations 2 100 3 1,000 4 10,000 5 100,000
No Random Search ● Theoretically more effective than grid search ● Large variance in results ● No intelligence
Bayesian Optimization ● Explore/exploit ● Ideal for "expensive" optimization ● No requirements on: convex, differentiable,...
Alternatives to Bayesian Optimization Genetic algorithms Particle-based methods Convex optimizers Simulated annealing To n...
Takeaways ●Optimize the entire pipeline ●Ensure generalization ●Use Bayesian optimization
Thank You! blog.sigopt.com sigopt.com/research
Upcoming SlideShare
Loading in …5
×

Alexandra Johnson, Software Engineer, SigOpt at MLconf ATL 2017

281 views

Published on

Best Practices for Hyperparameter Optimization:
All machine learning and artificial intelligence pipelines – from reinforcement agents to deep neural nets – have tunable hyperparameters. Optimizing these hyperparameters provides tremendous performance gains, but only if the optimization is done correctly. This presentation will discuss topics including selecting performance criteria, why you should always use cross validation, and choosing between state of the art optimization methods.

Published in: Technology
no profile picture user

  • Be the first to comment

  • Be the first to like this

Alexandra Johnson, Software Engineer, SigOpt at MLconf ATL 2017

  1. 1. Best Practices for Hyperparameter Optimization Alexandra Johnson @alexandraj777
  2. 2. Example: Beating Vegas Scott Clark. Using Model Tuning to Beat Vegas.
  3. 3. Terminology ● Optimization ● Hyperparameter Optimization ● Hyperparameter tuning ● Model tuning
  4. 4. Tune the Whole Pipeline
  5. 5. Optimize all Parameters at Once TensorFlow Playground
  6. 6. Include Feature Parameters
  7. 7. Include Feature Parameters
  8. 8. Choosing a Metric ● Balance long-term and short-term goals ● Question underlying assumptions ● Example from Microsoft
  9. 9. Composite Metric Example: Lifetime Value clicks*wclicks + likes*wlikes + views*wviews
  10. 10. Choose Multiple Metrics ● Balance competing metrics ● Explore entire result space Image from PhD Comics
  11. 11. Avoiding Overfitting
  12. 12. Get A Suggestion
  13. 13. Shuffle and Split Data
  14. 14. Train the Model
  15. 15. Test the Performance
  16. 16. Repeat Shuffle Train Evaluate
  17. 17. Report An Observation
  18. 18. Repeat the Entire Process Shuffle Train Evaluate
  19. 19. Optimization Methods
  20. 20. Hand Tuning ● Hand tuning is time consuming and expensive ● Algorithms can quickly and cheaply beat expert tuning
  21. 21. Grid Search Random Search Bayesian Optimization Alternatives to Hand Tuning
  22. 22. No Grid Search Hyper- parameters Model Evaluations 2 100 3 1,000 4 10,000 5 100,000
  23. 23. No Random Search ● Theoretically more effective than grid search ● Large variance in results ● No intelligence
  24. 24. Bayesian Optimization ● Explore/exploit ● Ideal for "expensive" optimization ● No requirements on: convex, differentiable, continuous
  25. 25. Alternatives to Bayesian Optimization Genetic algorithms Particle-based methods Convex optimizers Simulated annealing To name a few...
  26. 26. Takeaways ●Optimize the entire pipeline ●Ensure generalization ●Use Bayesian optimization
  27. 27. Thank You! blog.sigopt.com sigopt.com/research

×