Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Plotcon 2016 Visualization Talk by Alexandra Johnson

1,507 views

Published on

Machine learning is full of ideas that are far abstracted away from the underlying data and difficult to understand. Luckily, this represents an amazing opportunity for visualization! These slides dive into the machine learning meta-problem of hyperparameter optimization. We'll show 4 opportunities for visualization in helping people understand, implement, and evaluate hyperparameter optimization strategies.

Published in: Software
  • Be the first to comment

Plotcon 2016 Visualization Talk by Alexandra Johnson

  1. 1. Visualizing Abstract Concepts in Machine Learning  PIC Alexandra Johnson ___________ Software Engineer @ SigOpt #MachineLearning #MLViz Visualizing Abstract Concepts in Machine Learning | 1
  2. 2. Visualizing Abstract Concepts in Machine Learning | 2 What is Machine Learning? Versicolor Setosa Virginica Training Data + Model -> Labels (Classification) or Numbers (Regression)
  3. 3. Why is this so Intimidating? Visualizing Abstract Concepts in Machine Learning | 3 In-brower deep neural net from playground.tensorflow.org Hyperparameters = your model's magic numbers Examples: learning rate, ratio of train to test data, number of hidden layers, neurons per hidden layer Hyperparameter values must be set before training
  4. 4. Solution: Hyperparameter Optimization And four visualization challenges Visualizing Abstract Concepts in Machine Learning | 4
  5. 5. Values you choose for your hyperparameters have a direct effect on the performance of your model Hard to capture interactions of 20 hyperparameters 20 Dimensional Math is Hard Visualizing Abstract Concepts in Machine Learning | 5
  6. 6. −15 −10 −5 0 5 0.2 0.4 0.6 0.8 1 log_C Accuracy Visualizing Abstract Concepts in Machine Learning | 6 20 Dimensional Math is Hard First try: graph model performance vs hyperparameter value For every hyperparameter Good for understanding indivudal hyperparameters, bad for understanding interactions
  7. 7. 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Accuracy Visualizing Abstract Concepts in Machine Learning | 7 20 Dimensional Math is Hard Graph up to 4 dimensions at once: x, y, z axis + color Hard to visualize 4 dimensions at once, imagine 20! Maybe you want to use an algorithm to handle hyperparameter optimization
  8. 8. Visualizing Abstract Concepts in Machine Learning | 8 Hyperparameter Optimization Strategies are Different Grid Search Random Search Bayesian Optimization
  9. 9. Some Strategies Produce Better Results 0.96 0.97 0.98 0.99 0 5 10 15 20 25 Distribution of Best Found Values over Experiments of 25 Iterations Maximum Accuracy Experiments Visualizing Abstract Concepts in Machine Learning | 9 Experiment = optimizing hyperparameters of your model, results in some maximum performance Some hyperparameter optimization strategies are stochastic, can't just look at one experiment Look at distribution of maximum performance over many experiments optimizing hyperparameters of the same model
  10. 10. Some Strategies Produce Better Results 0.92 0.93 0.94 0.95 0.96 0.97 0.98 0.99 1 0 5 10 15 20 25 Distribution of Best Found Values over Experiments of 25 Iterations Maximum Accuracy Experiments Random Search Grid Search Bayesian Optimization Visualizing Abstract Concepts in Machine Learning | 10 Use the Mann-Whitney U Test to compare distributions of maximum performance
  11. 11. Some Strategies Produce Better Results, Faster 0 5 10 15 20 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Best Seen Trace Timestep BestSeenAccuracy Visualizing Abstract Concepts in Machine Learning | 11 How much time do you have for optimization? Strategies that reliably produce better results faster can optimize the hyperparameters of your model in less time
  12. 12. Some Strategies Produce Better Results, Faster 0 5 10 15 20 0.4 0.5 0.6 0.7 0.8 0.9 1 Interquartile Range of Best Seen Traces Timestep BestSeenAccuracy Visualizing Abstract Concepts in Machine Learning | 12 Again, consider a distribution of optimization experiments 25th - 75th percentile of performance our model could acheive if we stopped early
  13. 13. Some Strategies Produce Better Results, Faster 0 5 10 15 20 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Interquartile Ranges of Best Seen Traces Timestep BestSeenAccuracy Grid Search Random Search Bayesian Optimization Visualizing Abstract Concepts in Machine Learning | 13 Compare the area under the curve of different strategies  Further reading at sigopt.com/research
  14. 14. Takeaways Visualizing Abstract Concepts in Machine Learning | 14 Hyperparameter optimization is an invaluable part of any modern machine learning pipeline Concepts like comparing hyperparameter optimization strategies are extremely abstract and difficult to understand  Visualizations are in their infancy, but are an important part of explaining these ideas
  15. 15. Thank You! Visualizing Abstract Concepts in Machine Learning | 14 Email: alexandra@sigopt.com Twitter: @alexandraj777 www.sigopt.com

×