Downloaded 100 times
![● E[f (λ)] after 20 runs, each run consisting of 60 function
evaluations
● For Grid Search : 64 evenly spaced parameter
configurations (order shuffled randomly)
● SigOpt statistical significance over grid and rnd
(p = 0.0001, Mann-Whitney U test)
PERFORMANCE
SigOpt Rnd. Search Grid Search No Tuning (Baseline)
Best Found 0.8760 (+5.72%) 0.8673 (+4.67%) 0.8680 (+4.76%) 0.8286](https://image.slidesharecdn.com/odkukdaxqywrhdrnjabs-signature-0bea2a94f96cf5a9f536918d6ef0707d749c8f2c3c6861466fa273a7271fe4a0-poli-161114204535/75/Scott-Clark-Co-Founder-and-CEO-SigOpt-at-MLconf-SF-2016-64-2048.jpg)
Uploaded byMLconf
Scott Clark, Co-Founder and CEO, SigOpt at MLconf SF 2016
The document discusses the challenges of tuning machine learning models and introduces Bayesian global optimization as an efficient method for hyperparameter tuning. It details the process of building Gaussian processes to identify optimal parameters while minimizing costly evaluations, highlighting examples from real-world applications in loan default prediction and algorithmic trading. Additionally, it compares various tuning methods and underscores the performance improvements and cost savings that can be achieved using SigOpt's services.
More Related Content
Similar to Scott Clark, Co-Founder and CEO, SigOpt at MLconf SF 2016
Scott Clark, Co-Founder and CEO, SigOpt at MLconf SF 2016
- 1. BAYESIAN GLOBAL OPTIMIZATION UsingOptimal Learning to Tune ML Models Scott Clark scott@sigopt.com
- 2. OUTLINE 1. Why isTuning ML Models Hard? 2. Standard Tuning Methods 3. Bayesian Global Optimization 4. Comparing Optimizers 5. Real World Examples
- 4. Machine Learning is extremelypowerful
- 5. Machine Learning is extremelypowerful Tuning Machine Learning systems is extremely non-intuitive
- 6. https://www.quora.com/What-is-the-most-important-unresolved-problem-in-machine-learning-3 What is themost important unresolved problem in machine learning? “...we still don't really know why some configurations of deep neural networks work in some case and not others, let alone having a more or less automatic approach to determining the architectures and the hyperparameters.” Xavier Amatriain, VP Engineering at Quora (former Director of Research at Netflix)
- 7.
- 8. TUNABLE PARAMETERS INDEEP LEARNING
- 9. TUNABLE PARAMETERS INDEEP LEARNING
- 10. TUNABLE PARAMETERS INDEEP LEARNING
- 11.
- 12.
- 13.
- 14. Grid Search RandomSearch
- 15. Predictive Models Predictive Models TUNING MACHINE LEARNINGMODELS New parameters Objective Metric Better Models Big Data
- 16.
- 17. … the challengeof how to collect information as efficiently as possible, primarily for settings where collecting information is time consuming and expensive. Prof. Warren Powell - Princeton What is the most efficient way to collect information? Prof. Peter Frazier - Cornell How do we make the most money, as fast as possible? Scott Clark - CEO, SigOpt OPTIMAL LEARNING
- 18. ● Optimize objectivefunction ○ Loss, Accuracy, Likelihood ● Given parameters ○ Hyperparameters, feature parameters ● Find the best hyperparameters ○ Sample function as few times as possible ○ Training on big data is expensive BAYESIAN GLOBAL OPTIMIZATION
- 19. 1. Build GaussianProcess (GP) with points sampled so far 2. Optimize the fit of the GP (covariance hyperparameters) 3. Find the point(s) of highest Expected Improvement within parameter domain 4. Return optimal next best point(s) to sample HOW DOES IT WORK?
- 20.
- 21.
- 22.
- 23.
- 24.
- 25.
- 26.
- 27.
- 28. overfit good fitunderfit GAUSSIAN PROCESSES
- 29.
- 30.
- 31.
- 32.
- 33.
- 34.
- 35.
- 36. What is thebest value found after optimization completes? METRIC: BEST FOUND BLUE RED BEST_FOUND 0.7225 0.8949
- 37. How quickly isoptimum found? (area under curve) METRIC: AUC BLUE RED BEST_FOUND 0.9439 0.9435 AUC 0.8299 0.9358
- 38. ● Optimization functions(eg Branin, Ackeley, Rosenbrock) ● ML datasets (LIBSVM) BENCHMARK SUITE TEST FUNCTION TYPE COUNT Continuous Params 184 Noisy Observations 188 Parallel Observations 45 Integer Params 34 Categorical Params / ML 47 Failure Observations 30 TOTAL 489
- 39. ● On-demand cluster inAWS for parallel eval function optimization ● Full eval consists of ~10000 optimizations, taking ~4 hours INFRASTRUCTURE
- 40. VIZ TOOL :BEST SEEN TRACES
- 41. METRICS: STOCHASTICITY ● Runeach 20 times ● Mann-Whitney U test for significance
- 42. RANKING OPTIMIZERS ● Alternatemethods exist for black box optimization : Spearmint, TPE, SMAC, PSO, RND Search, Grid Search ● Important to understand / track method performance disparity on high-level categories of functions ● For a given test function, want a partial ranking (allowing for ties) of method performance
- 43. RANKING OPTIMIZERS ● First,Mann-Whitney U tests using BEST_FOUND ● Tied results then partially ranked using AUC ● Any remaining ties, stay as ties for final ranking
- 44. RANKING AGGREGATION ● Aggregatepartial rankings across all eval functions using Borda count (sum of methods ranked lower)
- 45.
- 46.
- 47. Predictive Models Predictive Models HOW DOES SIGOPTINTEGRATE? New parameters Objective Metric Better Models Big Data
- 48.
- 49. DISTRIBUTED MODEL TRAINING ●SigOpt serves as an AWS-ready distributed scheduler for training models across workers ● Each worker accesses the SigOpt API for the latest parameters to try ● Enables distributed training of non-distributed algorithms
- 50.
- 51.
- 52.
- 53. EXAMPLE: LOAN DATA Loan Applications Default Prediction withtunable ML parameters ● Income ● Credit Score ● Loan Amount New parameters Prediction Accuracy Better Accuracy
- 54. COMPARATIVE PERFORMANCE Accuracy Grid Search RandomSearch AUC .698 .690 .683 .675 $1,000 100 hrs $10,000 1,000 hrs $100,000 10,000 hrs Cost ● Better: 22% fewer bad loans vs baseline ● Faster/Cheaper: 100x less time and AWS cost than standard tuning methods
- 55. EXAMPLE: ALGORITHMIC TRADING MarketData Trading Strategy with tunable weights and thresholds ● Closing Prices ● Day of Week ● Market Volatility New parameters Expected Revenue Higher Returns
- 56. COMPARATIVE PERFORMANCE Standard Method Expert ●Better: 200% Higher model returns than expert ● Faster/Cheaper: 10x faster than standard methods
- 57. 1. SigOpt LiveDemo 2. More Examples a. Text Classification b. Unsupervised + Supervised c. Neural Nets with TensorFlow ADDITIONAL TOPICS
- 58. AUTOMATICALLY TUNING TEXT SENTIMENTCLASSIFIER
- 59. ● Automatically tunetext sentiment classifier ● Amazon product review dataset (35K labels) eg : “This DVD is great. It brings back all the memories of the holidays as a young child.” ● Logistic regression is generally a good place to start PROBLEM
- 60. ● Maximize meanof k-fold cross-validation accuracies ● k = 5 folds, train and valid randomly split 70%, 30% OBJECTIVE FUNCTION
- 61. ● n-gram vocabularyselection parameters ● (min_n_gram, ngram_offset) determine which n-grams ● (log_min_df, df_offset) filter for n-grams within df range TEXT FEATURE PARAMETERS Original Text “SigOpt optimizes any complicated system” 1-grams { “SigOpt”, “optimizes”, “any”, “complicated”, “system”} 2-grams { “SigOpt_optimizes”, “optimizes_any”, “any_complicated” … } 3-grams { “SigOpt_optimizes_any”, “optimizes_any_complicated” … }
- 62. ● Logistic regressionerror cost parameters M = number of training examples θ = vector of weights the algorithm will learn for each n-gram in vocabulary yi - training data label : {-1, 1} for our two class problem xi - training data input vector: BOW vectors described in previous section α - weight of regularization term (log_reg_coef in our experiment) ρ - weight of l1 norm term (l1_coef in our experiment) ERROR COST PARAMETERS
- 63. ● 50 linepython snippet to train and tune classifier with SigOpt ● 20 lines to define 6 parameter experiment and run optimization loop using SigOpt PYTHON CODE
- 64. ● E[f (λ)]after 20 runs, each run consisting of 60 function evaluations ● For Grid Search : 64 evenly spaced parameter configurations (order shuffled randomly) ● SigOpt statistical significance over grid and rnd (p = 0.0001, Mann-Whitney U test) PERFORMANCE SigOpt Rnd. Search Grid Search No Tuning (Baseline) Best Found 0.8760 (+5.72%) 0.8673 (+4.67%) 0.8680 (+4.76%) 0.8286
- 65.
- 66. ● Classify housenumber digits with lack of labelled data ● Challenging digit variations, image clutter with neighboring digits PROBLEM
- 67. ● In generalwe’ll search for an optimized ML pipeline OBJECTIVE
- 68. ● Transform imagepatches into vectors of centroid distances, then pool to form final representation ● SigOpt optimizes selection of w, pool_r, K UNSUPERVISED MODEL PARAMS
- 69. ● Whitening transformoften useful as image data pre-processing step, expose εZCA to SigOpt UNSUPERVISED MODEL PARAMS
- 70. ● Tune sparsityof centroid distance transform ● SigOpt optimizes threshold (active_p) selection UNSUPERVISED MODEL PARAMS
- 71. ● learning rate numberof trees tree parameters (max_depth, sub_sample_sz), exposed to SigOpt SUPERVISED MODEL PARAMS
- 72.
- 73. ● 20 optimizationruns, each run consisting of 90 / 40 function evaluations for Unsup / Raw feature settings ● Optimized single CV fold on training set, ACC reported on test set as hold out PERFORMANCE SigOpt (Unsup Feats) Rnd Search (Unsup Feats) SigOpt (Raw Feats) Rnd Search (Raw Feats) No Tuning RF (Raw Feats) Hold Out ACC 0.8708 (+51.4%) 0.8583 0.6844 0.6739 0.5751
- 74.
- 75. ● Classify housenumbers with more training data and more sophisticated model PROBLEM
- 76. ● TensorFlow makesit easier to design DNN architectures, but what structure works best on a given dataset? CONVNET STRUCTURE
- 77. ● Per parameter adaptiveSGD variants like RMSProp and Adagrad seem to work best ● Still require careful selection of learning rate (α), momentum (β), decay (γ) terms STOCHASTIC GRADIENT DESCENT
- 78. ● Comparison ofseveral RMSProp SGD parametrizations ● Not obvious which configurations will work best on a given dataset without experimentation STOCHASTIC GRADIENT DESCENT
- 79.
- 80. ● Avg Holdout accuracy after 5 optimization runs consisting of 80 objective evaluations ● Optimized single 80/20 CV fold on training set, ACC reported on test set as hold out PERFORMANCE SigOpt (TensorFlow CNN) Rnd Search (TensorFlow CNN) No Tuning (sklearn RF) No Tuning (TensorFlow CNN) Hold Out ACC 0.8130 (+315.2%) 0.5690 0.5278 0.1958
- 81. COST ANALYSIS Model Performance (CVAcc. threshold) Random Search Cost SigOpt Cost SigOpt Cost Savings Potential Savings In Production (50 GPUs) 87 % $275 $42 84% $12,530 85 % $195 $23 88% $8,750 80 % $46 $21 55% $1,340 70 % $29 $21 27% $400
- 82.
- 83. MORE EXAMPLES Automatically TuningText Classifiers (with code) A short example using SigOpt and scikit-learn to build and tune a text sentiment classifier. Tuning Machine Learning Models (with code) A comparison of different hyperparameter optimization methods. Using Model Tuning to Beat Vegas (with code) Using SigOpt to tune a model for predicting basketball scores. Learn more about the technology behind SigOpt at https://sigopt.com/research