Download as PDF, PPTX
![Pipeline on DDF and BayesOpt
@arimoinc
train_ddf = session.get_ddf(…)
valid_ddf = session.get_ddf(…)
optimizer = SpearmintOptimizer(chooser_name=‘GPEIperSecChooser',
max_finished_jobs=max_iters, grid_size=5000, ..)
best_params, trace = auto_model(
optimizer, train_ddf, 'arrdelay',
classification=True,
excluded_columns=['actualelapsedtime','arrtime', 'year'],
validation_ddf=val_ddf)](https://image.slidesharecdn.com/7inguyenpham-160615004246/85/Automatic-Features-Generation-And-Model-Training-On-Spark-A-Bayesian-Approach-21-320.jpg)
![Experiment 1: SF Crimes dataset
@arimoinc
Hyper-parameter Type Range
Number of hidden layers INT 1, 2, 3
Number of hidden units INT 64, 128, 256
Dropout at the input layer FLOAT [0, 0.5]
Dropout at the hidden layers FLOAT [0,0.75]
Learning rate FLOAT [0.01, 0.1]
L2 Weight decay FLOAT [0, 0.01]
Logloss on the validation set
Running time (hours) ~ 40 iterations](https://image.slidesharecdn.com/7inguyenpham-160615004246/85/Automatic-Features-Generation-And-Model-Training-On-Spark-A-Bayesian-Approach-24-320.jpg)
![Experiment 1: SF Crimes dataset
@arimoinc
Hyper-parameter Type Range Spearmint
Number of hidden layers INT 1, 2, 3 2
Number of hidden units INT 64, 128, 256 256
Dropout at the input layer FLOAT [0, 0.5] 0.423678
Dropout at the hidden layers FLOAT [0,0.75] 0.091693
Learning rate FLOAT [0.01, 0.1] 0.025994
L2 Weight decay FLOAT [0, 0.01] 0.00238
Logloss on the validation set 2.1502
Running time (hours) ~ 40 iterations 15.8](https://image.slidesharecdn.com/7inguyenpham-160615004246/85/Automatic-Features-Generation-And-Model-Training-On-Spark-A-Bayesian-Approach-25-320.jpg)
![Experiment #1: With SigOpt
@arimoinc
Hyper-parameter Type Range Spearmint
Number of hidden layers INT 1, 2, 3 2 3
Number of hidden units INT 64, 128, 256 256 256
Dropout at the input layer FLOAT [0, 0.5] 0.423678 0.3141
Dropout at the hidden layers FLOAT [0,0.75] 0.091693 0.0944
Learning rate FLOAT [0.01, 0.1] 0.025994 0.0979
L2 Weight decay FLOAT [0, 0.01] 0.00238 0.0039
Logloss on the validation set 2.1502 2.14892
Running time (hours) ~ 40 iterations 15.8 20.1](https://image.slidesharecdn.com/7inguyenpham-160615004246/85/Automatic-Features-Generation-And-Model-Training-On-Spark-A-Bayesian-Approach-27-320.jpg)
![Experiment #2: Hyper-parameters
@arimoinc
Hyperparameter Type Range BayesOpt
Number of SVD dimensions INT [5, 100] 98
Top feature percentage FLOAT [0.1, 1] 0.8258
k (# of clusters) INT [1, 6] 2
Number of trees (RF) INT [50, 500] 327
Max. depth (RF) INT [1, 20] 12
Min. instances per node (RF) INT [1, 1000] 414
F1-score on validation set 0.8736](https://image.slidesharecdn.com/7inguyenpham-160615004246/85/Automatic-Features-Generation-And-Model-Training-On-Spark-A-Bayesian-Approach-31-320.jpg)
Uploaded bySpark Summit
Automatic Features Generation And Model Training On Spark: A Bayesian Approach
The document discusses automating data science on Spark using a Bayesian approach for hyper-parameter tuning, featuring various methods such as Bayesian optimization and a machine learning pipeline. It includes experimental results from two case studies: one focusing on SF crime data and the other on airline delays, highlighting hyper-parameter selections and experiments conducted for optimization. The authors promote the use of tools like Spearmint and Hyperopt for enhancing automated data science workflows.
More Related Content
![Samantha Wang [InfluxData] | Best Practices on How to Transform Your Data Usi...](https://cdn.slidesharecdn.com/ss_thumbnails/influxdaysnorthamerica2020-201005185945-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Automatic Features Generation And Model Training On Spark: A Bayesian Approach
![[ppt]](https://cdn.slidesharecdn.com/ss_thumbnails/ppt2931-thumbnail.jpg?width=640&height=640&fit=bounds)
![[ppt]](https://cdn.slidesharecdn.com/ss_thumbnails/ppt3441-thumbnail.jpg?width=640&height=640&fit=bounds)
![제 23회 보아즈(BOAZ) 빅데이터 컨퍼런스 - [MBOAX] : ABSA를 활용한 소비자 반응 분석 기반 운영 효율화 대시보드 설계](https://cdn.slidesharecdn.com/ss_thumbnails/3-1boaz23rdconferencemboax-260203102709-9d519923-thumbnail.jpg?width=640&height=640&fit=bounds)
Automatic Features Generation And Model Training On Spark: A Bayesian Approach
- 1. Automating Data Scienceon Spark: A Bayesian Approach Vu Pham, Huan Dao, Christopher Nguyen San Francisco, June 8th 2016
- 2.
- 3.
- 4. Agenda @arimoinc 1. For Hyper-parameterTuning 2. For “Automating” Data Science on Spark 3. Experiments
- 5.
- 6.
- 7. Hyper-parameters @arimoinc K-means K Neural Network# of layers, dropout, momentum… Random Forest Feature set, # of trees, max depth… SVM Regularization term (C) Gradient Descent Learning rate, number of iterations…
- 8.
- 9.
- 10.
- 11. Hyper-parameters tuning @arimoinc U: SR c U(c) Hyper-parameter Space Performance onValidation Set How to intelligently select the next configuration? (Given the observations in the past) Maximize a utility functions over hyper-parameter space:
- 12.
- 13. Bayesian Optimization Explained @arimoinc 1.Incorporate a prior over the space of possible objective functions (GP) 2. Combine the prior with the likelihood to obtain a posterior over function values given observations 3. Select next configuration to evaluate based on the posterior • According to an acquisition function
- 14.
- 15. @arimoinc expensive multi-modal noisy blackbox Bayesian Optimization isto globally optimize functions that are:
- 16. @arimoinc Bayesian Optimization for “Automating”Data Science on Spark
- 17. Automatic Data ScienceWorkflow @arimoinc Feature Engineering Model Selection Hyperparameter Tuning
- 18. Machines doing partof Data Science @arimoinc Dataset Feature preprocessing Data Cleaning Generate new features Feature selection Dimensionality reduction Predictive modeling Scoring
- 19. A generic MachineLearning pipeline @arimoinc Training set Val/Test set Feature preprocessing Truncated SVD n: # of dims f-score with target variable Keep alpha percents top features K-means k: # of clusters RandomForest #1 number of trees maximum depth percentage of features RandomForest #k … Scoring
- 20. Spark WorkersSpark Workers Pipelineon DDF and BayesOpt @arimoinc Spark Driver DDF Arimo Pipeline Manager … Client Bayesian Optimizer Client uses Bayesian Optimizer to select the hyper-parameters of the pipeline so that it maximizes the performance on a validation set Spark Workers
- 21. Pipeline on DDFand BayesOpt @arimoinc train_ddf = session.get_ddf(…) valid_ddf = session.get_ddf(…) optimizer = SpearmintOptimizer(chooser_name=‘GPEIperSecChooser', max_finished_jobs=max_iters, grid_size=5000, ..) best_params, trace = auto_model( optimizer, train_ddf, 'arrdelay', classification=True, excluded_columns=['actualelapsedtime','arrtime', 'year'], validation_ddf=val_ddf)
- 22.
- 23. Experiment 1: SFCrimes @arimoinc Dates Category Descript DayOfWeek PdDistrict Resolution Address X Y 2015-05-13 23:53:00 WARRANTS WARRANT ARREST Wednesday NORTHERN ARREST, BOOKED OAK ST / LAGUNA ST -122.4258 37.7745 2015-05-13 23:53:00 OTHER OFFENSES TRAFFICVIOLATION ARREST Wednesday NORTHERN ARREST, BOOKED OAK ST / LAGUNA ST -122.4258 37.7745 2015-05-13 23:33:00 OTHER OFFENSES TRAFFICVIOLATION ARREST Wednesday NORTHERN ARREST, BOOKED VANNESS AV / GREENWICH ST -122.4243 37.8004
- 24. Experiment 1: SFCrimes dataset @arimoinc Hyper-parameter Type Range Number of hidden layers INT 1, 2, 3 Number of hidden units INT 64, 128, 256 Dropout at the input layer FLOAT [0, 0.5] Dropout at the hidden layers FLOAT [0,0.75] Learning rate FLOAT [0.01, 0.1] L2 Weight decay FLOAT [0, 0.01] Logloss on the validation set Running time (hours) ~ 40 iterations
- 25. Experiment 1: SFCrimes dataset @arimoinc Hyper-parameter Type Range Spearmint Number of hidden layers INT 1, 2, 3 2 Number of hidden units INT 64, 128, 256 256 Dropout at the input layer FLOAT [0, 0.5] 0.423678 Dropout at the hidden layers FLOAT [0,0.75] 0.091693 Learning rate FLOAT [0.01, 0.1] 0.025994 L2 Weight decay FLOAT [0, 0.01] 0.00238 Logloss on the validation set 2.1502 Running time (hours) ~ 40 iterations 15.8
- 26. Experiment 1: SFCrimes dataset @arimoinc 2.149 2.152 2.155 2.157 2.16 1 7 17 18 39 2.1600 2.1555 2.1552 2.1551 2.1519 2.1502 Completed jobs 1 7 16 17 18 39 Elapsed time 1021 8003 12742 1623 1983 31561 Number of layers 1 3 3 2 3 2 Hidden units 64 256 256 256 256 256 Learning rate 0.01 0.1 0.01 0.01 0.021 0.026 Input dropout 0 0.5 0.5 0.463 0.5 0.424 Hidden dropout 0 0 0 0.024 0.089 0.092 Weight decay 0 0 0.002 0.003 0 0.002
- 27. Experiment #1: WithSigOpt @arimoinc Hyper-parameter Type Range Spearmint Number of hidden layers INT 1, 2, 3 2 3 Number of hidden units INT 64, 128, 256 256 256 Dropout at the input layer FLOAT [0, 0.5] 0.423678 0.3141 Dropout at the hidden layers FLOAT [0,0.75] 0.091693 0.0944 Learning rate FLOAT [0.01, 0.1] 0.025994 0.0979 L2 Weight decay FLOAT [0, 0.01] 0.00238 0.0039 Logloss on the validation set 2.1502 2.14892 Running time (hours) ~ 40 iterations 15.8 20.1
- 28. SF Crimes -Time to results @arimoinc
- 29. Experiment #2: Airlinesdata @arimoinc Year 1987-2008 DepDelay departure delay, in minutes Month 1-12 Origin origin IATA airport code DayofMonth 1-31 Dest destination IATA airport code DayOfWeek 1 (Monday) - 7 (Sunday) Distance in miles DepTime actual departure time TaxiIn taxi in time, in minutes CRSDepTime scheduled departure time TaxiOut taxi out time in minutes ArrTime actual arrival time Cancelled was the flight cancelled? CRSArrTime scheduled arrival time CancellationCode reason for cancellation UniqueCarrier unique carrier code Diverted 1 = yes, 0 = no FlightNum flight number CarrierDelay in minutes TailNum plane tail number WeatherDelay in minutes ActualElapsedTime in minutes NASDelay in minutes CRSElapsedTime in minutes SecurityDelay in minutes AirTime in minutes LateAircraftDelay in minutes ArrDelay arrival delay, in minutes Delayed Is the flight delayed
- 30. Experiment #2 @arimoinc Training set Val/Testset Feature preprocessing ~900 features Truncated SVD n: # of dims f-score with target variable Keep alpha percents top features K-means k: # of clusters RandomForest #1 number of trees maximum depth percentage of features RandomForest #k … Scoring
- 31. Experiment #2: Hyper-parameters @arimoinc HyperparameterType Range BayesOpt Number of SVD dimensions INT [5, 100] 98 Top feature percentage FLOAT [0.1, 1] 0.8258 k (# of clusters) INT [1, 6] 2 Number of trees (RF) INT [50, 500] 327 Max. depth (RF) INT [1, 20] 12 Min. instances per node (RF) INT [1, 1000] 414 F1-score on validation set 0.8736
- 32. Summary @arimoinc 1. Bayesian Optimizationfor Hyper-parameter Tuning 2. Bayesian Optimization for “Automating” Data Science on Spark 3. Experiments
- 33. Getting Started @arimoinc • Blogpost:http://goo.gl/PFyBKI • Open-source: spearmint, hyperopt, SMAC, AutoML • Commercial: Whetlab, SigOpt, …
- 34.