SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.
SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.
Successfully reported this slideshow.
Activate your 14 day free trial to unlock unlimited reading.
SigOpt founder and CEO, Scott Clark, PhD, explains the tradeoffs you'll want to consider when designing your modeling platform and integrating hyperparameter optimization to enhance data scientist productivity.
SigOpt founder and CEO, Scott Clark, PhD, explains the tradeoffs you'll want to consider when designing your modeling platform and integrating hyperparameter optimization to enhance data scientist productivity.
1.
Accelerate and amplify the
impact of modelers everywhere
2.
SigOpt. Confidential.3
How I got here: 10+ years of tuning models
3.
Differentiated
Models
Tailored Models
10,000x
Analytics 2.0 Models
100x
1x
Modelers by Segment Value per Model
Enterprise AI
Goals:
Differentiate Products
Generate Revenue
Requirements:
Modelers with Expertise
Best-in-Class Solutions
4.
SigOpt. Confidential.
Your firewall
Training
Data
AI, ML, DL,
Simulation Model
Model Evaluation
or Backtest
Testing
Data
New
Configurations
Objective
Metric
Better
Results
EXPERIMENT INSIGHTS
Track, organize, analyze and
reproduce any model
ENTERPRISE PLATFORM
Built to fit any stack and scale
with your needs
OPTIMIZATION ENGINE
Explore and exploit with a
variety of techniques
RESTAPI
Configuration
Parameters or
Hyperparameters
Your data
and models
stay private
Iterative, automated optimization
Integrates
with any
modeling
stack
5.
$300B+
in assets under management
Current SigOpt algorithmic
trading customers represent
$500B+
in market capitalization
Current SigOpt enterprise customers
across six industries represent
6.
SigOpt. Confidential.
“We’ve integrated SigOpt’s optimization service and
are now able to get better results faster and cheaper
than any solution we’ve seen before.”
Matt Adereth
Managing Director
7.
Lessons
1. Balance flexibility with standardization
2. Maximize resource utilization
3. Unlock new modeling capabilities
9.
Simulations & Evaluation
Modeling FrameworkData
Management
Data access
Data pipelines
Data labeling
Feature repo.
Feature prov.
Backfills,
versioning
Compute Environment
Model Execution
Model Serving
Instrumentation
Data validation
Phased deploys
Online A/B tests
Batch scoring
On-Premise
Optimization & Experimentation
Hybrid Multi-Cloud
Tracking, Visualization,
Version Control
Automated Hyperparameter
Optimization
Distributed Tuning & Job
Scheduling
Workflows
Auto Feature &
AutoML
Coding
Environment
Framework
Support
Library
Backtests Metric Iteration
Model
Evaluation
Portfolio
Optimization
10.
Simulations & Evaluation
Strategy: Customize and differentiate
Modeling Framework
Strategy: Modeler choice and flexibility
Data
Management
Strategy:
Customize and
differentiate
Compute Environment
Strategy: Modeler choice and flexibility
Model
Execution
Strategy:
Customize and
differentiate
Optimization & Experimentation
Strategy: Standardize and scale
11.
Simulations & Evaluation
Strategy: Customize and differentiate
Modeling Framework
Strategy: Modeler choice and flexibility
Data
Management
Strategy:
Customize and
differentiate
Compute Environment
Strategy: Modeler choice and flexibility
Model
Execution
Strategy:
Customize and
differentiateOptimization & Experimentation
Strategy: Standardize and scale
12.
Simulations & Evaluation
Strategy: Customize and differentiate
Modeling Framework
Strategy: Modeler choice and flexibility
Data
Management
Strategy:
Customize and
differentiate
Compute Environment
Strategy: Modeler choice and flexibility
Model
Execution
Strategy:
Customize and
differentiate
Optimization & Experimentation
Strategy: Standardize and scale
13.
Simulations & Evaluation
Strategy: Customize and differentiate
Modeling Framework
Strategy: Modeler choice and flexibility
Data
Management
Strategy:
Customize and
differentiate
Compute Environment
Strategy: Modeler choice and flexibility
Model
Execution
Strategy:
Customize and
differentiate
Optimization & Experimentation
Strategy: Standardize and scale
14.
Data
Management
Data access
Data pipelines
Data labeling
Feature repo.
Feature prov.
Backfills,
versioning
Simulation & Evaluation
Modeling Framework
Hardware Environment
On-Premise Hybrid Multi-Cloud
Optimization & Experimentation
Insights, Tracking,
Collaboration
Model Search,
Hyperparameter Tuning
Resource Scheduler,
Management
Backtests Metric Iteration
Model
Evaluation
Portfolio
Optimization
Model
Execution
Model Serving
Instrumentation
Data validation
Phased deploys
Online A/B tests
Batch scoring
Flexibility + Standardization
15.
SigOpt. Confidential.
Benefits of Robust Experimentation & Optimization
Learn fast, fail fast
Give yourself the best chance at finding good use
cases while avoiding false negatives
Connect outputs to outcomes
Define, select and iterate on your metrics
with end-to-end evaluation
Find the global maximum
Early non-optimized decisions in the process limit
your ability to maximize performance
Boost productivity
Automate modeling tasks so modelers spend
more time applying their expertise
19.
SigOpt. Confidential.
Hyperparameter Optimization
Model Tuning
Grid Search
Random Search Bayesian Optimization
Training & Tuning
Evolutionary Algorithms
Deep Learning Architecture Search
Hyperparameter Search
20.
SigOpt. Confidential.
Pro Con
Manual Search Leverages expertise Not scalable, inconsistent
Grid Search Simple to implement Not scalable, often infeasible
Random Search Scalable Inefficient
Evolutionary Algorithms Effective at architecture search Very resource intensive
Bayesian Optimization Efficient, effective Can be tough to parallelize
21.
SigOpt. Confidential.
Pro Con
Manual Search Leverages expertise Not scalable, inconsistent
Grid Search Simple to implement Not scalable, often infeasible
Random Search Scalable Inefficient
Evolutionary Algorithms Effective at architecture search Very resource intensive
Bayesian Optimization Efficient, effective Can be tough to parallelize
22.
SigOpt. Confidential.
Pro Con
Manual Search Leverages expertise Not scalable, inconsistent
Grid Search Simple to implement Not scalable, often infeasible
Random Search Scalable Inefficient
Evolutionary Algorithms Effective at architecture search Very resource intensive
Bayesian Optimization Efficient, effective Can be tough to parallelize
23.
SigOpt. Confidential.
Pro Con
Manual Search Leverages expertise Not scalable, inconsistent
Grid Search Simple to implement Not scalable, often infeasible
Random Search Scalable Inefficient
Evolutionary Algorithms Effective at architecture search Very resource intensive
Bayesian Optimization Efficient, effective Can be tough to parallelize
24.
SigOpt. Confidential.
Pro Con
Manual Search Leverages expertise Not scalable, inconsistent
Grid Search Simple to implement Not scalable, often infeasible
Random Search Scalable Inefficient
Evolutionary Algorithms Effective at architecture search Very resource intensive
Bayesian Optimization Efficient, effective Can be tough to parallelize
25.
Your firewall
Training
Data
AI, ML, DL,
Simulation Model
Model Evaluation
or Backtest
Testing
Data
New
Configurations
Objective
Metric
Better
Results
EXPERIMENT INSIGHTS
Track, organize, analyze and
reproduce any model
ENTERPRISE PLATFORM
Built to fit any stack and scale
with your needs
OPTIMIZATION ENGINE
Explore and exploit with a
variety of techniques
RESTAPI
Configuration
Parameters or
Hyperparameters
100x
Asynchronous
Parallelization
26.
SigOpt. Confidential.
24 Days
to optimize a model without
SigOpt
3 Days
to optimize a model with
SigOpt
Learn more: https://twosigma.com/news/article/why-two-sigma-is-using-sigopt-for-automated-parameter-tuning/
29.
SigOpt. Confidential.
Unlock new modeling capabilities
30
Accelerate Model Optimization
● Multitask optimization efficiently
optimizes “expensive” models
● Relevant docs
● Image classification use case
Solve for Competing Business Objectives
● Multimetric optimization empowers
you to evaluate trade-offs
● Relevant docs
● Sequence classification use case
Explore Model Architectures
● Conditional parameters result in more
efficient architecture search
● Relevant docs
● NLP use case
30.
SigOpt. Confidential.
Train and tune expensive models
31
Accelerate Model Optimization
● Multitask optimization efficiently
optimizes “expensive” models
● Relevant docs
● Image classification use case
Solve for Competing Business Objectives
● Multimetric optimization empowers
you to evaluate trade-offs
● Relevant docs
● Sequence classification use case
Explore Model Architectures
● Conditional parameters result in more
efficient architecture search
● Relevant docs
● NLP use case
Combine the intelligence of Bayesian
optimization with the efficiency of early
termination techniques
31.
SigOpt. Confidential.
Compare and select the right metric
32
Accelerate Model Optimization
● Multitask optimization efficiently
optimizes “expensive” models
● Relevant docs
● Image classification use case
Solve for Competing Business Objectives
● Multimetric optimization empowers
you to evaluate trade-offs
● Relevant docs
● Sequence classification use case
Explore Model Architectures
● Conditional parameters result in more
efficient architecture search
● Relevant docs
● NLP use case
Optimize multiple metrics at the same
time to inform your metric definition and
selection process
32.
SigOpt. Confidential.
Identify the right tool for the job
33
Accelerate Model Optimization
● Multitask optimization efficiently
optimizes “expensive” models
● Relevant docs
● Image classification use case
Solve for Competing Business Objectives
● Multimetric optimization empowers
you to evaluate trade-offs
● Relevant docs
● Sequence classification use case
Explore Model Architectures
● Conditional parameters result in more
efficient architecture search
● Relevant docs
● NLP use case
Take into account the conditionality of
certain parameter types in the
optimization process
33.
Lessons
1. Balance flexibility with standardization
2. Maximize resource utilization
3. Unlock new modeling capabilities
34.
Try our solution
Our team is here to help.
Find our table at Booth 02,
and sign up to demo SigOpt today.
Join our webinar on
modeling platforms
https://sigopt.com/company
/events/webinar-capabilities
-ml-platforms/
Listen to our recent podcast with Two Sigma
https://twimlai.com/twiml-talk-273-supporting
-rapid-model-development-at-two-sigma-
with-matt-adereth-scott-clark/
Download TWIML’s Modeling Platforms ebook
https://sigopt.com/guide-for-
machine-learning-platforms/