Property of SigOpt, Inc. - Private & Confidential
Lessons for an Enterprise
Approach to Modeling at Scale
Nick Payton
Head of Marketing & Partnerships

Property of SigOpt, Inc. - Private & ConfidentialProperty of SigOpt, Inc. - Private & Confidential
Empower experts everywhere to
amplify and accelerate their
modeling impact

Property of SigOpt, Inc. - Private & Confidential
DevOps Builds and Maintains Proprietary Infrastructure
Tasks that depend on your particular infrastructure
(e.g., model lifecycle management, model deployment)
Experts Focus on Data Science
Tasks that benefit from domain expertise
(e.g., metric-function selection)
Our model management philosophy
Software Automates Repeatable Tasks
Tasks that do not benefit from domain expertise
(e.g., training orchestration, model tuning)

Property of SigOpt, Inc. - Private & Confidential
We never
access your
data or models
Iterative, automated optimization
Built specifically
for scalable
enterprise use
cases

Property of SigOpt, Inc. - Private & Confidential
Benefits: Better, cheaper, faster model development
90% Cost Savings
Maximize utilization of compute
https://aws.amazon.com/blogs/machine-learning/fast
-cnn-tuning-with-aws-gpu-instances-and-sigopt/
10x Faster Time to Tune
Less expert time per model
https://devblogs.nvidia.com/sigopt-deep-learning-hy
perparameter-optimization/
Better Performance
No free lunch, but optimize any model
https://arxiv.org/pdf/1603.09441.pdf

Property of SigOpt, Inc. - Private & ConfidentialProperty of SigOpt, Inc. - Private & Confidential
How does the enterprise maximize
the value of their AI/ML investment?

Property of SigOpt, Inc. - Private & Confidential
Source: Kai-Fu Lee, “AI Superpowers: China, Silicon Valley and the New World Order”
Four “waves” of AI progress
Wave 1
Internet AI
Wave 2
Business AI
Wave 3
Perception AI
Wave 4
Autonomous AI
General Data
General Purpose
General Problems
Proprietary Data
Proprietary Purpose
Proprietary Problems
Sensor Data
IoT/Edge Purpose
IoT/Edge Problems
Integrated Data
Multi-Purpose
Real-World Problems

Property of SigOpt, Inc. - Private & ConfidentialProperty of SigOpt, Inc. - Private & Confidential
“Differentiated”
Models
Augment Experts
“Repeatable”
Models
Replace Experts

Property of SigOpt, Inc. - Private & Confidential
Hypothesis
Differentiated models will unlock ROI on AI

Property of SigOpt, Inc. - Private & Confidential
But differentiated models require a different workflow
Source: Nick Elprin Presentation at Domino REV 2018

Property of SigOpt, Inc. - Private & Confidential
This workflow may require a modeling platform

Property of SigOpt, Inc. - Private & Confidential
Source: Gartner, “How to Operationalize Machine Learning and Data Science Projects,” July 2018

Property of SigOpt, Inc. - Private & Confidential
Source: Gartner, “How to Operationalize Machine Learning and Data Science Projects,” July 2018
3 distinct processes
>20 individual tasks
1 of many approaches
Varies by team
Where to start?

Property of SigOpt, Inc. - Private & Confidential
5 Lessons for an Enterprise
Approach to Modeling at Scale

Property of SigOpt, Inc. - Private & ConfidentialProperty of SigOpt, Inc. - Private & Confidential
Divide labor between
machines and experts
1

Source: Indeed, AI Jobs Report

Property of SigOpt, Inc. - Private & Confidential
ML
Engineer
Data Features Models Training Tuning Deploy Monitor
ML
Engineer
ML
Engineer
ML
Engineer
ML
Engineer
DevOps DevOps

Property of SigOpt, Inc. - Private & Confidential
Experimentation Production
Data Features Models
Training Tuning
Deploy Monitor
ML Engineer DevOps
Objective Metric
Objective Function
Business Outcome
Domain
Expertise
Solutions
Experiment Management, Infrastructure Orchestration, Optimization

Property of SigOpt, Inc. - Private & Confidential
Maximize domain expertise

Property of SigOpt, Inc. - Private & ConfidentialProperty of SigOpt, Inc. - Private & Confidential
2
Solve for flexibility
(with plug-and-play APIs)

Property of SigOpt, Inc. - Private & Confidential
Source: AI & Compute, OpenAI Blog, May 2018

Property of SigOpt, Inc. - Private & Confidential
GBMs Neural Nets GANs
Reinforcement
Learning

Property of SigOpt, Inc. - Private & Confidential
Source: Shivon Zilis, http://www.shivonzilis.com/

Property of SigOpt, Inc. - Private & Confidential
Source: Shivon Zilis, http://www.shivonzilis.com/

Property of SigOpt, Inc. - Private & Confidential
Lock yourself into a closed system at
your own risk

Property of SigOpt, Inc. - Private & ConfidentialProperty of SigOpt, Inc. - Private & Confidential
3 Analyze and
reproduce any model

Property of SigOpt, Inc. - Private & Confidential
Your models are a significant investment
Source: HTTPS://WWW.STATISTA.COM/STATISTICS/607612/WORLDWIDE-ARTIFICIAL-INTELLIGENCE-FOR-ENTERPRISE-APPLICATIONS/

Property of SigOpt, Inc. - Private & Confidential
And a growing need to interpret, understand models

Property of SigOpt, Inc. - Private & Confidential
Example in SigOpt’s solution
Uncover model insights with
parameter importance
Monitor performance improvement as
the experiment progresses via API, the
web or your mobile phone
Cycle through analysis, suggestions,
history, and other experiment insights

Property of SigOpt, Inc. - Private & Confidential
Experiment management is model
analysis and reproducibility

Property of SigOpt, Inc. - Private & ConfidentialProperty of SigOpt, Inc. - Private & Confidential
4 Optimize throughout
the process

Property of SigOpt, Inc. - Private & Confidential
The “suboptimal optimization” problem
Random forest
Grid search
75%
CNN
Grid search
65%
CNN
Bayesian optimization
85%

Property of SigOpt, Inc. - Private & Confidential
The “leaving optimization to the last mile” problem
Data Features Models Training Tuning Deploy Monitor
FIXED FIXED FIXED
Performance Leakage

Property of SigOpt, Inc. - Private & Confidential
The “performance drift in production” problem
Data Features Models Training Tuning Deploy Monitor
Static
Performance Drift

Property of SigOpt, Inc. - Private & Confidential
Retune withOptimize with
Optimization impacts every step in your process
Data Features Models Training Tuning Deploy Monitor
Automate Experimentation
Cluster
Management
Hyperparameter
Optimization
Web UX with Insights, Metadata, Visuals

Property of SigOpt, Inc. - Private & Confidential
Advanced optimization techniques are critical
Multitask Optimization
Tune “expensive” deep learning models
Multimetric Optimization
Solve for competing business objectives
Conditional Parameters
Perform optimized architecture search
100 Parameters, 100x Parallelism
Efficiently optimize high-dimensional models

Property of SigOpt, Inc. - Private & Confidential
The “competing objective” problem
Accuracy Training Time
ROC AUC Inference Time
Loss Model Complexity
Conversion Rate Lifetime Value
Engagement Profit
Profit Drawdown
VS.

Property of SigOpt, Inc. - Private & Confidential
Finding the frontier
Accuracy v Training Time Accuracy v Inference Time % Loss v Per-Loss Magnitude

Property of SigOpt, Inc. - Private & ConfidentialProperty of SigOpt, Inc. - Private & Confidential
5 Build for variety and
reliability

Property of SigOpt, Inc. - Private & Confidential
Source: https://machinelearningmastery.com/a-tour-of-machine-learning-algorithms/

Property of SigOpt, Inc. - Private & Confidential
Open source may work Open source may not work
Programming Languages
Client Libraries
Modeling Frameworks
Notebook Management
Hyperparameter Optimization
Experiment Management
Training Management
Model Deployment

Property of SigOpt, Inc. - Private & Confidential

Property of SigOpt, Inc. - Private & Confidential
Considerations
On-Premise
Hybrid Infrastructure
Single-Cloud
Multi-Cloud
Single User(s)
One Team
Multi-team needs
Platform-driven modeling
Center of Excellence
Number of use cases
Variety of model types
Diversity of expertise
Sources of data

Property of SigOpt, Inc. - Private & Confidential
Standardization is critical to
modeling at scale

Property of SigOpt, Inc. - Private & Confidential
5 Lessons for an Enterprise
Approach to Modeling at Scale

Property of SigOpt, Inc. - Private & Confidential
Divide labor between machines and experts
Solve for flexibility
Analyze and reproduce any model
Optimize throughout the process
Build for variety and reliability

Property of SigOpt, Inc. - Private & Confidential
Realize the virtuous cycle of model development
1. Invest in tools to automate, optimize and manage the process
2. Improve team productivity and throughput
3. Free up capacity to apply expertise to metrics, outcomes
4. Amplify the business impact of models

Property of SigOpt, Inc. - Private & Confidential
Thank you