SigOpt. Confidential.
Exploring the spectrum of black-box optimization
through customer engagements
Harvey Cheng
05/03

SigOpt. Confidential.
Bayesian optimization

SigOpt. Confidential.
Bayesian optimization

SigOpt. Confidential.
Black-box optimization
In the research literature

SigOpt. Confidential.
Black-box optimization
A spectrum from customer engagement
Can you solve this
problem for me?
Glass box
Set and forget;
leave us alone.
Black box

SigOpt. Confidential.
Black-box optimization
A spectrum from customer engagement
Glass box Black box
Academic
collaboration

SigOpt. Confidential.
Material science research
Our Pitt collaborators wanted to use BO to
accelerate their research.
• Started using SigOpt as is and got some OK
results.
• Complicated experimental set-up and
SigOpt was not a 1-to-1 perfect fit.
• Were uninspired and had a lot of questions.

SigOpt. Confidential.
Find the fabrication process for an optical device (e.g., glass) that exhibits desirable physical and optical
properties.
• High transmission
• Low haze
• High liquid contact angle
Material science research

SigOpt. Confidential.
Find the fabrication process for an optical device (e.g., glass) that exhibits desirable physical and optical
properties.
• High transmission
• Low haze
• High liquid contact angle
Material science research
500 nm 100 nm 100 nm

SigOpt. Confidential.
Bayesian optimization for material science
What are our collaborator’s limitations? Their preferences? Their aspirations? Their fears?

SigOpt. Confidential.
Bayesian optimization for material science
What are our collaborator’s limitations? Their preferences? Their aspirations? Their fears?
Extremely budget conscious.

SigOpt. Confidential.
Bayesian optimization for material science
What are our collaborator’s limitations? Their preferences? Their aspirations? Their fears?
Fear of exploration.

SigOpt. Confidential.
Bayesian optimization for material science
What are our collaborator’s limitations? Their preferences? Their aspirations? Their fears?
Expertise is a double-edged sword.

SigOpt. Confidential.
Efficient search for the desirable material
Modification and Adaptation of BO
The researchers want:
• To leverage their expertise.
How we can help:
• Careful consideration of the input parameter
space.

SigOpt. Confidential.
Efficient search for the desirable material
Modification and Adaptation of BO
The researchers want:
• Lab equipment has limited precision.
How we can help:
• Understand how equipment precision
demands a discrete domain.

SigOpt. Confidential.
Efficient search for the desirable material
Modification and Adaptation of BO
The researchers want:
• Multiobjective optimization, on a budget.
How we can help:
• Pose the problem as a constrained
optimization problem to identify key points
on the Pareto frontier.

SigOpt. Confidential.
Efficient search for the desirable material

SigOpt. Confidential.
Optimization platform
Some of our customers cannot reveal anything
about their problems.
• Masking the input parameter names

SigOpt. Confidential.
Optimization platform
Some of our customers cannot reveal anything
about their problems.
• Masking the input parameter names
• Misusage sometimes

SigOpt. Confidential.
Optimization platform
Some of our customers cannot reveal anything
about their problems.
• Masking the input parameter names
• Misusage sometimes
• Vastly different access patterns

SigOpt. Confidential.
Black-box optimization
A spectrum from customer engagement
Academic
collaboration
Glass box Black box
Optimization
platform

SigOpt. Confidential.
Optimization platform
True black-box optimization
What can we do in the absence of customer interaction?
• Nonstandard benchmarking of our optimizer
• Flexible design of the API
• Scalable computation workflow

SigOpt. Confidential.
Scalable computation workflow
Online/Offline computation

SigOpt. Confidential.
Scalable computation workflow
Online/Offline computation

SigOpt. Confidential.
Black-box optimization
A spectrum from customer engagement
Academic
collaboration
Glass box Black box
Professional
services
Optimization
platform

SigOpt. Confidential.
Customer engagement at scale
Lessons we learned from working with our friends in Academia.
1. Customers’ problems may not be addressed immediately by the existing service.
2. Customers have inherent preferences to what is considered as success and what is considered as
failure.
Can we apply the lessons we have learned and serve a broad array of customers?

SigOpt. Confidential.
Professional services
When a customer’s problem cannot be immediately solved by plugging in the SigOpt API, the PS team can
help them by
• Understanding the customer’s success criteria.
• Building one-off projects to better interface SigOpt API with the customers.

SigOpt. Confidential.
Adjusting to customer expectations
Customers may judge their experience very differently from how we perceive it.

SigOpt. Confidential.
Adjusting to customer expectations
Customers may judge their experience very differently from how we perceive it.
• “Want more exploiting, less exploring because there is one region where there is an optimal value.
Not interested in exploring poor areas of performance.” - customer A
• “Want to make sure the optimizer is effectively exploring the parameter space. Don’t mind if there’s a
bit of extra work being done so long as it is sufficiently explored.” - customer B

SigOpt. Confidential.
Production throttles
Adjusting to customer biases
Production throttles are hooks that we can build
into the system to empower PS.
• Adjust the optimization behavior to meet the
customer’s demand.
• No redeployment of code.

SigOpt. Confidential.
Our spectrum thus far
Collaboration - Services - Black box
We have some customers with whom we have a collaborative relationship.
For most of customers, we treat their problems as completely black box.
We have identified opportunities to allow professional services to change the black box behavior without
changing the product structure.
Is this all?

SigOpt. Confidential.
Beyond black box
2016 BayesOpt workshop: We should “Open the black-box” and go “Beyond” black-box optimization.
What are some options?
• Address problems with a different goal (e.g., balancing competing objectives).
• Provide more information (level of noise for observations).
• Operate in a different workflow.
To do this, we must understand customer’s success criteria and failure modes.

SigOpt. Confidential.
Special feature for neural networks
Beyond black box
We wanted to build a feature to effectively address neural network developers. We needed to identify the
appropriate part of the black box spectrum on which this feature should lie.
Academic
collaboration
Glass box Black box
Professional
services
Optimization
platform

SigOpt. Confidential.
Special feature for neural networks
Beyond black box
Option 1: Hyperband
• Some customers had mentioned Hyperband, which could be implemented using a Bayesian
optimization strategy in the background.

SigOpt. Confidential.
Special feature for neural networks
Beyond black box
Option 1: Hyperband
• Some customers had mentioned Hyperband, which could be implemented using a Bayesian
optimization strategy in the background.
Complication:
• When confronted with the required change in workflow (storing weights, work split across generations,
idle machines), and the prospect of stopping SGD before convergence, customers balked.
Conclusion:
• Workflow change was too much towards glass box.

SigOpt. Confidential.
Special feature for neural networks
Beyond black box
Option 2: Multi-task BO
• We have a multi-task BO feature already in place; maybe neural network customers could be
convinced to use that.

SigOpt. Confidential.
Special feature for neural networks
Beyond black box
Option 2: Multi-task BO
• We have a multi-task BO feature already in place; maybe neural network customers could be
convinced to use that.
Complication:
• During customer interviews, we found customers confused by the definition of tasks and how to define
them effectively. They also disliked stopping SGD runs before convergence.
Conclusion:
• Existing multi-task feature was too black box.

SigOpt. Confidential.
Special feature for neural networks
Beyond black box
Resolution: Training Monitor
• Respect the customer’s workflow (no change required).
• Customers report progress during training.
• Allow customers to monitor training and provide status updates regarding convergence.
• Better internal models are built from all the progress information.
Conclusion:
• Does this fall in the best location of the spectrum?

SigOpt. Confidential.
Black-box optimization
A spectrum from customer engagement
Academic
collaboration
Glass box Black box
Professional
services
Optimization
platform
HPO for neural
networks

SigOpt. Confidential.
Thank You
Paul Leu
Sajad Haghanifar
@University of Pittsburgh
The entire SigOpt team.
Special thanks to our gracious hosts, Jake, Matthias, and Uber.
Hope to see you next month at ICML & CVPR! SigOpt events are being planned ...

SigOpt. Confidential.
Questions