SigOpt. Confidential.
Why is Experiment Management
important for NLP?

SigOpt. Confidential.
First, a quick overview of the
problem

SigOpt. Confidential.
Two main questions
3
Can we understand the
trade-offs made during
model compression?
Can we find a model
architecture that fits our
needs?

SigOpt. Confidential.
The Data: SQUAD 2.0
4
SQUAD 2.0

SigOpt. Confidential.
Distilling BERT for Question Answering
5
BERT
Pre-trained for language
modeling
Student Model
SQUAD 2.0
SQUAD 2.0
Soft
target
loss
Hard
target
loss
BERT
Fine-tuned for SQUAD 2.0
Trained Student Model
For more on distillation: Hinton et al 2015, DistilBERT

SigOpt. Confidential.
Multimetric Bayesian Optimization
Optimizing for two competing metrics
6
SigOpt’s Multimetric Optimization

SigOpt. Confidential.
Experiment Management was critical
throughout this process

SigOpt. Confidential.
Experiment management was critical for
8
Model Development
Understanding the Problem
Space
Monitoring Long Cycles

SigOpt. Confidential.
Model Development

SigOpt. Confidential.
Establishing a Baseline
10
BERT
Pre-trained for language
modeling
Student Model
SQUAD 2.0
SQUAD 2.0
Soft
target
loss
Hard
target
loss
BERT
Fine-tuned for SQUAD 2.0
Trained Student Model
?
?
?
? ?

SigOpt. Confidential.
Establishing a Baseline: Training from scratch
11
BERT
Pre-trained for language
modeling
DistilBERT
SQUAD 2.0
SQUAD 2.0
Standard
soft target
loss
Standard
hard
target
loss
BERT
Fine-tuned for SQUAD 2.0
Trained Model
? ?
?
?
?

SigOpt. Confidential.
Baseline #1: Trained from scratch
Dashboard link

SigOpt. Confidential.
Establishing a Baseline: Warm starting the model
13
BERT
Pre-trained for language
modeling
DistilBERT
SQUAD 2.0
SQUAD 2.0
Standard soft
target loss
Standard
Hard target
loss
BERT
Fine-tuned for SQUAD 2.0
Trained Model
DistilBERT
Pre-trained for language
modeling
Pretrained Weights
?
?

SigOpt. Confidential.
Baseline #2: Pretrained weights
Dashboard link

SigOpt. Confidential.
Understanding the problem space

SigOpt. Confidential.
Running HPO to understand the problem space
16

SigOpt. Confidential.
Let’s take a look at the experiment
dashboard

SigOpt. Confidential.
Correlations in the parameter space
18

SigOpt. Confidential.
Exploring specific parameter areas
19
Runs dashboard

SigOpt. Confidential.
Taking data properties into account
20

SigOpt. Confidential.
Providing feedback to the optimizer
21

SigOpt. Confidential.
Monitoring the full experiment

SigOpt. Confidential.
Monitoring the full experiment
23
Run dashboard

SigOpt. Confidential.
SigOpt found dozens of viable models
24
Baseline Exact
Baseline
Size
Metric Threshold

SigOpt. Confidential.
How did experiment management
help throughout my process?

SigOpt. Confidential.26
Model Development
Understanding the
Problem Space
Monitoring Long Cycles
Experiment Validation
Experiment design and
exploring the
parameter space
Tracking and
Debugging

SigOpt. Confidential.
So why Experiment Management?
27

SigOpt. Confidential.
Check out our
YouTube channel:
Learn more about SigOpt
Read our research and product blog.
See more videos here.
Sign up to try out SigOpt
for free!
Join the Experiment Management
beta
Click Here
Read the full work on Nvidia’s
dev blog

SigOpt. Confidential.
Thank you!