1.
Terry Taewoong Um (terry.t.um@gmail.com)
University of Waterloo
Department of Electrical & Computer Engineering
Terry T. Um
UNDERSTANDING BLACK-BOX PRED
-ICTION VIA INFLUENCE FUNCTIONS
1

2.
TODAY’S PAPER
Terry Taewoong Um (terry.t.um@gmail.com)
ICML2017 best paper
https://youtu.be/0w9fLX_T6tY

3.
QUESTIONS
Terry Taewoong Um (terry.t.um@gmail.com)
• How can we explain the predictions of a black-box model?
• Why did the system make this prediction?
• How can we explain where the model came from?
• What would happen if the values of a training point where
slightly changed?

4.
INTERPRETATION OF DL RESULTS
Terry Taewoong Um (terry.t.um@gmail.com)
• Retrieving images that maximally activate a neuron [Girshick et al. 2014]
• Finding the most influential part from the image [Zhou et al. 2016]
• Learning a simpler model around a test point [Ribeiro et al. 2016]
But, they assumed a
fixed model
 My NN is a function
of training inputs

5.
INFLUENCE OF A TRAINING POINT
Terry Taewoong Um (terry.t.um@gmail.com)
• What is the influence of a training example for
the model (or for the loss of a test example)?
Optimal model param. :
Model param. by training w/o z :
Model param. by upweighting z :
without z == (𝜖 = −
1
𝑛
)
• The influence of upweighting z on the parameters 𝜃

6.
INFLUENCE OF A TRAINING POINT
• Influence vs. Euclidean distance

7.
INFLUENCE OF A TRAINING POINT
Terry Taewoong Um (terry.t.um@gmail.com)
• The influence of upweighting z on the parameters 𝜃
• The influence of upweighting z on the loss at a test point

8.
PERTURBING A TRAINING POINT
Terry Taewoong Um (terry.t.um@gmail.com)
• Move 𝜖 mass from 𝑧 to 𝑧 𝛿
• If x is continuous and 𝛿 is small
• The effect of 𝑧  𝑧 𝛿 on the loss at a test point

9.
SUMMARY
Terry Taewoong Um (terry.t.um@gmail.com)
• The influence of 𝑧  𝑧 𝛿 on the loss at a test point
• The influence of upweighting z on the parameters 𝜃
• The influence of upweighting z on the loss at a test point

10.
EXAMPLE
Terry Taewoong Um (terry.t.um@gmail.com)
• The influence of upweighting z
• In logistic regression,
• Test : 7, Train : 7 (green), 1 (red)

11.
SEVERAL PROBLEMS
Terry Taewoong Um (terry.t.um@gmail.com)
• Calculation of
 Use Hessian-vector products (HVPs)

precompute 𝑠𝑡𝑒𝑠𝑡 by optimizing
or sampling-based approximation

12.
SEVERAL PROBLEMS
Terry Taewoong Um (terry.t.um@gmail.com)
• What if is non-convex, so H < 0
 Assuming that is a local minimum point, define a quadratic loss
Then calculate using the above
 empirically working!
• Influence function vs. retraining

13.
SEVERAL PROBLEMS
Terry Taewoong Um (terry.t.um@gmail.com)
• What if is non-differentiable?
e.g.) hinge loss
 Use a differentiable variation of the hinge loss

14.
APPLICATIONS
Terry Taewoong Um (terry.t.um@gmail.com)
• Understanding model behavior

15.
APPLICATIONS
Terry Taewoong Um (terry.t.um@gmail.com)
• Adversarial examples
c.f.) The effect of 𝑧  𝑧 𝛿 on the loss at a test point

16.
APPLICATIONS
Terry Taewoong Um (terry.t.um@gmail.com)
• Debugging domain mismatch

17.
APPLICATIONS
Terry Taewoong Um (terry.t.um@gmail.com)
• Fixing mislabeled examples