There is always some uncertainty in our predictions. No model is perfect! The following topics are covered- 1. How uncertainty in deep learning can be a threat to human life? 2. Types of Uncertainties 3. Bayesian Neural Networks 4. Challenges and Future Work

1. UNCERTAINTIES IN
DEEP LEARNING
There is always some uncertainty in
our predictions
No model is Perfect!

2. WHY DO WE
NEED TO CARE
ABOUT
UNCERTAINTY??

3. A THREAT TO
HUMAN LIFE
With the rise of computation
power and ability to collect more
data and store it, Deep Learning
has grown rapidly.
Currently, Deep learning models
are used in all the fields present
in the market in some or the
other form.
With their use in safety systems
and healthcare, their faults pose a
threat to human life.

4. In May 2016, a self-driving car
made by Tesla was not able
to recognize a white truck
against a brightly lit sky
injuring the person inside
which caused a huge safety
issue on the use of AI

5. In july 2015, Google Photos
erroneously identified two
African-Americans as Gorillas
raising racism issues

6. TYPES OF UNCERTAINTY
● EPISTEMIC UNCERTAINTY a.k.a MODEL UNCERTAINTY
● ALEATORIC UNCERTAINTY a.k.a DATA UNCERTAINTY
○ HOMOSCEDASTIC ALEATORIC UNCERTAINTY
○ HETEROSCEDASTIC ALEATORIC UNCERTAINTY

7. EPISTEMIC UNCERTAINTY
➔ Uncertainty due to limited knowledge and data
➔ Can be reduced by collecting more data intelligently

8. Intuition Behind Epistemic Uncertainty

9. LESS DATA vs LARGE DATA

10. ALEATORIC UNCERTAINTY
➔ Uncertainty representative of noise inherent in data
➔ Cannot be reduced by collecting more data
➔ Homoscedastic Model : The noise is constant for all data points, i.e,
independent of the input
➔ Heteroscedastic Model : The noise is different for all data points, i.e,
input-dependent

11. Heteroscedastic vs Homoscedastic

12. SUMMARY

13. BAYESIAN NEURAL NETWORKS

14. BAYES BY BACKPROP

15. VARIATIONAL INFERENCE
● It is an approximation method so that the result of the integration in
bayesian neural networks can be calculated
● Make T forward passes through the network for each data point
● Now use this set of predictions to estimate mean(final prediction) and
variance (which is the uncertainty measure)

16. DROPOUT AS BAYESIAN APPROXIMATION
● Keep dropout on at test time too
● Make T forward passes for each data
point
● Use this set of predictions to get the
actual final prediction and the uncertainty
measures

17. MODELLING ALEATORIC UNCERTAINTY
● For Homoscedastic, just add a noise term to the variance as a
hyperparameter and tune it according to the validation data
● For Heteroscedastic, add a standard deviation term alongside the output but
it will require some tweaking to the loss function
L(ŷ,y,𝛔) = ||ŷ-y||2
/(2*𝛔2
) + log(𝛔2
)/2

18. QUANTILES AND CONFIDENCE INTERVALS
❏ Quantiles are cuts which divide the distribution into two parts
❏ x% Confidence Interval mean an interval [a,b] st the value you are measuring
will lie in this interval x% of times

19. QUANTILE REGRESSION
● We develop a model to predict a certain quantile of the output we want and
then combine them to generate 95/99% confidence intervals
● Loss function becomes :
L(ŷ,y) = max( q(ŷ-y) , (q-1)(ŷ-y) )

20. SOME METRICS FOR CLASSIFICATION
● Variation Ratio = 1 - frequency of mode/T
● Entropy of mean of T vectors = - Σc
pc
log(pc
)
● Mutual Information = Entropy of mean of T vectors - Mean of
entropies of T vectors

21. SOME EXPERIMENTAL DATA

22. MY EXPERIMENT WITH MC DROPOUT
● Accuracy with Standard Dropout -
96.17%
● Accuracy with MC Dropout - 98.12%
● With uncertainty estimates, we were
able to reduce wrong predictions to
only 34 out of 10000
● Also, 98% CIFAR-10 dataset was
rejected by the model
Results on
MNIST
dataset
Certain Uncertain
Correct 9464 348
Incorrect 34 154

23. WHEN TO USE WHICH TYPE?
Aleatoric Uncertainty should be used
in the cases of :
● Large data situations
● Real-time applications
Epistemic Uncertainty should be used
in the cases of :
● Safety-critical applications
● Small Datasets

24. How can this
uncertainty
help us??
● Tells whether our model is
certain or not
● Helps in developing a strategy
for collection of more data
● Bayesian Active Learning

25. HUMAN INTERFERENCE
Model output
Certain Uncertain
Human CheckingConfident Prediction

26. EPISTEMIC UNCERTAINTY TO THE RESCUE
● Analyse all the data which results in high epistemic uncertainty
● Try to find patterns in this data

27. BAYESIAN ACTIVE LEARNING

28. CHALLENGES
AND
FUTURE WORK
● Real time epistemic uncertainty
estimates are yet to be
achieved
● There is still scope of
improvement in the uncertainty
estimates
● Not all the current models can
be successfully transformed
into Bayesian Neural Networks