In this presentation we will learn about what is self-supervision and how thinking in innovative yet very simple ways can reduce the dependency of ML models on labelled data.

1. Silversparro Technologies Pvt. Ltd.
The Promise of Self-
supervised Learning
Research Head
Ravikant Bhargava

2. Silversparro Technologies Pvt. Ltd.
The Promise of Self-Supervised
Learning
Research Head
Ravikant Bhargava

3. Silversparro Technologies Pvt. Ltd.
"A computer program is said to learn from
experience E with respect to some class
of tasks T and performance measure P if
its performance at tasks in T, as measured
by P, improves with experience E."
What is learning?

4. Silversparro Technologies Pvt. Ltd.
Supervised learning is the machine
learning task of learning a function that
maps an input to an output based on
example input-output pairs.
What is Supervised Learning?

5. Silversparro Technologies Pvt. Ltd.
What is Self-supervised Learning?
● A form of unsupervised learning where the data provides the
supervision
● In general, withhold some part of the data, and task the network with
predicting it
● The task defines a proxy loss, and the network is forced to learn what
we really care about, e.g. a semantic representation, in order to solve it

6. Silversparro Technologies Pvt. Ltd.

7. Silversparro Technologies Pvt. Ltd.
Why Self-supervision?
● Expense of producing a new dataset for each new task
● Some areas are supervision-starved, e.g. medical data, where it is hard to
obtain annotation
● Untapped/availability of vast numbers of unlabelled images/videos
● Facebook: one billion images uploaded per day
● 300 hours of video are uploaded to YouTube every minute

8. Silversparro Technologies Pvt. Ltd.
How do we do it?
● From images
● From videos
● From videos with audios

9. Silversparro Technologies Pvt. Ltd.
From Images
Context Prediction

10. Silversparro Technologies Pvt. Ltd.

11. Silversparro Technologies Pvt. Ltd.

12. Silversparro Technologies Pvt. Ltd.

13. Silversparro Technologies Pvt. Ltd.

14. Silversparro Technologies Pvt. Ltd.
From videos
Shuffle & Learn

15. Silversparro Technologies Pvt. Ltd.

16. Silversparro Technologies Pvt. Ltd.
From videos
Colorize a video

17. Silversparro Technologies Pvt. Ltd.
From videos
Arrow of Time

18. Silversparro Technologies Pvt. Ltd.

19. Silversparro Technologies Pvt. Ltd.
From videos
Temporal Cycle Consistency

20. Silversparro Technologies Pvt. Ltd.
From videos
Temporal Cycle Consistency, continued...

21. Silversparro Technologies Pvt. Ltd.
● Important to select informative data in training
○ Hard negatives and positives
○ Otherwise, most data is too easy or has no information and the network will not learn
○ Often use heuristics for this, e.g. motion energy
● Consider how the network can possibly solve the task (without cheating)
○ This determines what it must learn, e.g. human keypoints in `shuffle and learn’
● Choose the proxy task to encourage learning the features of interest
Things to keep in mind

22. Silversparro Technologies Pvt. Ltd.
1. Mitchell, T. (1997). Machine Learning. McGraw Hill. p. 2. ISBN 978-0-07-042807-2
2. Stuart J. Russell, Peter Norvig (2010) Artificial Intelligence: A Modern Approach, Third Edition, Prentice Hall ISBN
9780136042594
3. https://project.inria.fr/paiss/files/2018/07/zisserman-self-supervised.pdf
4. https://arxiv.org/abs/1603.08561
5. https://arxiv.org/abs/1904.07846
6. https://arxiv.org/abs/1505.05192
7. https://arxiv.org/abs/1603.08511
8. http://www.robots.ox.ac.uk/~vgg/publications/2018/Wei18/wei18.pdf
9. https://www.facebook.com/722677142/posts/10155934004262143/
10. https://arxiv.org/abs/1406.6909
11. https://arxiv.org/abs/1708.07860
12. https://arxiv.org/abs/1806.09594
References