Transfer learning is specifically very helpful when there is a scarcity of data, limited bandwidth that might not allow training deep models from scratch, and so on. In the world of computer vision, ImageNet pre-training has been widely successful across a number of different tasks, image classification being the most popular one. All of that success has been possible mainly because of the ImageNet dataset which is a collection of images spanning across 1000 labels. This is where a stern limitation comes in - the need for having labeled data. In this session, we want to take a deep dive into the world of self-supervised learning which allows models to exploit the implicit labels of input data. In the first half of the session, we will be covering the basics of transfer learning, its successes, and its challenges. We will then start by formulating the problem that self-supervised learning tries to address. In the second half of the session, we will be discussing the ABCs of self-supervised learning along with some examples. We will conclude by a shortcode walk-through and a discussion on the challenges of self-supervised learning.

1. Redefining transfer learning
for computer vision
Sayak Paul
PyImageSearch
@RisingSayak
Doing it better with self-supervised learning
Souradip Chakraborty
Data Scientist @ WalmartLabs

2. Agenda
● The ImageNet pre-training era in Computer Vision
○ Success
○ Challenges
● Introduction to self-supervised learning
● Pre-text tasks
● Downstream tasks
● Examples
● Code demo with image inpainting
● Challenges and directions

3. The ImageNet pre-training era
Transfer learning
traditionally

4. ImageNet pre-training era:
Success● Less data
● Architectural decision making
● Faster prototyping

5. ImageNet pre-training era:
Success● Some architectures widely used for the purpose:
○ ResNet and its different variants
○ EfficientNet
○ MobileNet
○ ...

6. Challenges
Transfer learning
traditionally

7. ImageNet pre-training era:
Challenges● Knowledge of the pre-trained models not applicable to the target data
domain
● Source data distribution of the pre-trained models differ from the target
data distribution
● Shortage of labeled data

8. Can we leverage the power of
inherent patterns of
(unlabeled) data?

9. Self-supervised learning 101
● Pre-text tasks
● Downstream tasks
● Examples (one from both of the following domains)
○ Computer Vision
○ Natural Language Processing

10. Remarkable results

11. Remarkable results

12. The idea of training models with
masked inputs and having the
models learn to unmask them
(courtesy of Yann LeCun*)

13. The idea of training models with
masked inputs and having the
models learn to unmask them
(courtesy of Yann LeCun*)

14. Using image inpainting as a pre-
text task

15. Image inpainting as a pre-text task
● Image inpainting basics
● Our approach
○ Partial Convs
○ U-Net based architectures for information propagation

16. Image inpainting data preparation
Base dataset: CIFAR10

17. Image inpainting model schematics
● Partial convs
● U-Net like architecture
● Loss function
● Performance metric

18. Image inpainting results

19. Now, can we leverage the
knowledge of image inpainting
for other tasks like image
classification?

20. Using an image inpainting model for
other tasks
● Training an image classification model just like we train a text-
classification model with pre-trained embeddings

21. Image classification model schematics
WIP

22. Image classification results
WIP

23. Self-supervised learning
challenges

24. How to choose a pre-text task?
● Pre-text invariant representation
● Jeremy Howard’s suggestions via his blog - Self-supervised learning and
computer vision

25. Measuring performance
● How to evaluate a self-supervised learning system?
● Noise contrastive estimation/consistency loss

26. Self-supervised learning: “the
next step in AI” - Yann LeCun

27. Recent approaches
● FixMatch
● SimCLR