The document discusses unsupervised learning, particularly in the context of video representation and temporal order verification. It highlights the significance of unsupervised learning frameworks like autoencoders and GANs, and the challenges involved in effectively sampling frame tuples for training. The conclusions emphasize the potential of temporal verification networks to understand video sequences better, while suggesting the need for further exploration in capturing longer temporal logics.

1. Shuffle and Learn: Unsupervised Learning
using Temporal Order Verification
Slides by Xunyu Lin
ReadCV, UPC
20th February, 2017
Ishan Misra, C. Lawrence Zitnick, Martial Hebert
[arxiv] (26 July 2016) [code] [demo]

2. Index
1. Introduction
2. Unsupervised Representations Learning
3. Video Representations Learning
4. Temporal Order Verification
5. In Practice
6. Evaluations
7. Conclusions

3. Index
1. Introduction
2. Unsupervised Representations Learning
3. Video Representations Learning
4. Temporal Order Verification
5. In Practice
6. Evaluations
7. Conclusions

4. Introduction
What is Unsupervised Learning?
● Unsupervised Learning is the machine learning task of inferring a function
to describe hidden structure from unlabeled data.
● The key of Unsupervised Learning is how to do clustering:

5. Introduction
Why Unsupervised Learning?
“Most of human and animal learning is unsupervised learning. If
intelligence was a cake, unsupervised learning would be the cake,
supervised learning would be the icing on the cake, and
reinforcement learning would be the cherry on the cake. We know
how to make the icing and the cherry, but we don’t know how to make
the cake.”
—— Yann LeCun

6. Introduction
Why Unsupervised Learning?
● It is the nature of how intelligent beings percept
the world.
● It can save us tons of efforts to build a
human-alike intelligent agent compared to a
totally supervised fashion.
● It’ll be the new breakthroughs to get true AI!

7. Index
1. Introduction
2. Unsupervised Representations Learning
3. Video Representations Learning
4. Temporal Order Verification
5. In Practice
6. Evaluations
7. Conclusions

8. Unsupervised Representations Learning
Popular Unsupervised Representations Learning frameworks
Auto-Encoder

9. Unsupervised Representations Learning
Popular Unsupervised Representations Learning frameworks
Variational
Auto-Encoder
(VAE)
Tutorial

10. Unsupervised Representations Learning
Popular Unsupervised Representations Learning frameworks
GAN

11. Index
1. Introduction
2. Unsupervised Representations Learning
3. Video Representations Learning
4. Temporal Order Verification
5. In Practice
6. Evaluations
7. Conclusions

12. Video Representations Learning
● Human percept the world through observing the dynamic changing of our
daily lives, which can be regarded as videos.
● Thus the unsupervised video representations learning plays an
unneglectable role in building a human-alike intelligent agent.

13. Video Representations Learning
Related Works
Video Prediction with LSTMs

14. Video Representations Learning
Related Works
Spatiotemporally Coherent Reconstruction

15. Index
1. Introduction
2. Unsupervised Representations Learning
3. Video Representations Learning
4. Temporal Order Verification
5. In Practice
6. Evaluations
7. Conclusions

16. Temporal Order Verification
The internal temporal order of videos

17. Temporal Order Verification

18. Temporal Order Verification
Take temporal order as the supervisory signals for learning
Shuffled
sequences
Binary classification
In order
Not in order

19. Index
1. Introduction
2. Unsupervised Representations Learning
3. Video Representations Learning
4. Temporal Order Verification
5. In Practice
6. Evaluations
7. Conclusions

20. In Practice
How to sample the tuple of frames?
1. The number of frames for each tuple
- 2 frames: may be ambiguous (picking up or placing down a cup?)
- 3 frames: practically useful, but still not enough for a cyclical case
- ...

21. In Practice
How to sample the tuple of frames?
a b c d e
b c d
ab d
eb d
Positive
Negative
Original
Video

22. In Practice
How to sample the tuple of frames?
2. Ambiguity in frames with small motion
- The order of a small motion is indistinguishable.
- Only sample from frames with high motion (smart sampling).
- Use coarse frame level optical flow as a proxy to measure the motion
between frames.

23. In Practice
How to sample the tuple of frames?
3. The distance of frames in positive tuples (difficulty of the task)
- Too close: results in ambiguous small motion or overly easy task
- Too far: consecutive frames are not highly related which makes the
learning task too difficult.

24. In Practice
How to sample the tuple of frames?
3. The distance of frames in positive tuples (difficulty of the task)
- Two metrics which control the difficulty of positive and negative samples.
b c d
ab d
eb d

25. In Practice
How to sample the tuple of frames?
4. Ratio of negative and positive samples

26. Index
1. Introduction
2. Unsupervised Representations Learning
3. Video Representations Learning
4. Temporal Order Verification
5. In Practice
6. Evaluations
7. Conclusions

27. Evaluation
Action Recognition on UCF-101 & HMDB-51
- Comparison to random initialization & transfer learning
- Pre-trained on ImageNet and finetuned on UCF-101 gives an accuracy of 67.1%.
- Pre-trained on ImageNet and finetuned on HMDB-51 gives an accuracy of 28.5%.
+ 11.6 %
+ 4.8 %
* UCF-101 is two times larger than HMDB-51

28. Evaluation
Action Recognition on UCF-101 & HMDB-51
- Comparison to other unsupervised frameworks
- Two Close: measure if two frames are close or not.
- Two Order: temporal verification with only 2 frames.
- DrLim: measure temporal coherency with L2 loss.
- TempCoh: measure temporal coherency with L1 loss.
- Obj. Patch: basically imitates human’s instinct eyes fixation ability. Paper link

29. Evaluation
Nearest Neighbor retrieval

30. Evaluation
Visualizing pool5 Unit Responses

31. Evaluation
Pose Estimation on FLIC & MPII

32. Index
1. Introduction
2. Unsupervised Representations Learning
3. Video Representations Learning
4. Temporal Order Verification
5. In Practice
6. Evaluations
7. Conclusions

33. Conclusions
● Temporal verification exploits the potential of a network to capture the
sequential logics in videos.
● Further works should be explored by capturing a longer temporal logics.
For now it only utilizes single frames in less than around 60 frames.
Architectures like RNN could be further utilized to extend the temporal
range.
● The only drawbacks lie in its weak constraint and tedious sampling
techniques.
● More general constraint with simplified procedure? → My research line