Group Reading for "A flexible model for training action localization with varying levels of supervision"

1. A flexible model
for training action localization
with varying levels of supervision
IVUL Group Reading– December 4, 2018
Presented by Frost

2. Content
• Review
• Spatio-temporal action detection
• Weakly supervised Spatio-temporal action detection
• Discriminative clustering
• Frank–Wolfe algorithm
• Method
• Formulation
• Constraints
• Setup
• Experiments
• Conclusion

3. Content
• Review
• Spatio-temporal action detection
• Weakly supervised Spatio-temporal action detection
• Discriminative clustering
• Frank–Wolfe algorithm
• Method
• Formulation
• Constraints
• Setup
• Experiments
• Conclusion

4. Review: Spatio-temporal action detection
What are the performed actions?
When do they happen?
Where do they take place?

5. Review: Spatio-temporal action detection
Kalogeiton, V., Weinzaepfel, P., Ferrari, V., & Schmid, C. (n.d.). Action Tubelet Detector for Spatio-Temporal Action Localization.

6. Review: Spatio-temporal action detection
Kalogeiton, V., Weinzaepfel, P., Ferrari, V., & Schmid, C. (n.d.). Action Tubelet Detector for Spatio-Temporal Action Localization.

7. Review: Spatio-temporal action detection
Kalogeiton, V., Weinzaepfel, P., Ferrari, V., & Schmid, C. (n.d.). Action Tubelet Detector for Spatio-Temporal Action Localization.

8. Content
• Review
• Spatio-temporal action detection
• Weakly supervised Spatio-temporal action detection
• Discriminative clustering
• Frank–Wolfe algorithm
• Method
• Formulation
• Constraints
• Setup
• Experiments
• Conclusion

9. Review: Actor-Supervision for Spatiotemporal
Action Localization
Escorcia, V., Dao, C. D., Jain, M., Ghanem, B., & Snoek, C. (n.d.). Guess Where? Actor-Supervision for Spatiotemporal
Action Localization. Retrieved from https://arxiv.org/pdf/1804.01824.pdf

10. Review: Actor-Supervision for Spatiotemporal
Action Localization
Escorcia, V., Dao, C. D., Jain, M., Ghanem, B., & Snoek, C. (n.d.). Guess Where? Actor-Supervision for Spatiotemporal
Action Localization. Retrieved from https://arxiv.org/pdf/1804.01824.pdf

11. Content
• Review
• Spatio-temporal action detection
• Weakly supervised Spatio-temporal action detection
• Discriminative clustering
• Frank–Wolfe algorithm
• Method
• Formulation
• Constraints
• Setup
• Experiments
• Conclusion

12. Review: Discriminative clustering
Bach, F. R., Za¨ıd, Z., & Ltci, H. (n.d.). DIFFRAC : a discriminative and flexible framework for clustering. Retrieved from
http://papers.nips.cc/paper/3269-diffrac-a-discriminative-and-flexible-framework-for-clustering.pdf

13. Review: Discriminative clustering
Bach, F. R., Za¨ıd, Z., & Ltci, H. (n.d.). DIFFRAC : a discriminative and flexible framework for clustering. Retrieved from
http://papers.nips.cc/paper/3269-diffrac-a-discriminative-and-flexible-framework-for-clustering.pdf
a multi-label classification problem with square loss functions

14. Content
• Review
• Spatio-temporal action detection
• Weakly supervised Spatio-temporal action detection
• Discriminative clustering
• Frank–Wolfe algorithm
• Method
• Formulation
• Constraints
• Setup
• Experiments
• Conclusion

15. Review: Frank–Wolfe algorithm
Bojanowski, P., Lajugie, R., Bach, F., Laptev, I., Ponce, J., Schmid, C., & Sivic, J. (n.d.). Weakly Supervised Action Labeling
in Videos Under Ordering Constraints. Retrieved from https://arxiv.org/pdf/1407.1208.pdf
Also known as the conditional gradient method,
reduced gradient algorithm and the convex combination
algorithm.
Link: https://www.wikiwand.com/en/Frank%E2%80%93Wolfe_algorithm

16. Review: Frank–Wolfe algorithm
Bojanowski, P., Lajugie, R., Bach, F., Laptev, I., Ponce, J., Schmid, C., & Sivic, J. (n.d.). Weakly Supervised Action Labeling
in Videos Under Ordering Constraints. Retrieved from https://arxiv.org/pdf/1407.1208.pdf
Also known as the conditional gradient method,
reduced gradient algorithm and the convex combination
algorithm.
Link: https://www.wikiwand.com/en/Frank%E2%80%93Wolfe_algorithm

17. Review: Frank–Wolfe algorithm
Bojanowski, P., Lajugie, R., Bach, F., Laptev, I., Ponce, J., Schmid, C., & Sivic, J. (n.d.). Weakly Supervised Action Labeling
in Videos Under Ordering Constraints. Retrieved from https://arxiv.org/pdf/1407.1208.pdf
B is known

18. Review: Frank–Wolfe algorithm
Bojanowski, P., Lajugie, R., Bach, F., Laptev, I., Ponce, J., Schmid, C., & Sivic, J. (n.d.). Weakly Supervised Action Labeling
in Videos Under Ordering Constraints. Retrieved from https://arxiv.org/pdf/1407.1208.pdf
B is known

19. Content
• Review
• Spatio-temporal action detection
• Weakly supervised Spatio-temporal action detection
• Discriminative clustering
• Frank–Wolfe algorithm
• Method
• Formulation
• Constraints
• Setup
• Experiments
• Conclusion

20. Method: Formulation

21. Method: Formulation

22. Content
• Review
• Spatio-temporal action detection
• Weakly supervised Spatio-temporal action detection
• Discriminative clustering
• Frank–Wolfe algorithm
• Method
• Formulation
• Constraints
• Setup
• Experiments
• Conclusion

23. Method: Constraints
Go to 4.3 Supervision as constraints
Chéron, G., Alayrac, J.-B., Laptev, I., & Schmid, C. (n.d.). A flexible model for training action localization with varying levels of
supervision. Retrieved from https://arxiv.org/pdf/1806.11328.pdf

24. Method: WSL + Constraints

25. Method: Constraints
Go to 4.3 Supervision as constraints
Chéron, G., Alayrac, J.-B., Laptev, I., & Schmid, C. (n.d.). A flexible model for training action localization with varying levels of
supervision. Retrieved from https://arxiv.org/pdf/1806.11328.pdf

26. Content
• Review
• Spatio-temporal action detection
• Weakly supervised Spatio-temporal action detection
• Discriminative clustering
• Frank–Wolfe algorithm
• Method
• Formulation
• Constraints
• Setup
• Experiments
• Conclusion

27. Method: Setup
Chéron, G., Alayrac, J.-B., Laptev, I., & Schmid, C. (n.d.). A flexible model for training action localization with varying levels of
supervision. Retrieved from https://arxiv.org/pdf/1806.11328.pdf
Detector: FasterRCNN
pretrained on COCO
Tracker: KLT
or other online linker
Video Tracklets
8 frames
I3D
Pretrained on Kinetics
Classifier
Formulation

28. Method: Setup
Chéron, G., Alayrac, J.-B., Laptev, I., & Schmid, C. (n.d.). A flexible model for training action localization with varying levels of
supervision. Retrieved from https://arxiv.org/pdf/1806.11328.pdf
Detector: FasterRCNN
pretrained on COCO
Tracker: KLT
or other online linker
Video Tracklets
8 frames
I3D
Pretrained on Kinetics
Classifier
Formulation
Fixed

29. Content
• Review
• Spatio-temporal action detection
• Weakly supervised Spatio-temporal action detection
• Discriminative clustering
• Frank–Wolfe algorithm
• Method
• Formulation
• Constraints
• Setup
• Experiments
• Conclusion

30. Experiments
parenthesis: fine-tuning on spatial annotations
the bounding boxes have been annotated
with a large margin around the person

31. Experiments

32. Content
• Review
• Spatio-temporal action detection
• Weakly supervised Spatio-temporal action detection
• Discriminative clustering
• Frank–Wolfe algorithm
• Method
• Formulation
• Constraints
• Setup
• Experiments
• Conclusion

33. Conclusion
• A flexible model with ability to adopt and combine various types of
supervision for action localization
• An experimental study demonstrating the strengths and weaknesses
of a wide range of supervisory signals and their combinations

34. Thank you