Shreyank N Gowda, Marcus Rohrbach, Frank Keller, Laura Sevilla-Lara, "Learn2Augment: Learning to Composite Videos for Data Augmentation in Action Recognition" ECCV2022 https://www.ecva.net/papers/eccv_2022/papers_ECCV/papers/136910234.pdf https://arxiv.org/abs/2206.04790

1. Learn2Augment:
Learning to Composite Videos
for Data Augmentation
in Action Recognition
Shreyank N Gowda, Marcus Rohrbach, Frank Keller, and Laura
Sevilla-Lara , ECCV2022
2022/11/25

2. nLearn2Augment
•
•

3. nData Augmentation
• ActorCut [Zou+, arXiv2021] VideoMix [Yun+, arXiv2020]
•
• [Zhang+, arXiv2019]
• GAN
• self-paced selection
nSemi-supervised Video Action Recognition
•
• VideoSSL [Jing+, WACV2021]
•
• Temporal Contrastive Learning (TCL) [Singh+, CVPR2021]
• 2

4. nSample selection
• [Huang+, CVPR2018]
•
• SMART [Gowda+, arXiv2020]
•
• SCSampler [Korbar+, ICCV2019]
•
•
• RL [Yoon+, PMLR2020]
•
•

5. Learn2Augment
1. Semantic Match
•
2. Selector
1. Selector 𝜔
2. Video Composite
1.
•
•
2.
3. Classifier & Selector Reword
3.

6. Semantic Matching
n[Choi+, NIPS2019]
•
•
nSen2vec [Pagliardini+, arXiv2018]
•
• 𝑐! 𝑐" 𝑉!, 𝑉"

7. Selector
nSelector Architecture
• 3D ResNet-18 [He+, arXiv2016] + MLP
•
n
• 3D ResNet-18 validation loss
n
•

8. Training Selector
nSelector
• RL
1.
• 𝐷#$%: validation set ℒ&%':
• 𝑓(: 𝑉): 𝑦):
• 𝛿:
• 𝑆:

9. Training Selector
2. REINFORCE [Williams+, Machine learning1992]
•
• 𝐷*:
• 𝐷+:

10. Video Compositing
1.
• MaskRCNN [He+, ICCV2017]
• MaskRCNN COCO [Lin+, ECCV2014]
•
2.
•
• [Liu+, ECCV2018]
3.

11. Training Classifier
n
• ,
𝑦
• 𝛾 = ∑
*!
,-.
𝛼 = 4
• 𝑦/ 𝑦0

12. n
•
•
•
• Few-shot
• n+k k
• Novel-class
• 1~5
• Seen-class
•
•
• Standard split [Zhang+, arXiv2020]
• Truze split [Gowda+, arXiv2021]
•
•
•
•
• Sports1M [Karpathy+,
CVPR2014]
n
• HMDB51 [Jhuang+, ICCV2011]
• UCF101 [Soomro+, arXiv, 2012]
• Kinetics-400 [Kay+, arXiv2017]
• Kinetics-100

13. 1
n
• 13.4%
•

14. 2
n
• Top-1 accuracy
• 5~50%
• L2A Pre-training
• Kinetics-400 Selector

15. 3
nFew-shot
• Top-1 accuracy
• S: Standard split, T: Truze split

16. 4
n
• L2A

17. nLearn2Augment
•
•
n
•
• 8.6%
• Few-shot
• 3.7%
•
• 17.4%