AJ Piergiovanni, Weicheng Kuo, Anelia Angelova, "Rethinking Video ViTs: Sparse Video Tubes for Joint Image and Video Learning" arXiv2022 https://arxiv.org/abs/2212.03229

1. Rethinking Video ViTs:
Sparse Video Tubes
for Joint Image and Video Learning
AJ Piergiovanni, Weicheng Kuo, Anelia Angelova
arXiv2022
2023/6/8

2. ◼Vision Transformer (ViT) [Dosovitskiy+, ICLR 2021]
•
•
ViT
Conv2D
Conv3D
...
ViViT [Arnab+, ICCV2021]

3. ◼ViT TubeViT
• 2D
• 2D 3D
◼3D
• 3D

4. ◼ViViT [Arnab+, ICCV2021]
• ViT
• 3D
•
◼
•
• 3D 2D
◼
•
... ...

5. ◼ViT TubeViT
1.
• 2D
• 3D
•
•
2. 3D
•
3.
• Video
•
•

6. 1.
◼2D
• (1, 𝐻, 𝑊) (𝑇𝑠, 𝐻, 𝑊) 2D Conv
• 𝑇𝑠
◼3D
• (𝑇, 𝐻, 𝑊) (𝑇𝑠, 𝐻𝑠, 𝑊
𝑠) (𝑡, 𝑥, 𝑦) 3D Conv
•
• 3D
◼
•
•
•

7. 2.
◼
•
•
◼
•
◼
• (𝑡, 𝑥, 𝑦)
• sine, cosine
• 1 6
• 2D 3D
◼
•

8. 2.
(𝑡, 𝑥, 𝑦)
Conv 3D
𝑧𝑖 ∈ 𝑅𝑑
1 2 3 d
4 5 6 …
7 8 9 10 11 12
+
𝑗 = 1 …
𝑗 = 2
cos(𝑦 ∗ 𝑤1)
sin(𝑦 ∗ 𝑤1)
cos(𝑥 ∗ 𝑤1)
sin(𝑥 ∗ 𝑤1)
cos(𝑡 ∗ 𝑤1)
sin(𝑡 ∗ 𝑤1)
• 𝜏 = 10000
• 𝑗 = 1 ~ 𝑑/6
• (𝑡, 𝑥, 𝑦)
𝑤𝑗 = 1/𝜏𝑗
𝑝𝑖,𝑡 = sin 𝑡 ∗ 𝑤𝑗 , cos(𝑡 ∗ 𝑤𝑗)
𝑝𝑖,𝑥 = sin 𝑥 ∗ 𝑤𝑗 , cos(𝑥 ∗ 𝑤𝑗)
𝑝𝑖,𝑦 = sin 𝑦 ∗ 𝑤𝑗 , cos(𝑦 ∗ 𝑤𝑗)
𝑥
𝑦 𝑡

9. 3.

10. 3.

11. 3.

12. SOTA
◼
•
•
•
• ViT-B, L, H
◼
•
• ImageNet-1k [Deng+, CVPR2009]
•
• Kinetics400, 600, 700
(K400,600,700) [Kay+, arXiv2017,
Carreira+, arXiv2018, Carreira+, arXiv2019]
• Something Something V2 (SSv2)
[Goyal+, ICCV2017]
◼
• 256
•
• K400, 600, 700 64
• SSv2 32
• 300000
• Adam
•
• K400, 600, 700 5e-5
• SSv2 2e-5
◼
• val top-1, top-5 accuracy

13. ◼Video Tube
•
◼ViT-B, L, H
Tube size stride offset
1 8 8 8 16 32 32 (0, 0, 0)
2 16 4 4 6 32 32 (4, 8, 8)
3 4 12 12 16 32 32 (0, 16, 16)
4 1 16 16 32 16 16 (0, 0, 0)

14. ◼ImageNet-1k Kinetics400
• SOTA

15. K600 K700
SSv2

16. ◼
• Kinetics600
• ImageNet-1k Kinetics600
• ImageNet-1k Kinetics600
• 2D ImageNet-1k
Kinetics600
• ImageNet-1k 3D
Kinetics600
◼
• ViT-L
◼
•
• head
• head+ 4
• head+ 8
◼
• ViT-H
•
• ImageNet pretrained weight
◼
• Kinetics600

17. ◼
• ImageNet-1k Kinetics600
◼
•
• Head+ 8
48%

18. ◼ViT TubeViT
• 3D
•
•
◼
• Kinetics400, 600, 700, SSv2 SOTA
• Token