Prompt Switch: Efficient CLIP Adaptation for Text-Video Retrieval Chaorui Deng, Qi Chen, Pengda Qin, Da Chen, Qi Wu; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 15648-15658 https://openaccess.thecvf.com/content/ICCV2023/html/Deng_Prompt_Switch_Efficient_CLIP_Adaptation_for_Text-Video_Retrieval_ICCV_2023_paper.html

1. Prompt Switch: Efficient CLIP
Adaptation for Text-Video Retrieval
Chaorui Deng,Qi Chen, Pengda Qin, Da Chen,Qi Wu, ICCV2023
2023/10/30

2. ◼ Prompt
•
• Prompt Head
• Prompt
• Prompt
◼ Text-image CLIP Prompt
• Video Token Prompt Token
•
[Jia+, ECCV2022]

3. ◼ CLIP [Radford+, ICML2021]

4. ◼ Text-image CLIP Prompt
1. Prompt Token
• Video Token token
2. Prompt Switch
• Video Encoder prompt token
3. Prompt Aggregation
• Class token prompt token
4. Captioning loss
◼
• Video up
•

6. 1. Prompt Token

7. 2. Prompt Switch
𝑉 : Visual Token
𝑃 : Prompt Token

8. 3. Prompt Aggregation

9. Prompt Aggregation
◼ Prompt Aggregation
•
• 𝑐𝑖 : Visual class token 𝑖 ∈ 1, 𝑁𝑓 , 𝑁𝑓:
• ෠
𝑃 : Prompt Token
• 𝑀𝐻𝐴: Multi-Head Attention
• 𝐿𝑁: Layer Normalization
• Multi-Head Attention + Add LayerNorm
• Query: 𝑐𝑖, key, value : ෠
𝑃
• Mean-Pooling

10. 4. Captioning Loss
Captioning output

11. Loss
◼ Loss
• Contrastive Loss : 𝐿𝑐𝑜𝑛
• Captioning Loss : 𝐿𝑐𝑎𝑝
◼
• 𝑥𝑖 : video
• 𝑦𝑖 : text
• 𝐵 :
• 𝑁𝑓 :
• ഥ
𝑝𝑖 : prompt Aggregation
• 𝜏 :
• 𝑤<𝑙 : 𝑙 caption
• 𝑤𝑙 : 𝑙 caption
• 𝑁𝑤 : token

12. ◼
• MSR-VTT [Xu+, CVPR2016]
• Video: 10k, caption: 200k
• Train: 9k, Val: 1k
• MSVD [Chen&Dolan, ACL2011]
• Video: 1970, caption: 120k
• Train: 1200, Val: 100, test: 670
• LSMDC [Rohrbach+, arXiv2015]
• Video: 118081, caption: 11808
• Train: 109673, Val: 7408, test: 1k
◼ Video
• 6
• 224×224
◼ : CLIP
•
• Text encoder, video encoder
• CLIP (ViT-B/32)
• Prompt Token
• ( 0, 0.02)
◼
• Batch size : 128
• Optimizer : AdamW
• Learning rate : 3𝑒−5
• Scheduler : CosinAnnealing

13. 1: Ablation
◼ ablation
1. Baseline CLIP
2. 1 + Prompt Switch
3. 2 + Prompt Aggregation
4. 3 + Captioning Loss
◼
• MSR-VTT 1K-A
◼
•

14. 2
◼ temporal modeling
• Temporal Transformer
• Attention
• Token Shift [Liu+, ECCV2022]
• Video Proxy [Xue+, ICLR2023]
• Full Attention
• video Attention
• ours (Prompt Switch + Aggregation)
◼
•
◼
• MSR-VTT 1K-A
◼
• Recall (𝑘 = 1, 5, 10)

15. 3: SOTA
◼ MSRVTT

16. 3: SOTA
◼ MSVD

17. ◼ Text-image CLIP Prompt
• Prompt Token
• Prompt Switch
• Prompt Aggregation
• Captioning loss
◼ SOTA
•
• MSR-VTT
• MSVD