論文紹介：Multitask Vision-Language Prompt Tuning
•
0 likes•13 views
Sheng Shen, Shijia Yang, Tianjun Zhang, Bohan Zhai, Joseph E. Gonzalez, Kurt Keutzer, Trevor Darrell, " Multitask Vision-Language Prompt Tuning " WACV2024 https://openaccess.thecvf.com/content/WACV2024/html/Shen_Multitask_Vision-Language_Prompt_Tuning_WACV_2024_paper.html
Recommended
Recommended
More Related Content
More from Toru Tamaki
More from Toru Tamaki (20)
Recently uploaded
Recently uploaded (20)
論文紹介：Multitask Vision-Language Prompt Tuning
- 1. Multitask Vision-Language Prompt Tuning Sheng Shen, Shijia Yang, Tianjun Zhang, Bohan Zhai, Joseph E. Gonzalez, Kurt Keutzer, Trevor Darrell WACV2024 2024/1/11
- 2. ◼ zeroshot, fewshot CLIP • Transformer image-text • ◼ Prompt • ◼ Prompt • •
- 3. Prompt ◼ CLIP Prompt • CoOp (textual prompt tunning) • VPT (visual prompt tuning) • UPT (unified prompt tuning) CoOp VPT UPT CLIP [Radford+,arXiv2021]
- 4. ◼ Prompt CLIP MVLPT 1. Multitask Prompt Initialization 2. Multitask Prompt Adaptation
- 5. ◼ Prompt CLIP 1. Multitask Prompt Initialization 2. Multitask Prompt Adaptation
- 6. ◼ Prompt CLIP 1. Multitask Prompt Initialization 2. Multitask Prompt Adaptation
- 7. 1. Multitask Prompt Initialization ◼ Prompt • Task A, B, C… • CLIP Prompt • Source Prompts ◼ • Prompt ◼ • Prompt Source prompts
- 8. 2. Multitask Prompt Adaptation ◼ Single-task Adaptation • Source Prompts • ◼ Multi-task Adaptation • Source Prompts • Source prompts Source prompts
- 9. Multi-task Adaptation ( ) ◼ 20 • 1 (Step1) • (Step2) • 20 20 1 2 20 2 1 20 … … 90 50 60 … 80 70 … … … …
- 10. ◼ 2
- 12. ◼ • ◼ • ELEVATER [Li+, arXiv2022] 20 • • Caltech101, OxfordPets, StanfordCars, Flowers102, Food101, FGVCAircraft, DTD ,EuroSAT • Hateful Memes, PatchCamelyon, Rendered-SST2, KITTI Distance, FER 2013, CIFAR-10/100, VOC 2007 Classification, Country-211, MNIST, GTSRB, Resisc45
- 13. ◼ • 200 • 32 • Optimizer Adam • Scheduker CosinAnearing • 2e-3 • WarmUp 1 ◼ image • 224 • 224 224 ◼ Text • Label ”a photo of [CLASS]”
- 14. ◼ Multitask Prompt Initialization • 2 • Single task prompt tuning (CoOp, VPT, UTP ) • Prompt Initialization + Single-task Prompt Adaptation(CoOp, VPT, UTP )
- 16. ◼ Step1, 2 Ablation • Step1 (Initilization) • “-” • “M” • Step2 (Adaptation) • “S” single task • “M” multi-task
- 17. ◼ Source, Adaptation ”M”
- 18. ◼ Prompt CLIP MVLPT • Multi-task Prompt Initialization • Multi-task Prompt Adaptation • ◼ Prompt Tuning • Zeroshot • Single Task Prompt Tuning
- 20. Abation ◼ • Single task • • Prompt • K-means