mPLUG is a new vision-language pre-trained model proposed by the authors that achieves state-of-the-art performance on various vision-language tasks through an asymmetric architecture using novel cross-modal skip connections. The model introduces skip-connected fusion blocks to address information asymmetry and computation inefficiency problems in multi-modal fusion. mPLUG is pre-trained using contrastive learning on image-text pairs and masked language modeling, and shows strong zero-shot transfer ability on tasks like image captioning and image-text retrieval. Evaluation shows mPLUG outperforms prior work on tasks including visual question answering, image captioning, image-text retrieval, visual grounding and visual reasoning.

1. mPLUG: Effective and Efficient Vision-Language Learning by
Cross-modal Skip-connections
2023.04.09
이미지 처리팀
이찬혁, 김준철, 류채은, 이해원, 최승준, 현청천

2. Introduction

3. 03
Introduction Vision-Language Task

4. 04
Introduction Problem of previous works

5. 05
Introduction Problem of previous works
•
•

6. 06
Introduction Proposed method
•
•
•

7. 07
Introduction Contribution
• We propose a unified vision-language pretrained model mPLUG of cross-modal understanding and
generation for both effectiveness and efficiency in cross-modal learning.
• We introduce a new asymmetric vision language architecture with novel cross-modal skip-connections,
to address two fundamental problems of information asymmetry and computation inefficiency in multi-
modal fusion.
• mPLUG achieves state-of-the-art performance on a wide range of vision-language tasks, including
Image captioning, image-text retrieval, visual grounding and visual question answering in zero-shot
manner

8. Proposed method

9. 09
Proposed method Overall architecture
skip-connected fusion block
skip-connected fusion block

10. 010
Proposed method Model architecture
𝐿𝐼𝑇𝐶 + 𝐿𝐼𝑇𝑀 + 𝐿𝑀𝑎𝑠𝑘𝐿𝑀 𝐿𝑃𝑟𝑒𝑓𝑖𝑥𝐿𝑀

11. 011
Proposed method Model architecture
𝑳𝑰𝑻𝑪 + 𝐿𝐼𝑇𝑀 + 𝐿𝑀𝑎𝑠𝑘𝐿𝑀 𝐿𝑃𝑟𝑒𝑓𝑖𝑥𝐿𝑀

12. 012
Proposed method Pre-training methods (Image-Text Contrastive learning)
•
𝐼1, 𝐼2, 𝐼3, 𝐼4
𝑇1, 𝑇2, 𝑇3, 𝑇4
𝐼1, 𝐼2, 𝐼3, 𝐼4
𝑇1, 𝑇2, 𝑇3, 𝑇4
𝐼1
𝑇2, 𝑇3, 𝑇4
𝐼3
𝑇1, 𝑇2, 𝑇4

13. 013
Proposed method Pre-training methods (Image-Text Contrastive learning)
𝑻𝟏 𝑻𝟐 𝑻𝟑 𝑻𝟒
𝑰𝟏 𝑺𝟏𝟏 𝑺𝟏𝟐 𝑺𝟏𝟑 𝑺𝟏𝟒
𝑰𝟐 𝑺𝟐𝟏 𝑺𝟐𝟐 𝑺𝟐𝟑 𝑺𝟐𝟒
𝑰𝟑 𝑺𝟑𝟏 𝑺𝟑𝟐 𝑺𝟑𝟑 𝑺𝟑𝟒
𝑰𝟒 𝑺𝟒𝟏 𝑺𝟒𝟐 𝑺𝟒𝟑 𝑺𝟒𝟒
𝐼𝑚𝑎𝑔𝑒
𝑇𝑒𝑥𝑡
𝑳𝑰𝑻𝑪 + 𝐿𝐼𝑇𝑀 + 𝐿𝑀𝑎𝑠𝑘𝐿𝑀 𝐿𝑃𝑟𝑒𝑓𝑖𝑥𝐿𝑀

14. 014
Proposed method Model architecture
𝐿𝐼𝑇𝐶 + 𝑳𝑰𝑻𝑴 + 𝐿𝑀𝑎𝑠𝑘𝐿𝑀 𝐿𝑃𝑟𝑒𝑓𝑖𝑥𝐿𝑀

15. 015
Proposed method Pre-training methods (Image-Text Matching)
𝑻𝟏 𝑻𝟐 𝑻𝟑 𝑻𝟒
𝑰𝟏 𝑺𝟏𝟏 𝑺𝟏𝟐 𝑺𝟏𝟑 𝑺𝟏𝟒
𝑰𝟐 𝑺𝟐𝟏 𝑺𝟐𝟐 𝑺𝟐𝟑 𝑺𝟐𝟒
𝑰𝟑 𝑺𝟑𝟏 𝑺𝟑𝟐 𝑺𝟑𝟑 𝑺𝟑𝟒
𝑰𝟒 𝑺𝟒𝟏 𝑺𝟒𝟐 𝑺𝟒𝟑 𝑺𝟒𝟒
𝑻𝟏 𝑻𝟐 𝑻𝟑 𝑻𝟒
𝑰𝟏
𝑰𝟐
𝑰𝟑
𝑰𝟒
𝐼𝑚𝑎𝑔𝑒
𝑇𝑒𝑥𝑡
𝐼𝑚𝑎𝑔𝑒
𝑇𝑒𝑥𝑡
[Hard negative Image-Text pair]
𝐼1 − 𝑇2 𝐼2 − 𝑇1 𝐼3 − 𝑇1 𝐼4 − 𝑇3
𝑻𝟏 𝑻𝟐 𝑻𝟑 𝑻𝟒
𝑰𝟏
𝑰𝟐
𝑰𝟑
𝑰𝟒
𝐼𝑚𝑎𝑔𝑒
𝑇𝑒𝑥𝑡
[Hard negative Text-Image pair]
𝑇1 − 𝐼2 𝑇2 − 𝐼1 𝑇3 − 𝐼4 𝑇4 − 𝐿1

16. 016
Proposed method Pre-training methods (Image-Text Matching)
→
𝐿𝐼𝑇𝐶 + 𝑳𝑰𝑻𝑴 + 𝐿𝑀𝑎𝑠𝑘𝐿𝑀 𝐿𝑃𝑟𝑒𝑓𝑖𝑥𝐿𝑀
𝑳𝑰𝑻𝑴

17. 017
Proposed method Model architecture
𝐿𝐼𝑇𝐶 + 𝐿𝐼𝑇𝑀 + 𝑳𝑴𝒂𝒔𝒌𝑳𝑴 𝐿𝑃𝑟𝑒𝑓𝑖𝑥𝐿𝑀

18. 018
Proposed method Pre-training methods (Masked Language Modeling)
𝐿𝐼𝑇𝐶 + 𝐿𝐼𝑇𝑀 + 𝑳𝑴𝒂𝒔𝒌𝑳𝑴 𝐿𝑃𝑟𝑒𝑓𝑖𝑥𝐿𝑀

19. 019
Proposed method Model architecture
𝐿𝐼𝑇𝐶 + 𝐿𝐼𝑇𝑀 + 𝐿𝑀𝑎𝑠𝑘𝐿𝑀 𝑳𝑷𝒓𝒆𝒇𝒊𝒙𝑳𝑴

20. 020
Proposed method Pre-training methods (Prefix Language Modeling)

21. 021
Proposed method Pre-training methods
𝑳𝑰𝑻𝑪
𝑳𝑴𝑳𝑴 𝑳𝑰𝑻𝑴
𝑳𝑷𝒓𝒆𝒇𝒊𝒙𝑳𝑴

22. Question

23. Experiments

24. 024
Experiments Data & Setup
•
•
• )
•
𝐵𝐸𝑅𝑇𝑏𝑎𝑠𝑒
Multi-modal network
(Last 6-layers in 𝐵𝐸𝑅𝑇𝑏𝑎𝑠𝑒)
Decoder
(12-layers Transformer)
•
• NVIDIA A100 GPUs (16 GPUS)
• AdamW (Weight decay 0.02)
• 1e-5 (ViT) / 1e-4 (BERT)
• Random image crop, RandAugment
• 65,536
•

25. 025
Experiments Distributed learning on a large scale

26. 026
Evaluation on Vision-Language Tasks Visual Question Answering

27. 027
Evaluation on Vision-Language Tasks Image captioning

28. 028
Evaluation on Vision-Language Tasks Image-Text retrieval

29. 029
Evaluation on Vision-Language Tasks Visual grounding

30. 030
Evaluation on Vision-Language Tasks Visual reasoning

31. 031
Effectiveness and Efficiency Analysis of Stride for Skip

32. 032
Effectiveness and Efficiency Analysis of Cross-modal Fusion

33. 033
Zero-shot Transferability Image caption

34. 034
Zero-shot Transferability Image-Text retrieval

35. 035
Conclusion
• Presents mPLUG with novel cross-modal skip-connections, an effective and efficient VLP
framework for both cross-modal understanding and generation.
• mPLUG achieves state-of-the-art performance on a wide range of vision-language tasks with
zero-shot manner

36. THANK YOU