[NS][Lab_Seminar_250428]Geometry Sensitive Cross-Modal Reasoning fo...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Integrating Multi-Label
Contrastive Learning With Dual
Adversarial Graph Neural
Networks for Cross-Modal
Retrieval
Tien-Ba...

2
Introduction
Fig. 1. The left picture illustrates the co-occurrence of
“Window” and “Buildings” in the NUS-WIDE dataset....

3
Proposed Method
Figure 2. The overall architecture of P-GNN-CON model

4
Proposed Method
Problem Statement
● Training set include n image-text pairs
● Image feature and text feature of ith samp...

5
Proposed Method
Image and Text Encoding Network
● Image Encoding Network
○ Extract image features using VGGNet
○ Common ...

6
Proposed Method
Dual GANs
● ImgGAN
○ Rebuild pseudo image feature; it will be sent through generator of ImgGAN
○ Discrim...

7
Proposed Method
Multi-Hop GNNs
● Observed: category labels such as plant, tree, and flower include rich semantic correla...

8
Proposed Method
Multi-Hop GNNs
● Employ Multi-hop GNN to transform node features to corresponding inter-dependent
classi...

9
Proposed Method
Probabilistic Correlation Matrix of Multi-Hop GNN (P-GNN)
● Calculate conditional probability
● Denode c...

10
Proposed Method
Iterative Correlation Graph of Multi-Hop GNN (I-GNN)
● Idea: better node embeddings can be learned base...

11
Proposed Method
Iterative Correlation Graph of Multi-Hop GNN (I-GNN)
● Filter out noise and extract correlation matrix
...

12
Proposed Method
Multi-Label Cross-Model Contrastive Learning
● Propose contrastive loss by selecting pairs of sample th...

13
Proposed Method
Multi-Label Cross-Model Contrastive Learning
● For cross-modal, propose Inter-modality InfoNCE
● Inter-...

14
Proposed Method
Noise in Multi-Label Supervised Contrastive Learning
● Use label information; supervised contrastive le...

15
Proposed Method
Noise in Multi-Label Supervised Contrastive Learning
● Assume positive samples retrieved by taking cate...

16
Proposed Method
Soft Positive Sampling
● In contrastive learning, different positive samples have same sampling probabi...

17
Proposed Method
Soft Positive Sampling
● Soft InfoNCE can be viewed as assigning a weight of to positive samples
● Modi...

18
Proposed Method
Modal Reconstruction for Incomplete-Modal Learning
● Refer to a situation where some training samples l...

19
Proposed Method
Objective Function
● Modality gap can be bridged while generators attempt to produce modality-invariant...

20
Proposed Method
Objective Function
● Optimization procedure is implemented as minimax 2 concurrent sub-processes

21
Experiments
● Dataset
○ NUS-WIDE
○ MIRFlickr
○ MS-COCO
● Baseline
○ Cross-modal retrieval
■ CFA
■ CCA
■ PLS-C2A
■ JRL
■...

22
Experiments
SOTA Comparison
Table 1. The mAP Results on NUS-WIDE, MIRFlikcr, and MS-COCO Datasets

23
Experiments
SOTA Comparison
Table 2. The mAP Results of I-GNN-CON Variants on NUS-WIDE and MIRFlikcr

24
Experiments
SOTA Comparison
Fig. 7. Effects of various GNN models on NUS-WIDE and
MIRFlickr. Fig. 8. Effects of various...

25
Conclusion
● Propose a novel GNN and GAN based method integrated with multi-label contrastive learning to learn the
com...

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1011447 views
Storytelling For The Web: Integrate... by Chiara Aliotta 919244 views
Artificial Intelligence, Data and C... by OECD Directorate ... 820562 views
How to Leverage AI to Boost Employe... by SocialHRCamp 342424 views
2024 State of Marketing Report – by... by Marius Sescu 195380 views
Everything You Need To Know About C... by Expeed Software 218594 views

View on Slideshare

1 of 25 Ad

View on Slideshare

1 of 25 Ad