This document summarizes a presentation on multimodal language analysis using multimodal transformers for unaligned multimodal language sequences. It discusses how multimodal transformers can analyze text, vision, and audio inputs simultaneously using crossmodal attention. The proposed method is shown to generate embeddings that capture the relationships between different input modalities without strict alignment between the sequences. Various experiments are conducted to evaluate the model using different metrics and ablation studies.

1. Multimodal Transformer for
Unaligned Multimodal Language Sequences
Tsai et al., ACL 2019
딥러닝 논문읽기 자연어처리팀
발표자 : 주정헌 팀원: 황소현, 백지윤
2021.05.16

2. Introduction | Multimodal?
text
vision
acoustic
Human

3. Introduction | Multimodal?
Demo

4. Introduction | Multimodal?
Data2text

5. Proposed Method|Crossmodal Attention
seq2seq
Multimodal Transformer

6. Proposed Method|Crossmodal Attention
Human Multimodal Language Analysis.

7. Proposed Method|Crossmodal Attention
Aligned sequence
Today the weather is gorgeous and I see a beautiful blue sky.
Today the Weather is gorgeous And I see a

8. Proposed Method|Crossmodal Attention
Unaligned sequence
Today
the
Weather
is
gorgeous
And
0.513 0.323 0.422 0.145 0.612
0.425 0.122 0.432 0.6756 0.46
0.25 0.342 0.335 0.2345 0.26278
0.5124 0.312 0.245 0.113 0.7834
0.78 0.355 0.734 0.534 0.642
0.35 0.666 0.7665 0.4565 0.25223

9. Proposed Method|Crossmodal Attention
“Spectacle” 라는 단어의
Alignment visualization

10. Overall Architecture |Temporal Convolutions
X L : 텍스트 input
X V : 이미지 input
X A : 오디오 input
Input sequences has sufficient awareness of its
neighborhood elements.

11. Overall Architecture |Positional Embedding
PE = Augment positional embedding
Z = Low-level position-aware features for different
modalities

12. Overall Architecture |Crossmodal Transformers

13. Overall Architecture |Crossmodal Transformers
Target modality
(text)
Source modality
(Audio)
1) Audio -> text
2) Vision -> text
1) Text -> vision
2) audio -> vision
1) Text -> audio
2) vision -> audio

14. Overall Architecture |Crossmodal Transformers

15. Experiments |Evaluation metrics

16. Experiments |Evaluation metrics & Ablation study

17. Experiments |Ablation study

18. Experiments |Features and Hyper-parameters
Language : pre-trained Glove(840B.300d)
Vision: Facet, 35 facial action units, per-frame basic and
emotions
Audio : COVAREP MFCC, voice/unvoiced segmenting features

19. Q & A

20. Thank you