Multimodal deep learning is a contemporary technique of feature extraction from multiple modalities. Previously neural networks were able to analyze monolithic source of input such as audio or video or text. However, this new model multiplex multiple source of input and perform better quality prediction.

1. Multimodal Deep Learning
Akhter Al Amin
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee &
Andrew Ng

2. Audio-Visual Speech Recognition

3. Feature Challenge
Classifier (e.g. SVM)

4. Representing Lips
• Can we learn better representations for
audio/visual speech recognition?
• How can multimodal data (multiple sources
of input) be used to find better features?

5. Unsupervised Feature Learning
5
1.1
.
.
.
10
9
1.67
.
.
.
3

6. Unsupervised Feature Learning
5
1.1
.
.
.
10
9
1.67
.
.
.
3

7. Multimodal Features
1
2.1
5
9
.
.
.
.
.
.
.
6.5
9

8. Cross-Modality Feature Learning
5
1.1
.
.
.
10

9. Feature Learning Models

10. Feature Learning with Autoencoders
...
...
Audio Input
...
...
Video Input
... ...
Audio Reconstruction Video Reconstruction

11. Bimodal Autoencoder
...
... ...
... ...
Audio Input Video Input
Hidden
Representation
Audio Reconstruction Video Reconstruction
Adapted from: MIT 191

12. Bimodal Autoencoder
...
... ...
... ...
Audio Input Video Input
Hidden
Representation
Audio Reconstruction Video Reconstruction
Adapted from: MIT 191

13. Shallow Learning
Hidden
Units
Video Input Audio Input
• Mostly unimodal features learned
Adapted from: MIT 191

14. Bimodal Autoencoder
...
... ...
... ...
Audio Input Video Input
Hidden
Representation
Audio Reconstruction Video Reconstruction
Adapted from: MIT 191

15. Bimodal Autoencoder
...
...
... ...
Video Input
Hidden
Representation
Audio Reconstruction Video Reconstruction
Cross-modality Learning:
Learn better video features by using audio as a cue
Adapted from: MIT 191

16. Cross-modality Deep Autoencoder
...
...
...
...
... ...
...
Video Input
Learned
Representation
Audio Reconstruction Video Reconstruction
Adapted from: MIT 191

17. Cross-modality Deep Autoencoder
...
...
...
...
... ...
...
Audio Input
Learned
Representation
Audio Reconstruction Video Reconstruction
Adapted from: MIT 191

18. Bimodal Deep Autoencoders
...
...
... ...
...
...
... ...
...
Audio Input Video Input
Shared
Representation
Audio Reconstruction Video Reconstruction
“Visemes”
(Mouth Shapes)
“Phonemes”
Adapted from: MIT 191

19. Bimodal Deep Autoencoders
...
...
...
...
... ...
...
Video Input
Audio Reconstruction Video Reconstruction
“Visemes”
(Mouth Shapes)
Adapted from: MIT 191

20. “Phonemes”
Bimodal Deep Autoencoders
...
...
...
...
... ...
...
Audio Input
Audio Reconstruction Video Reconstruction
Adapted from: MIT 191

21. Bimodal Deep Autoencoders
...
...
... ...
...
...
... ...
...
Audio Input Video Input
Shared
Representation
Audio Reconstruction Video Reconstruction
“Visemes”
(Mouth Shapes)
“Phonemes”
Adapted from: MIT 191

22. Training Bimodal Deep Autoencoder
...
...
...
...
... ...
...
Audio Input
Shared
Representation
Audio Reconstruction Video Reconstruction
...
...
...
...
... ...
...
Video Input
Shared
Representation
Audio Reconstruction Video Reconstruction
...
...
... ...
...
...
... ...
...
Audio Input Video Input
Shared
Representation
Audio Reconstruction Video Reconstruction
• Train a single model to perform all 3 tasks
• Similar in spirit to denoising autoencoders
(Vincent et al., 2008)

23. Evaluations

24. Visualizations of Learned Features
0 ms 33 ms 67 ms 100 ms
0 ms 33 ms 67 ms 100 ms
Audio (spectrogram) and Video features
learned over 100ms windows

25. Lip-reading with AVLetters
● AVLetters:
○ 26-way Letter Classification
○ 10 Speakers
○ 60x80 pixels lip regions
● Cross-modality learning
...
...
...
...
... ...
...
Video Input
Learned
Representation
Audio Reconstruction Video Reconstruction
Feature Learning Supervised Learning Testing
Audio + Video Video Video

26. Lip-reading with AVLetters
Feature Representation Classification Accuracy
Multiscale Spatial Analysis
(Matthews et al., 2002)
44.6%
Local Binary Pattern
(Zhao & Barnard, 2009)
58.5%

27. Lip-reading with AVLetters
Feature Representation Classification Accuracy
Multiscale Spatial Analysis
(Matthews et al., 2002)
44.6%
Local Binary Pattern
(Zhao & Barnard, 2009)
58.5%
Video-Only Learning
(Single Modality Learning)
54.2%

28. Lip-reading with AVLetters
Feature Representation Classification Accuracy
Multiscale Spatial Analysis
(Matthews et al., 2002)
44.6%
Local Binary Pattern
(Zhao & Barnard, 2009)
58.5%
Video-Only Learning
(Single Modality Learning)
54.2%
(Cross Modality Learning) 64.4%

29. Lip-reading with CUAVE
● CUAVE:
○ 10-way Digit Classification
○ 36 Speakers
● Cross Modality Learning
...
...
...
...
... ...
...
Video Input
Learned
Representation
Audio Reconstruction Video Reconstruction
Feature Learning Supervised Learning Testing
Audio + Video Video Video

30. Lip-reading with CUAVE
Feature Representation Classification Accuracy
Baseline Preprocessed Video 58.5%
Video-Only Learning
(Single Modality Learning)
65.4%

31. Lip-reading with CUAVE
Feature Representation Classification Accuracy
Baseline Preprocessed Video 58.5%
Video-Only Learning
(Single Modality Learning)
65.4%
(Cross Modality Learning) 68.7%

32. Lip-reading with CUAVE
Feature Representation Classification Accuracy
Baseline Preprocessed Video 58.5%
Video-Only Learning
(Single Modality Learning)
65.4%
(Cross Modality Learning) 68.7%
Discrete Cosine Transform
(Gurban & Thiran, 2009)
64.0%
Visemic AAM
(Papandreou et al., 2009)
83.0%

33. Multimodal Recognition
● CUAVE:
○ 10-way Digit Classification
○ 36 Speakers
● Evaluate in clean and noisy audio scenarios
○ In the clean audio scenario, audio performs extremely well alone
Feature Learning Supervised Learning Testing
Audio + Video Audio + Video Audio + Video
...
...
... ...
...
...
... ...
...
Audio Input Video Input
Shared
Representation
Audio Reconstruction Video Reconstruction

34. Multimodal Recognition
Feature Representation
Classification Accuracy
(Noisy Audio at 0db SNR)
Audio Features (RBM) 75.8%
Our Best Video Features 68.7%

35. Multimodal Recognition
Feature Representation
Classification Accuracy
(Noisy Audio at 0db SNR)
Audio Features (RBM) 75.8%
Our Best Video Features 68.7%
Bimodal Deep Autoencoder 77.3%

36. Multimodal Recognition
Feature Representation
Classification Accuracy
(Noisy Audio at 0db SNR)
Audio Features (RBM) 75.8%
Our Best Video Features 68.7%
Bimodal Deep Autoencoder 77.3%
Bimodal Deep Autoencoder
+ Audio Features (RBM)
82.2%

37. Shared Representation Evaluation
Supervise
d
Testing
Audio
Shared
Representation
Video Audio
Shared
Representation
Video
Linear Classifier
Trainin
g
Testin
g
Feature Learning Supervised Learning Testing
Audio + Video Audio Video

38. Shared Representation Evaluation
Supervise
d
Testing
Audio
Shared
Representation
Video Audio
Shared
Representation
Video
Linear Classifier
Trainin
g
Testin
g
● Method: Learned Features + Canonical Correlation
Analysis
Feature Learning
Supervised
Learning
Testing Accuracy
Audio + Video Audio Video 57.3%
Audio + Video Video Audio 91.7%

39. McGurk Effect
A visual /ga/ combined with an audio /ba/ is often
perceived as /da/.
Audio
Input
Video
Input
Model Predictions
/ga/ /ba/ /da/
/ga/ /ga/ 82.6% 2.2% 15.2%
/ba/ /ba/ 4.4% 89.1% 6.5%

40. McGurk Effect
A visual /ga/ combined with an audio /ba/ is often
perceived as /da/.
Audio
Input
Video
Input
Model Predictions
/ga/ /ba/ /da/
/ga/ /ga/ 82.6% 2.2% 15.2%
/ba/ /ba/ 4.4% 89.1% 6.5%
/ga/ /ba/ 28.3% 13.0% 58.7%

41. Conclusion
● Applied deep autoencoders to
discover features in multimodal
data
● Cross-modality Learning:
We obtained better video features
(for lip-reading) using audio as a
cue
● Multimodal Feature Learning:
Learn representations that relate
across audio and video data
...
...
...
...
... ...
...
Video Input
Learned
Representation
Audio Reconstruction Video Reconstruction
...
...
... ...
...
...
... ...
...
Audio Input Video Input
Shared
Representation
Audio Reconstruction Video Reconstruction

42. Bimodal Learning with RBMs
…...
...
Audio Input
Hidden Units
...
Video Input

43. Discussion Topic
● What are the current state-of-the-art multimodal deep learning model?
● How current models get improved than this one?