berlinmeetup2020

slideshare.net/JannikWolff/berlinmeetup2020
Multimodal Hierarchical
Variational Autoencoders
JannikWolff Rahul G.
Krishnan
Jan Nikolas
Morshuis
Tassilo Klein
Moin Nabi
www.linkedin.com/in/jannik-wolff

OUTLINE
1 Motivation
Modalities?
Variational Autoencoders?
Research Question?
2 Method
Hierarchies!
3 Experiments

OUTLINE
1 Motivation
Modalities?
Variational Autoencoders?
Research Question?
2 Method 3 Experiments

What Are Modalities?
4

5
Google.com
Definition

6
https://twitter.com/karpathy
Images
Visual Data

7
https://www.pinterest.de/pin/471259548496679439/, https://en.wikipedia.org/wiki/Monochrome, https://www.pinterest.com/pin/378161699930406001/
Paintings
Monochrome Pictures
Old Photographs
Other Visual Data

8
http://www.vision.caltech.edu/visipedia/CUB-200.html [Shi et al., 2019]; [Xian et al., 2018].
Descriptions
Text

Exemplary Tasks?
9

Helping the Visually Impaired
10
https://towardsdatascience.com/image-captioning-in-deep-learning-9cd23fb4d8d2
Input: Image
Output: Caption

Art: Style Transfer
11
[Zhu et al., 2017]

12
Language
Model
Plausible
Text

13
Language
Model
Plausible
Text

Approaches?
14
“What I cannot create, I do not understand.”
—Richard Feynman

One Modality: Autoencoders
Encoder Decoder
Latent Representation
[Kingma and Welling, 2014]
15

One Modality: Variational Autoencoders (VAE)
Encoder Decoder
[Kingma and Welling, 2014]
Loss 16
Latent Representation

Examplary Samples
Imagined Faces
[Vahdat and Kautz, 2020]
17
Related Work

Multiple Modalities
[Xian et al., 2018]. We later add the species “bird” for illustrative purposes.
18
Images and Attributes

Multimodal VAE
19
Attribute
Encoder
Image
Encoder
Image
Decoder
Attribute
Decoder
horns
spots
fast
swims
timid
domestic
HumanEstimate
87%
9%
71%
1%
39%
8%
horns
spots
fast
swims
timid
domestic
HumanEstimate
87%
9%
71%
1%
39%
8%
[Wu and Goodman, 2018] among others

Multimodal VAE
20
Attribute
Encoder
Image
Encoder
Image
Decoder
Attribute
Decoder
https://www.dreamstime.com/stickman-man-simple-stickman-man-simple-smiling-vector-image107200458
Modality Expert
Modality Experts Are Latent Distributions
Modality Expert
Latent
Representation

VAE: Graphical Models
21
Generative Model
(Decoder)
Inference Model
(Encoder)

Example
22
FashionMNIST
[Wu and Goodman, 2018]

Example: Conditional Generation
23
Inference Model

24
Generative Model Inference Model

25
Input: Label “Ankle Boot”
Output: Images
[Wu and Goodman, 2018]
Related Work

Research Question?
26

How to Represent Shared and Individual Structure?
27
horns
spots
fast
swims
timid
domestic
HumanEstimate
87%
9%
71%
1%
39%
8%
Visual Modality Attribute Modality
https://cvml.ist.ac.at/AwA2/

Example: Why is Diversity Important?
28
Average Image
(per-pixel)
Many “Antelope”
Images

RECAP
• Multimodal World
• Variational Autoencoders
• Shared and Individual Structure

OUTLINE
1 Motivation 2 Method
Hierarchies!
3 Experiments

Outline
1. Generative Model
2. Inference Model
3. Loss
31

Outline
1. Generative Model
2. Inference Model
3. Loss
32

How To Build The Generative Model?
33

Single Shared Variable
34
[Wu and Goodman, 2018], [Shi et al., 2019]
Related Work

Single Shared Variable
35
Related Work
Balancing Variation and Shared
Structure?

Disentangled Variables
36
[Mahajan et al., 2019], [Daunhawer et al., 2019]
Related Work

Shared and Disentangled Variables
37
• z1: bird variation
• g: lion structure
Related Work

Ignored Dependencies
38
https://www.deviantart.com/unkeptgirl/art/Flying-Lion-491864413

Proposed Hierarchy
39
Concurrent work: [Vasco et al., 2020]
Shared
structure
Unimodal
variation

Proposed Hierarchy
40
Conditional Variation
Shared
structure
Unimodal
variation
Hypothesis

41
Increasing
Complexity

42
e.g., furriness, size or paws

43
e.g., texture,
background

44
e.g., texture,
background e.g., behavior

Optimization?
45

Recap: Standard VAE Optimization
46
Bound
Unimodal
Standard VAE
1.

47
1.
2.
Arithmetics -> LossUnimodal
Standard VAE
Bound

48
1.
2.
Arithmetics -> LossUnimodal
Standard VAE
Bound

Bedmonsters and Math
49
http://learnenglishkids.britishcouncil.org/jokes/monsters-under-your-bed

Proposed Optimization
50
Bound

Generative Model
51
Bound

Generative Model
52
Simpler As it Looks!

Generative Model
53
Unconditional
Prior

Generative Model
54
Unconditional
Prior
Conditional
Priors

Generative Model
55
Unconditional
Prior
Conditional
Priors
Reconstruction

Outline
1. Generative Model
2. Inference Model
3. Loss
56

Proposed Inference Model
57
Unimodal HVAE by
[Sønderby, 2016]

58
1. Shared variable g
Example:
furriness, size or paws

59
2. Unimodal variables z_i

• Top-down: shared concepts
60
Example:
furriness, size or paws

• Top-down: shared concepts
• Bottom-up: unimodal variation
61
Example:
Background or texture

How to Infer Shared Variable?
62
Combining Modality Experts

63
One Modality
Expert

64
Another Modality
Expert

How to Infer the Shared Variable?
65
[Wu and Goodman, 2018], [Shi et al., 2019]
- Mixture of Experts
- Product of Experts
Several Approaches

Inference Model
66
Bound

Inference Model
67
Simpler As It Looks!

Inference Model
68
Top-Level Posterior
(combines modality experts)

Inference Model
69
Top-Level Posterior
Lower-Level Posteriors

Outline
1. Generative Model
2. Inference Model
3. Loss
70

Loss
71
Generative
Model
Inference Model

Loss
72
Generative
Model
Inference Model

Loss
73

Loss
74
KL-Divergence
between
Priors and
Posteriors

Loss
75
Reconstructions
KL-Divergence
between
Priors and Posteriors

RECAP
• Generative Model
• ConditionalVariable
• Inference Model
1. SharedVariable
2. UnimodalVariables

OUTLINE
1 Motivation 2 Method 3 Experiments

Hypotheses
1. Meaningful Latent Representation
2. Diverse Generations
78

Dataset: Animals with Attributes 2
79
[Xian et al., 2018]. We currently use a random subset of classes. We use the ResNet features provided by the authors. We span the hierarchy solely over the first modality.

Hypothesis 1:
Meaningful Latent Represenation
80

What To Look Out For?
81
Latent Spaces Look
Different
Hierarchy
Identical Modality
Experts
Alignment

Latent Space Plots
82
Generative Model

AWA2: Dimensionality Reduction
83
Shared Variable g Unimodal Variable z_1
We reduce the amount of datapoints and classes for clarity.
Full Resolution onNext Slide!

Alignment
Different Structure
AWA2: Dimensionality Reduction
84
Shared
https://www.vecteezy.com/vector-art/554272-check-mark-vector-icon
Unimodal

Hypothesis 2:
Diverse Generations
85

Generate x1 from x2
86
Inference Model

Generate x1 from x2
87
Generative Model
Inference Model
We plot this
variable!

Diverse Generations
AWA2: Image Features from Attribute “Antelope”
88
We maximize the likelihood of ResNet-101 features. We plot test images corresponding to nearest neighbors in feature space.
Product VAE Proposed Method

RECAP
Confirmed Hypotheses:
1. Meaningful Latent
Representation
2. Diverse Generations
Outlook:
• Robustness by Regularization

SUMMARY
Motivation:
• Shared and Individual Structure
Method:
• Generative Model
• Inference Model
• Loss
Experiments:
• Meaningful Latent
Representation
• Diverse Generations

ThankYou!
Questions?
www.linkedin.com/in/jannik-wolff

berlinmeetup2020

Recommended

Recommended

More Related Content

Recently uploaded

Recently uploaded (20)

Featured

Featured (20)

berlinmeetup2020