The document summarizes a research paper that establishes formal connections between deep generative models like GANs and VAEs. It shows that GANs and VAEs can be viewed as special cases of a more general framework called adversarial domain adaptation (ADA). The ADA objective functionally connects GANs and VAEs by showing their objectives are equivalent when viewed from different perspectives. This connection provides opportunities to exchange ideas across GANs and VAEs to improve each approach.

1. Algorithmic Intelligence Lab Seminar:
On Unifying Deep Generative Models
2017.06.21.
Sangwoo Mo
1/21

2. Overview
• On Unifying Deep Generative Models (arXiv, 2 Jun 2017)
• Author: Zhiting Hu, Zichao Yang, Ruslan Salakhutdinov, Eric P. Xing
• Contribution
1. Establish formal connection between GAN and VAE
2. Enables to exchange ideas across models in principled way
(apply ideas in VAE to GAN, and ideas in GAN to VAE)
2/21

3. Table of Contents
• Bridging the Gap
• ADA (Adversarial Domain Adaptation)
• GAN (Generative Adversarial Network)
• VAE (Variational Autoencoder)
• WS (Wake Sleep Algorithm)
• Applications
• IWGAN (Importance Weighted GAN)
• AAVAE (Adversary Activated VAE)
• Experiments
• Conclusion
3/21

4. Table of Contents
• Bridging the Gap
• ADA (Adversarial Domain Adaptation)
• GAN (Generative Adversarial Network)
• VAE (Variational Autoencoder)
• WS (Wake Sleep Algorithm)
• Applications
• IWGAN (Importance Weighted GAN)
• AAVAE (Adversary Activated VAE)
• Experiments
• Conclusion
4/21

5. ADA (Adversarial Domain Adaptation)
• Goal: Transfer knowledge from source domain to target domain
• => Learn feature extractor whose output cannot be distinguished
by a discriminator of source and target domains
• Notations
• 𝒛: data (either from source or target domain)
• 𝒚 ∈ {0,1}: domain indicator (source: 1, target: 0)
• 𝒙 = 𝐺,(𝑧): feature (𝐺,: feature extractor)
5/21

6. ADA (Adversarial Domain Adaptation)
• More Notations
• 𝒑(𝒚): prior distribution (assume uniform)
• 𝒑(𝒛|𝒚): data distribution (for given domain)
• 𝒑 𝜽(𝒙|𝒚): implicit feature distribution
• 𝒒 𝝓(𝒚|𝒙): discriminator of domains
• 𝒒 𝝓
𝒓
𝒚 𝒙 = 𝑞7(1 − 𝑦|𝑥)
• 𝑫 𝝓 𝒙 = 𝑞7(𝑦 = 1|𝑥)
6/21

7. ADA (Adversarial Domain Adaptation)
• The discriminator should guess the true domain,
and the feature extractor should fool the discriminator
• Thus, the objective function is
(omitted supervised learning part of feature extractor)
7/21

8. GAN (Generative Adversarial Network)
• GAN can be seen as a special case of ADA
• Let real data to be source, and generated data to be target
• Note that 𝑝=(𝑥) is parametrized by 𝜃, while 𝑝?@A@(𝑥) is fixed
(code space and generator is degenerated for 𝑦 = 1)
• Here, ADA objective is identical to GAN (unsaturated version)
8/21

9. GAN (Generative Adversarial Network)
• GAN objective = 𝐾𝐿(𝑝, 𝑥 𝑦 ||𝑞D 𝑥 𝑦 ) − 𝐽𝑆𝐷(𝑝=||𝑝?@A@)
• Let 𝑦 as visible and 𝑥 as latent
• Then it is variational inference where 𝑞D 𝑥 𝑦 is posterior
• Since 𝑞D 𝑥 𝑦 ∝ 𝑝,I
𝑥 =
J
K
(𝑝= 𝑥 + 𝑝?@A@ 𝑥 ), 𝑝= goes to 𝑝?@A@
• Remark that it is reverse KL, thus occurs mode collapse problem
9/21

10. GAN (Generative Adversarial Network)
• InfoGAN: additionally recover the latent code 𝑧
• Simply introduce extra conditional 𝑝M(𝑧|𝑥, 𝑦)
• Then the objective is
where 𝑞D 𝑥 𝑧, 𝑦 ∝ 𝑞MI
𝑧 𝑥, 𝑦 𝑞7I
D
𝑦 𝑥 𝑝,I
(𝑥)
10/21

11. VAE (Variational Autoencoder)
• Assume VAE has optimal (degenerated) discriminator 𝑞∗(𝑦|𝑥)
• VAE detects every false data, and only learns from real data
• The original objective is
•
and identical to
11/21

12. Compare GAN and VAE
• In summary
• InfoGAN objective: 𝐾𝐿(𝑝, 𝑥 𝑧, 𝑦 ||𝑞D 𝑥 𝑧, 𝑦 )
• VAE objective: 𝐾𝐿(𝑞D 𝑧, 𝑦 𝑥 ||𝑝, 𝑧, 𝑦 𝑥 )
• Remark that (1) position of 𝑝 and 𝑞 are reversed, and
(2) hidden/visible variables 𝑥 and 𝑦, 𝑧 are inverted
• VAE minimizes KL -> smoothed output
• GAN minimizes reverse KL -> mode collapse
=> VAE/GAN joint model
12/21

13. WS (Wake Sleep Algorithm)
• Classic wake-sleep algorithm is
• VAE = wake step
• Let ℎ be 𝑧 and 𝜆 be 𝜂. VAE objective is 𝑝,, but also optimize 𝑞M
• GAN = sleep step
• Let ℎ be 𝑦 and 𝜆 be 𝜙. GAN objective is 𝑞7, but also optimize 𝑝,
13/21

14. Table of Contents
• Bridging the Gap
• ADA (Adversarial Domain Adaptation)
• GAN (Generative Adversarial Network)
• VAE (Variational Autoencoder)
• WS (Wake Sleep Algorithm)
• Applications
• IWGAN (Importance Weighted GAN)
• AAVAE (Adversary Activated VAE)
• Experiments
• Conclusion
14/21

15. IWGAN (Importance Weighted GAN)
• Importance Weighted GAN
• In practice, just assign weights for each samples in mini-batch
15/21

16. AAVAE (Adversary Activated VAE)
• Adversary Activated VAE
• Motivation: Utilize fake samples
• => Use discriminator network 𝑞7(𝑦|𝑥) instead of optimal 𝑞∗(𝑦|𝑥)
• Objective
16/21

17. AAVAE (Adversary Activated VAE)
• However, (18) is intractable since 𝑝, 𝑥 𝑧, 𝑦 = 1 = 𝑝?@A@(𝑥) is
an implicit distribution (cannot estimate likelihood)
• In practice, AAVAE use binary classifier same as GAN
• In my opinion, it is just a GAN variant using different 𝐺 objective
17/21

18. Table of Contents
• Bridging the Gap
• ADA (Adversarial Domain Adaptation)
• GAN (Generative Adversarial Network)
• VAE (Variational Autoencoder)
• WS (Wake Sleep Algorithm)
• Applications
• IWGAN (Importance Weighted GAN)
• AAVAE (Adversary Activated VAE)
• Experiments
• Conclusion
18/21

19. Experiment Results
19/21

20. Table of Contents
• Bridging the Gap
• ADA (Adversarial Domain Adaptation)
• GAN (Generative Adversarial Network)
• VAE (Variational Autoencoder)
• WS (Wake Sleep Algorithm)
• Applications
• IWGAN (Importance Weighted GAN)
• AAVAE (Adversary Activated VAE)
• Experiments
• Conclusion
20/21

21. Conclusion & Discussions
• Conclusion
• Traditional models usually distinguish visible/latent variables
• However, it may not necessary to make clear boundary between
visible/latent and generator/discriminator
• GAN and VAE can be thought as (in some sense) dual
• Research Directions
• Generalize framework to connect to other learning paradigms
e.g. Reinforcement Learning, Energy-based model, etc.
21/21

22. Appendix
22/21

23. Appendix: ADA objective = GAN objective
23/21

24. Appendix: Reverse KL divergence
24/21

25. Appendix: Proof of Lemma 1.
25/21

26. Appendix: Proof of Lemma 2.
26/21