The document discusses different types of generative models including auto-regressive models, variational auto-encoders, and generative adversarial networks. It provides examples of each type of model and highlights some of their features and issues during training. Specific models discussed in more detail include PixelRNNs, DCGANs, WGANs, BEGANs, Pix2Pix, and CycleGANs. The document aims to introduce deep generative models and their applications.

1. GAN Introduction
Hyungjoo Cho

2. Generative Model

3. CAT
Generative Model

4. OWL
Generative Model

5. CAT
OWL
Generative Model

6. Let’s Think

7. A CAT ? 😺

8. A CAT ? 😺

9. A CAT ! 😺

10. Ideal Generative Model
Model
CAT
Short
Hair
Big
Ear

11. Ideal Generative Model
Model
CAT
Short
Hair
Big
Ear

12. Ideal Generative Model
Model
CAT
Short
Hair
Big
Ear
Tabby

13. Ideal Generative Model
Model
CAT
Short
Hair
Big
Ear
Tabby Savannah Cat

14. Why Generative
• Use high-dimensional, complicated probability distributions
• Combining with Reinforcement learning
• Missing data
• Semi-supervised learning
• Multi-modal outputs
• Code
• Can make data with realistic generation

15. –Richard Feynman
“What I cannot create, I do not understand.”

16. Deep Generative Models

17. What is Generative Model
https://blog.openai.com/generative-models/

18. Toy Example

19. Generative model

20. Generator (TF-code)

21. Result … 😢
Maybe we need more conditions…

22. Deep Generative Models
• Auto-Regressive Models
• Variational Auto-Encoder
• Generative Adversarial Networks

23. Auto-Regressive Models

24. Auto-Regressive Models
http://slazebni.cs.illinois.edu/spring17/lec13_advanced.pdf

25. Multi-Dimensional RNNs
<Graves et al, Multi-Dimensional Recurrent Neural Networks, 2013>
2D RNN Forward and Backward passes
Sequence ordering (not fixed) Multi-directional MDRNNs

26. Spatial LSTM
<Theis et al, Generative Image Modeling Using Spatial LSTMs, 2015>

27. Pixel RNN
<Aaron et al, Pixel Recurrent Neural Networks, 2016>

28. Sampling
• Feed the 2D vector of zeros( I ) to the generator
➔ The output vector : O
• O is softmax activations for each pixel
➔ Probability of first pixel value : p(0, 0)
• Random sampling from p(0, 0)
➔ Set the very first pixel of I with sampled value
• Feed I to the generator
• Iteration…

29. Results
<Aaron et al, Pixel Recurrent Neural Networks, 2016>

30. Features
• Simple and stable training process
• Best log likelihoods so far
• Inefficient during sampling
• Don’t easily provide simple low-dimensional codes for images

31. Variational Auto-Encoder

32. Variational Inference

33. Variational Inference

34. Latent variables
Model
CAT
Short
Hair
Big
Ear
Tabby Savannah Cat

35. Latent variables
Model
CAT
Short
Hair
Big
Ear
Tabby Savannah Cat
Latent Space Data Space

36. Latent variables
Model
Latent Space Data Space
Low Dimension High Dimension
Mapping Function

37. Variational Inference
Well Known Distribution
: Multivariate Gaussian

38. Variational Inference
Well Known Distribution
: Multivariate Gaussian
Sampling

39. Variational Inference
Well Known Distribution
: Multivariate Gaussian
Sampling
Make Distribution
: Pairs of
Mean, Variance

40. Kullback Leibler Divergence

41. Kullback Leibler Divergence

42. Kullback Leibler Divergence

43. Variational Inference
Well Known Distribution
: Multivariate Gaussian
Sampling
Make Distribution
: Pairs of
Mean, Variance
Optimize

44. Variational Auto-Encoder

45. Objective (Evidence Lower BOund)

46. Objective (Evidence Lower BOund)
Maximize Log-likelihood Minimize the distance of p and q

47. Auto-Encoder

48. Reparameterization Trick

49. Results

50. Features
• Simple and stable training process
• Can check log likelihood
• Latent variable
• Low quality

51. Generative Adversarial Nets

52. GANs

53. GANs

54. Vanilla GANs

55. Vanilla GANs

56. Vanilla GANs
<Goodfellow et al, Generative Adversarial Networks, 2014>

57. Results
<Goodfellow et al, Generative Adversarial Networks, 2014>

58. Features
• Advanced quality
• Unstable training
• Mode collapsing
• Cannot check log likelihood

59. Vanilla GANs
Model
Loss
Hyper
Parameters
Code

60. DCGAN

61. Deep Convolutional GAN
<Radford et al, Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, 2015>

62. Tips !!

63. Issues during Training
• Mode Collapsing / Oscillating
• Intractable Training Loss
• Balance btw Generator & Discriminator
• Manipulation
• Not Enough Quality

64. Mode Collapsing / Oscillating
The generator rotates through the modes of the data distribution.
<Metz et al, Unrolled Generative Adversarial Networks, 2016>

65. Mode Collapsing / Oscillating
< https://www.slideshare.net/HyungjooCho2/deep-generative-modelpdf >

66. Mode Collapsing / Oscillating
GAN uses Jenen-Shannon Divergence

67. Mode Collapsing / Oscillating
Target MLE JS R-KL

68. Mode Collapsing / Oscillating
Target MLE JS R-KL
JS and Reverse KL Divergence tend to favor under-generalization.
It never converges to a fixed distribution, and only ever assigns significant probability mass
to a single data mode at once.

69. Intractable Loss

70. Intractable Loss
< https://www.slideshare.net/ssuser77ee21/generative-adversarial-networks-70896091 >

71. Intractable Loss

72. Intractable Loss

73. Intractable Loss
GAN
LSGAN WGAN

74. Intractable Loss
The Wasserstein distance(left plot) is continuous and provides a usable gradient everywhere.
The JS plot(right) is not continuous and does not provide a usable gradient.
<Arjovsky et al, Wasserstein Generative Adversarial Networks, 2017>

75. Intractable Loss
The WGAN’s loss decreases consistently as training progresses
and sample quality increases.

76. Balance
Boundary Equilibrium GAN
<Berthelot et al, BEGAN, 2017>

77. Manipulation
Conditional GAN
<Mirza et al, Conditional Generative Adversarial Networks, 2014>

78. Quality
Progressive Growing of GAN
<Karras et al, Progressive Growing of GANs For Improved Quality, Stability, and Variation, 2017>

79. Quality

80. Significant Variants

81. Info GAN
InfoGAN successfully disentangles writing styles
<Chen et al, InfoGAN, 2017>

82. Info GAN
Real fake
Classifying time series data through unsupervised way
Clustering
< https://github.com/buriburisuri/timeseries_gan >

83. Pix2Pix
Most successful GAN architecture !!
<Isola et al, Image-to-image translation with conditional GAN, 2016>

84. Stain Style Transfer

85. Stain Style Transfer
SST achieves the highest performance on
original images on tumor classification

86. Domain Cross GAN
Unsupervised version of Pix2Pix
<Taigman et al, Unsupervised Cross-Domain Image Generation, 2016>

87. DiscoGAN / CycleGAN
<Kim et al, Learning to Discover Cross Domain Relations with Generative Adversarial Networks, 2017>

88. DiscoGAN / CycleGAN
<Zhu et al, Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks, 2017>

89. Simulated + Unsupervised Learning
<Shrivastava et al, Learning from Simulated and Unsupervised Images through Adversarial Training, 2016>

90. AnoGAN
<Schlegl et al,Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker discovery, 2017>

91. AmbientGAN

92. Generative Replay
<Shin et al,Continual Learning with Deep Generative Replay, 2017>

93. Thanks ☺