Overview of generative models with the accent to the GANs and deep learning. Includes autoencoders, VAE, normalizing flows, autoregressive models, and a lot of GAN architectures.

1. Generative models
Vitaly Bondar
DL Enthusiast & Senior Researcher @ Neuromation
johngull @ gmail | ODS

2. Generative task
Discriminative task
P(X, Y) or P(Y)
P(Y | X)

3. Solve as discriminative task

4. Solve as discriminative task

5. Autoregressive models

6. Autoregressive models
Training Inference

7. Autoregressive models

8. Autoregressive models

9. Autoregressive models

10. Autoregressive models. PixelCNN++
Multiscale processing for long dependencies

11. Autoregressive models. PixelSnail
PixelCNN/PixelCNN++ PixelSnail

12. Autoregressive models. PixelSnail

13. Autoregressive models. PixelSnail

14. Autoregressive models. Fast PixelCNN

15. Autoregressive models
Pros:
● Likelihood models
● Generate nice details
● Generalization
Cons:
● Slow inference
● Tend to lost global features

16. Autoencoder

17. Variational autoencoder (VAE)

18. Variational autoencoder (VAE)

19. Variational autoencoder (VAE)

20. Beta-VAE

21. Beta-VAE

22. VQ-VAE/VQ-VAE2

23. VQ-VAE2

24. NVAE

25. NVAE

26. NVAE

27. Latent variable models (autoencoders)
Pros:
● Direct loss
● Simple latent recovery
Cons:
● Encoder output distribution is limited (diagonal-covariance gaussian)
● Blurred images

28. Normalizing flows

29. Normalizing flows

30. Normalizing flows
f(x) - invertible and differentiable

31. Normalizing flows

32. Normalizing flows

33. Normalizing flows

34. Normalizing flows

35. Normalizing flows

36. Normalizing flows. HINT (Hierarchical coupling flow)

37. Normalizing flows. Glow

38. Normalizing Flows. Glow

39. Normalizing Flows. SRFlow

40. Normalizing Flows. SRFlow

41. Normalizing flows
Pros:
● Direct loss
● Simple latent recovery
● Exact distribution recovery
Cons:
● Lower quality than GANs for now
● Limitations for the model blocks

42. Generative adversarial networks
“I came up with this idea in a bar in Montreal, Canada
called "Three Winemakers." Ian Goodfellow

43. Generative adversarial networks

44. Generative adversarial networks

45. GAN. WGAN

46. GAN. Relativistic losses

47. GAN. Relativistic losses

48. GAN. Critic penalties
WGAN-GP
R1/R2
Any auxiliary target!

49. GAN architectures: DCGAN

50. GAN architectures: pix2pix (vid2vid)

51. GAN architectures: CycleGAN

52. GAN architectures: TransGAGA

53. GAN architectures: StarGAN

54. GAN architectures: AEGAN

55. GAN architectures: BicycleGAN

56. GAN architectures: BicycleGAN

57. GAN architectures: Self-supervised GAN

58. GAN architectures: Self-attention GAN

59. GAN architectures: ESRGAN

60. GAN architectures: Real-SR

61. GAN architectures: Text-to-Image

62. GAN architectures: BigGAN

63. GAN architectures: BigGAN (fun) results

64. GAN architectures: Progressive GAN

65. GAN architectures: StyleGAN

66. GAN architectures: StyleGAN noise effect

67. GAN architectures: StyleGAN truncation trick

68. GAN architectures: MSG-GAN

69. GAN architectures: StyleGAN2

71. GAN architectures: SinGan

72. GAN architectures: SinGan

73. GAN architectures: COCO-GAN

74. GAN architectures: GauGAN

75. GAN architectures: GauGAN

76. GAN architectures: infinite nature

77. GAN architectures: infinite nature

78. GAN architectures: FUNIT

79. GAN architectures: FUNIT

80. GAN architectures: Everybody dance now

81. GAN architectures: Everybody dance now

82. GAN. Data problem
● CelebA - 202,599
● CelebA-HQ - 300K
● FFHQ - 70K
● LSUN-Bedroom - 3,033,042
● LSUN-Church - 126,227
● LSUN-Car - 2,067,710

83. GAN. Data problem
What we usually do

84. GAN. Data problem
What GAN will generate

85. GAN. Data problem
Zhao et al. Improved Consistency Regularization for GANs

86. GAN. Adaptive Discriminator Augmentation (StyleGAN2-ADA)

87. GAN. Adaptive Discriminator Augmentation

88. GANs: tricks
● ReLU -> LeakyRELU
● MaxPool -> AvgPool or Conv+stride
● Upsampling -> Deconvolution, PixelShuffle
● Use additional data if you have it
● Averaged generator
● Discriminator regularization
● Weights equalizing/special normalizations

89. Thank you.