The detailed overview of the whole family of StyleGANs starting from the ProgressiveGAN to the latest StyleGAN3. Such a continuous look at the StyleGAN improvements gives an excellent understanding of the research principles and approaches for improving your own models.

1. Evolution of the
StyleGAN family
presented by
Vitaly Bondar
Generative DL enthusiast
johngull @ gmail

2. Recap: GAN
Goodfellow et al. Generative Adversarial Networks, 2014

3. Recap: DCGAN
Radford et al. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, 2015

4. ProgressiveGAN
Karras et al. Progressive Growing of GANs for Improved Quality, Stability, and Variation, 2017

5. ProgressiveGAN
Karras et al. Progressive Growing of GANs for Improved Quality, Stability, and Variation, 2017

6. ProgressiveGAN
Karras et al. Progressive Growing of GANs for Improved Quality, Stability, and Variation, 2017
● Leaky ReLU
● Add minibatch standard deviation as value layer at the end of
discriminator
● Equalized learning rate (initialize weights as N(0;1), normalize with
He at runtime)
● PixelNorm
● Exponential running average for the weights of the generator
● Small mini-batches
● WGAN-GP + discriminator drift regularization | LSGAN + noise at
discriminator
● lr and batch size changes with the resolution changes

7. StyleGAN
Karras et al. A Style-Based Generator Architecture for Generative Adversarial Networks, 2018

8. StyleGAN
Karras et al. A Style-Based Generator Architecture for Generative Adversarial Networks, 2018

9. StyleGAN
Karras et al. A Style-Based Generator Architecture for Generative Adversarial Networks, 2018
● Leaky ReLU
● Equalized learning rate (initialize weights as N(0;1), normalize with
He at runtime)
● Exponential running average for the weights of the generator
● Small mini-batches
● lr and batch size changes with the resolution changes
● NS GAN Loss + R1 regularization
● lr for mapping network = 0.01*lr
● Truncation trick

10. StyleGAN
Karras et al. A Style-Based Generator Architecture for Generative Adversarial Networks, 2018

11. StyleGAN
Karras et al. A Style-Based Generator Architecture for Generative Adversarial Networks, 2018

12. StyleGAN
Karras et al. A Style-Based Generator Architecture for Generative Adversarial Networks, 2018

13. StyleGAN2
Karras et al.Analyzing and Improving the Image Quality of StyleGAN, 2019
StyleGAN waterdrops

14. StyleGAN2
Karras et al.Analyzing and Improving the Image Quality of StyleGAN, 2019
StyleGAN2 solution: weights demodulation

15. StyleGAN2
Karras et al.Analyzing and Improving the Image Quality of StyleGAN, 2019
StyleGAN2 solution: weights demodulation

16. StyleGAN2
Karras et al.Analyzing and Improving the Image Quality of StyleGAN, 2019
StyleGAN: low FID, but high PPL

17. StyleGAN2
Karras et al.Analyzing and Improving the Image Quality of StyleGAN, 2019
StyleGAN2 solution: lazy path length regularization
...

18. StyleGAN2
Karras et al.Analyzing and Improving the Image Quality of StyleGAN, 2019
StyleGAN: “phase” artifacts

19. StyleGAN2
Karras et al.Analyzing and Improving the Image Quality of StyleGAN, 2019
StyleGAN2 solution: skip connections

20. StyleGAN2
Karras et al.Analyzing and Improving the Image Quality of StyleGAN, 2019

21. StyleGAN2
Karras et al.Analyzing and Improving the Image Quality of StyleGAN, 2019

22. StyleGAN2
Karras et al.Analyzing and Improving the Image Quality of StyleGAN, 2019
● Almost all previous :)
● Weights modulation/demodulation (no demodulation in tRGB)
● NS GAN Loss + R1 regularization + PPL regularization
● Regularization rare applying
● Skip and residual connections instead of progressive growing
● Double size of last resolution

23. StyleGAN2
Karras et al.Analyzing and Improving the Image Quality of StyleGAN, 2019

24. StyleGAN2-ADA
Karras et al. Training Generative Adversarial Networks with Limited Data, 2020
GANs problem: data hungry

25. StyleGAN2-ADA
Karras et al. Training Generative Adversarial Networks with Limited Data, 2020
Usual CV vs GANs
Dataset augmentation

26. StyleGAN2-ADA
Karras et al. Training Generative Adversarial Networks with Limited Data, 2020
Usual CV vs GANs
Dataset augmentation Generator output

27. StyleGAN2-ADA
Karras et al. Training Generative Adversarial Networks with Limited Data, 2020

28. StyleGAN2-ADA
Karras et al. Training Generative Adversarial Networks with Limited Data, 2020
Leaking of augmetation

29. StyleGAN2-ADA
Karras et al. Training Generative Adversarial Networks with Limited Data, 2020

30. StyleGAN2-ADA
Karras et al. Training Generative Adversarial Networks with Limited Data, 2020

31. StyleGAN3 (Alias-Free GAN)
Karras et al. Alias-Free Generative Adversarial Networks, 2021
StyleGAN2 problem

32. StyleGAN3 (Alias-Free GAN)
Karras et al. Alias-Free Generative Adversarial Networks, 2021
StyleGAN2 problem
Despite their hierarchical convolutional nature, the synthesis process of typical generative
adversarial networks depends on absolute pixel coordinates in an unhealthy manner.
Unintentional positional references available to the intermediate layers:
● through image borders
● per-pixel noise inputs
● positional encodings, and aliasing
Authors identified two sources:
● non-ideal upsampling filters (e.g., nearest, bilinear, strided convolutions)
● the pointwise application of nonlinearities such as ReLU.

33. StyleGAN3 (Alias-Free GAN)
One-line idea:
Use contiguous space equivariant operations to remove alias and
borders-using effects.
Karras et al. Alias-Free Generative Adversarial Networks, 2021

34. StyleGAN3 (Alias-Free GAN)
Karras et al. Alias-Free Generative Adversarial Networks, 2021
StyleGAN2 (without skip)

35. StyleGAN3 (Alias-Free GAN)
Karras et al. Alias-Free Generative Adversarial Networks, 2021

36. StyleGAN3 (Alias-Free GAN)
Convolution - everything good
Karras et al. Alias-Free Generative Adversarial Networks, 2021

37. StyleGAN3 (Alias-Free GAN)
Upsampling/downsampling.
● Identity operations in the continuous space, we just need to change sampling
rate.
● Upsampling ~= add zeros for interleaving and convolve with discretized filter
● Downsampling = low-pass filter and sampling =
~= discrete convolution + dropping points
Karras et al. Alias-Free Generative Adversarial Networks, 2021

38. StyleGAN3 (Alias-Free GAN)
Nonlinearity
Karras et al. Alias-Free Generative Adversarial Networks, 2021

39. StyleGAN3 (Alias-Free GAN)
Karras et al. Alias-Free Generative Adversarial Networks, 2021

40. StyleGAN3 (Alias-Free GAN)

41. StyleGAN3 (Alias-Free GAN)
Karras et al. Alias-Free Generative Adversarial Networks, 2021

42. StyleGAN3 (Alias-Free GAN)
● In FFHQ (1024×1024) the three generators (StyleGan2, Alias-Free-T,
Alias-Free-R) had 30.0M, 22.3M and 15.8M parameters, while the training times
were 1106, 1576 (+42%) and 2248 (+103%) GPU hours.
● The present research consumed 92 GPU years and 225 MWh of electricity on
an in-house cluster of NVIDIA V100s.
Karras et al. Alias-Free Generative Adversarial Networks, 2021

43. Karras et al. Alias-Free Generative Adversarial Networks, 2021

45. Evolution of the StyleGAN family (oversimplified and as for October 2022)