This document summarizes a research paper on consistency regularization for GANs. The key points are: 1. Consistency regularization enforces the discriminator to be unchanged by arbitrary semantic-preserving perturbations of input data in order to focus more on semantic and structural changes between real and fake data. 2. Experiments show that consistency regularization improves baseline GAN performance across different loss functions, optimizer settings, and neural architectures on CIFAR-10 and CelebA datasets. 3. Consistency regularization is more effective and robust than other regularization methods such as gradient penalty, and can be used as a plug-and-play technique to boost state-of-the-art GAN models.

1. Yoonho Na
210604
Journal Club

2. 2

3. 1. GAN training implicitly requires
fi
nding the Nash equilibrium in a
continuous and high dimensional space.
2. Characterizing the convergence properties of the GAN training procedure
is mostly an open problem.
Instability
Introduction
Major problem with GAN
→ Normalization & Regularization

4. • Gradient based regularizatio
n
• Penalize the gradient norm of straight lines between real data and generated
data.
Gulrajani et al. (2017)
• Directly regularize the squared gradient another form of gradient penalty where
the gradients at Gaussian perturbations of training data are penalized.
Roth et al. (2017
)
• Most of the gradient based regularization methods lighter provide marginal gains or
fail to introduce improvement when normalization is used.
Kurach et al. (2019)
Introduction
Regularization

5. Introduction
Consistency Regularization
The classi
fi
er output remains una
ff
ected for an unlabeled example even it is augmented in semantic-preserving ways.
Enforces the discriminator to be unchanged by arbitrary semantic-preserving perturbations.
to focus more on semantic and structural changes between real and fake data.

6. • Propose con. reg. for GAN discriminators to yield a simple, effective regularizer with
lower computational cost than gradient-based regularization methods
.
• Conduct extensive experiments with different GAN variants to demonstrate that our
technique interacts effectively with spectral normalization
.
• Show that simply applying the proposed technique can further boost the
performance of SOTA GAN models.
Introduction
Contribution

7. • The goal of D ➞ distinguish real data from fake data produced by
G
• The decision should be invariant to any valid domain-speci
fi
c data augmentations
.
• Randomly augment training images as they are passed to the discriminator and
penalize the sensitivity of the discriminator to those augmentations.
Methods
Consistency Regularization for GANs
Ɗ (
𝑥
) : the output vector before activation of the
𝑗
th layer of the discriminator given input
𝑥
.
𝑇
(
𝑥
) : stochastic data augmentation function.
λ : weight coe
ffi
cient for
𝑗
th layer.
∥·∥: L2 norm

8. Methods
Consistency Regularization for GANs
Ɗ (
𝑥
) : the output vector before activation of the
𝑗
th layer of the discriminator given input
𝑥
.
𝑇
(
𝑥
) : stochastic data augmentation function.
λ : weight coe
ffi
cient for
𝑗
th layer.
∥·∥: L2 norm
• The proposed con. reg.
• Con. Reg. On the last layer of D is su
ffi
cient.
• The objective of Consistency Regularized GAN (CR-GAN)

9. • Datase
t
• CIFAR-10 : 60K of 32 x 32 images in 10 classe
s
• CELEBA-HQ-128 : 30K images of 128 x 128 (27K for training, and 3K for testing
)
• ImageNet-2012 : 1.2 million images with 1000 categories ➞ resized to 128 x 12
8
• Evaluation Metri
c
• Fréchet Inception distance (FID
)
• 10K images each on CIFAR-10, 3K on CelebA, 50K on ImageNe
t
Augmentation used in con. reg. is combination of random shifting,
fl
ipping
Experiments
Datasets and Evaluation Metrics

10. Compare 3 GAN reg. technique
s
• Gradient Penalty (GP
)
• DRAGAN Regularizer (DR
)
• JS-Regularizer (JSR
)
• Evaluate 3 reg. methods across different optimizer parameters,
loss functions,
regularization coef
fi
cient,
neural architecture
s
• Adam optimizer with batch size of 64 for all experiments
.
• Stop training after 200k generator update steps for CIFAR-10 and 100k steps for CelebA
.
• Spectral normalization is used in the discriminator.
Experiments
Comparison with other GAN Regularization Methods

11. • Evaluate reg. Method using 3 loss functions
.
• Non-Saturating loss (NS
)
• Wasserstein loss (WAS
)
• Hinge loss (Hinge
)
• Evaluate 7 hyper-parameter settings of Adam optimizer
.
• For Reg. coef
fi
cient, use the best value reported in the corresponding paper
.
• 10 for GP, DR, CR and 0.1 for JS
R
• Network architecture : SNDCGAN (Miyato et al., 2018)
Experiments
Impact of Loss Function

12. Experiments
Impact of Loss Function
CIFAR-10
CelebA
The con. reg. Improves the baseline across all di
ff
erent loss functions and both datasets.

13. Experiments
Hyperparameter Settings of Optimizer

14. Experiments
Impact of the Regularization Coe
ffi
cient
• Con. Reg. Is more robust to changes in λ than other GAN regularization techniques.
• Also has the best FID for both datasets.
➞ Con. Reg. Can be used as a plug-and-play technique to improve GAN performance.

15. Experiments
Impact of the Neural Architectures
SNDCGAN
ResNet
CIFAR-10

16. Experiments
Comparison with State-Of-The-Art GAN Models

17. Ablation Studies and Discussion
How much does augmentation matter by itself?
• Con. reg. has two parts :
(1) data augmentation, (2) enforce consistency between augmented data and original data.
• The performance gains shown in experiment due to data augmentation?
Compare 3 GANs : (1) GAN, (2) GAN with Augmentation, (3) GAN with Con. Reg.

18. Ablation Studies and Discussion
How does the type of augmentation a
ff
ect results?
• Ablation study on the CIFAR-10 dataset using four different types of data augmentatio
n
• Adding Gaussian nois
e
• Random shifting &
fl
ippin
g
• Applying cutou
t
• Cutout + random shifting &
fl
ipping
Adding Gaussian noise is not good semantic preserving transformation in the image manifold
The generator sometimes also generates samples with augmented artifacts (e.g., cutout). ➞ lead to worse FID

19. Conclusion
• Proposed a simple, effective, and computationally cheap method — consistency
regularization — to improve the performance of GANs
.
• Con. reg. is compatible with spectral normalization and results in improvements
in all of the many contexts
.
• Demonstrated con. reg. is more effective than other reg. methods under different
loss functions, neural architectures and optimizer hyper-parameter settings
.
• shown simply applying con. reg. on top of state-of-the-art GAN models can
further boost the performance.