My summary of the paper "Infogan-CR : Disentangling Generative Adversarial Networks with Contrastive Regularizers"

1. Paper Summary of :
Infogan-CR : Disentangling Generative Adversarial
Networks with Contrastive Regularizers
Jun-sik Choi
Department of Brain and Cognitive Engineering,
Korea University
November 9, 2019

2. InfoGAN

3. Overview of Vanila InfoGAN [1]
InfoGAN learns disentangled representation of data without
any supervision.
By maximizing mutual information between c and G(z, c),
InfoGAN achieves latent representation of the data.
If ci ∼ Cat(k), each category of latent code represents class of
the data.
If ci is continuous variable, the variation of latent code can
represent continual change of some attributes that is
represented by the code.

4. Variational Mutual Information Maximization
V (D, G) = Ex∼Pdata
[log D(x)] + Ez∼ noise [log(1 − D(G(z)))]
I(c; G(z, c))
= H(c) − H(c|G(z, c))
= Ex∼G(z,c) Ec ∼P(c|x) [log P (c |x)] + H(c)
= Ex∼G(z,c)


DKL(P(·|x) Q(·|x))
≥0
+Ec ∼P(c|x) [log Q (c |x)]


 + H(c)
≥ Ex∼G(z,c) Ec ∼P(c|x) [log Q (c |x)] + H(c)
= Lower bound of Mutual Information
= Ec∼P(c),x∼G(z,c) Ec ∼P(c|x) [log Q (c |x)] + H(c) (Lemma 5.1 from [1])
= LI (G, Q)
Minimax game of InfoGAN
= min
G
max
D
VI (D, G) = V (D, G) − λI(c; G(z, c))

5. Results of InfoGAN

6. InfoGAN-CR

7. Overview of InfoGAN-CR [2]
InfoGAN-CR provides additional contrastive regularizer to
enhance InfoGAN’s disentangled representation.
Also, this paper shows that the InfoGAN can show better
disentanglement than VAE based models with proper
techniques to stabilizing training procedure (spectral
normalization, two time-scale update rules).
InfoGAN-CR showed state-of-the-art performance for
disentanglement on dSprite dataset.

8. Contrastive regularizer I
InfoGAN-CR added contrastive regularizer to the target
function of vanila InfoGAN.
min
G,H
max
D
LAdv(G, D) − λI(c; G(c, z)) − αLc(G, H)
Key insight of contrastive loss is that the disentanglement is
fundamentally measured by the changes made when traversing
the latent space.
The changes from different latent code ci should be
well-distinguishable in the disentangled latent space.
The CR discriminator H is fed with two images which are
share one latent code and predict the shared code index.
The Generator G should generate images that have
distinguishable features along the latent code to diminish the
Lc

9. Contrastive regularizer II
Calculating Contrastive Loss
1. Draw a random index I over k(number of latent code) indices.
2. Sample the chosen latent code cI ∈ R.
3. Generate image m ∈ {1, 2} from latent code cm
j where ith
code is fixed to cI .
4. The contrastive gap is defined as minj∈[k]{I} c1
j − c2
j .
5. Generated images x, x are fed into discriminator H which try
to identify which code was fixed.
6. Generator G and CR discriminator H define contrastive loss
using cross entropy loss:
Lc(G, H) = EI∼U([k]),(x,x )∼Q(I)[ I,log H(x,x ) ]
where Q(I)
denotes the joint distribution of the paired images
and I denotes the one-hot encoding, and H is k-dimensional
vector normalized to be 1, H (x, x ) = 1.

10. Results
Figure: Comparison of disentanglement metric on the dSprite dataset.
The modified InfoGAN trained with stabilizing techniques
performs much better than the vanila InfoGAN.
InfoGAN-CR showed state-of-the-art disentanglement
compared to other methods.

11. References

12. References
X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever,
and P. Abbeel, “Infogan: Interpretable representation learning
by information maximizing generative adversarial nets,” in
Advances in neural information processing systems,
pp. 2172–2180, 2016.
Z. Lin, K. K. Thekumparampil, G. Fanti, and S. Oh,
“Infogan-cr: Disentangling generative adversarial networks with
contrastive regularizers,” arXiv preprint arXiv:1906.06034,
2019.