The document discusses FactorVAE, a method for disentangling latent representations in variational autoencoders (VAEs). It introduces Total Correlation (TC) as a penalty term that encourages independence between latent variables. TC is added to the standard VAE objective function to guide the model to learn disentangled representations. The document provides details on how TC is defined and computed based on the density-ratio trick from generative adversarial networks. It also discusses how FactorVAE uses TC to learn disentangled representations and can be evaluated using a disentanglement metric.

1. DEEP LEARNING JP
[DL Papers]
Disentangling by Factorising
Hirono Okamoto, Matsuo Lab
http://deeplearning.jp/
1

2. : Disentangling by Factorising
n 2018 ICML accepted
n : Hyunjik Kim, Andriy Mnih
n β-VAE disentanglement metric
n disentangle
n
rotation position x scale position y
Shape
( )
gif
https://github.com/1Konny/FactorVAE

3. : beta-VAE
n : β-VAE: LEARNING BASIC VISUAL CONCEPTS WITH A CONSTRAINED
VARIATIONAL FRAMEWORK
n ICLR 2017(poster)
n :
n VAE disentangle
n disentanglement VAE …
azimuth entangle
??

4. β-VAE :
n : disentangle
n ⇒
n p(z) N(0, I)
n β 1 βVAE VAE
n β z
disentangle
n β trade off …

5. β-VAE : disentangle
n accuracy disentangle
n 1. k ( scale)
n 2. L
n
n encode
n encode
n 3. 2 z ( )
y (y=Wz )
n disentangle z
y
n = metric score
n K-1 disentangle
100% …
Disentangle Metric
Scale Scale z
0

6. FactorVAE β-VAE ( )
n β-VAE disentanglement
n disentanglement
n ⇒ Total Correlation Penalty
n β-VAE disentanglement metric
n ( : L iteration )
n ( … )
n (K K-1 disentangle
100% )
n ⇒ a new metric for disentanglement
new metric
old metric

7. FactorVAE : Total Correlation Penalty
n FactorVAE
n Total Correlation (TC):
n GAN z
density-ratio trick
q(z)
¯q(z)
=
p(z|y = 1)
p(z|y = 0)
=
p(y = 1|z)
p(y = 0|z)
⇡
D(z)
1 D(z)
density-ratio trick
q(z) =
Z
pdata(x)q(z|x)dx
<latexit sha1_base64="PEMDYUO5OQ+UOtLO81MQvQtmmIU=">AAAChHicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQvIF2pUaSrYKsRk5pUoFMRXpySWJNZqVGgChWsqNFMq4gWUDfQMwEABk2EIZSgzQEFAvsByhhiGFIZ8hmSGUoZchlSGPIYSIDuHIZGhGAijGQwZDBgKgGKxDNVAsSIgKxMsn8pQy8AF1FsKVJUKVJEIFM0GkulAXjRUNA/IB5lZDNadDLQlB4iLgDoVGFQNrhqsNPhscMJgtcFLgz84zaoGmwFySyWQToLoTS2I5++SCP5OUFcukC5hyEDowuvmEoY0BguwWzOBbi8Ai4B8kQzRX1Y1/XOwVZBqtZrBIoPXQPcvNLhpcBjog7yyL8lLA1ODZjNwASPAED24MRlhRnqGBnqGgSbKDk7QqOBgkGZQYtAAhrc5gwODB0MAQyjQ3laG1QxbGLYysTHpMBkzmUKUMjFC9QgzoAAmOwDSDpSU</latexit><latexit sha1_base64="PEMDYUO5OQ+UOtLO81MQvQtmmIU=">AAAChHicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQvIF2pUaSrYKsRk5pUoFMRXpySWJNZqVGgChWsqNFMq4gWUDfQMwEABk2EIZSgzQEFAvsByhhiGFIZ8hmSGUoZchlSGPIYSIDuHIZGhGAijGQwZDBgKgGKxDNVAsSIgKxMsn8pQy8AF1FsKVJUKVJEIFM0GkulAXjRUNA/IB5lZDNadDLQlB4iLgDoVGFQNrhqsNPhscMJgtcFLgz84zaoGmwFySyWQToLoTS2I5++SCP5OUFcukC5hyEDowuvmEoY0BguwWzOBbi8Ai4B8kQzRX1Y1/XOwVZBqtZrBIoPXQPcvNLhpcBjog7yyL8lLA1ODZjNwASPAED24MRlhRnqGBnqGgSbKDk7QqOBgkGZQYtAAhrc5gwODB0MAQyjQ3laG1QxbGLYysTHpMBkzmUKUMjFC9QgzoAAmOwDSDpSU</latexit><latexit sha1_base64="PEMDYUO5OQ+UOtLO81MQvQtmmIU=">AAAChHicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQvIF2pUaSrYKsRk5pUoFMRXpySWJNZqVGgChWsqNFMq4gWUDfQMwEABk2EIZSgzQEFAvsByhhiGFIZ8hmSGUoZchlSGPIYSIDuHIZGhGAijGQwZDBgKgGKxDNVAsSIgKxMsn8pQy8AF1FsKVJUKVJEIFM0GkulAXjRUNA/IB5lZDNadDLQlB4iLgDoVGFQNrhqsNPhscMJgtcFLgz84zaoGmwFySyWQToLoTS2I5++SCP5OUFcukC5hyEDowuvmEoY0BguwWzOBbi8Ai4B8kQzRX1Y1/XOwVZBqtZrBIoPXQPcvNLhpcBjog7yyL8lLA1ODZjNwASPAED24MRlhRnqGBnqGgSbKDk7QqOBgkGZQYtAAhrc5gwODB0MAQyjQ3laG1QxbGLYysTHpMBkzmUKUMjFC9QgzoAAmOwDSDpSU</latexit><latexit sha1_base64="PEMDYUO5OQ+UOtLO81MQvQtmmIU=">AAAChHicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQvIF2pUaSrYKsRk5pUoFMRXpySWJNZqVGgChWsqNFMq4gWUDfQMwEABk2EIZSgzQEFAvsByhhiGFIZ8hmSGUoZchlSGPIYSIDuHIZGhGAijGQwZDBgKgGKxDNVAsSIgKxMsn8pQy8AF1FsKVJUKVJEIFM0GkulAXjRUNA/IB5lZDNadDLQlB4iLgDoVGFQNrhqsNPhscMJgtcFLgz84zaoGmwFySyWQToLoTS2I5++SCP5OUFcukC5hyEDowuvmEoY0BguwWzOBbi8Ai4B8kQzRX1Y1/XOwVZBqtZrBIoPXQPcvNLhpcBjog7yyL8lLA1ODZjNwASPAED24MRlhRnqGBnqGgSbKDk7QqOBgkGZQYtAAhrc5gwODB0MAQyjQ3laG1QxbGLYysTHpMBkzmUKUMjFC9QgzoAAmOwDSDpSU</latexit>
Total Correlation Penalty
VAE objective

8. FactorVAE : Total Correlation Penalty
n FactorVAE
n Total Correlation (TC):
n GAN z
density-ratio trick
q(z) =
Z
pdata(x)q(z|x)dx
<latexit sha1_base64="PEMDYUO5OQ+UOtLO81MQvQtmmIU=">AAAChHicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQvIF2pUaSrYKsRk5pUoFMRXpySWJNZqVGgChWsqNFMq4gWUDfQMwEABk2EIZSgzQEFAvsByhhiGFIZ8hmSGUoZchlSGPIYSIDuHIZGhGAijGQwZDBgKgGKxDNVAsSIgKxMsn8pQy8AF1FsKVJUKVJEIFM0GkulAXjRUNA/IB5lZDNadDLQlB4iLgDoVGFQNrhqsNPhscMJgtcFLgz84zaoGmwFySyWQToLoTS2I5++SCP5OUFcukC5hyEDowuvmEoY0BguwWzOBbi8Ai4B8kQzRX1Y1/XOwVZBqtZrBIoPXQPcvNLhpcBjog7yyL8lLA1ODZjNwASPAED24MRlhRnqGBnqGgSbKDk7QqOBgkGZQYtAAhrc5gwODB0MAQyjQ3laG1QxbGLYysTHpMBkzmUKUMjFC9QgzoAAmOwDSDpSU</latexit><latexit sha1_base64="PEMDYUO5OQ+UOtLO81MQvQtmmIU=">AAAChHicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQvIF2pUaSrYKsRk5pUoFMRXpySWJNZqVGgChWsqNFMq4gWUDfQMwEABk2EIZSgzQEFAvsByhhiGFIZ8hmSGUoZchlSGPIYSIDuHIZGhGAijGQwZDBgKgGKxDNVAsSIgKxMsn8pQy8AF1FsKVJUKVJEIFM0GkulAXjRUNA/IB5lZDNadDLQlB4iLgDoVGFQNrhqsNPhscMJgtcFLgz84zaoGmwFySyWQToLoTS2I5++SCP5OUFcukC5hyEDowuvmEoY0BguwWzOBbi8Ai4B8kQzRX1Y1/XOwVZBqtZrBIoPXQPcvNLhpcBjog7yyL8lLA1ODZjNwASPAED24MRlhRnqGBnqGgSbKDk7QqOBgkGZQYtAAhrc5gwODB0MAQyjQ3laG1QxbGLYysTHpMBkzmUKUMjFC9QgzoAAmOwDSDpSU</latexit><latexit sha1_base64="PEMDYUO5OQ+UOtLO81MQvQtmmIU=">AAAChHicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQvIF2pUaSrYKsRk5pUoFMRXpySWJNZqVGgChWsqNFMq4gWUDfQMwEABk2EIZSgzQEFAvsByhhiGFIZ8hmSGUoZchlSGPIYSIDuHIZGhGAijGQwZDBgKgGKxDNVAsSIgKxMsn8pQy8AF1FsKVJUKVJEIFM0GkulAXjRUNA/IB5lZDNadDLQlB4iLgDoVGFQNrhqsNPhscMJgtcFLgz84zaoGmwFySyWQToLoTS2I5++SCP5OUFcukC5hyEDowuvmEoY0BguwWzOBbi8Ai4B8kQzRX1Y1/XOwVZBqtZrBIoPXQPcvNLhpcBjog7yyL8lLA1ODZjNwASPAED24MRlhRnqGBnqGgSbKDk7QqOBgkGZQYtAAhrc5gwODB0MAQyjQ3laG1QxbGLYysTHpMBkzmUKUMjFC9QgzoAAmOwDSDpSU</latexit><latexit sha1_base64="PEMDYUO5OQ+UOtLO81MQvQtmmIU=">AAAChHicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQvIF2pUaSrYKsRk5pUoFMRXpySWJNZqVGgChWsqNFMq4gWUDfQMwEABk2EIZSgzQEFAvsByhhiGFIZ8hmSGUoZchlSGPIYSIDuHIZGhGAijGQwZDBgKgGKxDNVAsSIgKxMsn8pQy8AF1FsKVJUKVJEIFM0GkulAXjRUNA/IB5lZDNadDLQlB4iLgDoVGFQNrhqsNPhscMJgtcFLgz84zaoGmwFySyWQToLoTS2I5++SCP5OUFcukC5hyEDowuvmEoY0BguwWzOBbi8Ai4B8kQzRX1Y1/XOwVZBqtZrBIoPXQPcvNLhpcBjog7yyL8lLA1ODZjNwASPAED24MRlhRnqGBnqGgSbKDk7QqOBgkGZQYtAAhrc5gwODB0MAQyjQ3laG1QxbGLYysTHpMBkzmUKUMjFC9QgzoAAmOwDSDpSU</latexit>
z
GAN
z

9. FactorVAE : A New Metric for Disentanglement
n 1. k x L
n 2. x encode z
n 3. d k
Old metric
New metric
z
0

10. : β-VAE vs factorVAE, 2D Shapes
n β-VAE FactorVAE disentanglement metric
( )
n ( ) ( )
n
y
x
size
shape
shape entangle …

11. : InfoWGAN-GP, 2D Shapes
n Info-GAN + WGAN-GP
n
n infoGAN
( )
n (infoGAN
…( ))
InfoWGAN-GP β-VAE Factor-VAE
better

12. : β-VAE vs factorVAE, 3D Shapes
n disentangle factorVAE ( )
n shape scale disentangle ( )

13. : β-VAE vs factorVAE, 3D Chairs
βVAE …?
leg style ??

14. : β-VAE vs factorVAE, 3D Faces
βVAE …?
azimuth ??

15. : β-VAE vs factorVAE, CelebA
FactorVAE

16. n contribution
n 2D Shapes 3D Shapes factorVAE betaVAE
disentanglement scores
n β-VAE disentanglement metric metric
n failure mode
n VAE GAN
n limitation
n Total Correlation disentangling
n ex) q(z|x) p(z)=N(z|0,I) TC=0 x
n
n future work
n
n

17. n Adversarial Autoencoder
n infoGAN

18. : adversarial autoencoder
n
n AAE VAE
n ICLR 2016 workshop

19. : InfoGAN
n InfoGAN: Interpretable Representation Learning by Information Maximizing Generative
Adversarial Nets
n NIPS 2016
n
disentangle
n disentangle
n ex) mnist

20. :
n :
n I(X; Y) = H(X) - H(X|Y)
n H(X) H(X|Y) Y
X
n x, y
n KL
n p(x) p(x|y)
( )
n wikipedia

21. :
n : c
n :
n : I(c; G(c, z))
n GAN c c
n z
n disentangle …?
n L

22. :
n p(c|x)
Q(c|x)
n lemma A.1
P(c|x)
n Q D

23. : MNIST
n condition
n c ~ unif(-1, 1)
n 10
y ~ cat(10)
n D c c’
n https://github.com/znxlwm/pytorch-generative-model-
collections/blob/master/infoGAN.py
n
n c c’ disentangle
n c c’

24. : yz c ( ) yz c ( )
n s