The document discusses variational divergence minimization for training generative neural networks using f-GAN. It introduces f-divergence as a generalization of divergence measures used in GANs like KL divergence. F-divergence allows the training of generative models by minimizing the divergence between the generated distribution and real data distribution. The paper presents an algorithm for minimizing f-divergence in generative neural samplers.

1. 双対表現と深層学習

2. n
n ex1) GAN f-GAN
n ex2) NN

3. : f-GAN: Training Generative Neural Samplers
using Variational Divergence Minimization
n NIPS 2016 accepted
n : Sebastian Nowozin, Botond Cseke, Ryota Tomioka (Microsoft Research)
n : GAN f-divergence f-
divergence
f-divergence

4. : f-divergence
n f: R+ → R
n f ( )
n f(1) = 0
n ex)
n KL div: f(u) = u log u
n reverse KL div: f(u) = - log u
n
n x f(x) > -∞ f(x) < ∞ x
n
n :

5. :
n f f = f**
n ( )
u 2 @f⇤
(t) , t 2 @f(u) , uT
t = f(u) + f⇤
(t)<latexit sha1_base64="Yfx3yWt+OV18kbpcAr9IFTOxvH8=">AAACx3ichVHPaxNBFP6yaq1Rm9ReBC+LoZK2EN4WoSIIRS8KHvorbaFpw+x2kg7d7C6zs7E19OC1/0APnhRExD+jl/4DPfRPEMFLhfTQgy+bRWmD7Vt25pvvfd+bNzNu5KvYEJ3krBs3bw3dHr6Tv3vv/kihOPpgOQ4T7cmqF/qhXnVFLH0VyKpRxperkZai5fpyxd1+1cuvtKWOVRgsmd1IrrdEM1AN5QnDVL3oJnZNBXYtEtoo4duNjc7kXtlM2LW3smG0am4ZoXX4zjaXdOVkUJNsLBn7RT839bdUvViiCqVhDwInAyVkMRcWv6KGTYTwkKAFiQCGsQ+BmL81OCBEzK2jw5xmpNK8xB7y7E1YJVkhmN3mscmrtYwNeN2rGaduj3fx+dfstDFOx/SNTumIvtMPOv9vrU5ao9fLLs9u3yujemH/4WL3WleLZ4Otf64rezZo4Fnaq+Leo5TpncLr+9vvD04Xny+Md57QZ/rJ/X+iEzrkEwTt396XebnwEXl+AOfydQ+C5emKQxVn/mlp9mX2FMN4hMco833PYBavMYcq73uEX+jizHpjhVbb2ulLrVzmGcOFsD78Absmri4=</latexit><latexit sha1_base64="Yfx3yWt+OV18kbpcAr9IFTOxvH8=">AAACx3ichVHPaxNBFP6yaq1Rm9ReBC+LoZK2EN4WoSIIRS8KHvorbaFpw+x2kg7d7C6zs7E19OC1/0APnhRExD+jl/4DPfRPEMFLhfTQgy+bRWmD7Vt25pvvfd+bNzNu5KvYEJ3krBs3bw3dHr6Tv3vv/kihOPpgOQ4T7cmqF/qhXnVFLH0VyKpRxperkZai5fpyxd1+1cuvtKWOVRgsmd1IrrdEM1AN5QnDVL3oJnZNBXYtEtoo4duNjc7kXtlM2LW3smG0am4ZoXX4zjaXdOVkUJNsLBn7RT839bdUvViiCqVhDwInAyVkMRcWv6KGTYTwkKAFiQCGsQ+BmL81OCBEzK2jw5xmpNK8xB7y7E1YJVkhmN3mscmrtYwNeN2rGaduj3fx+dfstDFOx/SNTumIvtMPOv9vrU5ao9fLLs9u3yujemH/4WL3WleLZ4Otf64rezZo4Fnaq+Leo5TpncLr+9vvD04Xny+Md57QZ/rJ/X+iEzrkEwTt396XebnwEXl+AOfydQ+C5emKQxVn/mlp9mX2FMN4hMco833PYBavMYcq73uEX+jizHpjhVbb2ulLrVzmGcOFsD78Absmri4=</latexit><latexit sha1_base64="Yfx3yWt+OV18kbpcAr9IFTOxvH8=">AAACx3ichVHPaxNBFP6yaq1Rm9ReBC+LoZK2EN4WoSIIRS8KHvorbaFpw+x2kg7d7C6zs7E19OC1/0APnhRExD+jl/4DPfRPEMFLhfTQgy+bRWmD7Vt25pvvfd+bNzNu5KvYEJ3krBs3bw3dHr6Tv3vv/kihOPpgOQ4T7cmqF/qhXnVFLH0VyKpRxperkZai5fpyxd1+1cuvtKWOVRgsmd1IrrdEM1AN5QnDVL3oJnZNBXYtEtoo4duNjc7kXtlM2LW3smG0am4ZoXX4zjaXdOVkUJNsLBn7RT839bdUvViiCqVhDwInAyVkMRcWv6KGTYTwkKAFiQCGsQ+BmL81OCBEzK2jw5xmpNK8xB7y7E1YJVkhmN3mscmrtYwNeN2rGaduj3fx+dfstDFOx/SNTumIvtMPOv9vrU5ao9fLLs9u3yujemH/4WL3WleLZ4Otf64rezZo4Fnaq+Leo5TpncLr+9vvD04Xny+Md57QZ/rJ/X+iEzrkEwTt396XebnwEXl+AOfydQ+C5emKQxVn/mlp9mX2FMN4hMco833PYBavMYcq73uEX+jizHpjhVbb2ulLrVzmGcOFsD78Absmri4=</latexit><latexit sha1_base64="Yfx3yWt+OV18kbpcAr9IFTOxvH8=">AAACx3ichVHPaxNBFP6yaq1Rm9ReBC+LoZK2EN4WoSIIRS8KHvorbaFpw+x2kg7d7C6zs7E19OC1/0APnhRExD+jl/4DPfRPEMFLhfTQgy+bRWmD7Vt25pvvfd+bNzNu5KvYEJ3krBs3bw3dHr6Tv3vv/kihOPpgOQ4T7cmqF/qhXnVFLH0VyKpRxperkZai5fpyxd1+1cuvtKWOVRgsmd1IrrdEM1AN5QnDVL3oJnZNBXYtEtoo4duNjc7kXtlM2LW3smG0am4ZoXX4zjaXdOVkUJNsLBn7RT839bdUvViiCqVhDwInAyVkMRcWv6KGTYTwkKAFiQCGsQ+BmL81OCBEzK2jw5xmpNK8xB7y7E1YJVkhmN3mscmrtYwNeN2rGaduj3fx+dfstDFOx/SNTumIvtMPOv9vrU5ao9fLLs9u3yujemH/4WL3WleLZ4Otf64rezZo4Fnaq+Leo5TpncLr+9vvD04Xny+Md57QZ/rJ/X+iEzrkEwTt396XebnwEXl+AOfydQ+C5emKQxVn/mlp9mX2FMN4hMco833PYBavMYcq73uEX+jizHpjhVbb2ulLrVzmGcOFsD78Absmri4=</latexit>

6. : f-divergence
n (u=p(x)/q(x))
n T: Χ→R
n T* tight
n sup t t
n convex duality
sup
t T(x)
q0/p0 2 @f⇤
(t) , t 2 @f(q0/p0)<latexit sha1_base64="mMvB036JDUyl9itr2xUtSyARzbA=">AAACrHichVFNTxNBGH5YELEKrXgx8bKhwRQO5V1jovFE9KAHD3wVSNqymV1my4Tt7jo7rcGGP+Af8OBJE2OM4Vdw4W48kMAPIB4h4eLBt9tNDBD0nczMM8+8zzvPzHhJqFJDdDhkDY/cGL05dqtw+874RLF0d3I1jTvalzU/DmO97olUhiqSNaNMKNcTLUXbC+Wat/2iv7/WlTpVcbRidhLZbItWpALlC8OUW3r5xqW5xCW7oSK7kQhtlAjtYKM3u1sxM3bjtQyMVq0tI7SO39rmUl4ll8+4pTJVKQv7KnByUEYeC3HpKxrYRAwfHbQhEcEwDiGQcqvDASFhrokec5qRyvYldlFgbYezJGcIZrd5bPGqnrMRr/s100zt8ykhd81KG9P0k77RKR3Qdzqh39fW6mU1+l52ePYGWpm4xff3l8//q2rzbLD1V/VPzwYBnmZeFXtPMqZ/C3+g7777cLr8bGm695A+0y/2/4kOaZ9vEHXP/C+LcukjCvwBzuXnvgpWH1UdqjqLj8vzz/OvGMMDTKHC7/0E83iFBdT43D38wBGOraq1YtWt5iDVGso193AhrOAPyM+jyA==</latexit><latexit sha1_base64="mMvB036JDUyl9itr2xUtSyARzbA=">AAACrHichVFNTxNBGH5YELEKrXgx8bKhwRQO5V1jovFE9KAHD3wVSNqymV1my4Tt7jo7rcGGP+Af8OBJE2OM4Vdw4W48kMAPIB4h4eLBt9tNDBD0nczMM8+8zzvPzHhJqFJDdDhkDY/cGL05dqtw+874RLF0d3I1jTvalzU/DmO97olUhiqSNaNMKNcTLUXbC+Wat/2iv7/WlTpVcbRidhLZbItWpALlC8OUW3r5xqW5xCW7oSK7kQhtlAjtYKM3u1sxM3bjtQyMVq0tI7SO39rmUl4ll8+4pTJVKQv7KnByUEYeC3HpKxrYRAwfHbQhEcEwDiGQcqvDASFhrokec5qRyvYldlFgbYezJGcIZrd5bPGqnrMRr/s100zt8ykhd81KG9P0k77RKR3Qdzqh39fW6mU1+l52ePYGWpm4xff3l8//q2rzbLD1V/VPzwYBnmZeFXtPMqZ/C3+g7777cLr8bGm695A+0y/2/4kOaZ9vEHXP/C+LcukjCvwBzuXnvgpWH1UdqjqLj8vzz/OvGMMDTKHC7/0E83iFBdT43D38wBGOraq1YtWt5iDVGso193AhrOAPyM+jyA==</latexit><latexit sha1_base64="mMvB036JDUyl9itr2xUtSyARzbA=">AAACrHichVFNTxNBGH5YELEKrXgx8bKhwRQO5V1jovFE9KAHD3wVSNqymV1my4Tt7jo7rcGGP+Af8OBJE2OM4Vdw4W48kMAPIB4h4eLBt9tNDBD0nczMM8+8zzvPzHhJqFJDdDhkDY/cGL05dqtw+874RLF0d3I1jTvalzU/DmO97olUhiqSNaNMKNcTLUXbC+Wat/2iv7/WlTpVcbRidhLZbItWpALlC8OUW3r5xqW5xCW7oSK7kQhtlAjtYKM3u1sxM3bjtQyMVq0tI7SO39rmUl4ll8+4pTJVKQv7KnByUEYeC3HpKxrYRAwfHbQhEcEwDiGQcqvDASFhrokec5qRyvYldlFgbYezJGcIZrd5bPGqnrMRr/s100zt8ykhd81KG9P0k77RKR3Qdzqh39fW6mU1+l52ePYGWpm4xff3l8//q2rzbLD1V/VPzwYBnmZeFXtPMqZ/C3+g7777cLr8bGm695A+0y/2/4kOaZ9vEHXP/C+LcukjCvwBzuXnvgpWH1UdqjqLj8vzz/OvGMMDTKHC7/0E83iFBdT43D38wBGOraq1YtWt5iDVGso193AhrOAPyM+jyA==</latexit><latexit sha1_base64="mMvB036JDUyl9itr2xUtSyARzbA=">AAACrHichVFNTxNBGH5YELEKrXgx8bKhwRQO5V1jovFE9KAHD3wVSNqymV1my4Tt7jo7rcGGP+Af8OBJE2OM4Vdw4W48kMAPIB4h4eLBt9tNDBD0nczMM8+8zzvPzHhJqFJDdDhkDY/cGL05dqtw+874RLF0d3I1jTvalzU/DmO97olUhiqSNaNMKNcTLUXbC+Wat/2iv7/WlTpVcbRidhLZbItWpALlC8OUW3r5xqW5xCW7oSK7kQhtlAjtYKM3u1sxM3bjtQyMVq0tI7SO39rmUl4ll8+4pTJVKQv7KnByUEYeC3HpKxrYRAwfHbQhEcEwDiGQcqvDASFhrokec5qRyvYldlFgbYezJGcIZrd5bPGqnrMRr/s100zt8ykhd81KG9P0k77RKR3Qdzqh39fW6mU1+l52ePYGWpm4xff3l8//q2rzbLD1V/VPzwYBnmZeFXtPMqZ/C3+g7777cLr8bGm695A+0y/2/4kOaZ9vEHXP/C+LcukjCvwBzuXnvgpWH1UdqjqLj8vzz/OvGMMDTKHC7/0E83iFBdT43D38wBGOraq1YtWt5iDVGso193AhrOAPyM+jyA==</latexit>

7. : f-divergence NN
n Q_θ generar T_ω ( f-GAN)
n f-divergence F θ ω
n GAN T(x) = log D(x)
sup

8. 1:
n : P Q
n D_f F

9. 2: (MNIST)
n KDE ( )
n (KDE) p(x)
n GAN (Theis, 2016)
KL
reverseKL
Hellinger
JS

10. 2: (LSUN)
(2016) GAN

11. : Mutual Information Neural Estimation
n ICML 2018 accepted
n : Yoshua Bengio
n :
n
n infoGAN
n Information Bottleneck
n ICA
n …

12. : KL
n
n P_XZ
n X Z 0
n
n KL

13. : KL
n 1: Donsker-Varadhan representation(Donsker & Varadhan, 1983)
n sup
n 2: f-div ( )
n KL f f*(t) = exp(t-1)
n 1 log bound tight

14. 1:
n MINE: DV
n MINE-f: f-div
n Kraskov: kNN
202
MINE True MI

15. 2:
n f
n σ
n : σ
MINE
f(x)

16. : DV
log
Gap
>= 0
(23)

17. :
n
n ex) φ(x) = |x| x=0
n t=0 φ(0) = 0 |s| >= zs -1<= z <= 1
-1 1
φ(x) = |x|