Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Intel® AI: Non-Parametric Priors for Generative Adversarial Networks

58 views

Published on

This presentation proposes a novel prior which is derived using basic theorems from probability theory and off-the-shelf optimizers, to improve fidelity of image generation using GANs by interpolating along any Euclidean straight line without any additional training and architecture modifications

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Intel® AI: Non-Parametric Priors for Generative Adversarial Networks

  1. 1. Rajhans Singh, Pavan Turaga, Suren Jayasuriya, Ravi Garg, Martin Braun In collaboration with:
  2. 2. 2 © Intel Corporation Intel, the Intel logo, Intel Inside, the Intel Inside logo, Intel Atom, Intel Core, Iris, Movidius, Myriad, Intel Nervana, OpenVINO, Intel Optane, Stratix, and Xeon are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. GAN Overview • GANs trained on images can generate images by mapping a point in low-dimensional latent space to a point in high dimensional image space • Low dimensional latent spaces are also known as prior distributions or priors • GANs has shown applications in super resolution, image-to- image translation, etc. Discriminator loss Generator loss
  3. 3. 3 © Intel Corporation Intel, the Intel logo, Intel Inside, the Intel Inside logo, Intel Atom, Intel Core, Iris, Movidius, Myriad, Intel Nervana, OpenVINO, Intel Optane, Stratix, and Xeon are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. Why Priors are Important? • Priors choice can help in generating diverse and realistic output using interpolation in latent spaces – Interpolation can help transfer certain semantic feature of image to another [Radford et al.] – Successful interpolation also shows that GANs don’t overfit and reproduce training data • Distribution Mismatch – Simple parametric latent space distributions like Normal and Uniform suffer from distribution mismatch issue in interpolation – The prior distribution does not match the interpolated point’s distribution Radford, A., Metz, L., and Chintala, S. Unsupervised representation learning with deep convolutional generative adversarial networks. In International Conference on Learning Representations (ICLR), 2016
  4. 4. 4 © Intel Corporation Intel, the Intel logo, Intel Inside, the Intel Inside logo, Intel Atom, Intel Core, Iris, Movidius, Myriad, Intel Nervana, OpenVINO, Intel Optane, Stratix, and Xeon are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. Distribution Mismatch Issue • The GANs trained on the prior points to generate realistic images, when used to interpolate between points, often loose fidelity in image quality due to distribution mismatch ‘Ghosty’ faces near the origin during interpolation from left to right on CelebA dataset using Normal prior distribution Bad fidelity due to distribution mismatch at the center • The problem worsens as we use higher dimensional latent spaces - mid point and prior distributions distance increases
  5. 5. 5 © Intel Corporation Intel, the Intel logo, Intel Inside, the Intel Inside logo, Intel Atom, Intel Core, Iris, Movidius, Myriad, Intel Nervana, OpenVINO, Intel Optane, Stratix, and Xeon are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. Addressing Distribution Mismatch Issue Designing better interpolation schemes that traverse through high probability mass density regions of the distribution • Geodesic paths [White et al.]; Traversed paths can be complicated and long By utilizing latent spaces such that the distribution mismatch is minimum, e.g., Gamma [Kilcher et al.] and Cauchy distribution[Lesniak et al.] • Cauchy distribution has undefined moments and heavy tails; may lead to undesirable outputs, training difficulties We propose a novel non-parametric prior that generates more realistic and diversified images using interpolated points in the latent space White, T., “Sampling generative networks” https://arxiv.org/vc/arxiv/papers/1609/1609.04468, 2016 Kilcher, Y., Lucchi, A., and Hofmann, T., “Semantic interpolation in implicit models”, in International Conference on Learning Representations (ICLR), 2018. Lesniak, D., Sieradzki, I., and Podolak, I., ”Distribution-interpolation trade off in generative models”, in International Conference on Learning Representations (ICLR), 2019
  6. 6. 6 © Intel Corporation Intel, the Intel logo, Intel Inside, the Intel Inside logo, Intel Atom, Intel Core, Iris, Movidius, Myriad, Intel Nervana, OpenVINO, Intel Optane, Stratix, and Xeon are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. Proposed Non-parametric Prior • If 𝑥𝑥1, 𝑥𝑥2~𝑓𝑓𝑋𝑋(𝑥𝑥), then the pdf of the interpolated point: 1 − 𝜆𝜆 𝑥𝑥1 + 𝜆𝜆𝜆𝜆2, for some 𝜆𝜆 𝜖𝜖 0,1 , is given by 𝑄𝑄 𝑥𝑥; 𝜆𝜆 = 1 𝜆𝜆(1 − 𝜆𝜆) 𝑓𝑓𝑋𝑋 𝑥𝑥 𝜆𝜆 ∗ 𝑓𝑓𝑋𝑋 𝑥𝑥 1 − 𝜆𝜆 • The goal is to minimize the KL divergence between 𝑃𝑃 𝑥𝑥 = 𝑓𝑓𝑋𝑋(𝑥𝑥) and 𝑄𝑄 𝑥𝑥; 𝜆𝜆 − 𝑃𝑃(𝑥𝑥) is restricted a compact domain, i.e. [0,1] and discretize into 210 bins to obtain a tractable solution - Variance constraint is added to avoid delta function as solution min 𝑃𝑃 𝑓𝑓(𝑃𝑃||𝑄𝑄) 𝑠𝑠. 𝑡𝑡. ∑𝑖𝑖=1 𝑛𝑛 𝑝𝑝𝑖𝑖 = 1, 1 𝑛𝑛 ∑𝑖𝑖=1 𝑛𝑛 𝑖𝑖2 𝑝𝑝𝑖𝑖 − ∑𝑖𝑖=1 𝑛𝑛 𝑖𝑖𝑖𝑖𝑖𝑖 2 ≥ 𝜀𝜀, 𝑝𝑝𝑖𝑖 ≥ 0
  7. 7. 7 © Intel Corporation Intel, the Intel logo, Intel Inside, the Intel Inside logo, Intel Atom, Intel Core, Iris, Movidius, Myriad, Intel Nervana, OpenVINO, Intel Optane, Stratix, and Xeon are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. Proposed Non-parametric Prior • Symmetric - Did not enforce as a constraint; came out as a solution • Combine two properties from other priors for interpolation - Sharp decaying – similar to Gamma distribution - Side lobes – may represent heavy tail structure of Cauchy Density of the prior and its midpoint distribution similar Distribution KL divergence Uniform .3065 Normal .1544 Proposed .0075
  8. 8. 8 © Intel Corporation Intel, the Intel logo, Intel Inside, the Intel Inside logo, Intel Atom, Intel Core, Iris, Movidius, Myriad, Intel Nervana, OpenVINO, Intel Optane, Stratix, and Xeon are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. GAN Generated Images Interpolation (left to right) through the origin on CelebA dataset  DC-GAN architecture [Radford et. al.’ 2016] – CelebA dataset Origin
  9. 9. 9 © Intel Corporation Intel, the Intel logo, Intel Inside, the Intel Inside logo, Intel Atom, Intel Core, Iris, Movidius, Myriad, Intel Nervana, OpenVINO, Intel Optane, Stratix, and Xeon are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. Measured using standard metrics used for GAN quality • Inception Score, FID score Quantitative Results Distribution CelebA LSUN Bedroom Inception Score FID Score Inception Score FID Score Prior Mid- Point Prior Mid- Point Prior Mid- Point Prior Mid- Point Uniform 1.843 1.369 24.055 40.371 2.969 2.649 42.998 76.412 Normal 1.805 1.371 26.173 42.136 2.812 2.591 64.682 108.49 Gamma 1.776 1.618 29.912 28.608 2.930 2.808 162.44 161.37 Cauchy 1.625 1.628 59.601 60.128 3.148 3.149 97.057 97.109 Non- parametric 1.933 1.681 17.735 19.115 3.028 2.769 27.857 31.472 Best performance on CelebA dataset among all priors tested Best or close to the best, while more consistent results at prior point and mid- point
  10. 10. 10 © Intel Corporation Intel, the Intel logo, Intel Inside, the Intel Inside logo, Intel Atom, Intel Core, Iris, Movidius, Myriad, Intel Nervana, OpenVINO, Intel Optane, Stratix, and Xeon are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. Conclusions •We derived a non-parametric approach to search for a prior which addresses the distribution mismatch problem •The proposed prior distribution provides better qualitative and quantitative results as compared to the standard priors such as Normal and Uniform distributions •We showed that our proposed prior yields better quality and diversity in generated output, without any additional training data and added architectural complexity
  11. 11. 11 © Intel Corporation Intel, the Intel logo, Intel Inside, the Intel Inside logo, Intel Atom, Intel Core, Iris, Movidius, Myriad, Intel Nervana, OpenVINO, Intel Optane, Stratix, and Xeon are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. Thank You
  12. 12. 12 © Intel Corporation Intel, the Intel logo, Intel Inside, the Intel Inside logo, Intel Atom, Intel Core, Iris, Movidius, Myriad, Intel Nervana, OpenVINO, Intel Optane, Stratix, and Xeon are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. Non-Parametric Priors For Generative Adversarial Networks Rajhans Singh, Pavan Turaga, Suren Jayasuriya, Ravi Garg, Martin Braun
  13. 13. 13 © Intel Corporation Intel, the Intel logo, Intel Inside, the Intel Inside logo, Intel Atom, Intel Core, Iris, Movidius, Myriad, Intel Nervana, OpenVINO, Intel Optane, Stratix, and Xeon are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. Backup
  14. 14. 14 © Intel Corporation Intel, the Intel logo, Intel Inside, the Intel Inside logo, Intel Atom, Intel Core, Iris, Movidius, Myriad, Intel Nervana, OpenVINO, Intel Optane, Stratix, and Xeon are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. Effect of prior dimensionality
  15. 15. 15 © Intel Corporation Intel, the Intel logo, Intel Inside, the Intel Inside logo, Intel Atom, Intel Core, Iris, Movidius, Myriad, Intel Nervana, OpenVINO, Intel Optane, Stratix, and Xeon are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. Euclidean Norm Distribution Euclidean norm distribution for normal prior Euclidean norm distribution for the proposed prior d  overlap in Norm- distribution Preserves overlap at higher dimensions

×