Seeing What a GAN Cannot Generate [cdm]
•
0 likes•114 views
1. The document discusses research on understanding limitations in generative adversarial networks (GANs) by analyzing what types of images or image features GANs cannot generate. 2. Mode collapse is identified as a problem where GAN outputs are identical. The paper presents techniques for understanding omissions in the distribution and individual images generated by GANs. 3. By inverting a GAN to find the closest latent vector that generates a real image, differences between the real and generated images reveal what the GAN cannot capture. This helps analyze limitations in GANs and understand how they can be improved.
Recommended
Recommended
More Related Content
More from Dongmin Choi
More from Dongmin Choi (15)
Recently uploaded
Recently uploaded (20)
Seeing What a GAN Cannot Generate [cdm]
- 1. Yonsei University Severance Hospital CCIDS Seeing What a GAN Cannot Generate David Bau MIT http://ganseeing.csail.mit.edu//
- 2. Seeing What a GAN Cannot Generate Yonsei University Severance Hospital CCIDS https://software.intel.com/en-us/blogs/2017/08/21/mode-collapse-in-gans Mode Collapse in GAN is serious Problem ※ Mode Collapse : A problem when all the generator outputs are identical (all of them or most of the samples are equal)
- 3. Seeing What a GAN Cannot Generate Yonsei University Severance Hospital CCIDS http://ganseeing.csail.mit.edu//
- 4. Seeing What a GAN Cannot Generate Yonsei University Severance Hospital CCIDS Fake Images Real Images http://ganseeing.csail.mit.edu//
- 5. Seeing What a GAN Cannot Generate Yonsei University Severance Hospital CCIDS Fake Images Real Images Inception Inception Fake Inception Feature Space Real Inception Feature Space http://ganseeing.csail.mit.edu// FID (Frechet Inception Distance) Measuring GAN Quality
- 6. Seeing What a GAN Cannot Generate Yonsei University Severance Hospital CCIDS Fake Images Real Images http://ganseeing.csail.mit.edu// 1. What is actually missing in the distribution? 2. What is actually missing in each image?
- 7. Seeing What a GAN Cannot Generate Yonsei University Severance Hospital CCIDS http://ganseeing.csail.mit.edu// 1. Understanding Omissions in the Distribution Real Image Semantic segmentation
- 8. Seeing What a GAN Cannot Generate Yonsei University Severance Hospital CCIDS http://ganseeing.csail.mit.edu// 1. Understanding Omissions in the Distribution Generated Image Semantic segmentation Real Image Semantic segmentation
- 9. Seeing What a GAN Cannot Generate Yonsei University Severance Hospital CCIDS http://ganseeing.csail.mit.edu// 1. Understanding Omissions in the Distribution Generated Image Semantic segmentation Real Image Semantic segmentation
- 10. Seeing What a GAN Cannot Generate Yonsei University Severance Hospital CCIDS http://ganseeing.csail.mit.edu// 2. Understanding Omissions in Individual Images Synthesized Image G(z)
- 11. Seeing What a GAN Cannot Generate Yonsei University Severance Hospital CCIDS http://ganseeing.csail.mit.edu// 2. Understanding Omissions in Individual Images Real Image x Synthesized Image G(z)
- 12. Seeing What a GAN Cannot Generate Yonsei University Severance Hospital CCIDS http://ganseeing.csail.mit.edu// 2. Understanding Omissions in Individual Images Real Image x Synthesized Image G(z) Pairs (x, G(z*)) reveals omissions Objective : z* = argminz Loss(x, G(z))
- 13. Seeing What a GAN Cannot Generate Yonsei University Severance Hospital CCIDS http://ganseeing.csail.mit.edu// Three steps to layer-wise invert a large generator
- 14. Seeing What a GAN Cannot Generate Yonsei University Severance Hospital CCIDS http://ganseeing.csail.mit.edu// Three steps to layer-wise invert a large generator
- 15. Seeing What a GAN Cannot Generate Yonsei University Severance Hospital CCIDS http://ganseeing.csail.mit.edu// Three steps to layer-wise invert a large generator
- 16. Seeing What a GAN Cannot Generate Yonsei University Severance Hospital CCIDS http://ganseeing.csail.mit.edu// x = G(z) x = G(z*) x = real x = G(z*) Generated Reconstruction Real Photo Reconstruction When G generates x, reconstruction is precise When reconstruction is imperfect we know G cannot generate x
- 17. Seeing What a GAN Cannot Generate Yonsei University Severance Hospital CCIDS http://ganseeing.csail.mit.edu// GANs don’t like people
- 18. Seeing What a GAN Cannot Generate Yonsei University Severance Hospital CCIDS http://ganseeing.csail.mit.edu// The Cheese Hypothesis Original Image
- 19. Seeing What a GAN Cannot Generate Yonsei University Severance Hospital CCIDS http://ganseeing.csail.mit.edu// The Cheese Hypothesis Original Image Optimized z
- 20. Seeing What a GAN Cannot Generate Yonsei University Severance Hospital CCIDS http://ganseeing.csail.mit.edu// The Cheese Hypothesis Original Image Adapted Cheese
- 21. Seeing What a GAN Cannot Generate Yonsei University Severance Hospital CCIDS Real Image x
- 22. Seeing What a GAN Cannot Generate Yonsei University Severance Hospital CCIDS Optimized vector z* G Real Image x Reconstructed Image G(z*) z* = argminz Loss(x, G(z))
- 23. Seeing What a GAN Cannot Generate Yonsei University Severance Hospital CCIDS Optimized vector z* G Real Image x Reconstructed Image G(z*, θ*) θ* z*, θ* = argminz,θ Loss(x, G(z)) + R(θ) Regularizer Inspired by Deep Image Prior [Ulyanove et al, 2018]
- 24. CNN-generated images are surprisingly easy to spot... for now Yonsei University Severance Hospital CCIDS
- 25. CNN-generated images are surprisingly easy to spot... for now Yonsei University Severance Hospital CCIDS This paper shows that a classifier trained to detect images generated by only one CNN (ProGAN, far left) can detect those generated by many other models (remaining columns).
- 26. CNN-generated images are surprisingly easy to spot... for now Yonsei University Severance Hospital CCIDS
- 27. CNN-generated images are surprisingly easy to spot... for now Yonsei University Severance Hospital CCIDS Discussion - Suggest CNN-generated images have common artifacts - Artifacts can be detected by a simple classifier! - Situation may not persist
- 28. Thank You