Image generative modeling for design inspiration and image editing by Camille Couprie, Research Scientist @Facebook AI
•
1 like•509 views
Abstract: Generative models, and in particular adversarial ones, are becoming prevalent in computer vision as they enable enhancing artistic creation, inspire designers, prove usefulness in semi-supervised learning or robotics applications. We will see how to develop the abilities of Generative Adversarial Networks (GANs) to deviate from training examples to generate more original images of fashion designs. As a limitation of GANs is the production of raw images of low resolution, we also present solutions to produce vectorized results, and show how the developed method may be useful for image editing.
Recommended
Recommended
More Related Content
Similar to Image generative modeling for design inspiration and image editing by Camille Couprie, Research Scientist @Facebook AI
Similar to Image generative modeling for design inspiration and image editing by Camille Couprie, Research Scientist @Facebook AI (20)
More from Paris Women in Machine Learning and Data Science
More from Paris Women in Machine Learning and Data Science (20)
Recently uploaded
Recently uploaded (20)
Image generative modeling for design inspiration and image editing by Camille Couprie, Research Scientist @Facebook AI
- 1. Generative modeling for design inspiration and image editing Camille Couprie Facebook AI Research Joint works with O. Sbai, M. Aubry, M. Riviere, A. Bordes, M. Elhoseiny, Y. LeCun 2019 1
- 2. 2 Menlo park Seattle New York Pittsburg Montreal Boston London Paris Tel Aviv Facebook AI Research
- 4. 4 Research areas Core machine learning Theory and optimization Reinforcement learning Artificial general intelligence / cognitive science Computer Vision Speech processing Natural Language processing Grounding, Interaction, Communication Robotics System research
- 5. Why do we care about generative models 5 • Scene Understanding can be assessed by checking the ability to generate plausible new scenes. • Generative models are interesting if they can be used to go beyond training data: data of higher resolution, data augmentation to help train better classifiers, use the learned representations in other tasks, make prediction about uncertain events, or producing original data content that can be inspiring...
- 6. • 1/ Design inspiration from adversarial generative networks • 2/ Vector image generation by learning parametric layer decomposition Outline 6
- 7. 7 Sbai, Elhoseiny, Bordes, LeCun, Couprie, ECCV workshop 17 1/ Design inspiration from generative networks Novelty Hedonic Value 1 2 30 4
- 8. Training with pictures of about 2000 Clothing items
- 9. Floral StripedTiled Uniform Dotted Animal Print Graphical Texture and shape labels DressSkirt JacketPullover T-Shirt Coat Top
- 10. RANDOM NUMBERS Real INPUT Generator AdVERSARIAL NETWORK GENERATED IMAGE 0 . 3 0 . 7 0 . 1 0 . 8 S h a p e C L A S S 0 / 1 ( R E A L / FA K E ) T E X T U R E C L A S S Class conditioned GAN
- 11. Without conditioning With class conditioning
- 12. RANDOM NUMBERS Generator 0 . 3 0 . 7 0 . 1 0 . 8 AdVERSARIAL NETWORK Generated Image Dotted Floral graphical uniform tiled striped Animal print 1 0 0 % Introduction of a Style Deviation criterion
- 13. 8 % 6 % 7 % 5 7 % 6 % 1 1 % 9 % RANDOM NUMBERS Generator 0 . 3 0 . 7 0 . 1 0 . 8 AdVERSARIAL NETWORK Generated Image Dotted Floral graphical uniform tiled striped Animal print Introduction of a Style Deviation criterion
- 14. RANDOM NUMBERS Generator 0 . 3 0 . 7 0 . 1 0 . 8 AdVERSARIAL NETWORK Generated Image D o t t e d F l o r a l g r a p h i c a l u n i f o r m t i l e d s t r i p e d A n i m a l p r i n t With the Style Deviation criterion (CAN H)
- 15. O v e r a l L L i k a b i l i t y ( % ) 6 5 7 0 7 56 0 8 0 GAN: DCGAN [Radford et al.] CAN: GAN with "Creativity" loss [Elgammal et al.], CAN (H) stands for the use of our "holistic" loss. CAN texTURE GAN Mask CAN(h) texTURE CAN(h) texTUREr e a l i s t i c A p p e a r a n c e 4 8 5 3 . 5 5 9 6 4 . 5 7 0 Human Evaluation Study
- 16. Models with texture deviation are Most Popular N o v e l t y0 1 L i k e a b i l i t y 6 2 6 6 7 0 7 4 7 8 judged by humans and measured as a distance to similar training images Can texture Can (H) Shape Can (H) texture mask can (h) GAN
- 18. 1024x1024 generations on the RTW dataset 2 0 Using Morgane Riviere’s pytorch code “progressive growing of GANs”, Karras et al., ICLR’18
- 19. 2 1 Sbai, Couprie, Aubry, arxiv dec18 3/ Vector Image Generation by learning parametric layer decomposition Current deep generative models are great but… … are limited in resolution, and control in generations
- 20. Kanan et al: Layered GANs (LR-GANs), ICLR’17 GANIN et al. SPIRAL, ICML’18 Related work 2 2
- 21. Our approach 2 3
- 22. Iterative generation : 𝐼𝑡 = 𝑔(𝐼, 𝐼𝑡−1) Vectorized mask generation 𝑀𝑡(𝑥, 𝑦) = 𝑔(𝑥, 𝑦, 𝑝𝑡) Alpha-blending 𝐼𝑡(𝑥, 𝑦) = 𝐼𝑡−1 𝑥, 𝑦 . (1-𝑀𝑡(𝑥, 𝑦))+ 𝑐𝑡 𝑀𝑡(𝑥, 𝑦) Our iterative pipeline for image reconstruction 2 4 x x ⇥ ⇥ + ct ct Mt Mask Blending Module
- 23. 2 5 Our iterative pipeline for image generation
- 25. Results using a perceptual loss 2 8
- 27. Image vectorization: Reconstruction results on MNIST images. Our model learns a vectorized mask representation of digits that can be generated at any resolution without interpolation artifacts. Vector output 3 0
- 28. Baselines 3 1
- 30. CelebA generations trained on 64x64 images, sampled at 256x256 GAN results 3 3 CIFAR10 generations trained on 32x32 images, sampled at 256x256
- 31. GAN Results on ImageNet 3 4 trained on 64x64 images, sampled at 1024x1024
- 32. Conclusions 3 5 Some open problems: - Automatic metrics to evaluate generative models performances - How to improve GANs resolution? Torch code online : - Pytorch GAN zoo - Vector image generation: available soon on Othman Sbai’s github