Neural Art Generation Using Convolutional Neural Networks
•
15 likes•4,940 views
slides for MLDM Monday NeuralArt English Version https://www.youtube.com/watch?v=qzGuYuCpy1M
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Neural Art Generation Using Convolutional Neural Networks
Similar to Neural Art Generation Using Convolutional Neural Networks (20)
More from Mark Chang
More from Mark Chang (9)
Recently uploaded
Recently uploaded (20)
Neural Art Generation Using Convolutional Neural Networks
- 1. Neural Art by Mark Chang
- 2. A Neural Algorithm of Ar6s6c Style • Author： – Leon A. Gatys. – Alexander S. Ecker. – Ma@hias Bethge • Organiza6on： – Werner Reichardt Centre for Integra6ve Neuroscience and Ins6tute of Theore6cal Physics, University of Tubingen, Germany. – Bernstein Center for Computa6onal Neuroscience, Tubingen, Germany.
- 3. The Mechanism of Pain6ng Brain Ar6st Scene Style ArtWork Computer Neural Networks
- 4. Overview • Visual Percep6on • Computer Vision • Neural Art • Demo
- 5. Visual Percep6on • Neuron • Visual Pathway • Misconcep6on
- 6. Neuron • Neuron • Ac6on Poten6al Dendrite Axon Cell Body Time Voltage Threshold
- 7. Visual Pathway Re6na Visual Area V1 Visual Area V4 Inferior Temporal Gyrus (IT)
- 8. Visual Pathway Visual Area V1 Inferior Temporal Gyrus (IT) Recep6ve Fields Visual Area V4
- 9. Misconcep6on
- 10. Computer Vision • Neural Networks • Convolu6onal Neural Networks • VGG 19
- 11. Neural Networks n W1 W2 x1 x2 b Wb y nin = w1x1 + w2x2 + wb nout = 1 1 + e nin Sigmoid Rec6fied Linear nout = ⇢ nin if nin > 0 0 otherwise
- 12. Neural Networks x y n11 n12 n21 n22 b b z1 z2 Input Layer Hidden Layer Output Layer W12,y W12,x W11,y W11,b W12,b W11,x W21,11 W22,12 W21,12 W22,11 W21,b W22,b
- 13. Convolu6onal Neural Networks • Convolu6onal Layer depth width width depth weights weights height shared weight
- 14. Convolu6onal Neural Networks • Stride • Padding Stride = 1 Stride = 2 Padding = 0 Padding = 1
- 15. Convolu6onal Neural Networks • Pooling Layer 1 3 2 4 5 7 6 8 0 0 4 4 6 6 0 0 4 5 3 2 no overlap no padding no weights depth = 1 7 8 6 4 Maximum Pooling Average Pooling
- 16. Convolu6onal Neural Networks Convolu6onal Layer Convolu6onal Layer Pooling Layer Pooling Layer Recep6ve Fields Recep6ve Fields Input Layer
- 17. Convolu6onal Neural Networks Input Layer Convolu6onal Layer with Recep6ve Fields: Max-‐pooling Layer with Width =3, Height = 3 Filter Responses Filter Responses Input Image
- 18. VGG 19 • Karen Simonyan & Andrew Zisserman. Very Deep Convolu6onal Networks for Large-‐scale Image Recogni6on. • ImageNet Challenge 2014 • 19 (+5) layers – 16 Convolu6onal layers (width=3, height=3) – 5 Max-‐pooling layers (width=2, height=2) – 3 Fully-‐connected layers
- 19. VGG 19 depth=64 conv1_1 conv1_2 maxpool depth=128 conv2_1 conv2_2 maxpool depth=256 conv3_1 conv3_2 conv3_3 conv3_4 depth=512 conv4_1 conv4_2 conv4_3 conv4_4 depth=512 conv5_1 conv5_2 conv5_3 conv5_4 maxpool maxpool maxpool size=4096 FC1 FC2 size=1000 sogmax
- 20. Neural Art • Content Genera6on • Style Genera6on • Artwork Genera6on
- 21. Content Genera6on Brain Ar6st Content Canvas Minimize the difference Neural S6mula6on Draw
- 22. Content Genera6on Filter Responses VGG19 Update the color of the pixels Content Canvas Result Width*Height Depth Minimize the difference
- 23. Content Genera6on Layer l’s Filter l Responses: ent(Pl , Cl ) = 1 2 X i,j (Cl i,j Pl i,j)2 Layer l’s Filter Responses: Lcontent(P l , Cl ) = 1 2 X i,j (Cl i,j P l i,j)2 Input Photo: Lcontent(p, c, l) = 1 2 X i,j (Cl i,j Pl i,j)2 Lcontent(p, x, l) = 1 2 X i,j (Xl i,j Pl i,j)2 @Lcontent(p, x, l) @Xl i,j = Xl i,j Pl i,j Xl Xl i,j Input Canvas: x Width*Height (j) Depth (i) Width*Height (j) Depth (i)
- 24. Content Genera6on • Backward Propaga6on Layer l’s Filter l Responses: Xl Input Canvas: x VGG19 @Lcontent @x = @Lcontent @Xl @Xl x x x ⌘ @Lcontent @x Update Canvas Learning Rate
- 26. Content Genera6on VGG19 conv1_2 conv2_2 conv3_4 conv4_4 conv5_2 conv5_1
- 27. Style Genera6on • ”Style” is posi6on-‐independent style extrac6on
- 28. Style Genera6on VGG19 Artwork G G Filter Responses Gram Matrix Width*Height Depth Depth Depth Posi6on-‐ dependent Posi6on-‐ independent
- 29. Style Genera6on 1. .5 .5 .5 1. 1. .5 .25 1. .5 .25 .5 .25 .25 1. .5 1. Width*Height Depth k1 k2 k1 k2 Depth Depth Layer l’s Filter Responses Gram Matrix Fl 1 Fl 2 Fl 3 Fl 4 Fl 1 Fl 2 Fl 3 Fl 4 G Gl i,j = Fl i · Fl j Gl 4,1 = Fl 4 · Fl 1 = 1 ⇥ 1 + 0 ⇥ 0.5 + 0 ⇥ 0 + ... = 1
- 30. Style Genera6on VGG19 Filter Responses Gram Matrix Minimize the difference G G Style Canvas Update the color of the pixels Result
- 31. Style Genera6on Lstyle(a, x, l) = 1 2 X i,j (Xl i,j Al i,j)2 @Lstyle(a, x, l) @Fl i,j = ((Fl )T (Xl Al ))j,i Layer l’s Filter Responses Layer l’s Gram Matrix Layer l’s Gram Matrix Fl i,j Al i,j Xl i,j Input Artwork: Input Canvas: a x
- 32. Style Genera6on
- 33. Style Genera6on VGG19 Conv1_1 Conv1_1 Conv2_1 Conv1_1 Conv2_1 Conv3_1 Conv1_1 Conv2_1 Conv3_1 Conv4_1 Conv1_1 Conv2_1 Conv3_1 Conv4_1 Conv5_1
- 34. Artwork Genera6on Ltotal = ↵Lcontent + Lstyle a ent(p, c, l) = 1 2 X i,j (Cl i,j Pl i,j)2 x x x ⌘ @Ltotal @x x Filter Responses VGG19 Lcontent(p, x) Lstyle(a, x) Gram Matrix
- 35. Artwork Genera6on VGG19 VGG19 Lcontent(p, x) Lstyle(a, x) Conv1_1 Conv2_1 Conv3_1 Conv4_1 Conv5_1 Conv4_2 Ltotal = ↵Lcontent + Lstyle
- 37. Demo • Content v.s. Style • Different Ini6al State • Different VGG Layers • Sketch & Watercolor • Pain6ng & Poem
- 38. Content v.s. Style 0.15 0.05 0.02 0.007 ↵
- 39. Different Ini6al State noise 0.9 *noise + 0.1*photo photo
- 40. Different VGG Layers Conv1_1 Conv2_1 Conv1_1 Conv2_1 Conv3_1 Conv1_1 Conv2_1 Conv3_1 Conv4_1 Conv1_1 Conv2_1 Conv3_1 Conv4_1 Conv5_1 ↵ = 0.002
- 42. Pain6ng & Poem
- 43. Further Reading • A Neural Algorithm of Ar6s6c Style – h@p://arxiv.org/abs/1508.06576 • Texture Synthesis Using Convolu6onal Neural Networks – h@p://arxiv.org/abs/1505.07376 • Convolu6onal Neural Network – h@p://cs231n.github.io/convolu6onal-‐networks/ • Neural Network Back Propaga6on – h@p://cpmarkchang.logdown.com/posts/277349-‐neural-‐ network-‐backward-‐propaga6on • Computa6onal Poetry： – h@p://www.slideshare.net/ckmarkohchang/computa6onal-‐ poetry
- 44. Code • Python Tensorflow – h@ps://github.com/ckmarkoh/neuralart_tensorflow • Python Theano – h@ps://github.com/woonketwong/ar6fy • Python Theano (ipython notebook) – h@ps://github.com/Lasagne/Recipes/blob/master/ examples/styletransfer/Art%20Style %20Transfer.ipynb • Python deeppy – h@ps://github.com/andersbll/neural_ar6s6c_style
- 45. Image URL • h@p:// www.taipei-‐101.com.tw/ upload/news/ 201502/2015021711505431 705145.JPG • h@ps://github.com/ andersbll/ neural_ar6s6c_style/blob/ master/images/ starry_night.jpg?raw=true
- 46. Acknowledgement • NTU imlab
- 47. About the Speaker • Email: ckmarkoh at gmail dot com • Blog: h@p://cpmarkchang.logdown.com • Github: h@ps://github.com/ckmarkoh Mark Chang • Facebook: h@ps://www.facebook.com/ckmarkoh.chang • Slideshare: h@p://www.slideshare.net/ckmarkohchang • Linkedin: h@ps://www.linkedin.com/pub/mark-‐chang/85/25b/847