Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Neural	
  Art	
  
	
  by	
  Mark	
  Chang	
  
A	
  Neural	
  Algorithm	
  of	
  Ar6s6c	
  Style	
  
•  Author:	
  
– Leon	
  A.	
  Gatys.	
  	
  
– Alexander	
  S.	
  E...
The	
  Mechanism	
  of	
  Pain6ng	
  
Brain	
  Ar6st	
  
Scene	
   Style	
   ArtWork	
  
Computer	
   Neural	
  Networks	
...
Overview	
  
•  Visual	
  Percep6on	
  
•  Computer	
  Vision	
  
•  Neural	
  Art	
  
•  Demo	
  
Visual	
  Percep6on	
  
•  Neuron	
  
•  Visual	
  Pathway	
  
•  Misconcep6on	
  
Neuron	
  
•  Neuron	
   •  Ac6on	
  Poten6al	
  
Dendrite	
  
Axon	
  
Cell	
  Body	
  
Time	
  
Voltage	
  
Threshold	
  
Visual	
  Pathway	
  
Re6na	
  
Visual	
  Area	
  V1	
  
Visual	
  Area	
  V4	
  
Inferior	
  
Temporal	
  Gyrus	
  
(IT)	...
Visual	
  Pathway	
  
Visual	
  Area	
  V1	
  
Inferior	
  
Temporal	
  
Gyrus	
  (IT)	
  
Recep6ve	
  Fields	
  
Visual	
...
Misconcep6on	
  
Computer	
  Vision	
  
•  Neural	
  Networks	
  	
  
•  Convolu6onal	
  Neural	
  Networks	
  
•  VGG	
  19	
  
Neural	
  Networks	
  	
  
n
W1
W2
x1
x2
b
 Wb
y
nin = w1x1 + w2x2 + wb
nout =
1
1 + e nin
Sigmoid	
  
Rec6fied	
  Linear	
...
Neural	
  Networks	
  	
  
x
y
n11
n12
n21
n22
b b
z1
z2
Input	
  	
  
Layer
Hidden	
  
Layer
Output	
  
Layer
W12,y
W12,x...
Convolu6onal	
  Neural	
  Networks	
  
•  Convolu6onal	
  Layer
depth	
  
width	
  width	
  	
  depth	
  
weights	
  weigh...
Convolu6onal	
  Neural	
  Networks	
  
•  Stride
 •  Padding
Stride	
  =	
  1	
  
Stride	
  =	
  2	
  
Padding	
  =	
  0	
...
Convolu6onal	
  Neural	
  Networks	
  
•  Pooling	
  Layer	
  
1
 3
 2
 4
5
 7
 6
 8
0
 0
 4
 4
6
 6
 0
 0
4
 5
3
 2
no	
 ...
Convolu6onal	
  Neural	
  Networks	
  
Convolu6onal	
  
Layer	
  
Convolu6onal	
  
Layer	
  	
   Pooling	
  
Layer	
  	
  ...
Convolu6onal	
  Neural	
  Networks	
  
Input	
  Layer	
  
Convolu6onal	
  
Layer	
  with	
  
Recep6ve	
  Fields:	
  
Max-­...
VGG	
  19	
  
•  Karen	
  Simonyan	
  &	
  Andrew	
  Zisserman.	
  Very	
  Deep	
  
Convolu6onal	
  Networks	
  for	
  Lar...
VGG	
  19	
  
depth=64	
  
conv1_1	
  
conv1_2	
  
maxpool
depth=128	
  
conv2_1	
  
conv2_2
maxpool
depth=256	
  
conv3_1...
Neural	
  Art	
  
•  Content	
  Genera6on	
  
•  Style	
  Genera6on	
  
•  Artwork	
  Genera6on	
  
Content	
  Genera6on	
  
Brain	
  Ar6st	
  Content	
  
Canvas	
  
Minimize	
  
the	
  
difference	
  
Neural	
  
S6mula6on	...
Content	
  Genera6on	
  
Filter	
  	
  
Responses	
  VGG19	
  
Update	
  the	
  
color	
  of	
  	
  
the	
  pixels	
  
Con...
Content	
  Genera6on	
  
Layer	
  l’s	
  Filter	
  l	
  
Responses:	
  
ent(Pl
, Cl
) =
1
2
X
i,j
(Cl
i,j Pl
i,j)2
Layer	
...
Content	
  Genera6on	
  
•  Backward	
  Propaga6on	
  
Layer	
  l’s	
  Filter	
  l	
  
Responses:	
  Xl
Input	
  
Canvas:	...
Content	
  Genera6on	
  
Content	
  Genera6on	
  
VGG19	
  
conv1_2	
   conv2_2	
   conv3_4	
   conv4_4	
   conv5_2	
  conv5_1	
  
Style	
  Genera6on	
  
•  ”Style”	
  is	
  posi6on-­‐independent	
  
style	
  
extrac6on	
  
Style	
  Genera6on	
  
VGG19	
  Artwork	
  
G	
  
G	
  
Filter	
  Responses	
   Gram	
  Matrix	
  
Width*Height	
  
Depth	...
Style	
  Genera6on	
  
1.
 .5
.5
.5
1.
1.	
   .5
 .25
 1.
.5
 .25
 .5
.25
 .25
1.
 .5
 1.
Width*Height	
  
Depth	
  
k1	
 ...
Style	
  Genera6on	
  
VGG19	
  
Filter	
  
	
  Responses	
  
Gram	
  	
  
Matrix	
  
Minimize	
  
the	
  
difference	
  
G...
Style	
  Genera6on	
  
Lstyle(a, x, l) =
1
2
X
i,j
(Xl
i,j Al
i,j)2
@Lstyle(a, x, l)
@Fl
i,j
= ((Fl
)T
(Xl
Al
))j,i
Layer	...
Style	
  Genera6on	
  
Style	
  Genera6on	
  
VGG19	
  
Conv1_1	
   Conv1_1	
  
Conv2_1	
  
Conv1_1	
  
Conv2_1	
  
Conv3_1	
  
	
  
Conv1_1	
  
...
Artwork	
  Genera6on	
  
Ltotal = ↵Lcontent + Lstyle
a
ent(p, c, l) =
1
2
X
i,j
(Cl
i,j Pl
i,j)2
x
x x ⌘
@Ltotal
@x
x
Filt...
Artwork	
  Genera6on	
  
VGG19	
   VGG19	
  
Lcontent(p, x) Lstyle(a, x)
Conv1_1	
  
Conv2_1	
  
Conv3_1	
  
Conv4_1	
  
C...
Artwork	
  Genera6on	
  
Demo	
  
•  Content	
  v.s.	
  Style	
  
•  Different	
  Ini6al	
  State	
  
•  Different	
  VGG	
  Layers	
  
•  Sketch	
  ...
Content	
  v.s.	
  Style	
  
0.15	
   0.05	
  
0.02	
   0.007	
  
↵
Different	
  Ini6al	
  State	
  
noise	
   0.9	
  *noise	
  +	
  0.1*photo	
   photo	
  
Different	
  VGG	
  Layers	
  
Conv1_1	
  
Conv2_1	
  
Conv1_1	
  
Conv2_1	
  
Conv3_1	
  
Conv1_1	
  
Conv2_1	
  
Conv3_1	...
Sketch	
  &	
  Watercolor	
  
Pain6ng	
  &	
  Poem	
  
Further	
  Reading	
  
•  A	
  Neural	
  Algorithm	
  of	
  Ar6s6c	
  Style	
  
–  h@p://arxiv.org/abs/1508.06576	
  
•  T...
Code
•  Python	
  Tensorflow	
  
– h@ps://github.com/ckmarkoh/neuralart_tensorflow	
  
•  Python	
  Theano	
  
– h@ps://gith...
Image	
  URL	
  
•  h@p://
www.taipei-­‐101.com.tw/
upload/news/
201502/2015021711505431
705145.JPG	
  	
  
	
  
•  h@ps:/...
Acknowledgement	
  
•  NTU	
  imlab	
  
About	
  the	
  Speaker	
  
•  Email:	
  ckmarkoh	
  at	
  gmail	
  dot	
  com	
  
•  Blog:	
  h@p://cpmarkchang.logdown.c...
You’ve finished this document.
Download and read it offline.
Upcoming SlideShare
Neural Doodle
Next
Upcoming SlideShare
Neural Doodle
Next
Download to read offline and view in fullscreen.

Share

Neural Art (English Version)

Download to read offline

slides for MLDM Monday NeuralArt
English Version
https://www.youtube.com/watch?v=qzGuYuCpy1M

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Neural Art (English Version)

  1. 1. Neural  Art    by  Mark  Chang  
  2. 2. A  Neural  Algorithm  of  Ar6s6c  Style   •  Author:   – Leon  A.  Gatys.     – Alexander  S.  Ecker.     – Ma@hias  Bethge     •  Organiza6on:   – Werner  Reichardt  Centre  for  Integra6ve  Neuroscience   and  Ins6tute  of  Theore6cal  Physics,  University  of   Tubingen,  Germany.       – Bernstein  Center  for  Computa6onal  Neuroscience,   Tubingen,  Germany.  
  3. 3. The  Mechanism  of  Pain6ng   Brain  Ar6st   Scene   Style   ArtWork   Computer   Neural  Networks  
  4. 4. Overview   •  Visual  Percep6on   •  Computer  Vision   •  Neural  Art   •  Demo  
  5. 5. Visual  Percep6on   •  Neuron   •  Visual  Pathway   •  Misconcep6on  
  6. 6. Neuron   •  Neuron   •  Ac6on  Poten6al   Dendrite   Axon   Cell  Body   Time   Voltage   Threshold  
  7. 7. Visual  Pathway   Re6na   Visual  Area  V1   Visual  Area  V4   Inferior   Temporal  Gyrus   (IT)  
  8. 8. Visual  Pathway   Visual  Area  V1   Inferior   Temporal   Gyrus  (IT)   Recep6ve  Fields   Visual  Area  V4  
  9. 9. Misconcep6on  
  10. 10. Computer  Vision   •  Neural  Networks     •  Convolu6onal  Neural  Networks   •  VGG  19  
  11. 11. Neural  Networks     n W1 W2 x1 x2 b Wb y nin = w1x1 + w2x2 + wb nout = 1 1 + e nin Sigmoid   Rec6fied  Linear   nout = ⇢ nin if nin > 0 0 otherwise
  12. 12. Neural  Networks     x y n11 n12 n21 n22 b b z1 z2 Input     Layer Hidden   Layer Output   Layer W12,y W12,x W11,y W11,b W12,b W11,x W21,11 W22,12 W21,12 W22,11 W21,b W22,b
  13. 13. Convolu6onal  Neural  Networks   •  Convolu6onal  Layer depth   width  width    depth   weights  weights   height   shared  weight  
  14. 14. Convolu6onal  Neural  Networks   •  Stride •  Padding Stride  =  1   Stride  =  2   Padding  =  0   Padding  =  1  
  15. 15. Convolu6onal  Neural  Networks   •  Pooling  Layer   1 3 2 4 5 7 6 8 0 0 4 4 6 6 0 0 4 5 3 2 no  overlap   no  padding   no  weights   depth  =  1   7 8 6 4 Maximum   Pooling   Average   Pooling  
  16. 16. Convolu6onal  Neural  Networks   Convolu6onal   Layer   Convolu6onal   Layer     Pooling   Layer     Pooling   Layer     Recep6ve  Fields   Recep6ve  Fields   Input   Layer  
  17. 17. Convolu6onal  Neural  Networks   Input  Layer   Convolu6onal   Layer  with   Recep6ve  Fields:   Max-­‐pooling   Layer  with   Width  =3,  Height  =  3   Filter  Responses   Filter  Responses   Input  Image  
  18. 18. VGG  19   •  Karen  Simonyan  &  Andrew  Zisserman.  Very  Deep   Convolu6onal  Networks  for  Large-­‐scale  Image   Recogni6on.   •  ImageNet  Challenge  2014   •  19  (+5)  layers   – 16  Convolu6onal  layers  (width=3,  height=3)   – 5  Max-­‐pooling  layers  (width=2,  height=2)   – 3  Fully-­‐connected  layers  
  19. 19. VGG  19   depth=64   conv1_1   conv1_2   maxpool depth=128   conv2_1   conv2_2 maxpool depth=256   conv3_1   conv3_2   conv3_3   conv3_4 depth=512   conv4_1   conv4_2   conv4_3   conv4_4 depth=512   conv5_1   conv5_2   conv5_3   conv5_4 maxpool maxpool maxpool size=4096   FC1   FC2   size=1000   sogmax
  20. 20. Neural  Art   •  Content  Genera6on   •  Style  Genera6on   •  Artwork  Genera6on  
  21. 21. Content  Genera6on   Brain  Ar6st  Content   Canvas   Minimize   the   difference   Neural   S6mula6on   Draw  
  22. 22. Content  Genera6on   Filter     Responses  VGG19   Update  the   color  of     the  pixels   Content   Canvas   Result   Width*Height   Depth   Minimize   the   difference  
  23. 23. Content  Genera6on   Layer  l’s  Filter  l   Responses:   ent(Pl , Cl ) = 1 2 X i,j (Cl i,j Pl i,j)2 Layer  l’s  Filter     Responses:  Lcontent(P l , Cl ) = 1 2 X i,j (Cl i,j P l i,j)2 Input   Photo:  Lcontent(p, c, l) = 1 2 X i,j (Cl i,j Pl i,j)2 Lcontent(p, x, l) = 1 2 X i,j (Xl i,j Pl i,j)2 @Lcontent(p, x, l) @Xl i,j = Xl i,j Pl i,j Xl Xl i,j Input   Canvas:  x Width*Height  (j)   Depth  (i)   Width*Height  (j)   Depth  (i)  
  24. 24. Content  Genera6on   •  Backward  Propaga6on   Layer  l’s  Filter  l   Responses:  Xl Input   Canvas:   x VGG19   @Lcontent @x = @Lcontent @Xl @Xl x x x ⌘ @Lcontent @x Update   Canvas   Learning  Rate  
  25. 25. Content  Genera6on  
  26. 26. Content  Genera6on   VGG19   conv1_2   conv2_2   conv3_4   conv4_4   conv5_2  conv5_1  
  27. 27. Style  Genera6on   •  ”Style”  is  posi6on-­‐independent   style   extrac6on  
  28. 28. Style  Genera6on   VGG19  Artwork   G   G   Filter  Responses   Gram  Matrix   Width*Height   Depth   Depth   Depth   Posi6on-­‐   dependent   Posi6on-­‐   independent  
  29. 29. Style  Genera6on   1. .5 .5 .5 1. 1.   .5 .25 1. .5 .25 .5 .25 .25 1. .5 1. Width*Height   Depth   k1   k2   k1   k2   Depth   Depth   Layer  l’s  Filter  Responses   Gram  Matrix   Fl 1 Fl 2 Fl 3 Fl 4 Fl 1 Fl 2 Fl 3 Fl 4 G         Gl i,j = Fl i · Fl j Gl 4,1 = Fl 4 · Fl 1 = 1 ⇥ 1 + 0 ⇥ 0.5 + 0 ⇥ 0 + ... = 1
  30. 30. Style  Genera6on   VGG19   Filter    Responses   Gram     Matrix   Minimize   the   difference   G   G   Style   Canvas   Update  the  color  of     the  pixels  Result  
  31. 31. Style  Genera6on   Lstyle(a, x, l) = 1 2 X i,j (Xl i,j Al i,j)2 @Lstyle(a, x, l) @Fl i,j = ((Fl )T (Xl Al ))j,i Layer  l’s     Filter  Responses   Layer  l’s     Gram  Matrix   Layer  l’s     Gram  Matrix   Fl i,j Al i,j Xl i,j Input   Artwork:   Input   Canvas:  a x
  32. 32. Style  Genera6on  
  33. 33. Style  Genera6on   VGG19   Conv1_1   Conv1_1   Conv2_1   Conv1_1   Conv2_1   Conv3_1     Conv1_1   Conv2_1   Conv3_1   Conv4_1   Conv1_1   Conv2_1   Conv3_1   Conv4_1   Conv5_1  
  34. 34. Artwork  Genera6on   Ltotal = ↵Lcontent + Lstyle a ent(p, c, l) = 1 2 X i,j (Cl i,j Pl i,j)2 x x x ⌘ @Ltotal @x x Filter  Responses  VGG19   Lcontent(p, x) Lstyle(a, x) Gram  Matrix  
  35. 35. Artwork  Genera6on   VGG19   VGG19   Lcontent(p, x) Lstyle(a, x) Conv1_1   Conv2_1   Conv3_1   Conv4_1   Conv5_1   Conv4_2   Ltotal = ↵Lcontent + Lstyle
  36. 36. Artwork  Genera6on  
  37. 37. Demo   •  Content  v.s.  Style   •  Different  Ini6al  State   •  Different  VGG  Layers   •  Sketch  &  Watercolor   •  Pain6ng  &  Poem  
  38. 38. Content  v.s.  Style   0.15   0.05   0.02   0.007   ↵
  39. 39. Different  Ini6al  State   noise   0.9  *noise  +  0.1*photo   photo  
  40. 40. Different  VGG  Layers   Conv1_1   Conv2_1   Conv1_1   Conv2_1   Conv3_1   Conv1_1   Conv2_1   Conv3_1   Conv4_1   Conv1_1   Conv2_1   Conv3_1   Conv4_1   Conv5_1   ↵ = 0.002
  41. 41. Sketch  &  Watercolor  
  42. 42. Pain6ng  &  Poem  
  43. 43. Further  Reading   •  A  Neural  Algorithm  of  Ar6s6c  Style   –  h@p://arxiv.org/abs/1508.06576   •  Texture  Synthesis  Using  Convolu6onal  Neural  Networks   –  h@p://arxiv.org/abs/1505.07376   •  Convolu6onal  Neural  Network   –  h@p://cs231n.github.io/convolu6onal-­‐networks/   •  Neural  Network  Back  Propaga6on   –  h@p://cpmarkchang.logdown.com/posts/277349-­‐neural-­‐ network-­‐backward-­‐propaga6on   •  Computa6onal  Poetry:   –  h@p://www.slideshare.net/ckmarkohchang/computa6onal-­‐ poetry  
  44. 44. Code •  Python  Tensorflow   – h@ps://github.com/ckmarkoh/neuralart_tensorflow   •  Python  Theano   – h@ps://github.com/woonketwong/ar6fy   •  Python  Theano  (ipython  notebook)   – h@ps://github.com/Lasagne/Recipes/blob/master/ examples/styletransfer/Art%20Style %20Transfer.ipynb   •  Python  deeppy   – h@ps://github.com/andersbll/neural_ar6s6c_style  
  45. 45. Image  URL   •  h@p:// www.taipei-­‐101.com.tw/ upload/news/ 201502/2015021711505431 705145.JPG       •  h@ps://github.com/ andersbll/ neural_ar6s6c_style/blob/ master/images/ starry_night.jpg?raw=true  
  46. 46. Acknowledgement   •  NTU  imlab  
  47. 47. About  the  Speaker   •  Email:  ckmarkoh  at  gmail  dot  com   •  Blog:  h@p://cpmarkchang.logdown.com   •  Github:  h@ps://github.com/ckmarkoh   Mark  Chang   •  Facebook:  h@ps://www.facebook.com/ckmarkoh.chang   •  Slideshare:  h@p://www.slideshare.net/ckmarkohchang   •  Linkedin:  h@ps://www.linkedin.com/pub/mark-­‐chang/85/25b/847  
  • IsabellHerdl

    May. 16, 2021
  • DukeYang1

    Jun. 16, 2018
  • slowcampus

    Apr. 11, 2018
  • MarcellCsiszr

    Apr. 4, 2018
  • JunoBae

    Jul. 5, 2017
  • KeatonKim

    Jun. 20, 2017
  • damianborth

    Feb. 5, 2017
  • LizziePerez1

    Oct. 19, 2016
  • TseJenChen

    Jul. 10, 2016
  • AquaMac

    Jul. 10, 2016
  • rightson

    Jun. 4, 2016
  • jihoBak

    May. 11, 2016
  • kmirmir1

    May. 11, 2016
  • ChengTaoChung

    Apr. 21, 2016
  • sasbluesea

    Apr. 21, 2016

slides for MLDM Monday NeuralArt English Version https://www.youtube.com/watch?v=qzGuYuCpy1M

Views

Total views

4,773

On Slideshare

0

From embeds

0

Number of embeds

127

Actions

Downloads

272

Shares

0

Comments

0

Likes

15

×