Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Neural Art (English Version)

4,027 views

Published on

slides for MLDM Monday NeuralArt
English Version
https://www.youtube.com/watch?v=qzGuYuCpy1M

Published in: Technology

Neural Art (English Version)

  1. 1. Neural  Art    by  Mark  Chang  
  2. 2. A  Neural  Algorithm  of  Ar6s6c  Style   •  Author:   – Leon  A.  Gatys.     – Alexander  S.  Ecker.     – Ma@hias  Bethge     •  Organiza6on:   – Werner  Reichardt  Centre  for  Integra6ve  Neuroscience   and  Ins6tute  of  Theore6cal  Physics,  University  of   Tubingen,  Germany.       – Bernstein  Center  for  Computa6onal  Neuroscience,   Tubingen,  Germany.  
  3. 3. The  Mechanism  of  Pain6ng   Brain  Ar6st   Scene   Style   ArtWork   Computer   Neural  Networks  
  4. 4. Overview   •  Visual  Percep6on   •  Computer  Vision   •  Neural  Art   •  Demo  
  5. 5. Visual  Percep6on   •  Neuron   •  Visual  Pathway   •  Misconcep6on  
  6. 6. Neuron   •  Neuron   •  Ac6on  Poten6al   Dendrite   Axon   Cell  Body   Time   Voltage   Threshold  
  7. 7. Visual  Pathway   Re6na   Visual  Area  V1   Visual  Area  V4   Inferior   Temporal  Gyrus   (IT)  
  8. 8. Visual  Pathway   Visual  Area  V1   Inferior   Temporal   Gyrus  (IT)   Recep6ve  Fields   Visual  Area  V4  
  9. 9. Misconcep6on  
  10. 10. Computer  Vision   •  Neural  Networks     •  Convolu6onal  Neural  Networks   •  VGG  19  
  11. 11. Neural  Networks     n W1 W2 x1 x2 b Wb y nin = w1x1 + w2x2 + wb nout = 1 1 + e nin Sigmoid   Rec6fied  Linear   nout = ⇢ nin if nin > 0 0 otherwise
  12. 12. Neural  Networks     x y n11 n12 n21 n22 b b z1 z2 Input     Layer Hidden   Layer Output   Layer W12,y W12,x W11,y W11,b W12,b W11,x W21,11 W22,12 W21,12 W22,11 W21,b W22,b
  13. 13. Convolu6onal  Neural  Networks   •  Convolu6onal  Layer depth   width  width    depth   weights  weights   height   shared  weight  
  14. 14. Convolu6onal  Neural  Networks   •  Stride •  Padding Stride  =  1   Stride  =  2   Padding  =  0   Padding  =  1  
  15. 15. Convolu6onal  Neural  Networks   •  Pooling  Layer   1 3 2 4 5 7 6 8 0 0 4 4 6 6 0 0 4 5 3 2 no  overlap   no  padding   no  weights   depth  =  1   7 8 6 4 Maximum   Pooling   Average   Pooling  
  16. 16. Convolu6onal  Neural  Networks   Convolu6onal   Layer   Convolu6onal   Layer     Pooling   Layer     Pooling   Layer     Recep6ve  Fields   Recep6ve  Fields   Input   Layer  
  17. 17. Convolu6onal  Neural  Networks   Input  Layer   Convolu6onal   Layer  with   Recep6ve  Fields:   Max-­‐pooling   Layer  with   Width  =3,  Height  =  3   Filter  Responses   Filter  Responses   Input  Image  
  18. 18. VGG  19   •  Karen  Simonyan  &  Andrew  Zisserman.  Very  Deep   Convolu6onal  Networks  for  Large-­‐scale  Image   Recogni6on.   •  ImageNet  Challenge  2014   •  19  (+5)  layers   – 16  Convolu6onal  layers  (width=3,  height=3)   – 5  Max-­‐pooling  layers  (width=2,  height=2)   – 3  Fully-­‐connected  layers  
  19. 19. VGG  19   depth=64   conv1_1   conv1_2   maxpool depth=128   conv2_1   conv2_2 maxpool depth=256   conv3_1   conv3_2   conv3_3   conv3_4 depth=512   conv4_1   conv4_2   conv4_3   conv4_4 depth=512   conv5_1   conv5_2   conv5_3   conv5_4 maxpool maxpool maxpool size=4096   FC1   FC2   size=1000   sogmax
  20. 20. Neural  Art   •  Content  Genera6on   •  Style  Genera6on   •  Artwork  Genera6on  
  21. 21. Content  Genera6on   Brain  Ar6st  Content   Canvas   Minimize   the   difference   Neural   S6mula6on   Draw  
  22. 22. Content  Genera6on   Filter     Responses  VGG19   Update  the   color  of     the  pixels   Content   Canvas   Result   Width*Height   Depth   Minimize   the   difference  
  23. 23. Content  Genera6on   Layer  l’s  Filter  l   Responses:   ent(Pl , Cl ) = 1 2 X i,j (Cl i,j Pl i,j)2 Layer  l’s  Filter     Responses:  Lcontent(P l , Cl ) = 1 2 X i,j (Cl i,j P l i,j)2 Input   Photo:  Lcontent(p, c, l) = 1 2 X i,j (Cl i,j Pl i,j)2 Lcontent(p, x, l) = 1 2 X i,j (Xl i,j Pl i,j)2 @Lcontent(p, x, l) @Xl i,j = Xl i,j Pl i,j Xl Xl i,j Input   Canvas:  x Width*Height  (j)   Depth  (i)   Width*Height  (j)   Depth  (i)  
  24. 24. Content  Genera6on   •  Backward  Propaga6on   Layer  l’s  Filter  l   Responses:  Xl Input   Canvas:   x VGG19   @Lcontent @x = @Lcontent @Xl @Xl x x x ⌘ @Lcontent @x Update   Canvas   Learning  Rate  
  25. 25. Content  Genera6on  
  26. 26. Content  Genera6on   VGG19   conv1_2   conv2_2   conv3_4   conv4_4   conv5_2  conv5_1  
  27. 27. Style  Genera6on   •  ”Style”  is  posi6on-­‐independent   style   extrac6on  
  28. 28. Style  Genera6on   VGG19  Artwork   G   G   Filter  Responses   Gram  Matrix   Width*Height   Depth   Depth   Depth   Posi6on-­‐   dependent   Posi6on-­‐   independent  
  29. 29. Style  Genera6on   1. .5 .5 .5 1. 1.   .5 .25 1. .5 .25 .5 .25 .25 1. .5 1. Width*Height   Depth   k1   k2   k1   k2   Depth   Depth   Layer  l’s  Filter  Responses   Gram  Matrix   Fl 1 Fl 2 Fl 3 Fl 4 Fl 1 Fl 2 Fl 3 Fl 4 G         Gl i,j = Fl i · Fl j Gl 4,1 = Fl 4 · Fl 1 = 1 ⇥ 1 + 0 ⇥ 0.5 + 0 ⇥ 0 + ... = 1
  30. 30. Style  Genera6on   VGG19   Filter    Responses   Gram     Matrix   Minimize   the   difference   G   G   Style   Canvas   Update  the  color  of     the  pixels  Result  
  31. 31. Style  Genera6on   Lstyle(a, x, l) = 1 2 X i,j (Xl i,j Al i,j)2 @Lstyle(a, x, l) @Fl i,j = ((Fl )T (Xl Al ))j,i Layer  l’s     Filter  Responses   Layer  l’s     Gram  Matrix   Layer  l’s     Gram  Matrix   Fl i,j Al i,j Xl i,j Input   Artwork:   Input   Canvas:  a x
  32. 32. Style  Genera6on  
  33. 33. Style  Genera6on   VGG19   Conv1_1   Conv1_1   Conv2_1   Conv1_1   Conv2_1   Conv3_1     Conv1_1   Conv2_1   Conv3_1   Conv4_1   Conv1_1   Conv2_1   Conv3_1   Conv4_1   Conv5_1  
  34. 34. Artwork  Genera6on   Ltotal = ↵Lcontent + Lstyle a ent(p, c, l) = 1 2 X i,j (Cl i,j Pl i,j)2 x x x ⌘ @Ltotal @x x Filter  Responses  VGG19   Lcontent(p, x) Lstyle(a, x) Gram  Matrix  
  35. 35. Artwork  Genera6on   VGG19   VGG19   Lcontent(p, x) Lstyle(a, x) Conv1_1   Conv2_1   Conv3_1   Conv4_1   Conv5_1   Conv4_2   Ltotal = ↵Lcontent + Lstyle
  36. 36. Artwork  Genera6on  
  37. 37. Demo   •  Content  v.s.  Style   •  Different  Ini6al  State   •  Different  VGG  Layers   •  Sketch  &  Watercolor   •  Pain6ng  &  Poem  
  38. 38. Content  v.s.  Style   0.15   0.05   0.02   0.007   ↵
  39. 39. Different  Ini6al  State   noise   0.9  *noise  +  0.1*photo   photo  
  40. 40. Different  VGG  Layers   Conv1_1   Conv2_1   Conv1_1   Conv2_1   Conv3_1   Conv1_1   Conv2_1   Conv3_1   Conv4_1   Conv1_1   Conv2_1   Conv3_1   Conv4_1   Conv5_1   ↵ = 0.002
  41. 41. Sketch  &  Watercolor  
  42. 42. Pain6ng  &  Poem  
  43. 43. Further  Reading   •  A  Neural  Algorithm  of  Ar6s6c  Style   –  h@p://arxiv.org/abs/1508.06576   •  Texture  Synthesis  Using  Convolu6onal  Neural  Networks   –  h@p://arxiv.org/abs/1505.07376   •  Convolu6onal  Neural  Network   –  h@p://cs231n.github.io/convolu6onal-­‐networks/   •  Neural  Network  Back  Propaga6on   –  h@p://cpmarkchang.logdown.com/posts/277349-­‐neural-­‐ network-­‐backward-­‐propaga6on   •  Computa6onal  Poetry:   –  h@p://www.slideshare.net/ckmarkohchang/computa6onal-­‐ poetry  
  44. 44. Code •  Python  Tensorflow   – h@ps://github.com/ckmarkoh/neuralart_tensorflow   •  Python  Theano   – h@ps://github.com/woonketwong/ar6fy   •  Python  Theano  (ipython  notebook)   – h@ps://github.com/Lasagne/Recipes/blob/master/ examples/styletransfer/Art%20Style %20Transfer.ipynb   •  Python  deeppy   – h@ps://github.com/andersbll/neural_ar6s6c_style  
  45. 45. Image  URL   •  h@p:// www.taipei-­‐101.com.tw/ upload/news/ 201502/2015021711505431 705145.JPG       •  h@ps://github.com/ andersbll/ neural_ar6s6c_style/blob/ master/images/ starry_night.jpg?raw=true  
  46. 46. Acknowledgement   •  NTU  imlab  
  47. 47. About  the  Speaker   •  Email:  ckmarkoh  at  gmail  dot  com   •  Blog:  h@p://cpmarkchang.logdown.com   •  Github:  h@ps://github.com/ckmarkoh   Mark  Chang   •  Facebook:  h@ps://www.facebook.com/ckmarkoh.chang   •  Slideshare:  h@p://www.slideshare.net/ckmarkohchang   •  Linkedin:  h@ps://www.linkedin.com/pub/mark-­‐chang/85/25b/847  

×