Study Neural Style Transfer with TensorFlow 2.0 @GDG Songdo DL Study

1. Neural Style Transfer
Olivia Seji OH
June 8, 2019

2. By Olivia Seji OH
What I am going to do here...
∙ I would like to study and discuss what neural style transfer is
and how to do programming it using with TensorFlow 2.0.
CONTENTS
⟜ The reason I chose this topic
⟜ About the paper
⟜ The code
⟜ Conclusion
⟜ References
2

3. By Olivia Seji OH
The reason I chose this topic is...
Originally, I was introduced to the technique with this tutorial by
a study friend.
Reference: Android & TensorFlow: Artistic Style Transfer @ Google Codelabs
https://codelabs.developers.google.com/codelabs/tensorflow-style-transfer-android/index.html#0
3

4. By Olivia Seji OH
But I do know nothing about
making an Android app!!!
(But I do not need to give it up here,
and, of course, I will not:)
4

5. By Olivia Seji OH
So, let’s move on
and
find an another way
to learn about It. 5

6. By Olivia Seji OH
Instead of making an android app...
I found a tutorial about Neural
style transfer at the
TensorFlow official webpage.
Here is the link:
https://www.tensorflow.org/alph
a/tutorials/generative/style_tr
ansfer
The Result:
Beautiful mixture of images
You can also try this here:
https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/r2/tutorials/generative/style_transfer.ipynb
6

7. By Olivia Seji OH
Hold on!
Before we dive into the codes,
let’s see what is inside the logic!
7
(At first, we are going to study the paper.)

8. By Olivia Seji OH
A Neural Algorithm of Artistic Style
By Leon A. Gatys, Alexander S. Ecker, Matthias Bethge
(2015: https://arxiv.org/abs/1508.06576)
8

9. By Olivia Seji OH
The paper introduces...
An artificial system
based on a
Deep Neural Network (DNN)
9
Artistic images
of
high perceptual quality
creates
Especially, Convolutional Neural Networks (CNN)
∙ are trained an object recognition.
∙ develop a representation of the image that makes object information
increasingly explicit along the processing hierarchy.

10. By Olivia Seji OH
The paper introduces...
10
Convolutional Neural
Networks (CNN)
∙ The number of different
filters increases along
the processing
hierarchy.
∙ The size of the filtered
images is reduced by
some downsampling
mechanism leading to a
decrease in the total
number of units per
layer of the network.

11. By Olivia Seji OH
The paper introduces...
11
Content Reconstructions
∙ It is able to visualize
the information at
different processing
stages in the CNN by
reconstructing the input
image from only knowing
the network’s responses
in a particular layer.

12. By Olivia Seji OH
The paper introduces...
12
Content Reconstructions
∙ In higher layers of the
network, detailed pixel
information is lost
while the high-level
content of the image is
preserved (d, e).

13. By Olivia Seji OH
The paper introduces...
Style Reconstructions
13
Compute correlations
The style of an input images are captured
on top of the original CNN representations
Reconstruct
The different features in different layers of the CNN
The style of the input image from style represations
built on different subsets of CNN layers

14. By Olivia Seji OH
The paper introduces...
14
Style Reconstructions
∙ The images are created
that match the style of
a given images on an
increasing scale while
discarding information
of the global
arrangement of the
scene.
∙ Multi-scale
representation
(multiple layers of
the neural network)

15. By Olivia Seji OH
The key finding of the paper is...
∙ The representations of content and style in the Convolutional
Neural Network (CNN) are separable.
∙ Both representations can be manipulated independently to produce
new, perceptually meaningful images.
∙ The images are synthesised by finding an image that
simultaneously matches the content representation of the
photograph and the style representation of the respective piece
of art.
15

16. By Olivia Seji OH
The key finding of the paper is...
∙ The global arrangement of the original photograph is preserved.
∙ The colours and local structures that compose the global scenery
are provided by the artwork.
∙ Effectively, this renders the photograph in the style of the
artwork, such that the appearance of the synthesised images
resembles the work of art, even though it shows the same content
as the photograph.
16

17. By Olivia Seji OH
The key finding of the paper is...
17

18. By Olivia Seji OH
The key finding of the paper is...
∙ Image content and style cannot be completely disentangled.
∙ The loss function which is minimised during image synthesis
contains two terms for content and style respectively, that are
well separated.
∙ It is able to be regulated smoothly the emphasis on either
reconstructing the content or the style.
∙ For specific pair of source images one can adjust the trade-off
between content and style to create visually appealing images.
18

19. By Olivia Seji OH
The key finding of the paper is...
19
∙ The local image structures captured
by the style representation
increase in size and complexity
when including style features from
higher layers of the network.
∙ This can be explained by the
increasing receptive field size and
feature complexity along the
network’s processing hierarchy.
∙ The column show different relative
weightings between the content and
style reconstruction.

20. By Olivia Seji OH
OKAY, now we understand
What ‘neural style transfer’ is.
Let’s try to figure out
how it can be accomplished.
20
(Yes, we are going to go deeper instead of stopping here.)

21. By Olivia Seji OH
The method is...
∙ The results in the main text were generated on the basis of the
VGG-Network.
∙ A Convolutional Neural Network (CNN).
∙ The feature space provided by 16 convolutional and 5 pooling
layers of the 19 layer VGG-Network.
∙ For image synthesis that replacing the max-pooling operation by
averaging pooling improves the gradient flow and one obtains
slightly more appealing results.
21

22. By Olivia Seji OH
The method is...
∙ A given input image is encoded in each layer of the CNN by the
filter responses to that image.
∙ Gradient descent is performed on a white noise image to find
another image that matches the feature responses of the original
image to visualise the image information that is encoded at
different layers of the hierarchy.
∙ The squared-error loss between the two feature representations
(defined)
22

23. By Olivia Seji OH
The method is...
∙ The derivative of this loss with respect to the activations in
layer l equals
from which the gradient with respect to the image can be
computed using standard error back-propagation.
∙ Thus the initially random image can be changed until it
generates the same response in a certain layer of the CNN as the
original image .
23

24. By Olivia Seji OH
The method is...
∙ The derivative of this loss with respect to the activations in
layer l equals
from which the gradient with respect to the image can be
computed using standard error back-propagation.
∙ Thus the initially random image can be changed until it
generates the same response in a certain layer of the CNN as the
original image .
24

25. By Olivia Seji OH
The method is...
∙ On top of the CNN responses in each layer of the Network which
is built a style representation that computes the correlations
between the different filter responses, where the expectation is
taken over the spatial extent of the input image.
∙ These feature correlations are given by the Gram matrix
where it is the inner product between the vectorised feature map
i and j in layer l.
25

26. By Olivia Seji OH
The method is...
∙ Gradient descent from a white noise image can be used to find
another image that matches the style representation of the
original image by minimising the mean-squared distance between
the entries of the Gram matrix from the original image and the
Gram matrix of the image to be generated.
26

27. By Olivia Seji OH
The method is...
∙ The contribution of that layer to the total loss is
∙ The total loss is
where are weighting factors of the contribution of each
layer to the total loss.
27

28. By Olivia Seji OH
The method is...
∙ The derivative of El
with respect to the activations in layer l
can be computed analytically:
∙ The gradients of El
with respect to the activations in lower
layers of the network can be readily computed using standard
error back-propagation.
28

29. By Olivia Seji OH
The method is...
∙ The loss function which is minimised here is
where 𝛼 and 𝛽 are the weighting factors for content and style
reconstruction respectively.
29

30. By Olivia Seji OH
The method is...
30
∙ Results for
different relative weightings
of the content
and style reconstruction
(the ratio 𝛼/𝛽) and
for matching
the style representations.
∙ The factor was always equal
to one divided by the number of
active layers with a non-zero
loss weight .

31. By Olivia Seji OH
VGG19 Model in KERAS with TensorFlow 2.0 is...
∙ This is a model with weights pre-trained on ImageNet.
∙ keras.applications.vgg19.VGG19(include_top=True,
weights='imagenet', input_tensor=None, input_shape=None,
pooling=None, classes=1000)
∙ This model can be built both with 'channels_first' data format
(channels, height, width) or 'channels_last' data format
(height, width, channels).
∙ The default input size for this model is 224x224.
31

32. By Olivia Seji OH
Now we can revisit the code.
32

33. By Olivia Seji OH
the code is...
A tutorial about Neural style
transfer at the TensorFlow
official webpage.
Here is the link:
https://www.tensorflow.org/alph
a/tutorials/generative/style_tr
ansfer
The Result:
Beautiful mixture of images
You can also try this here:
https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/r2/tutorials/generative/style_transfer.ipynb
33

34. By Olivia Seji OH
It is time to conclude.
34

35. By Olivia Seji OH
With the method...
∙ Render a given photograph in the style of a range of well-known
artworks by using Deep Neural Networks trained on object
recognition.
∙ Carry out manipulations in feature spaces that explicitly
represent the high level content of an image.
35

36. By Olivia Seji OH
What this method provides...
With the method of synthesising images that mix content and
style from different sources
∙ A new and fascinating tool to study the perception and neural
representation of art, style and content-independent image
appearance in general.
∙ An algorithmic understanding of how neural representations can
independently capture the content of an image and the style in
which it is represented.
36

37. By Olivia Seji OH
What this method provides...
With the method of synthesising images that mix content and
style from different sources
∙ The mathematical form of our style representations generates a
clear, testable hypothesis about the representation of image
appearance down to the single neuron level.
∙ The style representations simply compute the correlations
between different types of neurons in the network.
∙ Extracting correlations between neurons is a biologically
plausible computation.
37

38. By Olivia Seji OH
What this method provides...
With the method of synthesising images that mix content and
style from different sources
∙ The ability to abstract content from style and therefore the
ability to create and enjoy art might be primarily a preeminent
signature of the powerful inference capabilities of our visual
system.
38

39. By Olivia Seji OH
Where I get the information are... (References)
⟣ Neural style transfer @TensorFlow 2.0 Tutorial:
https://www.tensorflow.org/alpha/tutorials/generative/style_tran
sfer
⟣ A Neural Algorithm of Artistic Style by Leon A. Gatys, Alexander
S. Ecker, Matthias Bethge, 2015:
https://arxiv.org/abs/1508.06576
⟣ Applied Deep Learning: A Case-Based Approach to Understanding
Deep Neural Networks, written by Umberto Michelucci, Apress,
2018
39