My final seminar on the course "Fundamentals and Trends in Vision and Image Processing" (IMPA). In this presentation, I focused on deep learning applications for image processing and creative goals.

1. DEEP LEARNING
applications
Julia Rabetti Giannella
Research assistant at VISGRAF Lab
PhD in Design and Technology (PPDESDI-UERJ)
juliagiannella@gmail.com

2. APPLICATIONS
• Colorization of Black and White Images
• Adding Sounds To Silent Movies
• Object Classification in Photographs
• Automatic Handwriting Generation
• Character Text Generation.
• Image Caption Generation.
• Automatic Game Playing
• Artistic style transfer
Source: http://machinelearningmastery.com/inspirational-applications-deep-learning/

3. 1) Colorization of Black and White Images
• problem of adding color to black and white photographs
• traditionally, this was done by hand with human effort
• CV task attacked by different approaches
• topic of relative importance in SIGGRAPH and EUROGRAPH
• DL approach involves the use of very large CNN
and supervised layers that recreate the image with
the addition of color

4. Paper Colorful Image Colorization (ECCV, 2016)
Source: http://richzhang.github.io/colorization/

5. Network architecture
Source: https://arxiv.org/pdf/1603.08511.pdf

6. Semantic interpretability of results
Source: http://richzhang.github.io/colorization/

7. [Algorithmia] Demo
Source: http://demos.algorithmia.com/colorize-photos/

8. Dana Keller - designer and photo colorizer
Source: https://www.youtube.com/watch?v=bYHnWhZkAIc
Source: http://www.danarkeller.com/about/

9. Comparing
Keller Algorithmia

10. Comparing
Keller Algorithmia

11. Comparing
Keller Algorithmia

12. Comparing
Keller Algorithmia

13. Comparing
Keller Algorithmia

14. Comparing
Keller Algorithmia

15. 2) Object Classification in Photographs
• task requires the classification of objects within a
photograph as one of a set of previously known objects
• State-of-the-art results have been achieved on benchmark
examples of this problem using very large CNN
• derives from image classification task
• breakthrough: ImageNet Classification with Deep
Convolutional Neural Networks (Krizhevsky et al., 2012)
• AlexNet won ILSVRC-2012 challenge
Source: http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf

16. Classification with localization
• more complex variation
of this task involves
specifically identifying
one or more objects
within the scene of the
photograph and drawing
a box around them
• GoogLeNet won
ILSVRC-2014 challenge
in this task
Source: https://research.googleblog.com/2014/09/building-deeper-understanding-of-images.html

17. 2.1) DL and RIO2016
• VISGRAF project (out 2016)
• task: automatically classify and cluster images by subject
features related to the Olympic Games, Olympic Torch
• CNN model and supervised learning
• TensorFlow (open source software library)
• Inception-v3 (Going Deeper with Convolutions, 2015)
• transfer learning (manually labeled 100 examples)
Source: http://lvelho.impa.br/dl_rio2016/index.html
Source: https://arxiv.org/abs/1409.4842

18. Confidence score
Source: http://lvelho.impa.br/dl_rio2016/metodologia.html
A subset of 12
from 2091
images with
confidence
score over 83%
for the Olympic
torch category

19. Torch Mosaic
Source: http://lvelho.impa.br/dl_rio2016/mosaico.html

20. Torch Mosaic
Source: http://lvelho.impa.br/dl_rio2016/mosaico.html

21. 2.2) Twitter Facial Analysis Reveals Demographics
of Presidential Campaign Followers
• (Mit Technology Review, march 2016)
• IN: Conference on Web and Social Media
• understand follower demographics of Trump and Clinton by
crossing Twitter metadata and facial features
• a CNN model on followers’ profile images extracts
information on gender, race and age
Source: https://www.technologyreview.com/s/601074/twitter-facial-analysis-reveals-demographics-of-presidential-
campaign-followers/?utm_campaign=add_this&utm_source=email&utm_medium=post
Source: https://arxiv.org/abs/1603.03097

22. A Comparison of the Trumpists and Clintonists
Source: https://arxiv.org/abs/1603.03097
C"lintonists"
in the Twitter
Sphere

23. 2.3) NVIDIA DRIVENet Demo - Visualizing
a Self-Driving Car
Source: https://www.youtube.com/watch?v=HJ58dbd5g8g

24. 3) Artistic style transfer
• task: separate and recombine content and style of arbitrary
images, providing a neural algorithm for the creation of
artistic images
• A Neural Algorithm of Artistic Style (Gatys et al., 2015)
Source: https://arxiv.org/abs/1508.06576

25. Convolutional Neural Network (CNN)
Source: https://arxiv.org/abs/1508.06576

26. An example
Source: https://research.googleblog.com/2016/02/exploring-intersection-of-art-and.html
The style
transfer
algorithm
crosses a photo
with a painting
style; for
example Neil
deGrasse Tyson
in the style of
Kadinsky’s Jane
Rouge Bleu.
Photo by
Guillaume Piolle,
used with
permission.

27. 3.1) DeepDream
Source: http://deepdreamgenerator.com/
Source: https://en.wikipedia.org/wiki/DeepDream
• computer vision program created by Google
• given an input image returns a version with h"allucinogenic"
appearance
• originates in a CNN codenamed Inception after the film of
the same name developed for the ILSVRC-2014
• CNN can also be run in reverse, to do synthesis
• enhance faces and certain animals -> pareidolia results

28. 3.1) DeepDream
Source: http://deepdreamgenerator.com/
Source: https://en.wikipedia.org/wiki/DeepDream

29. 3.2) Prisma App
Source: http://prisma-ai.com/
Source: https://en.wikipedia.org/wiki/Prisma_(app)
• photo-editing application that utilizes a neural network and
to transform the image into an artistic effect
• became popular on July 2016
• created by Alexey Moiseenkov
• reference A Neural Algorithm of Artistic Style (2016)

30. 3.2) Prisma App

31. 3.2) Prisma App

32. 3.3) Artistic style transfer (video)
Source: https://arxiv.org/abs/1604.08610
Source: https://www.youtube.com/watch?v=Khuj4ASldmU
• Artistic style transfer for videos (Ruder et al.,2016)

33. 3.4) Supercharging Style Transfer for video
Source: https://arxiv.org/abs/1610.07629
Source: https://research.googleblog.com/2016/10/supercharging-style-transfer.html
• A Learned Representation For Artistic Style (Dumoulin et al.,
2016)
• CNN that learns multiple styles at the same time
• method enables style interpolation

34. 3.4) Supercharging Style Transfer for video
Source: https://www.youtube.com/watch?v=6ZHiARZmiUI