It Works well on images while you want to edit an image or to repair old images. it also has great results on occluded images and good to use on censorship purposes. Appropriate reconstruction is one of its features. one of the main and effective purposes is to complete images which have been destroyed during a time on SSDs or during transferring data in a transmission line or during transferring data between two devices such as laptop or Cellphones Hope you all enjoy and make it as a reference

1. B.Sc. Final Project – Mohammad Pooya Malek 2
University of IRIB
Faculty of Broadcast Engineering
A Thesis Presented for the Degree of
Bachelor of Science in Telecommunications Engineering
Image Inpainting
Based on Deep Learning
By
Mohammad Pooya Malek
Supervisor Master Reviewer
Dr. Azam Bastanfard Dr. Rahil Mahdian
Summer 2018

2. B.Sc. Final Project – Mohammad Pooya Malek 2
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
Aim of Thesis:
 Censorships
Occluded Images
Appropriate reconstruction
To Complete the image which has been destroyed
To repair old Images

3. B.Sc. Final Project – Mohammad Pooya Malek
Index
3
Introduction
Related work
Problems
Recommended Algorithm (Proposed Model)
Samples From Model
Conclusion

4. B.Sc. Final Project – Mohammad Pooya Malek
Introduction
4
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion

5. B.Sc. Final Project – Mohammad Pooya Malek
Introduction
5
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
Computer Vision (Machine Vision)
• Artificial Intelligence (AI)
• Deep Learning
• Machine Learning

6. B.Sc. Final Project – Mohammad Pooya Malek
6
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion

7. B.Sc. Final Project – Mohammad Pooya Malek
7
Neural
Network
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion

8. B.Sc. Final Project – Mohammad Pooya Malek
8
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
Neural Networks: Architectures
“Fully-connected” layers
“2-layer Neural Net”, or
“1-hidden-layer Neural Net”
“3-layer Neural Net”, or
“2-hidden-layer Neural Net”

9. B.Sc. Final Project – Mohammad Pooya Malek
9
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
Neural Networks: Architectures

10. B.Sc. Final Project – Mohammad Pooya Malek
10
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
32
32
3
Convolution Layer
5x5x3 filter
32x32x3 image

11. B.Sc. Final Project – Mohammad Pooya Malek
11
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
32
32
3
Convolution Layer
32x32x3 image
5x5x3 filter
convolve (slide) over all spatial
locations
Activation map
1
28
28

12. B.Sc. Final Project – Mohammad Pooya Malek
12
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
32
32
3
Convolution Layer
32x32x3 image
5x5x3 filter
convolve (slide) over all spatial
locations
activation maps
2
28
28
consider a second, green filter
1

13. B.Sc. Final Project – Mohammad Pooya Malek
13
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
32
32
3
Convolution Layer
activation maps
6
28
28
For example, if we had 6, 5x5 filters, we’ll get 6 separate activation maps:
We stack these up to get a “new image” of size 28x28x6!

14. B.Sc. Final Project – Mohammad Pooya Malek
14
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
Preview: ConvNet is a sequence of Convolutional Layers, interspersed with activation
functions
32
32
3
CONV,
ReLU
e.g. 6
5x5x3
filters 28
28
6
CONV,
ReLU
e.g. 10
5x5x6
filters
CONV,
ReLU
….
10
24
24

15. B.Sc. Final Project – Mohammad Pooya Malek
15
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion

16. B.Sc. Final Project – Mohammad Pooya Malek
16
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
Convolutional Layer

17. B.Sc. Final Project – Mohammad Pooya Malek
17
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
Convolutional Layer
http://setosa.io/ev/image-kernels/

18. B.Sc. Final Project – Mohammad Pooya Malek
18
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
Activation Functions
Sigmoid
tanh tanh(x)
ReLU max(0,x)
Leaky ReLU
max(0.1x, x)
ELU

19. B.Sc. Final Project – Mohammad Pooya Malek
19
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
Activation Functions
ReLU
(Rectified Linear Unit)
- Computes f(x) = max(0,x)
- Does not saturate (in +region)
- Very computationally efficient
- Converges much faster than sigmoid/tanh
in practice (e.g. 6x)
- Not zero-centered output
- And the gradient when x < 0 is 0

20. B.Sc. Final Project – Mohammad Pooya Malek
20
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
Activation Functions
Leaky ReLU
- Does not saturate
- Computationally efficient
- Converges much faster than
sigmoid/tanh in practice! (e.g. 6x)
- will not “die”.
Parametric Rectifier (PReLU)
backprop into alpha
(parameter)

21. B.Sc. Final Project – Mohammad Pooya Malek
21
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
Activation Functions
Exponential Linear Units (ELU)
- All benefits of ReLU
- Does not die
- Closer to zero mean outputs
- Computation requires exp()

22. B.Sc. Final Project – Mohammad Pooya Malek 22
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
ReLU (Rectified Linear Units)
max(0,x)

23. B.Sc. Final Project – Mohammad Pooya Malek
23
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
Pooling layer

24. B.Sc. Final Project – Mohammad Pooya Malek
24
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
Fully
Connected
layer

25. B.Sc. Final Project – Mohammad Pooya Malek
25
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
LeNet-5, a pioneering convolutional network by LeCun. in 1998
classifies digits
Recognize hand-written numbers on 32x32 pixel images.
LeNet Architecture

26. B.Sc. Final Project – Mohammad Pooya Malek
26
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
AlexNet Architecture
Alex Krizhevsky, Geoffrey Hinton, and Ilya Sutskever
ImageNet Large Scale Visual Recognition Challenge (ILSVRC) 2012 Winner
AlexNet uses ReLU(Rectified Linear Unit) for the non-linear part,
instead of a Tanh or Sigmoid function
The advantage of Using ReLU over sigmoid is that it trains much faster
Full (simplified) AlexNet architecture:
[227x227x3] INPUT
[55x55x96] CONV1: 96 11x11 filters at stride 4, pad 0
[27x27x96] MAX POOL1: 3x3 filters at stride 2
[27x27x96] NORM1: Normalization layer
[27x27x256] CONV2: 256 5x5 filters at stride 1, pad 2
[13x13x256] MAX POOL2: 3x3 filters at stride 2
[13x13x256] NORM2: Normalization layer
[13x13x384] CONV3: 384 3x3 filters at stride 1, pad 1
[13x13x384] CONV4: 384 3x3 filters at stride 1, pad 1
[13x13x256] CONV5: 256 3x3 filters at stride 1, pad 1
[6x6x256] MAX POOL3: 3x3 filters at stride 2
[4096] FC6: 4096 neurons
[4096] FC7: 4096 neurons
[1000] FC8: 1000 neurons (class scores)

27. B.Sc. Final Project – Mohammad Pooya Malek
27
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
VGGNet16 Architecture
Visual Geometry Group
Made improvement over AlexNet
Increases the depth of the network which enable it to
learn more complex features

28. B.Sc. Final Project – Mohammad Pooya Malek
28
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
VGGnet19 Architecture

29. B.Sc. Final Project – Mohammad Pooya Malek
29
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
CNN Architectures

30. B.Sc. Final Project – Mohammad Pooya Malek 30
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
Related work
Criminisi Algorithm

31. B.Sc. Final Project – Mohammad Pooya Malek
31
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion

32. B.Sc. Final Project – Mohammad Pooya Malek
32
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
Problems

33. B.Sc. Final Project – Mohammad Pooya Malek
33
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
Algorithm
Supervised vs
Unsupervised

34. B.Sc. Final Project – Mohammad Pooya Malek
34
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
OpenCV (Open Source Computer Vision Library)
BSD license: free for both academic and commercial usage
Open Source: C++, Python and Java interfaces
Cross-platform: Windows, Linux, Mac OS, iOS and Android.
focus on real-time applications
Originally developed by Intel 1999; supported by Willow Garage
maintained by Itseez; Adopted all around the world: 47 thousand user and community
downloads exceeding 14 million times; last release: 3.4.0 - 23 December 2017
Tensorflow
BSD license: free for both academic and commercial usage
Open Source: Python, C++, CUDA
Cross-platform: Linux, macOS, Windows, Android, iOS and website.
open source software library for high performance numerical computation
TensorFlow was developed by the Google Brain team in 2011
Initial released on November 9, 2015; 2 years ago
TensorFlow can run on multiple CPUs and GPUs
last release: August 8, 2018; 34 days ago

35. B.Sc. Final Project – Mohammad Pooya Malek
35
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
Supervised vs Unsupervised
Supervised Learning
Data: (x, y)
x is data, y is label
Goal: Learn a function to map
x -> y
Examples: Classification, regression,
object detection, semantic
segmentation, image captioning, etc.
Unsupervised Learning
Data: x
Just data, no labels!
Goal: Learn some structure of
the data
Examples: Clustering, dimensionality
reduction, feature learning, generative
models, etc.

36. B.Sc. Final Project – Mohammad Pooya Malek
Traditional Neural Network, Decision tree, Nearest Neighbor
36
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
Generative Adversarial Networks

37. B.Sc. Final Project – Mohammad Pooya Malek
37
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion

38. B.Sc. Final Project – Mohammad Pooya Malek
38
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
Yann LeCun Director of Facebook AI Research
“The most important breakthrough in deep learning, in my opinion, is
adversarial training (also called GAN for Generative Adversarial Networks).
This, and the variations that are now being proposed is the most interesting
idea in the last 10 years in ML.”
OpenAI
“In the process of training generative
models, we will endow the
computer with an understanding of
the world and what it is made up of.”

39. B.Sc. Final Project – Mohammad Pooya Malek
39
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
GAN
Generative Adversarial Network
Two-player game
Generator network: try to fool the discriminator by generating
real-looking images
Discriminator network: try to distinguish between real and fake
images

40. B.Sc. Final Project – Mohammad Pooya Malek
40
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
GAN
Generative Adversarial Network
Input
Noise
Latent space coordinates

41. B.Sc. Final Project – Mohammad Pooya Malek
41
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
GAN
Generative Adversarial Network
Input
Noise
Latent space coordinates
Generator
(Counterfeit)

42. B.Sc. Final Project – Mohammad Pooya Malek
42
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
GAN
Generative Adversarial Network
Input
Noise
Latent space coordinates
Generator
(Counterfeit)
Discriminator
(Detective)

43. B.Sc. Final Project – Mohammad Pooya Malek
43
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
GAN
Generative Adversarial Network
Input
Noise
Latent space coordinates
Generator
(Counterfeit)
Discriminator
(Detective)
Trained Data set
Training Set

44. B.Sc. Final Project – Mohammad Pooya Malek
44
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
CelebFaces Attributes Dataset
(CelebA)
more than 200K celebrity images,
each with 40 attribute annotations
10,177 number of identities,
202,599 number of face images, and
5 landmark locations, 40 binary
attributes annotations per image.
5_o_Clock_Shadow, Arched_Eyebrows, Attractive
Bags_Under_Eyes, Bald, Bangs, Big_Lips, Big_Nose, Black_Hai,r
Blond_Hair, Blurry Brown_Hair, Bushy_Eyebrow,s Chubby
Double_Chin, Eyeglasses, Goatee, Gray_Hair, Heavy_Makeup,
High_Cheekbones, Male, Mouth_Slightly_Open, Mustache,
Narrow_Eyes, No_Beard, Oval_Fac,e Pale_Skin Pointy_Nose
Receding_Hairline Rosy_Cheeks Sideburns, Smiling, Straight_Hair,
Wavy_Hair, Wearing_Earrings, Wearing_Hat, Wearing_Lipstick,
Wearing_Necklace, Wearing_Necktie,Young.
CelebA Dataset

45. B.Sc. Final Project – Mohammad Pooya Malek
45
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
MSCOCO Dataset
COCO-Stuff dataset: It
includes all 164K images
from COCO 2017 (train
118K, val 5K, test-dev
20K, test-challenge 20K).
It covers 172 classes: 80
thing classes, 91 stuff
classes and 1 class
'unlabeled'.

46. B.Sc. Final Project – Mohammad Pooya Malek
46
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion

47. B.Sc. Final Project – Mohammad Pooya Malek
47
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
GAN
Generative Adversarial Network
Input
Noise
Latent space coordinates
Generator
(Counterfeit)
Discriminator
(Detective)
Real
Fake
Trained Data set
Training Set

48. B.Sc. Final Project – Mohammad Pooya Malek
48
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
GAN
Generative Adversarial Network
Input
Noise
Latent space coordinates
Generator
(Counterfeit)
Discriminator
(Detective)
Real
Fake
Trained Data set
(Training Set)

49. B.Sc. Final Project – Mohammad Pooya Malek
49
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
f
activations
gradients
“local gradient”

50. B.Sc. Final Project – Mohammad Pooya Malek
50
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
activations
Another example:

51. B.Sc. Final Project – Mohammad Pooya Malek
51
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
GAN
Generative Adversarial Network
Input
Noise
Latent space coordinates
Generator
(Counterfeit)
Discriminator
(Detective)
Real
Fake
Trained Dataset
(Training Set)

52. B.Sc. Final Project – Mohammad Pooya Malek
52
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
GAN
Generative Adversarial Network
Input
Noise
Latent space coordinates
Generator
(Counterfeit)
Discriminator
(Detective)
Real
Fake
Trained Dataset
Training Set

53. B.Sc. Final Project – Mohammad Pooya Malek
53
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion

54. B.Sc. Final Project – Mohammad Pooya Malek
54
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion

55. B.Sc. Final Project – Mohammad Pooya Malek
55
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion

56. B.Sc. Final Project – Mohammad Pooya Malek
56
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
VGGNet19 Architecture GAN
Recommended
Algorithm

57. B.Sc. Final Project – Mohammad Pooya Malek
57
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
Transfer Learning with CNNs
1. Train on
ImageNet
2. If small dataset:
fix all weights
(treat CNN as fixed
feature extractor),
retrain only the
classifier
i.e. swap the
Softmax layer at
the end
3. If you have medium
sized dataset,
“finetune” instead: use
the old weights as
initialization, train the
full network or only
some of the higher
layers
retrain bigger portion
of the network, or even
all of it.

58. B.Sc. Final Project – Mohammad Pooya Malek
58
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
Samples

59. B.Sc. Final Project – Mohammad Pooya Malek
Application of GAN in IRIB
Visual Effects (VFX) : GAN Can converts day to night theme
caused decrease cost of movies
Style transfer
Image Captioning
Image Inpainting
Robotics
,…
59
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
Conclusion

60. B.Sc. Final Project – Mohammad Pooya Malek
60
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
References
 CS231n, Stanford University Deep learning Course
 MS COCO DataSet.
 CelebA DataSet.
 Criminisi A, Perez P, Toyama K. Region filling and object removal by exemplar-based
image inpainting[J]. IEEE Transactions on Image Processing, 2004, 13(9):1200-1212.
 Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,
Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In
Advances in neural information processing systems, 2014.

61. B.Sc. Final Project – Mohammad Pooya Malek
61
Image Inpainting Using Deep Learning
Introduction | Related work | Problems | Algorithm | Samples | Conclusion
Way of Connecting to Author
MOPOMA1995@gmail.com
+989379595379
https://www.linkedin.com/in/mopoma1995/