The document discusses inpainting, a technique for reconstructing damaged or missing areas in images, and presents various algorithms, including one developed by the author that reduces processing time and avoids image blurring. It reviews traditional and convolution-based methods, outlining their applications in image restoration. The proposed method demonstrates improved execution times and results, indicating its potential for future use in both 2D and 3D image processing.

1. Date : December 6, 2017
Location : USM
VISUAL TECHNIQUES, COS-701
1
Presented by:
Md. Shohel Rana
Instructed By:
Dr. Parthapratim Biswas
DE-CONVOLUTION ON DIGITAL IMAGES

2. CONTENTS
• Motivation and Contribution
• What is Inpainting? Why?
• Convolution? Why?
• Convolution based inpainting
• Various Inpainting Techniques on Image
• Image Measurement Matrices
• Proposed Method
• Experiment and Result
• Discussion on Result
• Conclusion and Future Works
VISUAL TECHNIQUES, COS-701
2

3. MOTIVATION AND CONTRIBUTION
• Given image with significant portions missing or damaged
• Dynamically detect the damaged regions to be inpainted
• Reconstitute missing regions with data consistent with the rest of the
image
• Proposed a method which restore damaged area of the image
reducing processing time without blurring output
VISUAL TECHNIQUES, COS-701
3

4. INPAINTING? WHY?
• Reconstruction of missing or damaged portions of images is an ancient
practice used broadly in artwork restoration
• The activity consists of filling in the missing areas or modifying the damaged
ones in a non-detectable way by an observer not familiar with the original
images
• Applications of image inpainting range from restoration of photographs, films
and paintings, to removal of occlusions, such as text, subtitles, stamps and
publicity from images.
• Inpainting is an artistic synonym for image interpolation, and has been
circulated among museum restoration artists for a long time
• As an ancient painting gets older, the pigments in certain parts start falling
off the canvas, and the painting becomes incomplete
• The human act of filling in the missing parts of a painting is called
"inpainting“ as first introduced to image processing by Bertalmio, Sapiro,
Caselles, and Ballester at the University of Minnesota (SIGGRAPH 2000)
[1]-[2] VISUAL TECHNIQUES, COS-701
4

5. INPAINTING? WHY? (CONT.)
• Inpainting technique can be categorized as follows:
 Convolution Filter based method
 Partial Differential Equation (PDE)
 Others Algorithm
• In general all convolution based methods provide good results
• The convolution based equation is as follows
Where Iout is inpainted image, I is the input image W is the Mask and M x N is size of the Image
VISUAL TECHNIQUES, COS-701
5
∑∑= =
=
M
i
N
j
out jiWjiIjiI
0 0
),(),(),(

6. CONVOLUTION? WHY?
• Convolution is a mathematical way of combining two signals to form a third signal. It
is the single most important technique in Digital Signal Processing.
• A system changes an input signal into an output signal.
• First, the input signal can be decomposed into a set of impulses, each of which can be viewed
as a scaled and shifted delta function.
• Second, the output resulting from each impulse is a scaled and shifted version of the impulse
response.
• Third, the overall output signal can be found by adding these scaled and shifted impulse
responses. In other words, if we know a system's impulse response, then we can calculate
what the output will be for any possible input signal
• The impulse response goes by a different name in some applications. If the system
being considered is a filter, the impulse response is called the filter kernel, the
convolution kernel. In image processing, the impulse response is called the point
spread function.
• Convolution helps to understand a system’s behavior based on current and past
events
VISUAL TECHNIQUES, COS-701
6

7. CONVOLUTION? WHY? (CONT.)
• The below figure shows a simple convolution problem: a 9 point input
signal x[n] is passed through a system with a 4 point impulse response,
h[n], resulting in a 9+4-1=12 point output signal, y[n]. In mathematical
terms, x[n] is convolved with h[n] to produce y[n]. This first viewpoint of
convolution is based on the fundamental concept of DSP: decompose
the input, pass the components through the system, and synthesize the
output.
VISUAL TECHNIQUES, COS-701
7

8. CONVOLUTION BASED INPAINTING
• Convolution of two functions f(x) and g(x)
VISUAL TECHNIQUES, COS-701
8

9. VARIOUS INPAINTING TECHNIQUES ON
IMAGE
Let us assume that image I is of size M × N. Let (i, j) be the pixel location
inside the inpainting region Ω, and Ω is the area to be inpainted where δΩ its
boundary.
•Bertalmio’s algorithm: The equation for image I with inpainting region Ω
Where Ixx(i,j) and Iyy(i,j) are second-order derivatives of image at pixel (i, j): 1 ≤ i ≤ M,1≤ j ≤ N along axes x
and y, respectively. The change in L along the direction is given by
•Let, In (i, j) denote each of the image pixels inside the region Ω at the
inpainting “time” n. Then, with an improvement factor the inpainting equation is
given by
VISUAL TECHNIQUES, COS-701
9
( ) ( )
Ω∈∀∆∆+=+
),(|,),(|),(),(1
jijiItjiIjiI nnnn
β
|),,(|
),,(
),(:),(
njiN
njiN
jiLji nn



⋅= δβ
),(),(),( jiIjiIjiL n
yy
n
xx
n
+=

10. VARIOUS INPAINTING TECHNIQUES ON
IMAGE
• Oliveira Algorithm: Proposed an inpainting algorithm deleting color
information inside the mask, followed by edge detection for the
occluded/damaged area. Starting from the pixels on the edge, a convolution
operation is then applied, using a neighborhood centered on each contour
pixel and one of the proposed kernels [3]-[5]
• Oliviera method takes the image to inpaint on selected region by convolving
with averaging filter has a zero weight at the center
Where the values of a, b, and c for both kernels are 0.073235, 0.176765, and 0.125 respectively
VISUAL TECHNIQUES, COS-701
10
∑∑
∑∑
= =
= =
=
=
M
i
N
j
out
M
i
N
j
jiWjiIjiI
jiWjiIjiI
0 0
/
0 0
/
),(2).,(),(
),(1).,(),(

11. VARIOUS INPAINTING TECHNIQUES ON
IMAGE (CONT.)
• Hadhoud, Moustafa, and Shenoda’s Algorithm: They have proposed an
improvement of Oliveira’s method with reducing processing time with convoluting
the region with averaging filter has a zero weight at bottom right corner instead of
center. [4]-[6]
• The method uses a differently defined convolution kernel by using more known
neighbors, and the restoration process can be achieved even within a single
iteration
Where the values of a, b, and c for both kernels are 0.073235, 0.176765, and 0.125
respectively
VISUAL TECHNIQUES, COS-701
11
∑∑
∑∑
= =
= =
=
=
M
i
N
j
out
M
i
N
j
jiWjiIjiI
jiWjiIjiI
0 0
/
0 0
/
),(2).,(),(
),(1).,(),(

12. VARIOUS INPAINTING TECHNIQUES ON
IMAGE (CONT.)
• H. Noori, S. Saryazdi, H. Nezamabadi: Uses an adaptive kernel permitting a better
processing edge regions. To do this, it uses the gradient of known pixels in the
neighborhood of a missed pixel to compute weights in convolving mask W(x) by
proposing a function F(x) to compute weights from the image gradient. [7]-[8]
• Selecting a missed pixel on boundary of the damaged region, next considering a
neighborhood around it and central gradients for each recognized pixel in the mask W(x),
is then calculated. Then finally, a value for a damaged pixel is calculated as
Where k presents the pixel position, x is gradient value of the current pixel in the image, α is a parameter
giving an estimation of the missed pixel gradient control the softness of propagation and f'(p) is estimated
value, f(k) is value of a known pixel, n is the number of known pixel in the current neighborhood.
VISUAL TECHNIQUES, COS-701
12


















≥
≤≤−
≤−
=
α
α
α
α
α
α
||0
||
2
)1(
2
||)(1
)( 2
2
xif
xif
x
xif
x
xF
)(
1
)( kxF
n
xw =
)())()())(1()( 11
/
kfkwpfkwpf
n
k
n
k ∑∑ −−
+−=

13. IMAGE MEASUREMENT MATRICES
• RMSE: The root mean square error
• PSNR: Peak Signal to Noise Ratio is computed by
VISUAL TECHNIQUES, COS-701
13
),,.
1
(
1 1
2
)(∑∑ −
= =
=
M
i
N
j
yx jijiMN
RMSE
)max(.20
2
10
log RMSEPSNR g=

14. PROPOSED METHOD
In case of proposed method
•Inpaint damaged portion of the image without blurring output with taking less
time
•Dynamically detect the inpainted region with removing image’s noise
•Use only one Kernel/Mask where Oliveira method uses two kernels and
replacing the values of kernel
VISUAL TECHNIQUES, COS-701
14
0.080000 0.170000 0.080000
0.170000 0.080000 0.170000
0.080000 0.170000 0

15. PROPOSED METHOD (CONT.)
1. Input Image with damage information
2. Color Image Converted in to Gray scale
3. Filter the image to remove noise using median filter
4. Find edge of Imidusing “Canny Edge Detector” and smoothing image to
reduce the number of connected components producing Icanny
VISUAL TECHNIQUES, COS-701
15
)(
)(
]:1[]:1[
II
WI
W
sortmid
nsort
n
mid
sort
MNIF
=
=
=

16. PROPOSED METHOD (CONT.)
5. Resize the image to produce mask image Imaskwhich gives the target
region to be inpainted from Icanny
6. Calculate connected components to extract all the connected components.
7. Create 3x3 Window/kernel W and convolve the image using mask image
and following equation by checking damage using
8. Print Output and compute execution time
VISUAL TECHNIQUES, COS-701
16
∑∑= =
=
M
i
N
j
out jiWjiIjiI
0 0
),().,(),(

17. EXPERIMENT AND RESULT
VISUAL TECHNIQUES, COS-701
17

18. EXPERIMENT AND RESULT (CONT.)
VISUAL TECHNIQUES, COS-701
18
Figure/
Metho
d
Oliveira
(seconds)
Noori
(seconds)
Hadhoud
(seconds)
Proposed
(seconds)
Fig-a 98.2739 86.0736 9.6012 27.0833
Fig-b 42.7693 37.4856 7.1990 15.2771
Fig-c 77.6711 63.4485 9.0671 21.4831
Figure/
Method
Oliveira Noori Hadhoud Propose
d
Fig-a 12.0129 7.021
5
12.9902 13.2219
Fig-b 12.7494 9.124
8
14.4092 15.1507
Fig-c 15.1130 6.225
1
13.1350 13.0310TABLE 1: COMPARISON STUDY OF EXECUTION TIME OF PROPOSED METHOD
WITH OTHERS
TABLE 2: COMPARISON STUDY OF PSNROF PROPOSED METHOD WITH
OTHERS

19. RESULT AND DISCUSSION
During the last couple of years, a certain number of inpainting methods
have been proposed, but it is still difficult to determine the appropriate
one and also important to determine the algorithm parameters that lead
to the best PSNR results and selecting representative images to
provide relevant information. We have proposed a simple convolution
based model which faster than others’ algorithm with creating a
dynamic kernel to detect the damaged area to be inpainted. Our
proposed method can also substitute or restore the background when
removing the large object from the image by removing noise without
blurring the image.
VISUAL TECHNIQUES, COS-701
19

20. CONCLUSION AND FUTURE WORK
• It has been demonstrated that proposed technique is capable of
restoring damaged/occluded 2D image
• It gives best result among those algorithms with reducing time without
blurring images
• Used image extension was “.jpeg”
• It can be used in 3D images
• Try to preserve edge and inpaint image with asymmetric background
VISUAL TECHNIQUES, COS-701
20

21. REFERENCES
1. M. Bertalmio, Im ag e Inpainting , Proc. of SIGGRAPH 2000, Computer Graphics Processings, 417–424.
2. C. Ballester, M. Bertalmio, V. Caselles, G. Sapiro, and J. Verdera, IEEE Trans. Img. Proc., 10, 1200-1211, 2001.
3. M. Oliveira, B. Bowen, R. Mckenna, and Y. S. Chang, “Fast dig italim ag e inpainting ”, in Proc. VIIP, 2001, pp. 261–266.
4. R. Vreja and R. Brad, “Im ag e inpainting m e tho ds e valuatio n and im pro ve m e nt”, The Scientific World Journal, vol. 2014,
p. 11, 2014.
5. F. Hollaus, “Diffe re nt m e tho ds fo r im ag e inpainting ”, Vienna University of Technology.
6. M. M. Hadhoud, K. A. Moustafa, and S. Z. Shendoa, “Dig ital im ag e s inpainting using m o difie d co nvo lutio n base d
m e tho d”, International Journal of Signal Processing, Image Procesing and Pattern Recognition, 2005.
7. H. Noori, S. Saryazdi, and H. Nezamabadi-Pour, “A co nvo lutio n base d im ag e inpainting ”, in Proc. 1st
International
Conference on Communications Engineering, 22–24, December 2010.
8. R. Kamran, M. Nasri, H. Nezamabadi-pour, and S. Saryazdi, “Ane w ve cto r m e tho d fo r co lo r im ag e inpainting ”, in Proc.
First National Conference on New Ideas in Electrical Engineering, 2012.
9. A. Telea, “An im ag e inpainting te chniq ue base d o n the fast m arching m e tho d ”, Journal of Graphics Tools, vol. 9, no. 1,
2004.
VISUAL TECHNIQUES, COS-701
21

22. QUESTION?
VISUAL TECHNIQUES, COS-701
22