This presentation deals with examplar-based inpainting. It is based on the following papers: (i) C. Guillemot and O. Le Meur, Image inpainting: overview and recent advances, IEEE Signal Processing Magazine, Vol. 1, pp. 127-144, 2014. (ii) O. Le Meur, M. Ebdelli and C. Guillemot, Hierarchical super-resolution-based inpainting, IEEE TIP, vol. 22(10), pp. 3779-3790, 2013. (iii) O. Le Meur and C. Guillemot, Super-resolution-based inpainting, ECCV 2012.

1. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Examplar-based inpainting
Olivier Le Meur
olemeur@irisa.fr
IRISA - University of Rennes 1
June 19, 2014
1 / 44

2. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Inpainting: context and issues (1/3)
This talk is about inpainting. We will heavily rely upon these papers:
C. Guillemot & O. Le Meur, Image inpainting: overview
and recent advances, IEEE Signal Processing Magazine,
Vol. 1, pp. 127-144, 2014.
O. Le Meur, M. Ebdelli and C. Guillemot, Hierarchical
super-resolution-based inpainting, IEEE TIP, vol.
22(10), pp. 3779-3790, 2013.
O. Le Meur & C. Guillemot, Super-resolution-based
inpainting, ECCV 2012.
2 / 44

3. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Inpainting: context and issues (2/3)
Inpainting
Inpainting corresponds to filling holes (i.e. missing areas) in im-
ages (Bertalmio et al., 2000).
Let be an image I defined as
I : Ω ⊂ Rn
−→ Rm
Let be a degradation operator M
M : Ω −→ {0, 1}
M(x) =
0, if x ∈ U
1, otherwise
Let F the observed image:
F = M ◦ I
n = 2 for a 2D image
m = 3 for (R,G,B) image
Ω = S ∪ U,
• S being the known part
of I
• U the unknown part of I
◦ is the Hadamard product
3 / 44

4. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Inpainting: context and issues (3/3)
Different configurations according to the definition of M:
Original image
80% of the pixels
have been
removed.
damaged portions
in black, scratches
object removal
Sparsity and
low-rank methods
Diffusion-based
methods
Examplar-based
methods
4 / 44

5. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Outline of the presentation
1 Inpainting: context and issues
2 Examplar-based inpainting
3 Variants of Criminisi’s method
4 Super-resolution-based inpainting method
5 Results and comparison with existing methods
6 Conclusion
5 / 44

6. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Outline of the presentation
1 Inpainting: context and issues
2 Examplar-based inpainting
Presentation
Notation
Criminisi et al.’s method
3 Variants of Criminisi’s method
4 Super-resolution-based inpainting method
5 Results and comparison with existing methods
6 Conclusion
6 / 44

7. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Examplar-based inpainting (1/4)
Texture synthesis
Examplar-based inpainting methods rely on the assumption that the
known part of the image provides a good dictionary which could be
used efficiently to restore the unknown part (Efros and Leung, 1999).
The recovered texture is therefore
learned from similar regions.
ª This can be done simply by
sampling, copying or
combining patches from the
known part of the image;
Template Matching
ª Patches are then stitched
together to fill in the missing
area.
7 / 44

8. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Examplar-based inpainting (2/4)
Notations:
ª a patch ψpx is a discretized
N × N neighborhood
centered on the pixel px.
This patch can be vectorized
in a raster-scan order as a
mN2
-dimensional vector;
ª ψuk
px
denotes the unknown
pixels of the patch;
ª ψk
px
denotes its known
pixels;
ª ψpx(i)
denotes the ith
nearest neighbour of ψpx ;
ª δU is the front line;
8 / 44

9. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Examplar-based inpainting (3/4)
Criminisi et al.’s algorithm
Criminisi et al. (Criminisi et al., 2004) has brought a new momentum
to inpainting applications and methods. They proposed a new method
based on two sequential stages:
1 Filling order computation;
2 Texture synthesis.
1 Filling order computation: P(px) = C(px) × D(px)
Confidence term
C(px) =
q∈ψk
px
C(q)
|ψpx
|
where |ψpx
| is the area of ψpx
.
Data term
D(px) =
| I⊥
(px) · npx
|
α
where α is a normalization
constant in order to ensure that
D(px) is in the range 0 to 1.
9 / 44

10. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Examplar-based inpainting (4/4)
2 Texture synthesis:
A template matching is performed within a local neighborhood:
py = arg min
q∈W
d(ψk
pq
, ψk
px∗ )
ª W ⊆ S is the window search;
ª ψk
px∗ are the known pixels of the patch ψpx∗ with the highest
priority;
ª ψk
py
are the known pixels of the nearest patch neighbor;
ª d(a, b) is the sum of squared differences between patches a and
b.
The pixels of the patch ψuk
py
are then copied into the unknown pixels
of the patch ψpx∗ .
10 / 44

11. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Outline of the presentation
1 Inpainting: context and issues
2 Examplar-based inpainting
3 Variants of Criminisi’s method
Filling order computation
Texture synthesis
Some examples
Limitations
4 Super-resolution-based inpainting method
5 Results and comparison with existing methods
6 Conclusion
11 / 44

12. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Filling order computation (1/4)
P(px) = C(px) × D(px)
Two variants are here presented:
ª Tensor-based data term (Le Meur et al., 2011);
ª Sparsity-based data term (Xu and Sun, 2010).
Many others: edge-based data term, transformation of the data term
in a nonlinear fashion, entropy-based data term...
12 / 44

13. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Filling order computation (2/4)
Tensor-based data term
Instead of using the gradient, (Le Meur et al., 2011) used the structure
tensor which is more robust:
D(px) = α + (1 − α)exp −
η
(λ1 − λ2)2
where η is a positive value and α ∈ [0, 1].
The structure tensor is a symmetric, positive semi-definite
matrix (Weickert, 1999):
Jρ,σ [I] = Kρ ∗
m
i=1
(Ii ∗ Kσ) (Ii ∗ Kσ)T
where Ka is a Gaussian kernel with a standard deviation a. The
parameters ρ and σ are called integration scale and noise scale,
respectively.
13 / 44

14. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Filling order computation (3/4)
D(px) = α + (1 − α)exp −
η
(λ1 − λ2)2
When λ1 λ2, the data term tends to α. It tends to 1 when
λ1 >> λ2.
14 / 44

15. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Filling order computation (4/4)
Sparsity-based data term
Sparsity-based data term (Xu and Sun, 2010) is based on the sparse-
ness of nonzero patch similarities:
D(px) =
|Ns(px)|
|N(px)|
×
pj ∈Ws
w2
px ,pj
where Ns and N are the numbers of valid and candidate patches in
the search window.
Weight wpx ,pj is proportional to the similarity between the two patches
centered on px and pj ( j wpx ,pj
= 1).
A large value of the structure sparsity term means sparse similarity
with neighboring patches
⇒ a good confidence that the input patch is on some structure.
15 / 44

16. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Texture synthesis (1/4)
Texture synthesis with more than one candidate
From K patches ψpx(i)
which are the most similar to the known part
ψk
px
of the input patch, the unknown part of the patch to be filled ψuk
px
is then obtained by a linear combination of the sub-patches ψuk
px(i)
.
ψuk
px
=
K
i=1
wiψuk
px(i)
How can we compute the weights
wi of this linear combination?
Note: K is locally adjusted by using
an -ball including patches within a
certain radius.
16 / 44

17. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Texture synthesis (2/4)
ψuk
px
=
K
i=1
wiψuk
px(i)
Different solutions exist (Guillemot et al., 2013):
ª Average template matching: wi = 1
K , ∀i;
ª Non-local means approach (Buades et al., 2005):
wi = exp −
d(ψpk
x
, ψpk
x(i)
)
h2
ª Least-square method minimizing
E(w) = ψk
px
− Aw 2
2,a
w∗
= arg min
w
E(w)
17 / 44

18. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Texture synthesis (3/4)
ψuk
px
=
K
i=1
wiψuk
px(i)
ª Constrained Least-square optimization with the sum-to-one
constraint of the weight vector ⇒ LLE method (Saul and
Roweis, 2003)
E(w) = ψk
px
− Aw 2
2,a
w∗
= arg min
w
E(w) s.t. wT
1K = 1
ª Constrained Least-square optimization with positive weights ⇒
NMF method (Lee and Seung, 2001)
w∗
= arg min
w
E(w) s.t. wi ≥ 0
18 / 44

19. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Texture synthesis (4/4)
Similarity metrics:
ª Using a Gaussian weighted Euclidean distance
dL2 (ψpx
, ψpy
) = ψpx
− ψpy
2
2,a
where a controls the decay of the Gaussian function
g(k) = e−
|k|
2a2
, a > 0;
ª A better distance introduced in (Bugeau et al., 2010, Le Meur
and Guillemot, 2012):
d(ψpx , ψpy ) = dL2 (ψpx , ψpy ) × (1 + dH (ψpx , ψpy ))
where dH (ψpx
, ψpy
) is the Hellinger distance
dH (ψpx
, ψpy
) = 1 −
k
p1(k)p2(k)
where p1 and p2 represent the histograms of patches ψpx
, ψpy
,
respectively.
19 / 44

20. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Some Examples (1/2)
Inpainted pictures with (Criminisi et al., 2004)’s method (Courtesy of
P. P´erez):
20 / 44

21. Results from (Le Meur et al., 2011).

22. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Limitations
Very sensitive to the parameter settings such as the filling order
and the patch size:
Examplar-based methods are a one-pass greedy algorithms.
22 / 44

23. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Outline of the presentation
1 Inpainting: context and issues
2 Examplar-based inpainting
3 Variants of Criminisi’s method
4 Super-resolution-based inpainting method
Proposed approach
More than one inpainting
Loopy Belief Propagation
Super-resolution
5 Results and comparison with existing methods
6 Conclusion
23 / 44

24. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Proposed approach (1/1)
Objectives of the proposed method
We apply an examplar-based inpainting algorithm several times and
fuse together the inpainted results.
less sensitive to the inpainting setting;
relax the greedy constraint.
The inpainting method is applied on a coarse version of the input
picture:
less demanding of computational resources;
less sensitive to noise;
K candidates for the texture synthesis without introducing blur.
Need to fuse the inpainted images and to retrieve the highest
frequencies
Loopy Belief Propagation and Super-Resolution algorithms.
24 / 44

25. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
More than one inpainting (1/1)
The baseline algorithm is an
examplar-based method:
ª Filling order
computation;
ª Texture synthesis.
ª Decimation factor n = 3
ª 13 sets of parameters
Table: Thirteen inpainting configurations.
Setting Parameters
1
Patch’s size 5 × 5
Decimation factor n = 3
Search window 80 × 80
Sparsity-based filling order
2 default + rotation by 180 degrees
3 default + patch’s size 7 × 7
4
default + rotation by 180 degrees
+ patch’s size 7 × 7
5 default + patch’s size 11 × 11
6
default + rotation by 180 degrees
+ patch’s size 11 × 11
7 default + patch’s size 9 × 9
8
default + rotation by 180 degrees
+ patch’s size 9 × 9
9
default + patch’s size 9 × 9
+ Tensor-based filling order
10
default + patch’s size 7 × 7
+ Tensor-based filling order
11
default + patch’s size 5 × 5
+ Tensor-based filling order
12
default + patch’s size 11 × 11
+ Tensor-based filling order
13
default + rotation by 180 degrees
+ patch’s size 9 × 9
+ Tensor-based filling order
25 / 44

26. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Loopy Belief Propagation (1/5)
. . . . . .
Loopy Belief Propagation is used to fuse together the 13 inpainted
images.
Let be a finite set of labels L composed of M = 13 values.
E(l) =
p∈ν
Vd(lp) + λ
(n,m)∈N4
Vs(ln, lm)
where, lp the label of pixel px, ν represents the pixel in U and N4 is
a neighbourhood system. λ is a weighting factor.
26 / 44

27. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Loopy Belief Propagation (2/5)
E(l) =
p∈ν
Vd(lp) + λ
(n,m)∈N4
Vs(ln, lm)
ª Vd(lp) represents the cost of assigning a label lp to a pixel px:
Vd(lp) =
n∈L u∈υ
I(l)
(x + u) − I(n)
(x + u)
2
ª Vs(ln, lm) is the discontinuity cost:
Vs(ln, lm) = (ln − lm)
2
The minimization is performed iteratively (less than 15
iterations) (Boykov and Kolmogorov, 2004, Boykov et al., 2001,
Yedidia et al., 2005).
27 / 44

28. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Loopy Belief Propagation (3/5)
LBP convergence:
ª 13 inpainted image in
input;
ª 25 iterations;
ª resolution=80 × 120.
28 / 44

29. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Loopy Belief Propagation (4/5)
LBP convergence:
ª 13 inpainted image in
input;
ª 25 iterations;
ª resolution=120 × 80.
29 / 44

30. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Loopy Belief Propagation (5/5)
LBP convergence:
ª 13 inpainted image in
input;
ª 25 iterations;
ª resolution=200 × 135.
30 / 44

31. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Super-resolution (1/2)
For the LR patch corresponding
to the HR patch having the
highest priority:
ª We look for its best
neighbour;
ª Only the best candidate is
kept;
ª The corresponding HR
patch is simply deduced.
ª Its pixel values are then
copied into the unknown
parts of the current HR
patch.
31 / 44

32. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Super-resolution (2/2)
To speed-up the process, we can perform the
search:
ª within a search window;
ª within a dictionary (as illustrated on the
right) composed of LR patches with
their corresponding HR patches.
32 / 44

33. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Outline of the presentation
1 Inpainting: context and issues
2 Examplar-based inpainting
3 Variants of Criminisi’s method
4 Super-resolution-based inpainting method
5 Results and comparison with existing methods
Results
Comparison with existing methods
6 Conclusion
33 / 44

34. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Results (1/4)
34 / 44

35. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Results (2/4)
35 / 44

36. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Results (3/4)
36 / 44

37. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Results (4/4)
37 / 44

38. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Comparison with existing methods (1/5)
Three methods have been tested:
ª [Komodakis] N. Komodakis, and G. Tziritas, Image Completion
using Global Optimization. in CVPR 2007 (Komodakis and
Tziritas, 2007);
ª [Pritch] Y. Pritch, E. Kav-Venaki, S. Peleg, Shift-Map Image
Editing. in ICCV 2009 (Pritch et al., 2009);
ª [He] K. He and J. Sun, Statistics of Patch Offsets for Image
Completion. in ECCV 2012 (He and Sun, 2012).
38 / 44

39. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Comparison with existing methods (2/5)
From left to right: Komodakis, Pritch, He, Ours.
39 / 44

40. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Comparison with existing methods (3/5)
From left to right: Komodakis, Pritch, He, Ours.
40 / 44

41. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Comparison with existing methods (4/5)
Much more results on the link:
http://people.irisa.fr/Olivier.Le_Meur/publi/2013_TIP/
indexSoA.html
41 / 44

42. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Comparison with existing methods (5/5)
Limitations and failure cases:
From left to right: original, He’s method and proposed one.
ª No semantic information are used...
ª No objective quality metric.
42 / 44

43. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Outline of the presentation
1 Inpainting: context and issues
2 Examplar-based inpainting
3 Variants of Criminisi’s method
4 Super-resolution-based inpainting method
5 Results and comparison with existing methods
6 Conclusion
43 / 44

44. Examplar-based
inpainting
O. Le Meur
Inpainting:
context and
issues
Examplar-based
inpainting
Presentation
Notation
Criminisi et al.’s
method
Variants of
Criminisi’s
method
Filling order
computation
Texture synthesis
Some examples
Limitations
Super-resolution-
based inpainting
method
Proposed approach
More than one
inpainting
Loopy Belief
Propagation
Super-resolution
Conclusion
ª A new framework to perform inpainting of still color pictures:
coarse inpainting + super-resolution.
Binary file could be downloaded:
http://people.irisa.fr/Olivier.Le_Meur/publi/2013_
TIP/index.html
ª A natural extension is to deal with video inpainting.
A paper dealing with video inpainting under revision in IEEE TIP.
44 / 44

45. Examplar-based
inpainting
O. Le Meur
References
References
M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester. Image inpainting. In SIGGRPAH 2000, 2000.
Y. Boykov and V. Kolmogorov. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision.
IEEE Trans. On PAMI, 26(9):1124–1137, 2004.
Y. Boykov, O. Veksler, and R.Zabih. Efficient approximate energy minimization via graph cuts. IEEE Trans. On PAMI, 20(12):
1222–1239, 2001.
A. Buades, B. Coll, and J.M. Morel. A non local algorithm for image denoising. In IEEE Computer Vision and Pattern Recognition
(CVPR), volume 2, pages 60–65, 2005.
A. Bugeau, M. Bertalm´ıo, V. Caselles, and G. Sapiro. A comprehensive framework for image inpainting. IEEE Trans. on Image
Processing, 19(10):2634–2644, 2010.
A. Criminisi, P. P´erez, and K. Toyama. Region filling and object removal by examplar-based image inpainting. IEEE Trans. On
Image Processing, 13:1200–1212, 2004.
A. A. Efros and T. K. Leung. Texture synthesis by non-parametric sampling. In IEEE Computer Vision and Pattern Recognition
(CVPR), pages 1033–1038, 1999.
C. Guillemot, M. Turkan, O. Le Meur, and M. Ebdelli. Object removal and loss concealment using neigbor embedding methods.
Signal processing: image communication, 28:1405–1419, 2013.
K. He and J. Sun. Statistics of patch offsets for image completion. In ECCV, 2012.
N. Komodakis and G. Tziritas. Image completion using efficient belief propagation via priority scheduling and dynamic pruning.
IEEE Trans. On Image Processing, 16(11):2649 – 2661, 2007.
O. Le Meur and C. Guillemot. Super-resolution-based inpainting. In ECCV, pages 554–567, 2012.
O. Le Meur, J. Gautier, and C. Guillemot. Examplar-based inpainting based on local geometry. In ICIP, 2011.
D. D. Lee and H. S. Seung. Algorithms for non-negative matrix factorization. In In NIPS, pages 556–562. MIT Press, 2001.
Y. Pritch, E. Kav-Venaki, and S. Peleg. Shift-map image editing. In ICCV’09, pages 151–158, Kyoto, Sept 2009.
L.K. Saul and S.T. Roweis. Think globally, fit locally: Unsupervised learning of low dimensional manifolds. Journal of Machine
Learning Research, 4:119–155, 2003.
J. Weickert. Coherence-enhancing diffusion filtering. International Journal of Computer Vision, 32:111–127, 1999.
Z. Xu and J. Sun. Image inpainting by patch propagation using patch sparsity. IEEE Trans. on Image Processing, 19(5):
1153–1165, 2010.
J.S. Yedidia, W.T. Freeman, and Y. Weiss. Constructing free energy approximations and generalized belief propagation algorithms.
IEEE Transactions on Information Theory, 51:2282–2312, 2005.
44 / 44