image denoising

1. Real-Time Exemplar-Based Face Sketch Synthesis
Pipeline illustration
Note: containing animations
Yibing Song1 Linchao Bao1 Qingxiong Yang1 Ming-Hsuan Yang2
1City University of Hong Kong
2University of California at Merced

2. Our assumption: a database containing photo-sketch pairs
1. photo database 2. sketch database
Aligned

3. Coarse Sketch Generation
Step 1: KNN search
p
Test photo patch 𝑻𝒑
Test photo
Training photo dataset
𝑻𝒑
𝑻𝒑
𝑻𝒑
Matched photo patch 𝑰𝑷
𝟏
Relative
position
∆𝒑
𝟏 Similarly
Matched photo patch 𝑰𝑷
𝟐
Relative
position
∆𝒑
𝟐 ∆𝒑
𝑲
[ ]
∆𝒑 =

4. Test photo patch 𝑻𝒑
∆𝒑
𝟏
Matched photo patch 𝑰𝑷
𝟏
∆𝒑
𝟐
Matched photo patch 𝑰𝑷
𝟐
∆𝒑
𝑲
Matched photo patch 𝑰𝑷
𝑲
𝒙𝒑
𝟏 ∙ +𝒙𝒑
𝟐
∙ +𝒙𝒑
𝑲 ∙ =
2. Compute linear mapping function defined by 𝒙𝒑
𝟏, 𝒙𝒑
𝟐, ⋯ , 𝒙𝒑
𝑲
Coarse Sketch Generation
Step 2: Linear Estimation from Photos

5. Matched sketch pixel
∆𝒑
𝟏
p
Matched sketch pixel
Test photo
𝑺𝑷+∆𝒑
𝟏
∆𝒑
𝟐
𝑺𝑷+∆𝒑
𝟐
∆𝒑
𝑲
Matched sketch pixel 𝑺𝑷+∆𝒑
𝑲
𝒙𝒑
𝟏
∙ +𝒙𝒑
𝟐
∙ +𝒙𝒑
𝑲 ∙ =
Estimation on pixel p
Repeat for every pixel
Coarse sketch
Coarse Sketch Generation
Step 3: Apply Linear Mapping to Sketches
𝑬𝒑

6. Because: coarse sketch image is not natural.
𝑤(𝑝, 𝑟) is not a good similarity measurement
between p and r.
Denoising: State-of-the-art Image Denoising Algorithms
Coarse sketch
Nonlocal Means (NLM)
p
r
𝑆𝑝
𝑁𝐿𝑀 = 𝐸𝑟
𝑤(𝑝, 𝑟)
+ ⋯
For all pixels in the neighbor of p: Ψ𝑝
Little improvement
After NLM
q
𝐸𝑞
𝑤(𝑝, 𝑞)
+
[NLM] A. Buades, B. Coll and J.-M. Morel, A non-local algorithm for image denoising, CVPR 2005.

7. Motivation – BM3D
BM3D groups correlated patches in the noisy image to create multiple estimations.
Our idea for sketch denoising: group highly similar sketch estimations.
How BM3D works
[BM3D] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3D transform-
domain collaborative filtering,” IEEE Trans. Image Process., vol. 16, no. 8, pp. 2080-2095, August 2007.

8. 𝑤(𝑝, 𝑞) ∙
Proposed Spatial Sketch Denoising Algorithm (SSD)
Test photo
q
∆𝒒
𝟏
, 𝒙𝒒
𝟏
𝑺𝒒+∆𝒒
𝟏
p
Matched sketch
𝑺𝒑+∆𝒒
𝟏
Similarly
∆𝒒
𝟐, 𝒙𝒒
𝟐
𝑺𝒑+∆𝒒
𝟐
𝑺𝒒+∆𝒒
𝟐
∆𝒒
𝑲, 𝒙𝒒
𝑲
𝑺𝒒+∆𝒒
𝑲
𝑺𝒑+∆𝒒
𝑲
𝒙𝒒
𝟏 ∙ +𝒙𝒒
𝟐 ∙ +𝒙𝒒
𝑲 ∙ =
𝑬𝒑
𝒒
p
Estimations from pixels
in local region Ψ𝑝
r 𝑬𝒑
𝒓
Averaging estimations to
generate output sketch value.
Nonlocal Means (NLM):
𝑆𝑝
𝑁𝐿𝑀 = 𝐸𝑞 + ⋯
𝐸𝑟
𝑤(𝑝, 𝑟) ∙
+
Proposed SSD:
𝑆𝑝
𝑆𝑆𝐷
= + ⋯
+
1 ∙𝑬𝒑
𝒒
1 ∙ 𝑬𝒑
𝒓

9. p
Proposed SSD is robust to Ψ𝑝
Input 5x5 local region
𝜳𝒑 = 𝟐𝟓
11x11 local region
𝜳𝒑 = 𝟏𝟐𝟏
17x17 local region
𝜳𝒑 = 𝟐𝟖𝟗
23x23 local region
𝜳𝒑 = 𝟓𝟐𝟗
Note: When 𝛹𝑝 is sufficient large (i.e., 𝛹𝑝 >100), the proposed SSD can effectively
suppress noise while preserving facial details like the tiny eye reflections (see close-ups).
Robustness to the region size Ψ𝑝 - the only parameter involved