2. Outline
• Problem formulation
• Previous works
• Our System
– Blur identification
– L12TTV deblurring model
– Pixel selection
• Experimental Results
• Conclusions
3. ( , ) ( , ) ( , ) ( , )g x y D B x y l x y n x y
( , )g x y
( , )l x y
( , )B x y
( , )n x y
: observed image
: blur kernels
: latent image
: noise
:down-sample
: convolution
{}D
Problem Formulation
• According to the image formation process, we
have
IEEE Signal Processing Magazine, 2003
4. Previous Works
• Multiframe Super-Resolution [Tsai&Huang 84]
• Fast and Robust Multiframe + L1BTV [Farsiu et al. 04]
• Bandlet Super-Resolution [Mallat 06]
• Edge Statistics [Fattal 07]
• Fast Image/Video Upsampling [Shan et al. 08]
• Sparse Representation [Yang et al. 08]
• Self Similarity [Glanser&Irani 09]
• Local Self Example [Freedman&Fattal 11]
5. Our System: Overview
• What we have to do is to inverse the process!
( , ) ( , ) ( , ) ( , )g x y D B x y l x y n x y
1 1ˆ( , ) ( , ) ( , )l x y B D g x y n x y
7. Our System: Blur Identification
• Different to the previous works to deal with
the single blur kernel (PSF), we deal with
spatially-varying blur condition.
• Let’s model the blurred edges.
( ) ( )l t A u t C
( ) ( ) ( , )f t l t b t
2
2
1
( , ) exp( )
22
t
b t
( )l t
( )f t
A
C
( , )b t
A C
8. Our System: Blur Identification
• Take 1st derivatives:
• Edge Properties:
( ) ( ) ( , )f t l t b t
( ) ( ( ) ( , )) ( , )f t l t b t A b t
t t
max | exp(0) | 1
( )l t
( )f t
A
C
( , )b t
A C
9. Our System: Blur Identification
• We can adopt the local contrast prior:
• We can obtain our blur map:
• Guided Image Filtering
( , ) ( ', ')( , ) ( ', ')
2 2
( , ) ( ', ')
( , ) ( , )
1
( , )
2
( ( , )) ( ( , ))
max min
max
x y x yx y x y
x y x y
f x y f x y
x y
f x y f x y
x y
2 2
( , ) ( ', ')
( , ) ( ', ')( , ) ( ', ')
( ( , )) ( ( , ))
max | ( , ) | 1
( , ) ( , ) 2 ( , )
max
max min
x y x y
x y x yx y x y
f x y f x y
x y A b t
f x y f x y A C C x y
11. Our System: L12TTV Deblurring
• We can firstly interpolate the input image to
desired size. In this paper, we adopt the bicubic
interpolation with a smoothing operation.
However, the edge-directed interpolation
methods can also perform well.
1
( , )
( , ) ( , ) ( , )
( , ) ( , ) ( , )
f x y
D g x y n x y x y
B x y l x y x y
2 2
2 2
1
( , ) exp( )
2 ( , ) 2 ( , )
x y
B x y
x y x y
2D Gaussian-like blurring kernel B(x,y) with its blur level σ(x,y):
12. Our System: L12TTV Deblurring
• We propose a L12TTV deblurring model with
discrete B(x,y) (or ).
2
( , )
( , ) ( , )
2
( , ) ( , )
min | ( , ) ( , ) ( , ) | | ( , ) ( , ) ( , ) |
2
| ( , ) | (1 ) | ( , ) |
l x y
x y x y
x y x y
B x y l x y f x y B x y l x y f x y
s l x y s l x y
: L1 fidelity, L1 data term
: L2 fidelity, L2 data term
: Total variation
: Tikhonov-like regularizer
( , )
| ( , ) ( , ) ( , ) |
x y
B x y l x y f x y
2
( , )
| ( , ) ( , ) ( , ) |
x y
B x y l x y f x y
2
( , )
| ( , ) |
x y
l x y
( , )
| ( , ) |
x y
l x y
( , ) 0.5,1.0,...,4.5x y
13. Our System: L12TTV Deblurring
• It is not easy to solve the L1 fidelity and TV
term. As a result, we approximate L1 fidelity
to v(x,y) and TV term to w(x,y), respectively.
2
, ,
( , )
2
( , ) ( , )
2
( , ) ( , )
2
( , )
min | ( , ) ( , ) ( , ) ( , ) |
2
| ( , ) | | ( , ) ( , ) ( , ) |
2
| ( , ) | | ( , ) ( , ) |
2
(1 ) | ( , ) |
l v w
x y
x y x y
x y x y
x y
B x y l x y f x y v x y
v x y B x y l x y f x y
s
s w x y w x y l x y
s l x y
Equation
14. Our System: L12TTV Deblurring
• We adopt the alternating minimization to solve
equation ( ). Given l(x,y), we can have v(x,y) and
w(x,y):
• Since v(x,y) and w(x,y) are independent, we have:
2
,
( , )
( , ) ( , )
2
( , )
min | ( , ) ( , ) ( , ) ( , ) |
2
| ( , ) | | ( , ) |
| ( , ) ( , ) |
2
v w
x y
x y x y
x y
B x y l x y f x y v x y
v x y s w x y
s
w x y l x y
1 ( , ) ( , ) ( , )
( , ) max | ( , ) ( , ) ( , ) | ,0
| ( , ) ( , ) ( , ) |
B x y l x y f x y
v x y B x y l x y f x y
B x y l x y f x y
1 ( , )
( , ) max | ( , ) | ,0
| ( , ) |
l x y
w x y l x y
l x y
15. Our System: L12TTV Deblurring
• Since v(x,y) and w(x,y) are calculated, we can
obtain l(x,y) from equation ( ) .
• Since B(x,y) is a Gaussian-like blur kernel, we can
solve l(x,y) using 2D-FFTs.
2
, ,
( , )
2
2
( , )
2
( , )
min | ( , ) ( , ) ( , ) ( , ) |
2
| ( , ) ( , ) ( , ) |
2
| ( , ) ( , ) |
2
(1 ) | ( , ) |
l v w
x y
x y
x y
B x y l x y f x y v x y
B x y l x y f x y
s
w x y l x y
s l x y
* * *
1
* *
( , )
( 2 2 )
s F F w kF B F f F B F v
l x y F
s s F F kF B F B
16. Our System: L12TTV Deblurring
• s is the weighting parameter. We let it equal to
the proportion of the Salient pixels inside
image.
• If Sal(x,y) is high, we may meet the texture
areas so as to deblur stronger (on TV term).
2
,
( , ) ( , )
I M
Sal x y Lm x y
18. Our System: Pixel Selection
• We use the blur map as the index and
reconstruct the latent HD image.
ˆ ( , )
( , )
ˆ( , ) ( , )x y
x y
l x y l x y
…
…
33. Conclusions
• We propose a Spatially-Varying scheme to
Super-Resolution for HDTV.
• Different to previous works, we deal with
spatially-varying blur condition.
• We propose a L12TTV deblurring model.
• Pixel Selection ensures our final HD output.
• WeiBo : ShenChihTsung
• Gmail : shenchihtsung@gmail.com
