T.Q. Pham, M. Bezuijen, L.J. van Vliet, K. Schutte, and C.L. Luengo Hendriks, SPIE vol. 5817 Visual Information Processing XIV San Jose, CA, 2005.

1. Conference 5817-15: Visual
Information Processing XIV
Performance of
Optimal Registration Estimator
Tuan Pham
tuan@qi.tnw.tudelft.nl
Quantitative Imaging Group
Delft University of Technology
The Netherlands

2. Outline: accurate registration
•Image registration
–Bias of gradient-based shift estimation
–Bias correction by iterative shift estimation
•Cramer-Rao bound of registration:
–Optimal shift estimation
–Optimal 2D projective registration
•Applications:
–Panoramic image reconstruction
–Super-resolution
© 2004 Tuan Pham tuan@qi.tnw.tudelft.nl 2

3. Shift accuracy and precision
256x256 image at SNR=15dB
NCC: -0.16
NCC
Normalized
MAD
cross-correlation -0.18 Taylor
phase
MAD:
-0.2 true shift
Minimum
∆y = -0.2477
absolute
difference -0.22
Taylor: -0.24
Local Taylor
expansion -0.26
Phase:
-0.28 100 noise
Plane fit of
phase difference realizations
-0.3
-0.26 -0.24 -0.22 -0.2 -0.18 -0.16
© 2004 Tuan Pham ∆x = -0.2286
tuan@qi.tnw.tudelft.nl 3

4. 1D Taylor shift estimator
•For a small shift vx: (vx < 1 pixel)
∂s1 1 2 ∂ 2s1
s2 ( x ) = s1( x + v x ) = s1( x ) + v x + vx + ... ε
∂x 2! ∂x 2
Minimize mean squared error ε over a neighborhood S:
2
1  ∂s1 
MSE = ∑  s2 ( x ) − s1( x ) − v x
N S  ∂x 
yields a least-squares solution:
∂s1
∑ ( s2 ( x ) − s1( x )) ∂x
vx =
LS S
2
 ∂s1 
∑  ∂x 
S  
© 2004 Tuan Pham tuan@qi.tnw.tudelft.nl 4

5. Biases of Taylor method
•Bias due to truncation of Taylor expansion:
∂s
–Least squares solves ε = s2 ( x ) − s1( x ) − v x 1 = 0 but ε ≠ 0
∂x
•Bias due to noise: *
–Noise modifies signals:
s1 ( x) = s1 ( x) + n1 ( x)
s* ( x) = s1 ( x + ∆x) + n2 ( x)
2
•Total bias under brightness consistency:
2
 ∂n1  ∂ 3s1 ∂s1 ∂ 5s1 ∂s1
∑  ∂x 
S   1 3 ∑ ∂x 3 ∂x 1 5 ∑ ∂x 5 ∂x
bias = v x 2 2
+ vx S
2
+ vx S
2
+ ...
 ∂s1   ∂n  3!  ∂s1  5!  ∂s1 
∑  ∂x  S  ∂x 
S  
+ ∑ 1  ∑  ∂x 
S  
∑  ∂x 
S  
© 2004 Tuan Pham tuan@qi.tnw.tudelft.nl 5

6. Iterative bias correction
•Since bias = Av x + Bv x + Cv x + ...
3 5
bias → 0 when vx → 0
•Iterative shift refinement:
Initialization: ii = 0, Δx = 0, s2(0) = s2
Shift estimation: Δx(ii) = findshift ( s1 , s2(ii) )
Update displacement: Δx += Δx(ii)
Shift correction: s2(ii+1) = warp ( s2 , Δx )
Δx(ii) <
Break condition: threshold No, loop again
© 2004 Tuan Pham tuan@qi.tnw.tudelft.nl Yes, finish 6

7. Performance of iterative shift estimation
precision of shift estimation
0.58
Taylor
Bias = 0 but the iterative
true shift
0.56
estimated shift
0.54 is not precise:
0.52 stdev(Vx)> 0
∆y
0.5
0.48
100 noise
0.46
realizations
0.44
0.42
0.44 0.46 0.48 0.5 0.52 0.54 0.56 0.58
© 2004 Tuan Pham ∆x
tuan@qi.tnw.tudelft.nl 7

8. Cramér-Rao bound on registration
ˆ
•A lower bound on the variance of any unbiased estimatorm :
E (mi − mi )2  ≥ Fii−1(m)
 ˆ 
where m = [m1 m2 … mn]T & F is the Fisher Information Matrix (FIM).
•For 2D shift estimation: I2(x,y) = I1(x+vx,y+vy) the FIM is:
 ∑ I2 ∑ IxIy 
1  S x 
F( v ) = 2  S
σ n ∑ IxIy ∑ I2 
S
 S
y


where Ix Iy are image derivatives, σ n is noise variance
2
var(v x ) ≥ F111 = σ n ∑ I2 Det ≈ σ n
− 2 2
∑I 2
•Registration variance: S
y
S
x
var(v y ) ≥ F221 = σ n ∑ I2 Det ≈ σ n
− 2
x
2
∑I 2
y
S S
where Det = determinant ( F )
© 2004 Tuan Pham tuan@qi.tnw.tudelft.nl 8

9. Iterative shift estimator is optimal
0.55 σnoise=5 0.06
Cramer-Rao Lower Bound
0.5 without pre-smoothing
0.05 adaptive pre-smoothing σ = σ /20
s n
0.45
0.45 0.5 0.55
0.04
0.55 σnoise=10
std(v )
x
0.5 0.03
0.45
0.45 0.5 0.55 0.02
0.55 σnoise=20
0.01
0.5
0.45 0
0.45 0.5 0.55 0 10 20 30 40
σnoise
© 2004 Tuan Pham tuan@qi.tnw.tudelft.nl 9

10. 2D projective registration
• Transformation: p ' = M 2 D × p
 u   m0 m1 m2   x  x' = u / w
 v  = m m4 m5  ×  y  y'= v/w
   3   
 w  m6
   m7 1  1 
  
• Levenberg-Marquardt
iteratively minimizes:
E = ∑ [I '( pi ') − I ( pi )]2
i
© 2004 Tuan Pham tuan@qi.tnw.tudelft.nl mosaic image 10

11. 2D projective registration by
Levenberg-Marquardt optimization
Original Distorted image Register after 19 iters
W
a
r
p
e
r
r
o
r
© 2004 TuanRMSE
11 iters: Pham = 10.0 14tuan@qi.tnw.tudelft.nl
iters: RMSE = 1.5 19 iters: RMSE = 1.3
11

12. Levenberg-Marquardt is locally optimal
-3 -3
x 10 x 10
1 1 0.1
std( m2 )
std( m3 )
std( m1 )
0.5 0.5 0.05
0 0 0
0 -4 10 20 0 -3 10 20 0 10 20
x 10 x 10
5 1 0.1
std( m4 )
std( m5 )
std( m6 )
0.5 0.05
0 0 0
0 -6 10 20 0 -6 10 20 0 10 20
x 10 x 10
4 4
std( m8 )
std( m7 )
Cramer-Rao
2 2
estimated
0 0
0 10 20 0 10 20
σnoise
© 2004 Tuan Pham tuan@qi.tnw.tudelft.nl 12

13. Application: Panoramic image reconstruction
128 x 128 x 100 long-IR sequence
© 2004 Tuan Pham tuan@qi.tnw.tudelft.nl 13

14. Application: Super-resolution
Low-resolution input 2x High-Resolution output
© 2004 Tuan Pham tuan@qi.tnw.tudelft.nl 14

15. Conclusion and Future Research
•Conclusion
– All conventional shift estimators are inaccurate due to bias
– Iterative Taylor shift estimator reaches Cramer-Rao bound
– Good super-resolution is achieved with the iterative registration
•Future research
– Precision of optic flow for certain neighborhood size
– Improving registration of JPEG compressed images
© 2004 Tuan Pham tuan@qi.tnw.tudelft.nl 15

16. ?
© 2004 Tuan Pham tuan@qi.tnw.tudelft.nl 16

17. Image registration
•3-step approach:
Set of LR Irregular samples of LR Filtered Deblurred
images on a HR grid HR image HR image
images
...
Regis-
Fusion Deblur
tration
. ..
•Registration:
–Shift estimation
–2D projective
–Optical flow
•Inaccurate registration → poor super-resolution
© 2004 Tuan Pham tuan@qi.tnw.tudelft.nl 17