T.Q. Pham, L.J. van Vliet, and K. Schutte, SPIE vol. 5672 Image Processing: Algorithms and Systems IV Orlando, FL, 2005

1. Conf. 5672: Image Processing
Algorithms and Systems IV
Influence of Signal-to-Noise Ratio
and Point Spread Function on
Limits of Super-Resolution
Tuan Pham
Quantitative Imaging Group
Delft University of Technology
The Netherlands

2. Super-Resolution: an example
128x128x100 infra-red sequence 4x super-resolution
© 2004 Tuan Pham 2

3. Super-Resolution: an example
Low resolution 4x super-resolution
© 2004 Tuan Pham 3

4. Overview and Goal
System inputs
No. of inputs SNR PSF
Positioning SNR Resolving
limit limit limit
GOAL: Derive the Limits of Super-Resolution given system inputs
© 2004 Tuan Pham 4

5. Limit of registration
• Cramer-Rao Lower Bound for 2D shift: I2(x, y) = I1(x+vx, y+vy) :
var(v x ) ≥ F111 = σ n ∑ I2 Det (F)
− 2
y
S
var(v y ) ≥ F221 = σ n ∑ I2 Det (F)
− 2
x
S
where I x = ∂I / ∂x , I y = ∂I / ∂y , σ n is noise variance, and F is the
2
Fisher Information Matrix:
⎡ ∑ I2 ∑I I ⎤
1 ⎢ S x x y
⎥
F( v ) = 2 ⎢ S
σ n ∑ IxIy
⎢S
⎣
∑I ⎥
S
⎥
⎦
2
y
• Optimal registration is achievable by iterative optimization
• CRLB also exists for more complicated motion models:
- 2D projective - optic flow
© 2004 Tuan Pham 5

6. Noise of HR image after fusion
• Total noise = Intensity noise + Noise due to registration error
μ : zoom factor
μ2 2 N : # of LR images
σ 2
n = σ I
2
+ ∇I σ 2
reg
2
∇I : gradient energy
N
position error distribution
Intensity error distribution
σreg
I
σI
x
local
signal
∂I
σI = σ
∂x
re g
Blurred & mis-registered Noise due to
5x5 box blur, σ reg = 0.2 pixel mis-registration mis-registration → noise
© 2004 Tuan Pham 6

7. The need for deconvolution
• After fusion, the High-Resolution image is still blurry due to:
– Sensor integration blur (severe if high fill-factor)
– Optical blur (severe if high sampling factor)
On-chip microlens of
Sony Super HAD CCD
© 2004 Tuan Pham 7

8. The necessity of aliasing
• Spectrum is cut off beyond fc due to optics → data forever lost
1 1
OTF (sampling factor = 0.25) OTF (sampling factor = 1)
frequency spectra / transfer functions
frequency spectra / transfer functions
STF (fill factor = 1) STF (fill factor = 1)
0.8 Original scene spectrum 0.8 Original scene spectrum
Band−limited spectrum Band−limited spectrum
Aliased image spectrum Sampled image spectrum
0.6 0.6
0.4 0.4
0.2 0.2
0 0
−0.2 −0.2
−0.4 −0.4
0 0.5 1 1.5 0 0.5 1 1.5 2
frequency in unit of sampling frequency (f/fs) frequency in unit of sampling frequency (f/fs)
Aliasing due to No aliasing at
under-sampling (fs < 2fc) critical sampling (fs = 2fc)
© 2004 Tuan Pham 8

9. Limit of deconvolution
• Blur = attenuation of HF spectrum recoverable
• Deconvolution = amplify HF spectrum:
– noise is also amplified → limit the deconvolution PS>PN
• Deconvolution can only recover:
Not
– Spectrum whose signal power > noise power recoverable
resolution factor = 0.44
fusion result
© 2004 Tuan Pham after deconvolution simulated at resolution = 0.44
9

10. SR reconstruction experiment
• Aim: show that the attainable SR factor agrees with the prediction
• Experiment:
– Inputs: sufficient shifted LR images of the Pentagon
– Output: SR image and a measure of SR factor from edge width
64x64 LR input 4xHR after fusion 4xSR after deconvolution
sampling=1/4, fill = 100% BSNR = 20 dB SR factor = 3.4
© 2004 Tuan Pham 10

11. SR reconstruction experiment
• Aim: show that the attainable SR factor agrees with the prediction
• Experiment:
– Inputs: sufficient shifted LR images of the Pentagon
– Output: SR image and a measure of SR factor from edge width
64x64 LR input 4xHR after fusion 4xSR after deconvolution
sampling=1/4, fill = 100% BSNR = 20 dB SR factor = 3.4
© 2004 Tuan Pham 11

12. SR factor at BSNR=20dB
• Consistent results between prediction and measurement:
– Attainable SR factor depends mainly on sampling factor (i.e. level of aliasing)
6 3.2
6 3.0
3.4 2.5
4
SR limit
SR factor
4
1.7
1.9 2
2 1.0
1.0
0 0.6
0 0 0
0 0
0.6
0.5 1
0.5 1 sampling factor
sampling factor fill factor 1 2 (f /2f )
fill factor 1 2 (fs/2fc) s c
Measured SR factor Predicted SR factor
© 2004 Tuan Pham 12

13. Summary
• Limit of Super-Resolution depends on:
– input Signal-to-Noise Ratio
– System’s Point Spread Function and how well it can be estimated
• Procedure for estimating SR factor directly from inputs:
– Measure noise variance from LR images σ I2
– Derive registration error σ reg
2
– Determine SR factor from the Power Spectrum Density (PS > PN)
© 2004 Tuan Pham 13