Super Resolution in Digital Image processing

2,255 views

Published on

Super Resolution in Digital Image processing

Published in: Education
  • Be the first to comment

  • Be the first to like this

Super Resolution in Digital Image processing

  1. 1. Super-Resolution
  2. 2. Super-resolution • convolutions, blur, and de-blurring • Bayesian methods • Wiener filtering and Markov Random Fields • sampling, aliasing, and interpolation • multiple (shifted) images • prior-based methods • MRFs • learned models • domain-specific models (faces)- Gary 3/7/2003 Super-Resolution 2
  3. 3. Linear systems Basic properties • homogeneity T[a X] T[X1+X2] • additivity = a T[X] = T[X1]+T[X2] • superposition T[aX1+bX2] = aT[X1]+bT[X2] Linear system ⇔ superposition Examples: • matrix operations (additions, multiplication) • convolutions 3/7/2003 Super-Resolution 3
  4. 4. Signals and linear operators Continuous Discrete Vector form I(x) I[k] or Ik I Discrete linear operator y=Ax Continuous linear operator: convolution integral g(x) = s h(ξ,x) f(ξ) dξ, h(ξ,x): impulse response g(x) = s h(ξ-x) f(ξ) dξ= [f * h](x) shift invariant 3/7/2003 Super-Resolution 4
  5. 5. 2-D signals and convolutions Continuous Discrete I(x,y) I[k,l] or Ik,l 2-D convolutions (discrete) g[k,l] = ∑m,n f[m,n] h[k-m,l-n] = ∑m,n f[m,n] h1[k-m]h2[l-n] separable Gaussian kernel is separable and radial h(x,y) = (2πσ2)-1exp-(x2+y2)/σ2 3/7/2003 Super-Resolution 5
  6. 6. Convolution and blurring 3/7/2003 Super-Resolution 6
  7. 7. Separable binomial low-pass filter 3/7/2003 Super-Resolution 7
  8. 8. Fourier transforms Project onto a series of complex sinusoids F[m,n] = ∑k f[k,l] e-i 2π(km+ln) Properties: • shifting g(x-x0) ⇔ exp(-i 2πfxx0)G(fx) • differentiation dg(x)/dx ⇔ i 2πfxG(fx) • convolution 3/7/2003 [f * g](x) ⇔ [F G] (fx) Super-Resolution 8
  9. 9. Blurring examples Increasing amounts of blur + Fourier transform 3/7/2003 Super-Resolution 9
  10. 10. Sharpening Unsharp mask (darkroom photography): • remove some low-frequency content y’ = y + s (y – g * y) spatial (blur, sharp) 3/7/2003 Super-Resolution freq (blur,sharp) 10
  11. 11. Sharpening - result Unsharp mask: original, blur (σ=1), sharp(s=0, 1, 2) 3/7/2003 Super-Resolution 11
  12. 12. Deconvolution Filter by inverse of blur • easiest to do in the Fourier domain • problem: high-frequency noise amplification 3/7/2003 Super-Resolution 12
  13. 13. Bayesian modeling Use prior model for image and noise • y = g * x + n, x is original, y is blurred • p(x|y) = p(y|x)p(x) = exp(-|y – g*x|2/2σn-2) exp(-|x|2/2σx-2) • -log p(x|y) ∝ |y – g*x|2σn-2 + |x|2σx-2 where the norm || is summed squares over all pixels 3/7/2003 Super-Resolution 13
  14. 14. Parseval’s Theorem Energy equivalence in spatial ↔ frequency domain • |x|2 = |F(x)|2 • -log p(x|y) ∝ |Y(f) – G(f)X(f)|2σn-2 + |X(f)|2σx-2 • least squares solution (∂/∂X = 0) X(f) = G(f)Y(f) / [G2(f) + σn2/σx2] 3/7/2003 Super-Resolution 14
  15. 15. Wiener filtering Optimal linear filter given noise and signal statistics • X(f) = G(f)Y(f) / [G2(f) + σn2/σx2] • low frequencies: X(f) ≈ G-1(f)Y(f) boost by inverse gain (blur) X(f) ≈ G(f) σn-2σx2 Y(f) • high frequencies: attenuate by blur (gain) 3/7/2003 Super-Resolution 15
  16. 16. Wiener filtering – white noise prior Assume all frequencies equally likely • p(x) ~ N(0,σx2) • X(f) = G(f)Y(f) / [G2(f) + σn2/σx2] • solution is too noisy in high frequencies 3/7/2003 Super-Resolution 16
  17. 17. Wiener filtering – pink noise prior Assume frequency falloff (“natural statistics”) • p(X(f)) ~ N(0,|f|-βσx2) • X(f) = G(f)Y(f) / [G2(f) + |f|βσn2/σx2] • greater attenuation at high frequencies G(f) 3/7/2003 H(f) Super-Resolution 17
  18. 18. Markov Random Field modeling Use spatial neighborhood prior for image i • -log p(x) = ∑ij∈Cρ(xi-xj) where ρ(v) is a robust norm: • • • • j ρ(v) = v2: quadratic norm ↔ pink noise ρ(v) = |v|: total variation (popular with maths) ρ(v) = |v|β: natural statistics ρ(v) = v2,|v|: Huber norm [Schultz, R.R.; Stevenson, IEEE TIP, 1996] 3/7/2003 Super-Resolution 18
  19. 19. MRF estimation Set up discrete energy (quadratic or non-) • -log p(x|y) ∝ σn-2 |y – Gx|2 + ∑ij∈Cρ(xi-xj) where G is sparse convolution matrix • quadratic: solve sparse linear system • non-quadratic: use sparse non-linear least squares (Levenberg-Marquardt, gradient descent, conjugate gradient, …) 3/7/2003 Super-Resolution 19
  20. 20. Sampling a signal • sampling: • creating a discrete signal from a continuous signal • downsampling (decimation) • subsampling a discrete signal • upsampling • introducing zeros between samples • aliasing • two sampled signals that differ in their original form (many → one mapping) 3/7/2003 Super-Resolution 20
  21. 21. Sampling interpolation 3/7/2003 Super-Resolution 21
  22. 22. Nyquist sampling theorem Signal to be (down-) sampled must have a bandwidth no larger than twice the sample frequency ωs = 2π / ns > 2 ω0 3/7/2003 Super-Resolution 22
  23. 23. Box filter (top hat) 3/7/2003 Super-Resolution 23
  24. 24. Ideal low-pass filter 3/7/2003 Super-Resolution 24
  25. 25. Simplified camera optics 1. 2. 3. 4. Blur = pill-box*Bessel2 (diffr.) ≈ Gaussian Integrate = box filter Sample = produce single digital sample Noise = additive white noise 3/7/2003 Super-Resolution 25
  26. 26. Aliasing Aliasing (“jaggies” and “crawl”) is present if blur amount < sampling (σ = 1) • shift each image in previous pipeline by 1 3/7/2003 Super-Resolution 26
  27. 27. Aliasing - less Less aliasing (“jaggies” and “crawl”) is present if blur amount ~ sampling (σ = 2) • shift each image in previous pipeline by 1 3/7/2003 Super-Resolution 27
  28. 28. Multi-image super-resolution Exploit aliasing to recover frequencies above Nyquist cutoff ∀ ∑kσn-2 |yk – Gkx|2 + ∑ij∈Cρ(xi-xj) where Gk are sparse convolution matrices • quadratic: solve sparse linear system • non-quadratic: use sparse non-linear least squares (Levenberg-Marquardt, gradient descent, conjugate gradient, …) • projection onto convex sets (POCS) 3/7/2003 Super-Resolution 28
  29. 29. Multi-image super-resolution Need: • accurate (sub-pixel) motion estimates (Wednesday’s lecture) • accurate models of blur (pre-filtering) • accurate photometry • no (or known) non-linear pre-processing (Bayer mosaics) • sufficient images and low-noise relative to amount of aliasing 3/7/2003 Super-Resolution 29
  30. 30. Prior-based Super-Resolution “Classical” non-Gaussian priors: • robust or natural statistics • maximum entropy (least blurry) • constant colors (black & white images) 3/7/2003 Super-Resolution 30
  31. 31. Example-based Super-Resolution William T. Freeman, Thouis R. Jones, and Egon C. Pasztor, IEEE Computer Graphics and Applications, March/April, 2002 • learn the association between low-resolution patches and high-resolution patches • use Markov Network Model (another name for Markov Random Field) to encourage adjacent patch coherence 3/7/2003 Super-Resolution 31
  32. 32. Example-based Super-Resolution William T. Freeman, Thouis R. Jones, and Egon C. Pasztor, IEEE Computer Graphics and Applications, March/April, 2002 3/7/2003 Super-Resolution 32
  33. 33. References – “classic” Irani, M. and Peleg. Improving Resolution by Image Registration. Graphical Models and Image Processing, 53(3), May 1991, 231-239. Schultz, R.R.; Stevenson, R.L. Extraction of high-resolution frames from video sequences. IEEE Trans. Image Proc., 5(6), Jun 1996, 996-1011. Elad, M.; Feuer, A.. Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images. IEEE Trans. Image Proc., 6(12) , Dec 1997, 16461658. Elad, M.; Feuer, A.. Super-resolution reconstruction of image sequences. IEEE PAMI 21(9), Sep 1999, 817-834. Capel, D.; Zisserman, A.. Super-resolution enhancement of text image sequences. CVPR 2000, I-600-605 vol. 1. Chaudhuri, S. (editor). Super-Resolution Imaging. Kluwer Academic Publishers. 2001. 3/7/2003 Super-Resolution 33
  34. 34. References – strong priors Freeman, W.T.; Pasztor, E.C.. Learning low-level vision, CVPR 1999, 182-1189 vol.2 William T. Freeman, Thouis R. Jones, and Egon C. Pasztor, Example-based super-resolution, IEEE Computer Graphics and Applications, March/April, 2002 Baker, S.; Kanade, T. Hallucinating faces. Automatic Face Gesture Recognition, 2000, 83-88. Ce Liu; Heung-Yeung Shum; Chang-Shui Zhang. A two-step approach to hallucinating faces: global parametric model and local nonparametric model. CVPR 2001. I-192-8. 08/03/2014 Super-Resolution 34

×