Your SlideShare is downloading. ×
Super Resolution in Digital Image processing
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Super Resolution in Digital Image processing

509
views

Published on

Super Resolution in Digital Image processing

Super Resolution in Digital Image processing

Published in: Education

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
509
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
33
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Super-Resolution
  • 2. Super-resolution • convolutions, blur, and de-blurring • Bayesian methods • Wiener filtering and Markov Random Fields • sampling, aliasing, and interpolation • multiple (shifted) images • prior-based methods • MRFs • learned models • domain-specific models (faces)- Gary 3/7/2003 Super-Resolution 2
  • 3. Linear systems Basic properties • homogeneity T[a X] T[X1+X2] • additivity = a T[X] = T[X1]+T[X2] • superposition T[aX1+bX2] = aT[X1]+bT[X2] Linear system ⇔ superposition Examples: • matrix operations (additions, multiplication) • convolutions 3/7/2003 Super-Resolution 3
  • 4. Signals and linear operators Continuous Discrete Vector form I(x) I[k] or Ik I Discrete linear operator y=Ax Continuous linear operator: convolution integral g(x) = s h(ξ,x) f(ξ) dξ, h(ξ,x): impulse response g(x) = s h(ξ-x) f(ξ) dξ= [f * h](x) shift invariant 3/7/2003 Super-Resolution 4
  • 5. 2-D signals and convolutions Continuous Discrete I(x,y) I[k,l] or Ik,l 2-D convolutions (discrete) g[k,l] = ∑m,n f[m,n] h[k-m,l-n] = ∑m,n f[m,n] h1[k-m]h2[l-n] separable Gaussian kernel is separable and radial h(x,y) = (2πσ2)-1exp-(x2+y2)/σ2 3/7/2003 Super-Resolution 5
  • 6. Convolution and blurring 3/7/2003 Super-Resolution 6
  • 7. Separable binomial low-pass filter 3/7/2003 Super-Resolution 7
  • 8. Fourier transforms Project onto a series of complex sinusoids F[m,n] = ∑k f[k,l] e-i 2π(km+ln) Properties: • shifting g(x-x0) ⇔ exp(-i 2πfxx0)G(fx) • differentiation dg(x)/dx ⇔ i 2πfxG(fx) • convolution 3/7/2003 [f * g](x) ⇔ [F G] (fx) Super-Resolution 8
  • 9. Blurring examples Increasing amounts of blur + Fourier transform 3/7/2003 Super-Resolution 9
  • 10. Sharpening Unsharp mask (darkroom photography): • remove some low-frequency content y’ = y + s (y – g * y) spatial (blur, sharp) 3/7/2003 Super-Resolution freq (blur,sharp) 10
  • 11. Sharpening - result Unsharp mask: original, blur (σ=1), sharp(s=0, 1, 2) 3/7/2003 Super-Resolution 11
  • 12. Deconvolution Filter by inverse of blur • easiest to do in the Fourier domain • problem: high-frequency noise amplification 3/7/2003 Super-Resolution 12
  • 13. Bayesian modeling Use prior model for image and noise • y = g * x + n, x is original, y is blurred • p(x|y) = p(y|x)p(x) = exp(-|y – g*x|2/2σn-2) exp(-|x|2/2σx-2) • -log p(x|y) ∝ |y – g*x|2σn-2 + |x|2σx-2 where the norm || is summed squares over all pixels 3/7/2003 Super-Resolution 13
  • 14. Parseval’s Theorem Energy equivalence in spatial ↔ frequency domain • |x|2 = |F(x)|2 • -log p(x|y) ∝ |Y(f) – G(f)X(f)|2σn-2 + |X(f)|2σx-2 • least squares solution (∂/∂X = 0) X(f) = G(f)Y(f) / [G2(f) + σn2/σx2] 3/7/2003 Super-Resolution 14
  • 15. Wiener filtering Optimal linear filter given noise and signal statistics • X(f) = G(f)Y(f) / [G2(f) + σn2/σx2] • low frequencies: X(f) ≈ G-1(f)Y(f) boost by inverse gain (blur) X(f) ≈ G(f) σn-2σx2 Y(f) • high frequencies: attenuate by blur (gain) 3/7/2003 Super-Resolution 15
  • 16. Wiener filtering – white noise prior Assume all frequencies equally likely • p(x) ~ N(0,σx2) • X(f) = G(f)Y(f) / [G2(f) + σn2/σx2] • solution is too noisy in high frequencies 3/7/2003 Super-Resolution 16
  • 17. Wiener filtering – pink noise prior Assume frequency falloff (“natural statistics”) • p(X(f)) ~ N(0,|f|-βσx2) • X(f) = G(f)Y(f) / [G2(f) + |f|βσn2/σx2] • greater attenuation at high frequencies G(f) 3/7/2003 H(f) Super-Resolution 17
  • 18. Markov Random Field modeling Use spatial neighborhood prior for image i • -log p(x) = ∑ij∈Cρ(xi-xj) where ρ(v) is a robust norm: • • • • j ρ(v) = v2: quadratic norm ↔ pink noise ρ(v) = |v|: total variation (popular with maths) ρ(v) = |v|β: natural statistics ρ(v) = v2,|v|: Huber norm [Schultz, R.R.; Stevenson, IEEE TIP, 1996] 3/7/2003 Super-Resolution 18
  • 19. MRF estimation Set up discrete energy (quadratic or non-) • -log p(x|y) ∝ σn-2 |y – Gx|2 + ∑ij∈Cρ(xi-xj) where G is sparse convolution matrix • quadratic: solve sparse linear system • non-quadratic: use sparse non-linear least squares (Levenberg-Marquardt, gradient descent, conjugate gradient, …) 3/7/2003 Super-Resolution 19
  • 20. Sampling a signal • sampling: • creating a discrete signal from a continuous signal • downsampling (decimation) • subsampling a discrete signal • upsampling • introducing zeros between samples • aliasing • two sampled signals that differ in their original form (many → one mapping) 3/7/2003 Super-Resolution 20
  • 21. Sampling interpolation 3/7/2003 Super-Resolution 21
  • 22. Nyquist sampling theorem Signal to be (down-) sampled must have a bandwidth no larger than twice the sample frequency ωs = 2π / ns > 2 ω0 3/7/2003 Super-Resolution 22
  • 23. Box filter (top hat) 3/7/2003 Super-Resolution 23
  • 24. Ideal low-pass filter 3/7/2003 Super-Resolution 24
  • 25. Simplified camera optics 1. 2. 3. 4. Blur = pill-box*Bessel2 (diffr.) ≈ Gaussian Integrate = box filter Sample = produce single digital sample Noise = additive white noise 3/7/2003 Super-Resolution 25
  • 26. Aliasing Aliasing (“jaggies” and “crawl”) is present if blur amount < sampling (σ = 1) • shift each image in previous pipeline by 1 3/7/2003 Super-Resolution 26
  • 27. Aliasing - less Less aliasing (“jaggies” and “crawl”) is present if blur amount ~ sampling (σ = 2) • shift each image in previous pipeline by 1 3/7/2003 Super-Resolution 27
  • 28. Multi-image super-resolution Exploit aliasing to recover frequencies above Nyquist cutoff ∀ ∑kσn-2 |yk – Gkx|2 + ∑ij∈Cρ(xi-xj) where Gk are sparse convolution matrices • quadratic: solve sparse linear system • non-quadratic: use sparse non-linear least squares (Levenberg-Marquardt, gradient descent, conjugate gradient, …) • projection onto convex sets (POCS) 3/7/2003 Super-Resolution 28
  • 29. Multi-image super-resolution Need: • accurate (sub-pixel) motion estimates (Wednesday’s lecture) • accurate models of blur (pre-filtering) • accurate photometry • no (or known) non-linear pre-processing (Bayer mosaics) • sufficient images and low-noise relative to amount of aliasing 3/7/2003 Super-Resolution 29
  • 30. Prior-based Super-Resolution “Classical” non-Gaussian priors: • robust or natural statistics • maximum entropy (least blurry) • constant colors (black & white images) 3/7/2003 Super-Resolution 30
  • 31. Example-based Super-Resolution William T. Freeman, Thouis R. Jones, and Egon C. Pasztor, IEEE Computer Graphics and Applications, March/April, 2002 • learn the association between low-resolution patches and high-resolution patches • use Markov Network Model (another name for Markov Random Field) to encourage adjacent patch coherence 3/7/2003 Super-Resolution 31
  • 32. Example-based Super-Resolution William T. Freeman, Thouis R. Jones, and Egon C. Pasztor, IEEE Computer Graphics and Applications, March/April, 2002 3/7/2003 Super-Resolution 32
  • 33. References – “classic” Irani, M. and Peleg. Improving Resolution by Image Registration. Graphical Models and Image Processing, 53(3), May 1991, 231-239. Schultz, R.R.; Stevenson, R.L. Extraction of high-resolution frames from video sequences. IEEE Trans. Image Proc., 5(6), Jun 1996, 996-1011. Elad, M.; Feuer, A.. Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images. IEEE Trans. Image Proc., 6(12) , Dec 1997, 16461658. Elad, M.; Feuer, A.. Super-resolution reconstruction of image sequences. IEEE PAMI 21(9), Sep 1999, 817-834. Capel, D.; Zisserman, A.. Super-resolution enhancement of text image sequences. CVPR 2000, I-600-605 vol. 1. Chaudhuri, S. (editor). Super-Resolution Imaging. Kluwer Academic Publishers. 2001. 3/7/2003 Super-Resolution 33
  • 34. References – strong priors Freeman, W.T.; Pasztor, E.C.. Learning low-level vision, CVPR 1999, 182-1189 vol.2 William T. Freeman, Thouis R. Jones, and Egon C. Pasztor, Example-based super-resolution, IEEE Computer Graphics and Applications, March/April, 2002 Baker, S.; Kanade, T. Hallucinating faces. Automatic Face Gesture Recognition, 2000, 83-88. Ce Liu; Heung-Yeung Shum; Chang-Shui Zhang. A two-step approach to hallucinating faces: global parametric model and local nonparametric model. CVPR 2001. I-192-8. 08/03/2014 Super-Resolution 34