200081003 Friday Food@IBBT

  • 707 views
Uploaded on

Ghent University …

Ghent University
Image Processing and Interpretation Group
Aleksandra Pizurica
Advances and challenges in image and
video restoration

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
707
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
21
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Advances and challenges in image and video restoration Aleksandra Pizurica Ghent University Image Processing and Interpretation Group
  • 2. Median filter: Reduction of impulse noise impulse noise median over 3x3 Median filter removes isolated noise peaks, without blurring the image IBBT Friday food talk, October 3, 2008 Image and video restoration 2
  • 3. Median filter and reduction of white noise original median over 3x3 For not-isolated noise peaks (e.g., white Gaussian noise) median filter is not very efficient. IBBT Friday food talk, October 3, 2008 Image and video restoration 3
  • 4. Why is denoising important original denoised Not only visual enhancement, but also: automatic processing is facilitated! Example: edge detection IBBT Friday food talk, October 3, 2008 Image and video restoration 4
  • 5. Image restoration example ©Max Planck Institute for Biophysical Chemistry State of the art image restoration methods iterate wavelet domain denoising and Fourier domain deconvolution. From now on we focus on the denoising step [F. Rooms et al; Journal of Microscopy 2005] IBBT Friday food talk, October 3, 2008 Image and video restoration 5
  • 6. Overview • Wavelet domain image restoration • Gain from using other wavelet-like representations • Medical applications: MRI, CT, OCT • On noise and blur estimation • Video denoising and advances in 3D video IBBT Friday food talk, October 3, 2008 Image and video restoration 6
  • 7. Discrete Wavelet Transform (DWT) DWT algorithm: a filter bank iterated on the lowpass output wavelet coefficients highpass w j+1 g 2 w j+2 sj g 2 w j+3 h 2 g 2 lowpass s j+1 h 2 s j+1 h 2 s j+3 scaling coefficients IBBT Friday food talk, October 3, 2008 Image and video restoration 7
  • 8. Choosing a wavelet: Nv, support size K, symmetry ∞ Nv - number of vanishing moments: ∫−∞ t k ψ(t )dt = 0, 0 ≤ k ≤ Nv −1 A tradeoff: K ≥ 2Nv-1 Symmlets (Daubechies) Daubechies wavelets dbNv: 1.5 1.5 1.5 1.5 ϕ ψ ϕ 1 ψ 1 1 1 0.5 sym8 0.5 0.5 db1 0 0 0 0.5 K=1 -0.5 -0.5 -1 -1 -0.5 -5 0 5 0 5 10 15 0 -1.5 -0.5 0 0.5 1 1.5 -1.5 -1 -0.5 0 0.5 Biorthogonal wavelets ϕ ψ 1.5 2 0.8 1 1 K=3 1 0.6 ϕ 0.5 ψ db2 0.5 0 0.4 0 0 -1 0.2 -0.5 -0.5 -2 0 1 2 3 -1 0 1 2 0 -1 1.5 -5 0 5 -4 -2 0 2 4 1 ϕ 1 ψ 2 1.5 ~ ϕ 2 ~ ψ 0.5 1 db8 0.5 K=15 0 1 0 0.5 -0.5 -1 0 0 -1 -0.5 -2 -0.5 -4 -2 0 2 4 0 5 10 15 -5 0 5 -5 0 5 IBBT Friday food talk, October 3, 2008 Image and video restoration 8
  • 9. Two dimensional DWT APPROXIMATION scaling coefficients Wavelet coefficient values DETAIL IMAGES wavelet coefficients 0 Peaks indicate image edges IBBT Friday food talk, October 3, 2008 Image and video restoration 9
  • 10. Noise in the wavelet domain Noise-free reference Wavelet coefficient values 0 Peaks indicate image edges IBBT Friday food talk, October 3, 2008 Image and video restoration 10
  • 11. Marginal priors: Generalized Laplacian Generalized Laplacian (generalized Gaussian) distribution noise-free histogram f(y)=Aexp(-|y /s|ν ) s: scale parameter ν: shape parameter (0 ≤ ν ≤ 1) noisy Parameters accurately estimated from a signal corrupted by additive white Gaussian noise Often yields complicated expressions Extension to higher dimensions (joint histograms) difficult IBBT Friday food talk, October 3, 2008 Image and video restoration 11
  • 12. Gaussian Scale Mixture (GSM) models Efficient for modelling joint histograms wavelet coefficient Gaussian of the neighboring wavelet coefficients random variable f(y) y= zu f(u) y u z: mixture variable, random multiplier A state-of the art denoiser for many years BLS-GSM: Bayesian Least Squares estimator using GSM prior [Portilla et al, IEEE TIP’03] ∞ x = y + n = zu + n E ( y | x ) = ∫−∞ zE ( y | x, z ) f ( z )dz zC u C u , Cn : signal and noise noise E ( y | x, z ) = x zC u + C n covariances IBBT Friday food talk, October 3, 2008 Image and video restoration 12
  • 13. Locally adaptive denoising ProbShrink [Pizurica&Philips, IEEE TIP 2006] LSAI – Local Spatial Activity Indicator ESTIMATE LSAI H1 signal of interest present yl OBSERVATION H0 signal of interest absent LSAI zl ηξµ y = β + n, ˆ β = P( H1 | y, z ) y = y 1 + ηξµ f ( y | H1 ) P ( H1 ) η= f ( z | H1 ) µ= f ( y | H0 ) ξ= P( H 0 ) f (z | H0 ) f(z|H0) f(y|H0) P(H0) f(y|H1) f(z|H1) P(H1) noisy coefficient y LSAI z subband statistics IBBT Friday food talk, October 3, 2008 Image and video restoration 13
  • 14. Locally adaptive denoising: ProbShrink… [Pizurica&Philips, IEEE TIP 2006] IBBT Friday food talk, October 3, 2008 Image and video restoration 14
  • 15. ProbShrink for correlated noise… local window  X 22  20 40  X    23  60 80 100 X22 X23 120 140 vector of coefficients 160 180 50 100 150 H0 H1 X23 H1 X22 X22 [B. Goosens, A. Pizurica, W. Philips; IEEE TIP 2008, in press] IBBT Friday food talk, October 3, 2008 Image and video restoration 15
  • 16. … ProbShrink for correlated noise [B. Goosens, A. Pizurica, W. Philips; IEEE TIP 2008, in press] IBBT Friday food talk, October 3, 2008 Image and video restoration 16
  • 17. Denoising by singularity detection Input signal Rate of increase of the modulus of the wavelet transform across scales is w1 wavelet coefficients proportional to the local Lipschitz regularity w2 w3 [Mallat&Zhong, IEEE IT 1992] IBBT Friday food talk, October 3, 2008 Image and video restoration 17
  • 18. Statistics: magnitude and the rate of increase Noise standard deviation=25.5 0.05 150 0.04 noise 100 edges x=1 l Magnitude 0.03 0.02 50 0.01 edges noise x= 0 l 0 -50 0 50 100 150 200 250 300 0 noisy magnitude -3 -2 -1 0 1 2 3 ACR 1 0.8 scale 0.6 noise edges 0.4 Average Cone Ratio – an estimate 0.2 of the local Lipschitz exponent – 0 measures the rate of increase of -4 -2 0 2 4 6 the coefficients across the scales cone of influence ACR [A. Pizurica et al; IEEE TIP 2002] IBBT Friday food talk, October 3, 2008 Image and video restoration 18
  • 19. Inter- and intrascale dependencies • Bivariate models • Hidden Markov Tree models • Markov Random Field models IBBT Friday food talk, October 3, 2008 Image and video restoration 19
  • 20. Statistical modeling: MRF models x0 xMAP neighborhood cliques P(x) Prior model 1   P(x) = exp− ∑VC (x) Example: penalize isolated peaks Z  C∈ς  negative potential clique potentials positive potential IBBT Friday food talk, October 3, 2008 Image and video restoration 20
  • 21. Statistical modeling: MRF models x0 xMAP neighborhood cliques P(x) Prior model Initial edges Iteration 1 Iteration 2 Iteration 3 IBBT Friday food talk, October 3, 2008 Image and video restoration 21
  • 22. MRF based wavelet denoising Original Gamma MAP filter wavelet filter [A. Pizurica et al; ICIP 2001] IBBT Friday food talk, October 3, 2008 Image and video restoration 22
  • 23. Current trends in Bayesian wavelet denoising Gaussian Scale Mixture (GSM) model MRF models [Portilla et al, IEEE TIP 2003] Shortcoming: assumes the same but scaled covariance for the whole subband Fields of GSM [Liu and Simoncelli, Mixture of GSM PAMI’08] (MGSM) Spatially variant GSM [Portilla et al, Spie (SVGSM) 2008] Field of Experts (FoE) [Guerrero-Colon et al, Computationally expensive [Roth and Black, CVPR’05] IEEE TIP,08] GSM in non-overlapping blocks [Tappen, Adelson, Freeman, • Ignores non-local correlations CVPR’07, CVPR’08] • Block size? Mixture of projected GSM Dimension reduction (MPGSM) ? in MGSM [Goossens, in review TIP 2008] IBBT Friday food talk, October 3, 2008 Image and video restoration 23
  • 24. Overview • Wavelet domain image restoration • Gain from using other wavelet-like representations • Medical applications: MRI, CT, OCT • On noise and blur estimation • Video denoising and advances in 3D video IBBT Friday food talk, October 3, 2008 Image and video restoration 24
  • 25. Why other multiresolution representations Classical wavelets are well suited for point-like singularities, but not for curvilinear singularities in images • Poor orientation selectivity; no difference between 45 and -45o • Checkerboard pattern appears also as an artifact in denoising An example of wavelet base functions Many wavelet-like representations with a better orientation selectivity: complex wavelets [Kingsbury, Selesnick] , steerable pyramids [Freeman, Adelson], curvelets [Donoho, Candes], contourlets [Do, Vetterli], … IBBT Friday food talk, October 3, 2008 Image and video restoration 25
  • 26. Curvelet-domain image denoising… Curvelets: specific tiling of the frequency plane: localized + directional IBBT Friday food talk, October 3, 2008 Image and video restoration 26
  • 27. …Curvelet domain image denoising… Wavelet ProbShrink Noisy Image Curvelet Hard Thresholding Curvelet ProbShrink PSNR=29.50dB PSNR=22.16dB PSNR=29.02dB PSNR=30.43dB [L. Tessens, A. Pizurica, W. Philips, J Electr Imag 2008 in press] IBBT Friday food talk, October 3, 2008 Image and video restoration 27
  • 28. Results …Curvelet domain image denoising… Wavelet ProbShrink Curvelet ProbShrink IBBT Friday food talk, October 3, 2008 Image and video restoration 28
  • 29. …Curvelet domain image denoising Wavelet ProbShrink Curvelet ProbShrink IBBT Friday food talk, October 3, 2008 Image and video restoration 29
  • 30. Overview • Wavelet domain image restoration • Gain from using other wavelet-like representations • Medical applications: MRI, CT, OCT • On noise and blur estimation • Video denoising and advances in 3D video IBBT Friday food talk, October 3, 2008 Image and video restoration 30
  • 31. MRI denoising: signal dependent noise p(m) low SNR noisy m (f=0) magnitude contrast high SNR ( f =f1 ) noise-free f f1 m SNR IBBT Friday food talk, October 3, 2008 Image and video restoration 31
  • 32. MRI denosing: algorithm Step 1: Bias removal Square magnitude MRI image – after squaring constant bias, proportional to noise standard deviation. For better results: square root the result before denoising! Step 2: Denoising (coarse-to-fine, empirical density estimation) Mask A noisy detail T? p(z|H1) log( ) p(z|H0) histograms p(z|H0) Coarser, processed detail p(z|H1) [Pizurica et al IEEE TMI 2003] IBBT Friday food talk, October 3, 2008 Image and video restoration 32
  • 33. MRI denosing: some results Noisy image Denoised image Ground truth IBBT Friday food talk, October 3, 2008 Image and video restoration 33
  • 34. 3D MRI volume denoising using 3D dual-tree complex wavelet transform [J. Aelterman et al, EUSIPCO 2008] IBBT Friday food talk, October 3, 2008 Image and video restoration 34
  • 35. Denoising low-dose CT images • Reducing radiation dose increases noise level • Can we use denoising on low dose CT to obtain the same diagnostic quality as in a higher dose CT image? [IBBT Ica4dt project] • Difficulties: - non-stationary correlated noise - Streak artefacts - How to estimate noise IBBT Friday food talk, October 3, 2008 Image and video restoration 35
  • 36. Denoising algorithm segmentation wavelet Inverse wavelet Vector transform transform ProbShrink (WT) (IWT) (Dual-tree H0 complex) H1 H1 [B. Goossens et al, EMBS 2007] IBBT Friday food talk, October 3, 2008 Image and video restoration 36
  • 37. …Results Watershed segmentation denoised IBBT Friday food talk, October 3, 2008 Image and video restoration 37
  • 38. Qualitative Validation CT: psycho-visual experiment … Abdomen/Lung Blur Noise Structure Quality Low-dose CT +++ +++ -- --- +++ +++ + +++ Versatile Probshrink - + + -- + + ++ + Versatile Probshrink2 - -- + + + + ++ - Curvelet Filter - -- --- ++ + ++ -- -- Wavelet GSM Filter + - ++ +++ ++ +++ +++ --- The abdomen image judged better than the original by radiologists! IBBT Friday food talk, October 3, 2008 Image and video restoration 38
  • 39. Optical Coherence Tomography (OCT) images [IBBT Ica4dt project, with AGFA Healthcare] 3D OCT data 2D signal OCT – “echography with light” Noise: speckle similar to that in radar and ultrasound images IBBT Friday food talk, October 3, 2008 Image and video restoration 39
  • 40. Denoising OCT images A developed 3D OCT denoiser combines [IBBT Ica4dt project] • wavelet domain speckle filter • motion compensated video denoising method coarse-to-fine processing 3D OCT data Signal and noise statistics Image:g120406breastseconformolnogelLR 050 Detail:Dx1 Parameter:b=10.3752 0 Image:g120406breastseconformolnogelLR 050 Detail:Dx2 Parameter:a=4.8348 0 0.035 0.2 Gamma, b=10.3752 Laplace, a=4.8348 0.03 0.025 0.15 likelihood p(m|1) likelihood p(m|0) 0.02 0.1 0.015 0.01 0.05 0.005 0 0 50 100 150 magnitude m 200 250 0 0 50 magnitude m 100 150 Video denoising Locally adaptive denoising IBBT Friday food talk, October 3, 2008 Image and video restoration 40
  • 41. Results and evaluation of OCT denoising Noisy Image Our method GTF SAT RKT Lee IBBT Friday food talk, October 3, 2008 Image and video restoration 41
  • 42. Results and evaluation of OCT denoising Noisy Our method Lee SAT [Pizurica et al; CMIR 2008 in press] IBBT Friday food talk, October 3, 2008 Image and video restoration 42
  • 43. Results and evaluation of OCT denoising BLS-GSM input Our method (2D version) [Pizurica et al; CMIR 2008 in press] IBBT Friday food talk, October 3, 2008 Image and video restoration 43
  • 44. Overview • Wavelet domain image restoration • Gain from using other wavelet-like representations • Medical applications: MRI, CT, OCT • On noise and blur estimation • Video denoising and advances in 3D video IBBT Friday food talk, October 3, 2008 Image and video restoration 44
  • 45. Noise variance estimation Block-based Smoothing based Search for blocks of nearly uniform intensity smooth estimate σ Gradient distribution based Wavelet based noise (Rayleigh distr.) HL1 signal+noise use HH1 LH1 this part σ Median{|HH1|} or compensate for σ= ˆ 0.6745 the peak shift IBBT Friday food talk, October 3, 2008 Image and video restoration 45
  • 46. Blur estimation using wavelet coefficients • A well known approach: kurtosis of the wavelet coefficient histogram • An alternative: examine the propagation of the wavelet coefficients across the scales 0.35 0.3 original blurred 0.25 0.2 PDF 0.15 0.1 0.05 0 -4 -2 0 2 4 6 8 Original image Blurred image ACR 1-2 ACR - Average Cone Ratio – an estimate of the local Lipschitz exponent – measures the rate of increase of the coefficients across the scales IBBT Friday food talk, October 3, 2008 Image and video restoration 46
  • 47. Blur estimation using wavelet coefficients blur 6000 6000 reference reference blur 3 blur 5 solid – noise free blur 3 blur 5 5000 5500 blur 7 blur 0 + noise dashed – noisy blur 7 blur 0 + noise blur 3 + noise blur 3 + noise 4000 blur 5 + noise 5000 blur 5 + noise blur 7 + noise blur 7 + noise 3000 4500 2000 4000 Different colors - 1000 3500 different levels of 0 -1 0 1 2 3 4 5 3000 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 blur ACR 2-4 ACR 2-4 IBBT Friday food talk, October 3, 2008 Image and video restoration 47
  • 48. Overview • Wavelet domain image restoration • Gain from using other wavelet-like representations • Medical applications: MRI, CT, OCT • On noise and blur estimation • Video denoising and advances in 3D video IBBT Friday food talk, October 3, 2008 Image and video restoration 48
  • 49. Video denoising Input Noisy 2D Wavelet Noise Frame Transform Estimation Motion Time delay Estimation Recursive Temporal Filtering Time delay Adaptive Spatial Inverse 2D Wavelet Denoised Filtering Transform Frame [V. Zlokolica, A. Pizurica, W. Philips; IEEE TCSVT 2006] IBBT Friday food talk, October 3, 2008 Image and video restoration 49
  • 50. Video denoising: motion estimation… center of the motion block motion direction (smaller amplitude) motion direction (larger amplitude) Accurate motion estimation is essential for video denoising. Also important: reliability of the estimated motion at each point IBBT Friday food talk, October 3, 2008 Image and video restoration 50
  • 51. …Motion compensated video denoising Further development currently within IBBT project ISYSS [V. Zlokolica, A. Pizurica, W. Philips; IEEE TCSVT 2006] IBBT Friday food talk, October 3, 2008 Image and video restoration 51
  • 52. Reusing motion estimator from video codecs • Motion estimators from video codecs tolerate errors cannot be directly used in denoising • Can we still use them with some postprocessing? The core of our approach: • Motion field refinement step • Reliability to motion estimates controls the recursive filter • Competitive with state-of-the art video denoisers [LJ. Jovanov et al; IEEE TCSVT 2008, in press] IBBT Friday food talk, October 3, 2008 Image and video restoration 52
  • 53. Reusing motion estimator from video codecs noise-free input [Balster; TCSVT 2006] [Jovanov; TCSVT 2008] IBBT Friday food talk, October 3, 2008 Image and video restoration 53
  • 54. Denoising and outlier removal in 3D video Time-of-flight camera records simultaneously luminance and depth information Degradations in the depth image: noise, and outliers (similar to impulse noise but in bursts) 3D reconstructions using “surf” in Matlab The biggest errors in the depth measurement are induced by strong ambient light The measured distance is much smaller than the true distance) IBBT Friday food talk, October 3, 2008 Image and video restoration 54
  • 55. Noisy 3D video sequence (luminance and depth) Luminance image contains much less noise Luminance and depth images are correlated Use the luminance information for denoising depth data IBBT Friday food talk, October 3, 2008 Image and video restoration 55
  • 56. Denoised luminance and depth IBBT Friday food talk, October 3, 2008 Image and video restoration 56
  • 57. Acknowledgements Thanks to my colleagues for their contributions • Vladimir Zlokolica (video denoising) • Bart Goossens (removal of correlated noise) • Ljubomir Jovanov (video, 3D video, OCT) • Linda Tessens (curvelets) • Jan Aelterman (MRI denoising) • Filip Rooms (deblurring) • Ewout Vansteenkiste (quality evaluation CT) Related material available at: http://telin.ugent.be/~sanja IBBT Friday food talk, October 3, 2008 Image and video restoration 57