Image and Video Restoration Advances and Challenges

Advances and challenges in image and
video restoration

Aleksandra Pizurica

Ghent University
Image Processing and Interpretation Group

Median filter: Reduction of impulse noise

impulse noise median over 3x3

Median filter removes isolated noise peaks, without blurring the image

IBBT Friday food talk, October 3, 2008 Image and video restoration 2

Median filter and reduction of white noise

original median over 3x3

For not-isolated noise peaks (e.g., white Gaussian noise) median filter
is not very efficient.


Why is denoising important
original denoised
Not only visual
enhancement, but
also: automatic
processing is
facilitated!

Example:
edge detection


Image restoration example

©Max Planck Institute for Biophysical Chemistry

State of the art image restoration methods iterate wavelet domain denoising and
Fourier domain deconvolution. From now on we focus on the denoising step
[F. Rooms et al; Journal of Microscopy 2005]


Overview

• Wavelet domain image restoration
• Gain from using other wavelet-like representations
• Medical applications: MRI, CT, OCT
• On noise and blur estimation
• Video denoising and advances in 3D video


Discrete Wavelet Transform (DWT)

DWT algorithm: a filter bank iterated on the lowpass output

wavelet coefficients
highpass
w j+1
g 2
w j+2
sj g 2
w j+3
h 2 g 2
lowpass s j+1 h 2
s j+1 h 2 s j+3
scaling coefficients


Choosing a wavelet: Nv, support size K, symmetry
∞
Nv - number of vanishing moments: ∫−∞ t k ψ(t )dt = 0, 0 ≤ k ≤ Nv −1
A tradeoff: K ≥ 2Nv-1
Symmlets (Daubechies)
Daubechies wavelets dbNv: 1.5 1.5

1.5 1.5 ϕ ψ
ϕ 1 ψ 1
1

1 0.5 sym8 0.5
0.5

db1 0
0

0
0.5 K=1 -0.5
-0.5

-1 -1 -0.5
-5 0 5 0 5 10 15
0 -1.5
-0.5 0 0.5 1 1.5 -1.5 -1 -0.5 0 0.5
Biorthogonal wavelets
ϕ ψ
1.5 2
0.8 1
1

K=3
1
0.6 ϕ 0.5 ψ
db2 0.5 0
0.4 0
0 -1
0.2 -0.5

-0.5 -2
0 1 2 3 -1 0 1 2 0 -1
1.5 -5 0 5 -4 -2 0 2 4

1
ϕ 1
ψ 2

1.5
~
ϕ
2
~
ψ
0.5 1

db8 0.5
K=15 0
1
0
0.5

-0.5 -1
0 0

-1 -0.5 -2
-0.5 -4 -2 0 2 4
0 5 10 15 -5 0 5 -5 0 5


Two dimensional DWT

APPROXIMATION
scaling coefficients

Wavelet coefficient values

DETAIL IMAGES
wavelet coefficients
0

Peaks indicate image edges


Noise in the wavelet domain

Noise-free reference

Wavelet coefficient values

0

Peaks indicate image edges


Marginal priors: Generalized Laplacian
Generalized Laplacian (generalized Gaussian) distribution

noise-free histogram
f(y)=Aexp(-|y /s|ν )
s: scale parameter
ν: shape parameter
(0 ≤ ν ≤ 1)

noisy Parameters accurately
estimated from a signal
corrupted by additive
white Gaussian noise

Often yields complicated expressions
Extension to higher dimensions (joint histograms) difficult


Gaussian Scale Mixture (GSM) models
Efficient for modelling joint histograms
wavelet coefficient Gaussian of the neighboring wavelet coefficients
random variable
f(y) y= zu
f(u)

y u
z: mixture variable, random multiplier

A state-of the art denoiser for many years BLS-GSM: Bayesian Least
Squares estimator using GSM prior [Portilla et al, IEEE TIP’03]
∞
x = y + n = zu + n E ( y | x ) = ∫−∞ zE ( y | x, z ) f ( z )dz
zC u C u , Cn : signal and noise
noise E ( y | x, z ) = x
zC u + C n covariances

Locally adaptive denoising: ProbShrink…

[Pizurica&Philips, IEEE TIP 2006]


ProbShrink for correlated noise…
local window
 X 22 
20

40 
X  
 23 
60

80

100
X22 X23
120

140
vector of coefficients
160

180

50 100 150

H0

H1
X23 H1

X22
X22
[B. Goosens, A. Pizurica, W. Philips; IEEE TIP 2008, in press]


… ProbShrink for correlated noise

[B. Goosens, A. Pizurica, W. Philips; IEEE TIP 2008, in press]

Denoising by singularity detection

Input signal Rate of increase of the
modulus of the wavelet
transform across scales is
w1 wavelet coefficients proportional to the
local Lipschitz regularity

w2

w3

[Mallat&Zhong, IEEE IT 1992]


Statistics: magnitude and the rate of increase
Noise standard deviation=25.5
0.05 150

0.04
noise 100
edges
x=1 l

Magnitude
0.03

0.02

50
0.01 edges noise
x= 0
l

0
-50 0 50 100 150 200 250 300
0
noisy magnitude -3 -2 -1 0 1 2 3
ACR

1

0.8
scale

0.6

noise edges
0.4 Average Cone Ratio – an estimate
0.2 of the local Lipschitz exponent –
0
measures the rate of increase of
-4 -2 0 2 4 6
the coefficients across the scales
cone of influence ACR
[A. Pizurica et al; IEEE TIP 2002]

Inter- and intrascale dependencies

• Bivariate models
• Hidden Markov Tree models
• Markov Random Field models


Statistical modeling: MRF models

x0 xMAP neighborhood cliques

P(x)

Prior model

1  
P(x) = exp− ∑VC (x) Example: penalize isolated peaks
Z  C∈ς 
negative potential
clique
potentials positive potential


Statistical modeling: MRF models

x0 xMAP neighborhood cliques

P(x)

Prior model

Initial edges Iteration 1 Iteration 2 Iteration 3


MRF based wavelet denoising

Original
Gamma MAP filter wavelet filter

[A. Pizurica et al; ICIP 2001]

Current trends in Bayesian wavelet denoising
Gaussian Scale Mixture (GSM) model MRF models
[Portilla et al, IEEE TIP 2003]
Shortcoming: assumes the same but scaled
covariance for the whole subband Fields of GSM
[Liu and Simoncelli,
Mixture of GSM PAMI’08]
(MGSM)
Spatially variant GSM
[Portilla et al, Spie
(SVGSM) 2008] Field of Experts (FoE)
[Guerrero-Colon et al,
Computationally expensive [Roth and Black, CVPR’05]
IEEE TIP,08]
GSM in non-overlapping blocks [Tappen, Adelson, Freeman,
• Ignores non-local correlations CVPR’07, CVPR’08]
• Block size?
Mixture of projected GSM
Dimension reduction
(MPGSM) ?
in MGSM [Goossens, in review TIP 2008]


Overview



Why other multiresolution representations
Classical wavelets are well suited for point-like singularities,
but not for curvilinear singularities in images
• Poor orientation selectivity; no difference between 45 and -45o
• Checkerboard pattern appears also as an artifact in denoising

An example of wavelet base functions

Many wavelet-like representations with a better orientation selectivity:
complex wavelets [Kingsbury, Selesnick] , steerable pyramids [Freeman,
Adelson], curvelets [Donoho, Candes], contourlets [Do, Vetterli], …


Curvelet-domain image denoising…

Curvelets: specific tiling of the frequency plane:

localized + directional


…Curvelet domain image denoising…
Wavelet ProbShrink
Noisy Image Curvelet Hard Thresholding
Curvelet ProbShrink

PSNR=29.50dB
PSNR=22.16dB PSNR=29.02dB
PSNR=30.43dB

[L. Tessens, A. Pizurica, W. Philips, J Electr Imag 2008 in press]


Results

…Curvelet domain image denoising…
Wavelet ProbShrink Curvelet ProbShrink


…Curvelet domain image denoising
Wavelet ProbShrink Curvelet ProbShrink


Overview



MRI denoising: signal dependent noise
p(m) low SNR noisy m
(f=0)

magnitude

contrast
high SNR
( f =f1 )
noise-free f

f1 m SNR


MRI denosing: algorithm
Step 1: Bias removal
Square magnitude MRI image – after squaring constant bias, proportional
to noise standard deviation.
For better results: square root the result before denoising!

Step 2: Denoising (coarse-to-fine, empirical density estimation)
Mask
A noisy detail

T?
p(z|H1)
log( )
p(z|H0) histograms p(z|H0)

Coarser, processed detail p(z|H1)
[Pizurica et al IEEE TMI 2003]

MRI denosing: some results
Noisy image Denoised image Ground truth


3D MRI volume denoising
using 3D dual-tree complex wavelet transform

[J. Aelterman et al, EUSIPCO 2008]

Denoising low-dose CT images
• Reducing radiation dose increases noise level
• Can we use denoising on low dose CT to obtain the same diagnostic
quality as in a higher dose CT image? [IBBT Ica4dt project]

• Difficulties:
- non-stationary correlated noise
- Streak artefacts
- How to estimate noise


Denoising algorithm
segmentation

wavelet Inverse wavelet
Vector
transform transform
ProbShrink
(WT) (IWT)
(Dual-tree H0
complex) H1 H1

[B. Goossens et al, EMBS 2007]


…Results
Watershed segmentation
denoised


Qualitative Validation CT: psycho-visual experiment

…

Abdomen/Lung Blur Noise Structure Quality
Low-dose CT +++ +++ -- --- +++ +++ + +++
Versatile Probshrink - + + -- + + ++ +
Versatile Probshrink2 - -- + + + + ++ -
Curvelet Filter - -- --- ++ + ++ -- --
Wavelet GSM Filter + - ++ +++ ++ +++ +++ ---

The abdomen image judged better than the original by radiologists!

Optical Coherence Tomography (OCT) images
[IBBT Ica4dt project, with AGFA Healthcare]

3D OCT data 2D signal

OCT – “echography with light”

Noise: speckle similar to that in radar and ultrasound images


Denoising OCT images
A developed 3D OCT denoiser combines [IBBT Ica4dt project]
• wavelet domain speckle filter
• motion compensated video denoising method

coarse-to-fine
processing 3D OCT data

Signal and noise statistics
Image:g120406breastseconformolnogelLR 050 Detail:Dx1 Parameter:b=10.3752
0 Image:g120406breastseconformolnogelLR 050 Detail:Dx2 Parameter:a=4.8348
0
0.035 0.2
Gamma, b=10.3752 Laplace, a=4.8348
0.03

0.025 0.15
likelihood p(m|1)

likelihood p(m|0)

0.02
0.1
0.015

0.01
0.05
0.005

0
0 50 100 150
magnitude m
200 250
0
0 50
magnitude m
100 150
Video denoising

Locally adaptive denoising


Results and evaluation of OCT denoising

Noisy
Image Our method GTF
SAT
RKT
Lee



Noisy Our method Lee
SAT
[Pizurica et al; CMIR 2008 in press]



BLS-GSM
input Our method (2D version)

[Pizurica et al; CMIR 2008 in press]


Overview



Noise variance estimation
Block-based Smoothing based
Search for blocks of
nearly uniform intensity

smooth
estimate
σ

Gradient distribution based Wavelet based
noise (Rayleigh distr.)
HL1
signal+noise

use HH1
LH1
this part
σ Median{|HH1|}
or compensate for σ=
ˆ 0.6745
the peak shift

Blur estimation using wavelet coefficients
• A well known approach: kurtosis of the wavelet coefficient histogram
• An alternative: examine the propagation of the wavelet coefficients across
the scales

0.35
0.3 original
blurred
0.25

0.2
PDF
0.15

0.1

0.05

0
-4 -2 0 2 4 6 8
Original image Blurred image ACR 1-2

ACR - Average Cone Ratio – an estimate of the local Lipschitz exponent –
measures the rate of increase of the coefficients across the scales


Blur estimation using wavelet coefficients

blur

6000 6000
reference reference
blur 3
blur 5
solid – noise free
blur 3
blur 5
5000 5500
blur 7
blur 0 + noise dashed – noisy
blur 7
blur 0 + noise
blur 3 + noise blur 3 + noise
4000 blur 5 + noise 5000 blur 5 + noise
blur 7 + noise blur 7 + noise

3000 4500

2000 4000
Different colors -
1000 3500
different levels of
0
-1 0 1 2 3 4 5
3000
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2
blur
ACR 2-4 ACR 2-4


Overview



Video denoising

Input Noisy 2D Wavelet Noise
Frame Transform Estimation

Motion Time delay
Estimation

Recursive Temporal Filtering

Time delay

Adaptive Spatial Inverse 2D Wavelet Denoised
Filtering Transform Frame

[V. Zlokolica, A. Pizurica, W. Philips; IEEE TCSVT 2006]

Video denoising: motion estimation…

center of
the motion block

motion direction
(smaller amplitude)

motion direction
(larger amplitude)

Accurate motion estimation is essential for video denoising.
Also important: reliability of the estimated motion at each point


…Motion compensated video denoising
Further development currently within IBBT project ISYSS

[V. Zlokolica, A. Pizurica, W. Philips; IEEE TCSVT 2006]


Reusing motion estimator from video codecs
• Motion estimators from video codecs tolerate errors cannot be directly
used in denoising
• Can we still use them with some postprocessing? The core of our approach:
• Motion field refinement step
• Reliability to motion estimates controls the recursive filter
• Competitive with state-of-the art video denoisers

[LJ. Jovanov et al; IEEE TCSVT 2008, in press]

Reusing motion estimator from video codecs

noise-free input

[Balster; TCSVT 2006] [Jovanov; TCSVT 2008]

Denoising and outlier removal in 3D video
Time-of-flight camera
records simultaneously
luminance and depth information

Degradations in the depth image:
noise, and outliers (similar to
impulse noise but in bursts)
3D reconstructions using
“surf” in Matlab
The biggest errors in the
depth measurement are
induced by strong
ambient light
The measured
distance is much smaller
than the true distance)


Noisy 3D video sequence (luminance and depth)

Luminance image contains much less noise
Luminance and depth images are correlated
Use the luminance information for denoising depth data


Denoised luminance and depth


Acknowledgements
Thanks to my colleagues for their contributions
• Vladimir Zlokolica (video denoising)
• Bart Goossens (removal of correlated noise)
• Ljubomir Jovanov (video, 3D video, OCT)
• Linda Tessens (curvelets)
• Jan Aelterman (MRI denoising)
• Filip Rooms (deblurring)
• Ewout Vansteenkiste (quality evaluation CT)

Related material available at: http://telin.ugent.be/~sanja


Image and Video Restoration Advances and Challenges

Recommended

Recommended

More Related Content

What's hot

What's hot (15)

Viewers also liked

Viewers also liked (20)

Similar to Image and Video Restoration Advances and Challenges

Similar to Image and Video Restoration Advances and Challenges (9)

More from imec.archive

More from imec.archive (20)

Image and Video Restoration Advances and Challenges