Digital image processing involves techniques to restore degraded images. Image restoration aims to recover the original undistorted image from a degraded observation. The degradation is typically modeled as the original image being operated on by a degradation function and additive noise. Common restoration techniques include spatial domain filters like mean, median and order-statistic filters to remove noise, and frequency domain filtering to reduce periodic noise. The choice of restoration method depends on the type and characteristics of degradation in the image.

1. DIGITAL IMAGE PROCESSING
IMAGE RESTORATION

2. Basic Idea
• With Image Restoration one tries to repair errors or distortions in an image,
caused during the image creation process.
• In general our starting point is a degradation and noise model:
g(x, y) = H ( f(x, y) ) + (x, y)
Determined by quality of equipment and image taking conditions.
• Image restoration is computationally complex.
• Equipment as degradation free as possible.
• seen technical and financial limitations
• Medical: low radiation, little time in magnet-tube.
• lowest image quality to achieve medical goals
• Web-cams: cheap lens distortions corrected by CPU in cam.

3. Basic Idea
• Ultimate goal of restoration techniques is to
improve an image in some predefined sense.
• This above concept obviously have certain
overlapping with image enhancement, although
there are differences among the both.
• The fundamental differences in the two concepts
are illustrated in the next slide.

4. 4
Image Enhancement vs. Image
Restoration
• Image enhancement: process image so that the result is more
suitable for a specific application, is largely a subjective process.
• Image restoration: recover image from distortions to its original
image, is largely an objective process.
• Image enhancement techniques are basically heuristic procedure
designed to manipulate an image in order to take advantage of the
psychophysical aspects of human visual systems.
• In contrast, the restoration approach usually involves formulating a
criterion of goodness that will yield an optimal estimate of the
desired result.

5. Model of Image Degradation/Restoration
Degradation process is modeled as a degradation function
that together with an additive noise, operates on an input
image f(x, y) to produce a degraded image g(x, y).
If H is a linear, position-invariant process, then
( , ) ( , ) ( , ) ( , )g x y f x y h x y x y  
where h(x, y) is the spatial representation of the degradation
function and (x, y) is the noise.

6. Linear, Position-Invariant Degradation
• From the model of image degradation, one
may write:
where H is the degradation function and η is
the additive noise.
• Initially assume that which gives,
     , , ,g x y H f x y x y   
 , 0x y 
   , ,g x y H f x y   

7. Linear, Position-Invariant Degradation
• An operator having input-output relationship g(x,
y) = H[f(x, y)] is said to be position- (or space-)
invariant if
for any f(x, y) and any α, β.
• The definition indicates that the response at any
point in the image depends only on the value of
the input at that point, not on its position.
    , ,H f x y g x y       

8. 8
Linear, Position-Invariant Degradation
( , ) ( , )* ( , ) ( , )
( , ) ( , ) ( , ) ( , )
g x y f x y h x y x y
G u v F u v H u v N u v
 
 
 ( , , , ) ( , )h x y H x y      
The following function is called a Point Spread Function:
Now we may express the linear position-invariant
degradation (Spatial and Frequency Domain) as follows:
The above is nothing but the impulse response of H.

9. Properties of Noise
• A noise is a white noise when its Fourier
Spectrum is constant.
• We assume that the noise is independent of the
spatial coordinates, and that it is uncorrelated
with respect to image itself.
• Noise cannot be predicted, but can be
approximately described in statistical way using
the probability density function (PDF).

10. Noise Models
Gaussian Noise:
2 2
( ) /21
( )
2
z
p z e  

 

Rayleigh Noise
2
( ) /2
( ) for
( )
0 for z a
z a b
z a e z a
p z b
 
 
 
 
Erlang (Gamma) Noise
1
( ) for 0
( ) ( 1)!
0 for z 0
Here a > 0 and b is a positive integer.
b b
aza z
z a e z
p z b


 
 
 
2
mean =
variance =


 2
mean, = 4
4
variance, =
4
a b
b
 




2
2
mean, =
variance, =
b
a
b
a



11. Exponential Noise
Uniform Noise
Impulse (salt & pepper) Noise
, z 0
( )
0, z < 0
Here a > 0.
az
ae
p z

 
 

1
for a
( ) b - a
0 otherwise
z b
p z

 
 

for
( ) for
0 otherwise
a
b
P z a
p z P z b


 


Noise Models
2
2
1
mean, =
1
variance, =
a
a


 
2
2
mean, =
2
variance, =
12
a b
b a




If either Pa or Pb is zero, the impulse noise
is called Uni-polar.
Other wise it is called Bi-polar.
Bi-polar noise is also called salt & pepper
noise.

12. Sources of Noise
• Gaussian noise arises in an image due to factors such as
electronic circuit noise and sensor noise due to poor
illumination and/or high temperature.
• Raleigh noise is helpful in characterizing noise phenomena
in range imaging.
• Exponential and Gamma noises are more common is Laser
Imaging.
• Impulse noise is found in situations where quick transients,
such as faulty switching, take place during imaging.

13. Image Degradation with Additive Noise
Original image
Histogram
Degraded images
( , ) ( , ) ( , )g x y f x y x y 

14. Original image
Histogram
Degraded images
( , ) ( , ) ( , )g x y f x y x y 
Image Degradation with Additive Noise

15. Estimation of Noise Parameters
• The parameters of periodic noise typically are
estimated by inspection of the Fourier spectrum
of the image.
• The parameters of the noise PDFs may be known
partially from sensor specifications.
• But it is often necessary to estimate them for a
particular imaging environment.

16. Restoration in presence of Noise only
• Clearly, g(x, y) = f(x, y) + η(x, y).
• In frequency domain, G(u, v) = F(u, v) + N(u, v).
• In general, neither η(x, y) nor N(u, v) is known in advance.
• We choose spatial domain filtering techniques when
additive random noise is expected to be present.
• We use frequency domain technique only if we feel that
periodic noise is present.

17. Periodic Noise Reduction by Frequency
Domain Filtering
• Periodic noise appears as a concentrated bursts of energy
in the Fourier transform, at locations corresponding to
frequencies of the periodic interference.
• The approach for restoration is to use a selective filter to
isolate the noise.
• The type of selective filters we use is either band-reject or
band-pass filters.
• We also use notch-reject or notch-pass filters for the same
purpose.

18. Periodic Noise Reduction by Frequency Domain Filtering
Band reject filter Restored image
Degraded image DFT
Periodic noise can
be reduced by
setting frequency
components
corresponding to
noise to zero.

19. Restoration in presence of Noise only:
Spatial Domain Filtering
• Spatial domain filtering are chosen for
restoration if the degradation present in the
image is additive noise only.
• There are two general types of filters used in
this category:
– Mean Filters.
– Order-statistics Filters.

20. Mean Filters
Arithmetic mean filter or moving average filter
( , )
1ˆ( , ) ( , )
xys t S
f x y g s t
mn 
 
Geometric mean filter
1
( , )
ˆ( , ) ( , )
xy
mn
s t S
f x y g s t

 
  
 
 

mn = size of moving window
Degradation model:
( , ) ( , ) ( , ) ( , )g x y f x y h x y x y  
To remove this part

21. Geometric Mean Filter: Example
Original
image
Image
corrupted
by AWGN
Image
obtained
using a 3x3
geometric
mean filter
Image
obtained
using a 3x3
arithmetic
mean filter
AWGN: Additive White Gaussian Noise

22. Harmonic and Contra-harmonic Filters
Harmonic mean filter
( , )
ˆ( , )
1
( , )xys t S
mn
f x y
g s t


Contra-harmonic mean filter
1
( , )
( , )
( , )
ˆ( , )
( , )
xy
xy
Q
s t S
Q
s t S
g s t
f x y
g s t






mn = size of moving window
Works well for salt noise
but fails for pepper noise
Q = the filter order
Positive Q is suitable for
eliminating pepper noise.
Negative Q is suitable for
eliminating salt noise.
For Q = 0, the filter reduces to an arithmetic mean filter.
For Q = -1, the filter reduces to a harmonic mean filter.

23. Contra-harmonic Filters: Example
Image
corrupted
by pepper
noise with
prob. = 0.1
Image
corrupted
by salt
noise with
prob. = 0.1
Image
obtained
using a 3x3
contra-
harmonic
mean filter
With Q = 1.5
Image
obtained
using a 3x3
contra-
harmonic
mean filter
With Q=-1.5

24. Contra-harmonic Filters: Incorrect Use Example
Image
corrupted
by pepper
noise with
prob. = 0.1
Image
corrupted
by salt
noise with
prob. = 0.1
Image
obtained
using a 3x3
contra-
harmonic
mean filter
With Q=-1.5
Image
obtained
using a 3x3
contra-
harmonic
mean filter
With Q=1.5

25. Order-Statistic Filters: Revisited
subimage
Original image
Moving
window
Statistic parameters
Mean, Median, Mode,
Min, Max, Etc.
Output image

26. Order-Statistics Filters
Median filter
 ( , )
ˆ( , ) ( , )median
xys t S
f x y g s t


Max filter
 ( , )
ˆ( , ) ( , )max
xys t S
f x y g s t


Min filter
 ( , )
ˆ( , ) ( , )min
xys t S
f x y g s t


Mid-point filter
   ( , )( , )
1ˆ( , ) ( , ) ( , )max min
2 xyxy s t Ss t S
f x y g s t g s t

 
  
 
Reduce “dark” noise
(pepper noise)
Reduce “bright” noise
(salt noise)

27. Median Filter : How it works
A median filter is good for removing impulse, isolated noise
Degraded image
Salt noise
Pepper noise
Moving
window
Sorted
array
Salt noise
Pepper noise
Median
Filter output
Normally, impulse noise has high magnitude
and is isolated. When we sort pixels in the
moving window, noise pixels are usually at
the ends of the array.
Therefore, it’s rare that the noise pixel will be a median value.

28. Median Filter : Example
Image
corrupted
by salt-and-
pepper
noise with
pa=pb= 0.1
Images obtained using a 3x3 median filter
1
4
2
3

29. Max and Min Filters: Example
Image
corrupted
by pepper
noise with
prob. = 0.1
Image
corrupted
by salt
noise with
prob. = 0.1
Image
obtained
using a 3x3
max filter
Image
obtained
using a 3x3
min filter

30. Alpha-trimmed Mean Filter
( , )
1ˆ( , ) ( , )
xy
r
s t S
f x y g s t
mn d 



where gr(s, t) represent the remaining mn - d pixels after
removing the d/2 highest and d/2 lowest values of g(s, t).
This filter is useful in situations involving multiple types
of noise such as a combination of salt-and-pepper and
Gaussian noise.
Formula:
Here d can range from 0 to (mn – 1).
If d = 0, it reduces to Arithmetic mean filter.
If d = mn – 1, it reduces to median filter.

31. Alpha-trimmed Mean Filter: Example
Image
corrupted
by additive
uniform
noise
Image
obtained
using a 5x5
arithmetic
mean filter
Image
additionally
corrupted
by additive
salt-and-
pepper
noise
1 2
2 Image
obtained
using a 5x5
geometric
mean filter
2

32. Alpha-trimmed Mean Filter: Example (cont.)
Image
corrupted
by additive
uniform
noise
Image
obtained
using a 5x5
median filter
Image
additionally
corrupted
by additive
salt-and-
pepper
noise
1 2
2
Image
obtained
using a 5x5
alpha-
trimmed
mean filter
with d = 5
2

33. Alpha-trimmed Mean Filter: Example (cont.)
Image
obtained
using a 5x5
arithmetic
mean filter
Image
obtained
using a 5x5
geometric
mean filter
Image
obtained
using a 5x5
median filter
Image
obtained
using a 5x5
alpha-
trimmed
mean filter
with d = 5

34. Adaptive Median Filter
• Adaptive median filter has three main purposes:
– To remove salt-and-pepper noise.
– To provide smoothing of other noise that may not be
impulsive.
– To reduce distortion, such as excessive thinning or
thickening of object boundaries.
• The algorithm discussed previously consider zmin and
zmax to be “impulse-like” noise components.

35. Estimation of Degradation Model
Degradation model:
( , ) ( , ) ( , ) ( , )g x y f x y h x y x y  
Purpose: to estimate h(x, y) or H(u, v)
( , ) ( , ) ( , ) ( , )G u v F u v H u v N u v 
Methods:
1. Estimation by Image Observation
2. Estimation by Experiment
3. Estimation by Modeling
or
Why? If we know exactly h(x, y), regardless of noise, we can do
deconvolution to get f(x, y) back from g(x, y).

36. Restoration Using Degradation Model
• The degradations are modeled as being result of convolution, and
restoration seeks to find filters that apply process in reverse.
• Due to that reason, the term image deconvolution is used frequently to
signify linear image restoration.
• The filters used in the restoration process often are called deconvolution
function.
• The process of restoring an image by using a degradation function that has
been estimated in some way sometimes is called blind deconvolution.
• It is due to the fact that true degradation function is seldom known
completely.

37. Estimation by Image Observation
• We are given a degraded image without any knowledge about the
degradation function H.
• One way to estimate the function H is to gather information from
the image itself.
• We take a small section of the image with simple structures, where
there are strong signal contents.
• Using the sample gray levels of the object and background, we can
construct an un-blurred image of the same size and characteristics
as the observed sub-image.
• Then we can extend the transfer function to the whole image.

38. Estimation by Image Observation
f(x,y) f(x,y)*h(x,y) g(x,y)
Subimage
Reconstructed
Subimage
),( vuGs ),( yxgs
),(ˆ yxfs
DFT
DFT
),(ˆ vuFs
Restoration
process by
estimation
Original image (unknown) Degraded image
),(ˆ
),(
),(),(
vuF
vuG
vuHvuH
s
s
s 
Estimated Transfer
function
Observation
This case is used when we
know only g(x, y) and cannot
repeat the experiment!

39. Estimation by Experimentation
• If the equipment similar to the equipment used to acquire the
degraded image is available, it is possible in principle to obtain an
accurate estimate of the degradation.
• Images similar to the degraded image can be acquired with various
system settings until they are as closely as possible to the image we
wish to restore.
• The idea is to obtain the impulse response of the degradation by
imaging an impulse using the same system settings.
• An impulse is simulated by a bright dot of light, as bright as possible
to reduce the effect of noise.

40. Estimation by Experimentation
Used when we have the same equipment set up and can repeat the
experiment.
Input impulse image
System
H( )
Response image from
the system
),( vuG
),( yxg),( yxA
  AyxADFT ),(
A
vuG
vuH
),(
),( 
DFTDFT

41. Estimation by Modeling
Used when we know physical mechanism underlying the image
formation process that can be expressed mathematically.
Atmospheric
Turbulence model
2 2 5/6
( )
( , ) k u v
H u v e 

Example:Original image Severe turbulence
k = 0.00025k = 0.001
k = 0.0025
Low turbulenceMild turbulence K is a constant
that depends on
the nature of the
turbulence.
Proposed by Hufnagel and Stanley in 1964

42. Inverse Filter
After we obtain H(u, v), we can estimate F(u, v) by the inverse filter:
( , ) ( , )ˆ( , ) ( , )
( , ) ( , )
G u v N u v
F u v F u v
H u v H u v
  
From degradation model:
( , ) ( , ) ( , ) ( , )G u v F u v H u v N u v 
Noise is enhanced
when H(u, v) is small.
To avoid the side effect of enhancing
noise, we can apply this formulation
to frequency component (u, v) with in
a radius D0 from the center of (u, v).
In practice, the inverse filtering is not a popular technique.

43. Inverse Filter: Example
2 2 5/6
0.0025( )
( , ) u v
H u v e 

Original image
Blurred image
Due to Turbulence
Result of applying
the full filter
Result of applying
the filter with D0=70
Result of applying
the filter with D0=40
Result of applying
the filter with D0=85

44. Wiener Filter: Minimum Mean Square
Error Filter
• The inverse filtering approach discussed so far
makes no explicit provision for handling noise.
• The current approach incorporates both the
degradation function and statistical
characteristics of noise into the restoration
process.
• The method is founded on considering images
and noise as random processes.

45. Wiener Filter: Minimum Mean Square
Error Filter
• The objective is to find an estimate of the uncorrupted
f such that the mean square error between the original
and the estimate is minimized.
• It is assumed that:
– the noise and the image are uncorrelated;
– one or the other has zero mean;
– the gray levels in the estimate are a linear function of the
levels in the degraded image.

46. Wiener Filter: Minimum Mean Square Error Filter
Objective: optimize mean square error:  2 2ˆ( )e E f f 
*
2
*
2
2
2
( , ) ( , )ˆ( , ) ( , )
( , ) ( , ) ( , )
( , )
( , )
( , ) ( , ) / ( , )
( , )1
( , )
( , ) ( , ) ( , ) / ( , )
f
f
f
f
H u v S u v
F u v G u v
S u v H u v S u v
H u v
G u v
H u v S u v S u v
H u v
G u v
H u v H u v S u v S u v



 
  
  
 
  
  
 
  
  
Wiener Filter Formula:
where
H(u, v) = Degradation function
Sh(u, v) = Power spectrum of noise
Sf(u, v) = Power spectrum of the un-degraded image

47. Approximation of Wiener Filter
2
2
( , )1ˆ( , ) ( , )
( , ) ( , ) ( , ) / ( , )f
H u v
F u v G u v
H u v H u v S u v S u v
 
  
  
Wiener Filter Formula:
Approximated Formula:
2
2
( , )1ˆ( , ) ( , )
( , ) ( , )
H u v
F u v G u v
H u v H u v K
 
  
  
Difficult to estimate
Practically, K is chosen manually to obtained the best visual result!

48. Wiener Filter: Example
Original image
Blurred image
Due to Turbulence
Result of the
full inverse filter
Result of the inverse
filter with D0=70
Result of the
full Wiener filter

49. Wiener Filter: Example (cont.)
Original image
Result of the inverse
filter with D0=70
Result of the
Wiener filter
Blurred image
Due to Turbulence

50. Example: Wiener Filter and Motion Blurring
Image
degraded
by motion
blur +
AWGN
Result of the
inverse filter
Result of the
Wiener filter

2=650

2=325

2=130
Note: K is
chosen
manually

51. 51
Different restoration approaches
Frequency domain
– Inverse filter
– Wiener (minimum mean
square error) filter
Algebraic approaches
– Unconstrained
optimization
– Constrained
optimization

52. 52
The block-circulant matrix
 Stacking rows of image f, g, n to make MN x 1 column vectors
f, g, and n. (Also called lexicographic representation of the
original image). Correspondingly, H should be a MN x MN
matrix
 H is called block-circulant matrix
nHfg +=
ú
ú
ú
ú
ú
ú
û
ù
ê
ê
ê
ê
ê
ê
ë
é
=
---
-
--
0321
3012
2101
1210
HHHH
HHHH
HHHH
HHHH
H
MMM
M
MM
( ) ( ) ( )
( ) ( ) ( )
( ) ( ) ( )ú
ú
ú
ú
û
ù
ê
ê
ê
ê
ë
é
--
-
=
0,2,1,
2,0,1,
1,1,0,
jhNjhNjh
jhjhjh
jhNjhjh
eee
eee
eee
jH

53. 53
Algebraic approach – Unconstrained
restoration vs. Inverse filter
()
() ( )
( ) ( ) gHgHHHgHHHf
gHfHHfHgH
f
f
fHgnf
Hfgn
nHfg
f
gfHf
1111
ˆrespect towith
sidehandrightatedifferenti
22
sensesquaresleastain
esapproximatˆsuch thatˆseek
ˆ
ˆˆ20
ˆ
ˆ
ˆˆ
----
===
=Þ--==
¶
¶
¾¾¾¾¾¾¾ ®¾-==
¾¾¾¾¾¾¾¾ ®¾-=
+=
TTTT
TTTJ
J
ˆF(u,v) =
G(u,v)
H(u,v)
Compared to the inverse filter:

54. 54
Algebraic approach – Constrained
restoration vs. Wiener filter
()
() ( )
( ) gHQQHHf
fHgHfQQ
f
f
nfHgfQf
f
nHfg
fQfQ
TTT
TTJ
J
1
222
22
2
ˆ
ˆ2ˆ20
ˆ
ˆ
.multiplierLagrangethecalledconstant,ais
ˆˆˆ
functionobjective)(orcriteriontheminimizesthatˆseekWe
methodonoptimizatiLagrangeusingproblemthisModel
.constraintthesubject to
,onoperatorlinearaiswhere,ˆMinimizing
-
+=
--==
¶
¶
÷
ø
ö
ç
è
æ --+=
=-
g
a
a
a
( )
( )
( )
( ) ( ) ( )
( )vuG
vuSvuSvuH
vuH
vuH
vuF
fn
,
,/,,
,
,
1
,ˆ
2
2
+
=Compared to:

55. Improvement
• Wiener Filtering
• K = Sn(u,v)/Sf(u,v),
• Sn(u,v) = |N(u,v)|2
• Sf(u,v) = |F(u,v)|2
– Sn(u,v) & Sf(u,v) must be known
• Sn(u,v) the power spectrum of the noise,
• Sf(u,v) the power spectrum of the original image
),(
),(
),(
),(
1
),( 2
2
vuG
KvuH
vuH
vuH
vuF












56. Improvement – Cons.
• Constrained Least Squares Filtering
• P(u,v) is the fourier transform of the Laplacian operator
– Constrain:
• |g – H |2 = |η|2
• R(u,v) = G(u,v) – H(u,v)
• Adjust γ from the constrain – by Newton-Raphson root-finding
• Apply algorithm from Prof. Hsien-Sen Hung

f
),(
),(),(
),(
),( 22
*
vuG
vuPvuH
vuH
vuF













f