2. IMAGES AS MATRICES
Images can be represented as matrices, with each entry in the matrix
containing the intensity of the corresponding image’s pixel
Grayscale intensities lie on an interval [0, n], where 0 = black, n = white
1040 x 857
image
(length by
width)
857 x 1040
matrix
(rows by
columns)
x1 1 x1 2 x1 1040
x2 1
x857 1 x857 1040
x2 2 x2 1040
Grayscale images are represented by single matrices
Color images are represented by multidimensional matrices
3. BLURRING AS A LINEAR OPERATION
• We have an m x n blurred image B, and the corresponding ideal, sharp m x n
image X
• Their vector forms, b and x, have m n = N entries, constructed by stacking
the columns of B and X respectively
• Blur is defined by how information that should be contained in a single pixel
“spills over” into surrounding pixels
• All the information about how X is blurred to B is contained in some N x N
matrix A such that,
Ax = b
The blurring matrix A has two primary ingredients
• Point-Spread Functions
• Boundary Conditions
4. BOUNDARY CONDITIONS
What’s happening just beyond the edges of
our image X?
Different assumptions about the
boundary conditions will yield different
constructions of the blurring matrix A,
and thus affect our ability to accurately
model the behavior of the blur
5. X is surrounded by 0 at every edge
This is the “default” boundary assumption
This condition takes effect if no boundary condition is considered
at all
A pixel value of 0 corresponds with black, so this condition might
be useful for images with black backgrounds, like outer space
ZERO BOUNDARY CONDITION
6. X is surrounded by copies of itself, repeated around every edge
PERIODIC BOUNDARY CONDITION
7. Each edge of X faces its mirror image
With this condition, the pixels at the edge of an image are always next to
identically valued pixels just beyond the edge
REFLEXIVE BOUNDARY CONDITION
8. POINT SPREAD FUNCTIONS
• An image with all black pixels (represented by 0) except for a single
white pixel (represented by 1) is known as a Point Source
This image is represented as an m x n matrix
(the same dimensions as X and B)
This matrix could be rearranged into a vector
of length m n = N
Since this vector is all 0 except for a 1 as the ith
entry, this vector is the ith column of an N x N
identity matrix, and denoted ei
9. POINT SPREAD FUNCTIONS
• A point spread function (PSF) is a function that describes the blurring and the
resulting image of the point source.
PSF
Horizontal Motion BlurOut-of-Focus BlurAtmospheric Turbulence BlurMoffat Blur
For any PSF, the elements of the
resulting image must sum to 1, the
value of the original point source
If the PSF is the same regardless of
the location of the point source, the
blurring is spatially invariant
10. HOW A INCORPORATES THE PSF
So the columns of A are the same as the
blurred images of the various point sources
For each point source vector ei, we can easily determine Aei, the
blurred image vector of ei
It’s defined by the PSF itself
But we can also show that Aei is equivalent to the ith column of A
11. THE SIMPLE EXAMPLE
Last time, we constructed a 12 x 12 blurring matrix for a 4 x 3 image
matrix, where the PSF blurred the point source to a vertical line of 2 pixels
(the point source pixel and the pixel directly beneath)
We accomplished this
by determining the
blurred images of
every possible point
source (they made up
the columns of A)
e1
1
0
0
0
0
0
0
0
0
0
0
0
1/2
1/2
0
0
0
0
0
0
0
0
0
0
1 0
0 0
0
0
0 0 0
0 0 0
1/2 0
1/2 0
0
0
0 0 0
0 0 0
Point Source Blurred Point Source
PSF
First column of A
12. THE SIMPLE EXAMPLE
Create the long column
vector x from the image
matrix X
Multiply A by x to get b,
the vector form of the
blurred image
Rearrange b back into
matrix form to get the
blurred image matrix B
13. THE SIMPLE EXAMPLE
Our technique of constructing the 12 x 12 blurring matrix A column by
column worked well enough for a simple 4 x 3 image
But constructing A column by column is typically extremely time consuming
15. =
Notice how each blurred pixel can
be written as a weighted sum of
pixels from the sharp image
These weighted sums can be
attained through a process called
convolution!!!
16. 0 0
0 0.5
0
0
0 0.5 0
0 0 0
0 0
0 1
0
0
0 0 0
0 0 0
Start with a point source and its corresponding
blurred image (known as the PSF array)
The position of the nonzero pixel in the point
source is the “center” of the PSF array
Rotate the PSF array 180° and lay it
overtop of the sharp image matrix
Multiply each pair of entries lying atop
of one another, then add them all
together
The position of the entry from X that is
multiplied by the “center” of the PSF
array is the position of the entry in B
that assumes the value of the weighted
sum
0 0
0 0.5
0
0
0 0.5 0
0 0 0
b11 = (1/2)x11
b21 = (1/2)x11 + (1/2)x21
b31 = (1/2)x21 + (1/2)x31
b41 = (1/2)x31 + (1/2)x41
b12 = (1/2)x12
b22 = (1/2)x12 + (1/2)x22
b32 = (1/2)x22 + (1/2)x32
b42 = (1/2)x32 + (1/2)x42
b13 = (1/2)x13
b23 = (1/2)x13 + (1/2)x23
b33 = (1/2)x23 + (1/2)x33
b43 = (1/2)x33 + (1/2)x43
CONVOLUTION
17. NOISE REMOVAL AND EDGE DETECTION
Convolution of an image with the PSF array blurs the image, but
convolution can be used to accomplish other ends as well
Noise Removal (Low-Pass Filter)
Convolution of an image with a matrix that averages any given
pixel with its neighbors
The result is a “smoothing out” of pixels that differ greatly from
their neighbors
Edge Detection (High-Pass Filter)
Convolution of an image with a matrix that amplifies pixels that
differ greatly from their neighbors
The result is that edges in the image are amplified, but noise is
amplified as well
18. p11 p12
p21 p22
p13
p23
p31 p32 p33
p41 p42 p43
p43 p42
p33 p32
p41
p31
p23 p22 p21
p13 p12 p11
b11 = p22(x11) + p12(x21) + 0(x31) + 0(x41) + p21(x12) + p11(x22) + 0(x32) + 0(x42) + 0(x13) + 0(x23) + 0(x33) + 0(x43)
CONVOLUTION
Let’s repeat the PSF
convolution, now with a general
PSF array with center p22
20. =
We’ve arrived back at
Ax = b, and we can
now see that A is
filled with the entries
of the PSF array!
Is there any
generality to be
gleaned from A’s
structure?
Divide A into 4 x 4 blocks
All of the diagonals in each
block are constant
A matrix where every
diagonal is constant is
known as Toeplitz
So all blocks in A are
Toeplitz matrices
Also, each diagonal of blocks in
A is constant!
We say that A is a block
Toeplitz matrix with Toeplitz
blocks
Note: This structure of A
holds only when we assume
zero boundary conditions
• Different boundary
conditions yield different
general structures for A
21. SEPARABLE BLUR
The blur is separable into vertical and horizontal components if the PSF
array, P, can be written as an outer product of vectors c and r
If this holds, then each element of the PSF array, pij, can be expressed as cirj
22. Let’s assume that the blur is separable in our arbitrary 4 x 3 PSF
Then A can be rewritten as:
23. KRONECKER PRODUCTS
0
0
Each r component is
being multiplied by
the same matrix of c
components
This is a Kronecker
Product of the
matrix of r
components with
the matrix of c
components
With separable blur, we were able to rewrite the full blurring matrix A
as the Kronecker product of matrices Ar and Ac , both of which are
much smaller than the full matrix A.
The structure of both Ar and Ac is Toeplitz
Ar Ac
24. WHAT WE NEED:
For our m x n image matrix X, we need a corresponding PSF array P
If the blur is separable, we can determine vectors c and r such that
Then we can use the entries of c and r to create m x m Ac and n x n Ar
We can do this because we know the Toeplitz structure of those
matrices
25. WHAT WE NEED:
The Kronecker Product of Ar and Ac gives us the full mn x mn blurring
matrix A
Alternatively, we can use Ar and Ac to blur X directly
Ax = b
…is equivalent to…
Ac X Ar
T = B
26. SEPARATING THE BLUR
If the PSF array P is a rank 1 matrix, then c and r can be determined as
follows, using the Singular Value Decomposition of P:
c = σ1 u1
r = σ1 v1
where σ1 is the first (largest) singular value of P, and u1 and v1 are its
corresponding left and right singular vectors.
27. DEBLURRING
Ax = b
Now we’ve sufficiently built up our model of how A blurs a given image:
Then, given a blurry image, we may deduce that it can be deblurred by
However, such a reconstruction of the sharp image would be naïve,
because our recorded blurry image likely contains more than just blur
Our recorded image also contains noise, errors caused by factors
such as technical limitations and poor lighting
x = A-1b
28. where bexact = Ax and e represents the noise component.
Then our naïve reconstruction can be expressed as:
DEBLURRING
b = bexact + e
xnaïve = A-1b
Our recorded image can be expressed as:
xnaïve = A-1(bexact + e)
xnaïve = x + A-1e
xnaïve = A-1bexact + A-1e
29. SINGULAR VALUE DECOMPOSITION OF A
= Σi = 1 viui
TN
σi
A-1
A-1e= Σi = 1 ui
Te
N
σi
vi
where σi are the strictly positive singular values of A, arranged in a non-
increasing order, σ1 ≥ σ2 ≥ … ≥ σN > 0. Then the inverted noise can be
expressed as:
We’ll use the singular value decomposition of A to cut out as much of
the inverted noise as possible.
So the inverted noise can be written as a linear combination of the right
singular vectors, vi
30. A-1e= Σi = 1 ui
Te
N
σi
vi
SINGULAR VALUE DECOMPOSITION OF A
Because the singular vectors are ordered such that σ1 ≥ σ2 ≥ … ≥ σN > 0,
the coefficients on vi get very large as i gets large
So the later terms in the sum contribute more to the inverted
noise
Removing terms from the end of the sum will reduce the size
of the error, at the cost of sacrificing some of the image’s
information
31. TRUNCATION
xnaïve = A-1b = Σi = 1 ui
Tb
N
σi
vi
If we choose k < N, then our truncated reconstruction is,
Our “naïve” reconstruction can be represented as,
xk = Σi = 1 ui
Tb
k
σi
vi
This is called the Truncated SVD method, or the TSVD
32. SPECTRAL FILTERING
where Φi is the filter factor for the ith term in the sum
The truncated SVD method is actually an example of a more general class of
reconstruction methods called spectral filtering
The filtered reconstruction is defined as:
The TSVD can be defined in terms of this general model by allowing:
xfilt = Σi = 1
ui
TbN
σi
viΦi
Φi =
1, i = 1, … , k
0, i = k + 1, … , N{
33. “R” EXAMPLE
In the program R, I created a 9 x 13 image matrix X, and defined the PSF
to take the point source and blur it to a square of 4 pixels
Instead of constructing the 117 x 117 blurring matrix A column by
column, I created the 9 x 9 Ac and the 13 x 13 Ar , and used those
to blur my image
![IMAGES AS MATRICES
Images can be represented as matrices, with each entry in the matrix
containing the intensity of the corresponding image’s pixel
Grayscale intensities lie on an interval [0, n], where 0 = black, n = white
1040 x 857
image
(length by
width)
857 x 1040
matrix
(rows by
columns)
x1 1 x1 2 x1 1040
x2 1
x857 1 x857 1040
x2 2 x2 1040
Grayscale images are represented by single matrices
Color images are represented by multidimensional matrices](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)