Compressive Spectral Image Sensing, Processing, and Optimization

Compressive Spectral Image
Sensing and Optimization
Gonzalo R. Arce
Charles Black Evans Professor
University of Delaware
Newark, Delaware, USA 19716
Distinguished Lecturer Series
Aristotle University of Thessaloniki
March 14, 2014

Optical imaging and spectroscopy discovers the characteristics of scenes and
materials by capturing EM radiation in the 0.01 to 10000 nm spectrum window.
Sensitive not only to spatial and spectral information of a scene, but also to
polarization, tomographic, angular, and even chemical composition.
Multidimensional imaging provides dimension preserving mappings
y = Hf,
where H is the “forward model“, characterizing the focal plane data.
Optical coding shapes H, under some criteria, to match the computational tools
of inverse problems to signiﬁcantly improve the overall imaging performance.
MURA Hadamard SPECT
Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 2 / 50

THE SPECTRAL IMAGING PROBLEM?
Push broom spectral imaging: Traditional approach, expensive, low
sensing speed, senses N × N × L voxels
Optical Filters; Again senses N × N × L voxels; limited by number of
colors

Compressive Spectral Imaging (CASSI), New revolutionary method, CS
makes a signiﬁcant difference, senses only N2
N × N × L
Coded apertures are
the only variable
element.
Coded Apertures are
the key elements in
CASSI.

WHY IS THIS IMPORTANT?
Remote sensing and surveillance
Visible, NIR, SWIR
Devices are challenging in NIR and SWIR: cost, size,
resolution, cooling

WHY IS THIS IMPORTANT?
Medical Imaging: Vascular tissue imaging, angiography,
contrast agent
paint restoration
Compressive Spectral Imaging
Reduce sensing complexity

Introduction
Compressive sensing introduced by Donoho†, Candès‡, Tao,
Romberg...
Measurements are given by y = Φx
y
xΦ
M x 1
Measurements
M x N
Sampling Operator
N x 1
Sparse Signal
A sparse solution x is recovered from y by solving the inverse
problem
ˆx = min
x
x 1 s.t. y = Φx.
†
Donoho. IEEE Trans. on Information Theory. December 2006.
‡
Candès, Romberg and Tao. IEEE Trans. on Information Theory. April 2006.

Introduction
Measurements are given by y = Φx
y ΨΦ α
x
A sparse solution α is recovered from y by solving the inverse
problem
ˆα = min
α
α 1 s.t. y = ΦΨα.

Introduction
Datacube
f = Ψθ
Compressive Measurements
g = HΨθ + w
Underdetermined system of equations
ˆf = Ψ{min
θ
g − HΨθ 2 + τ θ 1}

Coded-Aperture Spectral Imaging (CASSI)

Undetermined system of equations: N × M × L Unknowns and
N(M + L − 1) Equations.

Computational Model
A single shot compressive measurement across the FPA:
Gnm =
L−1
i=0
Fi(n+m)mTi(n+m) + win
F is the N × M × L datacube
T is the binary code aperture
w is the sensing noise
In vector form, the FPA measurement can be written as
g = Hf + w
H accounts for the aperture code and the dispersive element.

CASSI Multishot Matrix Model





g0
g1
...
gk−1





=





H0
H1
...
Hk−1





f, (1)
g = Hf, (2)
where H ∈ {0, 1}N(M+L−1)K×NML
.
Multi-shot coding done by using multiple coded apertures or a
Digital-Micromirror-Device (DMD)

Matrix CASSI representation g = Hf
Data cube:
N × N × L
Spectral bands: L
Spatial resolution:
N × N
Sensor size
N × (N + L − 1)
V=N(N+L-1)

Coded Aperture Optimization: Sensing and
Reconstruction
Restricted Isometry Property in CASSI
Given
g = HΨ
A
θ with |θθθ| = S
The RIP of the CASSI matrix A is deﬁned as the smallest constant δs
such that
(1 − δs) ||θθθ||2
2 ≤ ||Aθθθ||2
2 ≤ (1 + δs) ||θθθ||2
2, (3)
where
δs = max
T⊂[n],|T|≤S
λmax A|T||T| − I (4)
A|T||T| = AT
|T|A|T|, A|T| is a m × |T| matrix whose columns are equal to
|T| columns of the CASSI matrix A, and λmax (.) denotes the largest
eigenvalue.

Let the entries of ΨΨΨ be Ψj,k and let the columns of ΨΨΨ be [ψψψ0, . . . ,ψψψn−1].
The entries of A|T| can be written as
A|T| jk
= (HψψψΩk
)j = hT
j ψψψΩk
=
L−1
r=0
ti
j−rN
Ψj+r(N ),Ωk
for j = 0, . . . , m − 1, k = 0, . . . , |T| − 1, where i = j/V , N = N2 − N,
and Ωk ∈ {0, . . . , n − 1}. The entries of A|T||T| can be expressed as
A|T||T| jk
=
K−1
i=0
V−1
=0
L−1
r=0
L−1
u=0
ti
−rN
ti
−uN
Ψ +rN ,Ωj
Ψ +uN ,Ωk
(5)
for j, k = 0, . . . , |T| − 1.
Note that the coded aperture products ti
−rN
ti
−uN
determine the
eigenvalues of A|T||T|, and consequently they determine the constant
δs in the RIP.

Boolean Coded Apertures
An optimal ensemble of four 64 × 64 boolean coded apertures.
Each spatial
coordinate in the
ensemble contains
only one 1-value
entry.

Bernoulli Coded Apertures
An ensemble of four 64 × 64 Bernoulli coded apertures.

Performance of coded apertures

Original datacube Boolean (40.4dB) Unsigned grayscale (31.2dB)
Binary (27.7dB) Hadamard (27.7dB) Signed grayscale (22.7dB)

Original datacube Boolean Unsigned grayscale
Binary Hadamard Signed grayscale

Coded Aperture Optimization: Spectral Selectivity
UAV sensor requirements depend on ﬂight duration, range,
altitude, etc.
Need: Hyperspectral imaging that dynamically adapts to optimal
spectral bands.

((a)) Original ((b)) 12 Random Codes ((c)) 9 Optimal Codes
The resulting spectral data cubes are shown as they would be viewed by a Stingray
F-033C CCD Color Camera.

((a)) Random Code ((b)) Original ((c)) Optimal Code
((d)) Random Codes ((e)) Optimized Codes

((a)) Random ((b)) Optimized
((c)) ((d))
Differences between the original and the reconstructed 3rd
spectral channel (479nm)

((a)) Original ((b)) 12 Random Codes ((c)) 12 Optimized Codes
The resulting spectral data cubes are shown as they would be viewed by a Stingray
F-033C CCD Color Camera.

Coded Aperture Optimization: Image Classiﬁcation
Goal: classiﬁcation of a spectral scene using
Compressive measurements
Optimal code aperture
Sparsity signal model

Given the FPA measurements g, every test pixel fi belongs to one of
the P known classes
{H(1)
, H(2)
, ..., H(P)
}
0 10 20 30 40 50 60 70
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Spectrum Band
Glutamine
0 10 20 30 40 50 60 70
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Spectrum Band
Histidine
0 10 20 30 40 50 60 70
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Spectrum Band
Isoleucine
If a pixel fi ∈ H(k), its spectral proﬁle lies in a low-dimensional
subspace spanned by the training samples: {s
(k)
j }j=1,...,Np
fi ≈ [s
(k)
1 , s
(k)
2 , ..., s
(k)
Nk
][α
(k)
1 , α
(k)
2 , ..., α
(k)
Np
]T
= S(k)
α(k)
where, α(k) is a sparse vector.Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 30 / 50

Sparsity Model
Combining all class sub-dictionaries
fi ≈ [S(1)
, ..., S(k)
, ..., S(P)
][α(1)
, ..., α(k)
, ..., α(P)
]T
= Sα
Ideally, if fi ∈ H(k), then
α(j) = 0; ∀j = 1, ..., P; j = k.

Superposition of 3 shots for the different codes
Optimal codes
5 10 15 20 25 30
5
10
15
20
25
30
0
0.5
1
1.5
2
2.5
3
Hadamard Codes
5 10 15 20 25 30
5
10
15
20
25
30
0
0.5
1
1.5
2
2.5
3
Bernoulli codes
5 10 15 20 25 30
5
10
15
20
25
30
0
0.5
1
1.5
2
2.5
3
Ck = K−1
k=0 (tk
m,n)2 at each (m, n) spatial location.
Optimal Hadamard Bernoulli

Image Classiﬁcation Results
200 spectral bands. 50 shots.
Ground-truth Single spectral band SVM-full datacube (73.5%)
Bernoulli-25%datacube
(64.3%)
Hadamard-25%datacube
(63.4%)
Optimal-25%datacube (73.7%)

Image Classiﬁcation Results
200 spectral bands. 50 shots.
Ground-truth FPA measurement SVM-full datacube (73.5%)
Bernoulli-25%datacube
(64.3%)
Hadamard-25%datacube
(63.4%)
Optimal-25%datacube (73.7%)

Colored Coded Aperture Spectral Imaging
Patterned coating combines micro-lithography with optical coating
technology.
Precision patterned coating
and patterns
Sub-pixel alignment accuracy
Ultraviolet, visible, NIR, SWIR
Multi-ﬁlter arrays on monolithic
substrates

NEW FAMILY OF CODED APERTURES
Boolean Spectrally Selective
Super-resolution Colored

Colored coded aperture model
Colored coded aperture is a color ﬁlter array
Each entry is a wavelength selective color ﬁlter
3D Mask model has the same dimensions than the objective discrete data cube

Linear dispersion and focal plane array integration
Linear shifting operation
Focal plane array (FPA) projections
The number of pixels of the FPA
detector is N(N + L − 1) N2
L
(size of the spectral data cube)

Random Boolean Code

4 Colors Random Code

Restricted Isometry Property of Colored CASSI
A = HΨΨΨ, fff = ΨΨΨθθθ, ΨΨΨ = W ⊗ ΨΨΨ2D
Deﬁnition
(1 − δs) ||θθθ||2
2 ≤ ||Aθθθ||2
2 ≤ (1 + δs) ||θθθ||2
2,
δs = max
T ⊂[N2L],|T |≤S
||AT
|T |A|T | − I||2
2,
A|T |, |T | columns of A indexed by the set T
δs = max
T ⊂[N2L],|T |≤S
λmax A|T ||T | − I
A|T | ir
= hiψψψΩr
=
L−1
k=0
t i
k mi −kN
Ψmi +k(N ),Ωr
(hi )j =



t i
kj i− i V−kj N
, if i − i V = j − kj N
0, otherwise,
A = HΨΨΨ2D
A = H W ⊗ ΨΨΨ2D

A|T ||T | = AT
|T |A|T |
A|T ||T | r,u
=
K−1
=0
V−1
i=0
L−1
k1=0
L−1
k2=0
tk1
i−k1N
tk2
i−k2N
Ψi+k1N ,ΩrΨi+k2N ,Ωu

Results: LH-Colored Coded Aperture
A geometric interpretation of the colored coded apertures for LH-Colored ﬁlters (3
shots).

Code Design Results B-Colored Coded Apertures (Band Pass Filters)
Geometric interpretation of colored coded apertures for B-ﬁlters (3 shots)

461nm 470nm 479nm 488nm
497nm 506nm 515nm 524nm
533nm 542nm 551nm 560nm
569nm 578nm 587nm 596nm
16 channels
461-596nm
256 × 256
pixels

Reconstruction From B-Colored Coded Apertures
2 4 6 8 10 12 14
30
40
50
60
70
80
90
Number of Shots
PSNR Block Unblock
Random B-colored
B-colored
Mean PSNR of the reconstructed data cubes.

Reconstructions from Real Measurements (RGB)
K = 1 K = 1
Sh=1
(i) 1 snapshot: Photomask, Coloring, Optimal
K = 4 K = 4
Sh=4
(j) 4 snapshots: Photoask, Coloring, Optimal

Reconstruction Results (Photomask vs Colored
Coded Apertures)
454 nm K = 1 468 nm K = 1 485 nm K = 1 506 nm K = 1 529 nm K = 1 557 nm K = 1 594 nm K = 1 639 nm K = 1
(a) One snapshot reconstruction. (First row) Photomask, (Second row) Coloring
(b) 4 snapshots reconstructions. (First row) Photomask, (Second row) Coloring

Spectral Reconstruction From Colored Coded Apertures
Sh=4
P1
P2
454 468 485 506 529 557 594 639
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Nanometers
Norm.Intensity
Spectral Comparison: 4 Snapshots
Spectrometer
Photomask
Coloring
Optimal
454 468 485 506 529 557 594 639
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Nanometers
Norm.Intensity
Spectral Comparison: 4 Snapshots
Spectrometer
Photomask
Coloring
Optimal
Pixel P1
Pixel P2

Conclusions
Optical Coding for Compressive Spectral Imaging
Good codes for reconstruction, classiﬁcation, unmixing
Colored codes offer multidimensional coding - open problem
Convolution Optical Coding (Projections)
Light ﬁeld imaging
X-ray tomography
Super-resolution microscopy

Compressive Spectral Image Sensing, Processing, and Optimization

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Compressive Spectral Image Sensing, Processing, and Optimization

Similar to Compressive Spectral Image Sensing, Processing, and Optimization (20)

Recently uploaded

Recently uploaded (20)

Compressive Spectral Image Sensing, Processing, and Optimization