Compressive Spectral Image Sensing, Processing, and Optimization

986 views

Published on

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
986
On SlideShare
0
From Embeds
0
Number of Embeds
43
Actions
Shares
0
Downloads
93
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Compressive Spectral Image Sensing, Processing, and Optimization

  1. 1. Compressive Spectral Image Sensing and Optimization Gonzalo R. Arce Charles Black Evans Professor University of Delaware Newark, Delaware, USA 19716 Distinguished Lecturer Series Aristotle University of Thessaloniki March 14, 2014
  2. 2. Optical imaging and spectroscopy discovers the characteristics of scenes and materials by capturing EM radiation in the 0.01 to 10000 nm spectrum window. Sensitive not only to spatial and spectral information of a scene, but also to polarization, tomographic, angular, and even chemical composition. Multidimensional imaging provides dimension preserving mappings y = Hf, where H is the “forward model“, characterizing the focal plane data. Optical coding shapes H, under some criteria, to match the computational tools of inverse problems to significantly improve the overall imaging performance. MURA Hadamard SPECT Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 2 / 50
  3. 3. THE SPECTRAL IMAGING PROBLEM? Push broom spectral imaging: Traditional approach, expensive, low sensing speed, senses N × N × L voxels Optical Filters; Again senses N × N × L voxels; limited by number of colors Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 3 / 50
  4. 4. Compressive Spectral Imaging (CASSI), New revolutionary method, CS makes a significant difference, senses only N2 N × N × L Coded apertures are the only variable element. Coded Apertures are the key elements in CASSI. Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 4 / 50
  5. 5. WHY IS THIS IMPORTANT? Remote sensing and surveillance Visible, NIR, SWIR Devices are challenging in NIR and SWIR: cost, size, resolution, cooling Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 5 / 50
  6. 6. WHY IS THIS IMPORTANT? Medical Imaging: Vascular tissue imaging, angiography, contrast agent paint restoration Compressive Spectral Imaging Reduce sensing complexity Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 6 / 50
  7. 7. Introduction Compressive sensing introduced by Donoho†, Candès‡, Tao, Romberg... Measurements are given by y = Φx y xΦ M x 1 Measurements M x N Sampling Operator N x 1 Sparse Signal A sparse solution x is recovered from y by solving the inverse problem ˆx = min x x 1 s.t. y = Φx. † Donoho. IEEE Trans. on Information Theory. December 2006. ‡ Candès, Romberg and Tao. IEEE Trans. on Information Theory. April 2006. Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 7 / 50
  8. 8. Introduction Measurements are given by y = Φx y ΨΦ α x A sparse solution α is recovered from y by solving the inverse problem ˆα = min α α 1 s.t. y = ΦΨα. Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 8 / 50
  9. 9. Introduction Datacube f = Ψθ Compressive Measurements g = HΨθ + w Underdetermined system of equations ˆf = Ψ{min θ g − HΨθ 2 + τ θ 1} Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 9 / 50
  10. 10. Coded-Aperture Spectral Imaging (CASSI) Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 10 / 50
  11. 11. Coded-Aperture Spectral Imaging (CASSI) Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 11 / 50
  12. 12. Undetermined system of equations: N × M × L Unknowns and N(M + L − 1) Equations. Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 12 / 50
  13. 13. Computational Model A single shot compressive measurement across the FPA: Gnm = L−1 i=0 Fi(n+m)mTi(n+m) + win F is the N × M × L datacube T is the binary code aperture w is the sensing noise In vector form, the FPA measurement can be written as g = Hf + w H accounts for the aperture code and the dispersive element. Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 13 / 50
  14. 14. CASSI Multishot Matrix Model      g0 g1 ... gk−1      =      H0 H1 ... Hk−1      f, (1) g = Hf, (2) where H ∈ {0, 1}N(M+L−1)K×NML . Multi-shot coding done by using multiple coded apertures or a Digital-Micromirror-Device (DMD) Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 14 / 50
  15. 15. Matrix CASSI representation g = Hf Data cube: N × N × L Spectral bands: L Spatial resolution: N × N Sensor size N × (N + L − 1) V=N(N+L-1) Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 15 / 50
  16. 16. Coded Aperture Optimization: Sensing and Reconstruction Restricted Isometry Property in CASSI Given g = HΨ A θ with |θθθ| = S The RIP of the CASSI matrix A is defined as the smallest constant δs such that (1 − δs) ||θθθ||2 2 ≤ ||Aθθθ||2 2 ≤ (1 + δs) ||θθθ||2 2, (3) where δs = max T⊂[n],|T|≤S λmax A|T||T| − I (4) A|T||T| = AT |T|A|T|, A|T| is a m × |T| matrix whose columns are equal to |T| columns of the CASSI matrix A, and λmax (.) denotes the largest eigenvalue. Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 16 / 50
  17. 17. Let the entries of ΨΨΨ be Ψj,k and let the columns of ΨΨΨ be [ψψψ0, . . . ,ψψψn−1]. The entries of A|T| can be written as A|T| jk = (HψψψΩk )j = hT j ψψψΩk = L−1 r=0 ti j−rN Ψj+r(N ),Ωk for j = 0, . . . , m − 1, k = 0, . . . , |T| − 1, where i = j/V , N = N2 − N, and Ωk ∈ {0, . . . , n − 1}. The entries of A|T||T| can be expressed as A|T||T| jk = K−1 i=0 V−1 =0 L−1 r=0 L−1 u=0 ti −rN ti −uN Ψ +rN ,Ωj Ψ +uN ,Ωk (5) for j, k = 0, . . . , |T| − 1. Note that the coded aperture products ti −rN ti −uN determine the eigenvalues of A|T||T|, and consequently they determine the constant δs in the RIP. Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 17 / 50
  18. 18. Boolean Coded Apertures An optimal ensemble of four 64 × 64 boolean coded apertures. Each spatial coordinate in the ensemble contains only one 1-value entry. Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 18 / 50
  19. 19. Bernoulli Coded Apertures An ensemble of four 64 × 64 Bernoulli coded apertures. Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 19 / 50
  20. 20. Performance of coded apertures Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 20 / 50
  21. 21. Original datacube Boolean (40.4dB) Unsigned grayscale (31.2dB) Binary (27.7dB) Hadamard (27.7dB) Signed grayscale (22.7dB) Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 21 / 50
  22. 22. Original datacube Boolean Unsigned grayscale Binary Hadamard Signed grayscale Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 22 / 50
  23. 23. Coded Aperture Optimization: Spectral Selectivity UAV sensor requirements depend on flight duration, range, altitude, etc. Need: Hyperspectral imaging that dynamically adapts to optimal spectral bands. Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 23 / 50
  24. 24. Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 24 / 50
  25. 25. ((a)) Original ((b)) 12 Random Codes ((c)) 9 Optimal Codes The resulting spectral data cubes are shown as they would be viewed by a Stingray F-033C CCD Color Camera. Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 25 / 50
  26. 26. ((a)) Random Code ((b)) Original ((c)) Optimal Code ((d)) Random Codes ((e)) Optimized Codes Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 26 / 50
  27. 27. ((a)) Random ((b)) Optimized ((c)) ((d)) Differences between the original and the reconstructed 3rd spectral channel (479nm) Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 27 / 50
  28. 28. ((a)) Original ((b)) 12 Random Codes ((c)) 12 Optimized Codes The resulting spectral data cubes are shown as they would be viewed by a Stingray F-033C CCD Color Camera. Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 28 / 50
  29. 29. Coded Aperture Optimization: Image Classification Goal: classification of a spectral scene using Compressive measurements Optimal code aperture Sparsity signal model Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 29 / 50
  30. 30. Given the FPA measurements g, every test pixel fi belongs to one of the P known classes {H(1) , H(2) , ..., H(P) } 0 10 20 30 40 50 60 70 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Spectrum Band Glutamine 0 10 20 30 40 50 60 70 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Spectrum Band Histidine 0 10 20 30 40 50 60 70 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Spectrum Band Isoleucine If a pixel fi ∈ H(k), its spectral profile lies in a low-dimensional subspace spanned by the training samples: {s (k) j }j=1,...,Np fi ≈ [s (k) 1 , s (k) 2 , ..., s (k) Nk ][α (k) 1 , α (k) 2 , ..., α (k) Np ]T = S(k) α(k) where, α(k) is a sparse vector.Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 30 / 50
  31. 31. Sparsity Model Combining all class sub-dictionaries fi ≈ [S(1) , ..., S(k) , ..., S(P) ][α(1) , ..., α(k) , ..., α(P) ]T = Sα Ideally, if fi ∈ H(k), then α(j) = 0; ∀j = 1, ..., P; j = k. Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 31 / 50
  32. 32. Superposition of 3 shots for the different codes Optimal codes 5 10 15 20 25 30 5 10 15 20 25 30 0 0.5 1 1.5 2 2.5 3 Hadamard Codes 5 10 15 20 25 30 5 10 15 20 25 30 0 0.5 1 1.5 2 2.5 3 Bernoulli codes 5 10 15 20 25 30 5 10 15 20 25 30 0 0.5 1 1.5 2 2.5 3 Ck = K−1 k=0 (tk m,n)2 at each (m, n) spatial location. Optimal Hadamard Bernoulli Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 32 / 50
  33. 33. Image Classification Results 200 spectral bands. 50 shots. Ground-truth Single spectral band SVM-full datacube (73.5%) Bernoulli-25%datacube (64.3%) Hadamard-25%datacube (63.4%) Optimal-25%datacube (73.7%) Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 33 / 50
  34. 34. Image Classification Results 200 spectral bands. 50 shots. Ground-truth FPA measurement SVM-full datacube (73.5%) Bernoulli-25%datacube (64.3%) Hadamard-25%datacube (63.4%) Optimal-25%datacube (73.7%) Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 34 / 50
  35. 35. Colored Coded Aperture Spectral Imaging Patterned coating combines micro-lithography with optical coating technology. Precision patterned coating and patterns Sub-pixel alignment accuracy Ultraviolet, visible, NIR, SWIR Multi-filter arrays on monolithic substrates Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 35 / 50
  36. 36. NEW FAMILY OF CODED APERTURES Boolean Spectrally Selective Super-resolution Colored Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 36 / 50
  37. 37. Colored coded aperture model Colored coded aperture is a color filter array Each entry is a wavelength selective color filter 3D Mask model has the same dimensions than the objective discrete data cube Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 37 / 50
  38. 38. Linear dispersion and focal plane array integration Linear shifting operation Focal plane array (FPA) projections The number of pixels of the FPA detector is N(N + L − 1) N2 L (size of the spectral data cube) Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 38 / 50
  39. 39. Random Boolean Code Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 39 / 50
  40. 40. 4 Colors Random Code Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 40 / 50
  41. 41. Restricted Isometry Property of Colored CASSI A = HΨΨΨ, fff = ΨΨΨθθθ, ΨΨΨ = W ⊗ ΨΨΨ2D Definition (1 − δs) ||θθθ||2 2 ≤ ||Aθθθ||2 2 ≤ (1 + δs) ||θθθ||2 2, δs = max T ⊂[N2L],|T |≤S ||AT |T |A|T | − I||2 2, A|T |, |T | columns of A indexed by the set T δs = max T ⊂[N2L],|T |≤S λmax A|T ||T | − I A|T | ir = hiψψψΩr = L−1 k=0 t i k mi −kN Ψmi +k(N ),Ωr (hi )j =    t i kj i− i V−kj N , if i − i V = j − kj N 0, otherwise, A = HΨΨΨ2D A = H W ⊗ ΨΨΨ2D Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 41 / 50
  42. 42. A|T ||T | = AT |T |A|T | A|T ||T | r,u = K−1 =0 V−1 i=0 L−1 k1=0 L−1 k2=0 tk1 i−k1N tk2 i−k2N Ψi+k1N ,ΩrΨi+k2N ,Ωu Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 42 / 50
  43. 43. Results: LH-Colored Coded Aperture A geometric interpretation of the colored coded apertures for LH-Colored filters (3 shots). Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 43 / 50
  44. 44. Code Design Results B-Colored Coded Apertures (Band Pass Filters) Geometric interpretation of colored coded apertures for B-filters (3 shots) Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 44 / 50
  45. 45. 461nm 470nm 479nm 488nm 497nm 506nm 515nm 524nm 533nm 542nm 551nm 560nm 569nm 578nm 587nm 596nm 16 channels 461-596nm 256 × 256 pixels Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 45 / 50
  46. 46. Reconstruction From B-Colored Coded Apertures 2 4 6 8 10 12 14 30 40 50 60 70 80 90 Number of Shots PSNR Block Unblock Random B-colored B-colored Mean PSNR of the reconstructed data cubes. Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 46 / 50
  47. 47. Reconstructions from Real Measurements (RGB) K = 1 K = 1 Sh=1 (i) 1 snapshot: Photomask, Coloring, Optimal K = 4 K = 4 Sh=4 (j) 4 snapshots: Photoask, Coloring, Optimal Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 47 / 50
  48. 48. Reconstruction Results (Photomask vs Colored Coded Apertures) 454 nm K = 1 468 nm K = 1 485 nm K = 1 506 nm K = 1 529 nm K = 1 557 nm K = 1 594 nm K = 1 639 nm K = 1 454 nm K = 1 468 nm K = 1 485 nm K = 1 506 nm K = 1 529 nm K = 1 557 nm K = 1 594 nm K = 1 639 nm K = 1 (a) One snapshot reconstruction. (First row) Photomask, (Second row) Coloring 454 nm K = 4 468 nm K = 4 485 nm K = 4 506 nm K = 4 529 nm K = 4 557 nm K = 4 594 nm K = 4 639 nm K = 4 454 nm K = 4 468 nm K = 4 485 nm K = 4 506 nm K = 4 529 nm K = 4 557 nm K = 4 594 nm K = 4 639 nm K = 4 (b) 4 snapshots reconstructions. (First row) Photomask, (Second row) Coloring Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 48 / 50
  49. 49. Spectral Reconstruction From Colored Coded Apertures Sh=4 P1 P2 454 468 485 506 529 557 594 639 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Nanometers Norm.Intensity Spectral Comparison: 4 Snapshots Spectrometer Photomask Coloring Optimal 454 468 485 506 529 557 594 639 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Nanometers Norm.Intensity Spectral Comparison: 4 Snapshots Spectrometer Photomask Coloring Optimal Pixel P1 Pixel P2 Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 49 / 50
  50. 50. Conclusions Optical Coding for Compressive Spectral Imaging Good codes for reconstruction, classification, unmixing Colored codes offer multidimensional coding - open problem Convolution Optical Coding (Projections) Light field imaging X-ray tomography Super-resolution microscopy Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 50 / 50

×