1) The document describes a spectral ghost imaging snapshot camera that uses a spatial random phase modulator to modulate a 3D data cube into a 2D plane and compressed sensing to acquire spectral information below the Nyquist rate. 2) It presents the system model and calibration process for the camera, including an interpolated calibration method. 3) An image reconstruction framework is described that uses total variation regularization and nuclear norm minimization to exploit the spatial and spectral correlations in hyperspectral images. A gradient projection algorithm with Barzilai-Borwein stepsizes is used for reconstruction.

1. Shiyu Tan1,2 Shensheng Han1 Ming Gu2
1 Shanghai Institute of Optics and Fine Mechanics, Chinese Academy of Sciences
2 Department of Mathematics, University of California Berkeley
April 19, 2017 UC Berkeley
中国科学院上海光学精密机械研究所
Shanghai Institute of Optics and Fine Mechanics, CAS
Image Reconstruction in Snapshot Spectral GISC
camera

2. Spectral GISC Camera: spectral imaging based on ghost imaging via sparsity constraints
 Spatial random phase modulator modulate 3D data-cube into a 2D data-plane
 Compressed sensing (CS) acquire information below Nyquist rate
Z Liu & S Tan. Spectral Camera based on Ghost Imaging via Sparsity Constraints. Scientific. Reports (2016).
point-point point-plane
Snapshot Spectral GISC Imaging
diagonal block square sensing matrix full rank fat sensing matrix
M N

3. (a) The object plane; (b) the first image plane; (c) the speckles plane
(1) an imaging system; (2) a spatial random phase modulator; (3) a microscope objective; (4) CCD detector
Schematic of Spectral GISC Camera
speckleobject
 Imaging system: project object to the first image plane
 Spatial random phase modulator : modulate image into a speckle
 Microscope objective and detector: magnify and record the speckle
Z Liu & S Tan. Spectral Camera based on Ghost Imaging via Sparsity Constraints. Scientific. Reports (2016).

4. System Mathematical Model
 Calibration of system response
 System model
first image plane speckles plane
 , ,i j 
Point light
source
Speckle generated
by point light
1 2M M
ij

S
31 2 NN N
ij ij
j i
 

 Y S X
 unknown image
 system Response
 measurement
1 2 3N N N 
X
  1 2M M
ij ij 

S S
1 2M M
Y
1 2N N -size of image -number of wavelengths -size of speckle3N 1 2M M
y Ax
 unknown image
 sensing Matrix
 measurement
N
x
M N
A
M
y
 weeks of calibration
 TB storage/memory
6
:300 300 50 4.5 10N    
6
:1024 1024 1.05 10M   
large-scale A
Z Liu & S Tan. Spectral Camera based on Ghost Imaging via Sparsity Constraints. Scientific. Reports (2016).

5. Interpolated Calibration
 Divide the first image plane into several blocks
  


  


 


1,1 1, 1k  1,2 1k 
1,1k 
1,N
 
1, 1k k   1,2 1k k 

 
   
2 1,1k   2 1, 1k k 
 1,k N
 2 1,2 1k k  2 1,k N
,1N  , 1N k   ,2 1N k    ,N N
 ,i j  ,i j k
 ,i k j  ,i k j k 
 ,i j 
k -interval
, ,
, ,
, ,
, ,
i j
i k j
i j k
i k j k






 
S
S
S
S
First Image Plane
 Calibrate and Storage
Four speckles generated by
the vertex point light of a
block for fixed wavelength

6.  Approximate shift-invariance
 Bilinear translational Interpolation (fixed wavelength)
 Fast Convolution (Blocked FFT)
      ', ', , ,, ,,i j i j x i i y j j
k i i k j j
x y
k k
 
    
    
  S S
    , , ' , 'i j k x i i y j j k
k i i k j j k
k k
     
     
  S
    , , ' , 'i k j x i i k y j j
k i i k k j j
k k
     
     
  S
    , , ' , '
' '
i k j k x i i k y j j k
k i i k k j j k
k k
       
     
  S
 ,i j  ,i j k
 ,i k j  ,i k j k 
 ,i j 
k -interval
Interpolated Calibration
   , , , ,i i j j ijx y x x y y       S S
 
3 2 1
2 13
2 1 13
' ' ' '
' '
/ /
, ,
/ / /
mat
( , )
ˆ ˆ( , ) ( , )
N N N
i j i j
j i
N k N kN k k
ij i m j n
j i m k n k
N k N k N N kN k k
ij ij ij ij
j i m k n k j i
k m k n
x m y n
k k
x m y n m n
 

 

   

      
 
 
          
 
 
   
      
 
 
      
 

    
     
Ax S X
S X
S X S X
23 /kN

  
 

7. Image Reconstruction Framework
 Spectral images: spatial correlation and spectral correlation
 TV_RANK reconstruction
 Total Variation (TV) norm
 Nuclear norm: low-rank convex approximation
 IRTV_RANK reconstruction
 Iterative reweighted method
1 2, 0   -nonnegative parameters1 2 3 1 2 3
, ,N N N N N NN   
  x X X -unknown spectral image
2
1 22 *
min . . 0TV
s t    
x
y Ax X X x
Z Liu & S Tan. Spectral Camera based on Ghost Imaging via Sparsity Constraints. Scientific. Reports (2016).
     
2 2
1, , , 1,
,
:,:, +i j ij i j ijTV
i j
   
 
      X X X X X X
* i
i
 X
   2
1 22 *
min . . 0TV
W W s t    
x
y Ax X X x
Total Variation
   
 
   
     
 
   
     
 
2 2
1, , , 1,
2 2
,
1, , , 1,
+
+
k k k k
i j ij i j ijk k
TV TV k k k ki j
i j ij i j ij
W
   

   
 
 
 

 

X X X X
X X
W W W W
   
     
 * *
k k k kT
w wW trX X U XV
           1 1 1
, ,k k k k k k
w w
  
  W X U U V V

8. Image Reconstruction Framework
Z Liu & S Tan. Spectral Camera based on Ghost Imaging via Sparsity Constraints. Scientific. Reports (2016).
 PICHCS reconstruction framework
prior image constrained hyperspectral compressed sensing
 primary image: solving IRTV_RANK
 prior image: averaging the adjacent spectral primary image
 difference image: subtracting image with corresponding prior image
   2
1 22 *
arg min . . 0T TV
W W s t     
X
X y Ax X X x
   0
1
2 1
p Ti i j
j LL 
 


X X
1
2
2
min + , s.t. 0pl


      
x
y A x x x x
p  x x x
difference images

9.  Gradient projection with Barzilai-Borwein stepsize (GPBB)
 Compute GP step
 Line search
 Update BB steps
     1k k k

 x x δ
 
   
 
   
   2
,
mid 0, ,1
,
k k
k
k k k
F
F

   
  
 
 
δ x
δ x δ
 
   
     
     
2
1 2
min max
2
mid , ,
,
k k
k
k k k k k
F

  
 

 
 
  
 
 
δ
δ x δ
 min . . 0F s t 
x
x x
     2
1 22 *
k k
w w
TV
F    x y Ax X X
Reconstruction Algorithm
       
    k k k k k
F

   δ x x x

10. SR=0.1 SNR=5 rRMSE=0.5247
Columbia University multispectral image dataset
Original Image 530nm-670nm 140*100*15
Numerical Results
SNR /y n 
2
2
ˆ
rRMSE
ˆ


x x
x
SR /M N M N
SR=0.3 SNR=10 rRMSE=0.1814
 Gaussian random matrix
 Gaussian white noise
 Sampling Ratio
 Signal-to-noise Ratio
 Relative RMSE
 y Ax n

11. Experimental Results
Reference Image
IRTV_RANK rRMSE=0.3681
SR=0.09
PICHCS rRMSE=0.1790
140*140*9
S Tan. Prior Images Constrained Hyperspectral Compressed Sensing. Acta Optica Sinica, (2015)

12. 550nm
650nm
Experimental Results
Aloe
CR=30%
140*140*18
Mario
Z Liu & S Tan. Spectral Camera based on Ghost Imaging via Sparsity Constraints. Sci. Rep (2016).

13. 1-KM Remote Sensing Outfield Experiment
Mooring Platform Remote Sensing
CR=30%
337*337*51
Ground Target

14. Conclusion
Snapshot Spectral GISC Camera
 System schematic & mathematical model
 Imaging system, spatial random phase modulator, microscope objective
 Linear equations model vs. calibration
 Convolution model vs. Interpolated calibration
 Image reconstruction frameworks & algorithm
 TV_RANK, IRTV_RANK, PICHCS
 GDP, GP_BB
 Experimental Results
 Numerical results
 Indoor & outdoor experimental results
 1-Km remote sensing outfield experimental results
 Other GISC Applications