Presentation at CAMSAP 2019, Guadaloupe, West Indies

1. Real-time 3D color imaging
with single-photon lidar data
J. Tachella1, Y. Altmann1, J.-Y. Tourneret2 and S. McLaughlin1
1School of Engineering and Physical Sciences, Heriot-Watt University, Edinburgh, UK
2INP-ENSEEHIT-IRIT-TeSA, University of Toulouse, Toulouse, France

2. Single-photon lidar
Basic working principle
laser
single-photon
detector
Time-of-flight histogram
𝑧 𝑛,𝑡 ∼ 𝑃𝑜𝑖𝑠𝑠𝑜𝑛 𝑟𝑘ℎ 𝑡 − 𝑡 𝑘 + 𝑏 𝑛
1/17Goal: estimate {𝑟𝑘, 𝑡 𝑘} 𝑘=1,2,…., 𝑏 𝑛

3. • Measure 𝐿 wavelengths per pixel
Spectral diversity via
• Multiple laser sources
• Spectral filters before detector
Multispectral single-photon lidar
photons
e.g. color imaging (RGB)
2/17

4. Low light-flux model
𝑧 𝑛,ℓ,𝑡 ∼ 𝑃𝑜𝑖𝑠𝑠𝑜𝑛 𝑔 𝑛,ℓ 𝑟𝑘,ℓℎℓ(𝑡 − 𝑡 𝑘) + 𝑏 𝑛,ℓ
𝑔 𝑛,ℓ ∈ 0,1 indicates if histogram is measured
𝑛 = 1, … , 𝑁𝑟 𝑁𝑐 pixels
ℓ = 1, … , 𝐿 wavelengths
𝑡 = 1, … , 𝑇 histogram bins
3/17
Observation model

5. Spectral subsampling
Very large datasets
• Data size = 𝑁𝑟 𝑁𝑐 𝐿 𝑇 > 109 !!!
• Slow acquisition + processing
• Large memory requirements
Solution [Tachella et al., 2019]
• Choose 𝑊 out of 𝐿 wavelengths per pixel: ℓ 𝑔 𝑛,ℓ = 𝑊 < 𝐿
• Uniform measurements across the spatial and spectral dimensions
• e.g. 𝑊 = 1 and 𝐿 = 4 (RGB+)
• same amount of data as the single wavelength case, but color
reconstructions!
Measured pixels
at a given wavelength
4/17

6. Reconstruction algorithm
Parametrization
• Depths: 𝒕 = 𝑡1, … , 𝑡 𝐾
𝑇
• Log-intensities: ( 𝑟𝑘,ℓ = log 𝑟𝑘,ℓ ∈ ℝ), 𝒓 = 𝑟1,1, … , 𝑟 𝐾,𝐿
𝑇
• Log-background levels: ( 𝑏 𝑛,ℓ = log 𝑏 𝑛,ℓ ∈ ℝ), 𝒃 = 𝑏1,1, … , 𝑏 𝑁 𝑟 𝑁 𝑐,𝐿
𝑇
Negative log-likelihood
𝑔 𝒕, 𝒓, 𝒃 ∝ −
𝑛,ℓ,𝑡
log 𝑝(𝑧 𝑛,ℓ,𝑡|𝒕, 𝒓, 𝒃)
Penalized likelihood
argmin 𝒕, 𝒓, 𝒃 𝑔 𝒕, 𝒓, 𝒃 + 𝜌1 𝒕 + 𝜌2 𝒓 + 𝜌3 𝒃
5/17

7. Reconstruction algorithm
6/17
How to choose 𝜌1 𝒕 and 𝜌2(𝒓)?
1. Depth and reflectivity images [Rapp and Goyal, 2017]
Advantages:
• Off-the-shelf image processing regularizers (TV, DCT, etc.)
Disadvantages:
• Assumption too restrictive!
𝒕 𝒓

8. Reconstruction algorithm
7/17
How to choose 𝜌1 𝒕 and 𝜌2(𝒓)?
1. Depth and reflectivity images [Rapp and Goyal, 2017]
2. Sparse intensity cube [Shin et al., 2016] [Halimi et al., 2017]
Advantages:
• Off-the-shelf sparse volume regularizers (ℓ1, ℓ21 norms)
Disadvantages:
• Prohibitive complexity
• Does not model manifold structure

9. Reconstruction algorithm
8/17
How to choose 𝜌1 𝒕 and 𝜌2(𝒓)?
1. Depth and reflectivity images [Rapp and Goyal, 2017]
2. Sparse intensity cube [Shin et al., 2016]
3. Point cloud [Tachella et al. 2019a]
Advantages:
• Off-the-shelf computer graphics modelling
• Better complexity
Disadvantages:
• Valid penalty functions?

10. Optimization algorithm
𝒕∗ = 𝒕 𝑘 − 𝜇 𝑡∇𝐭 𝑔 𝒕 𝑘, 𝒓 𝑘, 𝒃 𝑘
𝒕 𝑘+1= argmin𝐭 𝜌1 𝒕 + 𝒕 − 𝒕∗
𝟐
𝟐
𝒓∗ = 𝒓 𝑘 − 𝜇 𝑟∇ 𝒓 𝑔 𝒕 𝑘+1, 𝒓 𝑘, 𝒃 𝑘
𝒓 𝑘+1 = argmin 𝒓 𝜌2 𝒓 + 𝒓 − 𝒓∗
𝟐
𝟐
𝒃∗ = 𝒃 𝑘 − 𝜇 𝑏∇ 𝒃 𝑔 𝒕 𝑘+1, 𝒓 𝑘+1, 𝒃 𝑘
𝒃 𝑘+1 = argmin 𝒃 𝜌3 𝒃 + 𝒃 − 𝒃∗
𝟐
𝟐
point cloud denoising 𝒕∗
intensity denoising 𝒓∗
image denoising 𝒃∗
9/17
PALM: block proximal gradient steps [Bolte et al., 2013]

11. Algebraic point set surfaces (APSS)
Projects 3D points onto smooth surfaces, by fitting spheres
locally [Guennebaud and Gross 2007]
Sphere defined by
𝜙 𝒖 𝒄 𝑛 = 𝒖 𝑇
1, 𝒄 𝑛, 𝒄 𝑛
𝑇
𝒄 𝑛
𝑇
= 0
argmin 𝒖
𝑛
𝑤 𝑛 𝜙 𝒖 𝒄 𝑛
2
The weights are given by the distance to the centroid
Once 𝒖 is computed, we just project the points
10/17

12. Intensity & background
Intensity denoising
• Bilateral filter with color information [Tomas and Manduchi, 1998]
• Preserves edges + fast parallel implementation
Background denoising
• Wiener filtering
• Independently per wavelength
11/17

13. CRT3D algorithm
𝑻 number of bins
𝝀 number of photons per pixel
(always 𝜆 ≤ 𝑇)
Complexity:
• Parallel gradient 𝒪 𝑊𝜆
• Parallel denoising ≈ 𝒪 1
Memory requirements
• Data 𝒪 𝑊𝑁𝑟 𝑁𝑐 𝜆
• Parameters 𝒪 𝐿𝑁𝑟 𝑁𝑐
12/17
Photons per pixel

14. Experiments
Lego Dataset:
𝑁𝑟 = 𝑁𝑐 = 200 pixels
𝑇 = 1029 bins
𝐿 = 4 wavelengths (RGBY)
𝑊 = 1 wavelength per pixel
Competing algorithms
• MuSaPoP [Tachella et al., 2019b]
• Depth TV [Altmann et al., 2017]
• Single-wavelength, same acq. time
13/17

15. Experiments
Ground truth Proposed MuSaPoP Depth TV
65 ms 1 h 2.7 h
Surfaces per pixel = 1
Signal-to-background ratio = 2
Mean photons per pixel (𝜆) = 10
14/17
Execution time:

16. Experiments
Ground truth Proposed MuSaPoP Depth TV
30 min 1 h35 ms
Surfaces per pixel ≤ 1
Signal-to-background ratio = 22
Mean photons per pixel (𝜆) = 2
15/17
Execution time:

17. Experiments
Ground truth Proposed Single-wavelength
(green)
65 ms 44 ms 42 ms
Single-wavelength
(blue)
Surfaces per pixel = 1
Signal-to-background ratio = 2
Mean photons per pixel (𝜆) = 10
16/17
Execution time:

18. Conclusions and future work
First real-time color algorithm for single-photon lidar data
• Complexity 𝑂(𝑊𝜆)
• Subsampling: same 𝜆 as single-wavelength
• Plug-and-play point cloud denoiser framework
Future/ongoing work
• Reducing the complexity when 𝜆 ≫ 1
17/17

19. Contact: jat3@hw.ac.uk
Online codes, presentation and more: tachella.github.io
Thanks for your attention!