PhD defence public presentation, Bayesian methods for inverse problems with point clouds: applications to single-photon lidar,  ENSEEHIT, Toulouse, France

1. Bayesian methods for
inverse problems with point clouds:
applications to single-photon lidar
1School of Engineering and Physical Sciences, Heriot-Watt University, Edinburgh, UK
2INP-ENSEEIHT-IRIT-TeSA, University of Toulouse, Toulouse, France
Y. Altmann1 J.-Y. Tourneret2 S. McLaughlin1
Julián Tachella1
1 of 62

2. Introduction
2 of 62

3. Why single-photon lidar?
State-of-the-art 3D ranging technology
• Up to kilometre distance
• Centimetre precision
• Eye-safe power levels
Timing
electronics
Laser
SPAD
Collection
optics
Beamsplitter
Scanning
mirrors
Control
Computer
3 of 62

4. Why single-photon lidar?
4 of 62

5. Working principle
laser
single-photon
detector
Time-of-flight histogram
2 metres
5 of 62

6. Working principle
laser
single-photon
detector
Time-of-flight histogram
𝑧𝑡 ∼ 𝒫 𝑠𝑡 + 𝑏
𝑠𝑡 = 𝑟1ℎ 𝑡 − 𝑡1
Photons per pixel (PPP) =
Signal-to-background ratio (SBR) =
𝑡=1
𝑇
𝑧𝑡
𝑡=1
𝑇
𝑠𝑡
𝑡=1
𝑇
𝑏
6 of 62

7. Cross-correlation algorithm
Maximum likelihood estimation (MLE)
• Background levels assumed to be known ( 𝑏 = 𝑏)
𝑡1, 𝑟1 = argmax
𝑡=1
𝑇
𝑝(𝑧𝑡|𝑡1, 𝑟1, 𝑏)
Approximated by
𝑟1 = max(0,
𝑡=1
𝑇
𝑧𝑡 − 𝑏𝑇)
𝑡1 = argmax 𝑧𝑡 log( 𝑟1ℎ 𝑡 − 𝜏 + 𝑏)
𝜏
𝑡1, 𝑟1
But this does not always work…
7 of 62

8. Challenges
No target 𝑠𝑡 = 0 Highly scattering environments 𝑒−𝛼𝑡 𝑛 𝑟𝑛
exponential
attenuation
spatial
correlations
neighbouring
pixels
estimate
background
unmix signals
target
detection
problem
unknown
dimension
additional
parameters
to estimate
low signal
high background
Few detected photons 𝑠𝑡 ≪ 1
High background 𝑏 ≫ 𝑠𝑡
Multiple surfaces 𝑠𝑡 = 𝑟𝑛 ℎ(𝑡 − 𝑡 𝑛)
Broadening of the IRF ℎ 𝜂(𝑡 − 𝑡 𝑛)
8 of 62

9. Notation
Observed data
• Lidar cube 𝒁 ∈ ℤ+
𝑁 𝑟×𝑁 𝑐×𝑇
with 𝒁 𝑖,𝑗,𝑡 = 𝑧𝑖,𝑗,𝑡
Unknown parameters
• Depths 𝒕 = 𝑡1, … , 𝑡 𝑁
𝑇 ∈ 1, 𝑇 𝑁
• Intensities 𝒓 = 𝑟1, … , 𝑟 𝑁
𝑇 ∈ ℝ+
𝑁
• Background levels 𝒃 = 𝑏1,1, … , 𝑏 𝑁 𝑟 𝑁 𝑐
𝑇
∈ ℝ+
𝑁 𝑟 𝑁 𝑐
9 of 62

10. Bayesian framework
Likelihood
𝑝 𝒁 𝒕, 𝒓, 𝒃 =
𝑖,𝑗,𝑡
𝑝(𝑧𝑖,𝑗,𝑡|𝒕, 𝒓, 𝒃)
Prior model
𝑝 𝒕, 𝒓, 𝒃 = 𝑝(𝒕, 𝒓)𝑝(𝒃)
Posterior distribution
𝑝 𝒕, 𝒓, 𝒃 𝒁 =
𝑝 𝒁 𝒕, 𝒓, 𝒃 𝑝 𝒕, 𝒓, 𝒃
𝑝(𝒁)
10 of 62

11. Existing approaches
How to choose 𝑝 𝒕, 𝒓 ?
1. Depth 𝒕 and reflectivity 𝒓 images [Kirmani et al., 2014]
Advantages:
• Off-the-shelf image processing priors (e.g., TV)
Disadvantages:
• Assumptions too restrictive!
𝒕 𝒓 11 of 62

12. Existing approaches
How to choose 𝑝 𝒕, 𝒓 ?
1. Depth 𝒕 and reflectivity 𝒓 images [Kirmani et al., 2014]
2. Sparse intensity cube 𝒓 [Shin et al., 2016]
𝒛𝑖,𝑗 ∼ 𝒫(𝑯𝒓𝑖,𝑗 + 𝟏𝑏𝑖,𝑗)
where 𝒓𝑖,𝑗 = 𝑟𝑖,𝑗,1, … , 𝑟𝑖,𝑗,𝑇
𝑇
is very sparse!
Advantages:
• Sparsity-promoting regularisation (ℓ1, ℓ21, TV norms)
• Convex problem
Disadvantages:
• High complexity and memory requirements
• Does not model manifold structure 12 of 62

13. Existing approaches
How to choose 𝑝 𝒕, 𝒓 ?
1. Depth 𝒕 and reflectivity 𝒓 images [Kirmani et al., 2014]
2. Sparse intensity cube 𝒓 [Shin et al., 2016]
3. Point cloud [Hernandez-Marin, 2007]
Advantages:
• Smaller dimensionality
• Better complexity
• Capture correlations between points
Disadvantages:
• Suitable prior model?
• Unknown dimensionality
• Speed?
13 of 62

14. Contributions overview
General multi-depth reconstruction
• Bayesian formulation
• Markov chain Monte Carlo inference
• Reference state-of-the-art reconstructions
• Extensions (broadening, multispectral lidar)
Real-time algorithms
• Detection
• Optimisation-based multi-depth reconstruction
14 of 62

15. Manifold
point process
model
15 of 62

16. Bayesian model
The point cloud to recover is
𝚽 = { 𝒄 𝑛, 𝑟𝑛 | 𝑛 = 1, … , 𝑁}
where 𝒄 𝑛 = [𝑥 𝑛, 𝑦𝑛, 𝑡 𝑛] 𝑇∈ 1, 𝑁𝑟 × 1, 𝑁𝑐 × [1, 𝑇]
𝑟𝑛 ∈ ℝ+
Point cloud as a spatial point process
𝑟𝑛
𝒄 𝑛
16 of 62

17. Prior distributions
Point position
Prior knowledge:
• Correlation between points within a surface
• Sparsity in depth
𝑓1 Φ 𝑓2 Φ 𝜋 𝑐 Φ
Area interaction process
Strauss process
Poisson reference measure
Prior distribution: Area interaction process + Strauss process
Laser
beam
direction
17 of 62

18. Point intensity
Prior knowledge:
• Correlation between neighbouring points within a surface
• Positivity constraint
𝑟𝑛 = log 𝑟𝑛
𝑝 𝒓 𝜎 𝑚, 𝛽 𝑚 ∝ 𝒩(0, 𝜎 𝑚
2 𝑷−𝟏)
Prior distribution:
Gaussian Markov random field
where 𝑷 is the Laplacian operator w.r.t. the manifold
𝜎 𝑚, 𝛽 𝑚 are hyperparameters
𝑟1
𝑟2 𝑟3 𝑟4
𝑟5 𝑟6
𝑟7
𝑟8
Laser beam
direction
Prior distributions
18 of 62

19. Background levels
Prior knowledge:
• 2D image
• Correlation between neighbouring pixels
• Positivity constraint
𝑝 𝒃 𝛼 𝐵 ∝
𝑖,𝑗
𝑏𝑖,𝑗
𝛼 𝐵−1
𝑏𝑖,𝑗
𝛼 𝐵
Prior distribution:
Gamma Markov random field
[Dikmen and Cemgil, 2010]
where 𝑏𝑖,𝑗 is a low-pass version of 𝑏𝑖,𝑗
and 𝛼 𝐵 is a hyperparameter
background illumination
target
Prior distributions
19 of 62

20. Bayesian framework
Posterior given the data 𝒁:
𝑝 𝚽, 𝒃 𝒁 ∝ 𝑝 𝒁 𝚽, 𝒃 𝑝 𝚽 𝑝(𝒃)
We want the maximum-a-posteriori estimate
argmax 𝑝 𝚽, 𝒃 𝒁
No analytical expressions …
We gather samples 𝚽(s) for 𝑠 = 1, … , 𝑁MC
𝚽 = argmax 𝑝 𝚽 s , 𝒃(𝑠) 𝒁
𝚽. 𝒃
𝚽(𝑠)
20 of 62

21. Sampling strategy
How do we obtain the samples?
• Reversible-jump Markov chain Monte Carlo
o Propose random moves
o Accept or reject according to change in 𝑝 𝚽 s , 𝒃(𝑠) 𝒁
Standard moves
• Birth and death
• Shift
• Mark update
• Split and merge
21 of 62

22. Sampling the model
How do we obtain the samples?
• Reversible-jump Markov chain Monte Carlo
Problem:
Classical birth/death moves get rarely accepted
[Hernandez-Marin et al. 2007]
• New moves:
• Dilation and erosion
• Multiresolution approach
• Better scaling with cube size
Coarse scale Fine scale
22 of 62

23. Experiments
Data size: 100 x 100 x 4700
Stand-off distance: 4 m
PPP ≈ 45 photons
SBR ≈ 10
Data from [Shin et al., 2016]
Competing methods:
ℓ1 [Shin et al. 2016]
ℓ21 + TV [Halimi et al. 2017]
23 of 62

24. ℓ1
Exec. time: 2871 s
Multi-depth scene
ℓ21 + TV
Exec. time: 202 s
ManiPoP
Exec. time: 146 s
Tachella et al., SIAM Journal in Imaging Sciences (2019) 24 of 62

25. ManiPoP
extensions
25 of 62

26. Broadening of the IRF
Prior distribution: Gaussian Markov random field
ℎ 𝜂 𝑛
(𝑡 − 𝑡 𝑛) = ℎ 𝑡 − 𝑡 𝑛 ∗ exp(−
𝑡2
2 𝜂 𝑛−1 2)
𝚽 = { 𝒄 𝑛, 𝑟𝑛, 𝜂 𝑛 | 𝑛 = 1, … , 𝑁}
with 𝜂 𝑛 ∈ (1, ∞) indicates broadening
𝑟𝑛
𝒄 𝑛
𝜂 𝑛
26 of 62

27. Experiments
Data size: 123 x 96 x 800
Stand-off distance: 3 km
PPP ≈ 913 photons
SBR ≈ 1.64
27 of 62

28. Experiments
Intensity 𝑟𝑛 Broadening 𝜂 𝑛
Execution time: 195 s
Tachella et al., ICASSP (2019) 28 of 62

29. Multispectral lidar
Measure 𝐿 wavelengths per pixel 𝒁 ∈ ℤ+
𝑁 𝑟×𝑁 𝑐×𝐿×𝑇
Spectral diversity via
• Multiple laser sources
• Spectral filters before detectors
29 of 62

30. Multispectral lidar
laser(s)
single-photon
detector(s)
Time-of-flight histograms
30 of 62
Motivations:
• Spectral 3D
• Material classification
• Robust depth estimation

31. MuSaPoP algorithm
𝑟𝑛,3
𝑟𝑛,2
𝑟𝑛,1
𝒄 𝑛
ℓ
The intensity marks are a vector of 𝐿 values
𝚽 = { 𝒄 𝑛, 𝒓 𝑛 | 𝑛 = 1, … , 𝑁}
with 𝒓 𝑛 = 𝑟1, … , 𝑟𝐿
𝑇 ∈ ℝ+
𝑁
Prior distributions
• Gaussian Markov random fields
• Independently per wavelength
31 of 62

32. Subsampling strategies
A typical MSL with 𝐿 = 32 has 𝟏𝟎 𝟗 data voxels!
• Prohibitive memory requirements
• Very long acquisition time
Subsampling
• 𝑊 < 𝐿 wavelengths per pixel
• Incorporate in the observation model
32 of 62

33. Subsampling strategies
Subsampling Collaboration with H. Arguello and M. Marquez
• Completely random
• Blue noise patterns
+ + =
+ + =
RGB 𝐿 = 3
𝑊 = 1
unwanted cluster
33 of 62

34. Experiments
Data size: 198 x 198 x 32 x 4500
PPP ≈ 33 to 0.3 photons
SBR ≈ 25
Data from [Altmann et al., 2017]
34 of 62

35. Experiments
35 of 62Tachella et al., IEEE Transaction on Computational Imaging (2019)

36. Experiments
Depth TV [Altmann et al., 2017]
Exec. time: 1062 min
MuSaPoP
Exec. time: 40 min
Tachella et al., IEEE Transaction on Computational Imaging (2019) 36 of 62

37. Partial summary
ManiPoP algorithm
• State-of-the-art reconstructions for the multi-depth case
• Easily generalisable
• Peak broadening
• Multispectral lidar
But …
… too slow for real-time applications!
• Other existing methods also too slow
• Even a recent CNN takes minutes [Lindell et al., 2019]
37 of 62

38. Towards real-time analysis
Fast target detection [Tachella et al., EUSIPCO (2019)]
• Discard histograms without objects
• Complexity similar to standard cross-correlation
• Highly parallelisable
• Small overhead spatial correlation
But …
… what about full real-time reconstruction?
I want it all,
and I want it now 38 of 62

39. Plug-and-play
point cloud
denoisers
39 of 62

40. Computer graphics
Computer graphics algorithms
• Model correlations of 3D point clouds very well
• Handle very large point clouds in real-time
How can we profit from these methods?
40 of 62

41. Reconstruction algorithm
Reparametrization
• Depths: 𝒕 = 𝑡1, … , 𝑡 𝑁
𝑇
• Log-intensities: ( 𝑟𝑛 = log 𝑟𝑛 ∈ ℝ), 𝒓 = 𝑟1, … , 𝑟 𝑁
𝑇
• Log-background levels: ( 𝑏𝑖,𝑗 = log 𝑏𝑖,𝑗 ∈ ℝ), 𝒃 = 𝑏1,1, … , 𝑏 𝑁 𝑟 𝑁 𝑐
𝑇
Negative log-likelihood
𝑔 𝒕, 𝒓, 𝒃 ∝ −
𝑖,𝑗,𝑡
log 𝑝(𝑧𝑖,𝑗,𝑡|𝒕, 𝒓, 𝒃)
Penalised likelihood
argmin 𝑔 𝒕, 𝒓, 𝒃 + 𝜌1 𝒕 + 𝜌2 𝒓 + 𝜌3 𝒃
𝒕, 𝒓, 𝒃
41 of 62

42. Optimisation algorithm
𝒕∗ = 𝒕 𝑘 − 𝜇 𝑡∇𝐭 𝑔 𝒕 𝑘, 𝒓 𝑘, 𝒃 𝑘
𝒕 𝑘+1= argmin𝐭 2𝜇 𝑡 𝜌1 𝒕 + 𝒕 − 𝒕∗
𝟐
𝟐
𝒓∗ = 𝒓 𝑘 − 𝜇 𝑟∇ 𝒓 𝑔 𝒕 𝑘+1, 𝒓 𝑘, 𝒃 𝑘
𝒓 𝑘+1 = argmin 𝒓 2𝜇 𝑟 𝜌2 𝒓 + 𝒓 − 𝒓∗
𝟐
𝟐
𝒃∗ = 𝒃 𝑘 − 𝜇 𝑏∇ 𝒃 𝑔 𝒕 𝑘+1, 𝒓 𝑘+1, 𝒃 𝑘
𝒃 𝑘+1 = argmin 𝒃 2𝜇 𝑏 𝜌3 𝒃 + 𝒃 − 𝒃∗
𝟐
𝟐
PALM: block proximal gradient steps [Bolte et al., 2013]
42 of 62

43. Optimisation algorithm
PALM: block proximal gradient steps [Bolte et al., 2013]
Gradient step
• Data consistency step
• Lipschitz-differentiable 𝑔
𝒕∗ = 𝒕 𝑘 − 𝜇 𝑡∇𝐭 𝑔 𝒕 𝑘, 𝒓 𝑘, 𝒃 𝑘
43 of 62

44. Optimisation algorithm
𝒕 𝑘+1= argmin𝐭 2𝜇 𝑡 𝜌1 𝒕 + 𝒕 − 𝒕∗
𝟐
𝟐
PALM: block proximal gradient steps [Bolte et al., 2013]
Proximal operator
• Incorporates prior information
• Just a denoising of 𝒕∗! [Venkatakrishnan et al., 2013]
• Apply off-the-shelf denoiser
44 of 62
point cloud denoising 𝒕∗

45. Optimisation algorithm
𝒕∗ = 𝒕 𝑘 − 𝜇 𝑡∇𝐭 𝑔 𝒕 𝑘, 𝒓 𝑘, 𝒃 𝑘
𝒕 𝑘+1= argmin𝐭 2𝜇 𝑡 𝜌1 𝒕 + 𝒕 − 𝒕∗
𝟐
𝟐
𝒓∗ = 𝒓 𝑘 − 𝜇 𝑟∇ 𝒓 𝑔 𝒕 𝑘+1, 𝒓 𝑘, 𝒃 𝑘
𝒓 𝑘+1 = argmin 𝒓 2𝜇 𝑟 𝜌2 𝒓 + 𝒓 − 𝒓∗
𝟐
𝟐
𝒃∗ = 𝒃 𝑘 − 𝜇 𝑏∇ 𝒃 𝑔 𝒕 𝑘+1, 𝒓 𝑘+1, 𝒃 𝑘
𝒃 𝑘+1 = argmin 𝒃 2𝜇 𝑏 𝜌3 𝒃 + 𝒃 − 𝒃∗
𝟐
𝟐
point cloud denoising 𝒕∗
intensity denoising 𝒓∗
image denoising 𝒃∗
PALM: block proximal gradient steps [Bolte et al., 2013]
45 of 62

46. Algebraic point set surfaces
Projects 3D points onto smooth surfaces
[Guennebaud and Gross 2007]
For each point
1. Fit sphere using neighbours
2. Solve least squares problem
3. Project point into sphere
• Can handle multiple surfaces per pixel
• Easily parallelisable in GPU!
Collaboration with N. Mellado
46 of 62

47. Denoisers
Intensity denoising
• Low-pass filtering using manifold structure
𝑟𝑛 = 1 − 𝛽 𝑟𝑛 + 𝛽
𝑛′
𝑟 𝑛′
• Parallel implementation in GPU
Background denoising
• Wiener filtering
• Fast via FFT
47 of 62

48. RT3D algorithm
𝑇: number of bins
PPP: mean photons per pixel
(always PPP ≤ 𝑇)
Complexity:
• Parallel gradient 𝒪 PPP
• Parallel denoising ≈ 𝒪 1
Memory requirements
• Data 𝒪 𝑁𝑟 𝑁𝑐 PPP
• Parameters 𝒪 𝑁𝑟 𝑁𝑐
PPP
48 of 62

49. Experiments
Data size: 141 x 141 x 4500
Stand-off distance: 40 m
PPP ≈ 3 photons
SBR ≈ 5
Competing methods:
• Cross-correlation
• Rapp and Goyal (single-depth)
• ManiPoP (multi-depth)

50. Experiments
Cross-corr Rapp and GoyalManiPoP RT3D
1 ms
(parallel)
180 s 30 s 10 ms
Execution time
50 of 62

51. Real-time 3D imaging
Data size: 32 x 32 x 153
Stand-off distance: 320 m
Collaboration with R. Tobin,
A. McCarthy and G. S. Buller
PPP ≈ 900 photons
SBR ≈ 1
Super-resolution to 96 x 96 pixels
Execution time: 50 frames per second
51 of 62

52. Real-time 3D imaging
Tachella et al., Nature Communications (2019) 52 of 62

53. Multispectral RT3D
Extension to MSL data
• 𝑊 = 1 out of 𝐿 wavelengths per pixel
• Same amount of data
Intensity denoising
• Bilateral filter with colour information [Tomas and Manduchi, 1998]
• Preserves edges + fast parallel implementation
Background denoising
• Wiener filtering
• Independently per wavelength
53 of 62

54. Experiments
Data size: 200 x 200 x 1029
𝐿 = 4 wavelengths (RGBY)
𝑊 = 1 wavelength per pixel
Competing algorithms
• MuSaPoP
• Depth TV [Altmann et al., 2017]
• Single-wavelength, same acq. time
54 of 62

55. Experiments
Ground truth CRT3D
65 ms 42 ms
Single-wavelength
(blue)
Surfaces per pixel = 1
PPP ≈ 10 photons
SBR ≈ 2
Execution time:
55 of 62Tachella et al., CAMSAP (2019)

56. Experiments
Ground truth CRT3D MuSaPoP Depth TV
30 min 1 h35 ms
Surfaces per pixel ≤ 1
PPP ≈ 2 photons
SBR ≈ 22
Execution time:
Tachella et al., CAMSAP (2019) 56 of 62

57. Conclusions
57 of 62

58. Conclusions
Point cloud models
• Spatial point processes
• Plug-and-play point cloud denoisers
• Efficient methods via low-dimensional models
• Correlations between points within a surface
• MCMC and optimisation-based inference
58 of 62

59. Extensions
ManiPoP
• Broadening
• Underwater
• Multispectral lidar
Plug-and-play
• Real-time multispectral lidar
• Other point cloud denoisers?
Inverse problems
𝒁|𝚽, 𝜽 ∼ ℱ(𝚽, 𝜽)
• 𝒁 is the data
• 𝚽 is the point cloud
• 𝜽 fixed dimensional parameters
59 of 62

60. Image a scene hidden from view
with a single-photon lidar
𝒁|𝚽, 𝜽 ∼ ℱ(𝚽, 𝜽)
• 𝒁 is the lidar data
• 𝚽 is a set of hidden facets
• 𝜽 ceiling parameters
Collaboration with V. Goyal
J. Rapp, C. Saunders and
J. Murray-Bruce
Non-line-of-sight imaging
60 of 62

61. Hidden room Reconstruction
Experiments
61 of 62

62. Future work
Extension to other inverse problems
• Lidar with atmospheric turbulence
• Sonar
• Lensless depth cameras
Real-time imaging in high-flux conditions
• Large and dense histograms (PPP = 𝑇 ≫ 1)
• Compressive learning
62 of 62

63. Contact: julian.tachella@ed.ac.uk
Online codes, presentations and more: tachella.github.io
Thanks for your attention!
Collaborators: G. S. Buller, V. K. Goyal, H. Arguello, N. Mellado,
A. McCarthy, M. Pereyra, R. Tobin, M. Márquez, A. Maccarone,
D. Aguirre, J. Rapp, J. Murray-Bruce, C. Saunders