1. The document discusses deep learning applications for 3D data including classification, segmentation, shape deformation, and remeshing. 2. PointNet and PointNet++ are introduced for 3D point cloud classification and segmentation using deep neural networks. 3. Non-Euclidean methods for processing manifolds include spectral and spatial convolutional neural networks on graphs and manifolds. 4. The RoButcher project aims to use 3D data and deep learning for autonomous robotic cutting of meat in abattoirs.

2. Agenda
1. Introduction to 3D data
a) Data collection
b) Data representation
2. Deep Learning applications based on 3D data
a) Classification problem
b) Segmentation problem
3. Remeshing
4. Non-Euclidean methods for manifold processing
a) Spectral
b) Spatial
5. Libraries and datasets
6. RoButcher project
7. Q&A
2

3. Data collection
3
https://orlitany.github.io

4. Data representation – 1
R
G
B

5. v1
v2
v3
f1 f2
Base components:
1. Vertices – v1 = [x1 y1 z1]
2. Edges
3. Faces – f1 = [v1 v2 v3 ]
5
Data representation – 2
class TriangularMesh:
Vertices -> float array
Triangles -> int array
class PointCloud:
Vertices -> float array

6. 6
https://arxiv.org/pdf/1905.03304.pdf
https://libigl.github.io/tutorial
Applications
Geometry
processing
3D
data
Alignment

7. 7
PointNet
[Charles R. Qi ] PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
Properties
1. Permutation invariance
2. Rigid transformation invariance
3. No local context

8. 8
PointNet++
PointNet++, Qi et al., NeurIPS, 2018
Properties
1. Permutation invariance
2. Rigid transformation invariance
3. Hierarchical features learning
3. Acces to the local context

9. 9
PointNet & PointNet++ Results
PointNet++, Qi et al., NeurIPS, 2018

10. 10
PointNet & PointNet++ Results

11. 11
Shape deformation
Rigid transformations:
• Translation
• Rotation
• Scaling
Not Rigid transformations:
• Internal structure has changed
• Rotation of some parts

12. 12
Shape deformation
[Boscani] Learning shape correspondence with anisotropic convolutional neural networks
Extrinsic Intrinsic

13. Point cloud
13
Remeshing
[Edelsbrunner1983]; [Bernardini1999]
https://pytorch3d.readthedocs.io/en/latest/
Alpha shapes Ball Pivoting;
Poisson
Triangulation

14. 14
Non-Euclidean CNN
Spectral domain Spacial domain
Embedding domain

15. 15
Laplac-Bertrami operator
MacNeal 1949; Duffin 1959; Pinkall, Polthier 1993

16. 16
Laplac-Bertrami operator
∆𝚽 = 𝚽𝚲
• Λ – diag(𝜆1, … , 𝜆 𝑛) diagonal matrix of non-negative
eigenvalues
• Φ – (𝜙1, … , 𝜙 𝑛) orthonormal matrix of eigenvectors
a3
http://geometry.cs.ucl.ac.uk/SGP2017/?p=gradschool

17. 17
Spectral convolution
d’Alembert 1754; Borel 1899

18. 18
Spectral convolution
http://geometry.cs.ucl.ac.uk/SGP2017/?p=gradschool

19. 19
Eigen decomposition
http://www.lix.polytechnique.fr/~maks/fmaps_course/notes/siggraph_notes_course_fmaps.pdf

20. 20
Synchronized Fmaps
http://geometry.cs.ucl.ac.uk/SGP2017/

21. 21
Correspondences result
Functionalmaps:a flexiblerepresentationofmapsbetweenshapes, O.,Ben-Chen,Solomon,Butscher,Guibas,SIGGRAPH 2012

22. 22
Non-Euclidean convolutions
Embedding domain
Spacial domainSpectral domain
Geodesic convolutional neural networks on Riemannian manifolds

23. 23
Spatial convolutions

24. 24
Spacial patches
j
• Local system of coordinates Uij around vertex i
(geodesic polar)
• Spatial convolution with filter g
𝑥𝑖
′
∝
𝜃=1
Θ
𝑔 𝜃
𝑗∈𝑁(𝑥)
𝑤 𝜃 𝒖𝑖,𝑗 𝑥𝑗
Geometric deep learning on graphs and manifolds using mixture model CNNs

25. 25
Spatial convolutions
26
Spatial convolutions
Geodesic convolutional neural networks on Riemannian manifolds

26. 26
Spatial CNN
http://geometry.cs.ucl.ac.uk/SGP2017/

27. 27
Spatial convolutions
Geometric deep learning on graphs and manifolds using mixture model CNNs

28. 28
Spatial convolutions
Shape correspondence quality obtained by different methods on the FAUST humans dataset
Geometric deep learning on graphs and manifolds using mixture model CNNs

29. ModelNet (2015)
(ModelNet10: 4899
models, 10 categories)
ModelNet40: 12311
models, 40 categories
PartNet (2019)
A Large-scale
Benchmark for Fine-
grained and
Hierrarchical Part-level
3D Object
Understanding
ABC (2018)
A Big CAD
Model
Dataset For
Geometric
Deep
Learning
MPI FAUST
300 human meshes
10 objects un 30
poses
~7k vertices
29
Datasets

30. 1. Open3d
2. Pytorch3d
3. Nvidia-Kaolin
4. Pytorch-Points
5. Pytorch-Geometric
Python C++
1. LibIgl (has python bindings)
2. CGAL
3. Point cloud library (PCL)
4. Geometry Central
5. Open3d
30
Libraries
 Easy to install
 Contains visualizer tools
 Compute some descriptors, normals,
reconstructions
 Algorithms
o Pytorch integration easier
o Need to compile
 Contains visualizer tools
 Compute descriptors, normals,
reconstructions
 Algorithms
 Speed

32. 32
A Robust, Flexible and Scalable Cognitive Robotics
Platform
RoBUTCHER – a European funded Horizon 2020
innovation project that aims to replace the
conventional line production in abattoirs, with parallel
production in autonomous «meat factory cells».
In the RoBUTCHER concept the robotic system will
be able to understand and plan cutting trajectories
based on the carcass that is presented. To achieve this
it will use a combination of detailed computed
tomography (CT) data, real-time 3D imagery and
human-expert cutting data for neural network training
toward cutting trajectory planning.
https://robutcher.eu/

33. 33
Pipeline
Data
collection
Point cloud
segmentaiton
Point cloud to
CAD morphing
Extraction
correspondences
RealSense camera

34. 34
PointNet++
Data collection and Segmentation
Data
collection
Point cloud
segmentaiton
Point cloud to
CAD morphing
Extraction
correspondences

35. 35
• Point clouds from cameras is not
manifold.
• Reconstructing the mesh is hard.
Point cloud morphing and Correspondence
Data
collection
Point cloud
segmentaiton
Point cloud to
CAD morphing
Extraction
correspondences

36. 36
Point cloud morphing and Correspondence
Data
collection
Point cloud
segmentaiton
Point cloud to
CAD morphing
Extraction
correspondences

37. Partners
37

38. Thank you for your attention!
Дякую за увагу!
Спасибо за внимание!
Дзякуй за ўвагу!
38