Slide for study session given by Dr. Daisuke Sato at Arithmer inc. It is a summary of methods for semantic segmentation for 3D pointcloud using 2D weakly-supervised learning. Arithmer株式会社は東京大学大学院数理科学研究科発の数学の会社です。私達は現代数学を応用して、様々な分野のソリューションに、新しい高度AIシステムを導入しています。AIをいかに上手に使って仕事を効率化するか、そして人々の役に立つ結果を生み出すのか、それを考えるのが私たちの仕事です。 Arithmer began at the University of Tokyo Graduate School of Mathematical Sciences. Today, our research of modern mathematics and AI systems has the capability of providing solutions when dealing with tough complex issues. At Arithmer we believe it is our job to realize the functions of AI through improving work efficiency and producing more useful results for society.

1. 2021/7/18
Weakly Supervised Semantic Segmentation in 3D
Graph-Structured Point Clouds of Wild Scenes
Arithmer R3 team
Daisuke SATO
https://arxiv.org/abs/2004.12498v2
1

2. Introduction
Point clouds
● 3D sensors have been developed rapidly
these days.
○ iPhone/iPad has LiDAR
○ realsense/azure kinect is amazing
considering its low price.
● Raw data collected is point cloud
○ Collection of 3D points
○ (x0
, y0
, z0
)
(x1
, y1
, z1
)
…
(xn
, yn
, zn
)
● Processing point cloud is quite
important in robotics/measurement (測
量).
2

3. Introduction
Task: semantic segmentation of point clouds
● Classifying every point of 3D point clouds
○ If we perform this task with supervised
learning, we need to annotate each
point.
○ Labeling point clouds is super
exhausting.
○ Labeling 2D image is easier.
○ Let us do weakly supervised learning
with 2D projected image.
3

4. Introduction
4
Idea: weakly supervised learning with 2D projected image
Input is
3D point cloud
Output at
inference stage
is 3D segmented
point cloud
Label at training
stage is
segmented 2D
projected images

5. Introduction
Note: Projection of 3D pointcloud onto 2D image
● Projection is described by the two
parameters:
○ Internal camera parameters
■ Focal length (f)
■ principal point (c)
○ External camera parameters
■ Translation (t)
■ Rotation (R)
5

6. Method
Model architecture
6
Total loss
Visibility branch
(feature vector -> mask for visibility)
Input is 3D point cloud
Graph convolution encoder
(point cloud -> feature vector)
2D projection
(segmented pcd/mask for visibility
-> 2D segmented image)
Segmentation branch
(feature vector
-> mask for segmentation)

7. Method
Perspective rendering
● Problem: Multiple pointclouds can be
projected onto a single pixel. Which
class should we give the pixel?
7
● Solution: “Semantic fusion”
(sophiscated voting system)

8. Experiments
Datasets
● SUNCG Synthetic dataset
○ class number: 40 (furniture)
○ rooms: 404058
■ create 55000 2D rendering sets
● S3DIS Real-world dataset
○ class number: 13 (furniture)
○ rooms: 272
■ thousands of viewpoints are
provided
8
● Each point has
○ position: (x, y, z)
○ color: (r, g, b)
● Also normal is computed (u, v, w)

9. Experiments
Metrics
● mean accuracy of total classes (mAcc)
● overall accuracy (oAcc)
● mean per-class intersection-over-union
(mIoU)
○ IOU = TP / (TP + FP + FN)
9

10. Experiments
Comparison with other fully-supervised methods
Not so bad even compared with fully-supervised learning.
10

11. Experiments
Inference samples for SUNCG dataset
11

12. Experiments
Inference samples for S3DIS dataset
12

13. Conclusion
● To the best of our knowledge, this is the first work to apply 2D supervision for 3D
semantic point cloud segmentation of wild scenes without using any 3D pointwise
annotations.
● Extensive experiments are conducted and the proposed method achieves comparable
performance with the state-of-the-art 3D supervised methods on the popular SUNCG and
S3DIS benchmarks.
13

14. 14

15. BACK UP
15

16. Method
TITLE HERE
● Bullet list
16

17. Experiments
Ablation study: projection method/OBSNet decoder
● Bullet list
17

18. Experiments
Ablation study - Encoder Design
● Bullet list
18

19. Experiments
Ablation study - Amount of training data
● Bullet list
19

20. Experiments
Ablation study - Visibility detection by OBSNet
● Bullet list
20

21. Experiments
Transfer learning from synthesic to realistic dataset
● Bullet list
21