labelCloud – A Lightweight Domain-Independent Labeling Tool for 3D Object Detection in Point Clouds

Christoph Sager¹, Patrick Zschech², Niklas Kühl³ // 07.07.2021
CAD'21 - 18th annual International CAD Conference, Barcelona
labelCloud – A Lightweight Domain-Independent Labeling
Tool for 3D Object Detection in Point Clouds
¹ Technische Universität Dresden, ² Friedrich-Alexander-Universität Erlangen-Nürnberg, ³ Karlsruher Institut für Technologie

labelCloud – A Lightweight Domain-Independent Labeling Tool for 3D Object Detection in Point Clouds
CAD'21 - the 18th annual International CAD Conference, Barcelona // C. Sager, P. Zschech, N. Kühl
Paper & Software Presentation // 07.07.2021
Slide 2
Content – labelCloud
A Lightweight Domain-Independent Labeling Tool for 3D Object Detection in Point Clouds
1. Foundations
Point Clouds and Machine Learning
2. Related Work & Motivation
Existing Point Cloud Labeling Tools
3. Live Demonstration
Presentation of labelCloud’s Core Features
4. Evaluation & Discussion
Results of the User Study
5. Conclusion & Outlook
Applications & Further Development

Slide 3
Foundations & Motivation
Point Clouds & Machine Learning

Slide 4
Foundations – The need for Point Clouds in Object Detection
● Automatic object detection is increasingly used in safety-critical contexts
→ like autonomous driving, robotics and medicine
● Yet 2D-based approaches fail in estimating the size and distance of objects with high precision
● 3D-point clouds capture environments in real-world metrics
100px
300px 4 m
1.6 m
2 m
2D vs. 3D

Slide 5
Foundations – Point Clouds and Machine Learning
● Point clouds are recorded with depth sensors based on lidar or stereo vision
● New architectures of artiﬁcial neural network can handle this unstructured data
○ estimate the position, rotation and dimension of objects
(Friederich et al., 2020, Qi et al., 2019, 2020)
● VoteNet
● ImVoteNet
● 3D-CNN
● ...

Slide 6
Foundations – 3D Bounding Boxes as Result of 3D Object Detection
● Depth-based 3D object detection methods can predict distances, dimensions and rotations of
objects in real-world metrics (i.e., meter)
● To train neural networks, large amounts of labeled training data (known outcomes) are
necessary, which requires labeling tools specialized for point clouds
(Friederich et al., 2020; Geiger et al., 2013)
pitch

Slide 7
Related Work & Motivation
Existing Tools Point Cloud Labeling Tools

Slide 8
Related Work – Existing Point Cloud Labeling Tools
● Analysis of 3+ existing point cloud labeling tools
● All focus on the creation of training data for
autonomous driving applications
● This lead to …
→ few supported point cloud ﬁle formats
→ strong focus on lidar sensors
→ neglection the x- and y-rotation
(Arief et al., 2020; Wang et al., 2019; Zimmer et al., 2019)

Slide 9
Motivation – Design of a lightweight, domain-independent Labeling Tool
(Chandra et al., 2015; Pfeffers et al. 2007)
Major Requirements:
R1) Support intuitive and direct labeling inside the point cloud
R2) Allow manipulation of position, rotation and dimension of 3D bounding box
R3) Increase quality and precision of the drawn bounding boxes
R4) Decrease labeling time and necessary user interactions
R5) Support multiple point cloud and label file formats
R6) Allow for flexibility while keeping dependencies to minimum

Slide 10
✓ Works for 3DOD & 6DPE
✓ Full Rotation Support (x,y,z)
✓ Many Compatible Formats
✓ Fast & Easy Setup
Live Demonstration – Presentation of labelCloud’s Core Features

Slide 11
MODEL
CONTROL
VIEW
Software Architecture – Standing on the Shoulders of Pip
PyOpenGL PyQt Open3D numpy
PCD Viewer
Point Cloud Label
Math3D
PCD Manger* Label Manager* Label Mode*
Picking
Mode
Spanning
Mode
modular & extend-
able architecture

Slide 12
Evaluation & Discussion
Results of the User Study

Slide 13
Evaluation – Comparison of Label Methods Picking & Spanning
The Spanning Mode is 22% faster and requires less user interaction (-63%).

Slide 14
Discussion – labelCloud in the Context of Existing Labeling Tools
labelCloud Existing Point Cloud Labeling Tools
● Direct labeling inside the point cloud ● Indirect labeling on a 2D-projection of the point cloud
● Domain-independent and flexible ● Domain-specific and highly specialized
● Support of many depth sensors and ML-frameworks ● Focus on single sensor types and selected frameworks
● Easy to set up and few dependencies ● Sometimes difficult to set up due to many
dependencies
● So far no automatization features
→ difficult due to domain-independence
● Provide automatization features, like
○ pre-trained models
○ object tracking
○ 1-click clustering
○ projections of 2D labels

Slide 15
Conclusion & Outlook
Applications & Further Development

Slide 16
Conclusion & Outlook – Critical Discussion & Future Research
Conclusion
● Proven feasibility of direct labeling with 3D Bounding Boxes
● Provision of a software to quickly test 3D object detection in multiple domains
● Further reduction of the labeling time still possible with automatization features
Outlook
1. 3D object detection is more and more used in real applications outside of research
2. 3D sensors are getting more aﬀordable and integrated into consumer products (like the iPhone)
3. The application of 3D object detection requires eﬃcient and easy to use labeling tools

Slide 17
Interested?
… try it out:
Do you see a use case that might
beneﬁt from 3D object detection?
Do you want further information?
Let’s talk:
Christoph Sager
christophsager
christoph.sager@gmail.com
Running on Linux, Windows and Mac.
Sager, C.; Zschech, P.; Kühl, N. labelCloud: A Lightweight Domain-Independent Labeling Tool for 3D
Object Detection in Point Clouds. 2021. 18th Annual International CAD Conference (CAD'21)
Sager, C.; Zschech, P.; Kühl, N. labelCloud: A Lightweight Domain-Independent Labeling Tool for 3D
Object Detection in Point Clouds. 2021. Computer-Aided Design and Applications (forthcoming)

Slide 18
Selected Sources
● Arief, H. A., Arief, M., Zhang, G., Liu, Z., Bhat, M., Indahl, U. G., Tveite, H., & Zhao, D. (2020). SAnE: Smart Annotation and Evaluation Tools for
Point Cloud Data. IEEE Access, 8, 131848–131858. https://doi.org/10.1109/ACCESS.2020.3009914
● Chandra, L., Seidel, S., & Gregor, S. (2015). Prescriptive Knowledge in IS Research: Conceptualizing Design Principles in Terms of Materiality,
Action, and Boundary Conditions. 2015 48th Hawaii International Conference on System Sciences, 4039–4048.
https://doi.org/10.1109/HICSS.2015.485
● Friederich, J.; Zschech, P.: Review and Systematization of Solutions for 3D Object Detection. In 15th International Conference on
Wirtschaftsinformatik (WI), 1699–1711. GITO Verlag, Potsdam, Germany,2020. ISBN 978-3-95545-335-0.
http://doi.org/10.30844/wi_2020_r2-friedrich
● Geiger, A.; Lenz, P.; Stiller, C.; Urtasun, R.: Vision meets robotics: The KITTI dataset. The InternationalJournal of Robotics Research, 32(11),
1231–1237, 2013. ISSN 0278-3649, 1741-3176.http://doi.org/10.1177/0278364913491297
● Peﬀers, K., Tuunanen, T., Rothenberger, M. A., & Chatterjee, S. (2007). A Design Science Research Methodology for Information Systems
Research. Journal of Management Information Systems, 24(3), 45–77. https://doi.org/10.2753/MIS0742-1222240302
● Qi, C.R.; Chen, X.; Litany, O.; Guibas, L.J.: ImVoteNet: Boosting 3D Object Detection in Point Clouds With Image Votes. In 2020 IEEE/CVF
Conference on Computer Vision and Pattern Recognition(CVPR), 4403–4412. IEEE, Seattle, WA, USA, 2020. ISBN 9781728171685.
http://doi.org/10.1109/CVPR42600.2020.00446
● Qi, C.R.; Litany, O.; He, K.; Guibas, L.: Deep Hough Voting for 3D Object Detection in Point Clouds. In 2019 IEEE/CVF International Conference on
Computer Vision (ICCV), 9276–9285. IEEE, Seoul, Korea(South), 2019. ISBN 978-1-72814-803-8.http://doi.org/10.1109/ICCV.2019.00937
● Sager, C.; Zschech, P.; Kühl, N. labelCloud: A Lightweight Domain-Independent Labeling Tool for 3D Object Detection in Point Clouds. 2021.
18th Annual International CAD Conference (CAD'21)
● Sager, C.; Zschech, P.; Kühl, N. labelCloud: A Lightweight Domain-Independent Labeling Tool for 3D Object Detection in Point Clouds. 2021.
Computer-Aided Design and Applications (forthcoming)
● Wang, B., Wu, V., Wu, B., & Keutzer, K. (2019). LATTE: Accelerating LiDAR Point Cloud Annotation via Sensor Fusion, One-Click Annotation, and
Tracking. 2019 IEEE Intelligent Transportation Systems Conference (ITSC), 265–272. https://doi.org/10.1109/ITSC.2019.8916980
● Wirth, F., Quehl, J., Ota, J., & Stiller, C. (2019). PointAtMe: Eﬃcient 3D Point Cloud Labeling in Virtual Reality. 2019 IEEE Intelligent Vehicles
Symposium (IV), 1693–1698. https://doi.org/10.1109/IVS.2019.8814115
● Zimmer, W., Rangesh, A., & Trivedi, M. (2019). 3D BAT: A Semi-Automatic, Web-based 3D Annotation Toolbox for Full-Surround, Multi-Modal
Data Streams. 2019 IEEE Intelligent Vehicles Symposium (IV), 1816–1821. https://doi.org/10.1109/IVS.2019.8814071

labelCloud – A Lightweight Domain-Independent Labeling Tool for 3D Object Detection in Point Clouds

More Related Content

Similar to labelCloud – A Lightweight Domain-Independent Labeling Tool for 3D Object Detection in Point Clouds

Recently uploaded

labelCloud – A Lightweight Domain-Independent Labeling Tool for 3D Object Detection in Point Clouds