The rapid development of 3D sensors and object detection methods based on 3D point clouds has led to increasing demand for labeling tools that provide suitable training data. However, existing labeling tools mostly focus on a single use case and generate bounding boxes only indirectly from a selection of points, which often impairs the label quality. Therefore, this work describes labelCloud, a generic point cloud labeling tool that can process all common file formats and provides 3D bounding boxes in multiple label formats. labelCloud offers two labeling methods that let users draw rotated bounding boxes directly inside the point cloud. Compared to a labeling tool based on indirect labeling, labelCloud could significantly increase the label precision while slightly reducing the labeling time. Due to its modular architecture, researchers and practitioners can adapt the software to their individual needs. With labelCloud, we contribute to enabling convenient 3D vision research in novel application domains.
Analytical Profile of Coleus Forskohlii | Forskolin .pptx
labelCloud – A Lightweight Domain-Independent Labeling Tool for 3D Object Detection in Point Clouds
1. Christoph Sager¹, Patrick Zschech², Niklas Kühl³ // 07.07.2021
CAD'21 - 18th annual International CAD Conference, Barcelona
labelCloud – A Lightweight Domain-Independent Labeling
Tool for 3D Object Detection in Point Clouds
¹ Technische Universität Dresden, ² Friedrich-Alexander-Universität Erlangen-Nürnberg, ³ Karlsruher Institut für Technologie
2. labelCloud – A Lightweight Domain-Independent Labeling Tool for 3D Object Detection in Point Clouds
CAD'21 - the 18th annual International CAD Conference, Barcelona // C. Sager, P. Zschech, N. Kühl
Paper & Software Presentation // 07.07.2021
Slide 2
Content – labelCloud
A Lightweight Domain-Independent Labeling Tool for 3D Object Detection in Point Clouds
1. Foundations
Point Clouds and Machine Learning
2. Related Work & Motivation
Existing Point Cloud Labeling Tools
3. Live Demonstration
Presentation of labelCloud’s Core Features
4. Evaluation & Discussion
Results of the User Study
5. Conclusion & Outlook
Applications & Further Development
3. labelCloud – A Lightweight Domain-Independent Labeling Tool for 3D Object Detection in Point Clouds
CAD'21 - the 18th annual International CAD Conference, Barcelona // C. Sager, P. Zschech, N. Kühl
Paper & Software Presentation // 07.07.2021
Slide 3
Foundations & Motivation
Point Clouds & Machine Learning
4. labelCloud – A Lightweight Domain-Independent Labeling Tool for 3D Object Detection in Point Clouds
CAD'21 - the 18th annual International CAD Conference, Barcelona // C. Sager, P. Zschech, N. Kühl
Paper & Software Presentation // 07.07.2021
Slide 4
Foundations – The need for Point Clouds in Object Detection
● Automatic object detection is increasingly used in safety-critical contexts
→ like autonomous driving, robotics and medicine
● Yet 2D-based approaches fail in estimating the size and distance of objects with high precision
● 3D-point clouds capture environments in real-world metrics
100px
300px 4 m
1.6 m
2 m
2D vs. 3D
5. labelCloud – A Lightweight Domain-Independent Labeling Tool for 3D Object Detection in Point Clouds
CAD'21 - the 18th annual International CAD Conference, Barcelona // C. Sager, P. Zschech, N. Kühl
Paper & Software Presentation // 07.07.2021
Slide 5
Foundations – Point Clouds and Machine Learning
● Point clouds are recorded with depth sensors based on lidar or stereo vision
● New architectures of artificial neural network can handle this unstructured data
○ estimate the position, rotation and dimension of objects
(Friederich et al., 2020, Qi et al., 2019, 2020)
● VoteNet
● ImVoteNet
● 3D-CNN
● ...
6. labelCloud – A Lightweight Domain-Independent Labeling Tool for 3D Object Detection in Point Clouds
CAD'21 - the 18th annual International CAD Conference, Barcelona // C. Sager, P. Zschech, N. Kühl
Paper & Software Presentation // 07.07.2021
Slide 6
Foundations – 3D Bounding Boxes as Result of 3D Object Detection
● Depth-based 3D object detection methods can predict distances, dimensions and rotations of
objects in real-world metrics (i.e., meter)
● To train neural networks, large amounts of labeled training data (known outcomes) are
necessary, which requires labeling tools specialized for point clouds
(Friederich et al., 2020; Geiger et al., 2013)
pitch
7. labelCloud – A Lightweight Domain-Independent Labeling Tool for 3D Object Detection in Point Clouds
CAD'21 - the 18th annual International CAD Conference, Barcelona // C. Sager, P. Zschech, N. Kühl
Paper & Software Presentation // 07.07.2021
Slide 7
Related Work & Motivation
Existing Tools Point Cloud Labeling Tools
8. labelCloud – A Lightweight Domain-Independent Labeling Tool for 3D Object Detection in Point Clouds
CAD'21 - the 18th annual International CAD Conference, Barcelona // C. Sager, P. Zschech, N. Kühl
Paper & Software Presentation // 07.07.2021
Slide 8
Related Work – Existing Point Cloud Labeling Tools
● Analysis of 3+ existing point cloud labeling tools
● All focus on the creation of training data for
autonomous driving applications
● This lead to …
→ few supported point cloud file formats
→ strong focus on lidar sensors
→ neglection the x- and y-rotation
(Arief et al., 2020; Wang et al., 2019; Zimmer et al., 2019)
9. labelCloud – A Lightweight Domain-Independent Labeling Tool for 3D Object Detection in Point Clouds
CAD'21 - the 18th annual International CAD Conference, Barcelona // C. Sager, P. Zschech, N. Kühl
Paper & Software Presentation // 07.07.2021
Slide 9
Motivation – Design of a lightweight, domain-independent Labeling Tool
(Chandra et al., 2015; Pfeffers et al. 2007)
Major Requirements:
R1) Support intuitive and direct labeling inside the point cloud
R2) Allow manipulation of position, rotation and dimension of 3D bounding box
R3) Increase quality and precision of the drawn bounding boxes
R4) Decrease labeling time and necessary user interactions
R5) Support multiple point cloud and label file formats
R6) Allow for flexibility while keeping dependencies to minimum
10. labelCloud – A Lightweight Domain-Independent Labeling Tool for 3D Object Detection in Point Clouds
CAD'21 - the 18th annual International CAD Conference, Barcelona // C. Sager, P. Zschech, N. Kühl
Paper & Software Presentation // 07.07.2021
Slide 10
✓ Works for 3DOD & 6DPE
✓ Full Rotation Support (x,y,z)
✓ Many Compatible Formats
✓ Fast & Easy Setup
Live Demonstration – Presentation of labelCloud’s Core Features
11. labelCloud – A Lightweight Domain-Independent Labeling Tool for 3D Object Detection in Point Clouds
CAD'21 - the 18th annual International CAD Conference, Barcelona // C. Sager, P. Zschech, N. Kühl
Paper & Software Presentation // 07.07.2021
Slide 11
MODEL
CONTROL
VIEW
Software Architecture – Standing on the Shoulders of Pip
PyOpenGL PyQt Open3D numpy
PCD Viewer
Point Cloud Label
Math3D
PCD Manger* Label Manager* Label Mode*
Picking
Mode
Spanning
Mode
modular & extend-
able architecture
12. labelCloud – A Lightweight Domain-Independent Labeling Tool for 3D Object Detection in Point Clouds
CAD'21 - the 18th annual International CAD Conference, Barcelona // C. Sager, P. Zschech, N. Kühl
Paper & Software Presentation // 07.07.2021
Slide 12
Evaluation & Discussion
Results of the User Study
13. labelCloud – A Lightweight Domain-Independent Labeling Tool for 3D Object Detection in Point Clouds
CAD'21 - the 18th annual International CAD Conference, Barcelona // C. Sager, P. Zschech, N. Kühl
Paper & Software Presentation // 07.07.2021
Slide 13
Evaluation – Comparison of Label Methods Picking & Spanning
The Spanning Mode is 22% faster and requires less user interaction (-63%).
14. labelCloud – A Lightweight Domain-Independent Labeling Tool for 3D Object Detection in Point Clouds
CAD'21 - the 18th annual International CAD Conference, Barcelona // C. Sager, P. Zschech, N. Kühl
Paper & Software Presentation // 07.07.2021
Slide 14
Discussion – labelCloud in the Context of Existing Labeling Tools
labelCloud Existing Point Cloud Labeling Tools
● Direct labeling inside the point cloud ● Indirect labeling on a 2D-projection of the point cloud
● Domain-independent and flexible ● Domain-specific and highly specialized
● Support of many depth sensors and ML-frameworks ● Focus on single sensor types and selected frameworks
● Easy to set up and few dependencies ● Sometimes difficult to set up due to many
dependencies
● So far no automatization features
→ difficult due to domain-independence
● Provide automatization features, like
○ pre-trained models
○ object tracking
○ 1-click clustering
○ projections of 2D labels
15. labelCloud – A Lightweight Domain-Independent Labeling Tool for 3D Object Detection in Point Clouds
CAD'21 - the 18th annual International CAD Conference, Barcelona // C. Sager, P. Zschech, N. Kühl
Paper & Software Presentation // 07.07.2021
Slide 15
Conclusion & Outlook
Applications & Further Development
16. labelCloud – A Lightweight Domain-Independent Labeling Tool for 3D Object Detection in Point Clouds
CAD'21 - the 18th annual International CAD Conference, Barcelona // C. Sager, P. Zschech, N. Kühl
Paper & Software Presentation // 07.07.2021
Slide 16
Conclusion & Outlook – Critical Discussion & Future Research
Conclusion
● Proven feasibility of direct labeling with 3D Bounding Boxes
● Provision of a software to quickly test 3D object detection in multiple domains
● Further reduction of the labeling time still possible with automatization features
Outlook
1. 3D object detection is more and more used in real applications outside of research
2. 3D sensors are getting more affordable and integrated into consumer products (like the iPhone)
3. The application of 3D object detection requires efficient and easy to use labeling tools
17. labelCloud – A Lightweight Domain-Independent Labeling Tool for 3D Object Detection in Point Clouds
CAD'21 - the 18th annual International CAD Conference, Barcelona // C. Sager, P. Zschech, N. Kühl
Paper & Software Presentation // 07.07.2021
Slide 17
Interested?
… try it out:
Do you see a use case that might
benefit from 3D object detection?
Do you want further information?
Let’s talk:
Christoph Sager
christophsager
christoph.sager@gmail.com
Running on Linux, Windows and Mac.
Sager, C.; Zschech, P.; Kühl, N. labelCloud: A Lightweight Domain-Independent Labeling Tool for 3D
Object Detection in Point Clouds. 2021. 18th Annual International CAD Conference (CAD'21)
Sager, C.; Zschech, P.; Kühl, N. labelCloud: A Lightweight Domain-Independent Labeling Tool for 3D
Object Detection in Point Clouds. 2021. Computer-Aided Design and Applications (forthcoming)
18. labelCloud – A Lightweight Domain-Independent Labeling Tool for 3D Object Detection in Point Clouds
CAD'21 - the 18th annual International CAD Conference, Barcelona // C. Sager, P. Zschech, N. Kühl
Paper & Software Presentation // 07.07.2021
Slide 18
Selected Sources
● Arief, H. A., Arief, M., Zhang, G., Liu, Z., Bhat, M., Indahl, U. G., Tveite, H., & Zhao, D. (2020). SAnE: Smart Annotation and Evaluation Tools for
Point Cloud Data. IEEE Access, 8, 131848–131858. https://doi.org/10.1109/ACCESS.2020.3009914
● Chandra, L., Seidel, S., & Gregor, S. (2015). Prescriptive Knowledge in IS Research: Conceptualizing Design Principles in Terms of Materiality,
Action, and Boundary Conditions. 2015 48th Hawaii International Conference on System Sciences, 4039–4048.
https://doi.org/10.1109/HICSS.2015.485
● Friederich, J.; Zschech, P.: Review and Systematization of Solutions for 3D Object Detection. In 15th International Conference on
Wirtschaftsinformatik (WI), 1699–1711. GITO Verlag, Potsdam, Germany,2020. ISBN 978-3-95545-335-0.
http://doi.org/10.30844/wi_2020_r2-friedrich
● Geiger, A.; Lenz, P.; Stiller, C.; Urtasun, R.: Vision meets robotics: The KITTI dataset. The InternationalJournal of Robotics Research, 32(11),
1231–1237, 2013. ISSN 0278-3649, 1741-3176.http://doi.org/10.1177/0278364913491297
● Peffers, K., Tuunanen, T., Rothenberger, M. A., & Chatterjee, S. (2007). A Design Science Research Methodology for Information Systems
Research. Journal of Management Information Systems, 24(3), 45–77. https://doi.org/10.2753/MIS0742-1222240302
● Qi, C.R.; Chen, X.; Litany, O.; Guibas, L.J.: ImVoteNet: Boosting 3D Object Detection in Point Clouds With Image Votes. In 2020 IEEE/CVF
Conference on Computer Vision and Pattern Recognition(CVPR), 4403–4412. IEEE, Seattle, WA, USA, 2020. ISBN 9781728171685.
http://doi.org/10.1109/CVPR42600.2020.00446
● Qi, C.R.; Litany, O.; He, K.; Guibas, L.: Deep Hough Voting for 3D Object Detection in Point Clouds. In 2019 IEEE/CVF International Conference on
Computer Vision (ICCV), 9276–9285. IEEE, Seoul, Korea(South), 2019. ISBN 978-1-72814-803-8.http://doi.org/10.1109/ICCV.2019.00937
● Sager, C.; Zschech, P.; Kühl, N. labelCloud: A Lightweight Domain-Independent Labeling Tool for 3D Object Detection in Point Clouds. 2021.
18th Annual International CAD Conference (CAD'21)
● Sager, C.; Zschech, P.; Kühl, N. labelCloud: A Lightweight Domain-Independent Labeling Tool for 3D Object Detection in Point Clouds. 2021.
Computer-Aided Design and Applications (forthcoming)
● Wang, B., Wu, V., Wu, B., & Keutzer, K. (2019). LATTE: Accelerating LiDAR Point Cloud Annotation via Sensor Fusion, One-Click Annotation, and
Tracking. 2019 IEEE Intelligent Transportation Systems Conference (ITSC), 265–272. https://doi.org/10.1109/ITSC.2019.8916980
● Wirth, F., Quehl, J., Ota, J., & Stiller, C. (2019). PointAtMe: Efficient 3D Point Cloud Labeling in Virtual Reality. 2019 IEEE Intelligent Vehicles
Symposium (IV), 1693–1698. https://doi.org/10.1109/IVS.2019.8814115
● Zimmer, W., Rangesh, A., & Trivedi, M. (2019). 3D BAT: A Semi-Automatic, Web-based 3D Annotation Toolbox for Full-Surround, Multi-Modal
Data Streams. 2019 IEEE Intelligent Vehicles Symposium (IV), 1816–1821. https://doi.org/10.1109/IVS.2019.8814071