SlideShare a Scribd company logo
1 of 21
Download to read offline
2021/7/18
Weakly Supervised Semantic Segmentation in 3D
Graph-Structured Point Clouds of Wild Scenes
Arithmer R3 team
Daisuke SATO
https://arxiv.org/abs/2004.12498v2
1
Introduction
Point clouds
● 3D sensors have been developed rapidly
these days.
○ iPhone/iPad has LiDAR
○ realsense/azure kinect is amazing
considering its low price.
● Raw data collected is point cloud
○ Collection of 3D points
○ (x0
, y0
, z0
)
(x1
, y1
, z1
)
…
(xn
, yn
, zn
)
● Processing point cloud is quite
important in robotics/measurement (測
量).
2
Introduction
Task: semantic segmentation of point clouds
● Classifying every point of 3D point clouds
○ If we perform this task with supervised
learning, we need to annotate each
point.
○ Labeling point clouds is super
exhausting.
○ Labeling 2D image is easier.
○ Let us do weakly supervised learning
with 2D projected image.
3
Introduction
4
Idea: weakly supervised learning with 2D projected image
Input is
3D point cloud
Output at
inference stage
is 3D segmented
point cloud
Label at training
stage is
segmented 2D
projected images
Introduction
Note: Projection of 3D pointcloud onto 2D image
● Projection is described by the two
parameters:
○ Internal camera parameters
■ Focal length (f)
■ principal point (c)
○ External camera parameters
■ Translation (t)
■ Rotation (R)
5
Method
Model architecture
6
Total loss
Visibility branch
(feature vector -> mask for visibility)
Input is 3D point cloud
Graph convolution encoder
(point cloud -> feature vector)
2D projection
(segmented pcd/mask for visibility
-> 2D segmented image)
Segmentation branch
(feature vector
-> mask for segmentation)
Method
Perspective rendering
● Problem: Multiple pointclouds can be
projected onto a single pixel. Which
class should we give the pixel?
7
● Solution: “Semantic fusion”
(sophiscated voting system)
Experiments
Datasets
● SUNCG Synthetic dataset
○ class number: 40 (furniture)
○ rooms: 404058
■ create 55000 2D rendering sets
● S3DIS Real-world dataset
○ class number: 13 (furniture)
○ rooms: 272
■ thousands of viewpoints are
provided
8
● Each point has
○ position: (x, y, z)
○ color: (r, g, b)
● Also normal is computed (u, v, w)
Experiments
Metrics
● mean accuracy of total classes (mAcc)
● overall accuracy (oAcc)
● mean per-class intersection-over-union
(mIoU)
○ IOU = TP / (TP + FP + FN)
9
Experiments
Comparison with other fully-supervised methods
Not so bad even compared with fully-supervised learning.
10
Experiments
Inference samples for SUNCG dataset
11
Experiments
Inference samples for S3DIS dataset
12
Conclusion
● To the best of our knowledge, this is the first work to apply 2D supervision for 3D
semantic point cloud segmentation of wild scenes without using any 3D pointwise
annotations.
● Extensive experiments are conducted and the proposed method achieves comparable
performance with the state-of-the-art 3D supervised methods on the popular SUNCG and
S3DIS benchmarks.
13
14
BACK UP
15
Method
TITLE HERE
● Bullet list
16
Experiments
Ablation study: projection method/OBSNet decoder
● Bullet list
17
Experiments
Ablation study - Encoder Design
● Bullet list
18
Experiments
Ablation study - Amount of training data
● Bullet list
19
Experiments
Ablation study - Visibility detection by OBSNet
● Bullet list
20
Experiments
Transfer learning from synthesic to realistic dataset
● Bullet list
21

More Related Content

What's hot

Introduction to Digital Image Processing
Introduction to Digital Image ProcessingIntroduction to Digital Image Processing
Introduction to Digital Image ProcessingParamjeet Singh Jamwal
 
Thresholding.ppt
Thresholding.pptThresholding.ppt
Thresholding.pptshankar64
 
Chapter 1 and 2 gonzalez and woods
Chapter 1 and 2 gonzalez and woodsChapter 1 and 2 gonzalez and woods
Chapter 1 and 2 gonzalez and woodsasodariyabhavesh
 
Lec15: Medical Image Registration (Introduction)
Lec15: Medical Image Registration (Introduction)Lec15: Medical Image Registration (Introduction)
Lec15: Medical Image Registration (Introduction)Ulaş Bağcı
 
Digital Image Processing
Digital Image ProcessingDigital Image Processing
Digital Image ProcessingReshma KC
 
Lec3: Pre-Processing Medical Images
Lec3: Pre-Processing Medical ImagesLec3: Pre-Processing Medical Images
Lec3: Pre-Processing Medical ImagesUlaş Bağcı
 
Analysing of big data using map reduce
Analysing of big data using map reduceAnalysing of big data using map reduce
Analysing of big data using map reducePaladion Networks
 
Image sampling and quantization
Image sampling and quantizationImage sampling and quantization
Image sampling and quantizationBCET, Balasore
 
Introduction to digital image processing
Introduction to digital image processingIntroduction to digital image processing
Introduction to digital image processingHossain Md Shakhawat
 
Self-supervised Learning Lecture Note
Self-supervised Learning Lecture NoteSelf-supervised Learning Lecture Note
Self-supervised Learning Lecture NoteSangwoo Mo
 
CT Image reconstruction
CT Image reconstructionCT Image reconstruction
CT Image reconstructionSantosh Ojha
 
Automatic Blood Vessels Segmentation of Retinal Images
Automatic Blood Vessels Segmentation of Retinal ImagesAutomatic Blood Vessels Segmentation of Retinal Images
Automatic Blood Vessels Segmentation of Retinal ImagesHarish Rajula
 
Scale Invariant Feature Transform
Scale Invariant Feature TransformScale Invariant Feature Transform
Scale Invariant Feature Transformkislayabhi
 
Fundamental Steps of Digital Image Processing & Image Components
Fundamental Steps of Digital Image Processing & Image ComponentsFundamental Steps of Digital Image Processing & Image Components
Fundamental Steps of Digital Image Processing & Image ComponentsKalyan Acharjya
 
Adaptive Median Filters
Adaptive Median FiltersAdaptive Median Filters
Adaptive Median FiltersAmnaakhaan
 
fundamentals-of-pacs (1).ppt
fundamentals-of-pacs (1).pptfundamentals-of-pacs (1).ppt
fundamentals-of-pacs (1).pptPriyeshKumar91
 
Basic principle of ct and ct generations
Basic principle of ct and ct generationsBasic principle of ct and ct generations
Basic principle of ct and ct generationsTarunGoyal66
 
Learning Disentangled Representation for Robust Person Re-identification
Learning Disentangled Representation for Robust Person Re-identificationLearning Disentangled Representation for Robust Person Re-identification
Learning Disentangled Representation for Robust Person Re-identificationNAVER Engineering
 
Feature Extraction
Feature ExtractionFeature Extraction
Feature Extractionskylian
 

What's hot (20)

Introduction to Digital Image Processing
Introduction to Digital Image ProcessingIntroduction to Digital Image Processing
Introduction to Digital Image Processing
 
Thresholding.ppt
Thresholding.pptThresholding.ppt
Thresholding.ppt
 
Chapter 1 and 2 gonzalez and woods
Chapter 1 and 2 gonzalez and woodsChapter 1 and 2 gonzalez and woods
Chapter 1 and 2 gonzalez and woods
 
Lec15: Medical Image Registration (Introduction)
Lec15: Medical Image Registration (Introduction)Lec15: Medical Image Registration (Introduction)
Lec15: Medical Image Registration (Introduction)
 
Digital Image Processing
Digital Image ProcessingDigital Image Processing
Digital Image Processing
 
Lec3: Pre-Processing Medical Images
Lec3: Pre-Processing Medical ImagesLec3: Pre-Processing Medical Images
Lec3: Pre-Processing Medical Images
 
Analysing of big data using map reduce
Analysing of big data using map reduceAnalysing of big data using map reduce
Analysing of big data using map reduce
 
Image sampling and quantization
Image sampling and quantizationImage sampling and quantization
Image sampling and quantization
 
Introduction to digital image processing
Introduction to digital image processingIntroduction to digital image processing
Introduction to digital image processing
 
Self-supervised Learning Lecture Note
Self-supervised Learning Lecture NoteSelf-supervised Learning Lecture Note
Self-supervised Learning Lecture Note
 
CT Image reconstruction
CT Image reconstructionCT Image reconstruction
CT Image reconstruction
 
Automatic Blood Vessels Segmentation of Retinal Images
Automatic Blood Vessels Segmentation of Retinal ImagesAutomatic Blood Vessels Segmentation of Retinal Images
Automatic Blood Vessels Segmentation of Retinal Images
 
Scale Invariant Feature Transform
Scale Invariant Feature TransformScale Invariant Feature Transform
Scale Invariant Feature Transform
 
Basic image processing techniques
Basic image processing techniquesBasic image processing techniques
Basic image processing techniques
 
Fundamental Steps of Digital Image Processing & Image Components
Fundamental Steps of Digital Image Processing & Image ComponentsFundamental Steps of Digital Image Processing & Image Components
Fundamental Steps of Digital Image Processing & Image Components
 
Adaptive Median Filters
Adaptive Median FiltersAdaptive Median Filters
Adaptive Median Filters
 
fundamentals-of-pacs (1).ppt
fundamentals-of-pacs (1).pptfundamentals-of-pacs (1).ppt
fundamentals-of-pacs (1).ppt
 
Basic principle of ct and ct generations
Basic principle of ct and ct generationsBasic principle of ct and ct generations
Basic principle of ct and ct generations
 
Learning Disentangled Representation for Robust Person Re-identification
Learning Disentangled Representation for Robust Person Re-identificationLearning Disentangled Representation for Robust Person Re-identification
Learning Disentangled Representation for Robust Person Re-identification
 
Feature Extraction
Feature ExtractionFeature Extraction
Feature Extraction
 

Similar to Weakly supervised semantic segmentation of 3D point cloud

Making BIG DATA smaller
Making BIG DATA smallerMaking BIG DATA smaller
Making BIG DATA smallerTony Tran
 
Introduction to Applied Machine Learning
Introduction to Applied Machine LearningIntroduction to Applied Machine Learning
Introduction to Applied Machine LearningSheilaJimenezMorejon
 
物件偵測與辨識技術
物件偵測與辨識技術物件偵測與辨識技術
物件偵測與辨識技術CHENHuiMei
 
Deep learning @ University of Oradea - part I (16 Jan. 2018)
Deep learning @ University of Oradea - part I (16 Jan. 2018)Deep learning @ University of Oradea - part I (16 Jan. 2018)
Deep learning @ University of Oradea - part I (16 Jan. 2018)Vlad Ovidiu Mihalca
 
Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018
Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018
Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018Universitat Politècnica de Catalunya
 
“Introducing Machine Learning and How to Teach Machines to See,” a Presentati...
“Introducing Machine Learning and How to Teach Machines to See,” a Presentati...“Introducing Machine Learning and How to Teach Machines to See,” a Presentati...
“Introducing Machine Learning and How to Teach Machines to See,” a Presentati...Edge AI and Vision Alliance
 
ECCV2022 paper reading - MultiMAE: Multi-modal Multi-task Masked Autoencoders...
ECCV2022 paper reading - MultiMAE: Multi-modal Multi-task Masked Autoencoders...ECCV2022 paper reading - MultiMAE: Multi-modal Multi-task Masked Autoencoders...
ECCV2022 paper reading - MultiMAE: Multi-modal Multi-task Masked Autoencoders...Antonio Tejero de Pablos
 
Visualizing Data Using t-SNE
Visualizing Data Using t-SNEVisualizing Data Using t-SNE
Visualizing Data Using t-SNEDavid Khosid
 
Beginning direct3d gameprogramming01_thehistoryofdirect3dgraphics_20160407_ji...
Beginning direct3d gameprogramming01_thehistoryofdirect3dgraphics_20160407_ji...Beginning direct3d gameprogramming01_thehistoryofdirect3dgraphics_20160407_ji...
Beginning direct3d gameprogramming01_thehistoryofdirect3dgraphics_20160407_ji...JinTaek Seo
 
Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud
Point-GNN: Graph Neural Network for 3D Object Detection in a Point CloudPoint-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud
Point-GNN: Graph Neural Network for 3D Object Detection in a Point CloudNuwan Sriyantha Bandara
 
Semantic Segmentation on Satellite Imagery
Semantic Segmentation on Satellite ImagerySemantic Segmentation on Satellite Imagery
Semantic Segmentation on Satellite ImageryRAHUL BHOJWANI
 
Introduction to 3D Computer Vision and Differentiable Rendering
Introduction to 3D Computer Vision and Differentiable RenderingIntroduction to 3D Computer Vision and Differentiable Rendering
Introduction to 3D Computer Vision and Differentiable RenderingPreferred Networks
 
Embeddings the geometry of relational algebra
Embeddings  the geometry of relational algebraEmbeddings  the geometry of relational algebra
Embeddings the geometry of relational algebraNikolaos Vasiloglou
 
DALL-E.pdf
DALL-E.pdfDALL-E.pdf
DALL-E.pdfdsfajkh
 
Deep single view 3 d object reconstruction with visual hull
Deep single view 3 d object reconstruction with visual hullDeep single view 3 d object reconstruction with visual hull
Deep single view 3 d object reconstruction with visual hullHanqing Wang
 

Similar to Weakly supervised semantic segmentation of 3D point cloud (20)

Making BIG DATA smaller
Making BIG DATA smallerMaking BIG DATA smaller
Making BIG DATA smaller
 
Deep 3D Visual Analysis - Javier Ruiz-Hidalgo - UPC Barcelona 2017
Deep 3D Visual Analysis - Javier Ruiz-Hidalgo - UPC Barcelona 2017Deep 3D Visual Analysis - Javier Ruiz-Hidalgo - UPC Barcelona 2017
Deep 3D Visual Analysis - Javier Ruiz-Hidalgo - UPC Barcelona 2017
 
Deep 3D Analysis - Javier Ruiz-Hidalgo - UPC Barcelona 2018
Deep 3D Analysis - Javier Ruiz-Hidalgo - UPC Barcelona 2018Deep 3D Analysis - Javier Ruiz-Hidalgo - UPC Barcelona 2018
Deep 3D Analysis - Javier Ruiz-Hidalgo - UPC Barcelona 2018
 
Introduction to Applied Machine Learning
Introduction to Applied Machine LearningIntroduction to Applied Machine Learning
Introduction to Applied Machine Learning
 
物件偵測與辨識技術
物件偵測與辨識技術物件偵測與辨識技術
物件偵測與辨識技術
 
Deep learning @ University of Oradea - part I (16 Jan. 2018)
Deep learning @ University of Oradea - part I (16 Jan. 2018)Deep learning @ University of Oradea - part I (16 Jan. 2018)
Deep learning @ University of Oradea - part I (16 Jan. 2018)
 
The Perceptron (D1L2 Deep Learning for Speech and Language)
The Perceptron (D1L2 Deep Learning for Speech and Language)The Perceptron (D1L2 Deep Learning for Speech and Language)
The Perceptron (D1L2 Deep Learning for Speech and Language)
 
Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018
Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018
Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018
 
RWDA
RWDARWDA
RWDA
 
Eye deep
Eye deepEye deep
Eye deep
 
“Introducing Machine Learning and How to Teach Machines to See,” a Presentati...
“Introducing Machine Learning and How to Teach Machines to See,” a Presentati...“Introducing Machine Learning and How to Teach Machines to See,” a Presentati...
“Introducing Machine Learning and How to Teach Machines to See,” a Presentati...
 
ECCV2022 paper reading - MultiMAE: Multi-modal Multi-task Masked Autoencoders...
ECCV2022 paper reading - MultiMAE: Multi-modal Multi-task Masked Autoencoders...ECCV2022 paper reading - MultiMAE: Multi-modal Multi-task Masked Autoencoders...
ECCV2022 paper reading - MultiMAE: Multi-modal Multi-task Masked Autoencoders...
 
Visualizing Data Using t-SNE
Visualizing Data Using t-SNEVisualizing Data Using t-SNE
Visualizing Data Using t-SNE
 
Beginning direct3d gameprogramming01_thehistoryofdirect3dgraphics_20160407_ji...
Beginning direct3d gameprogramming01_thehistoryofdirect3dgraphics_20160407_ji...Beginning direct3d gameprogramming01_thehistoryofdirect3dgraphics_20160407_ji...
Beginning direct3d gameprogramming01_thehistoryofdirect3dgraphics_20160407_ji...
 
Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud
Point-GNN: Graph Neural Network for 3D Object Detection in a Point CloudPoint-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud
Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud
 
Semantic Segmentation on Satellite Imagery
Semantic Segmentation on Satellite ImagerySemantic Segmentation on Satellite Imagery
Semantic Segmentation on Satellite Imagery
 
Introduction to 3D Computer Vision and Differentiable Rendering
Introduction to 3D Computer Vision and Differentiable RenderingIntroduction to 3D Computer Vision and Differentiable Rendering
Introduction to 3D Computer Vision and Differentiable Rendering
 
Embeddings the geometry of relational algebra
Embeddings  the geometry of relational algebraEmbeddings  the geometry of relational algebra
Embeddings the geometry of relational algebra
 
DALL-E.pdf
DALL-E.pdfDALL-E.pdf
DALL-E.pdf
 
Deep single view 3 d object reconstruction with visual hull
Deep single view 3 d object reconstruction with visual hullDeep single view 3 d object reconstruction with visual hull
Deep single view 3 d object reconstruction with visual hull
 

More from Arithmer Inc.

コーディネートレコメンド
コーディネートレコメンドコーディネートレコメンド
コーディネートレコメンドArithmer Inc.
 
Arithmerソリューション紹介 流体予測システム
Arithmerソリューション紹介 流体予測システムArithmerソリューション紹介 流体予測システム
Arithmerソリューション紹介 流体予測システムArithmer Inc.
 
Arithmer NLP 自然言語処理 ソリューション紹介
Arithmer NLP 自然言語処理 ソリューション紹介Arithmer NLP 自然言語処理 ソリューション紹介
Arithmer NLP 自然言語処理 ソリューション紹介Arithmer Inc.
 
Arithmer Robo Introduction
Arithmer Robo IntroductionArithmer Robo Introduction
Arithmer Robo IntroductionArithmer Inc.
 
Arithmer AIチャットボット
Arithmer AIチャットボットArithmer AIチャットボット
Arithmer AIチャットボットArithmer Inc.
 
Arithmer R3 Introduction
Arithmer R3 Introduction Arithmer R3 Introduction
Arithmer R3 Introduction Arithmer Inc.
 
VIBE: Video Inference for Human Body Pose and Shape Estimation
VIBE: Video Inference for Human Body Pose and Shape EstimationVIBE: Video Inference for Human Body Pose and Shape Estimation
VIBE: Video Inference for Human Body Pose and Shape EstimationArithmer Inc.
 
Arithmer Inspection Introduction
Arithmer Inspection IntroductionArithmer Inspection Introduction
Arithmer Inspection IntroductionArithmer Inc.
 
全力解説!Transformer
全力解説!Transformer全力解説!Transformer
全力解説!TransformerArithmer Inc.
 
Arithmer NLP Introduction
Arithmer NLP IntroductionArithmer NLP Introduction
Arithmer NLP IntroductionArithmer Inc.
 
Introduction of Quantum Annealing and D-Wave Machines
Introduction of Quantum Annealing and D-Wave MachinesIntroduction of Quantum Annealing and D-Wave Machines
Introduction of Quantum Annealing and D-Wave MachinesArithmer Inc.
 
Arithmer OCR Introduction
Arithmer OCR IntroductionArithmer OCR Introduction
Arithmer OCR IntroductionArithmer Inc.
 
Arithmer Dynamics Introduction
Arithmer Dynamics Introduction Arithmer Dynamics Introduction
Arithmer Dynamics Introduction Arithmer Inc.
 
ArithmerDB Introduction
ArithmerDB IntroductionArithmerDB Introduction
ArithmerDB IntroductionArithmer Inc.
 
Summarizing videos with Attention
Summarizing videos with AttentionSummarizing videos with Attention
Summarizing videos with AttentionArithmer Inc.
 
3D human body modeling from RGB images
3D human body modeling from RGB images3D human body modeling from RGB images
3D human body modeling from RGB imagesArithmer Inc.
 
Object Pose Estimation
Object Pose EstimationObject Pose Estimation
Object Pose EstimationArithmer Inc.
 

More from Arithmer Inc. (20)

コーディネートレコメンド
コーディネートレコメンドコーディネートレコメンド
コーディネートレコメンド
 
Test for AI model
Test for AI modelTest for AI model
Test for AI model
 
最適化
最適化最適化
最適化
 
Arithmerソリューション紹介 流体予測システム
Arithmerソリューション紹介 流体予測システムArithmerソリューション紹介 流体予測システム
Arithmerソリューション紹介 流体予測システム
 
Arithmer NLP 自然言語処理 ソリューション紹介
Arithmer NLP 自然言語処理 ソリューション紹介Arithmer NLP 自然言語処理 ソリューション紹介
Arithmer NLP 自然言語処理 ソリューション紹介
 
Arithmer Robo Introduction
Arithmer Robo IntroductionArithmer Robo Introduction
Arithmer Robo Introduction
 
Arithmer AIチャットボット
Arithmer AIチャットボットArithmer AIチャットボット
Arithmer AIチャットボット
 
Arithmer R3 Introduction
Arithmer R3 Introduction Arithmer R3 Introduction
Arithmer R3 Introduction
 
VIBE: Video Inference for Human Body Pose and Shape Estimation
VIBE: Video Inference for Human Body Pose and Shape EstimationVIBE: Video Inference for Human Body Pose and Shape Estimation
VIBE: Video Inference for Human Body Pose and Shape Estimation
 
Arithmer Inspection Introduction
Arithmer Inspection IntroductionArithmer Inspection Introduction
Arithmer Inspection Introduction
 
全力解説!Transformer
全力解説!Transformer全力解説!Transformer
全力解説!Transformer
 
Arithmer NLP Introduction
Arithmer NLP IntroductionArithmer NLP Introduction
Arithmer NLP Introduction
 
Introduction of Quantum Annealing and D-Wave Machines
Introduction of Quantum Annealing and D-Wave MachinesIntroduction of Quantum Annealing and D-Wave Machines
Introduction of Quantum Annealing and D-Wave Machines
 
Arithmer OCR Introduction
Arithmer OCR IntroductionArithmer OCR Introduction
Arithmer OCR Introduction
 
Arithmer Dynamics Introduction
Arithmer Dynamics Introduction Arithmer Dynamics Introduction
Arithmer Dynamics Introduction
 
ArithmerDB Introduction
ArithmerDB IntroductionArithmerDB Introduction
ArithmerDB Introduction
 
Summarizing videos with Attention
Summarizing videos with AttentionSummarizing videos with Attention
Summarizing videos with Attention
 
3D human body modeling from RGB images
3D human body modeling from RGB images3D human body modeling from RGB images
3D human body modeling from RGB images
 
YOLACT
YOLACTYOLACT
YOLACT
 
Object Pose Estimation
Object Pose EstimationObject Pose Estimation
Object Pose Estimation
 

Recently uploaded

Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Hiroshi SHIBATA
 
Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Kaya Weers
 
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Mark Goldstein
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentPim van der Noll
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen
 
Why Agile? - A handbook behind Agile Evolution
Why Agile? - A handbook behind Agile EvolutionWhy Agile? - A handbook behind Agile Evolution
Why Agile? - A handbook behind Agile EvolutionDEEPRAJ PATHAK
 
Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfIngrid Airi González
 
React Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkReact Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkPixlogix Infotech
 
Which standard is best for your content?
Which standard is best for your content?Which standard is best for your content?
Which standard is best for your content?Rustici Software
 
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...Wes McKinney
 
Dynamical Context introduction word sensibility orientation
Dynamical Context introduction word sensibility orientationDynamical Context introduction word sensibility orientation
Dynamical Context introduction word sensibility orientationBuild Intuit
 
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Alkin Tezuysal
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch TuesdayIvanti
 
Laying the Data Foundations for Artificial Intelligence!
Laying the Data Foundations for Artificial Intelligence!Laying the Data Foundations for Artificial Intelligence!
Laying the Data Foundations for Artificial Intelligence!Memoori
 
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...panagenda
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...Nikki Chapple
 
A Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxA Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxAna-Maria Mihalceanu
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsRavi Sanghani
 
Digital Tools & AI in Career Development
Digital Tools & AI in Career DevelopmentDigital Tools & AI in Career Development
Digital Tools & AI in Career DevelopmentMahmoud Rabie
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024TopCSSGallery
 

Recently uploaded (20)

Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024
 
Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)
 
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examples
 
Why Agile? - A handbook behind Agile Evolution
Why Agile? - A handbook behind Agile EvolutionWhy Agile? - A handbook behind Agile Evolution
Why Agile? - A handbook behind Agile Evolution
 
Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdf
 
React Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkReact Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App Framework
 
Which standard is best for your content?
Which standard is best for your content?Which standard is best for your content?
Which standard is best for your content?
 
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
 
Dynamical Context introduction word sensibility orientation
Dynamical Context introduction word sensibility orientationDynamical Context introduction word sensibility orientation
Dynamical Context introduction word sensibility orientation
 
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch Tuesday
 
Laying the Data Foundations for Artificial Intelligence!
Laying the Data Foundations for Artificial Intelligence!Laying the Data Foundations for Artificial Intelligence!
Laying the Data Foundations for Artificial Intelligence!
 
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...
 
A Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxA Glance At The Java Performance Toolbox
A Glance At The Java Performance Toolbox
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and Insights
 
Digital Tools & AI in Career Development
Digital Tools & AI in Career DevelopmentDigital Tools & AI in Career Development
Digital Tools & AI in Career Development
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024
 

Weakly supervised semantic segmentation of 3D point cloud

  • 1. 2021/7/18 Weakly Supervised Semantic Segmentation in 3D Graph-Structured Point Clouds of Wild Scenes Arithmer R3 team Daisuke SATO https://arxiv.org/abs/2004.12498v2 1
  • 2. Introduction Point clouds ● 3D sensors have been developed rapidly these days. ○ iPhone/iPad has LiDAR ○ realsense/azure kinect is amazing considering its low price. ● Raw data collected is point cloud ○ Collection of 3D points ○ (x0 , y0 , z0 ) (x1 , y1 , z1 ) … (xn , yn , zn ) ● Processing point cloud is quite important in robotics/measurement (測 量). 2
  • 3. Introduction Task: semantic segmentation of point clouds ● Classifying every point of 3D point clouds ○ If we perform this task with supervised learning, we need to annotate each point. ○ Labeling point clouds is super exhausting. ○ Labeling 2D image is easier. ○ Let us do weakly supervised learning with 2D projected image. 3
  • 4. Introduction 4 Idea: weakly supervised learning with 2D projected image Input is 3D point cloud Output at inference stage is 3D segmented point cloud Label at training stage is segmented 2D projected images
  • 5. Introduction Note: Projection of 3D pointcloud onto 2D image ● Projection is described by the two parameters: ○ Internal camera parameters ■ Focal length (f) ■ principal point (c) ○ External camera parameters ■ Translation (t) ■ Rotation (R) 5
  • 6. Method Model architecture 6 Total loss Visibility branch (feature vector -> mask for visibility) Input is 3D point cloud Graph convolution encoder (point cloud -> feature vector) 2D projection (segmented pcd/mask for visibility -> 2D segmented image) Segmentation branch (feature vector -> mask for segmentation)
  • 7. Method Perspective rendering ● Problem: Multiple pointclouds can be projected onto a single pixel. Which class should we give the pixel? 7 ● Solution: “Semantic fusion” (sophiscated voting system)
  • 8. Experiments Datasets ● SUNCG Synthetic dataset ○ class number: 40 (furniture) ○ rooms: 404058 ■ create 55000 2D rendering sets ● S3DIS Real-world dataset ○ class number: 13 (furniture) ○ rooms: 272 ■ thousands of viewpoints are provided 8 ● Each point has ○ position: (x, y, z) ○ color: (r, g, b) ● Also normal is computed (u, v, w)
  • 9. Experiments Metrics ● mean accuracy of total classes (mAcc) ● overall accuracy (oAcc) ● mean per-class intersection-over-union (mIoU) ○ IOU = TP / (TP + FP + FN) 9
  • 10. Experiments Comparison with other fully-supervised methods Not so bad even compared with fully-supervised learning. 10
  • 13. Conclusion ● To the best of our knowledge, this is the first work to apply 2D supervision for 3D semantic point cloud segmentation of wild scenes without using any 3D pointwise annotations. ● Extensive experiments are conducted and the proposed method achieves comparable performance with the state-of-the-art 3D supervised methods on the popular SUNCG and S3DIS benchmarks. 13
  • 14. 14
  • 17. Experiments Ablation study: projection method/OBSNet decoder ● Bullet list 17
  • 18. Experiments Ablation study - Encoder Design ● Bullet list 18
  • 19. Experiments Ablation study - Amount of training data ● Bullet list 19
  • 20. Experiments Ablation study - Visibility detection by OBSNet ● Bullet list 20
  • 21. Experiments Transfer learning from synthesic to realistic dataset ● Bullet list 21