SlideShare a Scribd company logo

Cross-domain complementary learning with synthetic data for multi-person part segmentation

presented by Kevin Lin

Cross-domain complementary learning with synthetic data for multi-person part segmentation

1 of 37
Download to read offline
Cross-domain Complementary Learning
with Synthetic Data for Multi-Person
Part Segmentation
Kevin Lin, Lijuan Wang, Kun Luo, Yinpeng Chen, Zicheng Liu, Ming-Ting Sun
University of Washington, Seattle
Microsoft, Redmond
International Conference on Computer Vision (ICCV), Demonstration, 2019
1
Outline
• Introduction
• Related works
• Proposed method
• Experiments
• On-going work and Conclusion
2
Human part segmentation
• Human part segmentation aims at partitioning persons in the image
to multiple semantically consistent regions.
Typically 14 parts: Head, torso, left upper-arm, right upper-arm, left lower-arm, right lower-
arm, left hand, right hand, left thigh, right thigh, left shank, right shank, left foot, right foot
Input Image Part Segmentation
3
Challenges
• Training data labeling in pixel-level is very expensive and labor intensive.
4
Previous works
• People have been exploring synthetic data as an alternative.
• They trained deep CNN using the synthetic data.
Samples of the synthetic training data and the synthetic labels [CVPR17]
5
Previous works
Their method works well only on the well-controlled, single-person
scenario.
Learning from Synthetic Humans, CVPR 2017
Input
images
Output
results
6
Ad

Recommended

Learning Disentangled Representation for Robust Person Re-identification
Learning Disentangled Representation for Robust Person Re-identificationLearning Disentangled Representation for Robust Person Re-identification
Learning Disentangled Representation for Robust Person Re-identificationNAVER Engineering
 
Deep learning for person re-identification
Deep learning for person re-identificationDeep learning for person re-identification
Deep learning for person re-identification哲东 郑
 
Visual geometry with deep learning
Visual geometry with deep learningVisual geometry with deep learning
Visual geometry with deep learningNAVER Engineering
 
Backbone can not be trained at once rolling back to pre trained network for p...
Backbone can not be trained at once rolling back to pre trained network for p...Backbone can not be trained at once rolling back to pre trained network for p...
Backbone can not be trained at once rolling back to pre trained network for p...NAVER Engineering
 
Deep re-id: 关于行人重识别的深度学习方法
Deep re-id: 关于行人重识别的深度学习方法Deep re-id: 关于行人重识别的深度学习方法
Deep re-id: 关于行人重识别的深度学习方法哲东 郑
 
Synthesizing pseudo 2.5 d content from monocular videos for mixed reality
Synthesizing pseudo 2.5 d content from monocular videos for mixed realitySynthesizing pseudo 2.5 d content from monocular videos for mixed reality
Synthesizing pseudo 2.5 d content from monocular videos for mixed realityNAVER Engineering
 
PR100: SeedNet: Automatic Seed Generation with Deep Reinforcement Learning fo...
PR100: SeedNet: Automatic Seed Generation with Deep Reinforcement Learning fo...PR100: SeedNet: Automatic Seed Generation with Deep Reinforcement Learning fo...
PR100: SeedNet: Automatic Seed Generation with Deep Reinforcement Learning fo...광희 이
 

More Related Content

What's hot

보다 유연한 이미지 변환을 하려면?
보다 유연한 이미지 변환을 하려면?보다 유연한 이미지 변환을 하려면?
보다 유연한 이미지 변환을 하려면?광희 이
 
Sparse representation based human action recognition using an action region-a...
Sparse representation based human action recognition using an action region-a...Sparse representation based human action recognition using an action region-a...
Sparse representation based human action recognition using an action region-a...Wesley De Neve
 
Action Genome: Action As Composition of Spatio Temporal Scene Graphs
Action Genome: Action As Composition of Spatio Temporal Scene GraphsAction Genome: Action As Composition of Spatio Temporal Scene Graphs
Action Genome: Action As Composition of Spatio Temporal Scene GraphsSangmin Woo
 
Modeling perceptual similarity and shift invariance in deep networks
Modeling perceptual similarity and shift invariance in deep networksModeling perceptual similarity and shift invariance in deep networks
Modeling perceptual similarity and shift invariance in deep networksNAVER Engineering
 
Action Recognitionの歴史と最新動向
Action Recognitionの歴史と最新動向Action Recognitionの歴史と最新動向
Action Recognitionの歴史と最新動向Ohnishi Katsunori
 
Unsupervised image to-image translation via pre-trained style gan2 network
Unsupervised image to-image translation via pre-trained style gan2 networkUnsupervised image to-image translation via pre-trained style gan2 network
Unsupervised image to-image translation via pre-trained style gan2 network광희 이
 
Dario izzo - Machine Learning methods and space engineering
Dario izzo - Machine Learning methods and space engineeringDario izzo - Machine Learning methods and space engineering
Dario izzo - Machine Learning methods and space engineeringAdvanced-Concepts-Team
 
Dynamic Two-Stage Image Retrieval from Large Multimodal Databases
Dynamic Two-Stage Image Retrieval from Large Multimodal DatabasesDynamic Two-Stage Image Retrieval from Large Multimodal Databases
Dynamic Two-Stage Image Retrieval from Large Multimodal DatabasesKonstantinos Zagoris
 
IRJET - Object Detection using Deep Learning with OpenCV and Python
IRJET - Object Detection using Deep Learning with OpenCV and PythonIRJET - Object Detection using Deep Learning with OpenCV and Python
IRJET - Object Detection using Deep Learning with OpenCV and PythonIRJET Journal
 
Obscenity Detection in Images
Obscenity Detection in ImagesObscenity Detection in Images
Obscenity Detection in ImagesAnil Kumar Gupta
 
LFI-CAM: Learning Feature Importance for Better Visual Explanation
LFI-CAM: Learning Feature Importance for Better Visual ExplanationLFI-CAM: Learning Feature Importance for Better Visual Explanation
LFI-CAM: Learning Feature Importance for Better Visual Explanation광희 이
 
184816386 x mining
184816386 x mining184816386 x mining
184816386 x mining496573
 
Temporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks
Temporal Activity Detection in Untrimmed Videos with Recurrent Neural NetworksTemporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks
Temporal Activity Detection in Untrimmed Videos with Recurrent Neural NetworksUniversitat Politècnica de Catalunya
 
NumPyCNNAndroid: A Library for Straightforward Implementation of Convolutiona...
NumPyCNNAndroid: A Library for Straightforward Implementation of Convolutiona...NumPyCNNAndroid: A Library for Straightforward Implementation of Convolutiona...
NumPyCNNAndroid: A Library for Straightforward Implementation of Convolutiona...Ahmed Gad
 
Intelligent Multimedia Recommendation
Intelligent Multimedia RecommendationIntelligent Multimedia Recommendation
Intelligent Multimedia RecommendationWanjin Yu
 
Big-Data Analytics for Media Management
Big-Data Analytics for Media ManagementBig-Data Analytics for Media Management
Big-Data Analytics for Media Managementtechkrish
 
Usage of Generative Adversarial Networks (GANs) in Healthcare
Usage of Generative Adversarial Networks (GANs) in HealthcareUsage of Generative Adversarial Networks (GANs) in Healthcare
Usage of Generative Adversarial Networks (GANs) in HealthcareGlobalLogic Ukraine
 
ICCES 2017 - Crowd Density Estimation Method using Regression Analysis
ICCES 2017 - Crowd Density Estimation Method using Regression AnalysisICCES 2017 - Crowd Density Estimation Method using Regression Analysis
ICCES 2017 - Crowd Density Estimation Method using Regression AnalysisAhmed Gad
 
Object Tracking with Instance Matching and Online Learning
Object Tracking with Instance Matching and Online LearningObject Tracking with Instance Matching and Online Learning
Object Tracking with Instance Matching and Online LearningJui-Hsin (Larry) Lai
 
Face recognition v1
Face recognition v1Face recognition v1
Face recognition v1San Kim
 

What's hot (20)

보다 유연한 이미지 변환을 하려면?
보다 유연한 이미지 변환을 하려면?보다 유연한 이미지 변환을 하려면?
보다 유연한 이미지 변환을 하려면?
 
Sparse representation based human action recognition using an action region-a...
Sparse representation based human action recognition using an action region-a...Sparse representation based human action recognition using an action region-a...
Sparse representation based human action recognition using an action region-a...
 
Action Genome: Action As Composition of Spatio Temporal Scene Graphs
Action Genome: Action As Composition of Spatio Temporal Scene GraphsAction Genome: Action As Composition of Spatio Temporal Scene Graphs
Action Genome: Action As Composition of Spatio Temporal Scene Graphs
 
Modeling perceptual similarity and shift invariance in deep networks
Modeling perceptual similarity and shift invariance in deep networksModeling perceptual similarity and shift invariance in deep networks
Modeling perceptual similarity and shift invariance in deep networks
 
Action Recognitionの歴史と最新動向
Action Recognitionの歴史と最新動向Action Recognitionの歴史と最新動向
Action Recognitionの歴史と最新動向
 
Unsupervised image to-image translation via pre-trained style gan2 network
Unsupervised image to-image translation via pre-trained style gan2 networkUnsupervised image to-image translation via pre-trained style gan2 network
Unsupervised image to-image translation via pre-trained style gan2 network
 
Dario izzo - Machine Learning methods and space engineering
Dario izzo - Machine Learning methods and space engineeringDario izzo - Machine Learning methods and space engineering
Dario izzo - Machine Learning methods and space engineering
 
Dynamic Two-Stage Image Retrieval from Large Multimodal Databases
Dynamic Two-Stage Image Retrieval from Large Multimodal DatabasesDynamic Two-Stage Image Retrieval from Large Multimodal Databases
Dynamic Two-Stage Image Retrieval from Large Multimodal Databases
 
IRJET - Object Detection using Deep Learning with OpenCV and Python
IRJET - Object Detection using Deep Learning with OpenCV and PythonIRJET - Object Detection using Deep Learning with OpenCV and Python
IRJET - Object Detection using Deep Learning with OpenCV and Python
 
Obscenity Detection in Images
Obscenity Detection in ImagesObscenity Detection in Images
Obscenity Detection in Images
 
LFI-CAM: Learning Feature Importance for Better Visual Explanation
LFI-CAM: Learning Feature Importance for Better Visual ExplanationLFI-CAM: Learning Feature Importance for Better Visual Explanation
LFI-CAM: Learning Feature Importance for Better Visual Explanation
 
184816386 x mining
184816386 x mining184816386 x mining
184816386 x mining
 
Temporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks
Temporal Activity Detection in Untrimmed Videos with Recurrent Neural NetworksTemporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks
Temporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks
 
NumPyCNNAndroid: A Library for Straightforward Implementation of Convolutiona...
NumPyCNNAndroid: A Library for Straightforward Implementation of Convolutiona...NumPyCNNAndroid: A Library for Straightforward Implementation of Convolutiona...
NumPyCNNAndroid: A Library for Straightforward Implementation of Convolutiona...
 
Intelligent Multimedia Recommendation
Intelligent Multimedia RecommendationIntelligent Multimedia Recommendation
Intelligent Multimedia Recommendation
 
Big-Data Analytics for Media Management
Big-Data Analytics for Media ManagementBig-Data Analytics for Media Management
Big-Data Analytics for Media Management
 
Usage of Generative Adversarial Networks (GANs) in Healthcare
Usage of Generative Adversarial Networks (GANs) in HealthcareUsage of Generative Adversarial Networks (GANs) in Healthcare
Usage of Generative Adversarial Networks (GANs) in Healthcare
 
ICCES 2017 - Crowd Density Estimation Method using Regression Analysis
ICCES 2017 - Crowd Density Estimation Method using Regression AnalysisICCES 2017 - Crowd Density Estimation Method using Regression Analysis
ICCES 2017 - Crowd Density Estimation Method using Regression Analysis
 
Object Tracking with Instance Matching and Online Learning
Object Tracking with Instance Matching and Online LearningObject Tracking with Instance Matching and Online Learning
Object Tracking with Instance Matching and Online Learning
 
Face recognition v1
Face recognition v1Face recognition v1
Face recognition v1
 

Similar to Cross-domain complementary learning with synthetic data for multi-person part segmentation

AI Personal Trainer Using Open CV and Media Pipe
AI Personal Trainer Using Open CV and Media PipeAI Personal Trainer Using Open CV and Media Pipe
AI Personal Trainer Using Open CV and Media PipeIRJET Journal
 
Human Behavior Understanding: From Human-Oriented Analysis to Action Recognit...
Human Behavior Understanding: From Human-Oriented Analysis to Action Recognit...Human Behavior Understanding: From Human-Oriented Analysis to Action Recognit...
Human Behavior Understanding: From Human-Oriented Analysis to Action Recognit...Wanjin Yu
 
AI Personal Trainer Using Open CV and Media Pipe
AI Personal Trainer Using Open CV and Media PipeAI Personal Trainer Using Open CV and Media Pipe
AI Personal Trainer Using Open CV and Media PipeIRJET Journal
 
Partial Object Detection in Inclined Weather Conditions
Partial Object Detection in Inclined Weather ConditionsPartial Object Detection in Inclined Weather Conditions
Partial Object Detection in Inclined Weather ConditionsIRJET Journal
 
The Shot Boundary And Classification Of Digital Video Essay
The Shot Boundary And Classification Of Digital Video EssayThe Shot Boundary And Classification Of Digital Video Essay
The Shot Boundary And Classification Of Digital Video EssayMarisela Stone
 
KaoNet: Face Recognition and Generation App using Deep Learning
KaoNet: Face Recognition and Generation App using Deep LearningKaoNet: Face Recognition and Generation App using Deep Learning
KaoNet: Face Recognition and Generation App using Deep LearningVan Huy
 
Fast Parallel Similarity Calculations with FPGA Hardware
Fast Parallel Similarity Calculations with FPGA HardwareFast Parallel Similarity Calculations with FPGA Hardware
Fast Parallel Similarity Calculations with FPGA HardwareTigerGraph
 
Transfer Learning: Breve introducción a modelos pre-entrenados.
Transfer Learning: Breve introducción a modelos pre-entrenados.Transfer Learning: Breve introducción a modelos pre-entrenados.
Transfer Learning: Breve introducción a modelos pre-entrenados.Fernando Constantino
 
brief Introduction to Different Kinds of GANs
brief Introduction to Different Kinds of GANsbrief Introduction to Different Kinds of GANs
brief Introduction to Different Kinds of GANsParham Zilouchian
 
CNN FEATURES ARE ALSO GREAT AT UNSUPERVISED CLASSIFICATION
CNN FEATURES ARE ALSO GREAT AT UNSUPERVISED CLASSIFICATION CNN FEATURES ARE ALSO GREAT AT UNSUPERVISED CLASSIFICATION
CNN FEATURES ARE ALSO GREAT AT UNSUPERVISED CLASSIFICATION cscpconf
 
Analysis of Educational Robotics activities using a machine learning approach
Analysis of Educational Robotics activities using a machine learning approachAnalysis of Educational Robotics activities using a machine learning approach
Analysis of Educational Robotics activities using a machine learning approachLorenzo Cesaretti
 
Proactive Rescue Work by Enhancing Situational Awareness: Modeling Resources,...
Proactive Rescue Work by Enhancing Situational Awareness: Modeling Resources,...Proactive Rescue Work by Enhancing Situational Awareness: Modeling Resources,...
Proactive Rescue Work by Enhancing Situational Awareness: Modeling Resources,...Matti Luhtala
 
Age Estimation And Gender Prediction Using Convolutional Neural Network.pptx
Age Estimation And Gender Prediction Using Convolutional Neural Network.pptxAge Estimation And Gender Prediction Using Convolutional Neural Network.pptx
Age Estimation And Gender Prediction Using Convolutional Neural Network.pptxBulbul Agrawal
 
H2O with Erin LeDell at Portland R User Group
H2O with Erin LeDell at Portland R User GroupH2O with Erin LeDell at Portland R User Group
H2O with Erin LeDell at Portland R User GroupSri Ambati
 
HUMAN IDENTIFIER WITH MANNERISM USING DEEP LEARNING
HUMAN IDENTIFIER WITH MANNERISM USING DEEP LEARNINGHUMAN IDENTIFIER WITH MANNERISM USING DEEP LEARNING
HUMAN IDENTIFIER WITH MANNERISM USING DEEP LEARNINGIRJET Journal
 

Similar to Cross-domain complementary learning with synthetic data for multi-person part segmentation (20)

AI Personal Trainer Using Open CV and Media Pipe
AI Personal Trainer Using Open CV and Media PipeAI Personal Trainer Using Open CV and Media Pipe
AI Personal Trainer Using Open CV and Media Pipe
 
Human Behavior Understanding: From Human-Oriented Analysis to Action Recognit...
Human Behavior Understanding: From Human-Oriented Analysis to Action Recognit...Human Behavior Understanding: From Human-Oriented Analysis to Action Recognit...
Human Behavior Understanding: From Human-Oriented Analysis to Action Recognit...
 
AI Personal Trainer Using Open CV and Media Pipe
AI Personal Trainer Using Open CV and Media PipeAI Personal Trainer Using Open CV and Media Pipe
AI Personal Trainer Using Open CV and Media Pipe
 
Partial Object Detection in Inclined Weather Conditions
Partial Object Detection in Inclined Weather ConditionsPartial Object Detection in Inclined Weather Conditions
Partial Object Detection in Inclined Weather Conditions
 
Learning where to look: focus and attention in deep vision
Learning where to look: focus and attention in deep visionLearning where to look: focus and attention in deep vision
Learning where to look: focus and attention in deep vision
 
The Shot Boundary And Classification Of Digital Video Essay
The Shot Boundary And Classification Of Digital Video EssayThe Shot Boundary And Classification Of Digital Video Essay
The Shot Boundary And Classification Of Digital Video Essay
 
KaoNet: Face Recognition and Generation App using Deep Learning
KaoNet: Face Recognition and Generation App using Deep LearningKaoNet: Face Recognition and Generation App using Deep Learning
KaoNet: Face Recognition and Generation App using Deep Learning
 
BTP Report.pdf
BTP Report.pdfBTP Report.pdf
BTP Report.pdf
 
Fa19_P1.pptx
Fa19_P1.pptxFa19_P1.pptx
Fa19_P1.pptx
 
Fast Parallel Similarity Calculations with FPGA Hardware
Fast Parallel Similarity Calculations with FPGA HardwareFast Parallel Similarity Calculations with FPGA Hardware
Fast Parallel Similarity Calculations with FPGA Hardware
 
Transfer Learning: Breve introducción a modelos pre-entrenados.
Transfer Learning: Breve introducción a modelos pre-entrenados.Transfer Learning: Breve introducción a modelos pre-entrenados.
Transfer Learning: Breve introducción a modelos pre-entrenados.
 
MILA DL & RL summer school highlights
MILA DL & RL summer school highlights MILA DL & RL summer school highlights
MILA DL & RL summer school highlights
 
brief Introduction to Different Kinds of GANs
brief Introduction to Different Kinds of GANsbrief Introduction to Different Kinds of GANs
brief Introduction to Different Kinds of GANs
 
final ppt
final pptfinal ppt
final ppt
 
CNN FEATURES ARE ALSO GREAT AT UNSUPERVISED CLASSIFICATION
CNN FEATURES ARE ALSO GREAT AT UNSUPERVISED CLASSIFICATION CNN FEATURES ARE ALSO GREAT AT UNSUPERVISED CLASSIFICATION
CNN FEATURES ARE ALSO GREAT AT UNSUPERVISED CLASSIFICATION
 
Analysis of Educational Robotics activities using a machine learning approach
Analysis of Educational Robotics activities using a machine learning approachAnalysis of Educational Robotics activities using a machine learning approach
Analysis of Educational Robotics activities using a machine learning approach
 
Proactive Rescue Work by Enhancing Situational Awareness: Modeling Resources,...
Proactive Rescue Work by Enhancing Situational Awareness: Modeling Resources,...Proactive Rescue Work by Enhancing Situational Awareness: Modeling Resources,...
Proactive Rescue Work by Enhancing Situational Awareness: Modeling Resources,...
 
Age Estimation And Gender Prediction Using Convolutional Neural Network.pptx
Age Estimation And Gender Prediction Using Convolutional Neural Network.pptxAge Estimation And Gender Prediction Using Convolutional Neural Network.pptx
Age Estimation And Gender Prediction Using Convolutional Neural Network.pptx
 
H2O with Erin LeDell at Portland R User Group
H2O with Erin LeDell at Portland R User GroupH2O with Erin LeDell at Portland R User Group
H2O with Erin LeDell at Portland R User Group
 
HUMAN IDENTIFIER WITH MANNERISM USING DEEP LEARNING
HUMAN IDENTIFIER WITH MANNERISM USING DEEP LEARNINGHUMAN IDENTIFIER WITH MANNERISM USING DEEP LEARNING
HUMAN IDENTIFIER WITH MANNERISM USING DEEP LEARNING
 

More from 哲东 郑

Visual saliency
Visual saliencyVisual saliency
Visual saliency哲东 郑
 
Image Synthesis From Reconfigurable Layout and Style
Image Synthesis From Reconfigurable Layout and StyleImage Synthesis From Reconfigurable Layout and Style
Image Synthesis From Reconfigurable Layout and Style哲东 郑
 
Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval
Polysemous Visual-Semantic Embedding for Cross-Modal RetrievalPolysemous Visual-Semantic Embedding for Cross-Modal Retrieval
Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval哲东 郑
 
Weijian image retrieval
Weijian image retrievalWeijian image retrieval
Weijian image retrieval哲东 郑
 
Scops self supervised co-part segmentation
Scops self supervised co-part segmentationScops self supervised co-part segmentation
Scops self supervised co-part segmentation哲东 郑
 
Video object detection
Video object detectionVideo object detection
Video object detection哲东 郑
 
C2 ae open set recognition
C2 ae open set recognitionC2 ae open set recognition
C2 ae open set recognition哲东 郑
 
Sota semantic segmentation
Sota semantic segmentationSota semantic segmentation
Sota semantic segmentation哲东 郑
 
Deep randomized embedding
Deep randomized embeddingDeep randomized embedding
Deep randomized embedding哲东 郑
 
Semantic Image Synthesis with Spatially-Adaptive Normalization
Semantic Image Synthesis with Spatially-Adaptive NormalizationSemantic Image Synthesis with Spatially-Adaptive Normalization
Semantic Image Synthesis with Spatially-Adaptive Normalization哲东 郑
 
Instance level facial attributes transfer with geometry-aware flow
Instance level facial attributes transfer with geometry-aware flowInstance level facial attributes transfer with geometry-aware flow
Instance level facial attributes transfer with geometry-aware flow哲东 郑
 
Learning to adapt structured output space for semantic
Learning to adapt structured output space for semanticLearning to adapt structured output space for semantic
Learning to adapt structured output space for semantic哲东 郑
 
Unsupervised Learning of Object Landmarks through Conditional Image Generation
Unsupervised Learning of Object Landmarks through Conditional Image GenerationUnsupervised Learning of Object Landmarks through Conditional Image Generation
Unsupervised Learning of Object Landmarks through Conditional Image Generation哲东 郑
 
Graph based global reasoning networks
Graph based global reasoning networks Graph based global reasoning networks
Graph based global reasoning networks 哲东 郑
 
Variational Discriminator Bottleneck
Variational Discriminator BottleneckVariational Discriminator Bottleneck
Variational Discriminator Bottleneck哲东 郑
 
GNorm and Rethinking pre training-ruijie
GNorm and Rethinking pre training-ruijieGNorm and Rethinking pre training-ruijie
GNorm and Rethinking pre training-ruijie哲东 郑
 
Smoothed manifold
Smoothed manifoldSmoothed manifold
Smoothed manifold哲东 郑
 

More from 哲东 郑 (20)

Visual saliency
Visual saliencyVisual saliency
Visual saliency
 
Image Synthesis From Reconfigurable Layout and Style
Image Synthesis From Reconfigurable Layout and StyleImage Synthesis From Reconfigurable Layout and Style
Image Synthesis From Reconfigurable Layout and Style
 
Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval
Polysemous Visual-Semantic Embedding for Cross-Modal RetrievalPolysemous Visual-Semantic Embedding for Cross-Modal Retrieval
Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval
 
Weijian image retrieval
Weijian image retrievalWeijian image retrieval
Weijian image retrieval
 
Scops self supervised co-part segmentation
Scops self supervised co-part segmentationScops self supervised co-part segmentation
Scops self supervised co-part segmentation
 
Video object detection
Video object detectionVideo object detection
Video object detection
 
Center nets
Center netsCenter nets
Center nets
 
C2 ae open set recognition
C2 ae open set recognitionC2 ae open set recognition
C2 ae open set recognition
 
Sota semantic segmentation
Sota semantic segmentationSota semantic segmentation
Sota semantic segmentation
 
Deep randomized embedding
Deep randomized embeddingDeep randomized embedding
Deep randomized embedding
 
Semantic Image Synthesis with Spatially-Adaptive Normalization
Semantic Image Synthesis with Spatially-Adaptive NormalizationSemantic Image Synthesis with Spatially-Adaptive Normalization
Semantic Image Synthesis with Spatially-Adaptive Normalization
 
Instance level facial attributes transfer with geometry-aware flow
Instance level facial attributes transfer with geometry-aware flowInstance level facial attributes transfer with geometry-aware flow
Instance level facial attributes transfer with geometry-aware flow
 
Learning to adapt structured output space for semantic
Learning to adapt structured output space for semanticLearning to adapt structured output space for semantic
Learning to adapt structured output space for semantic
 
Unsupervised Learning of Object Landmarks through Conditional Image Generation
Unsupervised Learning of Object Landmarks through Conditional Image GenerationUnsupervised Learning of Object Landmarks through Conditional Image Generation
Unsupervised Learning of Object Landmarks through Conditional Image Generation
 
Graph based global reasoning networks
Graph based global reasoning networks Graph based global reasoning networks
Graph based global reasoning networks
 
Style gan
Style ganStyle gan
Style gan
 
Vi2vi
Vi2viVi2vi
Vi2vi
 
Variational Discriminator Bottleneck
Variational Discriminator BottleneckVariational Discriminator Bottleneck
Variational Discriminator Bottleneck
 
GNorm and Rethinking pre training-ruijie
GNorm and Rethinking pre training-ruijieGNorm and Rethinking pre training-ruijie
GNorm and Rethinking pre training-ruijie
 
Smoothed manifold
Smoothed manifoldSmoothed manifold
Smoothed manifold
 

Recently uploaded

AWS reInvent 2023 recaps from Chicago AWS user group
AWS reInvent 2023 recaps from Chicago AWS user groupAWS reInvent 2023 recaps from Chicago AWS user group
AWS reInvent 2023 recaps from Chicago AWS user groupAWS Chicago
 
5 Things You Shouldn’t Do at Salesforce World Tour Sydney 2024!
5 Things You Shouldn’t Do at Salesforce World Tour Sydney 2024!5 Things You Shouldn’t Do at Salesforce World Tour Sydney 2024!
5 Things You Shouldn’t Do at Salesforce World Tour Sydney 2024!XfilesPro
 
LLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdf
LLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdfLLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdf
LLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdfThomas Poetter
 
How we think about an advisor tech stack
How we think about an advisor tech stackHow we think about an advisor tech stack
How we think about an advisor tech stackSummit
 
DNA LIGASE BIOTECHNOLOGY BIOLOGY STUDY OF LIFE
DNA LIGASE BIOTECHNOLOGY BIOLOGY STUDY OF LIFEDNA LIGASE BIOTECHNOLOGY BIOLOGY STUDY OF LIFE
DNA LIGASE BIOTECHNOLOGY BIOLOGY STUDY OF LIFEandreiandasan
 
Building Products That Think- Bhaskaran Srinivasan & Ashish Gupta
Building Products That Think- Bhaskaran Srinivasan & Ashish GuptaBuilding Products That Think- Bhaskaran Srinivasan & Ashish Gupta
Building Products That Think- Bhaskaran Srinivasan & Ashish GuptaISPMAIndia
 
Zi-Stick UBS Dongle ZIgbee from Aeotec manual
Zi-Stick UBS Dongle ZIgbee from  Aeotec manualZi-Stick UBS Dongle ZIgbee from  Aeotec manual
Zi-Stick UBS Dongle ZIgbee from Aeotec manualDomotica daVinci
 
"Running Open-Source LLM models on Kubernetes", Volodymyr Tsap
"Running Open-Source LLM models on Kubernetes",  Volodymyr Tsap"Running Open-Source LLM models on Kubernetes",  Volodymyr Tsap
"Running Open-Source LLM models on Kubernetes", Volodymyr TsapFwdays
 
zigbee motion sensor user manual NAS-PD07B2.pdf
zigbee motion sensor user manual NAS-PD07B2.pdfzigbee motion sensor user manual NAS-PD07B2.pdf
zigbee motion sensor user manual NAS-PD07B2.pdfDomotica daVinci
 
Power of 2024 - WITforce Odyssey.pptx.pdf
Power of 2024 - WITforce Odyssey.pptx.pdfPower of 2024 - WITforce Odyssey.pptx.pdf
Power of 2024 - WITforce Odyssey.pptx.pdfkatalinjordans1
 
Q1 Memory Fabric Forum: Intel Enabling Compute Express Link (CXL)
Q1 Memory Fabric Forum: Intel Enabling Compute Express Link (CXL)Q1 Memory Fabric Forum: Intel Enabling Compute Express Link (CXL)
Q1 Memory Fabric Forum: Intel Enabling Compute Express Link (CXL)Memory Fabric Forum
 
Introduction to Serverless with AWS Lambda in C#.pptx
Introduction to Serverless with AWS Lambda in C#.pptxIntroduction to Serverless with AWS Lambda in C#.pptx
Introduction to Serverless with AWS Lambda in C#.pptxBrandon Minnick, MBA
 
My self introduction to know others abut me
My self  introduction to know others abut meMy self  introduction to know others abut me
My self introduction to know others abut meManoj Prabakar B
 
Microsoft Azure News - Feb 2024
Microsoft Azure News - Feb 2024Microsoft Azure News - Feb 2024
Microsoft Azure News - Feb 2024Daniel Toomey
 
21ST CENTURY LITERACY FROM TRADITIONAL TO MODERN
21ST CENTURY LITERACY FROM TRADITIONAL TO MODERN21ST CENTURY LITERACY FROM TRADITIONAL TO MODERN
21ST CENTURY LITERACY FROM TRADITIONAL TO MODERNRonnelBaroc
 
Q1 Memory Fabric Forum: SMART CXL Product Lineup
Q1 Memory Fabric Forum: SMART CXL Product LineupQ1 Memory Fabric Forum: SMART CXL Product Lineup
Q1 Memory Fabric Forum: SMART CXL Product LineupMemory Fabric Forum
 
Artificial-Intelligence-in-Marketing-Data.pdf
Artificial-Intelligence-in-Marketing-Data.pdfArtificial-Intelligence-in-Marketing-Data.pdf
Artificial-Intelligence-in-Marketing-Data.pdfIsidro Navarro
 
Z-Wave Fan coil Thermostat Heltun_HE-HT01_User_Manual.pdf
Z-Wave Fan coil Thermostat Heltun_HE-HT01_User_Manual.pdfZ-Wave Fan coil Thermostat Heltun_HE-HT01_User_Manual.pdf
Z-Wave Fan coil Thermostat Heltun_HE-HT01_User_Manual.pdfDomotica daVinci
 
Manual Eurotronic Thermostatic Valve Comry Z-Wave
Manual Eurotronic Thermostatic Valve Comry Z-WaveManual Eurotronic Thermostatic Valve Comry Z-Wave
Manual Eurotronic Thermostatic Valve Comry Z-WaveDomotica daVinci
 

Recently uploaded (20)

AWS reInvent 2023 recaps from Chicago AWS user group
AWS reInvent 2023 recaps from Chicago AWS user groupAWS reInvent 2023 recaps from Chicago AWS user group
AWS reInvent 2023 recaps from Chicago AWS user group
 
5 Tech Trend to Notice in ESG Landscape- 47Billion
5 Tech Trend to Notice in ESG Landscape- 47Billion5 Tech Trend to Notice in ESG Landscape- 47Billion
5 Tech Trend to Notice in ESG Landscape- 47Billion
 
5 Things You Shouldn’t Do at Salesforce World Tour Sydney 2024!
5 Things You Shouldn’t Do at Salesforce World Tour Sydney 2024!5 Things You Shouldn’t Do at Salesforce World Tour Sydney 2024!
5 Things You Shouldn’t Do at Salesforce World Tour Sydney 2024!
 
LLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdf
LLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdfLLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdf
LLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdf
 
How we think about an advisor tech stack
How we think about an advisor tech stackHow we think about an advisor tech stack
How we think about an advisor tech stack
 
DNA LIGASE BIOTECHNOLOGY BIOLOGY STUDY OF LIFE
DNA LIGASE BIOTECHNOLOGY BIOLOGY STUDY OF LIFEDNA LIGASE BIOTECHNOLOGY BIOLOGY STUDY OF LIFE
DNA LIGASE BIOTECHNOLOGY BIOLOGY STUDY OF LIFE
 
Building Products That Think- Bhaskaran Srinivasan & Ashish Gupta
Building Products That Think- Bhaskaran Srinivasan & Ashish GuptaBuilding Products That Think- Bhaskaran Srinivasan & Ashish Gupta
Building Products That Think- Bhaskaran Srinivasan & Ashish Gupta
 
Zi-Stick UBS Dongle ZIgbee from Aeotec manual
Zi-Stick UBS Dongle ZIgbee from  Aeotec manualZi-Stick UBS Dongle ZIgbee from  Aeotec manual
Zi-Stick UBS Dongle ZIgbee from Aeotec manual
 
"Running Open-Source LLM models on Kubernetes", Volodymyr Tsap
"Running Open-Source LLM models on Kubernetes",  Volodymyr Tsap"Running Open-Source LLM models on Kubernetes",  Volodymyr Tsap
"Running Open-Source LLM models on Kubernetes", Volodymyr Tsap
 
zigbee motion sensor user manual NAS-PD07B2.pdf
zigbee motion sensor user manual NAS-PD07B2.pdfzigbee motion sensor user manual NAS-PD07B2.pdf
zigbee motion sensor user manual NAS-PD07B2.pdf
 
Power of 2024 - WITforce Odyssey.pptx.pdf
Power of 2024 - WITforce Odyssey.pptx.pdfPower of 2024 - WITforce Odyssey.pptx.pdf
Power of 2024 - WITforce Odyssey.pptx.pdf
 
Q1 Memory Fabric Forum: Intel Enabling Compute Express Link (CXL)
Q1 Memory Fabric Forum: Intel Enabling Compute Express Link (CXL)Q1 Memory Fabric Forum: Intel Enabling Compute Express Link (CXL)
Q1 Memory Fabric Forum: Intel Enabling Compute Express Link (CXL)
 
Introduction to Serverless with AWS Lambda in C#.pptx
Introduction to Serverless with AWS Lambda in C#.pptxIntroduction to Serverless with AWS Lambda in C#.pptx
Introduction to Serverless with AWS Lambda in C#.pptx
 
My self introduction to know others abut me
My self  introduction to know others abut meMy self  introduction to know others abut me
My self introduction to know others abut me
 
Microsoft Azure News - Feb 2024
Microsoft Azure News - Feb 2024Microsoft Azure News - Feb 2024
Microsoft Azure News - Feb 2024
 
21ST CENTURY LITERACY FROM TRADITIONAL TO MODERN
21ST CENTURY LITERACY FROM TRADITIONAL TO MODERN21ST CENTURY LITERACY FROM TRADITIONAL TO MODERN
21ST CENTURY LITERACY FROM TRADITIONAL TO MODERN
 
Q1 Memory Fabric Forum: SMART CXL Product Lineup
Q1 Memory Fabric Forum: SMART CXL Product LineupQ1 Memory Fabric Forum: SMART CXL Product Lineup
Q1 Memory Fabric Forum: SMART CXL Product Lineup
 
Artificial-Intelligence-in-Marketing-Data.pdf
Artificial-Intelligence-in-Marketing-Data.pdfArtificial-Intelligence-in-Marketing-Data.pdf
Artificial-Intelligence-in-Marketing-Data.pdf
 
Z-Wave Fan coil Thermostat Heltun_HE-HT01_User_Manual.pdf
Z-Wave Fan coil Thermostat Heltun_HE-HT01_User_Manual.pdfZ-Wave Fan coil Thermostat Heltun_HE-HT01_User_Manual.pdf
Z-Wave Fan coil Thermostat Heltun_HE-HT01_User_Manual.pdf
 
Manual Eurotronic Thermostatic Valve Comry Z-Wave
Manual Eurotronic Thermostatic Valve Comry Z-WaveManual Eurotronic Thermostatic Valve Comry Z-Wave
Manual Eurotronic Thermostatic Valve Comry Z-Wave
 

Cross-domain complementary learning with synthetic data for multi-person part segmentation

  • 1. Cross-domain Complementary Learning with Synthetic Data for Multi-Person Part Segmentation Kevin Lin, Lijuan Wang, Kun Luo, Yinpeng Chen, Zicheng Liu, Ming-Ting Sun University of Washington, Seattle Microsoft, Redmond International Conference on Computer Vision (ICCV), Demonstration, 2019 1
  • 2. Outline • Introduction • Related works • Proposed method • Experiments • On-going work and Conclusion 2
  • 3. Human part segmentation • Human part segmentation aims at partitioning persons in the image to multiple semantically consistent regions. Typically 14 parts: Head, torso, left upper-arm, right upper-arm, left lower-arm, right lower- arm, left hand, right hand, left thigh, right thigh, left shank, right shank, left foot, right foot Input Image Part Segmentation 3
  • 4. Challenges • Training data labeling in pixel-level is very expensive and labor intensive. 4
  • 5. Previous works • People have been exploring synthetic data as an alternative. • They trained deep CNN using the synthetic data. Samples of the synthetic training data and the synthetic labels [CVPR17] 5
  • 6. Previous works Their method works well only on the well-controlled, single-person scenario. Learning from Synthetic Humans, CVPR 2017 Input images Output results 6
  • 7. The domain gap • The discrepancy of pixel-value distributions between the synthetic and real data makes transferring the knowledge from the synthetic to real domain challenging. Synthetic image Real images 7
  • 8. Related works on street-view segmentation • People are also trying to use graphics simulation for training a segmentation model for street-view images. • They also observe the domain-gap issue. Zhang et al, Fully Convolutional Adaptation Networks for Semantic Segmentation, CVPR 2018. 8
  • 9. Related works on street-view segmentation • Previous studies tried to address the domain-gap issue by using adversarial training. • They use a discriminator to distinguish whether the input is from the source or target domain. [Tsai et al, ICCV2019], [Tsai et al, CVPR 2018], [Ren et al, CVPR2018], [Tzeng et al, CVPR2017], [Ganin et al, ICML2015] Graphics simulation Real-world images 9
  • 10. Challenges • Can we learn human part segmentation without data labeling? • How to learn human part segmentation from graphics simulations, and make the resulting model work well on real world scenario? We propose a new approach, named cross-domain complementary learning (CDCL) to address the challenges. 10
  • 11. Our multi-person synthetic data • We create a new multi-person synthetic dataset which contains multiple persons performing various actions in a 3D room. 11
  • 12. The idea •We observe that real and synthetic humans both have a skeleton (pose) representation. 12
  • 13. Proposed method • We propose to bridge the domains with skeletons and learn part segmentation from synthetic data. 13
  • 14. Proposed network: Module 1 Backbone (ResNet101) Part Affinity Fields Keypoint Maps Skeletons Real Inputs Head networks The network architecture is similar to “Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields,” in CVPR 2017. 14
  • 15. Proposed network: Module 2 Backbone (ResNet101) Head networks Keypoint Maps Body Part Maps Skeletons Body Part Segmentation Synthetic Inputs Part Affinity Fields 15
  • 16. Two modules are trained interchangeably Backbone (ResNet101) Head networks Keypoint Maps Body Part Maps Skeletons Body Part Segmentation Backbone (ResNet101) Part Affinity Fields Keypoint Maps Skeletons Parameter Sharing Synthetic Inputs Real Inputs Head networks Module 2 Module 1 Part Affinity Fields 16
  • 17. • Intersection over Union (IoU) is one of the most commonly used metrics in semantic segmentation. • IoU is calculated for each body part category separately. • We average over all categories to provide a mean IoU. Evaluation metric IoU = 𝐴𝑟𝑒𝑎 𝑜𝑓 𝐼𝑛𝑡𝑒𝑟𝑠𝑒𝑐𝑡𝑖𝑜𝑛 𝐴𝑒𝑟𝑎 𝑜𝑓 𝑈𝑛𝑖𝑜𝑛 𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛 ∩ 𝐺𝑟𝑜𝑢𝑛𝑑 𝑡𝑟𝑢𝑡ℎ 𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛 ∪ 𝐺𝑟𝑜𝑢𝑛𝑑 𝑡𝑟𝑢𝑡ℎ 17
  • 18. Evaluation benchmarks • Pascal-Person-Parts dataset • 1716 training images • 1817 test images • COCO-DensePose dataset • 26151 training images • 1508 test images 18
  • 19. Comparison on Pascal and COCO (mIOU, %) Synthetic Only Adversarial Training Fang et al CVPR18 OursChen et al TPAMI18 Gong et al CVPR17 Ours + Real part labels Use real part labels Use additional real part labels Ideal 19
  • 20. Comparison on Pascal and COCO (mIOU, %) Synthetic Only Adversarial Training Fang et al CVPR18 OursChen et al TPAMI18 Gong et al CVPR17 Ours + Real part labels Use real part labels Use additional real part labels Ideal Performance Gap 20
  • 21. Comparison on Pascal and COCO (mIOU, %) Synthetic Only Adversarial Training Fang et al CVPR18 OursChen et al TPAMI18 Gong et al CVPR17 Ours + Real part labels Use real part labels Use additional real part labels Ideal Performance Gap 21
  • 22. Comparison on Pascal and COCO (mIOU, %) Synthetic Only Adversarial Training Fang et al CVPR18 OursChen et al TPAMI18 Gong et al CVPR17 Ours + Real part labels Use real part labels Use additional real part labels Ideal Relax labeling requirements! 22
  • 23. Comparison on Pascal and COCO (mIOU, %) Synthetic Only Adversarial Training Fang et al CVPR18 OursChen et al TPAMI18 Gong et al CVPR17 Ours + Real part labels Use real part labels Use additional real part labels Ideal Our performance upper bound 23
  • 24. Qualitative comparison Training with Synthetic Data Only [CVPR17] Ours 24
  • 25. Qualitative comparison Domain Adaptation with Adversarial Training [CVPR18] Ours 25
  • 27. Synthetic training data analysis 27
  • 28. Qualitative comparison [1] Learning from Synthetic Humans, CVPR17. 28
  • 29. Qualitative comparison [1] Learning from Synthetic Humans, CVPR17. 29
  • 30. General approach • Our proposed cross-domain training approach is general and can be extended to other applications, such as novel keypoint detection. We can simply generate new labels on the synthetic data 30
  • 31. Novel keypoint detection • In some applications, we need to detect other keypoints (e.g., joints) such as hand tips, toes, pelvis, spine. • We create novel keypoints using the graphics simulator and train our model to detect new human skeleton including those on the hands and feet. The definition of our newly created novel keypoints 31
  • 33. Conclusion • We discover human pose is very effective to bridge the real and synthetic domains for multi-person part segmentation. • We introduce an effective framework to leverage information in both real and synthetic images for multi-person part segmentation. • Our method can be extended to generate labels for keypoints such as those on hands and feet in real images without human labeling. 33
  • 34. On-going work and future directions • Reconstruct 3D human mesh from a single image without ground truth training labels 34
  • 35. On-going work and future directions • Training data labeling for 3D body shape is very expensive. First stage: Ask workers to label parts Second stage: Ask workers to label the corresponding points on 3D human model Sampled points: uniformly sampled points within the part Guler et al, “DensePose: Learning image-to-surface correspondence,” CVPR 2018. 35
  • 36. On-going work and future directions • We plan to explore different approaches to learn human 3D body shape from graphics simulations. 36