SlideShare a Scribd company logo
1 of 15
Download to read offline
July 9 and 16, 2020
3D human body modeling from RGB images
Arithmer R3 Div. - Enrico Rinaldi
Human Pose Estimation WG
2
Enrico Rinaldi (Ph.D.)
● Education
○ Bachelor of Science in Physics from the University of Milan
○ Master of Science in Theoretical Physics from the University of Milan
○ Ph.D. in Theoretical Particle Physics from the University of Edinburgh
■ Computational Physics and (Big) Data Analysis
■ Particle Physics with Composite Higgs, Dark Matter, and Extra Dimensions
■ Supported by Scottish Universities Physics Alliance (SUPA) fellowship and Japanese Society for the Promotion of
Science (JSPS) fellowship at the Kobayashi-Maskawa Institute of Nagoya University
● Associate Researcher at the Lawrence Livermore National Laboratory
○ High-Performance Computing simulations of particle physics, nuclear physics and string theory
○ Markov Chain Monte Carlo sampling and Optimization problems
● Special Postdoctoral Fellow at the RIKEN BNL Research Center
○ Junior PI of project on numerical simulations for composite dark matter theories
○ GPU computing for HPC simulations of nuclear physics properties
○ Leveraging machine learning techniques to accelerate physics discoveries
● R&D Researcher and Engineer at Arithmer Inc.
○ Research on Geometric Learning, Graph Learning, 3D perception, Computer Vision
○ Applications to robotics, automated measurements systems, optimization problems
Outline
● Introduction
○ Definition of the problem
○ Challenges
○ Common approaches
● Statistical body models
○ Definition of the setup
○ Aim of the methods
○ Typical solutions
○ SMPL
● 3D human pose
○ Challenges
○ 2D solutions
○ SMPLify
● Conclusions
The data
4
SHAPE POSE
The Model
5
Shape linear coefficients
Vector β
multiplies the principal
components of the learned
shape parameters that
modify the N vertices of the
mesh
Pose vector
Vector θ
represents the 3D angles
between the kinematic tree
of the K=23 joints in the
skeleton
Parameters of the model related
to pose are trained first:
Parameters of the model related
to shape are trained after:
Joint positions are defined as a function of the body shape:
SMPL
SMPL stands for Skinned Multi-Person Linear model.
It is a “statistical” body model, which means it is learned from data.
It captures 2 aspects of the human body:
1. Shape
2. Pose
6
SHAPE
The shape of a male and female body is learned from 3D
data in the CAESAR dataset
Shape is represented by a 3D mesh
POSE
The pose of a male and female body is learned from 3D data
in the FAUST dataset
Pose is represented by a skeleton.
Example: Joint regression
7
Example: Inference
8
Start from a mesh template T and
blend weights W
The weights represent how much
vertices are affected by joint movement
Modify the template T using the shape
vector and apply the joint regressor based
on the shape vector.
The mesh and joints change with shape.
Add the pose vector and the pose
blend shapes that will modify the mesh
vertices based on the location of the
joints
Apply Linear Blend
Skinning with the new
template, joints and
weights.
SMPLify
The goal of SMPLify is to automatically estimate both 3D pose and 3D
body shape from a single RGB image.
9
Remember:
3D pose from 2D
coordinates is an
ambiguous ill-defined
problem.
The missing depth
information makes it
difficult to place the
joints in the
perpendicular direction.
Ingredients
Input: RGB image (pixel coordinates and pixel values) 2D
Intermediate representations:
• Joint locations: 2D coordinates of 23 joints
• Body model: SMPL, it is just a function
• Capsules: each bone is replaced by a cylinder
Output: Posed body model (mesh vertices) 3D
Method: non-linear least-squares optimization using Powell’s dogleg
10
Objective functions
11
Joint-based error function: to minimize this term we require that the joint position in 3D is close to the estimated joint position in
the image when projected to 2D with a perspective camera projection with parameters K.
Three pose priors that minimize the following: unnatural bending of knees and elbows, improbable poses, and interpenetration
of capsules (avoid intersecting body parts)
A shape prior to keep the shape similar to the description given by the principal components used in the SMPL model
Example: comparison
12
Input OutputOther approaches without body shape
References
References
Models
● SMPL
○ Project website: https://smpl.is.tue.mpg.de/
● SMPLify
○ Project website: http://smplify.is.tue.mpg.de/
Datasets
● FAUST
○ Project website: http://faust.is.tue.mpg.de/
● CAESAR
○ Project website: http://store.sae.org/caesar/
My ongoing notes on Notion:
https://www.notion.so/erinaldi/Fundamentals-of-riggin
g-eb8acd313a444b6999ef1b0a7a13b892
Future discussion
A few links for modern motion capture papers:
• DeepCap: DeepCap: Monocular Human Performance Capture Using Weak Supervision, CVPR 2020
• EventCap: EventCap: Monocular 3D Capture of High-Speed Human Motions using an Event Camera, CVPR 2020
• HandCap: Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data, CVPR 2020
• StyleRig: http://gvv.mpi-inf.mpg.de/projects/StyleRig/
• XNect: XNect: Real-time Multi-Person 3D Motion Capture with a Single RGB Camera, SIGGRAPH 2020
• LiveCap: LiveCap: Real-time Human Performance Capture from Monocular Video, ToG 2019
• MonoPerfCap: MonoPerfCap: Human Performance Capture from Monocular Video, TOG 2018
14
15

More Related Content

What's hot

Image classification using CNN
Image classification using CNNImage classification using CNN
Image classification using CNNNoura Hussein
 
Articulated human pose estimation by deep learning
Articulated human pose estimation by deep learningArticulated human pose estimation by deep learning
Articulated human pose estimation by deep learningWei Yang
 
Motion planning and controlling algorithm for grasping and manipulating movin...
Motion planning and controlling algorithm for grasping and manipulating movin...Motion planning and controlling algorithm for grasping and manipulating movin...
Motion planning and controlling algorithm for grasping and manipulating movin...ijscai
 
Recent Breakthroughs in AI + Learning Visual-Linguistic Representation in the...
Recent Breakthroughs in AI + Learning Visual-Linguistic Representation in the...Recent Breakthroughs in AI + Learning Visual-Linguistic Representation in the...
Recent Breakthroughs in AI + Learning Visual-Linguistic Representation in the...Sangmin Woo
 
Semantic Segmentation on Satellite Imagery
Semantic Segmentation on Satellite ImagerySemantic Segmentation on Satellite Imagery
Semantic Segmentation on Satellite ImageryRAHUL BHOJWANI
 
Object classification using CNN & VGG16 Model (Keras and Tensorflow)
Object classification using CNN & VGG16 Model (Keras and Tensorflow) Object classification using CNN & VGG16 Model (Keras and Tensorflow)
Object classification using CNN & VGG16 Model (Keras and Tensorflow) Lalit Jain
 
ViT (Vision Transformer) Review [CDM]
ViT (Vision Transformer) Review [CDM]ViT (Vision Transformer) Review [CDM]
ViT (Vision Transformer) Review [CDM]Dongmin Choi
 
Improving access to satellite imagery with Cloud computing
Improving access to satellite imagery with Cloud computingImproving access to satellite imagery with Cloud computing
Improving access to satellite imagery with Cloud computingRAHUL BHOJWANI
 
Introduction to 3D Computer Vision and Differentiable Rendering
Introduction to 3D Computer Vision and Differentiable RenderingIntroduction to 3D Computer Vision and Differentiable Rendering
Introduction to 3D Computer Vision and Differentiable RenderingPreferred Networks
 
Image Classification using deep learning
Image Classification using deep learning Image Classification using deep learning
Image Classification using deep learning Asma-AH
 
Deep Learning and Automatic Differentiation from Theano to PyTorch
Deep Learning and Automatic Differentiation from Theano to PyTorchDeep Learning and Automatic Differentiation from Theano to PyTorch
Deep Learning and Automatic Differentiation from Theano to PyTorchinside-BigData.com
 
Deep Neural Networks Presentation
Deep Neural Networks PresentationDeep Neural Networks Presentation
Deep Neural Networks PresentationBohdan Klimenko
 
Neuromation.io AI Ukraine Presentation
Neuromation.io AI Ukraine PresentationNeuromation.io AI Ukraine Presentation
Neuromation.io AI Ukraine PresentationBohdan Klimenko
 
Transformer in Vision
Transformer in VisionTransformer in Vision
Transformer in VisionSangmin Woo
 
Sequential Reptile_Inter-Task Gradient Alignment for Multilingual Learning
Sequential Reptile_Inter-Task Gradient Alignment for Multilingual LearningSequential Reptile_Inter-Task Gradient Alignment for Multilingual Learning
Sequential Reptile_Inter-Task Gradient Alignment for Multilingual LearningMLAI2
 
Vision Transformer(ViT) / An Image is Worth 16*16 Words: Transformers for Ima...
Vision Transformer(ViT) / An Image is Worth 16*16 Words: Transformers for Ima...Vision Transformer(ViT) / An Image is Worth 16*16 Words: Transformers for Ima...
Vision Transformer(ViT) / An Image is Worth 16*16 Words: Transformers for Ima...changedaeoh
 
Accelerated Logistic Regression on GPU(s)
Accelerated Logistic Regression on GPU(s)Accelerated Logistic Regression on GPU(s)
Accelerated Logistic Regression on GPU(s)RAHUL BHOJWANI
 
Review: Incremental Few-shot Instance Segmentation [CDM]
Review: Incremental Few-shot Instance Segmentation [CDM]Review: Incremental Few-shot Instance Segmentation [CDM]
Review: Incremental Few-shot Instance Segmentation [CDM]Dongmin Choi
 
Introduction to Graph Neural Networks: Basics and Applications - Katsuhiko Is...
Introduction to Graph Neural Networks: Basics and Applications - Katsuhiko Is...Introduction to Graph Neural Networks: Basics and Applications - Katsuhiko Is...
Introduction to Graph Neural Networks: Basics and Applications - Katsuhiko Is...Preferred Networks
 

What's hot (20)

Image classification using CNN
Image classification using CNNImage classification using CNN
Image classification using CNN
 
Articulated human pose estimation by deep learning
Articulated human pose estimation by deep learningArticulated human pose estimation by deep learning
Articulated human pose estimation by deep learning
 
Motion planning and controlling algorithm for grasping and manipulating movin...
Motion planning and controlling algorithm for grasping and manipulating movin...Motion planning and controlling algorithm for grasping and manipulating movin...
Motion planning and controlling algorithm for grasping and manipulating movin...
 
Recent Breakthroughs in AI + Learning Visual-Linguistic Representation in the...
Recent Breakthroughs in AI + Learning Visual-Linguistic Representation in the...Recent Breakthroughs in AI + Learning Visual-Linguistic Representation in the...
Recent Breakthroughs in AI + Learning Visual-Linguistic Representation in the...
 
Semantic Segmentation on Satellite Imagery
Semantic Segmentation on Satellite ImagerySemantic Segmentation on Satellite Imagery
Semantic Segmentation on Satellite Imagery
 
Object classification using CNN & VGG16 Model (Keras and Tensorflow)
Object classification using CNN & VGG16 Model (Keras and Tensorflow) Object classification using CNN & VGG16 Model (Keras and Tensorflow)
Object classification using CNN & VGG16 Model (Keras and Tensorflow)
 
998-isvc16
998-isvc16998-isvc16
998-isvc16
 
ViT (Vision Transformer) Review [CDM]
ViT (Vision Transformer) Review [CDM]ViT (Vision Transformer) Review [CDM]
ViT (Vision Transformer) Review [CDM]
 
Improving access to satellite imagery with Cloud computing
Improving access to satellite imagery with Cloud computingImproving access to satellite imagery with Cloud computing
Improving access to satellite imagery with Cloud computing
 
Introduction to 3D Computer Vision and Differentiable Rendering
Introduction to 3D Computer Vision and Differentiable RenderingIntroduction to 3D Computer Vision and Differentiable Rendering
Introduction to 3D Computer Vision and Differentiable Rendering
 
Image Classification using deep learning
Image Classification using deep learning Image Classification using deep learning
Image Classification using deep learning
 
Deep Learning and Automatic Differentiation from Theano to PyTorch
Deep Learning and Automatic Differentiation from Theano to PyTorchDeep Learning and Automatic Differentiation from Theano to PyTorch
Deep Learning and Automatic Differentiation from Theano to PyTorch
 
Deep Neural Networks Presentation
Deep Neural Networks PresentationDeep Neural Networks Presentation
Deep Neural Networks Presentation
 
Neuromation.io AI Ukraine Presentation
Neuromation.io AI Ukraine PresentationNeuromation.io AI Ukraine Presentation
Neuromation.io AI Ukraine Presentation
 
Transformer in Vision
Transformer in VisionTransformer in Vision
Transformer in Vision
 
Sequential Reptile_Inter-Task Gradient Alignment for Multilingual Learning
Sequential Reptile_Inter-Task Gradient Alignment for Multilingual LearningSequential Reptile_Inter-Task Gradient Alignment for Multilingual Learning
Sequential Reptile_Inter-Task Gradient Alignment for Multilingual Learning
 
Vision Transformer(ViT) / An Image is Worth 16*16 Words: Transformers for Ima...
Vision Transformer(ViT) / An Image is Worth 16*16 Words: Transformers for Ima...Vision Transformer(ViT) / An Image is Worth 16*16 Words: Transformers for Ima...
Vision Transformer(ViT) / An Image is Worth 16*16 Words: Transformers for Ima...
 
Accelerated Logistic Regression on GPU(s)
Accelerated Logistic Regression on GPU(s)Accelerated Logistic Regression on GPU(s)
Accelerated Logistic Regression on GPU(s)
 
Review: Incremental Few-shot Instance Segmentation [CDM]
Review: Incremental Few-shot Instance Segmentation [CDM]Review: Incremental Few-shot Instance Segmentation [CDM]
Review: Incremental Few-shot Instance Segmentation [CDM]
 
Introduction to Graph Neural Networks: Basics and Applications - Katsuhiko Is...
Introduction to Graph Neural Networks: Basics and Applications - Katsuhiko Is...Introduction to Graph Neural Networks: Basics and Applications - Katsuhiko Is...
Introduction to Graph Neural Networks: Basics and Applications - Katsuhiko Is...
 

Similar to 3D human body modeling from RGB images

Solving the Pose Ambiguity via a Simple Concentric Circle Constraint
Solving the Pose Ambiguity via a Simple Concentric Circle ConstraintSolving the Pose Ambiguity via a Simple Concentric Circle Constraint
Solving the Pose Ambiguity via a Simple Concentric Circle ConstraintDr. Amarjeet Singh
 
01 4941 10014-1-sm ediqbal
01 4941 10014-1-sm ediqbal01 4941 10014-1-sm ediqbal
01 4941 10014-1-sm ediqbalIAESIJEECS
 
An iterative morphological decomposition algorithm for reduction of skeleton ...
An iterative morphological decomposition algorithm for reduction of skeleton ...An iterative morphological decomposition algorithm for reduction of skeleton ...
An iterative morphological decomposition algorithm for reduction of skeleton ...ijcsit
 
Strength Analysis and Optimization Design about the key parts of the Robot
Strength Analysis and Optimization Design about the key parts of the RobotStrength Analysis and Optimization Design about the key parts of the Robot
Strength Analysis and Optimization Design about the key parts of the RobotIJRES Journal
 
4 tracking objects of deformable shapes
4 tracking objects of deformable shapes4 tracking objects of deformable shapes
4 tracking objects of deformable shapesprjpublications
 
2D Shape Reconstruction Based on Combined Skeleton-Boundary Features
2D Shape Reconstruction Based on Combined Skeleton-Boundary Features2D Shape Reconstruction Based on Combined Skeleton-Boundary Features
2D Shape Reconstruction Based on Combined Skeleton-Boundary FeaturesCSCJournals
 
4 tracking objects of deformable shapes (1)
4 tracking objects of deformable shapes (1)4 tracking objects of deformable shapes (1)
4 tracking objects of deformable shapes (1)prj_publication
 
4 tracking objects of deformable shapes
4 tracking objects of deformable shapes4 tracking objects of deformable shapes
4 tracking objects of deformable shapesprj_publication
 
4 tracking objects of deformable shapes
4 tracking objects of deformable shapes4 tracking objects of deformable shapes
4 tracking objects of deformable shapesprj_publication
 
Beam and Grid Sabin Presentation.pptx
Beam and Grid Sabin Presentation.pptxBeam and Grid Sabin Presentation.pptx
Beam and Grid Sabin Presentation.pptxSabin Acharya
 
hpe3d_report.pdf
hpe3d_report.pdfhpe3d_report.pdf
hpe3d_report.pdfgsrawat
 
VARIATIONAL MONTE-CARLO APPROACH FOR ARTICULATED OBJECT TRACKING
VARIATIONAL MONTE-CARLO APPROACH FOR ARTICULATED OBJECT TRACKINGVARIATIONAL MONTE-CARLO APPROACH FOR ARTICULATED OBJECT TRACKING
VARIATIONAL MONTE-CARLO APPROACH FOR ARTICULATED OBJECT TRACKINGcscpconf
 
3D Reconstruction from Multiple uncalibrated 2D Images of an Object
3D Reconstruction from Multiple uncalibrated 2D Images of an Object3D Reconstruction from Multiple uncalibrated 2D Images of an Object
3D Reconstruction from Multiple uncalibrated 2D Images of an ObjectAnkur Tyagi
 
Procedural modeling using autoencoder networks
Procedural modeling using autoencoder networksProcedural modeling using autoencoder networks
Procedural modeling using autoencoder networksShuhei Iitsuka
 
VARIATIONAL MONTE-CARLO APPROACH FOR ARTICULATED OBJECT TRACKING
VARIATIONAL MONTE-CARLO APPROACH FOR ARTICULATED OBJECT TRACKINGVARIATIONAL MONTE-CARLO APPROACH FOR ARTICULATED OBJECT TRACKING
VARIATIONAL MONTE-CARLO APPROACH FOR ARTICULATED OBJECT TRACKINGcsandit
 
A NOVEL APPROACH TO SMOOTHING ON 3D STRUCTURED ADAPTIVE MESH OF THE KINECT-BA...
A NOVEL APPROACH TO SMOOTHING ON 3D STRUCTURED ADAPTIVE MESH OF THE KINECT-BA...A NOVEL APPROACH TO SMOOTHING ON 3D STRUCTURED ADAPTIVE MESH OF THE KINECT-BA...
A NOVEL APPROACH TO SMOOTHING ON 3D STRUCTURED ADAPTIVE MESH OF THE KINECT-BA...cscpconf
 
Low complexity features for jpeg steganalysis using undecimated dct
Low complexity features for jpeg steganalysis using undecimated dctLow complexity features for jpeg steganalysis using undecimated dct
Low complexity features for jpeg steganalysis using undecimated dctPvrtechnologies Nellore
 
Curve Fitting - Linear Algebra
Curve Fitting - Linear AlgebraCurve Fitting - Linear Algebra
Curve Fitting - Linear AlgebraGowtham Cr
 

Similar to 3D human body modeling from RGB images (20)

Solving the Pose Ambiguity via a Simple Concentric Circle Constraint
Solving the Pose Ambiguity via a Simple Concentric Circle ConstraintSolving the Pose Ambiguity via a Simple Concentric Circle Constraint
Solving the Pose Ambiguity via a Simple Concentric Circle Constraint
 
01 4941 10014-1-sm ediqbal
01 4941 10014-1-sm ediqbal01 4941 10014-1-sm ediqbal
01 4941 10014-1-sm ediqbal
 
An iterative morphological decomposition algorithm for reduction of skeleton ...
An iterative morphological decomposition algorithm for reduction of skeleton ...An iterative morphological decomposition algorithm for reduction of skeleton ...
An iterative morphological decomposition algorithm for reduction of skeleton ...
 
Strength Analysis and Optimization Design about the key parts of the Robot
Strength Analysis and Optimization Design about the key parts of the RobotStrength Analysis and Optimization Design about the key parts of the Robot
Strength Analysis and Optimization Design about the key parts of the Robot
 
4 tracking objects of deformable shapes
4 tracking objects of deformable shapes4 tracking objects of deformable shapes
4 tracking objects of deformable shapes
 
2D Shape Reconstruction Based on Combined Skeleton-Boundary Features
2D Shape Reconstruction Based on Combined Skeleton-Boundary Features2D Shape Reconstruction Based on Combined Skeleton-Boundary Features
2D Shape Reconstruction Based on Combined Skeleton-Boundary Features
 
proj525
proj525proj525
proj525
 
4 tracking objects of deformable shapes (1)
4 tracking objects of deformable shapes (1)4 tracking objects of deformable shapes (1)
4 tracking objects of deformable shapes (1)
 
4 tracking objects of deformable shapes
4 tracking objects of deformable shapes4 tracking objects of deformable shapes
4 tracking objects of deformable shapes
 
4 tracking objects of deformable shapes
4 tracking objects of deformable shapes4 tracking objects of deformable shapes
4 tracking objects of deformable shapes
 
Beam and Grid Sabin Presentation.pptx
Beam and Grid Sabin Presentation.pptxBeam and Grid Sabin Presentation.pptx
Beam and Grid Sabin Presentation.pptx
 
hpe3d_report.pdf
hpe3d_report.pdfhpe3d_report.pdf
hpe3d_report.pdf
 
VARIATIONAL MONTE-CARLO APPROACH FOR ARTICULATED OBJECT TRACKING
VARIATIONAL MONTE-CARLO APPROACH FOR ARTICULATED OBJECT TRACKINGVARIATIONAL MONTE-CARLO APPROACH FOR ARTICULATED OBJECT TRACKING
VARIATIONAL MONTE-CARLO APPROACH FOR ARTICULATED OBJECT TRACKING
 
Parametric Human spine.
Parametric Human spine.Parametric Human spine.
Parametric Human spine.
 
3D Reconstruction from Multiple uncalibrated 2D Images of an Object
3D Reconstruction from Multiple uncalibrated 2D Images of an Object3D Reconstruction from Multiple uncalibrated 2D Images of an Object
3D Reconstruction from Multiple uncalibrated 2D Images of an Object
 
Procedural modeling using autoencoder networks
Procedural modeling using autoencoder networksProcedural modeling using autoencoder networks
Procedural modeling using autoencoder networks
 
VARIATIONAL MONTE-CARLO APPROACH FOR ARTICULATED OBJECT TRACKING
VARIATIONAL MONTE-CARLO APPROACH FOR ARTICULATED OBJECT TRACKINGVARIATIONAL MONTE-CARLO APPROACH FOR ARTICULATED OBJECT TRACKING
VARIATIONAL MONTE-CARLO APPROACH FOR ARTICULATED OBJECT TRACKING
 
A NOVEL APPROACH TO SMOOTHING ON 3D STRUCTURED ADAPTIVE MESH OF THE KINECT-BA...
A NOVEL APPROACH TO SMOOTHING ON 3D STRUCTURED ADAPTIVE MESH OF THE KINECT-BA...A NOVEL APPROACH TO SMOOTHING ON 3D STRUCTURED ADAPTIVE MESH OF THE KINECT-BA...
A NOVEL APPROACH TO SMOOTHING ON 3D STRUCTURED ADAPTIVE MESH OF THE KINECT-BA...
 
Low complexity features for jpeg steganalysis using undecimated dct
Low complexity features for jpeg steganalysis using undecimated dctLow complexity features for jpeg steganalysis using undecimated dct
Low complexity features for jpeg steganalysis using undecimated dct
 
Curve Fitting - Linear Algebra
Curve Fitting - Linear AlgebraCurve Fitting - Linear Algebra
Curve Fitting - Linear Algebra
 

More from Arithmer Inc.

コーディネートレコメンド
コーディネートレコメンドコーディネートレコメンド
コーディネートレコメンドArithmer Inc.
 
Arithmerソリューション紹介 流体予測システム
Arithmerソリューション紹介 流体予測システムArithmerソリューション紹介 流体予測システム
Arithmerソリューション紹介 流体予測システムArithmer Inc.
 
Weakly supervised semantic segmentation of 3D point cloud
Weakly supervised semantic segmentation of 3D point cloudWeakly supervised semantic segmentation of 3D point cloud
Weakly supervised semantic segmentation of 3D point cloudArithmer Inc.
 
Arithmer NLP 自然言語処理 ソリューション紹介
Arithmer NLP 自然言語処理 ソリューション紹介Arithmer NLP 自然言語処理 ソリューション紹介
Arithmer NLP 自然言語処理 ソリューション紹介Arithmer Inc.
 
Arithmer Robo Introduction
Arithmer Robo IntroductionArithmer Robo Introduction
Arithmer Robo IntroductionArithmer Inc.
 
Arithmer AIチャットボット
Arithmer AIチャットボットArithmer AIチャットボット
Arithmer AIチャットボットArithmer Inc.
 
Arithmer R3 Introduction
Arithmer R3 Introduction Arithmer R3 Introduction
Arithmer R3 Introduction Arithmer Inc.
 
Arithmer Inspection Introduction
Arithmer Inspection IntroductionArithmer Inspection Introduction
Arithmer Inspection IntroductionArithmer Inc.
 
全力解説!Transformer
全力解説!Transformer全力解説!Transformer
全力解説!TransformerArithmer Inc.
 
Arithmer NLP Introduction
Arithmer NLP IntroductionArithmer NLP Introduction
Arithmer NLP IntroductionArithmer Inc.
 
Introduction of Quantum Annealing and D-Wave Machines
Introduction of Quantum Annealing and D-Wave MachinesIntroduction of Quantum Annealing and D-Wave Machines
Introduction of Quantum Annealing and D-Wave MachinesArithmer Inc.
 
Arithmer OCR Introduction
Arithmer OCR IntroductionArithmer OCR Introduction
Arithmer OCR IntroductionArithmer Inc.
 
Arithmer Dynamics Introduction
Arithmer Dynamics Introduction Arithmer Dynamics Introduction
Arithmer Dynamics Introduction Arithmer Inc.
 
ArithmerDB Introduction
ArithmerDB IntroductionArithmerDB Introduction
ArithmerDB IntroductionArithmer Inc.
 
Summarizing videos with Attention
Summarizing videos with AttentionSummarizing videos with Attention
Summarizing videos with AttentionArithmer Inc.
 
Recommendation algorithm using reinforcement learning
Recommendation algorithm using reinforcement learningRecommendation algorithm using reinforcement learning
Recommendation algorithm using reinforcement learningArithmer Inc.
 
Survey on self supervised image segmentation
Survey on self supervised image segmentationSurvey on self supervised image segmentation
Survey on self supervised image segmentationArithmer Inc.
 
dataScienceofPhysics
dataScienceofPhysicsdataScienceofPhysics
dataScienceofPhysicsArithmer Inc.
 

More from Arithmer Inc. (20)

コーディネートレコメンド
コーディネートレコメンドコーディネートレコメンド
コーディネートレコメンド
 
Test for AI model
Test for AI modelTest for AI model
Test for AI model
 
最適化
最適化最適化
最適化
 
Arithmerソリューション紹介 流体予測システム
Arithmerソリューション紹介 流体予測システムArithmerソリューション紹介 流体予測システム
Arithmerソリューション紹介 流体予測システム
 
Weakly supervised semantic segmentation of 3D point cloud
Weakly supervised semantic segmentation of 3D point cloudWeakly supervised semantic segmentation of 3D point cloud
Weakly supervised semantic segmentation of 3D point cloud
 
Arithmer NLP 自然言語処理 ソリューション紹介
Arithmer NLP 自然言語処理 ソリューション紹介Arithmer NLP 自然言語処理 ソリューション紹介
Arithmer NLP 自然言語処理 ソリューション紹介
 
Arithmer Robo Introduction
Arithmer Robo IntroductionArithmer Robo Introduction
Arithmer Robo Introduction
 
Arithmer AIチャットボット
Arithmer AIチャットボットArithmer AIチャットボット
Arithmer AIチャットボット
 
Arithmer R3 Introduction
Arithmer R3 Introduction Arithmer R3 Introduction
Arithmer R3 Introduction
 
Arithmer Inspection Introduction
Arithmer Inspection IntroductionArithmer Inspection Introduction
Arithmer Inspection Introduction
 
全力解説!Transformer
全力解説!Transformer全力解説!Transformer
全力解説!Transformer
 
Arithmer NLP Introduction
Arithmer NLP IntroductionArithmer NLP Introduction
Arithmer NLP Introduction
 
Introduction of Quantum Annealing and D-Wave Machines
Introduction of Quantum Annealing and D-Wave MachinesIntroduction of Quantum Annealing and D-Wave Machines
Introduction of Quantum Annealing and D-Wave Machines
 
Arithmer OCR Introduction
Arithmer OCR IntroductionArithmer OCR Introduction
Arithmer OCR Introduction
 
Arithmer Dynamics Introduction
Arithmer Dynamics Introduction Arithmer Dynamics Introduction
Arithmer Dynamics Introduction
 
ArithmerDB Introduction
ArithmerDB IntroductionArithmerDB Introduction
ArithmerDB Introduction
 
Summarizing videos with Attention
Summarizing videos with AttentionSummarizing videos with Attention
Summarizing videos with Attention
 
Recommendation algorithm using reinforcement learning
Recommendation algorithm using reinforcement learningRecommendation algorithm using reinforcement learning
Recommendation algorithm using reinforcement learning
 
Survey on self supervised image segmentation
Survey on self supervised image segmentationSurvey on self supervised image segmentation
Survey on self supervised image segmentation
 
dataScienceofPhysics
dataScienceofPhysicsdataScienceofPhysics
dataScienceofPhysics
 

Recently uploaded

Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr BaganFwdays
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 3652toLead Limited
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxLoriGlavin3
 
Time Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsTime Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsNathaniel Shimoni
 
A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersA Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersNicole Novielli
 
What is Artificial Intelligence?????????
What is Artificial Intelligence?????????What is Artificial Intelligence?????????
What is Artificial Intelligence?????????blackmambaettijean
 
Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Manik S Magar
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxLoriGlavin3
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek SchlawackFwdays
 
Generative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersGenerative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersRaghuram Pandurangan
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024Lorenzo Miniero
 
Artificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptxArtificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptxhariprasad279825
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.Curtis Poe
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxLoriGlavin3
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsPixlogix Infotech
 
unit 4 immunoblotting technique complete.pptx
unit 4 immunoblotting technique complete.pptxunit 4 immunoblotting technique complete.pptx
unit 4 immunoblotting technique complete.pptxBkGupta21
 

Recently uploaded (20)

Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
 
Time Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsTime Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directions
 
A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersA Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software Developers
 
What is Artificial Intelligence?????????
What is Artificial Intelligence?????????What is Artificial Intelligence?????????
What is Artificial Intelligence?????????
 
Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
 
Generative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersGenerative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information Developers
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024
 
Artificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptxArtificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptx
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptx
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and Cons
 
unit 4 immunoblotting technique complete.pptx
unit 4 immunoblotting technique complete.pptxunit 4 immunoblotting technique complete.pptx
unit 4 immunoblotting technique complete.pptx
 

3D human body modeling from RGB images

  • 1. July 9 and 16, 2020 3D human body modeling from RGB images Arithmer R3 Div. - Enrico Rinaldi Human Pose Estimation WG
  • 2. 2 Enrico Rinaldi (Ph.D.) ● Education ○ Bachelor of Science in Physics from the University of Milan ○ Master of Science in Theoretical Physics from the University of Milan ○ Ph.D. in Theoretical Particle Physics from the University of Edinburgh ■ Computational Physics and (Big) Data Analysis ■ Particle Physics with Composite Higgs, Dark Matter, and Extra Dimensions ■ Supported by Scottish Universities Physics Alliance (SUPA) fellowship and Japanese Society for the Promotion of Science (JSPS) fellowship at the Kobayashi-Maskawa Institute of Nagoya University ● Associate Researcher at the Lawrence Livermore National Laboratory ○ High-Performance Computing simulations of particle physics, nuclear physics and string theory ○ Markov Chain Monte Carlo sampling and Optimization problems ● Special Postdoctoral Fellow at the RIKEN BNL Research Center ○ Junior PI of project on numerical simulations for composite dark matter theories ○ GPU computing for HPC simulations of nuclear physics properties ○ Leveraging machine learning techniques to accelerate physics discoveries ● R&D Researcher and Engineer at Arithmer Inc. ○ Research on Geometric Learning, Graph Learning, 3D perception, Computer Vision ○ Applications to robotics, automated measurements systems, optimization problems
  • 3. Outline ● Introduction ○ Definition of the problem ○ Challenges ○ Common approaches ● Statistical body models ○ Definition of the setup ○ Aim of the methods ○ Typical solutions ○ SMPL ● 3D human pose ○ Challenges ○ 2D solutions ○ SMPLify ● Conclusions
  • 5. The Model 5 Shape linear coefficients Vector β multiplies the principal components of the learned shape parameters that modify the N vertices of the mesh Pose vector Vector θ represents the 3D angles between the kinematic tree of the K=23 joints in the skeleton Parameters of the model related to pose are trained first: Parameters of the model related to shape are trained after: Joint positions are defined as a function of the body shape:
  • 6. SMPL SMPL stands for Skinned Multi-Person Linear model. It is a “statistical” body model, which means it is learned from data. It captures 2 aspects of the human body: 1. Shape 2. Pose 6 SHAPE The shape of a male and female body is learned from 3D data in the CAESAR dataset Shape is represented by a 3D mesh POSE The pose of a male and female body is learned from 3D data in the FAUST dataset Pose is represented by a skeleton.
  • 8. Example: Inference 8 Start from a mesh template T and blend weights W The weights represent how much vertices are affected by joint movement Modify the template T using the shape vector and apply the joint regressor based on the shape vector. The mesh and joints change with shape. Add the pose vector and the pose blend shapes that will modify the mesh vertices based on the location of the joints Apply Linear Blend Skinning with the new template, joints and weights.
  • 9. SMPLify The goal of SMPLify is to automatically estimate both 3D pose and 3D body shape from a single RGB image. 9 Remember: 3D pose from 2D coordinates is an ambiguous ill-defined problem. The missing depth information makes it difficult to place the joints in the perpendicular direction.
  • 10. Ingredients Input: RGB image (pixel coordinates and pixel values) 2D Intermediate representations: • Joint locations: 2D coordinates of 23 joints • Body model: SMPL, it is just a function • Capsules: each bone is replaced by a cylinder Output: Posed body model (mesh vertices) 3D Method: non-linear least-squares optimization using Powell’s dogleg 10
  • 11. Objective functions 11 Joint-based error function: to minimize this term we require that the joint position in 3D is close to the estimated joint position in the image when projected to 2D with a perspective camera projection with parameters K. Three pose priors that minimize the following: unnatural bending of knees and elbows, improbable poses, and interpenetration of capsules (avoid intersecting body parts) A shape prior to keep the shape similar to the description given by the principal components used in the SMPL model
  • 12. Example: comparison 12 Input OutputOther approaches without body shape
  • 13. References References Models ● SMPL ○ Project website: https://smpl.is.tue.mpg.de/ ● SMPLify ○ Project website: http://smplify.is.tue.mpg.de/ Datasets ● FAUST ○ Project website: http://faust.is.tue.mpg.de/ ● CAESAR ○ Project website: http://store.sae.org/caesar/ My ongoing notes on Notion: https://www.notion.so/erinaldi/Fundamentals-of-riggin g-eb8acd313a444b6999ef1b0a7a13b892
  • 14. Future discussion A few links for modern motion capture papers: • DeepCap: DeepCap: Monocular Human Performance Capture Using Weak Supervision, CVPR 2020 • EventCap: EventCap: Monocular 3D Capture of High-Speed Human Motions using an Event Camera, CVPR 2020 • HandCap: Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data, CVPR 2020 • StyleRig: http://gvv.mpi-inf.mpg.de/projects/StyleRig/ • XNect: XNect: Real-time Multi-Person 3D Motion Capture with a Single RGB Camera, SIGGRAPH 2020 • LiveCap: LiveCap: Real-time Human Performance Capture from Monocular Video, ToG 2019 • MonoPerfCap: MonoPerfCap: Human Performance Capture from Monocular Video, TOG 2018 14
  • 15. 15