SlideShare a Scribd company logo
Bridging the gap between 2D and 3D
with Deep Learning
Evgeny Burnaev (PhD) <e.burnaev@skoltech.ru>
assoc. prof. Skoltech
Alexandr Notchenko <a.notchenko@skoltech.ru>
PhD student
[1]
ImageNet top-5 error over the years
- Deep learning based methods
- Feature based methods
- human performance
Supervised Deep Learning data
Type
2D Image classification,
detection segmentation
Pose Estimation
Supervision
class label , object detection box,
segmentation contours
Structure of “skeleton” on image
But world is in 3D
3D deep learning is gaining popularity
Workshops:
● Deep Learning for Robotic Vision Workshop
CVPR 2017
● Geometry Meets Deep Learning ECCV 2016
● 3D Deep Learning Workshop @ NIPS 2016
● Large Scale 3D Data: Acquisition, Modelling
and Analysis CVPR 2016
● 3D from a Single Image CVPR 2015
Google Scholar when searched for "3D" "Deep
Learning" returns:
year # articles
2012 410
2013 627
2014 1210
2015 2570
2016 5440
Representation of 3D data for Deep Learning
Method Pros (+) Cons (-)
Many 2D projections sustain surface texture,
There is a lot of 2D DL methods
Redundant representation,
vulnerable to optic illusions
Voxels simple, can be sparse, has
volumetric properties
losing surface properties
Point Cloud Can be sparse losing surface properties and
volumetric properties
2.5D images Cheap measurement devices,
senses depth
self occlusion of bodies in a
scene, a lot of Noise in
measurements
[6]
[2]
3D shape as dense Point Cloud
Learning Rich Features from RGB-D Images for
Object Detection and Segmentation
[10]
Latest development in
SLAM family of methods
LSD-SLAM (Large-Scale Direct Monocular Simultaneous Localization and Mapping)
[5]
LSD-SLAM - direct (feature-less) monocular SLAM
ElasticFusion
ElasticFusion - DenseSLAM without a pose-graph
[7]
Dynamic Fusion
The technique won the prestigious CVPR 2015 best paper award.
[9]
Problems of SLAM algorithms
● Don’t represent objects (only know surfaces)
● Mostly dense representation (requires a lot of data)
● Whole scene is one big surface, e.g. cannot separate different objects that
are close to each other.
3D Shape Retrieval
3D Design Phase
•
There exists massive storages with 3D CAD models, e.g. GrabCAD
Chairs Mechanical parts
3D Design Phase
•Designers spend about 60% of their time
searching for the right information
• Massive and complex CAD models are
usually disorderly archived in enterprises,
which makes design reuse a difficult task
3D Model retrieval can significantly shorten the product lifecycles
3D Shape-based Model Retrieval
•3D models are complex = No clear search rules
•The text-based search has its limitations: e.g. often 3D
models are poorly annotated
• There is some commercial software for 3D CAD modeling, e.g.
➢ Exalead OnePart by Dassault Systems,
➢ Geolus Search by Siemens PLM, and others
• However, used methods
➢ are time-consuming,
➢ are often based on hand-crafted descriptors,
➢ could be limited to a specific class of shapes,
➢ are not robust to scaling, rotations, etc.
Sparse 3D Convolutional Neural Networks for
Large-Scale Shape Retrieval
Alexandr Notchenko, Ermek Kapushev, Evgeny Burnaev
Presented at 3D Deep Learning Workshop at NIPS 2016
Sparsity of voxel representation
30^3 Voxels is already enough
to understand simple shape
But with texture information it
would be even easier
Sparsity for all classes of
ModelNet40 train dataset at
voxel resolution 40 is only
5.5%
Shape Retrieval
Precomputed
feature vector of
dataset.
(Vcar
, Vperson
,...)
Vplane
- feature vector
of plane
Sparse3DCNN
Query
Retrieved items
Cosine distance
Triplet loss
The representation can be efficiently learned by minimizing triplet loss.
Triplet is a set (a, p, n), where
● a - anchor object
● p - positive object that is similar to anchor object
● n - negative object that is not similar to anchor object
,
where is a margin parameter, and are distances between p and a and
n and a.
Our approach
● Use very large resolutions, and sparse representations.
● Used triplet learning for 3D shapes.
● Used Large Scale Shape Datasets ModelNet and ShapeNet.
Represent voxel shape as vector
Obligatory t-SNE
Conclusions
● For small datasets of shape or 3D sparse tensors voxels
can work.
● Voxels don’t scale for hundreds of “classes” and loose
texture information.
● Cannot encode complicated object domains.
Problems for next 5 years
Autonomous Vehicles
Augmented (Mixed) Reality
Robotics in human
environments
Robotic Control in Human Environments
Commodity sensors to create 2.5D images
Intel RealSense Series
Asus Xtion Pro
Microsoft Kinect v2
Structure Sensor
What they have in
common?
What they have in
common?
They require understanding the whole scene
Problem of “Holistic” Scene understanding
Lin D., Fidler S., Urtasun R. Holistic scene understanding for 3d object detection
with rgbd cameras //Proceedings of the IEEE International Conference on Computer
Vision. – 2013. – С. 1417-1424.
● Human environments often designed by humans
● A most of the objects are created by humans
● Context provides information by joint probability functions
● Textures caused by materials and therefore can explain a functions and
structure of an object
Problem of “Holistic” Scene understanding
Connecting 3 families of CV algorithms is inevitable
Learnable Computer
Vision Systems
(Deep Learning)
Geometric Computer Vision
(SLAMs)
Probabilistic Computer
Vision
(Bayesian methods)
Connecting 3 families of CV algorithms is inevitable
Learnable Computer
Vision Systems
(Deep Learning)
Geometric Computer Vision
(SLAMs)
Probabilistic Computer
Vision
(Bayesian methods)
Probabilistic
Inverse
Graphics
Probabilistic Inverse Graphics enables
● Takes into account setting information (shop: shelves and products | street: buildings,
cars, pedestrians)
● Make maximum likelihood estimates from data and model (or give directions on how
to reduce uncertainty the best way)
● Learns structure of objects (Materials and textures / 3D shape / intrinsic dynamics)
Thank you.
Alexandr Notchenko Ermek Kapushev Evgeny Burnaev
Citations and Links
1. Deep Learning NIPS’2015 Tutorial by Geoff Hinton, Yoshua Bengio & Yann LeCun
2. Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., & Xiao, J. (2015). 3D ShapeNets: A Deep Representation for Volumetric Shapes.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1912-1920).
3. C. Nash, C. Williams Generative Models of Part-Structured 3D Objects
4. Qin, Fei-wei, et al. "A deep learning approach to the classification of 3D CAD models." Journal of Zhejiang University SCIENCE C 15.2
(2014): 91-106.
5. Engel, Jakob, Thomas Schöps, and Daniel Cremers. "LSD-SLAM: Large-scale direct monocular SLAM." European Conference on Computer
Vision. Springer International Publishing, 2014.
6. Su, Hang, et al. "Multi-view convolutional neural networks for 3D shape recognition." Proceedings of the IEEE International Conference on
Computer Vision. 2015.
7. Whelan, Thomas, et al. "ElasticFusion: Dense SLAM Without A Pose Graph." Robotics: science and systems. Vol. 11. 2015.
8. Notchenko, Alexandr, Ermek Kapushev, and Evgeny Burnaev. "Sparse 3D Convolutional Neural Networks for Large-Scale Shape Retrieval."
arXiv preprint arXiv:1611.09159 (2016).
9. Newcombe, Richard A., Dieter Fox, and Steven M. Seitz. "Dynamicfusion: Reconstruction and tracking of non-rigid scenes in
real-time." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.
10. Gupta, Saurabh, et al. "Learning rich features from RGB-D images for object detection and segmentation." European Conference on
Computer Vision. Springer International Publishing, 2014.

More Related Content

Viewers also liked

JETSON : AI at the EDGE
JETSON : AI at the EDGEJETSON : AI at the EDGE
JETSON : AI at the EDGE
Skolkovo Robotics Center
 
Skolkovo Robotics V. International Conference
Skolkovo Robotics V. International ConferenceSkolkovo Robotics V. International Conference
Skolkovo Robotics V. International Conference
Skolkovo Robotics Center
 
сборка 3 транспорт логистика
сборка 3 транспорт логистикасборка 3 транспорт логистика
сборка 3 транспорт логистика
Skolkovo Robotics Center
 
Skolkovo Robotics 2017. Программный документ. 21 апреля 2017 года
Skolkovo Robotics 2017. Программный документ. 21 апреля 2017 годаSkolkovo Robotics 2017. Программный документ. 21 апреля 2017 года
Skolkovo Robotics 2017. Программный документ. 21 апреля 2017 года
Skolkovo Robotics Center
 
Лаборатория молодости.
Лаборатория молодости. Лаборатория молодости.
Лаборатория молодости.
Skolkovo Robotics Center
 
СОЦМЕДИКА. электронный клинический фармаколог экф V2
СОЦМЕДИКА. электронный клинический фармаколог экф  V2СОЦМЕДИКА. электронный клинический фармаколог экф  V2
СОЦМЕДИКА. электронный клинический фармаколог экф V2
Skolkovo Robotics Center
 
ai контакт-центр
ai контакт-центрai контакт-центр
ai контакт-центр
Skolkovo Robotics Center
 
Институт медицинской информатики.
Институт медицинской информатики. Институт медицинской информатики.
Институт медицинской информатики.
Skolkovo Robotics Center
 
Моделирование сложных систем и обработка больших объемов данных: ищем общие п...
Моделирование сложных систем и обработка больших объемов данных: ищем общие п...Моделирование сложных систем и обработка больших объемов данных: ищем общие п...
Моделирование сложных систем и обработка больших объемов данных: ищем общие п...
Skolkovo Robotics Center
 
Крекс, Фекс, Пекс или как заработать на нейронных сетях
Крекс, Фекс, Пекс или как заработать на нейронных сетях Крекс, Фекс, Пекс или как заработать на нейронных сетях
Крекс, Фекс, Пекс или как заработать на нейронных сетях
Skolkovo Robotics Center
 
БИОСОФТ. цифровая медицина
БИОСОФТ. цифровая медицинаБИОСОФТ. цифровая медицина
БИОСОФТ. цифровая медицина
Skolkovo Robotics Center
 
NVIDIA Deep Learning.
NVIDIA Deep Learning. NVIDIA Deep Learning.
NVIDIA Deep Learning.
Skolkovo Robotics Center
 
Skolkovo EdTech Projects BETT 2017
Skolkovo EdTech Projects BETT 2017Skolkovo EdTech Projects BETT 2017
Skolkovo EdTech Projects BETT 2017
Skolkovo Robotics Center
 
Нейронные сети в высокопроизводительных вычислениях
Нейронные сети в высокопроизводительных вычисленияхНейронные сети в высокопроизводительных вычислениях
Нейронные сети в высокопроизводительных вычислениях
Skolkovo Robotics Center
 
ФРУКТ-МД. Fructmd echo 231116
ФРУКТ-МД. Fructmd echo 231116ФРУКТ-МД. Fructmd echo 231116
ФРУКТ-МД. Fructmd echo 231116
Skolkovo Robotics Center
 
View in 3_d_asper_syllabus
View in 3_d_asper_syllabusView in 3_d_asper_syllabus
View in 3_d_asper_syllabus
lokesh503
 
Основные направления и перспективы работ в области искусственного интеллекта ...
Основные направления и перспективы работ в области искусственного интеллекта ...Основные направления и перспективы работ в области искусственного интеллекта ...
Основные направления и перспективы работ в области искусственного интеллекта ...
Skolkovo Robotics Center
 
Фонд Перспективных Исследований. 16 11 22 семинар по мед. данным в сколково
Фонд Перспективных Исследований. 16 11 22 семинар по мед. данным в сколковоФонд Перспективных Исследований. 16 11 22 семинар по мед. данным в сколково
Фонд Перспективных Исследований. 16 11 22 семинар по мед. данным в сколково
Skolkovo Robotics Center
 
Shapes in the world aorund you and me
Shapes in the world aorund you and meShapes in the world aorund you and me
Shapes in the world aorund you and me
sabreen2008
 
Geometry in Everyday Things
Geometry in Everyday ThingsGeometry in Everyday Things
Geometry in Everyday Things
VisualBee.com
 

Viewers also liked (20)

JETSON : AI at the EDGE
JETSON : AI at the EDGEJETSON : AI at the EDGE
JETSON : AI at the EDGE
 
Skolkovo Robotics V. International Conference
Skolkovo Robotics V. International ConferenceSkolkovo Robotics V. International Conference
Skolkovo Robotics V. International Conference
 
сборка 3 транспорт логистика
сборка 3 транспорт логистикасборка 3 транспорт логистика
сборка 3 транспорт логистика
 
Skolkovo Robotics 2017. Программный документ. 21 апреля 2017 года
Skolkovo Robotics 2017. Программный документ. 21 апреля 2017 годаSkolkovo Robotics 2017. Программный документ. 21 апреля 2017 года
Skolkovo Robotics 2017. Программный документ. 21 апреля 2017 года
 
Лаборатория молодости.
Лаборатория молодости. Лаборатория молодости.
Лаборатория молодости.
 
СОЦМЕДИКА. электронный клинический фармаколог экф V2
СОЦМЕДИКА. электронный клинический фармаколог экф  V2СОЦМЕДИКА. электронный клинический фармаколог экф  V2
СОЦМЕДИКА. электронный клинический фармаколог экф V2
 
ai контакт-центр
ai контакт-центрai контакт-центр
ai контакт-центр
 
Институт медицинской информатики.
Институт медицинской информатики. Институт медицинской информатики.
Институт медицинской информатики.
 
Моделирование сложных систем и обработка больших объемов данных: ищем общие п...
Моделирование сложных систем и обработка больших объемов данных: ищем общие п...Моделирование сложных систем и обработка больших объемов данных: ищем общие п...
Моделирование сложных систем и обработка больших объемов данных: ищем общие п...
 
Крекс, Фекс, Пекс или как заработать на нейронных сетях
Крекс, Фекс, Пекс или как заработать на нейронных сетях Крекс, Фекс, Пекс или как заработать на нейронных сетях
Крекс, Фекс, Пекс или как заработать на нейронных сетях
 
БИОСОФТ. цифровая медицина
БИОСОФТ. цифровая медицинаБИОСОФТ. цифровая медицина
БИОСОФТ. цифровая медицина
 
NVIDIA Deep Learning.
NVIDIA Deep Learning. NVIDIA Deep Learning.
NVIDIA Deep Learning.
 
Skolkovo EdTech Projects BETT 2017
Skolkovo EdTech Projects BETT 2017Skolkovo EdTech Projects BETT 2017
Skolkovo EdTech Projects BETT 2017
 
Нейронные сети в высокопроизводительных вычислениях
Нейронные сети в высокопроизводительных вычисленияхНейронные сети в высокопроизводительных вычислениях
Нейронные сети в высокопроизводительных вычислениях
 
ФРУКТ-МД. Fructmd echo 231116
ФРУКТ-МД. Fructmd echo 231116ФРУКТ-МД. Fructmd echo 231116
ФРУКТ-МД. Fructmd echo 231116
 
View in 3_d_asper_syllabus
View in 3_d_asper_syllabusView in 3_d_asper_syllabus
View in 3_d_asper_syllabus
 
Основные направления и перспективы работ в области искусственного интеллекта ...
Основные направления и перспективы работ в области искусственного интеллекта ...Основные направления и перспективы работ в области искусственного интеллекта ...
Основные направления и перспективы работ в области искусственного интеллекта ...
 
Фонд Перспективных Исследований. 16 11 22 семинар по мед. данным в сколково
Фонд Перспективных Исследований. 16 11 22 семинар по мед. данным в сколковоФонд Перспективных Исследований. 16 11 22 семинар по мед. данным в сколково
Фонд Перспективных Исследований. 16 11 22 семинар по мед. данным в сколково
 
Shapes in the world aorund you and me
Shapes in the world aorund you and meShapes in the world aorund you and me
Shapes in the world aorund you and me
 
Geometry in Everyday Things
Geometry in Everyday ThingsGeometry in Everyday Things
Geometry in Everyday Things
 

Similar to Burnaev and Notchenko. Skoltech. Bridging gap between 2D and 3D with Deep Learning

Computer vision introduction
Computer vision  introduction Computer vision  introduction
Computer vision introduction
Wael Badawy
 
Interactive Video Search: Where is the User in the Age of Deep Learning?
Interactive Video Search: Where is the User in the Age of Deep Learning?Interactive Video Search: Where is the User in the Age of Deep Learning?
Interactive Video Search: Where is the User in the Age of Deep Learning?
klschoef
 
最近の研究情勢についていくために - Deep Learningを中心に -
最近の研究情勢についていくために - Deep Learningを中心に - 最近の研究情勢についていくために - Deep Learningを中心に -
最近の研究情勢についていくために - Deep Learningを中心に -
Hiroshi Fukui
 
Deep Learning for X ray Image to Text Generation
Deep Learning for X ray Image to Text GenerationDeep Learning for X ray Image to Text Generation
Deep Learning for X ray Image to Text Generation
ijtsrd
 
Materi_01_VK_2223_3.pdf
Materi_01_VK_2223_3.pdfMateri_01_VK_2223_3.pdf
Materi_01_VK_2223_3.pdf
ichsan6
 
Closing, Course Offer 17/18 & Homework (D5 2017 UPC Deep Learning for Compute...
Closing, Course Offer 17/18 & Homework (D5 2017 UPC Deep Learning for Compute...Closing, Course Offer 17/18 & Homework (D5 2017 UPC Deep Learning for Compute...
Closing, Course Offer 17/18 & Homework (D5 2017 UPC Deep Learning for Compute...
Universitat Politècnica de Catalunya
 
Semantic segmentation with Convolutional Neural Network Approaches
Semantic segmentation with Convolutional Neural Network ApproachesSemantic segmentation with Convolutional Neural Network Approaches
Semantic segmentation with Convolutional Neural Network Approaches
Fellowship at Vodafone FutureLab
 
Introduction to 3D Computer Vision and Differentiable Rendering
Introduction to 3D Computer Vision and Differentiable RenderingIntroduction to 3D Computer Vision and Differentiable Rendering
Introduction to 3D Computer Vision and Differentiable Rendering
Preferred Networks
 
Satellite and Land Cover Image Classification using Deep Learning
Satellite and Land Cover Image Classification using Deep LearningSatellite and Land Cover Image Classification using Deep Learning
Satellite and Land Cover Image Classification using Deep Learning
ijtsrd
 
Overview of computer vision and machine learning
Overview of computer vision and machine learningOverview of computer vision and machine learning
Overview of computer vision and machine learning
smckeever
 
The Opportunities and Challenges of Putting the Latest Computer Vision and De...
The Opportunities and Challenges of Putting the Latest Computer Vision and De...The Opportunities and Challenges of Putting the Latest Computer Vision and De...
The Opportunities and Challenges of Putting the Latest Computer Vision and De...
Albert Y. C. Chen
 
TOP 5 Most View Article From Academia in 2019
TOP 5 Most View Article From Academia in 2019TOP 5 Most View Article From Academia in 2019
TOP 5 Most View Article From Academia in 2019
sipij
 
Introduction talk to Computer Vision
Introduction talk to Computer Vision Introduction talk to Computer Vision
Introduction talk to Computer Vision
Chen Sagiv
 
Data Con LA 2019 - State of the Art of Innovation in Computer Vision by Chris...
Data Con LA 2019 - State of the Art of Innovation in Computer Vision by Chris...Data Con LA 2019 - State of the Art of Innovation in Computer Vision by Chris...
Data Con LA 2019 - State of the Art of Innovation in Computer Vision by Chris...
Data Con LA
 
Weave-D - 2nd Progress Evaluation Presentation
Weave-D - 2nd Progress Evaluation PresentationWeave-D - 2nd Progress Evaluation Presentation
Weave-D - 2nd Progress Evaluation Presentation
lasinducharith
 
PointNet
PointNetPointNet
IRJET- Identification of Missing Person in the Crowd using Pretrained Neu...
IRJET-  	  Identification of Missing Person in the Crowd using Pretrained Neu...IRJET-  	  Identification of Missing Person in the Crowd using Pretrained Neu...
IRJET- Identification of Missing Person in the Crowd using Pretrained Neu...
IRJET Journal
 
Practical computer vision-- A problem-driven approach towards learning CV/ML/DL
Practical computer vision-- A problem-driven approach towards learning CV/ML/DLPractical computer vision-- A problem-driven approach towards learning CV/ML/DL
Practical computer vision-- A problem-driven approach towards learning CV/ML/DL
Albert Y. C. Chen
 
Kadir A_20160804_res_tea
Kadir A_20160804_res_teaKadir A_20160804_res_tea
Kadir A_20160804_res_tea
Kadir A Peker
 
Xiaoxin_Resume_combined_NCL
Xiaoxin_Resume_combined_NCLXiaoxin_Resume_combined_NCL
Xiaoxin_Resume_combined_NCL
Xiaoxin Ren
 

Similar to Burnaev and Notchenko. Skoltech. Bridging gap between 2D and 3D with Deep Learning (20)

Computer vision introduction
Computer vision  introduction Computer vision  introduction
Computer vision introduction
 
Interactive Video Search: Where is the User in the Age of Deep Learning?
Interactive Video Search: Where is the User in the Age of Deep Learning?Interactive Video Search: Where is the User in the Age of Deep Learning?
Interactive Video Search: Where is the User in the Age of Deep Learning?
 
最近の研究情勢についていくために - Deep Learningを中心に -
最近の研究情勢についていくために - Deep Learningを中心に - 最近の研究情勢についていくために - Deep Learningを中心に -
最近の研究情勢についていくために - Deep Learningを中心に -
 
Deep Learning for X ray Image to Text Generation
Deep Learning for X ray Image to Text GenerationDeep Learning for X ray Image to Text Generation
Deep Learning for X ray Image to Text Generation
 
Materi_01_VK_2223_3.pdf
Materi_01_VK_2223_3.pdfMateri_01_VK_2223_3.pdf
Materi_01_VK_2223_3.pdf
 
Closing, Course Offer 17/18 & Homework (D5 2017 UPC Deep Learning for Compute...
Closing, Course Offer 17/18 & Homework (D5 2017 UPC Deep Learning for Compute...Closing, Course Offer 17/18 & Homework (D5 2017 UPC Deep Learning for Compute...
Closing, Course Offer 17/18 & Homework (D5 2017 UPC Deep Learning for Compute...
 
Semantic segmentation with Convolutional Neural Network Approaches
Semantic segmentation with Convolutional Neural Network ApproachesSemantic segmentation with Convolutional Neural Network Approaches
Semantic segmentation with Convolutional Neural Network Approaches
 
Introduction to 3D Computer Vision and Differentiable Rendering
Introduction to 3D Computer Vision and Differentiable RenderingIntroduction to 3D Computer Vision and Differentiable Rendering
Introduction to 3D Computer Vision and Differentiable Rendering
 
Satellite and Land Cover Image Classification using Deep Learning
Satellite and Land Cover Image Classification using Deep LearningSatellite and Land Cover Image Classification using Deep Learning
Satellite and Land Cover Image Classification using Deep Learning
 
Overview of computer vision and machine learning
Overview of computer vision and machine learningOverview of computer vision and machine learning
Overview of computer vision and machine learning
 
The Opportunities and Challenges of Putting the Latest Computer Vision and De...
The Opportunities and Challenges of Putting the Latest Computer Vision and De...The Opportunities and Challenges of Putting the Latest Computer Vision and De...
The Opportunities and Challenges of Putting the Latest Computer Vision and De...
 
TOP 5 Most View Article From Academia in 2019
TOP 5 Most View Article From Academia in 2019TOP 5 Most View Article From Academia in 2019
TOP 5 Most View Article From Academia in 2019
 
Introduction talk to Computer Vision
Introduction talk to Computer Vision Introduction talk to Computer Vision
Introduction talk to Computer Vision
 
Data Con LA 2019 - State of the Art of Innovation in Computer Vision by Chris...
Data Con LA 2019 - State of the Art of Innovation in Computer Vision by Chris...Data Con LA 2019 - State of the Art of Innovation in Computer Vision by Chris...
Data Con LA 2019 - State of the Art of Innovation in Computer Vision by Chris...
 
Weave-D - 2nd Progress Evaluation Presentation
Weave-D - 2nd Progress Evaluation PresentationWeave-D - 2nd Progress Evaluation Presentation
Weave-D - 2nd Progress Evaluation Presentation
 
PointNet
PointNetPointNet
PointNet
 
IRJET- Identification of Missing Person in the Crowd using Pretrained Neu...
IRJET-  	  Identification of Missing Person in the Crowd using Pretrained Neu...IRJET-  	  Identification of Missing Person in the Crowd using Pretrained Neu...
IRJET- Identification of Missing Person in the Crowd using Pretrained Neu...
 
Practical computer vision-- A problem-driven approach towards learning CV/ML/DL
Practical computer vision-- A problem-driven approach towards learning CV/ML/DLPractical computer vision-- A problem-driven approach towards learning CV/ML/DL
Practical computer vision-- A problem-driven approach towards learning CV/ML/DL
 
Kadir A_20160804_res_tea
Kadir A_20160804_res_teaKadir A_20160804_res_tea
Kadir A_20160804_res_tea
 
Xiaoxin_Resume_combined_NCL
Xiaoxin_Resume_combined_NCLXiaoxin_Resume_combined_NCL
Xiaoxin_Resume_combined_NCL
 

More from Skolkovo Robotics Center

возможности и барьеры &quot;разговорного&quot; интеллекта а. сандлер, лаборат...
возможности и барьеры &quot;разговорного&quot; интеллекта а. сандлер, лаборат...возможности и барьеры &quot;разговорного&quot; интеллекта а. сандлер, лаборат...
возможности и барьеры &quot;разговорного&quot; интеллекта а. сандлер, лаборат...
Skolkovo Robotics Center
 
когнитивные технологии, Ibm
когнитивные технологии, Ibmкогнитивные технологии, Ibm
когнитивные технологии, Ibm
Skolkovo Robotics Center
 
влияние искусственного интеллекта на пользовательский опыт г. калугина, Yota
влияние искусственного интеллекта на пользовательский опыт  г. калугина, Yotaвлияние искусственного интеллекта на пользовательский опыт  г. калугина, Yota
влияние искусственного интеллекта на пользовательский опыт г. калугина, Yota
Skolkovo Robotics Center
 
Искусственный интеллект и пользовательский опыт
Искусственный интеллект и пользовательский опытИскусственный интеллект и пользовательский опыт
Искусственный интеллект и пользовательский опыт
Skolkovo Robotics Center
 
как вырастить и воспитать чатбота для дела а. власова, лаборатория наносемантика
как вырастить и воспитать чатбота для дела а. власова, лаборатория наносемантикакак вырастить и воспитать чатбота для дела а. власова, лаборатория наносемантика
как вырастить и воспитать чатбота для дела а. власова, лаборатория наносемантика
Skolkovo Robotics Center
 
состояние и перспективы машинного интеллекта с. шумский, нейронет
состояние и перспективы машинного интеллекта с. шумский, нейронетсостояние и перспективы машинного интеллекта с. шумский, нейронет
состояние и перспективы машинного интеллекта с. шумский, нейронет
Skolkovo Robotics Center
 
искусственный интеллект в каждый дом – как новые технологии помогают достиг...
искусственный интеллект в каждый дом – как новые технологии помогают достиг...искусственный интеллект в каждый дом – как новые технологии помогают достиг...
искусственный интеллект в каждый дом – как новые технологии помогают достиг...
Skolkovo Robotics Center
 
[Skolkovo Robotics V] Современное состояние и перспективы развития технологий...
[Skolkovo Robotics V] Современное состояние и перспективы развития технологий...[Skolkovo Robotics V] Современное состояние и перспективы развития технологий...
[Skolkovo Robotics V] Современное состояние и перспективы развития технологий...
Skolkovo Robotics Center
 
[Skolkovo Robotics V] Autonomous driving: context and technical challenges of...
[Skolkovo Robotics V] Autonomous driving: context and technical challenges of...[Skolkovo Robotics V] Autonomous driving: context and technical challenges of...
[Skolkovo Robotics V] Autonomous driving: context and technical challenges of...
Skolkovo Robotics Center
 
[Skolkovo Robotics V] Анализ задач и решений модульной, роевой и облачной роб...
[Skolkovo Robotics V] Анализ задач и решений модульной, роевой и облачной роб...[Skolkovo Robotics V] Анализ задач и решений модульной, роевой и облачной роб...
[Skolkovo Robotics V] Анализ задач и решений модульной, роевой и облачной роб...
Skolkovo Robotics Center
 
[Skolkovo Robotics V] Facial Expression Recognition in the Wild
[Skolkovo Robotics V] Facial Expression Recognition in the Wild[Skolkovo Robotics V] Facial Expression Recognition in the Wild
[Skolkovo Robotics V] Facial Expression Recognition in the Wild
Skolkovo Robotics Center
 
[Skolkovo Robotics V] Application of AI in Healthcare
[Skolkovo Robotics V] Application of AI in Healthcare[Skolkovo Robotics V] Application of AI in Healthcare
[Skolkovo Robotics V] Application of AI in Healthcare
Skolkovo Robotics Center
 
[Skolkovo Robotics V] Боевые роботы: угрозы учтенные или непредвиденные
[Skolkovo Robotics V] Боевые роботы: угрозы учтенные или непредвиденные[Skolkovo Robotics V] Боевые роботы: угрозы учтенные или непредвиденные
[Skolkovo Robotics V] Боевые роботы: угрозы учтенные или непредвиденные
Skolkovo Robotics Center
 
[Skolkovo Robotics V] Race for AI: What do VCs expect from AI startups?
[Skolkovo Robotics V] Race for AI:  What do VCs expect from AI startups?[Skolkovo Robotics V] Race for AI:  What do VCs expect from AI startups?
[Skolkovo Robotics V] Race for AI: What do VCs expect from AI startups?
Skolkovo Robotics Center
 
[Skolkovo Robotics V] Overview of the Modern Robotics Market
[Skolkovo Robotics V] Overview of the Modern Robotics Market[Skolkovo Robotics V] Overview of the Modern Robotics Market
[Skolkovo Robotics V] Overview of the Modern Robotics Market
Skolkovo Robotics Center
 
Финальная версия программы Skolkovo Robotics V
Финальная версия программы Skolkovo Robotics VФинальная версия программы Skolkovo Robotics V
Финальная версия программы Skolkovo Robotics V
Skolkovo Robotics Center
 
Презентация Альберта Ефимова на РИФ+КИБ 2017
Презентация Альберта Ефимова на РИФ+КИБ 2017Презентация Альберта Ефимова на РИФ+КИБ 2017
Презентация Альберта Ефимова на РИФ+КИБ 2017
Skolkovo Robotics Center
 
Брошюра для конференции Skolkovo.AI 14.11.16
Брошюра для конференции Skolkovo.AI 14.11.16Брошюра для конференции Skolkovo.AI 14.11.16
Брошюра для конференции Skolkovo.AI 14.11.16
Skolkovo Robotics Center
 

More from Skolkovo Robotics Center (18)

возможности и барьеры &quot;разговорного&quot; интеллекта а. сандлер, лаборат...
возможности и барьеры &quot;разговорного&quot; интеллекта а. сандлер, лаборат...возможности и барьеры &quot;разговорного&quot; интеллекта а. сандлер, лаборат...
возможности и барьеры &quot;разговорного&quot; интеллекта а. сандлер, лаборат...
 
когнитивные технологии, Ibm
когнитивные технологии, Ibmкогнитивные технологии, Ibm
когнитивные технологии, Ibm
 
влияние искусственного интеллекта на пользовательский опыт г. калугина, Yota
влияние искусственного интеллекта на пользовательский опыт  г. калугина, Yotaвлияние искусственного интеллекта на пользовательский опыт  г. калугина, Yota
влияние искусственного интеллекта на пользовательский опыт г. калугина, Yota
 
Искусственный интеллект и пользовательский опыт
Искусственный интеллект и пользовательский опытИскусственный интеллект и пользовательский опыт
Искусственный интеллект и пользовательский опыт
 
как вырастить и воспитать чатбота для дела а. власова, лаборатория наносемантика
как вырастить и воспитать чатбота для дела а. власова, лаборатория наносемантикакак вырастить и воспитать чатбота для дела а. власова, лаборатория наносемантика
как вырастить и воспитать чатбота для дела а. власова, лаборатория наносемантика
 
состояние и перспективы машинного интеллекта с. шумский, нейронет
состояние и перспективы машинного интеллекта с. шумский, нейронетсостояние и перспективы машинного интеллекта с. шумский, нейронет
состояние и перспективы машинного интеллекта с. шумский, нейронет
 
искусственный интеллект в каждый дом – как новые технологии помогают достиг...
искусственный интеллект в каждый дом – как новые технологии помогают достиг...искусственный интеллект в каждый дом – как новые технологии помогают достиг...
искусственный интеллект в каждый дом – как новые технологии помогают достиг...
 
[Skolkovo Robotics V] Современное состояние и перспективы развития технологий...
[Skolkovo Robotics V] Современное состояние и перспективы развития технологий...[Skolkovo Robotics V] Современное состояние и перспективы развития технологий...
[Skolkovo Robotics V] Современное состояние и перспективы развития технологий...
 
[Skolkovo Robotics V] Autonomous driving: context and technical challenges of...
[Skolkovo Robotics V] Autonomous driving: context and technical challenges of...[Skolkovo Robotics V] Autonomous driving: context and technical challenges of...
[Skolkovo Robotics V] Autonomous driving: context and technical challenges of...
 
[Skolkovo Robotics V] Анализ задач и решений модульной, роевой и облачной роб...
[Skolkovo Robotics V] Анализ задач и решений модульной, роевой и облачной роб...[Skolkovo Robotics V] Анализ задач и решений модульной, роевой и облачной роб...
[Skolkovo Robotics V] Анализ задач и решений модульной, роевой и облачной роб...
 
[Skolkovo Robotics V] Facial Expression Recognition in the Wild
[Skolkovo Robotics V] Facial Expression Recognition in the Wild[Skolkovo Robotics V] Facial Expression Recognition in the Wild
[Skolkovo Robotics V] Facial Expression Recognition in the Wild
 
[Skolkovo Robotics V] Application of AI in Healthcare
[Skolkovo Robotics V] Application of AI in Healthcare[Skolkovo Robotics V] Application of AI in Healthcare
[Skolkovo Robotics V] Application of AI in Healthcare
 
[Skolkovo Robotics V] Боевые роботы: угрозы учтенные или непредвиденные
[Skolkovo Robotics V] Боевые роботы: угрозы учтенные или непредвиденные[Skolkovo Robotics V] Боевые роботы: угрозы учтенные или непредвиденные
[Skolkovo Robotics V] Боевые роботы: угрозы учтенные или непредвиденные
 
[Skolkovo Robotics V] Race for AI: What do VCs expect from AI startups?
[Skolkovo Robotics V] Race for AI:  What do VCs expect from AI startups?[Skolkovo Robotics V] Race for AI:  What do VCs expect from AI startups?
[Skolkovo Robotics V] Race for AI: What do VCs expect from AI startups?
 
[Skolkovo Robotics V] Overview of the Modern Robotics Market
[Skolkovo Robotics V] Overview of the Modern Robotics Market[Skolkovo Robotics V] Overview of the Modern Robotics Market
[Skolkovo Robotics V] Overview of the Modern Robotics Market
 
Финальная версия программы Skolkovo Robotics V
Финальная версия программы Skolkovo Robotics VФинальная версия программы Skolkovo Robotics V
Финальная версия программы Skolkovo Robotics V
 
Презентация Альберта Ефимова на РИФ+КИБ 2017
Презентация Альберта Ефимова на РИФ+КИБ 2017Презентация Альберта Ефимова на РИФ+КИБ 2017
Презентация Альберта Ефимова на РИФ+КИБ 2017
 
Брошюра для конференции Skolkovo.AI 14.11.16
Брошюра для конференции Skolkovo.AI 14.11.16Брошюра для конференции Skolkovo.AI 14.11.16
Брошюра для конференции Skolkovo.AI 14.11.16
 

Recently uploaded

GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)
Javier Junquera
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Brandon Minnick, MBA
 
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid ResearchHarnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Neo4j
 
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing InstancesEnergy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Alpen-Adria-Universität
 
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
Edge AI and Vision Alliance
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
Antonios Katsarakis
 
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
Alex Pruden
 
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their MainframeDigital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Precisely
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
panagenda
 
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
saastr
 
Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Chart Kalyan
 
Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation ParametersEssentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
Safe Software
 
Skybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoptionSkybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoption
Tatiana Kojar
 
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
Edge AI and Vision Alliance
 
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdfMonitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Tosin Akinosho
 
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSFAppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
Ajin Abraham
 
Y-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PPY-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PP
c5vrf27qcz
 
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
Fwdays
 

Recently uploaded (20)

GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
 
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid ResearchHarnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
 
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing InstancesEnergy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
 
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
 
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
 
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their MainframeDigital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
 
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
 
Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
 
Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation ParametersEssentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
 
Skybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoptionSkybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoption
 
Artificial Intelligence and Electronic Warfare
Artificial Intelligence and Electronic WarfareArtificial Intelligence and Electronic Warfare
Artificial Intelligence and Electronic Warfare
 
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
 
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdfMonitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
 
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSFAppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
 
Y-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PPY-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PP
 
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
 

Burnaev and Notchenko. Skoltech. Bridging gap between 2D and 3D with Deep Learning

  • 1. Bridging the gap between 2D and 3D with Deep Learning Evgeny Burnaev (PhD) <e.burnaev@skoltech.ru> assoc. prof. Skoltech Alexandr Notchenko <a.notchenko@skoltech.ru> PhD student
  • 2. [1]
  • 3. ImageNet top-5 error over the years - Deep learning based methods - Feature based methods - human performance
  • 4. Supervised Deep Learning data Type 2D Image classification, detection segmentation Pose Estimation Supervision class label , object detection box, segmentation contours Structure of “skeleton” on image
  • 5. But world is in 3D
  • 6. 3D deep learning is gaining popularity Workshops: ● Deep Learning for Robotic Vision Workshop CVPR 2017 ● Geometry Meets Deep Learning ECCV 2016 ● 3D Deep Learning Workshop @ NIPS 2016 ● Large Scale 3D Data: Acquisition, Modelling and Analysis CVPR 2016 ● 3D from a Single Image CVPR 2015 Google Scholar when searched for "3D" "Deep Learning" returns: year # articles 2012 410 2013 627 2014 1210 2015 2570 2016 5440
  • 7. Representation of 3D data for Deep Learning Method Pros (+) Cons (-) Many 2D projections sustain surface texture, There is a lot of 2D DL methods Redundant representation, vulnerable to optic illusions Voxels simple, can be sparse, has volumetric properties losing surface properties Point Cloud Can be sparse losing surface properties and volumetric properties 2.5D images Cheap measurement devices, senses depth self occlusion of bodies in a scene, a lot of Noise in measurements
  • 8. [6]
  • 9. [2]
  • 10. 3D shape as dense Point Cloud
  • 11. Learning Rich Features from RGB-D Images for Object Detection and Segmentation [10]
  • 12. Latest development in SLAM family of methods
  • 13. LSD-SLAM (Large-Scale Direct Monocular Simultaneous Localization and Mapping) [5] LSD-SLAM - direct (feature-less) monocular SLAM
  • 14. ElasticFusion ElasticFusion - DenseSLAM without a pose-graph [7]
  • 15. Dynamic Fusion The technique won the prestigious CVPR 2015 best paper award. [9]
  • 16. Problems of SLAM algorithms ● Don’t represent objects (only know surfaces) ● Mostly dense representation (requires a lot of data) ● Whole scene is one big surface, e.g. cannot separate different objects that are close to each other.
  • 18. 3D Design Phase • There exists massive storages with 3D CAD models, e.g. GrabCAD Chairs Mechanical parts
  • 19. 3D Design Phase •Designers spend about 60% of their time searching for the right information • Massive and complex CAD models are usually disorderly archived in enterprises, which makes design reuse a difficult task 3D Model retrieval can significantly shorten the product lifecycles
  • 20. 3D Shape-based Model Retrieval •3D models are complex = No clear search rules •The text-based search has its limitations: e.g. often 3D models are poorly annotated • There is some commercial software for 3D CAD modeling, e.g. ➢ Exalead OnePart by Dassault Systems, ➢ Geolus Search by Siemens PLM, and others • However, used methods ➢ are time-consuming, ➢ are often based on hand-crafted descriptors, ➢ could be limited to a specific class of shapes, ➢ are not robust to scaling, rotations, etc.
  • 21. Sparse 3D Convolutional Neural Networks for Large-Scale Shape Retrieval Alexandr Notchenko, Ermek Kapushev, Evgeny Burnaev Presented at 3D Deep Learning Workshop at NIPS 2016
  • 22. Sparsity of voxel representation 30^3 Voxels is already enough to understand simple shape But with texture information it would be even easier Sparsity for all classes of ModelNet40 train dataset at voxel resolution 40 is only 5.5%
  • 23. Shape Retrieval Precomputed feature vector of dataset. (Vcar , Vperson ,...) Vplane - feature vector of plane Sparse3DCNN Query Retrieved items Cosine distance
  • 24. Triplet loss The representation can be efficiently learned by minimizing triplet loss. Triplet is a set (a, p, n), where ● a - anchor object ● p - positive object that is similar to anchor object ● n - negative object that is not similar to anchor object , where is a margin parameter, and are distances between p and a and n and a.
  • 25. Our approach ● Use very large resolutions, and sparse representations. ● Used triplet learning for 3D shapes. ● Used Large Scale Shape Datasets ModelNet and ShapeNet.
  • 28. Conclusions ● For small datasets of shape or 3D sparse tensors voxels can work. ● Voxels don’t scale for hundreds of “classes” and loose texture information. ● Cannot encode complicated object domains.
  • 29. Problems for next 5 years
  • 31.
  • 32.
  • 34.
  • 35.
  • 37. Robotic Control in Human Environments
  • 38. Commodity sensors to create 2.5D images Intel RealSense Series Asus Xtion Pro Microsoft Kinect v2 Structure Sensor
  • 39. What they have in common?
  • 40. What they have in common? They require understanding the whole scene
  • 41. Problem of “Holistic” Scene understanding
  • 42. Lin D., Fidler S., Urtasun R. Holistic scene understanding for 3d object detection with rgbd cameras //Proceedings of the IEEE International Conference on Computer Vision. – 2013. – С. 1417-1424. ● Human environments often designed by humans ● A most of the objects are created by humans ● Context provides information by joint probability functions ● Textures caused by materials and therefore can explain a functions and structure of an object Problem of “Holistic” Scene understanding
  • 43. Connecting 3 families of CV algorithms is inevitable Learnable Computer Vision Systems (Deep Learning) Geometric Computer Vision (SLAMs) Probabilistic Computer Vision (Bayesian methods)
  • 44. Connecting 3 families of CV algorithms is inevitable Learnable Computer Vision Systems (Deep Learning) Geometric Computer Vision (SLAMs) Probabilistic Computer Vision (Bayesian methods) Probabilistic Inverse Graphics
  • 45. Probabilistic Inverse Graphics enables ● Takes into account setting information (shop: shelves and products | street: buildings, cars, pedestrians) ● Make maximum likelihood estimates from data and model (or give directions on how to reduce uncertainty the best way) ● Learns structure of objects (Materials and textures / 3D shape / intrinsic dynamics)
  • 46. Thank you. Alexandr Notchenko Ermek Kapushev Evgeny Burnaev
  • 47. Citations and Links 1. Deep Learning NIPS’2015 Tutorial by Geoff Hinton, Yoshua Bengio & Yann LeCun 2. Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., & Xiao, J. (2015). 3D ShapeNets: A Deep Representation for Volumetric Shapes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1912-1920). 3. C. Nash, C. Williams Generative Models of Part-Structured 3D Objects 4. Qin, Fei-wei, et al. "A deep learning approach to the classification of 3D CAD models." Journal of Zhejiang University SCIENCE C 15.2 (2014): 91-106. 5. Engel, Jakob, Thomas Schöps, and Daniel Cremers. "LSD-SLAM: Large-scale direct monocular SLAM." European Conference on Computer Vision. Springer International Publishing, 2014. 6. Su, Hang, et al. "Multi-view convolutional neural networks for 3D shape recognition." Proceedings of the IEEE International Conference on Computer Vision. 2015. 7. Whelan, Thomas, et al. "ElasticFusion: Dense SLAM Without A Pose Graph." Robotics: science and systems. Vol. 11. 2015. 8. Notchenko, Alexandr, Ermek Kapushev, and Evgeny Burnaev. "Sparse 3D Convolutional Neural Networks for Large-Scale Shape Retrieval." arXiv preprint arXiv:1611.09159 (2016). 9. Newcombe, Richard A., Dieter Fox, and Steven M. Seitz. "Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015. 10. Gupta, Saurabh, et al. "Learning rich features from RGB-D images for object detection and segmentation." European Conference on Computer Vision. Springer International Publishing, 2014.