The document discusses recent trends and advancements in 3D deep learning, highlighting various techniques such as volumetric CNN regression, image style transfer, and differentiable rendering. It mentions key studies and frameworks like DeepMimic for reinforcement learning and point cloud processing through PointNet++. Additionally, it addresses the limitations of 3D deep learning, including challenges like the curse of dimensionality and GPU memory constraints.

1. 3D 딥러닝 동향
New Trends in 3D Deep Learning
홍정모

2. Lost in 3D Deep Learning
어느 40대 프로그래머의

3. “Software 2.0 is written in
neural network weights.”
Director of AI at Tesla.

4. Software 1.0
Manual(?)
differentiation

6. Large Pose 3D Face Reconstruction from a
Single Image via Direct Volumetric CNN
Regression
[Jackson et al. ICCV 2017] Training data
- 2D face images
- 3D points -> Voxels

8. Joint 3D Face Reconstruction and Dense
Alignment with Position Map Regression
Network [Feng et al. 2018]
Training data
- 2D face images
- 3D points -> UV map

9. Auto-differentiation
Data driven
Neural network
Open sources

11. 강화 학습

12. 주먹 지르기 강화학습

13. DeepMimic: Example-Guided Deep Reinforcement
Learning of Physics-Based Character Skills
[Peng et al. 2018]

14. How ?
Image Style Transfer Using Convolutional Neural Networks
Gatys et al. CVPR 2016
Neural Style Transfer
3D Models

15. 3D Deep Learning
Mostly based on 3D Deep Learning Tutorial @ CVPR 2017 and
Geometric Deep Learning SIGGRAPH Asia 2016 Course Notes

16. 2D Image
3D Representations

17. Volumetric Representation
Learning a Probabilistic Latent Space of
Object Shapes via
3D Generative-Adversarial Modeling
[Wu et al. NIPS 2016]

18. tempoGAN
600 × 400 × 400
Kim and Hong 20121024 × 720 × 720256 × 180 × 180

19. Interactive 3D Modeling with a Generative Adversarial Network [Liu et al. 2017]
OctNet: Learning Deep 3D Representations
at High Resolutions [Riegler et al. 2016]
Octree Generating Networkss: Efficient Convolutional Architectures
for High-resolution 3D Outputs [Tatarchenko et al. 2017]

20. Intrinsic Deep Learning on Manifolds
Extrinsic Representation Intrinsic Representation

21. PointNet++: Deep Hierarchical Feature Learning on Point Sets
in a Metric Space [Qi et al. NIPS 2017]
Shape Generation using Spatially
Partitioned Point Clouds
[Gadelha et al. 2017]

22. Multi-view Convolutional Neural Networks
for 3D Shape Recognition
[Su et al. ICCV 2015]
3D Shape Reconstruction
from Sketches
via Multi-view Convolutional Networks
[Lun et al. 2017]

23. How to Back-propagate ?
Movie ‘Lucy’ (2014)

24. Perspective Projection Loss
Perspective Transformer Nets:
Learning Single-View
3D Object Reconstruction
without 3D Supervision
[Yan et al. NIPS 2016]

25. Differentiable Renderer
Neural 3D Mesh Renderer [Kato et al. CVPR 2018]

26. Differentiable
Ray-tracer
Levelset
Back-propagation

27. Learning Category-Specific Mesh Reconstruction
from Image Collections [Kanazawa et al. 2018]

28. Graph-based CNN
Pixel2Mesh: Generating 3D Mesh Models
from Single RGB Images [Wang et al. 2018]

29. Limitations in 3D Deep Learning
• Curse of dimensionality
• GPU memory limitation
• Rotation invariance
• Insufficient data
• … …

30. Drama ‘West World’ (2016)
Atom Bit