The document discusses the SfMLearner++ model, which improves on the SfMLearner model for unsupervised learning of depth and ego-motion from monocular video. SfMLearner++ incorporates additional geometric constraints, including an epipolar weight, to regularize the depth and pose predictions. An evaluation on KITTI datasets shows SfMLearner++ achieves state-of-the-art performance in terms of both depth and pose estimation compared to other unsupervised methods like SfMLearner, GeoNet, and DDVO.

1. SfM Learner++
Depth Pose

2. u
u AI
u →
u MRI
u LK SOTA/ B4/
u Computer Vision
u D
u Engineering Management
@HO75013227

3. u SLAM
u
u
u
u

4. u “SfMLearner++: Learning Monocular Depth & Ego-Motion using
Meaningful Geometric Constraints”
u Vignesh Prasad, Brojeshwar Bhowmick.
u TCS Innovation Labs Tata Consultancy Services
u arXiv 20/Dec/2018. https://arxiv.org/pdf/1812.08370.pdf
u WACV 2019 Accepted
u
u Depth
u Unsupervised

5. u
u
u Photometric Consistency
u Geometric Deep
u Geometric Constraint
u Regularization
u SfMLearner++

6. u
u
u Photometric Consistency
u
u Geometric Constraint
u Regularization
u SfMLearner++

7. SLAM 3D ” https://www.slideshare.net/KenSakurada
DOF)

8. u
Scale Ambiguity
SLAM 3D ” https://www.slideshare.net/KenSakurada

9. u
u
u Photometric Consistency
u
u Geometric Constraint
u Regularization
u SfMLearner++

10. u
u
u Photometric Consistency
u
u Geometric Constraint
u Regularization
u SfMLearner++

11. Photometric Consistency
u Intensity
P
It
Is1
Is2
Physical
Space
ps1
pt
ps2
u Photometric Consistency Loss
Reprojection Error
Depth
t
!"# = %
"
&' ( − *&+ (

12. Photometric Consistency
u
u
u

13. u
u
u Photometric Consistency
u
u Geometric Constraint
u Regularization
u SfMLearner++

14. u Direct Visual Odometry [Steinbrucker]
u
u Photometric Consistency Loss
Depth
!"# = %
"
&' ( − *&+ (
,

15. u Direct Visual Odometry [Steinbrucker]
u Inverse Compositional Jacobian/Hessian
Differentiable DVO [Wang]
!"# = %
"
&' ( − *&+ (
,
*&+ ( = &' ( + (/ ( − ()∇&' (
Photometric Consistency Loss
6DOF

16. u
u
u Photometric Consistency
u
u Geometric Constraint
u Regularization
u SfMLearner++

17. Geometric Constraint
u x1 l2
Fundamental Matrix
Essential Matrix
SLAM 3D ” https://www.slideshare.net/KenSakurada

18. u
u
u Photometric Consistency
u
u Geometric Constraint
u Regularization
u SfMLearner++

19. Regularization
u ill-posed
u Depth Optical flow
Depth
Depth [Wang]

20. Depth Regularization
u Depth
u [Wang]
Iteration Depth
Scale Ambiguity

21. u
u
u Photometric Consistency
u
u Geometric Constraint
u Regularization
u SfMLearner++

22. SfMLearner
SfMLearner++
DVO DDVO
GeoNet
SfMNet
u
Mahjourian
20172011 2018
Depth CNN+DVO
Pose Net +Depth Net
3D Constraint
Geometric
Constraint
Flow
Constraint
Depth Net+Pose 3D

23. SfMLearner [Tinghui 2017]
u Depth Pose Loss

24. SfMLearner [Tinghui 2017]
u
Depth
2 L1 norm
Explainability Mask
BCE
Explainability Mask
Photometric Consistency Loss
Explainability Mask

25. SfMLearner [Tinghui 2017]
u CNN Depth Net, Pose Net
u Explainability Mask

26. GeoNet [Zhichao 2018]
u Depth Net Pose Net Depth, Pose
u Flow

27. GeoNet [Zhichao 2018]
u
Depth
Smoothness
Photometric
Consistency
Loss
Flow
Smoothness
Flow
Consistency
Flow
Photometric
Consistency
Loss

28. 3D Constraints [Mahjourian 2018]
u SfMLearner
Depth,Pose
u 3D
u 3D

29. 3D Constraints[Mahjourian 2018]
u
Photometric
Consistency
Loss
Depth
Smoothness
Structured
Similarity
3D
Consistency
ICP
ICP

30. Differentiable DVO [Wang, 2017]
u Geometric Depth Pose

31. Differentiable DVO [Wang, 2017]
u
Photometric Consistency Loss
Depth

32. SfM Learner++

33. SfM Learner++ [Prasad 2018]
u SfM Learner Architecture

34. Network Architecture
u SfMLearner
u Explainability Mask

35. SfM Learner++ [Prasad 2018]
u
Eplipolar Weight
Photo Consistency
Loss
Depth

36. SfM Learner
u
u

37. Make3D

38. Epipolar Weight
u
Epipolar Weight

39. CitySpace
u KITTI

40. Depth

41. Ablation Study
u Depth Normalization
u Epipolar

42. Depth
u KITTI 697
u GeoNet, DDVO Stereo Godard
u SfMLearner++ > SfMLearner

43. Pose
u KITTI Visual Odometry Benchmark
u Average Trajectory Error SOTA SfMLearner

44. u SfMLearner++
Photometric Consistency Loss
Geometry Constraint
u SOTA
u Flow 3D
Constraint

45. u [Prasad, 2018] “SfMLearner++: Learning Monocular Depth & Ego-Motion using Meaningful Geometric
Constraints”, Vignesh Prasad* Brojeshwar Bhowmick.
u [Tinghui 2017] “Unsupervised Learning of Depth and Ego-Motion from Video”, Tinghui Zhou, Matthew Brown,
Noah Snavely, David G. Lowe.
u [SfM-Net2017] “Learning of Structure and Motion from Video”, Sudheendra Vijayanarasimhan, Susanna Ricco,
Cordelia Schmid, Rahul Sukthankar, Katerina Fragkiadaki.
u [Zhichao 2018] “GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose”, Zhichao
Yin and Jianping Shi.
u [Chen-Hsuan 2016] “Inverse Compositional Spatial Transformer Networks”, Chen-Hsuan Lin Simon Lucey.
u [Wang et al, 2017] “Learning Depth from Monocular Videos using Direct Methods”, Chaoyang Wang, Jose
Miguel Buenaposada, Rui Zhu, Simon Lucey.
u [Steinbrucker et al, 2011] “Real-Time Visual Odometry from Dense RGB-D Images”, Frank Steinbrücker
Jürgen Sturm Daniel Cremers.
u [Engelhard et al, 2014] “A Benchmark for the Evaluation of RGB-D SLAM Systems Jurgen Sturm, Nikolas
Engelhard”, Felix Endres, Wolfram Burgard, and Daniel Cremers.
u [Mahjourian et al, 2018] “Unsupervised learning of depth and ego-motion from monocular video using 3d
geometric constraints”, R. Mahjourian, M. Wicke, and A. Angelova.