This document presents a new approach for dense non-rigid structure from motion (NRSfM) using monocular video. Previous NRSfM methods provided sparse 3D reconstructions, while this new approach enables dense 3D reconstruction for every frame. It formulates NRSfM as a variational energy minimization problem that estimates projection matrices and 3D shapes by minimizing reprojection error with spatial smoothness and low-rank priors. The energy is optimized iteratively to estimate rotations and refine shapes. Results on real sequences demonstrate improved accuracy over previous sparse NRSfM methods.

1. Dense Variational Reconstruction of Non-Rigid
Surfaces from Monocular Video
Ravi Garg Anastasios Roussos∗
Lourdes Agapito∗
Queen Mary, University of London
∗
Now at UCL
Before This Paper
1 / 17

2. Dense Non Rigid Structure from Motion
Input: monocular sequence of non-rigid surface.
...
Goal: dense 3D reconstruction for every frame.
2 / 17

3. Dense Non Rigid Structure from Motion
Input: monocular sequence of non-rigid surface.
...
Goal: dense 3D reconstruction for every frame.
2 / 17

4. Dense Non Rigid Structure from Motion
Input: monocular sequence of non-rigid surface.
...
Goal: dense 3D reconstruction for every frame.
2 / 17

5. Dense Non Rigid Structure from Motion
Input: monocular sequence of non-rigid surface.
...
Goal: dense 3D reconstruction for every frame.
2 / 17

6. Dense Non Rigid Structure from Motion
Input: monocular sequence of non-rigid surface.
...
Goal: dense 3D reconstruction for every frame.
2 / 17

7. Dense Non Rigid Structure from Motion
Input: monocular sequence of non-rigid surface.
...
Goal: dense 3D reconstruction for every frame.
Goal: dense 3D reconstruction for every frame.
NO additional sensors.
NO pre-trained shape models.
NO surface template.
2 / 17

8. Dense Non Rigid Structure from Motion
Input: monocular sequence of non-rigid surface.
...
NRSfM: ill posed problem
Goal: dense 3D reconstruction for every frame.
Goal: dense 3D reconstruction for every frame.
NO additional sensors.
NO pre-trained shape models.
NO surface template.
2 / 17

9. Traditional Sparse Non Rigid Structure from Motion
...
3 / 17

10. Traditional Sparse Non Rigid Structure from Motion
...Feature
Tracking
...
3 / 17

11. Traditional Sparse Non Rigid Structure from Motion
...Feature
Tracking
...
...3D Shape
Inference
3 / 17

12. Traditional Sparse Non Rigid Structure from Motion
Priors
...Feature
Tracking
... +
...3D Shape
Inference
3 / 17

13. Low Rank Prior for NRSfM
Shape Space
(Bregler, Hertzmann, Biermann, Recovering non-rigid 3D shape from image streams CVPR’00.)
Bregler et al. CVPR’00, Brand CVPR’01, Xiao et al. IJCV’06, Torresani et al. PAMI’08, Akhter et al.
CVPR’09, Bartoli et al. CVPR2008, Paladini et al. IJCV’12,Dai et al. CVPR’12...
4 / 17

14. Low Rank Prior for NRSfM
Shape Space
(Bregler, Hertzmann, Biermann, Recovering non-rigid 3D shape from image streams CVPR’00.)
(Park et al. ECCV’10)
4 / 17

15. Inspiration from Dense Rigid Reconstruction
(Newcombe, Lovegrove, Davison, DTAM: Dense Tracking and Mapping in Real-Time, ICCV’11)
5 / 17

16. Inspiration from Dense Rigid Reconstruction
(Newcombe, Lovegrove, Davison, DTAM: Dense Tracking and Mapping in Real-Time, ICCV’11)
Key features
Variational approach.
Use of smoothness priors.
Per pixel reconstruction.
Scalable and GPU friendly algorithm.
5 / 17

17. Leap from sparse to dense NRSfM
Sparse
Dai et al. CVPR’12
Dense
This work
6 / 17

18. Leap from sparse to dense NRSfM
Sparse
Dai et al. CVPR’12
Dense
This work
We take the best of both worlds:
Low rank prior from sparse non rigid SfM.
Variational framework from dense rigid SfM.
6 / 17

19. Leap from sparse to dense NRSfM
Sparse
Dai et al. CVPR’12
Dense
This work
We take the best of both worlds:
Low rank prior from sparse non rigid SfM.
Variational framework from dense rigid SfM.
Our contribution
First variational formulation to dense NRSfM.
Scalable algorithm which can be ported on GPU.
6 / 17

20. Our Approach in a Nutshell
...
7 / 17

21. Our Approach in a Nutshell
...
...
Step 1: Dense
Video Registration ...
7 / 17

22. Our Approach in a Nutshell
Step 2: Dense
Shape Inference
...
...
...
Step 1: Dense
Video Registration ...
7 / 17

23. Our Approach in a Nutshell
Step 2: Dense
Shape Inference
...
...
...
Step 1: Dense
Video Registration ...
Priors+
7 / 17

24. Our Approach in a Nutshell
Step 2: Dense
Shape Inference
...
...
...
Step 1: Dense
Video Registration ...
Low rank.
Spatial smoothness.+
7 / 17

25. Our Approach in a Nutshell
...
... Low rank.
Spatial smoothness.
Step 1: Dense
Video Registration ...
+
Garg, Roussos, Agapito, A variational approach to video registration with subspace constraints, IJCV’13.
7 / 17

26. Our Approach in a Nutshell
Step 2: Dense
Shape Inference
...
... Low rank.
Spatial smoothness.+
7 / 17

27. Orthographic Projection Model
8 / 17

28. Orthographic Projection Model
8 / 17

29. Orthographic Projection Model
8 / 17

30. Orthographic Projection Model
8 / 17

31. Orthographic Projection Model
W = RS
8 / 17

32. Energy Minimisation Approach to NRSfM
Formulation of a single unified energy to estimate:
Orthographic projection matrices
3D shapes for all the frames
E R , S = λ Edata R, S + Ereg S + τ Etrace S
reprojection error over all frames
spatial smoothness prior on 3D shapes
low rank prior on 3D shapes
9 / 17

33. Energy Minimisation Approach to NRSfM
Formulation of a single unified energy to estimate:
Orthographic projection matrices
3D shapes for all the frames
E R , S = λ Edata R, S + Ereg S + τ Etrace S
reprojection error over all frames
spatial smoothness prior on 3D shapes
low rank prior on 3D shapes
9 / 17

34. Energy Minimisation Approach to NRSfM
Formulation of a single unified energy to estimate:
Orthographic projection matrices
3D shapes for all the frames
E R , S = λ Edata R, S + Ereg S + τ Etrace S
reprojection error over all frames
spatial smoothness prior on 3D shapes
low rank prior on 3D shapes
9 / 17

35. Energy Minimisation Approach to NRSfM
Formulation of a single unified energy to estimate:
Orthographic projection matrices
3D shapes for all the frames
E R , S = λ Edata R, S + Ereg S + τ Etrace S
reprojection error over all frames
spatial smoothness prior on 3D shapes
low rank prior on 3D shapes
9 / 17

36. Reprojection Error
E
`
R, S
´
= λEdata
`
R, S
´
+ Ereg
`
S
´
+ τEtrace
`
S
´
Edata (R, S) = W − RS 2
F
10 / 17

37. Spatial Smoothness Prior
E
`
R, S
´
= λEdata
`
R, S
´
+ Ereg
`
S
´
+ τEtrace
`
S
´
Ereg S =
i
TV (Si)
−−−−−−→
Without regularisation With regularisation
11 / 17

38. Low Rank Prior
E
`
R, S
´
= λEdata
`
R, S
´
+ Ereg
`
S
´
+ τEtrace
`
S
´
Etrace S = S ∗ =
i
σi(S)
lies in−−−−→ span
K F
Angst et al. ECCV’12, Dai et al. CVPR’12, Angst et al. ICCV’11, Dai et al. ECCV’10
12 / 17

39. Minimisation of E R, S
min
R,S
λ W − RS 2
F
Reprojection
error
+
i
TV (Si)
Smoothness
prior
+ τ S ∗
Low rank
prior
13 / 17

40. Minimisation of E R, S
min
R,S
λ W − RS 2
F
Reprojection
error
+
i
TV (Si)
Smoothness
prior
+ τ S ∗
Low rank
prior
Our Algorithm
Initialize R and S using rigid factorisation.
Minimize energy via alternation:
Step 1: Rotation estimation.
Step 2: Shape estimation.
Efficient and highly parallelizable algorithm → GPU-friendly
13 / 17

41. Minimisation of E R, S
min
R
λ W − RS 2
F
Reprojection
error
+
i
TV (Si)
Smoothness
prior
+ τ S ∗
Low rank
prior
Step 1: Rotation estimation
Robust estimation by using dense data.
Solved via Levenberg-Marquardt algorithm.
Rotations are parametrised as quaternions.
13 / 17

42. Minimisation of E R, S
min
S
λ W − RS 2
F
Reprojection
error
+
i
TV (Si)
Smoothness
prior
+ τ S ∗
Low rank
prior
Step 2: Shape estimation
Convex sub-problem.
Optimisation via alternation between:
Per frame shape refinement: using primal dual algorithm
Enforcing low rank: using soft impute algorithm.
13 / 17

43. Results on real sequences
14 / 17

44. Quantitative Evaluation
Average RMS 3D reconstruction errors.
Sequence TB MP Ours Ours(τ = 0)
Non-smooth rotations 4.50% 5.13% 2.60% 3.32%
Smooth rotations 6.61% 5.81% 2.81% 3.89%
- TB: Akhter et al, Trajectory space: A dual representation for non-rigid structure from motion, PAMI’11.
- MP: Paladini et al, Optimal metric projections for deformable and articulated structure-from-motion,
IJCV’12.
15 / 17

45. Conclusions and future work
Conclusions:
First dense, template-free approach to Non-rigid Structure
from Motion.
Unified energy minimization for both rotation and shape
estimation.
Combination of low-rank and spatial regularization prior.
Using variational methods, we can do much more with
monocular sequences that one could expect
Future work:
Direct estimation from pixel intensities
Towards real-time: online formulation
Occlusion modelling
16 / 17

46. Thank You for Your Attention!
For more details and data
Visit: www.eecs.qmul.ac.uk/~rgarg
OR Come to our poster!
Questions?
Authors thank
Chris Russell and Sara Vicente for valuable discussions.
17 / 17