This document provides a summary of a CVPR 2016 tutorial on fitting surface models to data. The tutorial covered several applications of surface fitting including curve and surface fitting, parameter estimation, bundle adjustment, and more. It discussed fitting subdivision surfaces and polygon meshes to 2D and video data. Specific examples of fitting hand models to data for applications like hand tracking were presented. The tutorial aimed to teach attendees how to solve hard vision problems using tools that may seem inelegant but are smarter than they appear for fitting models to data.

1. CVPR 2016 TUTORIAL
FITTING SURFACE MODELS TO DATA
のご紹介
2016/10/02
@sumisumith

2. 2
• オリジナルのチュートリアルスライドから、
スライドを抜粋
• 適宜、説明のため、自前で作成した
スライドを追加
して作成したプレゼン資料です。
！ご注意！
http://awf.fitzgibbon.ie/cvpr16_tutorial
オリジナルの資料はココ

3. Fitting Surface Models to Data
3
CVPR 2016Tutorial
Andrew Fitzgibbon, Microsoft
JonathanTaylor, PerceptiveIO

4. PEOPLE
Finding Nemo: Deformable Object Class Modelling using Curve Matching CVPR ’10
Mukta Prasad,Andrew Fitzgibbon,AndrewZisserman, LucVan Gool
KinÊtre: Animating theWorld with the Human Body UIST ’12
Jiawen (Kevin) Chen, Shahram Izadi, Fitzgibbon
TheVitruvian Manifold: Inferring dense correspondences for one-shot human pose estimation CVPR ’12
JonathanTaylor, Jamie Shotton,TobySharp, Fitzgibbon
What shape are dolphins? Building 3D morphable models from 2D images PAMI ’13
Tom Cashman, Fitzgibbon
User-Specific Hand Modeling from Monocular Depth Sequences CVPR ’14
Taylor, Richard Stebbing,Varun Ramakrishna, Cem Keskin, Shotton, Izadi, Fitzgibbon,Aaron Hertzmann
Real-Time Non-Rigid Reconstruction Using an RGB-D Camera SIGGRAPH ’14
Michael Zollhöfer, Matthias Nießner, Izadi, Christoph Rhemann, Christopher Zach,
Matthew Fisher, ChengleiWu, Fitzgibbon,Charles Loop, ChristianTheobalt, Marc Stamminger
Learning an Efficient Model of Hand ShapeVariation from Depth Images CVPR ’15
Sameh Khamis,Taylor,Shotton, Keskin, Izadi, Fitzgibbon
Efficient and Precise Interactive Hand Tracking through Joint, Continuous Optimization of Pose SIGGRAPH ‘16
and Correspondences
Taylor, Lucas Bordeaux,Cashman, Bob Corish, Keskin, Sharp, Eduardo Soto, David Sweeney, JulienValentin,
Ben Luff, ArranTopalian, ErrollWood, Khamis, Kohli, Izadi, Richard Banks, Fitzgibbon, Shotton.
Fits Like a Glove: Rapid and Reliable Hand Shape Personalization. CVPR ’16
David JosephTan,Cashman,Taylor, Fitzgibbon, DanielTarlow, Khamis, Izadi, Shotton.

5. LEARN HOWTO SOLVE HARDVISION PROBLEMS,
USINGTOOLSTHAT MAY APPEAR INELEGANT,
BUT ARE MUCH SMARTERTHANTHEY LOOK.
Goal
5

6. APPLICATIONS
Curve/surface fitting Parameter estimation “Bundle adjustment”
(Video from our friends at Google)

7. KINÊTRE 7

8. KINÊTRE 8

9. KINÊTRE 9

10. VITRUVIANMANIFOLD,CVPR’12

13. FITTINGSUBDIVISIONSURFACESTO2DDATA

14. FITTINGSUBDIVISIONSURFACESTO2DDATA

15. 15

16. FITTINGPOLYGONMESHESTOVIDEO

17. 17
[3D Scanning Deformable Objects with a Single RGBD Sensor, Dou et al, CVPR15]
Input Kinect Stream KinectFusion Deformable Fusion

18. HANDTRACKING
• Hand Shape Personalization:
• CVPR 2014,CVPR 2015,CVPR 2016
• Discriminative Hand Pose Reinitialization
• ICCV 2015,CHI 2015
• Hand Pose Estimation via Model Fitting (read “HandTracking”)
• CHI 2015, SIGGRAPH 2016

19. IMAGEDENOISING 19[STRANDMARK & AGARWAL, 2014, arXiv:1403.5590]

20. MATRIXFACTORIZATION[HONG&F.,ICCV15]

21. MATRIXFACTORIZATION[HONG&F.,ICCV15]

22. MYTH:YOUDON’TNEEDTOOPTIMIZEFAR 22

23. Write energy describing the image collection
𝑓=1
𝐹
𝐸data 𝐼𝑓, 𝜽 𝑓 + 𝐸reg 𝜽 𝑓, 𝜽core
Where:
𝜽 𝑓 are (unknown) parameters of surface model in frame 𝑓
𝜽core are (unknown) parameters of some shape model (e.g. linear
combination) and 𝐸reg measures distance, e.g. ARAP
And optimize it using Levenberg-Marquardt
 (i.e. any Newton-like algorithm, making maximum use of problem
structure)
FOREACHTASK,THEMETHODISTHESAME 23

24.  So, you can do lots of things by “fitting models to
data”.
 How do you do it right?
 Let’s look at some examples.
24

25. CONTINUOUS
OPTIMIZATION
Andrew Fitzgibbon
Microsoft Research Cambridge

26. GOAL
Given function
𝑓 𝑥 : ℝ 𝑑
↦ ℝ,
Devise strategies for finding 𝑥 which minimizes 𝑓
• Gradient descent++: Stochastic, Block, Minibatch
• Coordinate descent++: Block
• Newton++:Gauss, Quasi, Damped, Levenberg Marquardt, dogleg,Trust
region, Doublestep LM, [L-]BFGS, NonlinCG
• Not covered
• Proximal methods: Nesterov, ADMM…

27. CLASSESOFFUNCTIONS
quadratic convex quasiconvex multi-
extremum
noisy horrible
Given function
𝑓 𝑥 : ℝ 𝑑 ↦ ℝ
Devise strategies for finding 𝑥 which minimizes 𝑓

28. CLASSESOFFUNCTIONS
quadratic convex quasiconvex multi-
extremum
noisy horrible
Given function
𝑓 𝑥 : ℝ 𝑑 ↦ ℝ
Devise strategies for finding 𝑥 which minimizes 𝑓

29. 29
quadratic
convex
quasiconvex
multi-
extremum

30. 30
quadratic
convex
quasiconvex
multi-
extremum
Easy Hard

31. DERIVATIVES
Fast minimization depends on derivatives
 Gradient 𝑓: ℝ 𝑛 ↦ ℝ
 When 𝑓 𝒙 = 𝑭 𝒙 2
𝑭: ℝ 𝑛
↦ ℝ 𝑚
use Jacobian
𝜕𝑭
𝜕𝒙
31

32. ALTERNATION
Easy Hard

33. 33
Rosenbrock関数
最急降下法
滑降シンプレックス法
レベンバーグ・マルカート法

34. GRADIENTDESCENT
 Alternation is slow
because valleys may not
be axis aligned
 So try gradient descent?
-5 0 5 10 15
-5
0
5
10
15

35. GRADIENTDESCENT
 Alternation is slow
because valleys may not
be axis aligned
 So try gradient descent?

36. GRADIENTDESCENT
 Alternation is slow
because valleys may not
be axis aligned
 So try gradient descent?
 Note that convergence
proofs are available for
both of the above
 But so what?

37. ANDONAHARDPROBLEM

38. USEABETTERALGORITHM
 (Nonlinear) conjugate
gradients
 Uses 1st derivatives only
 Avoids “undoing”
previous work

39. USEABETTERALGORITHM
 (Nonlinear) conjugate
gradients
 Uses 1st derivatives only
 And avoids “undoing”
previous work
 101 iterations on
this problem

40. BUTWE CAN DO BETTER…

41. USESECONDDERIVATIVES…
 Starting with 𝒙 how can I choose 𝜹
so that 𝑓 𝒙 + 𝜹 is better than 𝑓(𝒙)?
 So compute
min
𝜹∈ℝ 𝑑
𝑓 𝒙 + 𝜹
 But hang on, that’s the same problem we were trying to
solve?

42. USESECONDDERIVATIVES…
 Starting with 𝑥 how can I choose 𝛿
so that 𝑓 𝑥 + 𝛿 is better than 𝑓(𝑥)?
 So compute
min
𝛿
𝑓 𝑥 + 𝛿
≈ min
𝛿
𝑓 𝑥 + 𝛿⊤ 𝑔(𝑥) + 1
2 𝛿⊤ 𝐻 𝑥 𝛿
𝑔 𝑥 = 𝛻𝑓 𝑥
𝐻 𝑥 = 𝛻𝛻⊤ 𝑓(𝑥)

43. USESECONDDERIVATIVES…
 How does it look?
𝑓 𝑥 + 𝛿⊤
𝑔(𝑥) + 1
2
𝛿⊤
𝐻 𝑥 𝛿
𝑔 𝑥 = 𝛻𝑓 𝑥
𝐻 𝑥 = 𝛻𝛻⊤
𝑓(𝑥)

44. USESECONDDERIVATIVES…
 Choose 𝛿 so that 𝑓 𝑥 + 𝛿 is better than 𝑓(𝑥)?
 Compute
min
𝜹
𝑓 + 𝜹⊤ 𝑔 + 1
2 𝜹⊤ 𝐻 𝜹
[derive]

45. USESECONDDERIVATIVES…
 Choose 𝛿 so that 𝑓 𝑥 + 𝛿 is better than 𝑓(𝑥)?
 Compute
min
𝜹
𝑓 + 𝜹⊤ 𝑔 + 1
2 𝜹⊤ 𝐻 𝜹
𝜹 = −𝐻−1 𝑔

46. USESECONDDERIVATIVES…
 Choose 𝛿 so that 𝑓 𝑥 + 𝛿 is better than 𝑓(𝑥)?
 Updates:
𝜹Newton = −𝐻−1
𝑔
𝜹GradientDescent = −𝜆𝑔

47. USESECONDDERIVATIVES…
 Updates:
𝜹Newton = −𝐻−1
𝑔
𝜹GradientDescent = −𝜆𝑔
 So combine them:
𝜹DampedNewton = − 𝐻 + 𝜆−1
𝐼 𝑑
−1
𝑔
= −𝜆 𝜆𝐻 + 𝐼 𝑑
−1
𝑔
 𝜆 small ⇒conservative gradient step
 𝜆 large ⇒Newton step

48. UPDATING 𝜆
𝜆 = 10−3; 𝜆′ = 3;
while 𝜆 < 109
𝑓, 𝒈, 𝑯 = error_function(𝒙 𝑘) % Perhaps Gauss-Newton for H
𝜹 = − 𝑯 + 𝜆𝑰 𝒈 % Many ways to do this efficiently
𝒙 𝑛𝑒𝑤 = 𝒙 𝑘 + 𝜹
if error_function(𝒙 𝑛𝑒𝑤) < 𝑓:
𝒙 𝑘 = 𝒙 𝑛𝑒𝑤 % Decreased error, accept the new 𝑥
𝜆 = 𝜆/𝜆′; 𝜆′ = 3 % Doing well—decrease 𝜆
else
𝜆 = 𝜆𝜆′; 𝜆′ = 3𝜆′ % Doing badly—increase 𝜆 quick

49. 1ST DERIVATIVESAGAIN
Levenberg-Marquardt
 Just damped Newton with approximate 𝐻
 For a special form of 𝑓
𝑓 𝑥 =
𝑖
𝑓𝑖 𝑥 2
 where 𝑓𝑖(𝑥) are
 zero-mean
 small at the optimum

50. BACKTOFIRSTDERIVATIVES
Levenberg Marquardt
 Just damped Newton with approximate 𝐻
 For a special form of 𝑓
𝑓 𝑥 =
𝑖
𝑓𝑖 𝑥 2
𝛻𝑓 𝑥 =
𝛻𝛻⊤
𝑓 𝑥 =

51. BACKTOFIRSTDERIVATIVES
Levenberg Marquardt
 Just damped Newton with approximate 𝐻
 For a special form of 𝑓
𝑓 𝑥 =
𝑖
𝑓𝑖 𝑥 2
𝛻𝑓 𝑥 =
𝑖
2𝑓𝑖 𝑥 𝛻𝑓𝑖(𝑥)
𝛻𝛻⊤ 𝑓 𝑥 = 2
𝑖
𝑓𝑖 𝑥 𝛻𝛻⊤ 𝑓𝑖 𝑥 + 𝛻𝑓𝑖(𝑥)𝛻⊤ 𝑓𝑖 𝑥

52. ORDERNCUBED?
 Not 𝑂 𝑛3
if you exploit sparsity of Hessian or
Jacobian
J =
𝛻𝑓1(𝑥)
⋮
𝛻𝑓𝑛(𝑥)

53. TYPICALHESSIANSTRUCTURE

54. 54
参照：バンドルアジャストメント 岡谷 貴之
http://bit.ly/2dcaFr7
きっと言いたいことはコレ

55. CONCLUSION:YMMV

56. CONCLUSION:YMMV
GIRAFFE
500 runs
for k=1:500
𝑥0 = 𝑟𝑎𝑛𝑑𝑛 𝑛, 1 ;
𝑥∗
= 𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝑓, 𝑥0 ;
𝐸 𝑘 = 𝑓 𝑥∗
end
plot(sort(E));

57. CONCLUSION:YMMV
FACE
1000 runs
DINOSAUR
1000 runs
GIRAFFE
500 runs

58.  On many problems,
alternation is just fine
 Indeed always start with a
couple of alternation steps
 Computing 2nd derivatives is
a pain
 But you don’t need to for LM
 But just alternation is not
 Unless you’re willing to
problem-select
 Convergence guarantees
are fine, but practice is
what matters
 Inverting the Hessian is
rarely 𝑂(𝑛3
)
There is no universal optimizer

59. WHAT IS A SURFACE?
59

60. 0 0.2 0.4 0.6 0.8 1
1.9
2
2.1
2.2
2.3
2.4
CURVES
function y(x::Interval)::Real = .3*x + 2
function C(t::Interval)::Point2D =
Point2D(t^2 + 2, t^2 – t + 1)
function S(u::Interval, v::Real)::Point3D =
Point3D(cos(u), sin(u), v)
60
2 2.2 2.4 2.6 2.8 30.75
0.8
0.85
0.9
0.95
1

61.  Surface: mapping 𝑆 𝒖 from ℝ2
↦ ℝ3
 E.g. cylinder 𝑆 𝑢, 𝑣 = cos 𝑢 , sin 𝑢 , 𝑣
SURFACE 61
*the surface is actually the set {𝑀 𝑢; Θ |𝑢 ∈ Ω}
𝑢
𝑣

62.  Surface: mapping 𝑆 𝒖 from ℝ2
↦ ℝ3
 E.g. cylinder 𝑆 𝑢, 𝑣 = cos 𝑢 , sin 𝑢 , 𝑣
 Probably not all of ℝ2
, but a subset Ω
 E.g. square Ω = 0,2𝜋 × [0, 𝐻]
 But also any union of patch domains Ω = 𝑝
Ω 𝑝
SURFACE 62
*the surface is actually the set {𝑀 𝑢; Θ |𝑢 ∈ Ω}
𝑢
𝑣

63.  Surface: mapping 𝑆 𝒖 from ℝ2
↦ ℝ3
 E.g. cylinder 𝑆 𝑢, 𝑣 = cos 𝑢 , sin 𝑢 , 𝑣
 Probably not all of ℝ2
, but a subset Ω
 E.g. square Ω = 0,2𝜋 × [0, 𝐻]
 But also any union of patch domains Ω = 𝑝
Ω 𝑝
 And we’ll look at parameterised surfaces 𝑆 𝒖; Θ
 E.g. Cylinder 𝑆 𝑢, 𝑣; 𝑅, 𝐻 = 𝑅 cos 𝑢 , 𝑅 sin 𝑢 , 𝐻𝑣
with Ω = 0,2𝜋 × 0,1
 E.g. subdivision surface 𝑆 𝒖; 𝑋
where Θ = 𝑋 ∈ ℝ3×𝑛
is matrix of control vertices
SURFACE 63
*the surface is actually the set {𝑀 𝑢; Θ |𝑢 ∈ Ω}
𝑢
𝑣

64. TOOL: SUBDIVISION SURFACES
64

65. CONTROLMESH 65
Control mesh vertices 𝑋 ∈ ℝ3×𝑚
Here 𝑚 = 16

66. LIMITSURFACE 66
Control mesh vertices 𝑋 ∈ ℝ3×𝑚
Here 𝑚 = 16

67. SUBDIVRULE:STEP1.ADDNEWVERTICES 67

68. SUBDIVRULE:STEP2.AVERAGENEIGHBOURS 68

69. 2 SUBDIVISIONS 69

70. 3 SUBDIVISIONS 70

71. LIMITSURFACE 71
Control mesh vertices 𝑉 ∈ ℝ3×𝑚
Here 𝑚 = 16
Blue surface is 𝑀 𝒖; 𝑉 | 𝒖 ∈ Ω
Ω is the grey surface

72. CONTROLVERTICESDEFINETHESHAPE 72
Control mesh vertices 𝑉 ∈ ℝ3×𝑛
Here 𝑛 = 16
Blue surface is 𝑀 𝒖; 𝑉 | 𝒖 ∈ Ω
Ω is the grey surface

73.  Mostly, 𝑀 is quite simple:
𝑀 𝒖; 𝑋 = 𝑀 𝑡, 𝑢, 𝑣; 𝒙1, … , 𝒙 𝑛 =
𝑖+𝑗≤4
𝑘=1..𝑛
𝐴𝑖𝑗𝑘
𝑡
𝑢 𝑖
𝑣 𝑗
𝒙 𝑘
 Integer triangle id 𝑡
 Quartic in 𝑢, 𝑣
 Linear in 𝑋
 Easy derivatives
 But…
 2nd Derivatives unbounded although normals well defined
 Piecewise parameter domain
SUBDIVISIONSURFACE:PARAMETRICFORM 73

74. EXAMPLES 74

75. BACKTO DOLPHINS
75

76. WHAT SHAPE ARE DOLPHINS?
BUILDING 3D MORPHABLE MODELS
FROM 2D IMAGES
勝手な通称：イルカの論文
76

77. 77
• 概要
• Loop Subdivision Surface で表現された、基本3Dモデル：Bを用意
• シルエット制約などの各種制約を設計し、バランスを取れた
モデルができるように、エネルギー関数を設計
• Bを変形させるルール／パラメタに従い、変形させ、
エネルギーが最小／最適な3Dモデルを推定
• 入力：
• ヒューリスティックに与えた、キーポイント数個
• 一般 Non-rigid 物体の基本3Dモデル B
• シルエット既知の画像
• 各種変形に用いる、パラメタ値
• 出力：
• 入力されたシルエット画像に合う、変形された3Dモデル
イルカ ( 一般 Non-rigid 物体 ) の姿勢推定

78. MODELREPRESENTATION
𝑋 𝑛 =
𝑘=0
𝐾
𝛼𝑖𝑘ℬ 𝑘
𝛼𝑖1 ℬ1 𝛼𝑖2 ℬ2+ +𝑋𝑖 =
Linear blend shapes:
Image 𝑖 represented by coefficient
vector 𝜶𝑖 = 𝛼𝑖1, … , 𝛼𝑖𝐾
ℬ0
78

79. 79

80. 80

81. 81

82. 𝒔𝑖𝑗 2D point
𝒏𝑖𝑗 2D normal
DATATERMS
Image 𝑖
𝒖𝑖𝑗 Contour generator
preimage in 𝛀
(unknown)
c.g. point in 3D is 𝑀 𝒖𝑖𝑗; 𝑿𝑖
82

83. DATATERMS
Image 𝑖
𝒔𝑖𝑗, 𝒏𝑖𝑗
83
Camera
position
Silhouette:
𝐸𝑖
𝑠𝑖𝑙
=
𝑗=1
𝑆𝑖
𝒔𝑖𝑗 − 𝜋 𝜃𝑖, 𝑀 𝑢𝑖𝑗, 𝑿𝑖
2
Normal:
𝐸𝑖
𝑠𝑖𝑙
=
𝑗=1
𝑆𝑖
𝒏𝑖𝑗
0
− 𝑅 𝜃𝑖 𝑁 𝑢𝑖𝑗, 𝑿𝑖
2
Projection
e.g. Perspective

84. DATATERMS
Image 𝑖
𝒔𝑖𝑗, 𝒏𝑖𝑗
Linear Blend Shapes (PCA) Model:
𝑿𝑖 =
𝑘
𝛼𝑖𝑘 𝑩 𝑘
84
Silhouette:
𝐸𝑖
𝑠𝑖𝑙
=
𝑗=1
𝑆𝑖
𝒔𝑖𝑗 − 𝜋 𝜃𝑖, 𝑀 𝑢𝑖𝑗, 𝑿𝑖
2
Normal:
𝐸𝑖
𝑠𝑖𝑙
=
𝑗=1
𝑆𝑖
𝒏𝑖𝑗
0
− 𝑅 𝜃𝑖 𝑁 𝑢𝑖𝑗, 𝑿𝑖
2

85. Data fidelity
terms
𝑝 𝐼 𝑋𝑖; 𝑈
Gaussian shape
weights
Smooth
contour
Smooth Basis
𝑝 𝚯
85

86. 86

87. CONTINOUSOPTIMIZATION
 Can focus on this term to understand entire
optimization.
 Total number of residuals 𝑛 = number of silhouette points.
Say 300𝑁 (𝑁 = number of images) ≈ 10,000
 Total number of unknowns 2𝑛 + 𝐾𝑁 + 𝑚 where
𝑚 ≈ 3𝐾 × number of vertices ≈ 3,000
87

88. INITIALESTIMATEFORMEANSHAPE
This is true, but misleading
88

89. INITIALESTIMATEFORMEANSHAPE
True initial estimate: only the topology is really important.
But the easiest way to get the topology is to build a rough template.
89

90. INITIALESTIMATEFORMEANSHAPE
True initial estimate: only the topology is really important.
But the easiest way to get the topology is to build a rough template.
90

91. 91

92. EXAMPLERESULTS 92

93. EXAMPLERESULTS 93

94. OPTIMIZATION 94

95. NUMBEROFIMAGES 95
8 16 32

96. PARAMETERSENSITIVITY
“Pixel” terms: noise level params “Dimensionless” terms “Smoothness” terms
𝐸 = 𝑖=1
𝑛
𝐸𝑖
sil
+ 𝐸𝑖
norm
+ 𝐸𝑖
con
+ 𝑖=1
𝑛
𝐸𝑖
cg
+ 𝐸𝑖
reg
+ 𝝃 𝟎
𝟐
𝐸0
tp
+ 𝝃 𝐝𝐞𝐟
𝟐
𝑖=1
𝑛
𝐸 𝑚
tp
96

97. 97
サンプルコードを動かしてみた

98. 98
サンプルコードを動かしてみた

99. 99

100. 100

101. 101

102. 102
 Fitting Surface Models to Data について、抜粋して、ご紹介
 最適化には多種の手法があるが、研究段階や対象のモデルに
合わせて、適宜、利用するものを変えることが望ましい
 対象課題についての、なるべく単純なモデルを作れると、
最適化の戦略も練りやすい
 実行速度をリアルタイムにするため、モデル制約や行列の
スパース性を利用してやり、実現することも可能
まとめ

103. ありがとうございました
(^ω^) @sumisumith