notes surveying adjustments

1. Non-Linear Least Squares and
Sparse Matrix Techniques:
Fundamentals
Richard Szeliski
Microsoft Research
UW-MSR Course on
Vision Algorithms
CSE/EE 577, 590CV, Spring 2004

2. 4/30/2004 NLS and Sparse Matrix Techniques 2
Readings
• Press et al., Numerical Recipes, Chapter 15 (Modeling of Data)
• Nocedal and Wright, Numerical Optimization, Chapter 10
(Nonlinear Least-Squares Problems, pp. 250-273)
• Shewchuk, J. R. An Introduction to the Conjugate Gradient
Method Without the Agonizing Pain.
• Bathe and Wilson, Numerical Methods in Finite Element
Analysis, pp.695-717 (sec. 8.1-8.2) and pp.979-987 (sec. 12.2)
• Golub and VanLoan, Matrix Computations. Chapters 4, 5, 10.
• Nocedal and Wright, Numerical Optimization. Chapters 4 and 5.
• Triggs et al., Bundle Adjustment – A modern synthesis.
Workshop on Vision Algorithms, 1999.

3. 4/30/2004 NLS and Sparse Matrix Techniques 3
Outline
Nonlinear Least Squares
• simple application (motivation)
• linear (approx.) solution and least squares
• normal equations and pseudo-inverse
• LDLT, QR, and SVD decompositions
• correct linearization and Jacobians
• iterative solution, Levenberg-Marquardt
• robust measurements

4. 4/30/2004 NLS and Sparse Matrix Techniques 4
Outline
Sparse matrix techniques
• simple application (structure from motion)
• sparse matrix storage (skyline)
• direct solution: LDLT with minimal fill-in
• larger application (surface/image fitting)
• iterative solution: gradient descent
• conjugate gradient
• preconditioning

5. Non-linear Least Squares

6. 4/30/2004 NLS and Sparse Matrix Techniques 6
Triangulation – a simple example
Problem: Given some
image points {(uj,vj)}
in correspondence across
two or more images (taken from
calibrated cameras cj),
compute the 3D location X
cj
X
(uj,vj)

7. 4/30/2004 NLS and Sparse Matrix Techniques 7
Image formation equations
u
(Xc,Yc,Zc)
uc
f

8. 4/30/2004 NLS and Sparse Matrix Techniques 8
Simplified model
Let R=I (known rotation),
f=1, Y = vj = 0 (flatland)
How do we solve this set
of equations (constraints)
to find the best (X,Z)? (xj,zj)
X
uj

9. 4/30/2004 NLS and Sparse Matrix Techniques 9
“Linearized” model
Bring the denominator over to
the LHS
or
(Measures horizontal distance
to each line equation.)
How do we solve this set of
equations (constraints)?
(xj,zj)
X
uj

10. 4/30/2004 NLS and Sparse Matrix Techniques 10
Linear regression
Overconstrained set of linear
equations
or
Jx = r
where
Jj0=1, Jj1 = -uj
is the Jacobian and
rj = xj-ujzj
is the residual
(xj,zj)
X
uj

11. 4/30/2004 NLS and Sparse Matrix Techniques 11
Normal Equations
How do we solve Jx = r?
Least squares: arg minx ║Jx-r║2
E = ║Jx-r║2 = (Jx-r)T(Jx-r)
= xTJTJx – 2xTJTr – rTr
∂E/∂x = 2(JTJ)x – 2JTr = 0
(JTJ)x = JTr normal equations
A x = b ” (A is Hessian)
x = [(JTJ)-1JT]r pseudoinverse

12. 4/30/2004 NLS and Sparse Matrix Techniques 12
LDLT factorization
Factor A = LDLT, where L is lower triangular with
1s on diagonal, D is diagonal
How?
L is formed from columns of Gaussian
elimination
Perform (similar) forward and backward
elimination/substitution
LDLTx = b, DLTx = L-1b, LTx = D-1L-1b,
x = L-TD-1L-1b

13. 4/30/2004 NLS and Sparse Matrix Techniques 13
LDLT factorization – details

14. 4/30/2004 NLS and Sparse Matrix Techniques 14
LDLT factorization – details

15. 4/30/2004 NLS and Sparse Matrix Techniques 15
LDLT factorization – details

16. 4/30/2004 NLS and Sparse Matrix Techniques 16
LDLT factorization – details

17. 4/30/2004 NLS and Sparse Matrix Techniques 17
LDLT factorization – details

18. 4/30/2004 NLS and Sparse Matrix Techniques 18
LDLT factorization – details

19. 4/30/2004 NLS and Sparse Matrix Techniques 19
LDLT factorization – details

20. 4/30/2004 NLS and Sparse Matrix Techniques 20
LDLT factorization – details

21. 4/30/2004 NLS and Sparse Matrix Techniques 21
LDLT factorization – details

22. 4/30/2004 NLS and Sparse Matrix Techniques 22
LDLT factorization – details

23. 4/30/2004 NLS and Sparse Matrix Techniques 23
LDLT factorization – details

24. 4/30/2004 NLS and Sparse Matrix Techniques 24
LDLT and Cholesky
Variant: Cholesky: A = GGT, where G = LD1/2
(involves scalar √)
Advantages: more stable than Gaussian
elimination
Disadvantage: less stable than QR: (cond. #)2
Complexity: (m+n/3)n2 flops

25. 4/30/2004 NLS and Sparse Matrix Techniques 25
QR decomposition
Alternative solution for Jx = r
Find an orthogonal matrix Q s.t.
J = Q R, where R is upper triangular
Q R x = r
R x = QTr solve for x using back subst.
Q is usu. computed using Householder matrices,
Q = Q1…Qm, Qj = I – βvjvj
T
Advantages: sensitivity / condition number
Complexity: 2n2(m-n/3) flops

26. 4/30/2004 NLS and Sparse Matrix Techniques 26
SVD
Most stable way to solve system Jx = r.
J = UTΣ V, where U and V are orthogonal
Σ is diagonal (singular values)
Advantage: most stable (very ill conditioned
problems)
Disadvantage: slowest (iterative solution)

27. 4/30/2004 NLS and Sparse Matrix Techniques 27
“Linearized” model – revisited
Does the “linearized” model
which measures horizontal
distance to each line give the
optimal estimate?
No!
(xj,zj)
X
uj

28. 4/30/2004 NLS and Sparse Matrix Techniques 28
Properly weighted model
We want to minimize errors in the
measured quantities
Closer cameras (smaller
denominators) have more
weight / influence.
Weight each “linearized” equation
by current denominator?
(xj,zj)
X
uj

29. 4/30/2004 NLS and Sparse Matrix Techniques 29
Optimal estimation
Feature measurement equations
Likelihood of (X,Z) given {ui,xj,zj}

30. 4/30/2004 NLS and Sparse Matrix Techniques 30
Non-linear least squares
Log likelihood of (x,z) given {ui,xj,zj}
How do we minimize E?
Non-linear regression (least squares), because
ûi are non-linear functions of {ui,xj,zj}

31. 4/30/2004 NLS and Sparse Matrix Techniques 31
Levenberg-Marquardt
Iterative non-linear least squares
• Linearize measurement equations
• Substitute into log-likelihood equation:
quadratic cost function in (Δx,Δz)

32. 4/30/2004 NLS and Sparse Matrix Techniques 32
Levenberg-Marquardt
Linear regression (sub-)problem:
with
ûi
Similar to weighted regression, but not quite.

33. 4/30/2004 NLS and Sparse Matrix Techniques 33
What if it doesn’t converge?
• Multiply diagonal by (1 + λ), increase λ
until it does
• Halve the step size (my favorite)
• Use line search
• Other trust region methods
[Nocedal & Wright]
Levenberg-Marquardt

34. 4/30/2004 NLS and Sparse Matrix Techniques 34
Other issues:
• Uncertainty analysis: covariance Σ = A-1
• Is maximum likelihood the best idea?
• How to start in vicinity of global minimum?
• What about outliers?
Levenberg-Marquardt

35. 4/30/2004 NLS and Sparse Matrix Techniques 35
Robust regression
Data often have outliers (bad measurements)
• Use robust penalty applied
to each set of joint
measurements
[Black & Rangarajan, IJCV’96]
• For extremely bad data, use random sampling
[RANSAC, Fischler & Bolles, CACM’81]

36. Sparse Matrix Techniques
Direct methods

37. 4/30/2004 NLS and Sparse Matrix Techniques 37
Structure from motion
Given many points in correspondence across
several images, {(uij,vij)}, simultaneously
compute the 3D location Xi and camera (or
motion) parameters (K, Rj, tj)
Two main variants: calibrated, and uncalibrated
(sometimes associated with Euclidean and
projective reconstructions)

38. 4/30/2004 NLS and Sparse Matrix Techniques 38
Bundle Adjustment
Simultaneous adjustment of bundles of rays
(photogrammetry)
What makes this non-linear minimization hard?
• many more parameters: potentially slow
• poorer conditioning (high correlation)
• potentially lots of outliers
• gauge (coordinate) freedom

39. 4/30/2004 NLS and Sparse Matrix Techniques 39
Simplified model
Again, R=I (known rotation),
f=1, Z = vj = 0 (flatland)
This time, we have to solve
for all of the parameters
{(Xi,Zi), (xj,zj)}. (xj,zj)
Xi
uij

40. 4/30/2004 NLS and Sparse Matrix Techniques 40
Lots of parameters: sparsity
Only a few entries in Jacobian are non-zero

41. 4/30/2004 NLS and Sparse Matrix Techniques 41
Sparse LDLT / Cholesky
First used in finite element analysis [Bathe…]
Applied to SfM by [Szeliski & Kang 1994]
structure | motion fill-in

42. 4/30/2004 NLS and Sparse Matrix Techniques 42
Skyline storage [Bathe & Wilson]

43. 4/30/2004 NLS and Sparse Matrix Techniques 43
Sparse matrices–common shapes
Banded (tridiagonal), arrowhead, multi-banded
: fill-in
Computational complexity: O(n b2)
Application to computer vision:
• snakes (tri-diagonal)
• surface interpolation (multi-banded)
• deformable models (sparse)

44. 4/30/2004 NLS and Sparse Matrix Techniques 44
Sparse matrices – variable reordering
Triggs et al. – Bundle Adjustment

45. Sparse Matrix Techniques
Iterative methods

46. 4/30/2004 NLS and Sparse Matrix Techniques 46
Two-dimensional problems
Surface interpolation and Poisson blending

47. 4/30/2004 NLS and Sparse Matrix Techniques 47
Poisson blending

48. 4/30/2004 NLS and Sparse Matrix Techniques 48
Poisson blending
→ multi-banded (sparse) system

49. 4/30/2004 NLS and Sparse Matrix Techniques 49
One-dimensional example
Simplified 1-D height/slope interpolation
tri-diagonal system (generalized snakes)

50. 4/30/2004 NLS and Sparse Matrix Techniques 50
Direct solution of 2D problems
Multi-banded Hessian
: fill-in
Computational complexity: n x m image
O(nm m2)
… too slow!

51. 4/30/2004 NLS and Sparse Matrix Techniques 51
Iterative techniques
Gauss-Seidel and Jacobi
Gradient descent
Conjugate gradient
Non-linear conjugate gradient
Preconditioning
… see Shewchuck’s TR

52. 4/30/2004 NLS and Sparse Matrix Techniques 52
Conjugate gradient
… see Shewchuck’s TR for rest of notes …

53. 4/30/2004 NLS and Sparse Matrix Techniques 53
Iterative vs. direct
Direct better for 1D problems and relatively
sparse general structures
• SfM where #points >> #frames
Iterative better for 2D problems
• More amenable to parallel (GPU?)
implementation
• Preconditioning helps a lot (next lecture)

54. 4/30/2004 NLS and Sparse Matrix Techniques 54
Monday’s lecture (Applications)
Preconditioning
• Hierarchical basis functions (wavelets)
• 2D applications:
interpolation, shape-from-shading,
HDR, Poisson blending,
others (rotoscoping?)

55. 4/30/2004 NLS and Sparse Matrix Techniques 55
Monday’s lecture (Applications)
Structure from motion
• Alternative parameterizations (object-
centered)
• Conditioning and linearization problems
• Ambiguities and uncertainties
• New research: map correlation