This slide explains the detail of the implementation of MatLab Camera Calibration Toolbox for stereo camera pair developed by Bouguet.

1. Theory of Bouguet’s MatLab Camera Calibration
Toolbox: Stereo
Yuji Oyamada
1 HVRL, University
2 Chair for Computer Aided Medical Procedure (CAMP)
Technische Universit¨t M¨nchen
a u
June 5, 2012

2. Variables Optimization
Variables
Assumption:
• N cameras observing L key-points resulting M images for each
sequence.
• The cameras are fixed meaning relative positions among them
are same through the sequence.
Variables:
• aj : Intrinsic parameters of j-th camera.
• bij : Extrinsic parameters of i-th image of j-th camera.
• xijk : k-th key-point of i-th image of j-th camera.
where
• j = 1, . . . , N
• i = 1, . . . , M
• k = 1, . . . , Lij

3. Variables Optimization
Variables
The set of extrinsic parameters {bij } has redundancy because the
cameras are fixed.
Introduce another variable r:
• bi : Extrinsic parameters of i-th image of 1st camera.
• rj : Extrinsic parameters of j-th camera w.r.t. 1st camera.
Note that r1 is equivalent to identity matrix.

4. Variables Optimization
Variables
We have observation vector x and parameters vector p where
x = (x111 , . . . , x11L11 , x1N1 , . . . , x1NL1N , . . . , xMN1 , . . . , xMNLMN )
= (x11 , x1N , . . . , xMN , . . . , xMN )
p = (a1 , . . . , aN , r2 , . . . , rN , b1 , . . . , bM )
where xij = (xij1 , . . . , xijLij )

5. Variables Optimization
Non-linear optimization
Finds optimal parameters p as
M N Lij
a r ˆ
{{ˆj }{ˆj }{bi }} = arg min vijk dist(ˆijk , xijk )2
x
{aj }{rj }{bi }
i=1 j=1 k=1
where
• ˆijk = Q(aj , bi ) denotes a reprojected point of xijk with
x
parameters aj and bi ,
• a visibility term vijk = 1 iff k-th point is visible in i-th image
observed by j-th camera.

6. Variables Optimization
Normal equations
J Jδ = −J
where
∂ˆ
x ∂ˆ ∂ˆ ∂ˆ
x x x
J= = = [ARB]
∂p ∂a ∂r ∂b
∂ˆ
x ∂ˆ
x ∂ˆ
x
A= ,R = ,B =
∂a ∂r ∂b
∂ˆij
x ∂ˆij
x ∂ˆij
x
Aij = , Rij = , Bij =
∂aj ∂rj ∂bi

7. Variables Optimization
Structure of Jacobian matrix J

8. Variables Optimization
Sparse LM
• LM is suitable for minimization w.r.t. a small number of
parameters.
• The central step of LM, solving the normal equations,
• has complexity N 3 in the number of parameters and
• is repeated many times.
• The normal equation matrix has a certain sparse block
structure.

9. Variables Optimization
Sparse LM
• Let p ∈ RM be the parameter vector that is able to be
partitioned into parameter vectors as p = (a , b ) .
• Given a measurement vector x ∈ RN
• Let x be the covariance matrix for the measurement vector.
• A general function f : RM → RN takes p to the estimated
measurement vector ˆ = f (p).
x
• denotes the difference x − ˆ between the measured and the
x
estimated vectors.

10. Variables Optimization
Sparse LM
The set of equations Jδ = solved as the central step in the LM
has the form
δa
Jδ = [A|B] = .
δb
−1 −1
Then, the normal equations J x Jδ = J x to be solved
at each step of LM are of the form
−1 −1 −1
A x A A x B δa A x
−1 −1 = −1
B x A B x B
δb B x

11. Variables Optimization
Sparse LM
Let
−1
• U=A x A
−1
• W=A x B
−1
• V=B x B
and ·∗ denotes augmented matrix by λ.
The normal equations are rewritten as
U∗ W δa A
=
W V∗ δb B
∗ ∗−1
U − WV W 0 δa A − WV∗−1 B
→ =
W V∗ δb B
This results in the elimination of the top right hand block.

12. Variables Optimization
Sparse LM
The top half of this set of equations is
(U∗ − WV∗−1 W )δa = A − WV∗−1 B
Subsequently, the value of δa may be found by back-substitution,
giving
V ∗ δb = B − W δa

13. Variables Optimization
Sparse LM
p = (a , b ) , where a = (fc , cc , alpha c , kc ) and
b = ({omc i Tc i })
The Jacobian matrix is
∂ˆ
x ∂ˆ ∂ˆ
x x
J= = , = [A, B]
∂p ∂a ∂b
where
∂ˆ
x ∂ˆ ∂ˆ
x x ∂ˆ
x ∂ˆ
x
= , , ,
∂a ∂fc ∂cc ∂alpha c ∂kc
∂ˆ
x ∂ˆx ∂ˆ
x ∂ˆ
x ∂ˆ
x ∂ˆ
x ∂ˆ
x
= , ,··· , ··· ,
∂b ∂omc 1 ∂Tc 1 ∂omc i ∂Tc i ∂omc N ∂Tc N

14. Variables Optimization
Sparse LM
The normal equation is rewritten as
A A
A B + λI ∆p = − x
B B
 N N
 
 Ai Ai Ai Bi  
 + λI ∆p = − A x
  
i=1
→  N i=1
N B x
  
Bi Ai Bi Bi
  
i=1 i=1

15. Variables Optimization
Sparse LM
N
 
 A i Ai A1 B1 · · · Ai Bi ··· AN BN 
 
 i=1 

 B1 A1 B1 B1 

J J=
 .
. .. 
 . . 


 Bi Ai Bi Bi 

 .
. .. 
 . . 
BN AN BN BN

16. Variables Optimization
Sparse LM
N
 
 Ai x 
 
 i=1 

 B1 x


J =
 .
.

x . 
 

 Bi x


 .
.

 . 
BN x

17. Variables Optimization
Sparse LM
When each image has different number of corresponding points
(Mi = Mj , if i = j), each Ai and Bi have different size as

18. Variables Optimization
Sparse LM
However, the difference does not matter because
A A ∈ Rdint ×dint
A B ∈ Rdint ×dex
B A ∈ Rdex ×dint
B B ∈ Rdex ×dex
where dint denotes dimension of intrinsic params and dex denotes
dimension of extrinsic params.