Robotics Gimbal Camera Calibration

1. 1
Jason Rebello, Ph.D Candidate
Supervisor : Dr. Steven L. Waslander
Doctoral Examination Committee (DEC)
April 28, 2021
Dynamic Camera Calibration & Degeneracy

2. Overview 
2
| Toronto Robotics and AI Laboratory
• Motivation
• Previous Contributions
• Autonomous Active Calibration using Next-Best-View
• Encoderless Calibration of DCCs
• Calibration using Non-overlapping Field of view
• Degeneracy Analysis for Encoderless DCCs
• Recent Contributions
• DC-VINS Calibration
• Minimal Parameterization
• Future Work
• Calibration on Real Hardware
• DC-VINS Observability and Sensitivity

3. Motivation
3
| Toronto Robotics and AI Laboratory
Motivation

4. Motivation Camera Clusters
• Accurate SLAM depends on perceived information
• Uneven feature distribution and Limited FOV
| Toronto Robotics and AI Laboratory
4
Aerial View

5. Motivation Camera Clusters
• Accurate SLAM depends on perceived information
• Uneven feature distribution and Limited FOV
• Static Camera Cluster (SCCs)
• Increased FOV & 360° feature tracking
• Increased computational resources & viewpoint
coupling
Static Camera Cluster
| Toronto Robotics and AI Laboratory
5

6. Motivation Camera Clusters
• Accurate SLAM depends on perceived information
• Uneven feature distribution and Limited FOV
• Static Camera Cluster (SCCs)
• Increased FOV & 360° feature tracking
• Increased computational resources & viewpoint
coupling
• Dynamic Camera Clusters (DCCs)
• Viewpoint manipulation
• Reduced computational load
• Available on most drones
• Smoother image transitions
| Toronto Robotics and AI Laboratory
6
Dynamic Camera
Static Cameras
Dynamic Camera Cluster
Static Camera Cluster

7. Motivation Static vs Dynamic Camera
| Toronto Robotics and AI Laboratory
7
Static Camera Dynamic Camera

8. Motivation Static vs Dynamic Camera
| Toronto Robotics and AI Laboratory
8
Static Camera Dynamic Camera
Need to Determine Time-Varying Extrinsic Calibration

9. Motivation SCC vs DCC Extrinsic Transformation
Static Camera Cluster (SCC)
Transformation between cameras is
constant
| Toronto Robotics and AI Laboratory
9

10. Motivation SCC vs DCC Extrinsic Transformation
Static Camera Cluster (SCC) Dynamic Camera Cluster (DCC)
Transformation between cameras is
constant
Transformation between cameras is time-
varying and depends on joint angle
values ( 𝜆 )
Dynamic Camera extrinsic transformation uses DH Parameters
| Toronto Robotics and AI Laboratory
10

11. Related Work DCC Calibration Procedure [Das, Waslander 2016]
Forward Projection Error:
Dynamic camera
Pixel measurement
Target point in
static camera
Transformation from
static to dynamic camera
Camera Projection
Known joint angle inputs
Estimation Parameters
| Toronto Robotics and AI Laboratory
11

12. Related Work DCC Calibration Procedure
Forward Projection Error:
Dynamic camera
Pixel measurement
Target point in
static camera
Transformation from
static to dynamic camera
Camera Projection
Known joint angle inputs
Estimation Parameters
| Toronto Robotics and AI Laboratory
12
Dynamic
Camera (DC)
Static
Camera
DC Image Plane
True detected
pixels
Projected
pixels
Target
Common
Points

13. Related Work DCC Calibration Procedure
Forward Projection Error:
Dynamic camera
Pixel measurement
Target point in
static camera
Transformation from
static to dynamic camera
Camera Projection
Known joint angle inputs
Estimation Parameters
Backward Projection Error:
Total Error:
Total measurement
sets
| Toronto Robotics and AI Laboratory
13

14. Previous Contributions
14
| Toronto Robotics and AI Laboratory
Previous
Contributions

15. Active Calibration  NBV Calibration [Rebello, Das, Waslander 2017]
Previous Calibration Approach Proposed Calibration Approach
Manual Measurement
Collection
Automatic Measurement
Collection
Random Sample Strategy Next Best View Strategy
15
| Toronto Robotics and AI Laboratory
Contribution 1: Autonomous Active Calibration Using Next-Best-View

16. Active Calibration  Proposed Approach
Goal : Select view-points to minimize overall covariance of estimated parameters,
Two Step Approach:
● Predict covariance of a mechanism configuration
16
| Toronto Robotics and AI Laboratory
Predict covariance matrix from
mechanism configuration

17. Active Calibration  Proposed Approach
Goal : Select view-points to minimize overall covariance of estimated parameters,
Two Step Approach:
● Predict covariance of a mechanism configuration
● Optimize over the configuration to find joint angles that best minimize the covariance
17
| Toronto Robotics and AI Laboratory
Predict covariance matrix from
mechanism configuration
Formulate NBV Cost
Compute Entropy of
Covariance Matrix

18. Active Calibration  Simulation Results
Mechanism Configuration ( # joints = # DOF) :
1 DOF 2 DOF 3 DOF
NBV approach generates minimum parameter covariance
11 9
8
9
9
5
6
5
5
| Toronto Robotics and AI Laboratory
18

19. Active Calibration  Hardware Setup
DCC consisting of static camera 𝓕s and
dynamic camera 𝓕d mounted on a gimbal
19
| Toronto Robotics and AI Laboratory
10
11
5
Real Hardware: 3 DOF

20. Encoderless Calibration  Encoderless Calibration [Choi, Rebello et al 2018]
● Loss function, reprojection error :
● Optimize for calibration parameters and encoder angles:
Reprojection
Error
Calibration
Parameters
Unknown joint
angles
| Toronto Robotics and AI Laboratory
20
Contribution 2: Encoderless Calibration of DCCs

21. Encoderless Calibration  Hardware Experiments
21
| Toronto Robotics and AI Laboratory
2 DOF Dynamic Camera Cluster
Custom Built Drone

22. OKVIS with Dynamic Camera Cluster
Encoderless Calibration  OKVIS
| Toronto Robotics and AI Laboratory
22

23. Multi-camera Calibration  Overview [Rebello, Fung and Waslander 2020]
DJI Matrice 210 with 3 DOF Dynamic
Camera Cluster with 2 Static Cameras
23
| Toronto Robotics and AI Laboratory
• Previous approaches
• Overlapping FOV
• Calibrations repeated
Contribution 3: DCC Calibration with Non-overlapping FOVs

24. Multi-camera Calibration  Cost Function
24
| Toronto Robotics and AI Laboratory
• Multi-camera calibration
• Pose-loop error, no overlap in FOV
• Identity residual
• Previous approaches
• Overlapping FOV
• Calibrations repeated
Static
Camera
DC Image Plane
Dynamic Camera
(DC)
Static Camera
True DC Location
Estimated DC
Location
Pixel Error
Pose-Loop Error
( )
Estimated
True

25. Multi-camera Calibration  Cost Function
25
| Toronto Robotics and AI Laboratory
• Multi-camera calibration
• Pose-loop error, no overlap in FOV
• Abstract out features
• Greater joint space excitation
• Non-overlapping FOV
• Works with different sensors
• Previous approaches
• Overlapping FOV
• Calibrations repeated
Static
Camera
DC Image Plane
Dynamic Camera
(DC)
Static Camera
True DC Location
Estimated DC
Location
Pixel Error
Pose-Loop Error
( )
Estimated
True

26. Multi-camera Calibration  Simulation Experiments
26
| Toronto Robotics and AI Laboratory
• Configurations [Pitch] [Yaw] limits
• Target [-20 : 20] [-20 : 20]
• Drone [-120 : 30] [-180 : 180]
• Full Configuration [-180 : 180] [-180 : 180]
• Simulated Details
• 0.2 pixel error intrinsics
• 100 measurements randomly sampled
• Different environments
Checkerboard Cube

27. Multi-camera Calibration  Sensitivity Analysis
27
| Toronto Robotics and AI Laboratory
• Identify Critical Parameters
• 3 levels of noise for encoders and 2
levels of noise for camera
• Encoders [0.1 , 3 , 7 degrees]
• Camera [0.3 , 1.2 pixels]
• 70 measurements for evaluation
• Better Cameras are more important
than high quality encoders

28. Degeneracy  Overview
| Toronto Robotics and AI Laboratory
28
• State of systems needs to be uniquely recoverable
• Eg. J1 axis slides along Z
• Analyze the jacobian of measurement equation
• Need full rank jacobian
• DCC calibration without encoders.
x y
z
?
Base Frame
Calibration
Degeneracy
J1
Contribution 4: Encoderless DCC calibration degeneracy

29. Degeneracy  Overview
| Toronto Robotics and AI Laboratory
29
x y
z
?
Base Frame
Calibration
Degeneracy
J2
J1
• State of systems needs to be uniquely recoverable
• Eg. J1 axis slides along Z
• Analyze the jacobian of measurement equation
• Need full rank jacobian
• DCC calibration without encoders.
Contribution 4: Encoderless DCC calibration degeneracy

30. Degeneracy Contribution
| Toronto Robotics and AI Laboratory
30
Static camera to Base Transform First joint encoder angle
End effector to Gimbal Transform Last joint encoder angle
Rotation Translation

31. Degeneracy Static to Base
| Toronto Robotics and AI Laboratory
31
Static camera to Base Transform First joint encoder angle

32. Degeneracy Static to Base
| Toronto Robotics and AI Laboratory
32
Static camera to Base Transform First joint encoder angle
=
=
=
Jacobian of first joint angle is
linear dependent on jacobian
of rotation angles from static
camera to base

33. Recent Contributions
33
| Toronto Robotics and AI Laboratory
Recent
Contributions

34. DC-VINS Overview [Rebello, Li and Waslander 2021]
34
| Toronto Robotics and AI Laboratory
Contribution 5: DC-VINS : Dynamic Camera Visual Inertial Sensor Calibration
• VIN Systems are popular
• Helps with high dynamic motion
• Dynamic Cameras allow smoother
image transition
• Calibration between Dynamic
Camera and IMU
• Online Estimation in flight
Static vs Dynamic Camera Images

35. DC-VINS Offline Calibration
35
| Toronto Robotics and AI Laboratory
DCC Calibration
Kalibr
IMU
• Dynamic to Static Camera - DCC Calibration
• Static Camera to IMU - Kalibr
DJI matrice with IMU, Dynamic and
Static Cameras

36. DC-VINS Offline Calibration
36
| Toronto Robotics and AI Laboratory
DCC Calibration
Kalibr
IMU
• Dynamic to Static Camera - DCC Calibration
• Static Camera to IMU - Kalibr
Advantages :
• Proven methods high accuracy
Disadvantages :
• Requires a static camera
• Performed with Fiducial Targets
• Only performed offline
• Time intensive DJI matrice with IMU, Dynamic and
Static Cameras

37. DC-VINS  Static State Vector
| Toronto Robotics and AI Laboratory
37
Static VINS

38. DC-VINS  Dynamic State Vector
| Toronto Robotics and AI Laboratory
38
Static VINS DC VINS
Joint Angles
Static Parameters

39. DC-VINS  Parameter Comparison
| Toronto Robotics and AI Laboratory
39
Static VINS DC VINS
• 6 parameters • 6 + 6 + 3L + LN
• N : Window Size (10)
• L : Number of Links (3)
• Total : 51 parameters
Joint Angles
Static Parameters

40. DC-VINS  DC Visual Measurement Equation
| Toronto Robotics and AI Laboratory
40
Different Angles
Residual
Back projection
DC chain
Vsual Measurement across two time-steps

41. DC-VINS  Experiments
41
| Toronto Robotics and AI Laboratory
Environment 1 Environment 2
Drone Setup

42. DC-VINS  Static vs Dynamic VINS
42
| Toronto Robotics and AI Laboratory
• Dynamic vs Static Camera Odometry
• Tested in 2 Environments and 3 runs
• Assume known calibration
• Aggressive aerial motion
T1 Trajectory
T3 Trajectory

43. DC-VINS  Offline Calibration
43
| Toronto Robotics and AI Laboratory
Kalibr
Calibration
DCC Calibration
• 2 minute bag for Kalibr calibration
• DCC calibration built map using OpenVSLAM
• 39 measurements with gimbal excitation
• Results in Table 1

44. DC-VINS  Online vs Offline DC-VINS Calibration
44
| Toronto Robotics and AI Laboratory
Online DC-VINS with calibration recovery

45. DC-VINS  Degeneracy Summary
45
| Toronto Robotics and AI Laboratory
Common

46. DC-VINS  Degeneracy Summary
46
| Toronto Robotics and AI Laboratory
Chain 1 Chain 2

47. DC-VINS  Degeneracy Summary
47
| Toronto Robotics and AI Laboratory
d Parameter Jacobian Base to IMU Jacoian

48. DC-VINS  Degeneracy Summary
48
| Toronto Robotics and AI Laboratory
Complete Base to IMU
Jacobian
Complete d Jacobian

49. Minimal parameterization Overparameterization
49
| Toronto Robotics and AI Laboratory
Contribution 6: Minimal Parameterization Dynamic Camera Calibration
(6) (6)
(12)
(4) (4) (4)

50. Minimal parameterization Overparameterization
50
| Toronto Robotics and AI Laboratory
Contribution 6: Minimal Parameterization Dynamic Camera Calibration
(6) (6)
(12) (4) (4) (4)
• Total number of parameters = 24
• Total number of degeneracies = 6
• Minimal number of parameters = 24 – 6 = 18
• Last joint to camera
10 Parameters, 4 degeneracies
• Second joint to IMU
10 Parameters, 2 degeneracies

51. Minimal parameterization Dynamic Camera to Last Joint
51
| Toronto Robotics and AI Laboratory
= 4 + 6 = 10 parameters, 4 degeneracies
Need to preserve order
DH matrix

52. Minimal parameterization Dynamic Camera to Last Joint
52
| Toronto Robotics and AI Laboratory
= 4 + 6 = 10 parameters, 4 degeneracies
Modified DH matrix (6 parameters)
Rotation and translation along y-axis

53. Minimal parameterization First joint to IMU or Static Camera
53
| Toronto Robotics and AI Laboratory
Old Parameterization
= 6 + 4 = 10 parameters, 2 degeneracies
Actual Orientation

54. Minimal parameterization First joint to IMU or Static Camera
54
| Toronto Robotics and AI Laboratory
Old Parameterization
= 6 + 4 = 10 parameters, 2 degeneracies
XI
I
YI
ZI
XB
YB
ZB
ZJ2
XJ2
YJ2
dBJ2=0
Ambiguous 𝜃
B
J2

55. Minimal parameterization First joint to IMU or Static Camera
55
| Toronto Robotics and AI Laboratory
Old Parameterization
= 6 + 4 = 10 parameters, 2 degeneracies
XI
I
YI
ZI
XB
YB
ZB
ZJ2
XJ2
YJ2
dBJ2=0
Ambiguous 𝜃
B
J2

56. Minimal parameterization First joint to IMU or Static Camera
56
| Toronto Robotics and AI Laboratory
Old Parameterization
= 6 + 4 = 10 parameters, 2 degeneracies
XB
YB
ZB
ZJ2
XJ2
YJ2
dBJ2=0
Ambiguous 𝜃
B
J2
XI
I
YI
ZI

57. Minimal parameterization First joint to IMU or Static Camera
57
| Toronto Robotics and AI Laboratory
New Parameterization
= 6 + 4 = 10 parameters, 2 degeneracies
XB
YB
ZB
ZJ2
XJ2
YJ2
dBJ2=0
Ambiguous 𝜃
B
J2
XI
I
YI
ZI
𝜃offset

58. Minimal parameterization First joint to IMU or Static Camera
59
| Toronto Robotics and AI Laboratory
Old Parameterization New Parameterization
= 6 + 4 = 10 parameters, 2 degeneracies
4-DOF parameterization

59. Future Research
61
| Toronto Robotics and AI Laboratory
Future
Research

60. Future Research Preliminary Real Hardware Results
62
| Toronto Robotics and AI Laboratory
Static Camera Dynamic Camera
Projection Before Calibration Projection Errors
195 Pixels

61. Future Research Preliminary Real Hardware Results
63
| Toronto Robotics and AI Laboratory
Projection After Calibration Pixel Error after Calibration
3 Pixels
Better Calibration requires more excitation which requires a Map
High Skew image

62. Future Research 
| Toronto Robotics and AI Laboratory
64
Main Camera Left Camera
DJI Matrice 210 with 3 DOF Dynamic Camera Cluster with 2 Static Cameras
Joint work with Angus Fung.

63. Future Research Sensitivity DC-VINS Calibration
65
| Toronto Robotics and AI Laboratory
S1
Si
• Sufficient excitation for calibration
• Observability aware trajectory planning
• Selection of informed trajectories
Contribution 7: DC-VINS Sensitivity and Observability

64. Publication List
• J. Rebello, A. Das and S. L. Waslander, “Autonomous Active Calibration of a Dynamic Camera Cluster using
Next-Best-View”, in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017.
• C. L. Choi, J. Rebello, L. Koppel, P. Ganti, A. Das and S. L. Waslander, “Encoderless Gimbal Calibration of
Dynamic Multi-Camera Clusters”, in IEEE International Conference on Robotics and Automation (ICRA), 2018.
• J. Rebello, A. Fung, S. L. Waslander, “AC/DCC : Accurate Calibration of Dynamic Camera Clusters for Visual
SLAM”, in IEEE International Conference on Robotics and Automation (ICRA), 2020.
• J. Rebello, C. Li, S. L. Waslander, “DC-VINS: Dynamic Camera Visual Inertial Navigation System with Online
Calibration”, in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021. (Submission)
• A. Das, J. Rebello and S. L. Waslander, “Automatic Calibration of Dynamic Camera Clusters using Next-Best-
View”, 2021.
• J. Rebello, A. Fung, S. L. Waslander, “AC/DCC : Accurate Calibration of Dynamic Camera Clusters for Visual
SLAM”, in IJRR pre-submission, 2021.
• M. Pitropov, D. Garcia, J. Rebello, M. Smart, C. Wang, K. Czarnecki and S. L. Waslander, “Canadian Adverse
Driving Conditions Dataset”, in arXiv:2001.10117, 2020
66
| Toronto Robotics and AI Laboratory

65. Thank you 
67
| Toronto Robotics and AI Laboratory
Thank You