This paper proposes a semantic structure from motion (SSFM) approach to jointly estimate camera parameters, 3D point clouds, and object poses from multiple images. SSFM formulates the problem as a joint likelihood maximization over camera, point, and object parameters. It uses a Markov chain Monte Carlo method for inference, initializing parameters from object detections and solving the high-dimensional optimization. The approach is evaluated on datasets of cars, office objects, and people, outperforming baselines in camera pose estimation and 3D object localization accuracy.

1. Semantic Structure From Motion
Sid Yingze Bao and Silvio Savarese
VISION LAB Electrical and Computer Engineering, University of Michigan at Ann Arbor
Source code and data: http://www.eecs.umich.edu/vision/projects/ssfm/index.html
Introduction Model Overview Results
Goal: Assumption: Car Person
{O,Q,C}=arg max P(q,u,o|C,O,Q) Comparison Baselines
Office
Estimate 3D location and pose of objects, 3D location Given camera hypothesis, objects
=arg max P(q,u|C,Q)P(o|C,O) - Camera Pose Est.: Bundler [1]
of points, and camera parameters from 2 or more images. and points are independent
- Object Detection: LSVM [2]
Point Likelihood P(q,u|C,Q)
1. Car Dataset [3] (available online)
- Images and Dense Lidar Points
- ~500 testing images in 10 scenarios
Motivation: Object Likelihood P(o|C,O) Cam. T. est. error v.s. baseline 3D object localization
0.3
- Most 3D reconstruction methods do 40
e (degree)
T 0.25
Recall
- Estimate 3D object likelihood by 2D Bundler 0.2 4 Cam
not povide semantic information. SSFM 0.15 3 Cam
projection appearance: 20
0.1 2 Cam
- Most recognition methds do not Conventional 0.05
1 Cam
False Positive per Image
0
provide geometry and camera pose. Structure From Motion 1 2 3 4
0
0 2 4 6 8 10
- We propose to solve these two problem jointly. 2D object detection
Advantages:
- Improve camera pose estimation, compared to feature-point-based SFM.
- Improve object detections given multiple images, compared to independently
detecting objects from each single images.
- Establish object correspondences across views.
SSFM Problem Formulation 2. Kinect Office Dataset (available online) Cam. T. est. error v.s. baseline
20 e (degree)
3D object localization
Q
- Images and calibrated Kinect 3D range data T Recall
2 Cam
1 Cam
Bundler
Measurements - Mouse, Monitor, and Keyboard 10
- q: point features (e.g. DOG+SIFT) - 500 images in 10 scenarios 0
SSFM False Positive per Image
O 0.4 0.6 0.8 1
- u: point matches (e.g. threshold test)
- o: 2D objects (e.g. [2])
o
Joint Likelihood Maximization
Main challenge: image 1 object det. image 2 object det.
Model Parameters (unknowns) High dimensionality of unknowns =>
q
azimuth=20
- C: camera (K is known)
zenith =10
C
o q Sample P(q,u,o|C,O,Q) with MCMC azimuth=90
zenith =5
- Q: 3D points (locations) azimuth=-20
zenith =9
azimuth=70
Parameter Initialization
zenith =10
- O: 3D objects (locations, poses, categories) 3. Person Dataset
C - Use object detection scale and pose to one of camera - A pair of stereo cameras
Intuition: initialize cameras relative poses pose initializations
- 400 image pairs in 10 scenarios
In addition to point features, measurements of objects across views provide add- - Theorem: camera parameters can be
itional geometrical constraints that allow to relate cameras and scene parameters. estimated given:
i) 3 objects with scale; ii) 2 objects with
pose; iii) 1 object with scale and pose.
Reference Monte Carlo Markov Chain
[1] N. Snavely, S. M. Seitz, and R. S. Szeliski. Modeling the world from internet photo collections. IJCV. 2008. - Sampling starts from different initializations
[2] P. Felzenszwalb, R. Girshick, D. McAllester, and D. Ramanan. Object detection with discriminatively trained
- Proposal distribution P(q,u,o|C,O,Q) Ackowledgement
part based models. IEEE Transactions on Pattern Analysis and Machine Intelligence of Pattern Analysis, 2009. one of camera poses and objects initializations
[3] Gaurav Pandey, James McBride,,and Ryan Eustice,.Ford campus vision and lidar data set. International Journal - Combine all samples to identify the maximum We acknowledge the support of NSF CAREER #1054127 and the Gigascale Systems Research Center.
of Robotics Research. 2011 We thank Mohit Bagra for collecting the Kinect dataset and Min Sun for helpful feedback.