The document summarizes Kyle Ingersoll's master's thesis defense presentation on vision-based multiple target tracking using recursive RANSAC (R-RANSAC). The presentation covers an overview of the R-RANSAC approach, improvements made to R-RANSAC including different data association methods and handling of highly maneuverable objects. It also compares R-RANSAC to other multiple target tracking filters and discusses potential improvements like incorporating tracker-sensor feedback and machine learning.

1. Click to edit Master title style
Vision-Based Multiple Target Tracking
Using Recursive-RANSAC
Kyle Ingersoll
February 25, 2015
Master’s Thesis Defense

2. February 25, 2015 Thesis DefenseKyle Ingersoll
Outline
 Overview of tracking approach
 The multiple target tracking (MTT) problem
 Random sample consensus (RANSAC)
 Recursive-RANSAC (R-RANSAC)
 Improvements to R-RANSAC
 Data association
 Highly maneuverable objects
 Comparison to the GM-PHD filter
 Improvements to video processing
 Tracker-sensor feedback
 Stationary object detection
 Tracking with machine learning

3. February 25, 2015 Thesis DefenseKyle Ingersoll
Overview of Tracking Approach

4. February 25, 2015 Thesis DefenseKyle Ingersoll
The MTT Problem
 Estimate the number of targets, the target states, and arrange
the targets into continuous, temporal sequences
Sensor
Processing
• Produce measurements
Data
Association
• Assign measurements to tracks
Filtering
• Estimate target states
• Smooth noisy measurements
Track
Management
• Track initiation
• Merge and prune redundant tracks
• Remove dead tracks

5. February 25, 2015 Thesis DefenseKyle Ingersoll
RANSAC[1]
 Randomly samples data points
 Creates models with those data points
 Stores model with the highest number of inliers
https://canvas.instructure.com/courses/743674/assignments/1929377

6. February 25, 2015 Thesis DefenseKyle Ingersoll
RANSAC
 Able to reject outliers

7. February 25, 2015 Thesis DefenseKyle Ingersoll
Recursive-RANSAC[2,3]
 Able to robustly track multiple targets in the
presence of gross errors
 Performs data association, filtering, and track
initiation steps
 Requires model of target motion
 Does not require a priori knowledge of number
of targets, birth and death times of targets, or
distribution of signal clutter

8. February 25, 2015 Thesis DefenseKyle Ingersoll
R-RANSAC Steps
With each measurement scan:
Perform following steps for each measurement:
1. Determine if measurement is an inlier to an existing model
a. If yes, use that measurement to update existing model with filter
b. If no, use that measurement to construct a new model using
RANSAC with the measurements in the current measurement
window
i. Store the new model data: consensus set, model parameters
Manage track set:
1. Merge similar models
2. Store models with the highest number of inliers
3. Output models that exceed good model threshold & lifetime
requirements

9. February 25, 2015 Thesis DefenseKyle Ingersoll
R-RANSAC Steps

10. February 25, 2015 Thesis DefenseKyle Ingersoll
Other RANSAC Variants
1. Sequential-RANSAC[4]
 Estimates multiple signals from a single batch of data
2. Multiple-Model RANSAC[5]
 Roughly estimates signal states for classification and
to improve data association
3. Incremental-RANSAC[6]
 Only estimates one model; hypotheses generated at
each time step using newly arrived features
4. KALMANSAC[7]
 Estimates an optimal state and set of inliers for a
single model at each time step with RANSAC

11. February 25, 2015 Thesis DefenseKyle Ingersoll
Improvements to R-RANSAC

12. February 25, 2015 Thesis DefenseKyle Ingersoll
R-RANSAC Modularity
Track
Initialization
• RANSAC-based method
Data
Association
• All neighbors data association
• Nearest neighbor data association
• Probabilistic data association
• Joint probabilistic data association
Filtering
• Kalman filter
• Extended Kalman filter
• Interacting multiple models
• Particle filter

13. February 25, 2015 Thesis DefenseKyle Ingersoll
Data Association
 All neighbors
 Update Kalman filter with all validated measurements
 Nearest neighbor
 Update Kalman filter with nearest validated measurement
 Probabilistic data association (PDA) [8]
 Calculate measurement association probabilities of all
validated measurements. Incorporate information about
clutter distribution, gate probability, and probability of
detection.
 Joint probabilistic data association (JPDA) [9]
 Jointly calculate measurement association probabilities
across all targets.

14. February 25, 2015 Thesis DefenseKyle Ingersoll
Data Association – Comparison
1
Data
Association,
Model MOTP MOTA
False
Positives
Track Label
Switches
Missed
Detections OSPA-T
Execution
Time Per
Frame
CV, AN 9.0686 0.7540 0.0695 0.0110 0.1656 44.2893 0.0171
CJ, AN 7.0594 0.7450 0.0876 0.0087 0.1587 42.0448 0.0169
CJ, NN 7.0674 0.7221 0.1014 0.0095 0.1670 42.3409 0.0180
CJ, PDA 6.6295 0.7593 0.0747 0.0077 0.1583 41.3469 0.0188
CJ, JPDA 6.6137 0.7686 0.0670 0.0084 0.1560 41.7940 0.0329
 All results, including execution times, are from the Matlab
implementations
 Execution times do not include image processing
 OSPA and OSPA-T results are averages of 10 runs

15. February 25, 2015 Thesis DefenseKyle Ingersoll
Data Association - Comparison
Nearest Neighbor
All neighbor
PDA
PDA
All Neighbors

16. February 25, 2015 Thesis DefenseKyle Ingersoll
CJ-PDA R-RANSAC Results
1

17. February 25, 2015 Thesis DefenseKyle Ingersoll
Highly Maneuverable Objects
 Replace the nearly-
constant velocity
(CV) model with the
nearly-constant jerk
(CJ) model[10]
 Negligible change in
computational time
 Able to retain
RANSAC-based track
initialization method

18. February 25, 2015 Thesis DefenseKyle Ingersoll
Interacting Multiple Models[11]

19. February 25, 2015 Thesis DefenseKyle Ingersoll
Interacting Multiple Models
 High Q ensures maneuverability and high R
ensures smooth tracks and useable measurement
association probabilities

20. February 25, 2015 Thesis DefenseKyle Ingersoll
Comparison with GM-PHD Filter[12]
R-RANSAC, CJ-IMMPDA GM-PHD Filter

21. February 25, 2015 Thesis DefenseKyle Ingersoll
Improvements to Video Processing

22. February 25, 2015 Thesis DefenseKyle Ingersoll
Sensor Processing – Computer vision
 Foreground detector/background subtraction methods model the
background and create a foreground mask
 Foreground mask is processed by a blob detector to identify
distinct, contiguous blobs
 Blob centroids used as position measurements

23. February 25, 2015 Thesis DefenseKyle Ingersoll
Tracker-Sensor Feedback Loop[13]
 Use information from the tracker to inform how
we perform sensor processing.
Sensor
Processing
Data
Association
Filtering
Track
Management
Tracker

24. February 25, 2015 Thesis DefenseKyle Ingersoll
Modified Foreground Detector
 Based upon the Gaussian mixture models
(GMM) architecture[14,15]
 Zero the background update rate of valid
target-associated pixels
 Target extent measured directly from
foreground mask
 Kalman filter-based minimum blob area
threshold
 Measurement velocity estimated by
matching features and clustering optical
flow vectors[16] using sequential-
RANSAC[4]

25. February 25, 2015 Thesis DefenseKyle Ingersoll
Merged Measurements
 Interacting targets may
produce a merged
measurement that does
not accurately describe
the position of either
target Position
Velocity

26. February 25, 2015 Thesis DefenseKyle Ingersoll
Video Processing, Results
Method MOTP MOTA
False
Positives
Track Label
Switches
Missed
Detections OSPA-T
Baseline 10.6307 0.8067 0.0074 0.0141 0.1718 48.3186
Position 10.8491 0.9254 0.0043 0.0048 0.0655 28.1280
Velocity 10.1756 0.9232 0.0000 0.0030 0.0738 27.5439
 Results are averaged over 10 simulations

27. February 25, 2015 Thesis DefenseKyle Ingersoll
Video Processing, Results

28. February 25, 2015 Thesis DefenseKyle Ingersoll
Stationary Object Detection
Pets 2006 S7-T6

29. February 25, 2015 Thesis DefenseKyle Ingersoll
Tracking with Machine Learning

30. February 25, 2015 Thesis DefenseKyle Ingersoll
Tracking with Machine Learning
 Learn probable target trajectories with the Sequence Model[17,18] and use that
information to improve tracking, especially during scenarios with infrequent
measurement updates and high proportion of clutter

31. February 25, 2015 Thesis DefenseKyle Ingersoll
Simulation Environment
 4 well-spaced tracks
 Time steps of 9, 13, and
17 seconds
 Progressively more
intense periods of clutter
measurements
 Kalman filter and
Sequence Model
measurement association
probabilities combined

32. February 25, 2015 Thesis DefenseKyle Ingersoll
Learning Results – 17 second time steps
LearningNo learning

33. February 25, 2015 Thesis DefenseKyle Ingersoll
Future Research
 R-RANSAC
 Particle filter or UKF implementations
 R-RANSAC with multiple agents
 Tracking-based control
 Video processing
 Moving camera foreground detector
 Learning
 Multiple belief models to improve data association
among interacting targets

34. February 25, 2015 Thesis DefenseKyle Ingersoll
References
1. Fischler, M. A., and Bolles, R. C., 1981. “Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated
Cartography.” Communications of the ACM, 24(6), June, pp. 381–395. 3, 7
2. Niedfeldt, P. C., 2014. “A Novel Multiple Target Tracking Algorithm in Clutter: Recursive-RANSAC.” PhD thesis, Brigham Young University. 3, 10, 55, 60
3. Niedfeldt, P., and Beard, R., 2013. “Recursive RANSAC: Multiple Signal Estimation with Outliers.” In Nonlinear Control Systems, Vol. 9, pp. 430–435. 10
4. Torr, P. H. S. “Geometric Motion Segmentation and Model Selection.”. 3, 71
5. Wang, C.-C., 2009. “Multiple-Model RANSAC for Ego-Motion Estimation in Highly Dynamic Environments.” In 2009 IEEE International Conference on
Robotics and Automation, IEEE, pp. 3531–3538. 3
6. Tanaka, K., and Kondo, E., 2006. “Incremental RANSAC for Online Relocation in Large Dynamic Environments.” In Proceedings 2006 IEEE International
Conference on Robotics and Automation, 2006. ICRA 2006., IEEE, pp. 68–75. 4
7. Vedaldi, A., Favaro, P., and Soatto, S., 2005. “KALMANSAC: Robust Filtering by Consensus.” In Tenth IEEE International Conference on Computer Vision
(ICCV’05) Volume 1, Vol. 1, IEEE, pp. 633–640 Vol. 1. 4
8. Bar-Shalom, Y., Daum, F., and Huang, J., 2009. “The Probabilistic Data Association Filter.” IEEE Control Systems Magazine, 29(6), Dec., pp. 82–100. 11, 35
9. Fortmann, T., Bar-Shalom, Y., and Scheffe, M., 1980. “Multi-Target Tracking Using Joint Probabilistic Data Association.” In 1980 19th IEEE Conference on
Decision and Control including the Symposium on Adaptive Processes, IEEE, pp. 807–812. 2, 36
10. Mehrotra, K., 1997. “Jerk Model for Tracking Highly Maneuvering Targets.” IEEE Transactions on Aerospace and Electronic Systems, 33(4), pp. 1094 – 1105.
45
11. Pitre, R. R., Jilkov, V. P., and Li, X. R. “A Comparative Study of Multiple-Model Algorithms for Maneuvering Target Tracking.” Proc. SPIE 5809, Signal
Processing, Sensor Fusion, and Target Recognition XIV, 549. 48
12. Pasha, S. A., Vo, B.-N., Tuan, H. D., and Ma, W.-K., 2009. “A Gaussian Mixture PHD Filter for Jump Markov System Models.” IEEE Transactions on
Aerospace and Electronic Systems, 45(3), July, pp. 919–936. 55
13. Pnevmatikakis, A., and Polymenakos, L., 2006. Machine Learning for Multimodal Interaction., Vol. 4299 of Lecture Notes in Computer Science Springer Berlin
Heidelberg, Berlin, Heidelberg. 67, 69
14. Stauffer, C., and Grimson, W., 1999. “Adaptive Background Mixture Models for Real-Time Tracking.” In Proceedings. 1999 IEEE Computer Society Conference
on Computer Vision and Pattern Recognition (Cat. No PR00149), Vol. 2, IEEE Comput. Soc, pp. 246–252. 62
15. P. Kaewtrakulpong, R. B. “An Improved Adaptive Background Mixture Model for Realtime Tracking with Shadow Detection.” 2nd European Workshop on
Advanced Video Based Surveillance Systems, AVBS01, VIDEO BASED SURVEILLANCE SYSTEMS: Computer Vision and Distributed Processing. 62
16. Kim, J.,Wang, X.,Wang, H., Zhu, C., and Kim, D., 2012. “Fast Moving Object Detection with Non-Stationary Background.” Multimedia Tools and Applications,
67(1), Apr., pp. 311–335. 71
17. Cook, K., Bryan, E., Yu, H., Bai, H., Seppi, K., and Beard, R., 2013. “Intelligent Cooperative Control for Urban Tracking.” Journal of Intelligent & Robotic
Systems, 74(1-2), Sept., pp. 251–267. 1, 89, 91
18. Wood, F., Gasthaus, J., Archambeau, C., James, L., and Teh, Y. W., 2011. “The Sequence Memoizer.” Communications of the ACM, 54(2), Feb., p. 91. 89