This document summarizes a framework called TraMaL that was developed to acquire 3D motion trajectories of large numbers of swarming animals. TraMaL stands for Tracking-Matching-Linking and uses a three step process to obtain 4D (3D+time) trajectories for each individual in a swarm. The framework was able to successfully obtain near real-time 3D flying trajectories for hundreds of fruit flies. Evaluation on simulated and real-world datasets showed the framework can effectively handle challenges like occlusions and provide high quality reconstructed trajectories. Future work aims to improve the matching model and apply the framework to other swarming species.

1. Acquiring 3D Motion Trajectories of Large
Numbers of Swarming Animals
Fudan University
HaishanWu
8/27/2009

2. Social behavior: a spectacular sight of
nature

3. Scientists are interested in this
phenomenon
 Nature Method 2009 PNAS 2008 Nature 2005
 PRL 1995 Siggraph 1987

4. They need 4D motion trajectories!
 4D(3D+t) trajectory of each individual is imperative:
How they What is the flock
interact with property?
each other?
How do they make collision
avoidance decision?

5. Our results!
 We successfully obtained 3D flying trajectories of
Drosophila melanogaster (fruit fly) group comprising
hundreds of individuals

6. Multi-view Geometry Technique

7. It is , however, a challenging problem 
 Animal groups with collective action contain large
variable number of interacting individuals
 Moving targets themselves in a swarm usually
resemble each other in appearance
Tracking or matching
Frequent occlusions failures will lead to the
make event state-of-the broken or
art tracking method failed incomplete
trajectories in 3D
space
Appearance feature
will be also ineffectual for tracking or
matching across various views

8. TraMaL: Our framework to solve these
challenges
 TraMaL: Tracking-Matching-Linking

9. Novelties and contributions of our
approach
 Each of the three components of this framework can be
modeled as linear assignment problem (LAP), and our
method works in near real time.
 Our computationally efficient data association method, is
able to deal with large variable number of targets.
 MECL(maximum epipolar co-motion length) is used to
match the trajectories . It is able to handle tracking errors
and calibrations errors
 Linking model incorporate motion and temporal information
to make the final trajectories as complete as possible.

10. Related work
 Many complex models for tracking: JPDA,
MHT… Exponential complexity is a serious
disadvantage .
Most related papers:
 CVPR 2007: cluster based method for data
association: the accuracy of the tracking
trajectories is not reported
 Nature Method 2008: robust model for
SPT(single particle tracking): their data
association method did not consider particle true
and temporal disappearance, their track fragments
tend to incomplete.

11. Related work
 Numerous methods for stereo matching. A
survey is available in CVPR 2007. But
most the them combine the appearance
feature and epipolar constrains
Most related paper:
 ICCV 2007: REM will discard many potential
matching candidates (as shown in
experiments)
 IJCV 2006: aim to align video sequences
captured in various cameras, which is
achieved by matching partial trajectories of
several moving objects

12. Related work: other fields
 Computational fluid dynamics: A Spatial-
Temporal Matching Algorithm for 3D Particle
TrackingVelocimetry. PhD thesis : Reconstruct
from 4 different views, then track particles in
3D object space. Trajectories will be broken
frequently because of matching and tracking
ambiguities.
 Biology community:
 A SimpleVision-Based Algorithm for Decision Making in
Flying Drosophila ,Current Biology 2008: 5
cameras to track 20 fruit flies. But as reported,
their approach is only able to track up to 2 targets
simultaneously
 The STARFLAG handbook on collective animal behavior
, Animal Behaviour,2008: obtained the 3D
coordinates of a large bird flock, but 4D
trajectories is unavailable.

13. Our method: TraMaL
Tracking
 2 steps: single particle state updating and data
association
 State updating: Markov assumption and alpha-beta
filter (simple, yet efficient)

14. Tracking, the first LAP (linear
assignment problem )
 The first LAP is:
To handle these
difficulties:

15. Matching, the second LAP
 Now our problem is how to assign (match) a 2D trajectory in one view to
that of another view, which can be also formulated as a LAP .
 The appearance feature of targets, however, is useless in our experiments,
so how to define the matching cost?
 Observations:
 Two true matching trajectories will submit to epipolar constrains
 The longer the length of matched trajectories, the larger the possibility of
correct matching is.

16. Matching cost function
 MECL(maximum epipolar co-motion length)
For a rectified image pair in two different views:

17. Matching cost function
 Solve LAP recursively :

18. Matching overlap
 One observation:

19. Linking, the third LAP
 3D trajectories can be recovered by
using multi-view geometry technique.
 However, tracking errors make
these trajectories broken into
segments
 We solve this problem by formulating it
as pairwise tracklet matching problem,
namely a LAP

20. Linking, cost function
 We incorporate temporal and kinematic information into
linking cost.
 temporal information
 kinematic information

21. Results in simulating data sets
 Simulated particle swarms:
 Tracking Performance:
 Parameters:
1. : depends on the
intensity of noise
2. : determined in an
adaptive way
3. : 1-5 frame

22. Results in simulating data sets
 Matching Performance:
 Parameters:
 : depends on accuracy of camera calibration

23. Results in simulating data sets
 Linking Performance
 Parameters:
1. For non-overlap
candidates, maximum
frame difference
equals to
2. For overlap candidates,
maximum overlap
length also equals to

24. Results in simulating data sets
 Compared to:
 Evaluation metrics:
 Trajectory completeness factor (TCF), which measures on average the
ratio of a ground truth trajectory length covered by the reconstructed
trajectories. TCF is 100% when all reconstructed trajectories are correct.
 Trajectory fragmentation factor (TFF), which measures on average
the number of acquired trajectories used to match one ground truth
trajectory. The ideal score of TFF is 1, the larger this value is, the worse the
performance on maintaining particle identity.

25. Results in simulating data sets

26. Results in real world datasets
 We use our method to obtain 3D flying trajectories of
Drosophila melanogaster (fruit fly) group comprising
hundreds of individuals (more than 150 detected targets in
each frame).
 Experiment setup:
 The fruit flies flied freely in an acrylic box of size 35cm by 35cm by
25cm, where the background was illuminated by white plane lights. Two
synchronized and calibrated Sony HVR-V1C video cameras working in
high speed mode at 200fps were used to capture the scene from
different views. The image resolution was 960×540 and 200 frames
were captured for the experiments.

27. Results in real world datasets
 Near 400 4D trajectories are obtained
All the obtained trajectories Some longer trajectories
We hope our results could lead to deeper understanding of
fly social behavior of fruit fly group.

28. Future work
 Build an advanced probabilistic model for MECL
 Test our framework on bird flock or fish school data sets
 Maybe we have a chance to analyze the trajectories and
discover something interesting