3. Scientists are interested in this
phenomenon
Nature Method 2009 PNAS 2008 Nature 2005
PRL 1995 Siggraph 1987
4. They need 4D motion trajectories!
4D(3D+t) trajectory of each individual is imperative:
How they What is the flock
interact with property?
each other?
How do they make collision
avoidance decision?
5. Our results!
We successfully obtained 3D flying trajectories of
Drosophila melanogaster (fruit fly) group comprising
hundreds of individuals
7. It is , however, a challenging problem
Animal groups with collective action contain large
variable number of interacting individuals
Moving targets themselves in a swarm usually
resemble each other in appearance
Tracking or matching
Frequent occlusions failures will lead to the
make event state-of-the broken or
art tracking method failed incomplete
trajectories in 3D
space
Appearance feature
will be also ineffectual for tracking or
matching across various views
9. Novelties and contributions of our
approach
Each of the three components of this framework can be
modeled as linear assignment problem (LAP), and our
method works in near real time.
Our computationally efficient data association method, is
able to deal with large variable number of targets.
MECL(maximum epipolar co-motion length) is used to
match the trajectories . It is able to handle tracking errors
and calibrations errors
Linking model incorporate motion and temporal information
to make the final trajectories as complete as possible.
10. Related work
Many complex models for tracking: JPDA,
MHT… Exponential complexity is a serious
disadvantage .
Most related papers:
CVPR 2007: cluster based method for data
association: the accuracy of the tracking
trajectories is not reported
Nature Method 2008: robust model for
SPT(single particle tracking): their data
association method did not consider particle true
and temporal disappearance, their track fragments
tend to incomplete.
11. Related work
Numerous methods for stereo matching. A
survey is available in CVPR 2007. But
most the them combine the appearance
feature and epipolar constrains
Most related paper:
ICCV 2007: REM will discard many potential
matching candidates (as shown in
experiments)
IJCV 2006: aim to align video sequences
captured in various cameras, which is
achieved by matching partial trajectories of
several moving objects
12. Related work: other fields
Computational fluid dynamics: A Spatial-
Temporal Matching Algorithm for 3D Particle
TrackingVelocimetry. PhD thesis : Reconstruct
from 4 different views, then track particles in
3D object space. Trajectories will be broken
frequently because of matching and tracking
ambiguities.
Biology community:
A SimpleVision-Based Algorithm for Decision Making in
Flying Drosophila ,Current Biology 2008: 5
cameras to track 20 fruit flies. But as reported,
their approach is only able to track up to 2 targets
simultaneously
The STARFLAG handbook on collective animal behavior
, Animal Behaviour,2008: obtained the 3D
coordinates of a large bird flock, but 4D
trajectories is unavailable.
13. Our method: TraMaL
Tracking
2 steps: single particle state updating and data
association
State updating: Markov assumption and alpha-beta
filter (simple, yet efficient)
14. Tracking, the first LAP (linear
assignment problem )
The first LAP is:
To handle these
difficulties:
15. Matching, the second LAP
Now our problem is how to assign (match) a 2D trajectory in one view to
that of another view, which can be also formulated as a LAP .
The appearance feature of targets, however, is useless in our experiments,
so how to define the matching cost?
Observations:
Two true matching trajectories will submit to epipolar constrains
The longer the length of matched trajectories, the larger the possibility of
correct matching is.
16. Matching cost function
MECL(maximum epipolar co-motion length)
For a rectified image pair in two different views:
19. Linking, the third LAP
3D trajectories can be recovered by
using multi-view geometry technique.
However, tracking errors make
these trajectories broken into
segments
We solve this problem by formulating it
as pairwise tracklet matching problem,
namely a LAP
20. Linking, cost function
We incorporate temporal and kinematic information into
linking cost.
temporal information
kinematic information
21. Results in simulating data sets
Simulated particle swarms:
Tracking Performance:
Parameters:
1. : depends on the
intensity of noise
2. : determined in an
adaptive way
3. : 1-5 frame
22. Results in simulating data sets
Matching Performance:
Parameters:
: depends on accuracy of camera calibration
23. Results in simulating data sets
Linking Performance
Parameters:
1. For non-overlap
candidates, maximum
frame difference
equals to
2. For overlap candidates,
maximum overlap
length also equals to
24. Results in simulating data sets
Compared to:
Evaluation metrics:
Trajectory completeness factor (TCF), which measures on average the
ratio of a ground truth trajectory length covered by the reconstructed
trajectories. TCF is 100% when all reconstructed trajectories are correct.
Trajectory fragmentation factor (TFF), which measures on average
the number of acquired trajectories used to match one ground truth
trajectory. The ideal score of TFF is 1, the larger this value is, the worse the
performance on maintaining particle identity.
26. Results in real world datasets
We use our method to obtain 3D flying trajectories of
Drosophila melanogaster (fruit fly) group comprising
hundreds of individuals (more than 150 detected targets in
each frame).
Experiment setup:
The fruit flies flied freely in an acrylic box of size 35cm by 35cm by
25cm, where the background was illuminated by white plane lights. Two
synchronized and calibrated Sony HVR-V1C video cameras working in
high speed mode at 200fps were used to capture the scene from
different views. The image resolution was 960×540 and 200 frames
were captured for the experiments.
27. Results in real world datasets
Near 400 4D trajectories are obtained
All the obtained trajectories Some longer trajectories
We hope our results could lead to deeper understanding of
fly social behavior of fruit fly group.
28. Future work
Build an advanced probabilistic model for MECL
Test our framework on bird flock or fish school data sets
Maybe we have a chance to analyze the trajectories and
discover something interesting
