A look at the recently developed MPM method (an improvement over Detection & Association method) of tracking individual cell movements in cell cultures. The individual cells are identified using U-Nets, and a MPM created for each cell, and cell track data extracted from these maps using efficient algorithms. Has applications in Medical imaging, Biometrics, Wound imaging, Cell analysis and bacteriology.

1. Joint representation of MPM
in Cell-Tracking
発表者：NANDEDKAR PARTH SHIRISH
1
Department of
Intelligent Media,
Yagi Laboratory

2. 今日の流れーTopics of Focus
１．(Conventional) Cell Tracking – Detection & Association
２．Common Difficulties in Association
３．Motion and Position Map (MPM)
４．Cell Tracking using MPM
５．Interpolation for Undetected Cells
2

3. 1．(Conventional) Cell Tracking Method
→ Goal: To identify multiple cells (multi-object tracking)
• Process: Automatic tracking via tracking-by-detection.
1. Detection Phase - Trained CNNs detect cells in each frame
2. Association Phase – detection results associated in
successive time-frames
→ Difficulties:
• Cells tend to have similar shapes (thus low confidence level for
traditional Object tracking and Siamese Networks).
• Cells tend to deform.
• Blurry Intercellular boundaries – Clumped cells need to be detected.
• Cell Mitosis may occur (Mitosis tracker developed by S. Huh, 2011).3

4. 1．(Conventional) Cell Tracking – Detection
Detection Phase - Trained CNNs detect cells in each frame
a) Example Frame after CNN results (Manual detection as
well as automatic detection)
4

5. 1．(Conventional) Cell Tracking – Detection
Detection Phase - Trained CNNs detect cells in each frame
b) Graph of True detection rate for every frame of data.
5

6. 1．(Conventional) Cell Tracking – Detection
Manual Tracking vs Automated Tracking
Automated Tracking involves further Image pre-processing.
6

7. 1．(Conventional) Cell Tracking – Detection
Manual Tracking vs Automated Tracking
Manual Tracking is more precise in tracking Cell movements
(higher Mean Square Deviation), but is too time-consuming.
(using Fiji TrackMate)
7

8. 1．Conventional Cell Tracking - Association
→ Goal: To identify the track of each cell detected.
• Process [R. Bise, 2011] :
1. Tracklets generation: Each Detected Cell given an ID.
Cell Motion Fields (CMFs) generated for frame intervals
2,3,4,5…..and so on.
2. Global Association: Joining
tracklets to get overall
trajectories.
8

9. Tracklet Generation using CMF Intervals
9
Motion Interval Color shows Direction. [Hayashida,2019]

10. 1．(Conventional) Cell Tracking Overview
Detected Cells
Highlighted
10
Tracklets - Net Movement
in frame Interval

11. 2．Common Difficulties in Association
11
→ Possibility of Incoherence between Detection and
Association:
Although a cell is detected, there may be no response at the
detected position in the CMF, affecting the tracking
performance
Conventional CMF(Hayashida, 2019) and Multi Tasking learning
methods(Georgia, 2014) both have the above tendency, due to
having separate Detection and Association Phases.
This can be solved via the new Motion and Position Map
(MPM) Method

12. 3．Motion and Position Map (MPM)
12
MPM Algorithm:
Step 1) Take two successive frames and find the relative cell
position and motion (association), each as annotated in (a).
Step 2) Calculate the ‘Cell motion vector’ which is Motion
vectors from the neighbor points around the annotated point
at ‘t’ to the annotated point ‘t−1’. This is denoted by .

13. 3．Motion and Position Map (MPM)
13
MPM Algorithm:
Step 3) Next Calculate the ‘Cell position likelihood map’
which by the Random Walk Theory is a 3-D Gaussian
distribution around the ‘t-1’ position.
Step 4) Finally, calculate the MPM of the cell using
the formula:

14. 3．Motion and Position Map (MPM)
14
Step 4) Finally, calculate the MPM of the cell using the
formula:
Where is a weight function that takes each coordinate
at frame ‘t’ as input and returns a scalar value
are motion vectors for cell from ‘t’ to ‘t-1’.
Step 5) Training a CNN: A CNN (U-Net) is trained on the 3D
Vector Data of the MPM (blue in last slide) using MSE loss
function for the vectors and their magnitudes.
This is the ‘MPM-Net’.

15. 3．Motion and Position Map (MPM)
15
Advantages of MPM: Solves the Incoherence problem while
also simplifying the Learning process.
• The simplest of CNNs can function as MPM-Nets.
• By applying MPM-Net to the entire image sequence, we
can easily obtain the overall cell trajectories without a
multi-step process.
• Since MPM represents both detection and association, it
guarantees coherence; i.e., if a cell is detected, an
association of the cell is always produced.

16. 4．Cell Tracking using MPM
16
1. Detection Phase – The peak of the Gaussian likelihood
is the cell position.
2. Association Phase – The direction of each 3D vector of
MPM indicates thedirection from position of the cell at
‘t’ to position of the same cell at ‘t − 1’.
Tracklets generation: Each cell has its previous position
estimated (= ).
• We update the estimated position to a local
maximum that is the position of a tracked cell by
using the hill-climbing algorithm to find the peak (the
position of the tracked cell).

17. 4．Cell Tracking using MPM
17
Tracklets generation:
If an estimate was updated to a cell position we
associate with as the tracked cell.

18. 4．Cell Tracking using MPM
18
2. Association Phase –
Cell Division Tracking: Each
• If the likelihood value at the estimated position at ‘t − 1’
is zero, is registered as a newly appearing cell.
• If two estimated cell positions were updated to one
tracked cell position , these detected cells are
registered as new born cells
• The mitosis event information is also registered.

19. 5．Interpolation for undetected cells
19
The estimated MPM at ‘t’ may have a few false negatives
(the estimation is not perfect). In this case, the trajectory
is not associated with any cell and the frame-by-frame
tracking for the cell is terminated at ‘t−1’.
Solution: Assume there are newly detected cells at ‘t-1+q’.
Then, the method tries to associate the temporarily track-
terminated cell using MPM again, where the MPM is
estimated by inputting the images at ‘t−1’ and ‘t−1+q’
Next, the cell position is estimated and updated by using
frame-by-frame association.

20. 5．Interpolation for undetected cells
20
• Example: The tracking of the green trajectory is
terminated at ‘t − 1’ due to false negatives at ‘t’, ‘t + 1’,
and the red cell is detected at ‘t + 2’.
In this case, the MPM is estimated for three frame intervals
(inputs = frames ‘t − 1’ and ‘t + 2’), and it is associated with
the track-terminated cell. Then, the cell positions at ‘t’ and
‘t + 1’ are interpolated using the vector of MPM and the
tracking starts again.

21. References
[1] Detection Result Image:
https://www.researchgate.net/publication/323378424_L
ens-
free_Video_Microscopy_for_the_Dynamic_and_Quantit
ative_Analysis_of_Adherent_Cell_Culture/figures?lo=1&
utm_source=google&utm_medium=organic
[2] Automated vs Manual Tracking:
https://www.researchgate.net/publication/337310459_
A_simple_3D_cellular_chemotaxis_assay_and_analysis_
workflow_suitable_for_a_wide_range_of_migrating_cell
s/figures?lo=1
[3] Frame by Frame Data Association, CMF:
http://human.ait.kyushu-u.ac.jp/~bise/researches-bise-
CIA-en.html 21