SlideShare a Scribd company logo
1 of 41
Download to read offline
Large-scale cell tracking using an approximated
Sinkhorn-Knopp algorithm
Submitted to the
School of Engineering Science,
Information Science Department
Osaka University
February 2021
Parth Shirish Nandedkar
2
2020 Bachelor’s Thesis
Large-scale cell tracking using an approximated Sinkhorn-Knopp algorithm
Parth Shirish Nandedkar
Abstract
The popular field of Cell Tracking has a lot of recent developments, with the influx
of many different deep learning methods. However, not only are cells far more
numerous, the different kinds of organisms have cells with entirely different
characteristics. Apart from these practical considerations, the issues of blurry
intercellular boundaries and mitosis are also faced. It is for these reasons that much of
biological research on tracking the behaviour of cells still involves manual annotation of
cells in each photographed slide.
In this research, we focus on vastly speeding up the tracking of cells, basing our
experiments on the data for the multi-cellular pattern formation of over a hundred
thousand cells of a microorganism, the Dictyostelium discoideum species of amoeba, in
neurologically diseased animal subjects. The method follows the traditional two-phase
(detection phase and association phase) approach to cell tracking. However, due to the
sheer number of cells, faster, but less precise bipartite graph matching or linear
programming (optimal transport) approaches were tested and a final candidate - a
modified and greatly speeded-up Sinkhorn-Knopp algorithm, the Sparse Sinkhorn-
Knopp algorithm - was reached.
The proposed algorithm allows researchers to automate the annotation of cells, and
in the case of the Dictyostelium discoideum cell data, allows for the association of over
10,000 detected cells into continuous cell tracks in a mere 25 seconds per slide of data,
in 𝑂(𝑛) time and space complexity for 𝑛 cells.
Keywords: Cell tracking, Multi-cellular patterns, Hungarian algorithm, Bipartite
Matching, Optimal Transport, Sparse matrices, Sinkhorn-Knopp algorithm
3
Contents
1 Introduction
2 Related work
2.1 Deep Learning-based approaches
2.1.1 Motion and Position Map (MPM) method
2.1.2 CellNucleiTracker
2.2 Non-Deep Learning based approaches
2.2.1 CancerCellTracker
2.2.2 Macrophage tracker (Hungarian algorithm)
2.2.3 Candidate cell region detection-based association
2.2.4 Probabilistic event detection of stem cells
2.2.5 Tracking using Particle filters and Time-to-Event analysis
2.3 General multi object tracking methods
2.3.1 DeepMOT
2.3.2 Multi-object tracking using dynamical graph matching
3 Proposed method
3.1 Pipeline
3.2 Detection Phase
3.3 Tracklets generation
3.3.1 Features and cost matrix
3.3.2 Nearest Neighbor matching
3.3.3 Hungarian matching
3.3.4 Sinkhorn-Knopp matching
3.3.5 Sparse Sinkhorn-Knopp matching
3.4 Tracklets association
4
4 Experiments
4.1 Details on the dataset
4.2 Examples
4.3 Protocol
4.4 Tracking accuracy
4.5 Computational cost compared to pure Sinkhorn-Knopp algorithm
5 Conclusions
Acknowledgements
Bibliography
1
Chapter 1
Introduction
There have been many advances in the neurological study of diseased or malnourished
neural tissue. One of these advances is based on modelling the neurons with the
Dictyostelium discoideum species of amoeba. The lifecycle of D. discoideum is relatively
short, which allows for timely viewing of all stages. This makes it a model organism to
study cellular and biochemical processes in neural tissues. However, due to its ever-
changing nature and the large number of cells, the identification of cells fluorescent
imaging data for these is still done manually. The goal of this research thesis is to
automate this process of identifying, numbering and associating (joining the tracks of)
each cell across multiple frames of fluorescent imaging data.
When malnourished, thousands of dictyostelium cells collectively migrate and form
patterns as shown in Figure 1.1 below. In particular, only a few β€˜leader’ cells release a
small amount of Cyclic adenosine monophosphate (cAMP) as a signal to the remaining
β€˜follower’ cells which then leads to this migration. Such collective cell migration
contributes to many organismal phenomena in neural tissues, including morphogenesis,
wound healing, and cancer invasion [1, 2], making the timely analysis of the movement
of a 100,000 or more cells imperative for the study of cell-cell communication and
desieases like brain cancer. The provided data has over a 100,000 dictyostelium cells
laced with two different fluorescent protiens allowing for two different modes of
fluorescent imaging of the amoeba.
2
Figure 1.1: The formation of circular patterns in malnourished Dictyostelium cells.
Cells releasing cAMP are shown in red [15].
As this is a tell tracking problem, we will first discuss the basics method of cell
tracking. Cell tracking consists of two phases, the detection phase, and the Association
Phase [3]. The detection phase consists of identifying the bounding boxes of each cell
and assigning temporary IDs to them in each frame. Recently, trained CNNs and RNNs
are being used to detect the cells in each frame [4, 5, 6, 10, 17]. In the association
phase, these detection results associated in successive time-frames to form tracklets, in
a process called "Tracklets generation”. These tracklets are then associated wherever
possible, to form continuous overall trajectories for as many cells as possible in the
β€œGlobal Association” stage. This process is shown visually in Figure 1.2 below. The
global association stage always follows the detection phase and the tracklets generation
stage with a delay in order to gather as much data as required to associate a tracklet
ending in the current frame to a new tracklet starting in the next few frames.
Figure 1.2: The pipeline of most Cell Trackers [3]
The above method for tracking cells has many issues however, such as those shown
below. Figure 1.3 shows examples of these issues, with 1.3(a) showing the case of
3
similarly shaped cells, leading many deep learning based methods and traditional object
tracking astray. Figure 1.3(b), a part of our target dataset, shows the random shape
changes many types of cells undergo from frame to frame. Figure 1.3(c), depicts the
extremely blurry intercellular boundaries in our target dataset, which renders many cell
segmentation algorithms futile. Lastly, Figure 1.3(d) is an example of Cell Mitosis
occurring between frames, which needs its own algorithm to track newly separated
cells, such as the Mitosis tracker developed by S. Huh [7]. The fifth and most important
issue for practical purposes is temporal efficiency or computational cost, which rules out
most existing cell tracking or multi object tracking (MOT) methods for this research.
Figure 1.3: (a) Example cell data, with each cell showing nearly identical shapes [23]. (b)
The cells in the centre and the bottom right vastly change their shape and size
respectively. (c) Cells stuck to each other with no clear intercellular boundaries in our
dataset. (d) Cell mitosis occurs for the bright cell in the centre.
At present, there are many different methods to track (associate) detected cells, and
can be broadly classified into deep and non-deep. Deep learning methods for cell
tracking have been developed over the last decade, and are based on detection using a
RNN (or Bi-RNN), a U-net (a specialized CNN for medical image segmentation) or a
siamese network [4, 5, 10, 17, 18] trained on sample images of the type of cell and fed
cell ID annotation data. However, upon application of these methods to our target
dataset, the shortcomings of all deep learning methods are clear. Adding annotation
4
data for a 100,000 cells impractical. On top of this, these networks also almost
completely base their identification of cells upon existing characteristics of the specific
type of cell, which inevitably requires high resolution biomedical imaging. This is not
the case for our very low-resolution dataset as shown in Figure 1.1 or 1.3(b),(c). Since
cell imaging is low resolution when imaging such a vast number of cells, we can
conclude that deep learning methods will be inaccurate and largely overfitted. Apart
from these issues, neural networks may not be able to keep track of the constant cell
shape changes of the Dictyostelium discoideum amoeba, as discussed in detail in the
related work section.
In non-deep learning approaches, we have the basic K-nearest neighbour (KNN)
algorithm, bipartite graph matching algorithms, and the linear programming based
optimal transport algorithms [19 , 9, 12]. We use KNN as a baseline for testing the
other methods here, namely the Hungarian bipartite matching algorithm and the
Sinkhorn-Knopp optimal transport algorithm. The Hungarian algorithm, or the Kuhn-
Munkres algorithm [20] is a simple linear programming approach to solving the general
assignment problem provided the costs (based on similarity metrics) between every
combination of cells in two different groups of equal number of cells. The advantage
with this method is the ability to get a robust matching between two successive frames
of cells (current frame 𝑑 and previous frame 𝑑 βˆ’ 1) without impractical amounts of
annotation or training. However, the naΓ―ve Hungarian algorithm has 𝑂(𝑛 )
computational cost (where n is the number of cells). This can be reduced to 𝑂(π‘˜π‘› ) by
limiting the scope of matching to only the K nearest neighbours (Hungarian-on-KNN,
or the sparse assignment problem [21]). This makes for a small 𝑂(π‘˜π‘›) space cost matrix
and a decently fast match for a thousand cells (about 40 seconds per match for 2,000
cells), however it still takes more than 15 minutes to match two frames of 10,000 cells
each.
The Sinkhorn-Knopp algorithm [12, 13, 14] is also a linear programming based
approach, which, when provided the costs between cells in one frame and in another
frame, converts these costs into entropies, and maximizes the sum of these entropies,
closing in to an asymptotic solution, while never truly reaching the exact solution [13].
The advantages with this optimal transport algorithm are numerous, starting from
having no need of annotation or training, to the temporal and spatial cost [12]. It also
allows for specifying the weights of each object in the form of probability vectors, which
5
in our case corresponds to the total brightness of the cell. When two cells visually
merge, the Sinkhorn algorithm detects that their brightness has been added, and
matches these merger cells to the merged cell. The Sinkhorn-Knopp algorithm is 𝑂(𝑛 )
in time and space (due to the matrix operations), making it a much better candidate
for speeding it up by tinkering with the process.
The algorithm proposed in this paper for solving the issue of computational cost for
large-scale cell tracking is the Sparse Sinkhorn-Knopp algorithm. This is a modification
of the Sinkhorn-Knopp algorithm, in which we generate a sparse cost matrix by
calculating costs only for a limited number of candidate cells matching certain criteria.
This takes advantage of the faster 𝑂(π‘˜π‘›) sparse matrix multiplications, making the
computational cost of the match 𝑂(π‘˜π‘›) for n cells. Also, another major advantage of
the proposed algorithm also 𝑂(π‘˜π‘›) in spatial complexity since the sparse cost matrix
can be stored in any of the CSR, CSC, COO or LIL formats. This also tackles the
shape-changing nature of amoeba and neurons by not relying on learning cell shape and
structure. The issues of blurry intercellular boundaries and cell mitosis remain however,
making the global association of tracklets inaccurate for this method in its current
state.
6
Chapter 2
Related work
There have been several published methods in order to deal with issues such as
inaccurate cell segmentation and cell mitosis, however as this research focuses on
mainly the time and space complexity encountered in forming cell tracks, most of the
related work below was scrapped in order to generate the sparse version of the
Sinkhorn-Knopp algorithm. All the tried and tested methods have been listed and
discussed below. Some of these methods are general multi-object trackers.
2.1 Deep learning-based approaches
2.1.1 Motion and Position Map(MPM) method
The MPM method is a recently developed cutting-edge cell detection method [4]. It
uses U-nets to both detect and associate cells, the tracklet formation being based on a
likelihood estimation formula which uses the cell’s motion data for the next few frames.
In the detection phase, each detected cell is first given an ID using a Likelihood Map
(LM) generated by using a U-net trained for cell detection of the specific type of cell.
As the first step to associate the cells across frames, tracklets must be generated.
For this purpose, Cell Motion Fields (CMFs) for different frame intervals (2 frames ,3
frames and so on) are generated as shown in Figure 2.1. The CMF gives us data on the
direction and magnitude of each cell’s motion. A predicted matching is made based on
the Motion and Position Map calculated from a formula with trainable weights.
7
Figure 2.1: The Cell Motion Field (CMF) used for matching, the vectors show the
magnitude and direction of displacement in the duration of the interval [4]
The training of the MPM-net (the U-net performing the matches) consists of five
steps. We take two successive frames and find the relative cell position and motion,
each as annotated in Figure 2.2(a). Next, we calculate the β€˜Cell Motion Vectors’(CMV)
which are the Motion vectors from the neighbor points around the annotated cell of ID
β€˜π‘–β€™ in frame β€˜π‘‘β€™ to the annotated cell in frame β€˜π‘‘ βˆ’ 1’. This collection of vectors is shown
in blue in Figure 2.2(b). Next the β€˜Cell Position Likelihood Map’ (CPLM) is calculated
which, in a similar fashion to random walk theory, is a 3-D Gaussian distribution
around the β€˜π‘‘ βˆ’ 1’ position. as in Figure 2.2(c). The Motion and Position Map (MPM)
is formed from the above CMV and CPLM data using a formula which contains
trainable weights. Finally, a CNN (U-net) is trained on the 3D vector data of this
MPM using MSE loss which trains the weights in the formula. This is the β€˜MPM-Net’.
The main advantage of this method is since the number of trainable weights is
minimal, the simplest of CNNs can function as MPM-Nets, and with great accuracy.
Also, since MPM represents both detection and association, it guarantees coherence,
i.e., if a cell is detected, an association of the cell is always produced. Other similar
CMF-based or multi-tasking learning based methods [5] have the tendency to be
incoherent, due to having independent detection and association phases.
8
Figure 2.2: The calculation of the MPM from the CMV and the CPLM [4].
However, upon application of this method to a small sample target dataset, the
flaws of this approach in relation to the given problem will be met. The MPM-net,
while requiring fewer epochs and training samples to train than other deep learning
based methods, still needs large-scale annotations. Also, coherence between association
and detection phases does not solve the issues of shape-shifts and blurry boundaries in
the detection phase. On the contrary, the training and the results of the simpler MPM-
net architecture is highly susceptible to such detection lapses.
2.1.2 CellNucleiTracker
Another NN (U-net) based method [6], which is based on learning the features of the
cells, especially the location of the nuclei. As seen in Figure 2.3, this method was nearly
perfectly accurate for plant cells, and epithelial or circulatory tissue, which are largely
immobile cells. Also, the resolution of the frames is much higher, allowing the network
to learn the different features of each type of cell.
9
Figure 2.3: The pretrained network of CellNucleiTracker showed nearly perfect
detection and segmentation for high resolution images of plant and animal tissue [6].
Since such conditions are not present in our dataset, this method was very inaccurate
for the Dictyostelium cell dataset, as seen in the results of the pre-trained network
(Figure 2.4).
Figure 2.4: The pretrained network could not handle the provided low resolution frames
of Dictyostelium amoeba cells
10
2.2 Non-deep learning based approaches
2.2.1 CancerCellTracker
CancerCellTracker is an OpenSource project for solving the Fluo-H2DL-HeLa dataset
of the cell tracking challenge [8]. The dataset consists of cancerous cells with cell
mitosis constantly occurring. It uses the watershed algorithm to segment cells and
builds a feature vector for cell tracking including the information of position, shape,
spatial distribution and texture as discussed in Li et. al. [8]. It also has a cell family
history tracking particularly useful for cancerous cells.
Not requiring annotation and training, this method is also faster than the MPM and
DeepMOT methods discussed in this section, even when obtaining the results. However
identity switching is common in the results as in Figure 2.5, and cell family history
tracking is not useful for dictyostelium cells which, like the neural cells it is a model
for, rarely undergo mitosis, in stark contrast to cancerous cells. Due to major accuracy
concerns, this method is not robust enough for large-scale cell tracking.
Figure 2.5: CancerCellTracker shows many ID switches even in the first few frames.
11
2.2.2 Macrophage tracker (Hungarian algorithm)
This macrophage cell tracker [9] is based on one of the simplest bipartite graph
matching algorithm, the Hungarian algorithm, and is hence a good candidate for our
large-scale cell tracking problem. The proposed method is an automated single
macrophage tracking method for a dataset of a mouse brain (11.7 T time-lapse MR
images). The method detects macrophages using background subtraction algorithm,
and tracks macrophages using the Hungarian algorithm, backed up by a Kalman filter
for association. The results showed that the proposed method detected and tracked
macrophages in MR images successfully. The below table (Table 2.1) from the paper
shows an example matching obtained by the Hungarian algorithm between cells in
frame β€˜π‘‘β€™ and frame β€˜π‘‘ + 1’.
Table 2.1: An example matching using the Hungarian algorithm. The Red numbers
represent the highest probability match for the cells in β€˜π‘‘ + 1’, and the light red boxes
represent the final match chosen, after resolving conflicts [9].
However, despite the application of the Kalman filter in this method, the issues with
this method remain the same as for the optimized Hungarian algorithm (Hungarian-on-
KNN) discussed in the Introduction section. This algorithm is still 𝑂(𝑛 ) in time
complexity after limiting the scope of matching to the nearest neighbours, and hence,
despite the accuracy of association, is rendered unsuitable.
2.2.3 Candidate cell region detection-based association
In this paper [24], the proposed method tracks cells under highly confluent conditions
by using the candidate cell region detection-based association approach. While the
association phase is also a linear programming based approach like in our case, the
detection phase is it’s unique point. Unlike conventional segmentation-based association
tracking methods, the proposed method uses the tracking results from the previous
frame to segment the cell regions at the current frame. First, candidate cell regions are
12
detected, and while there may be many false positives, there are very few false
negatives. Next, optimized detection results are selected from the candidate regions and
associated with the tracking results of the previous frame by resolving a linear
programming problem.
This non-deep learning approach is very for our current research, as it has the
potential to be fast enough for the requisite number of cells. Many ideas discussed in
the proposed method section are similar to those discussed in this paper.
2.2.4 Probabilistic event detection of stem cells
In this non-deep learning approach [7], a probabilistic model for event detection is
proposed which can simultaneously identify spatio-temporal patch sequences that
contain a mitotic event and also specify the time and location of cell birth events. The
approach significantly outperforms previous approaches in terms of both detection
accuracy and computational efficiency, when applied to multipotent C3H10T1/2
mesenchymal and C2C12 myoblastic stem cell populations.
However, this is a method specialized for cells which undergo mitosis very
frequently, (as in the case of cancer cells shown in Figure 2.5), as the semi-Markov
model used adds a lot of unnecessary computation if the method is used on cells with
infrequent mitosis events.
2.2.5 Tracking using Particle filters and Time-to-Event
analysis
This paper [25] proposes a method for spatiotemporal tracking and analysis of FUCCI
(fluorescent ubiquitination-based cell cycle indicator)-labelled cells from time-lapse
videos. To address the colour transition of the FUCCI-labelled cell with the cell cycle
progression, the proposed method simultaneously estimates the location and the cell
cycle phase of the target cell. Furthermore, to analyse the cell phase transition, this
paper proposes to apply multistate time-to-event analysis to the information obtained
through our tracking method. This paper demonstrates the usefulness of the method
with application to FUCCI-labelled HuH7 cells (human hepatocellular carcinoma cell
line).
The method was deemed to be increasingly complex for the problem at hand, since
cell-cycle analysis is unnecessary for our research. The particle filters are a useful idea
13
for large-scale cell tracking, allowing global association in relatively quick time, and this
is hence one of the main proposed improvements to our cell tracker for the future.
2.3 General multi-object tracking methods
2.3.1 DeepMOT
DeepMOT [10] specializes in removing false positives and false negatives in general
multi-object tracking, while also preventing unnecessary ID changes during the cell
track. It achieves this by using a trained Bidirectional RNN (Bi-RNN) to perform the
assignment on a distance (cost) matrix). The detailed pipeline for this method is shown
in Figure 2.6.
Figure 2.6: The pipeline of DeepMOT. The distance matrix D is flattened and fed to a
Bi-RNN at two steps of the process for learning, and the recovered matrix is taken as
the assignment matrix A [10].
There are four steps to achieve this extremely accurate matching between the
objects. First, we randomly sample a pair of consecutive frames from the training video
sequences. These two images together with their ground-truth bounding boxes
constitute one training instance. Then, for each such instance, we first initialize the
tracks with ground-truth bounding boxes (at time 𝑑) and run the forward pass to
obtain the track's bounding-box predictions in the following video frame (time 𝑑 + 1).
To mimic the effect of imperfect detections, we add random perturbations to the
ground-truth bounding. Thirdly, we compute D (distance matrix in Figure 2.7) and use
the proposed Deep Hungarian Network (a Bi-RNN) to compute A (the assignment
matrix). Finally, we compute our proxy loss based on D and A. This provides us with a
gradient that accounts for the assignment, and that is used to update the weights of
the tracker.
Thus, the weights of the tracker are updated as the tracking results are generated,
simplifying the process of training and testing. Figure 2.7 depicts the flow of data and
14
the back-propagation of gradients with the Deep Hungarian Network (the Bi-RNN) in
the centre. Figure 2.8 is a frame of the result obtained.
Figure 2.7: The pipeline of DeepMOT including detection, association and training. The
pink box represents the training stage, where the DHN and the DeepMOT Loss
alternate till the training is complete [10].
As is evident in the small example frames in Figure 2.7 and the training data
proposed in [10], this method which uses a complex Bi-RNN architecture is suitable
only for a few dozen objects in-frame. Because of the use of a complex network, it is
very well suited for large-sized objects with many features to learn from. Neither of
these match with the conditions of large-scale cell tracking however, where the goal is a
100,000 cells with a very low resolution of imaging, few features and constantly
changing shapes.
2.3.2 Multi-object tracking using dynamical graph matching
This association part of this method is based on a tracking algorithm to address the
interactions among objects, and to track them individually for a static video camera
[11]. It is achieved by constructing an invariant bipartite graph to model the dynamics
of the tracking process, of which the nodes are classified into objects and profiles as
shown in Figure 2.9(a),(b) and (c).
15
Figure 2.9: (a) End of frame β€˜π‘‘ βˆ’ 1’. The matching information is used to update the
profiles used in frame β€˜π‘‘β€™. (b) Beginning of frame β€˜π‘‘β€™. The bipartite graph has only 2
profile nodes. (c) Now 3 objects are detected so the graph now has 3 new object nodes.
(d) A bipartite matching is carried out. Since there is one unmatched object node, a
profile node is created to match it as in (e) and (f) [11].
The matching occurs in as shown in Figure 2.9(d),(e) and (f), and includes ad-hoc
approaches to solve object interaction issues. First, the matching cost between a profile
node and an object node is measured by the Kullback-Leibler distance on the image
data distribution of the nodes. Now, if there exist unmatched object nodes, then new
profiles are is created for each newly detected object. When an object leaves the scene,
its corresponding profile will become unmatched. After finding the optimal bipartite
matching, we check all profiles to see if there are some profiles getting too close to
others. Those profiles will be marked as ”TBM” (to-be-merged). In the next image
frame, the TBM profiles will be processed separately to handle possible interactions.
This simple assignment method is designed for ever-changing numbers of cells and is
quite appropriate for the given dataset. However, as the number of objects for the video
data used in this research is small, the computational complexity has not been
considered. A simpler form of the TBM profile processing feature was implemented in
the form of a β€˜merge dictionary’ in the tracklet association stage (Chapter 3.4).
16
Chapter 3
Proposed method
3.1 Pipeline
The proposed method in order to solve the time and space complexity issue with large-
scale cell tracking is the Sparse Sinkhorn-Knopp algorithm. The pipeline is as shown in
Figure 3.1, and has the same broad structure of detection followed by tracklet
formation and ultimately global association of these tracklets.
Figure 3.1: The pipeline of the proposed method. It follows the same pattern as the
generic tracking pipeline in Figure 1.2.
3.2 Detection phase
We begin with a detection phase that is independent of the association phase, based on
a blob (maxima) detection algorithm on Laplacian of Gaussian (LoG) filtered frames
with non-maxima suppression. This method was chosen for being among the quickest
available methods. The LoG filter helps in boosting the brightness of the peaks
compared to the surroundings. The method used computes the LoG images with
successively increasing standard deviations and stacks them up in a cube data
structure. The blobs are the local maxima in this cube after non-maxima suppression..
The equation for the LoG filter is as follows directly from spatial image processing
theory, with an added Gaussian filter of standard deviation 𝜎. Various 𝜎 will be used in
the method. π‘₯, 𝑦 are the two dimensional variables used to specify the filter’s equation.
17
The 2D function looks as shown in Figure 3.2, and has the effect of highlighting
edges and peaks, the latter being our target of detection.
Figure 3.2: The LoG filter calculates the second derivatve of the image in the 2D space
domain. This leads to highlighted edges, and therefore, peaks.
The usual disadvantage of the LoG approach is that detecting larger blobs is
especially slower because of larger kernel sizes during convolution. However, since D.
discoideum cells very have similar sizes, the kernel size can be limited to a small range,
and the detection can occur smoothly in the expected 𝑂(𝑛) time. Alternative methods
are by applying the Difference of Gaussian (DoG) or the Determinant of Hessian (DoH)
filters instead, to speed up the detection even further. However, a simple comparison of
these three filters (Figure 3.3) shows the massive loss in accuracy for the DoG and DoH
filters.
18
Figure 3.3: A comparison of the LoG, the DoG and the DoH filters used in peak
extraction. The latter two while faster and widely used as approximations, are both
largely inaccurate for our low resolution target data.
These detected blobs are then sorted into an adjustable number of bins as shown in
Figure 3.4, based on their position on the frame, with bin 1 starting on the top left.
This is done so as to allow for optimized Sinkhorn-Knopp matches later on, by
considering matches for a cell in frame β€˜π‘‘β€™ only from frame β€˜π‘‘ βˆ’ 1’ cells in the same bin
or the surrounding bins. Hence, each detected blob consist of four parameters, the x
and y coordinates, the radius of the circular blob, and the bin in which the blob is
present.
Figure 3.4: The division of all the cells in each frame into bins of equal sizes, allowing
for the removal of the vast majority of the candidate matches faster than a KNN
approach. Bin numbering starts from the top left.
19
3.3 Tracklets generation
3.3.1 Features and cost matrix
As this is a bipartite matching or assignment problem, the costs or dissimilarities
between cells in frame β€˜π‘‘β€™ and frame β€˜π‘‘ βˆ’ 1’ must be arranged in a cost matrix. The
features used for calculating dissimilarity are the distance 𝑑 between the two cells in
the adjacent frames, the radii of the blobs, the brightness at the peaks (averaged with
the surrounding pixels), and the net brightness of all pixels that make up the cell blobs.
The last feature is useful because it is directly proportional to the amount of fluorescent
protein present inside the cell. The blob radii, peak brightness values and brightness
sum values are all averaged for each detection of each cell, to remove variance and get
more accurate data. A simple cost function that provides decent results is as shown
below.
Features: Euclidean distance moved 𝑑, radii π‘Ÿ and π‘Ÿ , peak brightness values 𝑏 and
𝑏 , brightness sum values 𝑠 and 𝑠 (distances in pixels, brightness from 0 to 255, 𝑀 are
the weights).
πΆπ‘œπ‘ π‘‘ = 𝑑 + 𝑀 Γ— (π‘Ÿ βˆ’ π‘Ÿ ) + 𝑀 Γ— (𝑏 βˆ’ 𝑏 ) + 𝑀 Γ— (𝑠 βˆ’ 𝑠 )
Figure 3.5 is an example cost matrix when the cells are divided into 100 bins.
Figure 3.5: An example cost matrix when the cells are sorted into 100 bins. The
columns represent the cell IDs in frame β€˜π‘‘ βˆ’ 1’, and the rows represent the temporary
IDs before matching in frame β€˜π‘‘β€™.
20
3.3.2 Nearest neighbour matching
The simplest method for matching the bipartite graph of cells is the K nearest
neighbour algorithm. In this approach, each cell in frame β€˜π‘‘β€™ is first matched with its
closest cell in frame β€˜π‘‘ βˆ’ 1’ terms of the cost function. If two or more cells match to the
same cell in frame β€˜π‘‘ βˆ’ 1’, an ad-hoc algorithm sorts out these cells to match their next
closest cells in frame β€˜π‘‘β€™, considering up to K nearest neighbours. If in the previous case,
however, the cost function for each of the K nearest neighbours is above a certain
threshold, then the cell is regarded as different from any corresponding cell in frame
β€˜π‘‘ βˆ’ 1’, and is given a new ID number.
These new IDs largely occur in three cases, in the case of a genuinely new cell (cell
mitosis or new cell appearance from the edges) or in case of a misdetection (a false
negative, causing track discontinuity) or a mismatch of a neighbouring cell, which may
cause no valid matches to remain for the current cell.
3.3.3 Hungarian matching
We use the cost matrix discussed in section 3.3.1 as the bipartite graph’s weights in the
assignment problem. As the algorithm needs square matrix, any extra rows (if frame β€˜π‘‘β€™
has more detected blobs) will have corresponding columns added with zero cost. The
Hungarian algorithm consisting of four steps is then applied to find the optimal match.
Step 1: Subtract row minima
For each row, find the lowest element and subtract it from each element in that row.
Step 2: Subtract column minima
Similarly, for each column, find the lowest element and subtract it from each element
in that column.
Step 3: Cover all zeros with a minimum number of lines
Cover all zeros in the resulting matrix using a minimum number of horizontal and
vertical lines. If n lines are required, an optimal assignment exists among the zeros. The
algorithm stops. If less than n lines are required, continue with Step 4.
Step 4: Create additional zeros
Find the smallest element (call it k) that is not covered by a line in Step 3.
Subtract k from all uncovered elements, and add k to elements that are covered twice.
21
The match then obtained is processed in the same manner as in the Nearest Neighbour
matching algorithm, to remove cases of multiple cells in β€˜π‘‘β€™ matching to the same cell in
β€˜π‘‘ βˆ’ 1’ and to generate new IDs, wherever necessary.
3.3.4 Sinkhorn-Knopp matching
The Sinkhorn-Knopp algorithm is an optimal transport algorithm based on a linear
programming approach to maximize the sum of entropies. This algorithm was chosen
because it solves the entropic regularization optimal transport problem at a speed that
is several orders of magnitude faster than other transportation solvers[12, 13]. Unlike
the Hungarian algorithm, a square matrix is not necessary, however in the steps shown
below a square matrix of size 𝑛 Γ— 𝑛 is assumed as input, along with two probability
vectors 𝑝 and π‘ž. These probability vectors correspond to the total brightness of the cell.
When two cells visually merge, the Sinkhorn algorithm detects that their brightness has
been added, and matches these merger cells to the merged cell. The algorithm has 8
steps shown below in pseudocode format.
Algorithm 1: Sinkhorn-Knopp algorithm
Inputs: Probability vectors 𝑝 and π‘ž, cost matrix 𝑐, hyperparameter πœ†
For each row, find the lowest element and subtract it from each element in that row.
1: Calculate entropic regularized matrix 𝐾 where 𝐾 ← 𝑒π‘₯𝑝(βˆ’πœ†. 𝑐 )
2: Initialize 𝑛-dimensional vector 𝑒 as [1,1, … .1]
3: While not convergence do
4: 𝑣 ← π‘ž./(𝐾 𝑒)
5: 𝑒 ← 𝑝./(𝐾𝑣)
6: end while
7: π‘ˆ ← π‘‘π‘–π‘Žπ‘”(𝑒), 𝑉 ← π‘‘π‘–π‘Žπ‘”(𝑣)
8: π‘šβˆ—
← π‘ˆπΎπ‘‰
Output: Probability matrix π‘šβˆ—
, with values from 0 to 1 for each combination of a
current frame cell and a previous frame cell, signifying the probability that this
combination is a match.
The hyperparameter πœ† is the regularizer, which allows for matching over a more
convex function as it’s value increases, speeding up the convergence (Figure 3.6).
22
Figure 3.6: The matching occurs faster over a more convex function as the value of the
hyperparameter rises from left to right. The blue and red curves represent the
probability vector input [22].
The results upon application of this algorithm were different from the KNN and the
Hungarian algorithm, with more continuous tracks, as described in section 4.2 where
specific examples for the different methods are compared.
3.3.5 Sparse Sinkhorn-Knopp matching
The algorithm in 3.3.4 is optimized by modifying the cost matrix in an advantageous
way. Only the cells in the current or neighboring bins (set to 20 in total at maximum)
will be considered for calculation of the regularized costs (entropies), and the rest of the
combinations will have exactly zero entropy (infinite cost). This allows us to form a
sparse entropic regularized cost matrix as shown below in Figure 3.6, which is much
simpler for calculations than the entropic regularized original cost matrix.
This sparse regularized cost matrix is then stored in a Column Sparse Row format
and the same procedure as in the original Sinkhorn-Knopp algorithm is followed.
23
Figure 3.6: The modified entropic regularized cost matrix is extremely sparse with only
three thin lines, representing the cells in the current bin and the 8 neighbouring bins
taken as candidates. Columns are previous frame IDs, rows are current frame IDs.
3.4 Tracklets association
For the proposed method, the global association of the tracklets formed in the previous
subsections is based on ad-hoc methods. These ad-hoc methods include simple case-by-
case analyses such as disallowing matching to cells beyond practically feasible
distances, preventing non-maximum detections from overtaking existing cells (causing
ID switches) and storing a list of merged blobs (Merge Dictionary) and dispersing the
correct IDs to the blobs at the time of their demerge.
The prevention of non-maxima breaking cell tracks is done by the following simple
method, leading to fixing many previously broken tracks.
Algorithm 2: Prevention of non-maxima overtaking of tracks
1: if new ID added π‘˜ frames ago
2: and old (correct) ID nearby is not updated from π‘˜ βˆ’ 1 or more frames ago
3: then remove new (incorrect) ID and reassign old (correct) ID
24
An example of a non-maxima detection (ID 736) overtaking an existing cell track
(ID 539) is shown in Figure 3.7, with the resolved track on the right. Cell 539 (deep
blue track) is now correctly the only cell following the specific track.
Figure 3.7: An example where the blue track is overtaken by a false positive non-
maximum detection, leading to a yellow track with a different ID.
The Merge Dictionary is a Heuristic method for detecting (visual) merging of cells.
First, a merge is detected when matching the cells in the new frame as follows:
Algorithm 3: Registering merges
1: if (cell not detected this frame):
2: and if it touched another detected cell in the last frame
3: then cells merged, IDs added to dictionary
(touch = distance between centres < sum of radii+1-2 pixels)
The cell in front is determined by which of the multiple cells is the closest KNN
match to the blob in the same position last frame. This front cell’s ID now represents a
merged cell, and the other ID(s) are stored in a data structure to be distributed upon
detection of a demerge as follows:
Algorithm 4: Detecting demerges
1: if (new cell detected this frame):
2: and touches a Merged cell in this frame
3: and brightness/radius similar to merged cell
4: then cell separation detected,
5: IDs in Merged Cell dictionary distributed appropriately
An example of the working of the merge dictionary is shown in Figure 3.8. The
demerge is detected correctly, and the track for cell 442 is recovered.
25
Figure 3.8: An example where two cells merge to form a single blob and then demerge
to continue their tracks. The track for ID 442 is recovered, using the merge dictionary.
26
Chapter 4
Experiments
4.1 Details on the dataset
In order to scientifically understand the mechanisms of these rare cell-driven systems,
simultaneous observation of huge populations of cells is essential. Hence, the dataset
consists of large pixel frames, with sub-million cells. Each frame is taken at an interval
of 30 seconds. The fluorescent imaging system named AMATERAS (A Multi-
scale/modal Analytical Tool for Every Rare Activity in Singularity) is used, which is a
novel type of imager, trans-scale-scope, which allows us to observe cell dynamics with
sub-cell resolution in a field of view greater than a centimetre [15].
The provided data[15, 16] has over a 100,000 dictyostelium cells laced with two
different fluorescent protiens (Flamindo and RFP), allowing for dual color imaging of
the amoeba, as shown in Figure 4.1.
Figure 4.1: The two proteins used in the fluorescent imaging. The method used is dual
color imaging, with a mixture of both, Flamindo2 and RFP proteins.
Flamindo2 shows strong reactivity to the cAMP released by the cells changing from
bright green to dark as cAMP concentration increases, and RFP shows weak reactivity
27
to it, maintining a reddish colour. In combination, these two show a change from
yellow to red based on cAMP concentration.
A comparison of the number of bright spots in the Flamindo2 and RFP frames below
confirms the high reactivity of Flamindo2 and the low reactivity of RFP.
Figure 4.2: The difference in the reactivity of Flamindo2 and RFP to cAMP is clear
from the raw monochrome data, as the former has many more bright spots.
4.2 Examples
We compare the results of the Nearest Neighbour matching algorithm and the proposed
Sparse Sinkhorn-Knopp matching algorithm in Figure 4.3 by looking at corresponding
frames, and at a substantial number of cells (over 600), for enough visual test cases. A
more detailed look at some of the examples in Figure 4.3 shows us the differences in the
results for these two methods.
28
Figure 4.3: A direct comparison of the output frame 13 of Flamindo2 data for the NN
and the sparse sinkhorn methods. The green highlights are examples of the
improvement observed, while the red area shows the pitfall of the proposed method.
29
Looking at the example results, a comparison of the Nearest Neighbour matching
and the Sparse Sinkhorn matching can be made. Below are the β€˜plus’ and β€˜minus’
points for the proposed algorithm in comparison to the ad-hoc nearest neighbour
implementation.
+ Sinkhorn gave better accuracy especially for longer tracks (Green in Figure 4.3).
+ Greater track continuity.
+ 2-3 times the computational speed. (the Nearest Neighbour algorithm can also be
implemented in 𝑂(𝑛) time and space using bin sorting of cells).
Worse performance in terms of new IDs generated (Red in Figure 4.3).
Fewer incorrect matches but reliance on new IDs in crowded areas.
4.3 Protocol
The experiments conducted to determine the tracking accuracy and the processing time
follow a simple protocol. The metrics used for tracking accuracy are the percentage of
new IDs generated every frame, the IDS (ID switch) metric, the precision and the
recall.
IDS: Number of times that a tracked trajectory changes its matched ground-truth
identity, in the sample size chosen [17].
Recall: Ratio of correctly matched detections to ground-truth detections [17].
Precision: Ratio of correctly matched detections to total result detections [17].
For calculating IDS, 50 random cells are chosen and the ground truth results are
compared to the results of the tracker over the interval of 15 frames. For precision and
recall, the algorithm was ran for random areas of the overall frame containing about 50
cells, and the mismatches till frame 15 were counted. The data shown in the graphs for
tracking accuracy and processing time is averaged over 30 runs, and 10-40 runs
(depending on the number of cells) respectively.
30
4.4 Tracking accuracy
The overall accuracy metric used during development was the number of new IDs
generated per frame, since this is an upper bound for all misdetections and mismatches.
More common and useful metrics were also calculated, as shown in the second half of
this subsection. Since new ID cells are those which had no decent match in the previous
frame, they either arise from being genuinely new (mitosis or entry at edges), from
being false positives, or from mismatches in the neighbouring area.
As false positives are rare in the LoG filter peak detection method used (false negatives
are more common), this percentage of new IDs is a simple and useful metric to compare
the accuracy of the tracking part for different algorithms.
Figure 4.4 shows the percentage of new IDs in each frame for the KNN, the Sinkhorn-
Knopp and the Sparse Sinkhorn-Knopp algorithms, averaged over a 50 runs.
Figure 4.4: The percentage of new IDs generated for the first 17 frames, averaged over
30 runs. This is a loose upper bound to the number of mismatches.
The percentage of new IDs per frame is slightly higher with the faster method or
with smaller sizes of bins, signifying a trade-off of matching accuracy and
computational cost. Since cell mitosis can be assumed to be at a constant rate in terms
of percentage of cells, it is clearly below 1%, as is discernible from the KNN new ID
0.00%
1.00%
2.00%
3.00%
4.00%
5.00%
6.00%
7.00%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
%
New
cell
IDs
Frame number
Sparse Sinkhorn Sinkhorn-Knopp KNN
31
rates at the right end of the graph. Hence, the proposed Sparse Sinkhorn introduced
around as many new IDs as the regular Sinkhorn-Knopp algorithm and the simplest
KNN matching algorithm. This accuracy depends on the size of each bin (bin size used:
20px*20px). The larger the size of each bin, the lower the chance of not finding a
suitable match within the bound of the current and neighbouring bins. The IDS, Recall
and Precision calculated were as in Table 4.1.
Table 4.1: A comparison of the accuracy of the two methods tested, the proposed
method, and the simpler but slower KNN. The metrics were quite similar for both.
Metric
(50 cells, 15 frame interval)
Sparse Sinkhorn-
Knopp
KNN
IDS 44 40
Recall 65.5% 68.1%
Precision 76.3% 79.2%
A clearly visible pattern in the graph of Figure 4.4 is the decreasing nature of the
number of new IDs generated. This happens due to a combination of many reasons.
Firstly, the features (radius, peak brightness and brightness sum) of every cell are not
fully defined by the first few frames as they are stochastic, with a large variance, as
shown in the example of the changing peak brightness values for two cells in Figure 4.5
below. This causes many matches to go wrong but only in the first few frames.
Secondly, the various global association method discussed in 3.4 start to take effect
after the first few frames as well. The combined effect of clearer data and track
recovery methods leads to a drop in the number of cells without a suitable match (new
ID cells).
Figure 4.5: A graph of the raw peak brightness values of two random cells chosen from
the dataset. To reduce this variance, these features are averaged over each frame.
32
4.5 Computational cost compared to pure
Sinkhorn-Knopp algorithm
The processing in each frame consists of the detection part, the tracking part, and the
display or output part. The runtimes for each of these parts was measured separately.
As expected, the processing time of the detection and output parts is the same for each
matching algorithm from chapter 3, at about 10ms and 2ms respectively per cell for
numbers of cells above 200. Hence, a direct comparison of the computational cost of the
tracking part can be made as in the following graph.
Figure 4.6 shows the average runtime for the association phase (tracklet formation
and global association) per frame.
Figure 4.6: The time taken to complete the association of one frame averaged over 10-
40 runs. The reduction to linear time is clearly seen.
0
20
40
60
80
100
120
Time
per
frame
(s)
Number of cells
Sparse Sinkhorn Sinkhorn-Knopp
33
Chapter 5
Conclusions
In our thesis, the goal was to replace visual cell data analysis by a fast automatic
method to track pattern formation. For this purpose, the fastest approach to accurately
track cells was found out via elimination of all the different candidate methods, such as
the deep learning methods and the Hungarian algorithm. The proposed method was
based on a particularly fast optimal transport algorithm, the Sinkhorn-Knopp
algorithm, which allows for decently accurate matching of bipartite graphs in 𝑂(𝑛 )
time. This, however can be improved substantially by pruning off extremely unlikely
match candidate cells, to obtain a sparse entropic regularized cost matrix, which can
then speed up the convergence of the Sinkhorn-Knopp algorithm due to substantially
faster matrix computations.
However, as is clear in the results and the metrics that follow it, the algorithm may
be fast but it is in a bare-bones state, being prone to ID switching. To solve this issue,
future work such as the addition of a Kalman filter can be proposed. During the final
results, it was found that the proposed matching algorithm is much faster than the
detection method used, with the detection taking nearly 8 times as long for larger cell
numbers, and high memory usage. A modification of the applied detection method, and
further research into large-scale blob detection can also be proposed.
34
Acknowledgements
I want to thank professor Yasushi Yagi. His insights during our
meeting always push me to look further and try to be genuinely a researcher.
I want to show my deep appreciation for professor Yasushi Makihara for his insightful
guidance and continuous support.
I also want to thank Mr. Yang Yu for helping me with many technical issues, and for
suggesting new approaches to better implement the algorithms discussed.
35
Bibliography
[1] Friedl, P. & Gilmour, D. Collective cell migration in morphogenesis, regeneration
and cancer. Nat. Rev. Mol. Cell Biol. 10, 445–457 (2009).
[2] RΓΈrth, P. Collective cell migration. Annu. Rev. Cell Dev. Biol. 25, 407–429 (2009).
[3] R. Bise, Z. Yin and T. Kanade, "Reliable cell tracking by global data
association," 2011 IEEE International Symposium on Biomedical Imaging: From Nano
to Macro, Chicago, IL, 2011, pp. 1004-1010, doi: 10.1109/ISBI.2011.5872571.
[4] J. Hayashida, K. Nishimura and R. Bise, "MPM: Joint Representation of Motion
and Position Map for Cell Tracking," 2020 IEEE/CVF Conference on Computer
Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2020, pp. 3822-3831, doi:
10.1109/CVPR42600.2020.00388.
[5] Tao He, Hua Mao, Jixiang Guo, Zhang Yi, Cell tracking using deep neural networks
with multi-task learning, Image and Vision Computing, Volume 60, 2017, Pages 142-
153, ISSN 0262-8856, https://doi.org/10.1016/j.imavis.2016.11.010.
[6] K.Chen, N. Zhang, L.S.Powers, J.M.Roveda, Cell Nuclei Detection and
Segmentation for Computational Pathology Using Deep Learning, SpringSim 2019
Modeling and Simulation in Medicine, Society for Modeling and Simuation (SCS)
International.
[7] Huh, S., Ker, D. F., Bise, R., Chen, M., & Kanade, T. (2011). Automated mitosis
detection of stem cell populations in phase-contrast microscopy images. IEEE
transactions on medical imaging, 30(3), 586–596.
https://doi.org/10.1109/TMI.2010.2089384
[8] Li F, Zhou X, Ma J, Wong ST. Multiple nuclei tracking using integer programming
for quantitative cancer cell cycle analysis. IEEE Trans Med Imaging. 2010
Jan;29(1):96-105. doi: 10.1109/TMI.2009.2027813. Epub 2009 Jul 28. PMID: 19643704;
PMCID: PMC2846554.
[9] A. Tashita et al., "Macrophage Tracking Using the Hungarian Algorithm in Time
Lapse MR Images," 2015 7th International Conference on Emerging Trends in
36
Engineering & Technology (ICETET), Kobe, 2015, pp. 169-173, doi:
10.1109/ICETET.2015.40.
[10] Xu, Y., Osep, A., Ban, Y., Horaud, R., Leal-TaixΒ΄e, L., Alameda-Pineda, X.: How
to train your deep multi-object tracker. In: CVPR (2020)
[11] Hwann-Tzong Chen, Horng-Horng Lin and Tyng-Luh Liu, "Multi-object tracking
using dynamical graph matching," Proceedings of the 2001 IEEE Computer Society
Conference on Computer Vision and Pattern Recognition. CVPR 2001, Kauai, HI,
USA, 2001, pp. II-II, doi: 10.1109/CVPR.2001.990962.
[12] Cuturi, M. (2013). Sinkhorn distances: Lightspeed computation of optimal
transport. In Advances in Neural Information Processing Systems (pp. 2292-2300).
[13] J. Altschuler, J.Weed, P. Rigollet, (2017) Near-linear time approximation
algorithms for optimal transport via Sinkhorn iteration, Advances in Neural
Information Processing Systems (NIPS) 31
[14] Alaya M. Z., BΓ©rar M., Gasso G., Rakotomamonjy A. (2019). Screening Sinkhorn
Algorithm for Regularized Optimal Transport, Advances in Neural Information
Processing Systems 33 (NeurIPS).
[15] Taro Ichimura, Taishi Kakizuka, Kazuki Horikawa, β€œTrans-scale-scope to find rare
cellular activity in sub-million cells”,
bioRxiv 2020.06.29.179044; doi: https://doi.org/10.1101/2020.06.29.179044
[16] Kakizuka T, Hara Y, Ohta Y, et al. Cellular logics bringing the symmetry breaking
in spiral nucleation revealed by trans-scale imaging. bioRxiv; 2020. DOI:
10.1101/2020.06.29.176891.
[17] Luo, W., Xing, J., Milan, A., Zhang, X., Liu, W., Zhao, X., & Kim, T. (2014).
Multiple Object Tracking: A Literature Review. arXiv: Computer Vision and Pattern
Recognition, p. 10
[18] Fiaz, M., Mahmood, A., & Jung, S. (2019). Deep Siamese Networks toward Robust
Visual Tracking.
[19] Ulman, V., MaΕ‘ka, M., Magnusson, K., Ronneberger, O., Haubold, C., Harder, N.,
Matula, P., Matula, P., Svoboda, D., Radojevic, M., Smal, I., Rohr, K., JaldΓ©n, J.,
37
Blau, H., Dzyubachyk, O., Lelieveldt, B., Xiao, P., Li, Y., Cho, S., Dufour, A.C., Olivo-
Marin, J., Reyes-Aldasoro, C.C., SolΓ­s-Lemus, J.A., Bensch, R., Brox, T., Stegmaier, J.,
Mikut, R., Wolf, S., Hamprecht, F., Esteves, T., Quelhas, P., Demirel, Γ–., MalmstrΓΆm,
L., Jug, F., Tomançak, P., Meijering, E., Muñoz-Barrutia, A., Kozubek, M., & Ortiz-
de-SolΓ³rzano, C. (2017). An Objective Comparison of Cell Tracking Algorithms. Nature
methods, 14, 1141 - 1152.
[20] Harold W. Kuhn, "The Hungarian Method for the assignment problem", Naval
Research Logistics Quarterly, 2: 83–97, 1955.
[21] Jonker, R.; Volgenant, A. (December 1987). "A shortest augmenting path
algorithm for dense and sparse linear assignment problems". Computing. 38 (4): 325–
340. doi:10.1007/BF02278710.
[22] Marino, S.D., Gerolin, A. An Optimal Transport Approach for the SchrΓΆdinger
Bridge Problem and Convergence of Sinkhorn Algorithm. J Sci Comput 85, 27 (2020).
https://doi.org/10.1007/s10915-020-01325-7
[23] Nasser, L., & Boudier, T. (2019). A novel generic dictionary-based denoising
method for improving noisy and densely packed nuclei segmentation in 3D time-lapse
fluorescence microscopy images. Scientific Reports, 9.
[24] Bise, R., Maeda, Y., Kim, M.H., & Kino-oka, M. (2013). Cell tracking under high
confluency conditions by candidate cell region detection-based association approach.
[25] Tracking and Analysis of FUCCI-Labeled Cells Based on Particle Filters and
Time-to-Event Analysis, Kenji Fujimoto, Shigeto Seno, Hironori Shigeta, Tomohiro
Mashita, Masaru Ishii, Hideo Matsuda, International Journal of Bioscience,
Biochemistry and Bioinformatics,10(2): 94-109, 2020.04, Papers.

More Related Content

What's hot

An Efficient Approach for Multi-Target Tracking in Sensor Networks using Ant ...
An Efficient Approach for Multi-Target Tracking in Sensor Networks using Ant ...An Efficient Approach for Multi-Target Tracking in Sensor Networks using Ant ...
An Efficient Approach for Multi-Target Tracking in Sensor Networks using Ant ...ijsrd.com
Β 
Data integration lab_meeting
Data integration lab_meetingData integration lab_meeting
Data integration lab_meetingLiangqun Lu
Β 
Comparison Between Clustering Algorithms for Microarray Data Analysis
Comparison Between Clustering Algorithms for Microarray Data AnalysisComparison Between Clustering Algorithms for Microarray Data Analysis
Comparison Between Clustering Algorithms for Microarray Data AnalysisIOSR Journals
Β 
Caenorhabditis elegans segmentation using texture based models for motility p...
Caenorhabditis elegans segmentation using texture based models for motility p...Caenorhabditis elegans segmentation using texture based models for motility p...
Caenorhabditis elegans segmentation using texture based models for motility p...ieeepondy
Β 
Bw4201485492
Bw4201485492Bw4201485492
Bw4201485492IJERA Editor
Β 
A comparative study of clustering and biclustering of microarray data
A comparative study of clustering and biclustering of microarray dataA comparative study of clustering and biclustering of microarray data
A comparative study of clustering and biclustering of microarray dataijcsit
Β 
D05222528
D05222528D05222528
D05222528IOSR-JEN
Β 

What's hot (8)

An Efficient Approach for Multi-Target Tracking in Sensor Networks using Ant ...
An Efficient Approach for Multi-Target Tracking in Sensor Networks using Ant ...An Efficient Approach for Multi-Target Tracking in Sensor Networks using Ant ...
An Efficient Approach for Multi-Target Tracking in Sensor Networks using Ant ...
Β 
Data integration lab_meeting
Data integration lab_meetingData integration lab_meeting
Data integration lab_meeting
Β 
Comparison Between Clustering Algorithms for Microarray Data Analysis
Comparison Between Clustering Algorithms for Microarray Data AnalysisComparison Between Clustering Algorithms for Microarray Data Analysis
Comparison Between Clustering Algorithms for Microarray Data Analysis
Β 
Caenorhabditis elegans segmentation using texture based models for motility p...
Caenorhabditis elegans segmentation using texture based models for motility p...Caenorhabditis elegans segmentation using texture based models for motility p...
Caenorhabditis elegans segmentation using texture based models for motility p...
Β 
Bw4201485492
Bw4201485492Bw4201485492
Bw4201485492
Β 
Thesis Presentation
Thesis PresentationThesis Presentation
Thesis Presentation
Β 
A comparative study of clustering and biclustering of microarray data
A comparative study of clustering and biclustering of microarray dataA comparative study of clustering and biclustering of microarray data
A comparative study of clustering and biclustering of microarray data
Β 
D05222528
D05222528D05222528
D05222528
Β 

Similar to Large scale cell tracking using an approximated Sinkhorn algorithm

8 A Cellular Neural Network based system for cell counting in culture of biol...
8 A Cellular Neural Network based system for cell counting in culture of biol...8 A Cellular Neural Network based system for cell counting in culture of biol...
8 A Cellular Neural Network based system for cell counting in culture of biol...Cristian Randieri PhD
Β 
Bo Dong-ISBI2015-CameraReady
Bo Dong-ISBI2015-CameraReadyBo Dong-ISBI2015-CameraReady
Bo Dong-ISBI2015-CameraReadyBo Dong
Β 
Segmenting Epithelial Cells in High-Throughput RNAi Screens (Miaab 2011)
Segmenting Epithelial Cells in High-Throughput RNAi Screens (Miaab 2011)Segmenting Epithelial Cells in High-Throughput RNAi Screens (Miaab 2011)
Segmenting Epithelial Cells in High-Throughput RNAi Screens (Miaab 2011)Kevin Keraudren
Β 
Melanoma Cell Detection in Lymph Nodes Histopathological Images using Deep Le...
Melanoma Cell Detection in Lymph Nodes Histopathological Images using Deep Le...Melanoma Cell Detection in Lymph Nodes Histopathological Images using Deep Le...
Melanoma Cell Detection in Lymph Nodes Histopathological Images using Deep Le...sipij
Β 
Articles -Signal & Image Processing: An International Journal (SIPIJ)
Articles -Signal & Image Processing: An International Journal (SIPIJ)Articles -Signal & Image Processing: An International Journal (SIPIJ)
Articles -Signal & Image Processing: An International Journal (SIPIJ)sipij
Β 
MELANOMA CELL DETECTION IN LYMPH NODES HISTOPATHOLOGICAL IMAGES USING DEEP LE...
MELANOMA CELL DETECTION IN LYMPH NODES HISTOPATHOLOGICAL IMAGES USING DEEP LE...MELANOMA CELL DETECTION IN LYMPH NODES HISTOPATHOLOGICAL IMAGES USING DEEP LE...
MELANOMA CELL DETECTION IN LYMPH NODES HISTOPATHOLOGICAL IMAGES USING DEEP LE...sipij
Β 
Intelligent algorithms for cell tracking and image segmentation
Intelligent algorithms for cell tracking and image segmentationIntelligent algorithms for cell tracking and image segmentation
Intelligent algorithms for cell tracking and image segmentationijcsit
Β 
Intelligent algorithms for cell tracking and image segmentation
Intelligent algorithms for cell tracking and image segmentationIntelligent algorithms for cell tracking and image segmentation
Intelligent algorithms for cell tracking and image segmentationijcsit
Β 
Optical flow based analysis of leukocyte
Optical flow based analysis of leukocyteOptical flow based analysis of leukocyte
Optical flow based analysis of leukocyteijaia
Β 
scRNA-Seq Workshop Presentation - Stem Cell Network 2018
scRNA-Seq Workshop Presentation - Stem Cell Network 2018scRNA-Seq Workshop Presentation - Stem Cell Network 2018
scRNA-Seq Workshop Presentation - Stem Cell Network 2018David Cook
Β 
ΞœΞΏΞ½Ο„Ξ­Ξ»Ξ± διάχυσης καρκινικών ΟŒΞ³ΞΊΟ‰Ξ½ ΡγκΡφάλου
ΞœΞΏΞ½Ο„Ξ­Ξ»Ξ± διάχυσης καρκινικών ΟŒΞ³ΞΊΟ‰Ξ½ Ξ΅Ξ³ΞΊΞ΅Ο†Ξ¬Ξ»ΞΏΟ…ΞœΞΏΞ½Ο„Ξ­Ξ»Ξ± διάχυσης καρκινικών ΟŒΞ³ΞΊΟ‰Ξ½ ΡγκΡφάλου
ΞœΞΏΞ½Ο„Ξ­Ξ»Ξ± διάχυσης καρκινικών ΟŒΞ³ΞΊΟ‰Ξ½ ΡγκΡφάλουManolis Vavalis
Β 
Tracking of Fluorescent Cells Based on the Wavelet Otsu Model
Tracking of Fluorescent Cells Based on the Wavelet Otsu ModelTracking of Fluorescent Cells Based on the Wavelet Otsu Model
Tracking of Fluorescent Cells Based on the Wavelet Otsu Modelrahulmonikasharma
Β 
Gaussian Multi-Scale Feature Disassociation Screening in Tuberculosise
Gaussian Multi-Scale Feature Disassociation Screening in TuberculosiseGaussian Multi-Scale Feature Disassociation Screening in Tuberculosise
Gaussian Multi-Scale Feature Disassociation Screening in Tuberculosiseijceronline
Β 
An Efficient Automatic Segmentation Method For Leukocytes
An Efficient Automatic Segmentation Method For LeukocytesAn Efficient Automatic Segmentation Method For Leukocytes
An Efficient Automatic Segmentation Method For LeukocytesCSCJournals
Β 
CONVOLUTIONAL NEURAL NETWORK BASED RETINAL VESSEL SEGMENTATION
CONVOLUTIONAL NEURAL NETWORK BASED RETINAL VESSEL SEGMENTATIONCONVOLUTIONAL NEURAL NETWORK BASED RETINAL VESSEL SEGMENTATION
CONVOLUTIONAL NEURAL NETWORK BASED RETINAL VESSEL SEGMENTATIONCSEIJJournal
Β 
Convolutional Neural Network based Retinal Vessel Segmentation
Convolutional Neural Network based Retinal Vessel SegmentationConvolutional Neural Network based Retinal Vessel Segmentation
Convolutional Neural Network based Retinal Vessel SegmentationCSEIJJournal
Β 
CLASSIFICATION AND SEGMENTATION OF LEUKEMIA USING CONVOLUTION NEURAL NETWORK
CLASSIFICATION AND SEGMENTATION OF LEUKEMIA USING CONVOLUTION NEURAL NETWORKCLASSIFICATION AND SEGMENTATION OF LEUKEMIA USING CONVOLUTION NEURAL NETWORK
CLASSIFICATION AND SEGMENTATION OF LEUKEMIA USING CONVOLUTION NEURAL NETWORKIRJET Journal
Β 
Statistical Analysis of Skin Cell Geometry and Motion
Statistical Analysis of Skin Cell Geometry and MotionStatistical Analysis of Skin Cell Geometry and Motion
Statistical Analysis of Skin Cell Geometry and MotionAmy Werner-Allen
Β 
An Efficient VLSI Design for Extracting Local Binary Pattern
An Efficient VLSI Design for Extracting Local Binary PatternAn Efficient VLSI Design for Extracting Local Binary Pattern
An Efficient VLSI Design for Extracting Local Binary PatternIJTET Journal
Β 

Similar to Large scale cell tracking using an approximated Sinkhorn algorithm (20)

8 A Cellular Neural Network based system for cell counting in culture of biol...
8 A Cellular Neural Network based system for cell counting in culture of biol...8 A Cellular Neural Network based system for cell counting in culture of biol...
8 A Cellular Neural Network based system for cell counting in culture of biol...
Β 
Bo Dong-ISBI2015-CameraReady
Bo Dong-ISBI2015-CameraReadyBo Dong-ISBI2015-CameraReady
Bo Dong-ISBI2015-CameraReady
Β 
Segmenting Epithelial Cells in High-Throughput RNAi Screens (Miaab 2011)
Segmenting Epithelial Cells in High-Throughput RNAi Screens (Miaab 2011)Segmenting Epithelial Cells in High-Throughput RNAi Screens (Miaab 2011)
Segmenting Epithelial Cells in High-Throughput RNAi Screens (Miaab 2011)
Β 
Melanoma Cell Detection in Lymph Nodes Histopathological Images using Deep Le...
Melanoma Cell Detection in Lymph Nodes Histopathological Images using Deep Le...Melanoma Cell Detection in Lymph Nodes Histopathological Images using Deep Le...
Melanoma Cell Detection in Lymph Nodes Histopathological Images using Deep Le...
Β 
Articles -Signal & Image Processing: An International Journal (SIPIJ)
Articles -Signal & Image Processing: An International Journal (SIPIJ)Articles -Signal & Image Processing: An International Journal (SIPIJ)
Articles -Signal & Image Processing: An International Journal (SIPIJ)
Β 
MELANOMA CELL DETECTION IN LYMPH NODES HISTOPATHOLOGICAL IMAGES USING DEEP LE...
MELANOMA CELL DETECTION IN LYMPH NODES HISTOPATHOLOGICAL IMAGES USING DEEP LE...MELANOMA CELL DETECTION IN LYMPH NODES HISTOPATHOLOGICAL IMAGES USING DEEP LE...
MELANOMA CELL DETECTION IN LYMPH NODES HISTOPATHOLOGICAL IMAGES USING DEEP LE...
Β 
Intelligent algorithms for cell tracking and image segmentation
Intelligent algorithms for cell tracking and image segmentationIntelligent algorithms for cell tracking and image segmentation
Intelligent algorithms for cell tracking and image segmentation
Β 
Intelligent algorithms for cell tracking and image segmentation
Intelligent algorithms for cell tracking and image segmentationIntelligent algorithms for cell tracking and image segmentation
Intelligent algorithms for cell tracking and image segmentation
Β 
Optical flow based analysis of leukocyte
Optical flow based analysis of leukocyteOptical flow based analysis of leukocyte
Optical flow based analysis of leukocyte
Β 
scRNA-Seq Workshop Presentation - Stem Cell Network 2018
scRNA-Seq Workshop Presentation - Stem Cell Network 2018scRNA-Seq Workshop Presentation - Stem Cell Network 2018
scRNA-Seq Workshop Presentation - Stem Cell Network 2018
Β 
ΞœΞΏΞ½Ο„Ξ­Ξ»Ξ± διάχυσης καρκινικών ΟŒΞ³ΞΊΟ‰Ξ½ ΡγκΡφάλου
ΞœΞΏΞ½Ο„Ξ­Ξ»Ξ± διάχυσης καρκινικών ΟŒΞ³ΞΊΟ‰Ξ½ Ξ΅Ξ³ΞΊΞ΅Ο†Ξ¬Ξ»ΞΏΟ…ΞœΞΏΞ½Ο„Ξ­Ξ»Ξ± διάχυσης καρκινικών ΟŒΞ³ΞΊΟ‰Ξ½ ΡγκΡφάλου
ΞœΞΏΞ½Ο„Ξ­Ξ»Ξ± διάχυσης καρκινικών ΟŒΞ³ΞΊΟ‰Ξ½ ΡγκΡφάλου
Β 
Tracking of Fluorescent Cells Based on the Wavelet Otsu Model
Tracking of Fluorescent Cells Based on the Wavelet Otsu ModelTracking of Fluorescent Cells Based on the Wavelet Otsu Model
Tracking of Fluorescent Cells Based on the Wavelet Otsu Model
Β 
Gaussian Multi-Scale Feature Disassociation Screening in Tuberculosise
Gaussian Multi-Scale Feature Disassociation Screening in TuberculosiseGaussian Multi-Scale Feature Disassociation Screening in Tuberculosise
Gaussian Multi-Scale Feature Disassociation Screening in Tuberculosise
Β 
An Efficient Automatic Segmentation Method For Leukocytes
An Efficient Automatic Segmentation Method For LeukocytesAn Efficient Automatic Segmentation Method For Leukocytes
An Efficient Automatic Segmentation Method For Leukocytes
Β 
CONVOLUTIONAL NEURAL NETWORK BASED RETINAL VESSEL SEGMENTATION
CONVOLUTIONAL NEURAL NETWORK BASED RETINAL VESSEL SEGMENTATIONCONVOLUTIONAL NEURAL NETWORK BASED RETINAL VESSEL SEGMENTATION
CONVOLUTIONAL NEURAL NETWORK BASED RETINAL VESSEL SEGMENTATION
Β 
Convolutional Neural Network based Retinal Vessel Segmentation
Convolutional Neural Network based Retinal Vessel SegmentationConvolutional Neural Network based Retinal Vessel Segmentation
Convolutional Neural Network based Retinal Vessel Segmentation
Β 
CLASSIFICATION AND SEGMENTATION OF LEUKEMIA USING CONVOLUTION NEURAL NETWORK
CLASSIFICATION AND SEGMENTATION OF LEUKEMIA USING CONVOLUTION NEURAL NETWORKCLASSIFICATION AND SEGMENTATION OF LEUKEMIA USING CONVOLUTION NEURAL NETWORK
CLASSIFICATION AND SEGMENTATION OF LEUKEMIA USING CONVOLUTION NEURAL NETWORK
Β 
Statistical Analysis of Skin Cell Geometry and Motion
Statistical Analysis of Skin Cell Geometry and MotionStatistical Analysis of Skin Cell Geometry and Motion
Statistical Analysis of Skin Cell Geometry and Motion
Β 
An Efficient VLSI Design for Extracting Local Binary Pattern
An Efficient VLSI Design for Extracting Local Binary PatternAn Efficient VLSI Design for Extracting Local Binary Pattern
An Efficient VLSI Design for Extracting Local Binary Pattern
Β 
AnoMalNet: outlier detection based malaria cell image classification method l...
AnoMalNet: outlier detection based malaria cell image classification method l...AnoMalNet: outlier detection based malaria cell image classification method l...
AnoMalNet: outlier detection based malaria cell image classification method l...
Β 

More from Parth Nandedkar

Discount tracker - Fujitsu Hackathon
Discount tracker - Fujitsu HackathonDiscount tracker - Fujitsu Hackathon
Discount tracker - Fujitsu HackathonParth Nandedkar
Β 
Line Detection in Computer Vision - Recent Developments and Applications
Line Detection in Computer Vision - Recent Developments and ApplicationsLine Detection in Computer Vision - Recent Developments and Applications
Line Detection in Computer Vision - Recent Developments and ApplicationsParth Nandedkar
Β 
Line Detection in Computer Vision
Line Detection in Computer VisionLine Detection in Computer Vision
Line Detection in Computer VisionParth Nandedkar
Β 
Speech at english_seminar
Speech at english_seminarSpeech at english_seminar
Speech at english_seminarParth Nandedkar
Β 
Permutations and Combinations IIT JEE+Olympiad Lecture 1
Permutations and Combinations IIT JEE+Olympiad Lecture 1 Permutations and Combinations IIT JEE+Olympiad Lecture 1
Permutations and Combinations IIT JEE+Olympiad Lecture 1 Parth Nandedkar
Β 
Permutations and Combinations IIT JEE+Olympiad Lecture 4
Permutations and Combinations IIT JEE+Olympiad Lecture 4Permutations and Combinations IIT JEE+Olympiad Lecture 4
Permutations and Combinations IIT JEE+Olympiad Lecture 4Parth Nandedkar
Β 
Permutations and Combinations IIT JEE+Olympiad Lecture 3
Permutations and Combinations IIT JEE+Olympiad Lecture 3 Permutations and Combinations IIT JEE+Olympiad Lecture 3
Permutations and Combinations IIT JEE+Olympiad Lecture 3 Parth Nandedkar
Β 
LSTM and GRU RNNs in Sentiment Analysis (Japanese)
LSTM and GRU RNNs in Sentiment Analysis (Japanese)LSTM and GRU RNNs in Sentiment Analysis (Japanese)
LSTM and GRU RNNs in Sentiment Analysis (Japanese)Parth Nandedkar
Β 
Deep Learning Demonstration using Tensorflow (7th lecture)
Deep Learning Demonstration using Tensorflow (7th lecture)Deep Learning Demonstration using Tensorflow (7th lecture)
Deep Learning Demonstration using Tensorflow (7th lecture)Parth Nandedkar
Β 
Basics of Deep Learning (2nd lecture)
Basics of Deep Learning (2nd lecture)Basics of Deep Learning (2nd lecture)
Basics of Deep Learning (2nd lecture)Parth Nandedkar
Β 

More from Parth Nandedkar (10)

Discount tracker - Fujitsu Hackathon
Discount tracker - Fujitsu HackathonDiscount tracker - Fujitsu Hackathon
Discount tracker - Fujitsu Hackathon
Β 
Line Detection in Computer Vision - Recent Developments and Applications
Line Detection in Computer Vision - Recent Developments and ApplicationsLine Detection in Computer Vision - Recent Developments and Applications
Line Detection in Computer Vision - Recent Developments and Applications
Β 
Line Detection in Computer Vision
Line Detection in Computer VisionLine Detection in Computer Vision
Line Detection in Computer Vision
Β 
Speech at english_seminar
Speech at english_seminarSpeech at english_seminar
Speech at english_seminar
Β 
Permutations and Combinations IIT JEE+Olympiad Lecture 1
Permutations and Combinations IIT JEE+Olympiad Lecture 1 Permutations and Combinations IIT JEE+Olympiad Lecture 1
Permutations and Combinations IIT JEE+Olympiad Lecture 1
Β 
Permutations and Combinations IIT JEE+Olympiad Lecture 4
Permutations and Combinations IIT JEE+Olympiad Lecture 4Permutations and Combinations IIT JEE+Olympiad Lecture 4
Permutations and Combinations IIT JEE+Olympiad Lecture 4
Β 
Permutations and Combinations IIT JEE+Olympiad Lecture 3
Permutations and Combinations IIT JEE+Olympiad Lecture 3 Permutations and Combinations IIT JEE+Olympiad Lecture 3
Permutations and Combinations IIT JEE+Olympiad Lecture 3
Β 
LSTM and GRU RNNs in Sentiment Analysis (Japanese)
LSTM and GRU RNNs in Sentiment Analysis (Japanese)LSTM and GRU RNNs in Sentiment Analysis (Japanese)
LSTM and GRU RNNs in Sentiment Analysis (Japanese)
Β 
Deep Learning Demonstration using Tensorflow (7th lecture)
Deep Learning Demonstration using Tensorflow (7th lecture)Deep Learning Demonstration using Tensorflow (7th lecture)
Deep Learning Demonstration using Tensorflow (7th lecture)
Β 
Basics of Deep Learning (2nd lecture)
Basics of Deep Learning (2nd lecture)Basics of Deep Learning (2nd lecture)
Basics of Deep Learning (2nd lecture)
Β 

Recently uploaded

Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...gurkirankumar98700
Β 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonetsnaman860154
Β 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxMalak Abu Hammad
Β 
Salesforce Community Group Quito, Salesforce 101
Salesforce Community Group Quito, Salesforce 101Salesforce Community Group Quito, Salesforce 101
Salesforce Community Group Quito, Salesforce 101Paola De la Torre
Β 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityPrincipled Technologies
Β 
Developing An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilDeveloping An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilV3cube
Β 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘RTylerCroy
Β 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreternaman860154
Β 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Drew Madelung
Β 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls
Β 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Servicegiselly40
Β 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)Gabriella Davis
Β 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024Rafal Los
Β 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdfhans926745
Β 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024The Digital Insurer
Β 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
Β 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationMichael W. Hawkins
Β 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Igalia
Β 
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptxFactors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptxKatpro Technologies
Β 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking MenDelhi Call girls
Β 

Recently uploaded (20)

Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Β 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
Β 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptx
Β 
Salesforce Community Group Quito, Salesforce 101
Salesforce Community Group Quito, Salesforce 101Salesforce Community Group Quito, Salesforce 101
Salesforce Community Group Quito, Salesforce 101
Β 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
Β 
Developing An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilDeveloping An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of Brazil
Β 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘
Β 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreter
Β 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Β 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
Β 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
Β 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
Β 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
Β 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
Β 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
Β 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
Β 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
Β 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Β 
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptxFactors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Β 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
Β 

Large scale cell tracking using an approximated Sinkhorn algorithm

  • 1. Large-scale cell tracking using an approximated Sinkhorn-Knopp algorithm Submitted to the School of Engineering Science, Information Science Department Osaka University February 2021 Parth Shirish Nandedkar
  • 2. 2 2020 Bachelor’s Thesis Large-scale cell tracking using an approximated Sinkhorn-Knopp algorithm Parth Shirish Nandedkar Abstract The popular field of Cell Tracking has a lot of recent developments, with the influx of many different deep learning methods. However, not only are cells far more numerous, the different kinds of organisms have cells with entirely different characteristics. Apart from these practical considerations, the issues of blurry intercellular boundaries and mitosis are also faced. It is for these reasons that much of biological research on tracking the behaviour of cells still involves manual annotation of cells in each photographed slide. In this research, we focus on vastly speeding up the tracking of cells, basing our experiments on the data for the multi-cellular pattern formation of over a hundred thousand cells of a microorganism, the Dictyostelium discoideum species of amoeba, in neurologically diseased animal subjects. The method follows the traditional two-phase (detection phase and association phase) approach to cell tracking. However, due to the sheer number of cells, faster, but less precise bipartite graph matching or linear programming (optimal transport) approaches were tested and a final candidate - a modified and greatly speeded-up Sinkhorn-Knopp algorithm, the Sparse Sinkhorn- Knopp algorithm - was reached. The proposed algorithm allows researchers to automate the annotation of cells, and in the case of the Dictyostelium discoideum cell data, allows for the association of over 10,000 detected cells into continuous cell tracks in a mere 25 seconds per slide of data, in 𝑂(𝑛) time and space complexity for 𝑛 cells. Keywords: Cell tracking, Multi-cellular patterns, Hungarian algorithm, Bipartite Matching, Optimal Transport, Sparse matrices, Sinkhorn-Knopp algorithm
  • 3. 3 Contents 1 Introduction 2 Related work 2.1 Deep Learning-based approaches 2.1.1 Motion and Position Map (MPM) method 2.1.2 CellNucleiTracker 2.2 Non-Deep Learning based approaches 2.2.1 CancerCellTracker 2.2.2 Macrophage tracker (Hungarian algorithm) 2.2.3 Candidate cell region detection-based association 2.2.4 Probabilistic event detection of stem cells 2.2.5 Tracking using Particle filters and Time-to-Event analysis 2.3 General multi object tracking methods 2.3.1 DeepMOT 2.3.2 Multi-object tracking using dynamical graph matching 3 Proposed method 3.1 Pipeline 3.2 Detection Phase 3.3 Tracklets generation 3.3.1 Features and cost matrix 3.3.2 Nearest Neighbor matching 3.3.3 Hungarian matching 3.3.4 Sinkhorn-Knopp matching 3.3.5 Sparse Sinkhorn-Knopp matching 3.4 Tracklets association
  • 4. 4 4 Experiments 4.1 Details on the dataset 4.2 Examples 4.3 Protocol 4.4 Tracking accuracy 4.5 Computational cost compared to pure Sinkhorn-Knopp algorithm 5 Conclusions Acknowledgements Bibliography
  • 5. 1 Chapter 1 Introduction There have been many advances in the neurological study of diseased or malnourished neural tissue. One of these advances is based on modelling the neurons with the Dictyostelium discoideum species of amoeba. The lifecycle of D. discoideum is relatively short, which allows for timely viewing of all stages. This makes it a model organism to study cellular and biochemical processes in neural tissues. However, due to its ever- changing nature and the large number of cells, the identification of cells fluorescent imaging data for these is still done manually. The goal of this research thesis is to automate this process of identifying, numbering and associating (joining the tracks of) each cell across multiple frames of fluorescent imaging data. When malnourished, thousands of dictyostelium cells collectively migrate and form patterns as shown in Figure 1.1 below. In particular, only a few β€˜leader’ cells release a small amount of Cyclic adenosine monophosphate (cAMP) as a signal to the remaining β€˜follower’ cells which then leads to this migration. Such collective cell migration contributes to many organismal phenomena in neural tissues, including morphogenesis, wound healing, and cancer invasion [1, 2], making the timely analysis of the movement of a 100,000 or more cells imperative for the study of cell-cell communication and desieases like brain cancer. The provided data has over a 100,000 dictyostelium cells laced with two different fluorescent protiens allowing for two different modes of fluorescent imaging of the amoeba.
  • 6. 2 Figure 1.1: The formation of circular patterns in malnourished Dictyostelium cells. Cells releasing cAMP are shown in red [15]. As this is a tell tracking problem, we will first discuss the basics method of cell tracking. Cell tracking consists of two phases, the detection phase, and the Association Phase [3]. The detection phase consists of identifying the bounding boxes of each cell and assigning temporary IDs to them in each frame. Recently, trained CNNs and RNNs are being used to detect the cells in each frame [4, 5, 6, 10, 17]. In the association phase, these detection results associated in successive time-frames to form tracklets, in a process called "Tracklets generation”. These tracklets are then associated wherever possible, to form continuous overall trajectories for as many cells as possible in the β€œGlobal Association” stage. This process is shown visually in Figure 1.2 below. The global association stage always follows the detection phase and the tracklets generation stage with a delay in order to gather as much data as required to associate a tracklet ending in the current frame to a new tracklet starting in the next few frames. Figure 1.2: The pipeline of most Cell Trackers [3] The above method for tracking cells has many issues however, such as those shown below. Figure 1.3 shows examples of these issues, with 1.3(a) showing the case of
  • 7. 3 similarly shaped cells, leading many deep learning based methods and traditional object tracking astray. Figure 1.3(b), a part of our target dataset, shows the random shape changes many types of cells undergo from frame to frame. Figure 1.3(c), depicts the extremely blurry intercellular boundaries in our target dataset, which renders many cell segmentation algorithms futile. Lastly, Figure 1.3(d) is an example of Cell Mitosis occurring between frames, which needs its own algorithm to track newly separated cells, such as the Mitosis tracker developed by S. Huh [7]. The fifth and most important issue for practical purposes is temporal efficiency or computational cost, which rules out most existing cell tracking or multi object tracking (MOT) methods for this research. Figure 1.3: (a) Example cell data, with each cell showing nearly identical shapes [23]. (b) The cells in the centre and the bottom right vastly change their shape and size respectively. (c) Cells stuck to each other with no clear intercellular boundaries in our dataset. (d) Cell mitosis occurs for the bright cell in the centre. At present, there are many different methods to track (associate) detected cells, and can be broadly classified into deep and non-deep. Deep learning methods for cell tracking have been developed over the last decade, and are based on detection using a RNN (or Bi-RNN), a U-net (a specialized CNN for medical image segmentation) or a siamese network [4, 5, 10, 17, 18] trained on sample images of the type of cell and fed cell ID annotation data. However, upon application of these methods to our target dataset, the shortcomings of all deep learning methods are clear. Adding annotation
  • 8. 4 data for a 100,000 cells impractical. On top of this, these networks also almost completely base their identification of cells upon existing characteristics of the specific type of cell, which inevitably requires high resolution biomedical imaging. This is not the case for our very low-resolution dataset as shown in Figure 1.1 or 1.3(b),(c). Since cell imaging is low resolution when imaging such a vast number of cells, we can conclude that deep learning methods will be inaccurate and largely overfitted. Apart from these issues, neural networks may not be able to keep track of the constant cell shape changes of the Dictyostelium discoideum amoeba, as discussed in detail in the related work section. In non-deep learning approaches, we have the basic K-nearest neighbour (KNN) algorithm, bipartite graph matching algorithms, and the linear programming based optimal transport algorithms [19 , 9, 12]. We use KNN as a baseline for testing the other methods here, namely the Hungarian bipartite matching algorithm and the Sinkhorn-Knopp optimal transport algorithm. The Hungarian algorithm, or the Kuhn- Munkres algorithm [20] is a simple linear programming approach to solving the general assignment problem provided the costs (based on similarity metrics) between every combination of cells in two different groups of equal number of cells. The advantage with this method is the ability to get a robust matching between two successive frames of cells (current frame 𝑑 and previous frame 𝑑 βˆ’ 1) without impractical amounts of annotation or training. However, the naΓ―ve Hungarian algorithm has 𝑂(𝑛 ) computational cost (where n is the number of cells). This can be reduced to 𝑂(π‘˜π‘› ) by limiting the scope of matching to only the K nearest neighbours (Hungarian-on-KNN, or the sparse assignment problem [21]). This makes for a small 𝑂(π‘˜π‘›) space cost matrix and a decently fast match for a thousand cells (about 40 seconds per match for 2,000 cells), however it still takes more than 15 minutes to match two frames of 10,000 cells each. The Sinkhorn-Knopp algorithm [12, 13, 14] is also a linear programming based approach, which, when provided the costs between cells in one frame and in another frame, converts these costs into entropies, and maximizes the sum of these entropies, closing in to an asymptotic solution, while never truly reaching the exact solution [13]. The advantages with this optimal transport algorithm are numerous, starting from having no need of annotation or training, to the temporal and spatial cost [12]. It also allows for specifying the weights of each object in the form of probability vectors, which
  • 9. 5 in our case corresponds to the total brightness of the cell. When two cells visually merge, the Sinkhorn algorithm detects that their brightness has been added, and matches these merger cells to the merged cell. The Sinkhorn-Knopp algorithm is 𝑂(𝑛 ) in time and space (due to the matrix operations), making it a much better candidate for speeding it up by tinkering with the process. The algorithm proposed in this paper for solving the issue of computational cost for large-scale cell tracking is the Sparse Sinkhorn-Knopp algorithm. This is a modification of the Sinkhorn-Knopp algorithm, in which we generate a sparse cost matrix by calculating costs only for a limited number of candidate cells matching certain criteria. This takes advantage of the faster 𝑂(π‘˜π‘›) sparse matrix multiplications, making the computational cost of the match 𝑂(π‘˜π‘›) for n cells. Also, another major advantage of the proposed algorithm also 𝑂(π‘˜π‘›) in spatial complexity since the sparse cost matrix can be stored in any of the CSR, CSC, COO or LIL formats. This also tackles the shape-changing nature of amoeba and neurons by not relying on learning cell shape and structure. The issues of blurry intercellular boundaries and cell mitosis remain however, making the global association of tracklets inaccurate for this method in its current state.
  • 10. 6 Chapter 2 Related work There have been several published methods in order to deal with issues such as inaccurate cell segmentation and cell mitosis, however as this research focuses on mainly the time and space complexity encountered in forming cell tracks, most of the related work below was scrapped in order to generate the sparse version of the Sinkhorn-Knopp algorithm. All the tried and tested methods have been listed and discussed below. Some of these methods are general multi-object trackers. 2.1 Deep learning-based approaches 2.1.1 Motion and Position Map(MPM) method The MPM method is a recently developed cutting-edge cell detection method [4]. It uses U-nets to both detect and associate cells, the tracklet formation being based on a likelihood estimation formula which uses the cell’s motion data for the next few frames. In the detection phase, each detected cell is first given an ID using a Likelihood Map (LM) generated by using a U-net trained for cell detection of the specific type of cell. As the first step to associate the cells across frames, tracklets must be generated. For this purpose, Cell Motion Fields (CMFs) for different frame intervals (2 frames ,3 frames and so on) are generated as shown in Figure 2.1. The CMF gives us data on the direction and magnitude of each cell’s motion. A predicted matching is made based on the Motion and Position Map calculated from a formula with trainable weights.
  • 11. 7 Figure 2.1: The Cell Motion Field (CMF) used for matching, the vectors show the magnitude and direction of displacement in the duration of the interval [4] The training of the MPM-net (the U-net performing the matches) consists of five steps. We take two successive frames and find the relative cell position and motion, each as annotated in Figure 2.2(a). Next, we calculate the β€˜Cell Motion Vectors’(CMV) which are the Motion vectors from the neighbor points around the annotated cell of ID β€˜π‘–β€™ in frame β€˜π‘‘β€™ to the annotated cell in frame β€˜π‘‘ βˆ’ 1’. This collection of vectors is shown in blue in Figure 2.2(b). Next the β€˜Cell Position Likelihood Map’ (CPLM) is calculated which, in a similar fashion to random walk theory, is a 3-D Gaussian distribution around the β€˜π‘‘ βˆ’ 1’ position. as in Figure 2.2(c). The Motion and Position Map (MPM) is formed from the above CMV and CPLM data using a formula which contains trainable weights. Finally, a CNN (U-net) is trained on the 3D vector data of this MPM using MSE loss which trains the weights in the formula. This is the β€˜MPM-Net’. The main advantage of this method is since the number of trainable weights is minimal, the simplest of CNNs can function as MPM-Nets, and with great accuracy. Also, since MPM represents both detection and association, it guarantees coherence, i.e., if a cell is detected, an association of the cell is always produced. Other similar CMF-based or multi-tasking learning based methods [5] have the tendency to be incoherent, due to having independent detection and association phases.
  • 12. 8 Figure 2.2: The calculation of the MPM from the CMV and the CPLM [4]. However, upon application of this method to a small sample target dataset, the flaws of this approach in relation to the given problem will be met. The MPM-net, while requiring fewer epochs and training samples to train than other deep learning based methods, still needs large-scale annotations. Also, coherence between association and detection phases does not solve the issues of shape-shifts and blurry boundaries in the detection phase. On the contrary, the training and the results of the simpler MPM- net architecture is highly susceptible to such detection lapses. 2.1.2 CellNucleiTracker Another NN (U-net) based method [6], which is based on learning the features of the cells, especially the location of the nuclei. As seen in Figure 2.3, this method was nearly perfectly accurate for plant cells, and epithelial or circulatory tissue, which are largely immobile cells. Also, the resolution of the frames is much higher, allowing the network to learn the different features of each type of cell.
  • 13. 9 Figure 2.3: The pretrained network of CellNucleiTracker showed nearly perfect detection and segmentation for high resolution images of plant and animal tissue [6]. Since such conditions are not present in our dataset, this method was very inaccurate for the Dictyostelium cell dataset, as seen in the results of the pre-trained network (Figure 2.4). Figure 2.4: The pretrained network could not handle the provided low resolution frames of Dictyostelium amoeba cells
  • 14. 10 2.2 Non-deep learning based approaches 2.2.1 CancerCellTracker CancerCellTracker is an OpenSource project for solving the Fluo-H2DL-HeLa dataset of the cell tracking challenge [8]. The dataset consists of cancerous cells with cell mitosis constantly occurring. It uses the watershed algorithm to segment cells and builds a feature vector for cell tracking including the information of position, shape, spatial distribution and texture as discussed in Li et. al. [8]. It also has a cell family history tracking particularly useful for cancerous cells. Not requiring annotation and training, this method is also faster than the MPM and DeepMOT methods discussed in this section, even when obtaining the results. However identity switching is common in the results as in Figure 2.5, and cell family history tracking is not useful for dictyostelium cells which, like the neural cells it is a model for, rarely undergo mitosis, in stark contrast to cancerous cells. Due to major accuracy concerns, this method is not robust enough for large-scale cell tracking. Figure 2.5: CancerCellTracker shows many ID switches even in the first few frames.
  • 15. 11 2.2.2 Macrophage tracker (Hungarian algorithm) This macrophage cell tracker [9] is based on one of the simplest bipartite graph matching algorithm, the Hungarian algorithm, and is hence a good candidate for our large-scale cell tracking problem. The proposed method is an automated single macrophage tracking method for a dataset of a mouse brain (11.7 T time-lapse MR images). The method detects macrophages using background subtraction algorithm, and tracks macrophages using the Hungarian algorithm, backed up by a Kalman filter for association. The results showed that the proposed method detected and tracked macrophages in MR images successfully. The below table (Table 2.1) from the paper shows an example matching obtained by the Hungarian algorithm between cells in frame β€˜π‘‘β€™ and frame β€˜π‘‘ + 1’. Table 2.1: An example matching using the Hungarian algorithm. The Red numbers represent the highest probability match for the cells in β€˜π‘‘ + 1’, and the light red boxes represent the final match chosen, after resolving conflicts [9]. However, despite the application of the Kalman filter in this method, the issues with this method remain the same as for the optimized Hungarian algorithm (Hungarian-on- KNN) discussed in the Introduction section. This algorithm is still 𝑂(𝑛 ) in time complexity after limiting the scope of matching to the nearest neighbours, and hence, despite the accuracy of association, is rendered unsuitable. 2.2.3 Candidate cell region detection-based association In this paper [24], the proposed method tracks cells under highly confluent conditions by using the candidate cell region detection-based association approach. While the association phase is also a linear programming based approach like in our case, the detection phase is it’s unique point. Unlike conventional segmentation-based association tracking methods, the proposed method uses the tracking results from the previous frame to segment the cell regions at the current frame. First, candidate cell regions are
  • 16. 12 detected, and while there may be many false positives, there are very few false negatives. Next, optimized detection results are selected from the candidate regions and associated with the tracking results of the previous frame by resolving a linear programming problem. This non-deep learning approach is very for our current research, as it has the potential to be fast enough for the requisite number of cells. Many ideas discussed in the proposed method section are similar to those discussed in this paper. 2.2.4 Probabilistic event detection of stem cells In this non-deep learning approach [7], a probabilistic model for event detection is proposed which can simultaneously identify spatio-temporal patch sequences that contain a mitotic event and also specify the time and location of cell birth events. The approach significantly outperforms previous approaches in terms of both detection accuracy and computational efficiency, when applied to multipotent C3H10T1/2 mesenchymal and C2C12 myoblastic stem cell populations. However, this is a method specialized for cells which undergo mitosis very frequently, (as in the case of cancer cells shown in Figure 2.5), as the semi-Markov model used adds a lot of unnecessary computation if the method is used on cells with infrequent mitosis events. 2.2.5 Tracking using Particle filters and Time-to-Event analysis This paper [25] proposes a method for spatiotemporal tracking and analysis of FUCCI (fluorescent ubiquitination-based cell cycle indicator)-labelled cells from time-lapse videos. To address the colour transition of the FUCCI-labelled cell with the cell cycle progression, the proposed method simultaneously estimates the location and the cell cycle phase of the target cell. Furthermore, to analyse the cell phase transition, this paper proposes to apply multistate time-to-event analysis to the information obtained through our tracking method. This paper demonstrates the usefulness of the method with application to FUCCI-labelled HuH7 cells (human hepatocellular carcinoma cell line). The method was deemed to be increasingly complex for the problem at hand, since cell-cycle analysis is unnecessary for our research. The particle filters are a useful idea
  • 17. 13 for large-scale cell tracking, allowing global association in relatively quick time, and this is hence one of the main proposed improvements to our cell tracker for the future. 2.3 General multi-object tracking methods 2.3.1 DeepMOT DeepMOT [10] specializes in removing false positives and false negatives in general multi-object tracking, while also preventing unnecessary ID changes during the cell track. It achieves this by using a trained Bidirectional RNN (Bi-RNN) to perform the assignment on a distance (cost) matrix). The detailed pipeline for this method is shown in Figure 2.6. Figure 2.6: The pipeline of DeepMOT. The distance matrix D is flattened and fed to a Bi-RNN at two steps of the process for learning, and the recovered matrix is taken as the assignment matrix A [10]. There are four steps to achieve this extremely accurate matching between the objects. First, we randomly sample a pair of consecutive frames from the training video sequences. These two images together with their ground-truth bounding boxes constitute one training instance. Then, for each such instance, we first initialize the tracks with ground-truth bounding boxes (at time 𝑑) and run the forward pass to obtain the track's bounding-box predictions in the following video frame (time 𝑑 + 1). To mimic the effect of imperfect detections, we add random perturbations to the ground-truth bounding. Thirdly, we compute D (distance matrix in Figure 2.7) and use the proposed Deep Hungarian Network (a Bi-RNN) to compute A (the assignment matrix). Finally, we compute our proxy loss based on D and A. This provides us with a gradient that accounts for the assignment, and that is used to update the weights of the tracker. Thus, the weights of the tracker are updated as the tracking results are generated, simplifying the process of training and testing. Figure 2.7 depicts the flow of data and
  • 18. 14 the back-propagation of gradients with the Deep Hungarian Network (the Bi-RNN) in the centre. Figure 2.8 is a frame of the result obtained. Figure 2.7: The pipeline of DeepMOT including detection, association and training. The pink box represents the training stage, where the DHN and the DeepMOT Loss alternate till the training is complete [10]. As is evident in the small example frames in Figure 2.7 and the training data proposed in [10], this method which uses a complex Bi-RNN architecture is suitable only for a few dozen objects in-frame. Because of the use of a complex network, it is very well suited for large-sized objects with many features to learn from. Neither of these match with the conditions of large-scale cell tracking however, where the goal is a 100,000 cells with a very low resolution of imaging, few features and constantly changing shapes. 2.3.2 Multi-object tracking using dynamical graph matching This association part of this method is based on a tracking algorithm to address the interactions among objects, and to track them individually for a static video camera [11]. It is achieved by constructing an invariant bipartite graph to model the dynamics of the tracking process, of which the nodes are classified into objects and profiles as shown in Figure 2.9(a),(b) and (c).
  • 19. 15 Figure 2.9: (a) End of frame β€˜π‘‘ βˆ’ 1’. The matching information is used to update the profiles used in frame β€˜π‘‘β€™. (b) Beginning of frame β€˜π‘‘β€™. The bipartite graph has only 2 profile nodes. (c) Now 3 objects are detected so the graph now has 3 new object nodes. (d) A bipartite matching is carried out. Since there is one unmatched object node, a profile node is created to match it as in (e) and (f) [11]. The matching occurs in as shown in Figure 2.9(d),(e) and (f), and includes ad-hoc approaches to solve object interaction issues. First, the matching cost between a profile node and an object node is measured by the Kullback-Leibler distance on the image data distribution of the nodes. Now, if there exist unmatched object nodes, then new profiles are is created for each newly detected object. When an object leaves the scene, its corresponding profile will become unmatched. After finding the optimal bipartite matching, we check all profiles to see if there are some profiles getting too close to others. Those profiles will be marked as ”TBM” (to-be-merged). In the next image frame, the TBM profiles will be processed separately to handle possible interactions. This simple assignment method is designed for ever-changing numbers of cells and is quite appropriate for the given dataset. However, as the number of objects for the video data used in this research is small, the computational complexity has not been considered. A simpler form of the TBM profile processing feature was implemented in the form of a β€˜merge dictionary’ in the tracklet association stage (Chapter 3.4).
  • 20. 16 Chapter 3 Proposed method 3.1 Pipeline The proposed method in order to solve the time and space complexity issue with large- scale cell tracking is the Sparse Sinkhorn-Knopp algorithm. The pipeline is as shown in Figure 3.1, and has the same broad structure of detection followed by tracklet formation and ultimately global association of these tracklets. Figure 3.1: The pipeline of the proposed method. It follows the same pattern as the generic tracking pipeline in Figure 1.2. 3.2 Detection phase We begin with a detection phase that is independent of the association phase, based on a blob (maxima) detection algorithm on Laplacian of Gaussian (LoG) filtered frames with non-maxima suppression. This method was chosen for being among the quickest available methods. The LoG filter helps in boosting the brightness of the peaks compared to the surroundings. The method used computes the LoG images with successively increasing standard deviations and stacks them up in a cube data structure. The blobs are the local maxima in this cube after non-maxima suppression.. The equation for the LoG filter is as follows directly from spatial image processing theory, with an added Gaussian filter of standard deviation 𝜎. Various 𝜎 will be used in the method. π‘₯, 𝑦 are the two dimensional variables used to specify the filter’s equation.
  • 21. 17 The 2D function looks as shown in Figure 3.2, and has the effect of highlighting edges and peaks, the latter being our target of detection. Figure 3.2: The LoG filter calculates the second derivatve of the image in the 2D space domain. This leads to highlighted edges, and therefore, peaks. The usual disadvantage of the LoG approach is that detecting larger blobs is especially slower because of larger kernel sizes during convolution. However, since D. discoideum cells very have similar sizes, the kernel size can be limited to a small range, and the detection can occur smoothly in the expected 𝑂(𝑛) time. Alternative methods are by applying the Difference of Gaussian (DoG) or the Determinant of Hessian (DoH) filters instead, to speed up the detection even further. However, a simple comparison of these three filters (Figure 3.3) shows the massive loss in accuracy for the DoG and DoH filters.
  • 22. 18 Figure 3.3: A comparison of the LoG, the DoG and the DoH filters used in peak extraction. The latter two while faster and widely used as approximations, are both largely inaccurate for our low resolution target data. These detected blobs are then sorted into an adjustable number of bins as shown in Figure 3.4, based on their position on the frame, with bin 1 starting on the top left. This is done so as to allow for optimized Sinkhorn-Knopp matches later on, by considering matches for a cell in frame β€˜π‘‘β€™ only from frame β€˜π‘‘ βˆ’ 1’ cells in the same bin or the surrounding bins. Hence, each detected blob consist of four parameters, the x and y coordinates, the radius of the circular blob, and the bin in which the blob is present. Figure 3.4: The division of all the cells in each frame into bins of equal sizes, allowing for the removal of the vast majority of the candidate matches faster than a KNN approach. Bin numbering starts from the top left.
  • 23. 19 3.3 Tracklets generation 3.3.1 Features and cost matrix As this is a bipartite matching or assignment problem, the costs or dissimilarities between cells in frame β€˜π‘‘β€™ and frame β€˜π‘‘ βˆ’ 1’ must be arranged in a cost matrix. The features used for calculating dissimilarity are the distance 𝑑 between the two cells in the adjacent frames, the radii of the blobs, the brightness at the peaks (averaged with the surrounding pixels), and the net brightness of all pixels that make up the cell blobs. The last feature is useful because it is directly proportional to the amount of fluorescent protein present inside the cell. The blob radii, peak brightness values and brightness sum values are all averaged for each detection of each cell, to remove variance and get more accurate data. A simple cost function that provides decent results is as shown below. Features: Euclidean distance moved 𝑑, radii π‘Ÿ and π‘Ÿ , peak brightness values 𝑏 and 𝑏 , brightness sum values 𝑠 and 𝑠 (distances in pixels, brightness from 0 to 255, 𝑀 are the weights). πΆπ‘œπ‘ π‘‘ = 𝑑 + 𝑀 Γ— (π‘Ÿ βˆ’ π‘Ÿ ) + 𝑀 Γ— (𝑏 βˆ’ 𝑏 ) + 𝑀 Γ— (𝑠 βˆ’ 𝑠 ) Figure 3.5 is an example cost matrix when the cells are divided into 100 bins. Figure 3.5: An example cost matrix when the cells are sorted into 100 bins. The columns represent the cell IDs in frame β€˜π‘‘ βˆ’ 1’, and the rows represent the temporary IDs before matching in frame β€˜π‘‘β€™.
  • 24. 20 3.3.2 Nearest neighbour matching The simplest method for matching the bipartite graph of cells is the K nearest neighbour algorithm. In this approach, each cell in frame β€˜π‘‘β€™ is first matched with its closest cell in frame β€˜π‘‘ βˆ’ 1’ terms of the cost function. If two or more cells match to the same cell in frame β€˜π‘‘ βˆ’ 1’, an ad-hoc algorithm sorts out these cells to match their next closest cells in frame β€˜π‘‘β€™, considering up to K nearest neighbours. If in the previous case, however, the cost function for each of the K nearest neighbours is above a certain threshold, then the cell is regarded as different from any corresponding cell in frame β€˜π‘‘ βˆ’ 1’, and is given a new ID number. These new IDs largely occur in three cases, in the case of a genuinely new cell (cell mitosis or new cell appearance from the edges) or in case of a misdetection (a false negative, causing track discontinuity) or a mismatch of a neighbouring cell, which may cause no valid matches to remain for the current cell. 3.3.3 Hungarian matching We use the cost matrix discussed in section 3.3.1 as the bipartite graph’s weights in the assignment problem. As the algorithm needs square matrix, any extra rows (if frame β€˜π‘‘β€™ has more detected blobs) will have corresponding columns added with zero cost. The Hungarian algorithm consisting of four steps is then applied to find the optimal match. Step 1: Subtract row minima For each row, find the lowest element and subtract it from each element in that row. Step 2: Subtract column minima Similarly, for each column, find the lowest element and subtract it from each element in that column. Step 3: Cover all zeros with a minimum number of lines Cover all zeros in the resulting matrix using a minimum number of horizontal and vertical lines. If n lines are required, an optimal assignment exists among the zeros. The algorithm stops. If less than n lines are required, continue with Step 4. Step 4: Create additional zeros Find the smallest element (call it k) that is not covered by a line in Step 3. Subtract k from all uncovered elements, and add k to elements that are covered twice.
  • 25. 21 The match then obtained is processed in the same manner as in the Nearest Neighbour matching algorithm, to remove cases of multiple cells in β€˜π‘‘β€™ matching to the same cell in β€˜π‘‘ βˆ’ 1’ and to generate new IDs, wherever necessary. 3.3.4 Sinkhorn-Knopp matching The Sinkhorn-Knopp algorithm is an optimal transport algorithm based on a linear programming approach to maximize the sum of entropies. This algorithm was chosen because it solves the entropic regularization optimal transport problem at a speed that is several orders of magnitude faster than other transportation solvers[12, 13]. Unlike the Hungarian algorithm, a square matrix is not necessary, however in the steps shown below a square matrix of size 𝑛 Γ— 𝑛 is assumed as input, along with two probability vectors 𝑝 and π‘ž. These probability vectors correspond to the total brightness of the cell. When two cells visually merge, the Sinkhorn algorithm detects that their brightness has been added, and matches these merger cells to the merged cell. The algorithm has 8 steps shown below in pseudocode format. Algorithm 1: Sinkhorn-Knopp algorithm Inputs: Probability vectors 𝑝 and π‘ž, cost matrix 𝑐, hyperparameter πœ† For each row, find the lowest element and subtract it from each element in that row. 1: Calculate entropic regularized matrix 𝐾 where 𝐾 ← 𝑒π‘₯𝑝(βˆ’πœ†. 𝑐 ) 2: Initialize 𝑛-dimensional vector 𝑒 as [1,1, … .1] 3: While not convergence do 4: 𝑣 ← π‘ž./(𝐾 𝑒) 5: 𝑒 ← 𝑝./(𝐾𝑣) 6: end while 7: π‘ˆ ← π‘‘π‘–π‘Žπ‘”(𝑒), 𝑉 ← π‘‘π‘–π‘Žπ‘”(𝑣) 8: π‘šβˆ— ← π‘ˆπΎπ‘‰ Output: Probability matrix π‘šβˆ— , with values from 0 to 1 for each combination of a current frame cell and a previous frame cell, signifying the probability that this combination is a match. The hyperparameter πœ† is the regularizer, which allows for matching over a more convex function as it’s value increases, speeding up the convergence (Figure 3.6).
  • 26. 22 Figure 3.6: The matching occurs faster over a more convex function as the value of the hyperparameter rises from left to right. The blue and red curves represent the probability vector input [22]. The results upon application of this algorithm were different from the KNN and the Hungarian algorithm, with more continuous tracks, as described in section 4.2 where specific examples for the different methods are compared. 3.3.5 Sparse Sinkhorn-Knopp matching The algorithm in 3.3.4 is optimized by modifying the cost matrix in an advantageous way. Only the cells in the current or neighboring bins (set to 20 in total at maximum) will be considered for calculation of the regularized costs (entropies), and the rest of the combinations will have exactly zero entropy (infinite cost). This allows us to form a sparse entropic regularized cost matrix as shown below in Figure 3.6, which is much simpler for calculations than the entropic regularized original cost matrix. This sparse regularized cost matrix is then stored in a Column Sparse Row format and the same procedure as in the original Sinkhorn-Knopp algorithm is followed.
  • 27. 23 Figure 3.6: The modified entropic regularized cost matrix is extremely sparse with only three thin lines, representing the cells in the current bin and the 8 neighbouring bins taken as candidates. Columns are previous frame IDs, rows are current frame IDs. 3.4 Tracklets association For the proposed method, the global association of the tracklets formed in the previous subsections is based on ad-hoc methods. These ad-hoc methods include simple case-by- case analyses such as disallowing matching to cells beyond practically feasible distances, preventing non-maximum detections from overtaking existing cells (causing ID switches) and storing a list of merged blobs (Merge Dictionary) and dispersing the correct IDs to the blobs at the time of their demerge. The prevention of non-maxima breaking cell tracks is done by the following simple method, leading to fixing many previously broken tracks. Algorithm 2: Prevention of non-maxima overtaking of tracks 1: if new ID added π‘˜ frames ago 2: and old (correct) ID nearby is not updated from π‘˜ βˆ’ 1 or more frames ago 3: then remove new (incorrect) ID and reassign old (correct) ID
  • 28. 24 An example of a non-maxima detection (ID 736) overtaking an existing cell track (ID 539) is shown in Figure 3.7, with the resolved track on the right. Cell 539 (deep blue track) is now correctly the only cell following the specific track. Figure 3.7: An example where the blue track is overtaken by a false positive non- maximum detection, leading to a yellow track with a different ID. The Merge Dictionary is a Heuristic method for detecting (visual) merging of cells. First, a merge is detected when matching the cells in the new frame as follows: Algorithm 3: Registering merges 1: if (cell not detected this frame): 2: and if it touched another detected cell in the last frame 3: then cells merged, IDs added to dictionary (touch = distance between centres < sum of radii+1-2 pixels) The cell in front is determined by which of the multiple cells is the closest KNN match to the blob in the same position last frame. This front cell’s ID now represents a merged cell, and the other ID(s) are stored in a data structure to be distributed upon detection of a demerge as follows: Algorithm 4: Detecting demerges 1: if (new cell detected this frame): 2: and touches a Merged cell in this frame 3: and brightness/radius similar to merged cell 4: then cell separation detected, 5: IDs in Merged Cell dictionary distributed appropriately An example of the working of the merge dictionary is shown in Figure 3.8. The demerge is detected correctly, and the track for cell 442 is recovered.
  • 29. 25 Figure 3.8: An example where two cells merge to form a single blob and then demerge to continue their tracks. The track for ID 442 is recovered, using the merge dictionary.
  • 30. 26 Chapter 4 Experiments 4.1 Details on the dataset In order to scientifically understand the mechanisms of these rare cell-driven systems, simultaneous observation of huge populations of cells is essential. Hence, the dataset consists of large pixel frames, with sub-million cells. Each frame is taken at an interval of 30 seconds. The fluorescent imaging system named AMATERAS (A Multi- scale/modal Analytical Tool for Every Rare Activity in Singularity) is used, which is a novel type of imager, trans-scale-scope, which allows us to observe cell dynamics with sub-cell resolution in a field of view greater than a centimetre [15]. The provided data[15, 16] has over a 100,000 dictyostelium cells laced with two different fluorescent protiens (Flamindo and RFP), allowing for dual color imaging of the amoeba, as shown in Figure 4.1. Figure 4.1: The two proteins used in the fluorescent imaging. The method used is dual color imaging, with a mixture of both, Flamindo2 and RFP proteins. Flamindo2 shows strong reactivity to the cAMP released by the cells changing from bright green to dark as cAMP concentration increases, and RFP shows weak reactivity
  • 31. 27 to it, maintining a reddish colour. In combination, these two show a change from yellow to red based on cAMP concentration. A comparison of the number of bright spots in the Flamindo2 and RFP frames below confirms the high reactivity of Flamindo2 and the low reactivity of RFP. Figure 4.2: The difference in the reactivity of Flamindo2 and RFP to cAMP is clear from the raw monochrome data, as the former has many more bright spots. 4.2 Examples We compare the results of the Nearest Neighbour matching algorithm and the proposed Sparse Sinkhorn-Knopp matching algorithm in Figure 4.3 by looking at corresponding frames, and at a substantial number of cells (over 600), for enough visual test cases. A more detailed look at some of the examples in Figure 4.3 shows us the differences in the results for these two methods.
  • 32. 28 Figure 4.3: A direct comparison of the output frame 13 of Flamindo2 data for the NN and the sparse sinkhorn methods. The green highlights are examples of the improvement observed, while the red area shows the pitfall of the proposed method.
  • 33. 29 Looking at the example results, a comparison of the Nearest Neighbour matching and the Sparse Sinkhorn matching can be made. Below are the β€˜plus’ and β€˜minus’ points for the proposed algorithm in comparison to the ad-hoc nearest neighbour implementation. + Sinkhorn gave better accuracy especially for longer tracks (Green in Figure 4.3). + Greater track continuity. + 2-3 times the computational speed. (the Nearest Neighbour algorithm can also be implemented in 𝑂(𝑛) time and space using bin sorting of cells). Worse performance in terms of new IDs generated (Red in Figure 4.3). Fewer incorrect matches but reliance on new IDs in crowded areas. 4.3 Protocol The experiments conducted to determine the tracking accuracy and the processing time follow a simple protocol. The metrics used for tracking accuracy are the percentage of new IDs generated every frame, the IDS (ID switch) metric, the precision and the recall. IDS: Number of times that a tracked trajectory changes its matched ground-truth identity, in the sample size chosen [17]. Recall: Ratio of correctly matched detections to ground-truth detections [17]. Precision: Ratio of correctly matched detections to total result detections [17]. For calculating IDS, 50 random cells are chosen and the ground truth results are compared to the results of the tracker over the interval of 15 frames. For precision and recall, the algorithm was ran for random areas of the overall frame containing about 50 cells, and the mismatches till frame 15 were counted. The data shown in the graphs for tracking accuracy and processing time is averaged over 30 runs, and 10-40 runs (depending on the number of cells) respectively.
  • 34. 30 4.4 Tracking accuracy The overall accuracy metric used during development was the number of new IDs generated per frame, since this is an upper bound for all misdetections and mismatches. More common and useful metrics were also calculated, as shown in the second half of this subsection. Since new ID cells are those which had no decent match in the previous frame, they either arise from being genuinely new (mitosis or entry at edges), from being false positives, or from mismatches in the neighbouring area. As false positives are rare in the LoG filter peak detection method used (false negatives are more common), this percentage of new IDs is a simple and useful metric to compare the accuracy of the tracking part for different algorithms. Figure 4.4 shows the percentage of new IDs in each frame for the KNN, the Sinkhorn- Knopp and the Sparse Sinkhorn-Knopp algorithms, averaged over a 50 runs. Figure 4.4: The percentage of new IDs generated for the first 17 frames, averaged over 30 runs. This is a loose upper bound to the number of mismatches. The percentage of new IDs per frame is slightly higher with the faster method or with smaller sizes of bins, signifying a trade-off of matching accuracy and computational cost. Since cell mitosis can be assumed to be at a constant rate in terms of percentage of cells, it is clearly below 1%, as is discernible from the KNN new ID 0.00% 1.00% 2.00% 3.00% 4.00% 5.00% 6.00% 7.00% 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 % New cell IDs Frame number Sparse Sinkhorn Sinkhorn-Knopp KNN
  • 35. 31 rates at the right end of the graph. Hence, the proposed Sparse Sinkhorn introduced around as many new IDs as the regular Sinkhorn-Knopp algorithm and the simplest KNN matching algorithm. This accuracy depends on the size of each bin (bin size used: 20px*20px). The larger the size of each bin, the lower the chance of not finding a suitable match within the bound of the current and neighbouring bins. The IDS, Recall and Precision calculated were as in Table 4.1. Table 4.1: A comparison of the accuracy of the two methods tested, the proposed method, and the simpler but slower KNN. The metrics were quite similar for both. Metric (50 cells, 15 frame interval) Sparse Sinkhorn- Knopp KNN IDS 44 40 Recall 65.5% 68.1% Precision 76.3% 79.2% A clearly visible pattern in the graph of Figure 4.4 is the decreasing nature of the number of new IDs generated. This happens due to a combination of many reasons. Firstly, the features (radius, peak brightness and brightness sum) of every cell are not fully defined by the first few frames as they are stochastic, with a large variance, as shown in the example of the changing peak brightness values for two cells in Figure 4.5 below. This causes many matches to go wrong but only in the first few frames. Secondly, the various global association method discussed in 3.4 start to take effect after the first few frames as well. The combined effect of clearer data and track recovery methods leads to a drop in the number of cells without a suitable match (new ID cells). Figure 4.5: A graph of the raw peak brightness values of two random cells chosen from the dataset. To reduce this variance, these features are averaged over each frame.
  • 36. 32 4.5 Computational cost compared to pure Sinkhorn-Knopp algorithm The processing in each frame consists of the detection part, the tracking part, and the display or output part. The runtimes for each of these parts was measured separately. As expected, the processing time of the detection and output parts is the same for each matching algorithm from chapter 3, at about 10ms and 2ms respectively per cell for numbers of cells above 200. Hence, a direct comparison of the computational cost of the tracking part can be made as in the following graph. Figure 4.6 shows the average runtime for the association phase (tracklet formation and global association) per frame. Figure 4.6: The time taken to complete the association of one frame averaged over 10- 40 runs. The reduction to linear time is clearly seen. 0 20 40 60 80 100 120 Time per frame (s) Number of cells Sparse Sinkhorn Sinkhorn-Knopp
  • 37. 33 Chapter 5 Conclusions In our thesis, the goal was to replace visual cell data analysis by a fast automatic method to track pattern formation. For this purpose, the fastest approach to accurately track cells was found out via elimination of all the different candidate methods, such as the deep learning methods and the Hungarian algorithm. The proposed method was based on a particularly fast optimal transport algorithm, the Sinkhorn-Knopp algorithm, which allows for decently accurate matching of bipartite graphs in 𝑂(𝑛 ) time. This, however can be improved substantially by pruning off extremely unlikely match candidate cells, to obtain a sparse entropic regularized cost matrix, which can then speed up the convergence of the Sinkhorn-Knopp algorithm due to substantially faster matrix computations. However, as is clear in the results and the metrics that follow it, the algorithm may be fast but it is in a bare-bones state, being prone to ID switching. To solve this issue, future work such as the addition of a Kalman filter can be proposed. During the final results, it was found that the proposed matching algorithm is much faster than the detection method used, with the detection taking nearly 8 times as long for larger cell numbers, and high memory usage. A modification of the applied detection method, and further research into large-scale blob detection can also be proposed.
  • 38. 34 Acknowledgements I want to thank professor Yasushi Yagi. His insights during our meeting always push me to look further and try to be genuinely a researcher. I want to show my deep appreciation for professor Yasushi Makihara for his insightful guidance and continuous support. I also want to thank Mr. Yang Yu for helping me with many technical issues, and for suggesting new approaches to better implement the algorithms discussed.
  • 39. 35 Bibliography [1] Friedl, P. & Gilmour, D. Collective cell migration in morphogenesis, regeneration and cancer. Nat. Rev. Mol. Cell Biol. 10, 445–457 (2009). [2] RΓΈrth, P. Collective cell migration. Annu. Rev. Cell Dev. Biol. 25, 407–429 (2009). [3] R. Bise, Z. Yin and T. Kanade, "Reliable cell tracking by global data association," 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro, Chicago, IL, 2011, pp. 1004-1010, doi: 10.1109/ISBI.2011.5872571. [4] J. Hayashida, K. Nishimura and R. Bise, "MPM: Joint Representation of Motion and Position Map for Cell Tracking," 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2020, pp. 3822-3831, doi: 10.1109/CVPR42600.2020.00388. [5] Tao He, Hua Mao, Jixiang Guo, Zhang Yi, Cell tracking using deep neural networks with multi-task learning, Image and Vision Computing, Volume 60, 2017, Pages 142- 153, ISSN 0262-8856, https://doi.org/10.1016/j.imavis.2016.11.010. [6] K.Chen, N. Zhang, L.S.Powers, J.M.Roveda, Cell Nuclei Detection and Segmentation for Computational Pathology Using Deep Learning, SpringSim 2019 Modeling and Simulation in Medicine, Society for Modeling and Simuation (SCS) International. [7] Huh, S., Ker, D. F., Bise, R., Chen, M., & Kanade, T. (2011). Automated mitosis detection of stem cell populations in phase-contrast microscopy images. IEEE transactions on medical imaging, 30(3), 586–596. https://doi.org/10.1109/TMI.2010.2089384 [8] Li F, Zhou X, Ma J, Wong ST. Multiple nuclei tracking using integer programming for quantitative cancer cell cycle analysis. IEEE Trans Med Imaging. 2010 Jan;29(1):96-105. doi: 10.1109/TMI.2009.2027813. Epub 2009 Jul 28. PMID: 19643704; PMCID: PMC2846554. [9] A. Tashita et al., "Macrophage Tracking Using the Hungarian Algorithm in Time Lapse MR Images," 2015 7th International Conference on Emerging Trends in
  • 40. 36 Engineering & Technology (ICETET), Kobe, 2015, pp. 169-173, doi: 10.1109/ICETET.2015.40. [10] Xu, Y., Osep, A., Ban, Y., Horaud, R., Leal-TaixΒ΄e, L., Alameda-Pineda, X.: How to train your deep multi-object tracker. In: CVPR (2020) [11] Hwann-Tzong Chen, Horng-Horng Lin and Tyng-Luh Liu, "Multi-object tracking using dynamical graph matching," Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, Kauai, HI, USA, 2001, pp. II-II, doi: 10.1109/CVPR.2001.990962. [12] Cuturi, M. (2013). Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems (pp. 2292-2300). [13] J. Altschuler, J.Weed, P. Rigollet, (2017) Near-linear time approximation algorithms for optimal transport via Sinkhorn iteration, Advances in Neural Information Processing Systems (NIPS) 31 [14] Alaya M. Z., BΓ©rar M., Gasso G., Rakotomamonjy A. (2019). Screening Sinkhorn Algorithm for Regularized Optimal Transport, Advances in Neural Information Processing Systems 33 (NeurIPS). [15] Taro Ichimura, Taishi Kakizuka, Kazuki Horikawa, β€œTrans-scale-scope to find rare cellular activity in sub-million cells”, bioRxiv 2020.06.29.179044; doi: https://doi.org/10.1101/2020.06.29.179044 [16] Kakizuka T, Hara Y, Ohta Y, et al. Cellular logics bringing the symmetry breaking in spiral nucleation revealed by trans-scale imaging. bioRxiv; 2020. DOI: 10.1101/2020.06.29.176891. [17] Luo, W., Xing, J., Milan, A., Zhang, X., Liu, W., Zhao, X., & Kim, T. (2014). Multiple Object Tracking: A Literature Review. arXiv: Computer Vision and Pattern Recognition, p. 10 [18] Fiaz, M., Mahmood, A., & Jung, S. (2019). Deep Siamese Networks toward Robust Visual Tracking. [19] Ulman, V., MaΕ‘ka, M., Magnusson, K., Ronneberger, O., Haubold, C., Harder, N., Matula, P., Matula, P., Svoboda, D., Radojevic, M., Smal, I., Rohr, K., JaldΓ©n, J.,
  • 41. 37 Blau, H., Dzyubachyk, O., Lelieveldt, B., Xiao, P., Li, Y., Cho, S., Dufour, A.C., Olivo- Marin, J., Reyes-Aldasoro, C.C., SolΓ­s-Lemus, J.A., Bensch, R., Brox, T., Stegmaier, J., Mikut, R., Wolf, S., Hamprecht, F., Esteves, T., Quelhas, P., Demirel, Γ–., MalmstrΓΆm, L., Jug, F., TomanΓ§ak, P., Meijering, E., MuΓ±oz-Barrutia, A., Kozubek, M., & Ortiz- de-SolΓ³rzano, C. (2017). An Objective Comparison of Cell Tracking Algorithms. Nature methods, 14, 1141 - 1152. [20] Harold W. Kuhn, "The Hungarian Method for the assignment problem", Naval Research Logistics Quarterly, 2: 83–97, 1955. [21] Jonker, R.; Volgenant, A. (December 1987). "A shortest augmenting path algorithm for dense and sparse linear assignment problems". Computing. 38 (4): 325– 340. doi:10.1007/BF02278710. [22] Marino, S.D., Gerolin, A. An Optimal Transport Approach for the SchrΓΆdinger Bridge Problem and Convergence of Sinkhorn Algorithm. J Sci Comput 85, 27 (2020). https://doi.org/10.1007/s10915-020-01325-7 [23] Nasser, L., & Boudier, T. (2019). A novel generic dictionary-based denoising method for improving noisy and densely packed nuclei segmentation in 3D time-lapse fluorescence microscopy images. Scientific Reports, 9. [24] Bise, R., Maeda, Y., Kim, M.H., & Kino-oka, M. (2013). Cell tracking under high confluency conditions by candidate cell region detection-based association approach. [25] Tracking and Analysis of FUCCI-Labeled Cells Based on Particle Filters and Time-to-Event Analysis, Kenji Fujimoto, Shigeto Seno, Hironori Shigeta, Tomohiro Mashita, Masaru Ishii, Hideo Matsuda, International Journal of Bioscience, Biochemistry and Bioinformatics,10(2): 94-109, 2020.04, Papers.