1. OPTICAL
FLOW
FOR
MOTION
ESTIMATION
AND
TRACKING
OF
SUBCELLULAR,
CELLULAR
AND
SUPRACELLULAR
DYNAMICS
Mauricio Cerda1, Jorge Jara1,4, Alex Córdova1, Jorge Toledo1,2, Eduardo Pulgar1,3, Carmen-Gloria Lemus1,3, Omar Ramírez1,2, Jarno Ralli5, Miguel Concha3 and Steffen Härtel1
References
1. Horn
BKP
&
Schunck
BG
(1981)
Determining
op>cal
flow.
Ar>ficial
Intelligence,
Vol.
17:
185–203.
2. Lucas
BD
(1985)
Generalized
image
matching
by
the
method
of
differences.
PhD
thesis,
Robo>cs
Ins>tute,
Carnegie
Mellon
University,
PiYsburgh,
PA,
USA.
3. Bruhn
A
&
Weickert
J
(2005)
Lucas/Kanade
Meets
Horn/Schunck:
Combining
Local
and
Global
Op>c
Flow
Methods.
Int.
J.
of
Computer
Vision
61(3):
211-­‐231.
4. Delpiano
J,
Jara
J,
Scheer
J,
Ramírez
O,
Ruiz-­‐del-­‐Solar
J
and
S
Härtel
(2012)
Performance
of
op>cal
flow
techniques
for
mo>on
analysis
of
fluorescent
point
signals
in
confocal
microscopy.
Machine
Vision
and
Applica>ons:
23(4):675-­‐689.
5. Márquez-­‐Valle
P,
Gil
D
&
Hernández-­‐Sabaté
A
(2012)
Error
analysis
for
Lucas-­‐Kanade
Based
Schemes.
LNCS
Vol.
7324:
184-­‐191.
6. Stuurman
N
(2003-­‐9)
ImageJ
MTrack2
tracking
plug-­‐in.
Ronald
D.
Vale
Lab.
at
U.
of
California,
San
Francisco.
hYp://valelab.ucsf.edu/~nico/IJplugins/MTrack2.html
7. Friedman
JR
et
al
(2010)
ER
sliding
dynamics
and
ER-­‐mitochondrial
contacts
occur
on
acetylated
microtubules.
J.
Cell
Biol.
Vol.
190
(3):
363–375.
8. Cai
D
et
al
(2009)
Single
molecule
Imaging
reveals
differences
in
microtubule
track
selec>on
between
kinesin
motors.
PLoS
Biology
7(10)
e1000216.
Figure
1.
Four
dynamical
structures
and
their
simplified
models.
a)
GABABR1
receptors
traffic
in
dendrites
(first
presented
by
Delpiano
et
al
[4].
b)
Cell
migra>on
in
the
Kupffer’s
vesicle.
c)
Microtubule
reorganiza>on
in
COS
cells.
d)
Cell
bleb
forma>on
in
the
parapineal
organ
of
zebra
fish.
IntroducXon
MS-­‐OF
improves
OF
mo>on
es>ma>on
range
by
a
factor
of
three,
and
the
performance
of
HS
and
CLG
methods
are
comparable
at
least
in
the
case
of
>p
growing.
We
quan>fy
and
bound
OF
error
for
mo>on
es>ma>on
in
model
structures.
These
results
can
be
used
directly
to
guide
biologists
in
defining
experimental
spa>o-­‐temporal
sampling
acquisi>on
rates
and
parameter
serngs
when
using
OF
for
mo>on
es>ma>on
and
segmenta>on
in
>me
series.
When
compared
with
automa>c
tracking,
OF
shows
to
be
less
sensi>ve
to
parameters,
and
its
performance
is
comparable
to
manual
segmenta>on
and
tracking
performed
by
an
expert.
Figure
2.
OF
methods
evalua>on
for
model
structures
(sample).
a) Parameter
op>miza>on
(α)
for
CLG-­‐OF
method
in
the
>p
model
(theore>cal
α
shown
in
black).
b) Maximum
detectable
speeds.
c) OF
for
the
>p
model
at
different
input
speeds,
measuring
all
of
the
described
OF
methods.
Figure
3.
Comparison
of
speed
es>ma>on
approaches
for
microtubule
>p
growing.
Reported
values
for
speed:
0.044
[µm/s],
std=0.018
[7]
and
0.08
[µm/s]
std=0.03
[8].
a)
Sample
image
of
the
microtubule
>ps
mo>on
sequence
(Fig.
1c).
b)
OF
vector
field
computed
for
the
segmented
microtubule
>ps.
c)
Es>mated
>p
speeds
with
the
tested
methods,
using
the
best
parameters
for
each
OF
approach.
d)
Frequency
histograms
of
es>mated
the
>p
mo>on
speeds.
I.
TheoreXcal
OF
parameter
opXmizaXon
Biological
systems.
COS-­‐7
cells
expressing
EB3-­‐GFP,
pineal
cells
expressing
GFP
and
dorsal
forerunner
cells
expressing
an
ac>ne
sensor
in
zebrafish
were
studied.
Live
imaging,
deconvolu>on
and
restoring
filters
were
applied.
Synthe3c
control
sequences.
Convolu>on
of
microscopic
point
spread
func>ons
with
basic
morphologic
models
of
single
molecules,
membranes
and
protrusions.
Different
MS-­‐OF
approaches
combined
with
ac>ve
contour
models
were
compared
to
evaluate
vector
fields
for
mo>on
es>ma>on
and
object
segmenta>on/tracking.
Model
Cell
migra*on
Biology
Bleb
forma*on
Microtubule
*ps
Protein
traffic
a)
b)
c)
d)
t=1
t=2
t=1
t=2
t=3
t=1
t=2
t=3
t=1
t=2
t=3
t=1
t=2
t=3
t=1
t=2
t=3
t=1
t=2
t=3
Growing
microtubule
*ps,
EB3-­‐GFP
(end
binding
protein)
Lucas
&
Kanade
(LK)
Horn
&
Schunk
(HS)
MulX-­‐scale
HS
Combined
Local
Global
(CLG)
MulX-­‐scale
CLG
1
pixel
jump
(v=1)
3
pixel
jump
(v=3)
6
pixel
jump
(v=6)
12
pixel
jump
(v=12)
20
pixel
jump
(v=20)
b)
c)
a)
a)
b)
Speed
[µm/s]
Speed
[µm/s]
We
show
that
the
parameters
of
the
OF
methods
can
be
automaXcally
tuned,
in
order
to
increase
moXon
range
and
precision.
Next,
we
compare
our
OF
results
with
standard
methods
used
by
biologists
(like
tracking)
to
compute
speed,
upon
manual
and
automaXc
segmentaXon.
t=1
t=2
2.
For
the
HS-­‐OF
itera>ve
scheme,
is
important
in
areas
where
,
and
thus
its
value
should
be
equal
to
the
gradient
in
the
areas
of
interest.
3.
For
the
CLG-­‐OF,
a
similar
argument
can
be
presented:
is
important
in
areas
where
,
thus
its
value
should
be
equal
to
the
gradient
in
the
areas
of
interest.
Note
that
from
HS-­‐OF
is
different
than
CLG-­‐OF,
being
1.
For
LK-­‐OF,
the
key
step
is
the
inversion
of
the
2x2
matrix,
A.
The
condi>on
number
quan>fies
the
upper
bound
error
[5].
where
denotes
the
eigenvalues
of
matrix
A
(func>on
of
the
edges
of
the
image).
Pixels
where
will
have
bounded
confidence.
Therefore,
selec>ng
those
pixels
in
the
area
of
interest
limits
the
measurement
error.
“Real”
speed
mean=.055
std=.022
Tracking-­‐es>mated
speed
mean=.15
std=.051
A
key
ques>on
to
quan>fy
mo>on
is
to
determine
the
movement
of
each
pixel…
If
we
assume
grey
value
constancy
between
two
successive
images
at
>me
t
and
>me
t+1,
or
the
search
for
the
“best”
movement
of
each
pixel
(u,v)
pixel,
can
be
formulated
as
a
minimiza>on
problem
for
f(u,v),
Look
for
(u,v)
vectors
which
are
similar
in
a
small
image
region
p,
giving
more
importance
to
the
center,
In
computer
vision,
methods
to
minimize
f(u,v)
have
been
proposed
assuming
different
constraints.
Solu>ons
for
(u,v)
are
required
to
be
also
smooth,
Solu>ons
for
(u,v)
are
required
to
give
more
importance
to
the
local
informa>on
and
also
to
be
smooth,
Lucas
&
Kanade
LK-­‐OF
[1]
Horn
&
Schunck
HS-­‐OF
[2]
Bruhn
et
al.
CLG-­‐OF
[3]
Time
1
Time
2
In
order
to
apply
OF
methods
in
biological
problems,
it
is
important
to
know:
(i) their
limits
(minimum/maximum
speed,
error),
in
order
to
correctly
setup
experimental
condiXons
like
sampling
rate.
(ii) how
to
esXmate
opXmal
method
parameters.
From
our
model
scenarios,
we
consider
the
case
of
>p
growing
or
filopodia
(Fig.
1c).
First,
we
use
the
error
in
the
speed
es>ma>on
to
find
an
op>mal
parameters
set
when
using
OF
methods
(Fig.
2a),
and
then
we
test
the
range
of
maximum
speeds
we
were
able
to
detect
with
OF
(Fig.
2b).
Microtubule
network
is
highly
dynamic
and
it
has
been
shown
that
it
has
a
typical
growing
speed
due
to
the
underlying
molecular
mechanism.
Our
goal
is
to
verify
that
the
>p
speed
can
be
retrieved
using
OF,
and
show
the
advantages
of
the
technique
when
compared
with
the
standard
technique
of
manually
marking
and
measuring
the
displacement
of
each
>p.
OF-­‐es>mated
speed
mean=.051
std=.029
Results
II.
Numerical
evaluaXon
of
OF
methods
and
parameter
selecXon
for
syntheXc
models
III.
ApplicaXon
to
microtubule
Xp
growing
(speed
detecXon)
Conclusion
MulX-­‐scale
CLG
(MS-­‐CLG)
yields
the
largest
range
and
lowest
error:
12
pixels/frame,
error
<
1.
pixel.
For
instance,
if
a
pixel
corresponds
to
0.04
[µm]
(63x
objecXve),
an
accurate
measurement
for
a
movement
of
0.08
[µm/s]
requires
at
least
1
image
acquired
each
6
seconds.
A
pixel’s
moXon
is
represented
with
a
vector
(u,v).
Over
an
image
this
forms
a
vector
field.
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
8.333333
33.333332
75
133.33333
208.33333
300
408.33334
533.33331
675
833.33331
1008.3333
1200
1408.3334
1633.3334
1875
2133.3333
2408.3333
2700
3008.3333
3333.3333
3675
4033.3333
4408.3335
4800
5208.3335
v=1
v=2
v=3
v=4
v=5
v=6
Mean
error
[pixels]
CLG-­‐
0
5
10
15
20
25
v=1
v=2
v=3
v=4
v=5
v=6
v=7
v=8
v=9
v=10
v=11
v=12
v=13
v=14
v=15
v=16
v=17
v=18
v=19
v=20
LK
LK-­‐MS(4)
HS
HS-­‐MS(4)
CLG
CLG-­‐MS(4)
Control
Est.
speed
[pixels/frame]
Input
speed
[pixels/frame]
Cell
1
Cell
2
Microtubule
>p
segmenta>on
approach:
• First,
automa>cally
segment
microtubule
>p
using
an
intensity
threshold
on
the
images.
• Second,
manual
refinement
leaving
only
the
>ps
that
appear
in
three
consecu>ve
frames.
We
compare
different
speed
esXmaXon
approaches
for
Xp
moXon:
• Manual
segmentaXon
+
standard
tracking
ImageJ
plug-­‐in
[6].
• AutomaXc
segmentaXon
+
OF
with
opXmum
parameters
• AutomaXc
segmentaXon
+
standard
tracking
ImageJ
plug-­‐in.
“Real”
CLG
OF-­‐esXm.
HS
OF-­‐esXm.
LK
OF-­‐esXm.
n=1475
n=38
n=14681
n=14681
n=14681
n=3467
Es*mated
speed
[µm/s]
histograms
c)
1Laboratory
for
Scien>fic
Image
Analysis
(SCIAN-­‐Lab),
2Laboratory
of
Cellular
and
Molecular
Neurobiology,
and
3Laboratory
of
Experimental
Ontogeny
(LEO)
at
Biomedical
Neuroscience
Ins>tute
(BNI)
and
Faculty
of
Medicine;
4Department
of
Computer
Sciences
(DCC)
at
Faculty
of
Physical
and
Mathema>cal
Sciences;
Universidad
de
Chile.
5Universidad
de
Granada.
Materials
&
Methods
Cell
migra>on,
forma>on
of
cellular
protrusions
(e.g.
blebs,
filopodia),
and
structural
reorganiza>on
are
important
phenomena
in
cell
biology.
Precise
quan>fica>ons
of
movement/deforma>on
are
crucial
to
understand
these
processes
at
different
levels
of
organiza>on.
We
apply
computer
vision
methods
for
combined
op>cal
flow
(OF)
and
mul>-­‐scale
(MS)
mo>on
es>ma>on
of
membrane
transla>ons,
end
growing
and
protrusion
forma>on
in
fluorescence
microscopy
images.
For
these
cases
we
bound
OF
error
and
op>mal
sampling
rate,
in
order
to
guide
biologists
on
their
experimental
condi>ons.
We
also
show
the
advantages
of
OF
methods
compared
with
manual
segmenta>on
and
tracking.
→
“Real”
speed
→
OF-­‐es3mated
speed
→
Tracking-­‐es3mated
speed
n=14681
n=14681
n=14681
n=40
Tracking-­‐
esXm.
“Real”
CLG
OF-­‐esXm.
HS
OF-­‐esXm.
LK
OF-­‐esXm.
Tracking-­‐
esXm.
Speed
[µm/s]
We
propose
how
to
address
both
ques>ons
in
five
cases
that
represent
any
cell
deforma>on,
as
shown
in
the
next
sec>on.
Funding:
ICM-­‐P09-­‐015-­‐F
(BNI);
CONICYT
scholarships
(JJ,
AC,
JT,
EP,
CGL);
FONDECYT
1120579
(SH),
3110157
(MC),
1120558
(OR)
Abstract
agreement
-­‐
disagreement