1. 000
001
002
003
004
005
006
007
008
009
010
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
035
036
037
038
039
040
041
042
043
044
045
046
047
048
049
050
051
052
053
Internal Review Copy of May 19, 2014: Do not distribute. (ryan.turner@ngc.com)
Variational Track Management
Anonymous Author(s)
Affiliation
Address
email
Abstract
We introduce a novel probabilistic tracking algorithm that incorporates combina-
torial data association constraints and model-based track management using vari-
ational Bayes. We use a Bethe entropy approximation to incorporate data associ-
ation constraints that are often ignored in previous probabilistic data association
algorithms. A novel aspect of our methodology is the model-based mechanism
to replace heuristic logic typically used to initiate and destroy tracks. The most
noteworthy aspect of our method is that it provides an assignment posterior with
linear computation cost in window length as opposed to the exponential scaling of
previous MAP-based approaches. We demonstrate the applicability of our method
on a radar tracking example and a computer vision problem.
1 Introduction
The field of tracking is broad and possesses many applications, particularly in radar/sonar systems
and computer vision. Imagine the following problem: A radar system is tracking a flying object,
referred to as a target, by getting measurements of range, bearing, and elevation; it may also have
Doppler measurements of radial velocity. We would like to construct a track which estimates the
trajectory of the object over time. As the measurements are noisy, methods such as the Kalman
filter [11], or its nonlinear extensions, can be used to “average out” the errors in the measurements.
The key difference between tracking and filtering is the presence of clutter (noise measurements)
and missed detections of true objects. We must determine which measurement to “plug in” to the
filter before applying it; this is known as data association. Additionally complicating the situation
is that we may be in a multi-target tracking scenario in which there are multiple objects to track and
we do not know which measurement originated from which object.
There is a large body of work on tracking algorithms given its standing as a long-posed and im-
portant problem. Algorithms vary primarily on their approach to data association. The dominant
approach uses a sliding window maximum a posteriori (MAP) estimate of the measurement-to-track
assignment, in particular the multiple hypothesis tracker (MHT) [1]. In the standard MHT, at every
frame the algorithm finds the most likely matching of measurements to tracks, in the form of an
assignment matrix, under a one-to-one constraint. One track can only result in one measurement,
and vice versa, which we refer to as framing constraints. As is typical in MAP estimation, once
an assignment is determined the filters are updated and the tracker proceeds as if these assignments
were known to be correct. The one-to-one constraint makes the MAP estimation problem a complex
discrete optimization task, although algorithms exist to solve it exactly in polynomial time [10].
However, the multi-frame MHT finds the joint MAP estimate over multiple frames, in which case
the assignment problem is known to NP-hard, although good approximate solvers exist [14].
Despite the complexity of the MHT, it only finds a sliding window MAP estimate of measurement-
to-track assignments. If a clutter measurement is by chance associated with a track for the duration
of a window then the tracker will assume with certainty that the measurement originated from that
track, and never reconsider despite all future evidence to the contrary. If multiple clutter (or oth-
erwise incorrect) measurements are associated with a track, then it may veer “off into space” and
1

2. 054
055
056
057
058
059
060
061
062
063
064
065
066
067
068
069
070
071
072
073
074
075
076
077
078
079
080
081
082
083
084
085
086
087
088
089
090
091
092
093
094
095
096
097
098
099
100
101
102
103
104
105
106
107
Internal Review Copy of May 19, 2014: Do not distribute. (ryan.turner@ngc.com)
result in spurious tracks. Likewise, an endemic problem in tracking is the issue of track swaps,
where two trajectories can cross and get mixed up. Naturally, with any MAP-based algorithm re-
searchers would like to extend it by taking uncertainty into account. This has been done by the
probabilistic MHT (PMHT) [5, Ch. 4], which uses an EM-like algorithm to get posteriors on tracks.
However, this requires dropping the one-to-one constraint in data association. Likewise, the proba-
bilistic data association (PDA) algorithm can better handle framing constraints, but only allows for
a single target. This lead to the development of the joint PDA (JPDA) algorithm for multiple targets,
which utilizes heuristic calculations of the assignment weights and does not scale to multiple frame
assignment. Particle filter implementations of the JPDA have tried to alleviate these issues, but they
have not been adopted into real-time systems due to their inefficiency and lack of robustness. The
probability hypothesis density (PHD) filter [13] alleviates many of these issues, but only estimates
the intensity of objects and does not model full trajectories; this is undesirable since the identity of
an object is required for many applications.
The variational algorithm of L´azaro-Gredilla et al. [12] is perhaps closest to our method. In their
approach every trajectory follows a Gaussian process (GP); measurements are thus modeled by
a mixture of GPs. Our algorithm utilizes a state space approach (e.g. Kalman filters) to model
tracks, providing linear rather than cubic time complexity in track length. Hartikainen and S¨arkk¨a
[8] showed there is little loss of modeling flexibility by following a state space approach over GPs.
Secondly, we develop additional variational Bayes (VB) machinery to retain the framing constraints,
which are dropped in L´azaro-Gredilla et al. [12] despite being viewed as important in many systems.
Most novel tracking algorithms neglect the critical issue of track management. Many tracking algo-
rithms unrealistically assume that the number of tracks NT is known apriori and fixed. Additional
“wrapper logic” is then placed around the trackers to initiate and destroy tracks. This logic involves
many heuristics such as M-of-N logic [1, Ch. 3]. We believe the area with the largest potential for
modernization are principled and model-based methods that replace the wrapper logic.
The outline of the paper is as follows: We first review the full joint probability distribution of the
tracking problem in Section 2. This includes how to solve the track management problem by aug-
menting tracks with an active and dormant state to address the issue of an unknown number of tracks.
We then recast this joint to develop a new conjugate prior on assignment matrices in Section 3. Us-
ing this new formulation we develop a variational algorithm for estimating the measurement-to-track
assignments and track states in Section 4. In order to efficiently scale our algorithm to a large num-
ber of tracks or measurements and retain the the framing constraints we modify the variational lower
bound in Section 5 using a Bethe entropy approximation. This results in a loopy belief propagation
(BP) algorithm being used as a subroutine in our method. We show that by taking the infinite limit
of this model we can handle an unbounded and unknown number of tracks. In this setup the pro-
cess describing active tracks can be shown to be a time series extension of the Indian buffet process
(IBP) [7] known as the Markov IBP (MIBP) [20]. Finally, we show results demonstrating the ac-
curacy of our approximate track assignment posterior (Section 6.1), show the improvements our
system can make on difficult radar tracking examples (Section 6.2), and show its applicability on a
real data computer vision problem in sports (Section 6.3).
Our paper presents the following novel contributions: Firstly, we develop the first efficient deter-
ministic approximate inference algorithm for solving the full tracking problem, which includes the
framing constraints and track management. The most important observation here is that the ap-
proximate assignment posterior has an induced factorization over time with regard to assignment
matrices. Therefore, the computational cost of our variational approach is linear in window length
as opposed to the exponential cost of the MAP approach. The most astounding aspect is that by
introducing a weaker approximation (VB factorization vs MAP) we lower the computational cost
from exponential to linear; this is a truly rare and noteworthy example. Secondly, in the process
we develop new approximate inference methods on assignment matrices and a new conjugate as-
signment prior (CAP). We believe these methods have much larger applicability beyond our current
tracking algorithm. Thirdly, we develop a process to handle the track management problem in a
model-based way, including a nonparametric extension. This provides a new practical application
of the Markov IBP, in which inference is much more practical than in many other uses of the IBP.
2

3. 108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
Internal Review Copy of May 19, 2014: Do not distribute. (ryan.turner@ngc.com)
2 Model Setup for the Tracking Problem
In this section we describe the full model used in the tracking problem and develop an unambiguous
notation. At each time step k ∈ N1, known as a frame, we observe NZ(k) ∈ N0 measurements,
Zk = {zj,k}
NZ (k)
j=1 , from both real targets and clutter (spurious measurements). The number of
clutter measurements is denoted Nc(k) ∈ N0. In the motivating radar example zj,k ∈ Z is a
vector of position measurements in R3
. In data association we estimate the assignment matrices
A ∈ {0, 1}NT +1×NZ +1
, where Aij = 1 if and only if track i is associated with measurement j.
Recall that each track is associated with at most one measurement, and vice versa, implying:
NT
i=0
Aij = 1 , j ∈ 1:NZ ,
NZ
j=0
Aij = 1 , i ∈ 1:NT . (1)
The zero indices are the “dummy row” and “dummy column” to represent the assignment of a
measurement to clutter and the assignment of a track to a missed detection, respectively.
Distribution on Assignments Although rarely stated explicitly in the literature a careful exam-
ination of the cost functions used in the MAP optimization in MHT yields a particular and in-
tuitive prior on the assignment matrices. The number of tracks NT is assumed known in ad-
vance and NZ is random. The corresponding generative process on assignment matrices is as
follows: 1) Start with a one-to-one mapping from measurements to tracks: A ← INT ×NT
. 2)
Each track is observed with probability PD ∈ [0, 1]NT
. Delete a column for every track not de-
tected: A ← A(·, d), di ∼ Bernoulli(PDi). 3) Sample a Poisson number of clutter measurements
(columns): A ←[A , 0NT ×Nc
], Nc ∼ Poisson(λ). 4) Use a random permutation vector π to make
the measurement order arbitrary: A ← A(·, π). 5) Append a dummy row and dummy column on A
to satisfy the summation constraints. This process gives rise to the following prior on assignments:
P(A|PD) = λNc
exp(−λ)/NZ!
NT
i=1
PD
di
i (1 − PDi)1−di
, (2)
Note that d, NZ, and Nc are deterministic functions of A.
Track Model We utilize a state-space formulation over K time steps. The latent states x1:K ∈ XK
follow a Markov process, while the measurements z1:K ∈ ZK
are iid conditional on the track state:
p(z1:K, x1:K) = p(x1)
K
k=2
p(xk|xk−1)
K
k=1
p(zk|xk) , (3)
where we have dropped the track and measurements indices i and j. Although more general models
are possible, throughout the paper we use a linear system (i.e. Kalman filter) formulation:
p(xk|xk−1) = N(xk|Fxk−1, Q) , p(zk|xk) = N(zk|Hxk, R) . (4)
Track Meta-States We address the motivating track management problem by augmenting track
states with a two-state Markov model with an “active” and “dormant” state:
P(s1:K) = P(s1)
K
k=2
P(sk|sk−1) , sk ∈ {0, 1}NS
, (5)
where sk follows a 1-of-N encoding. This allows us to, in effect, handle an unknown number of
tracks. More complex models with a larger number of states NS are also possible. We refer to the
collection of track meta-states over all tracks at frame k as Sk := {si,k}NT
i=1.
Full Model We combine the assignment process and track model to get the full model joint:
p(Z1:K, X1:K, A1:K, S1:K) =
K
k=1
p(Zk|Xk, Ak)p(Xk|Xk−1)p(Sk|Sk−1)P(Ak|Sk) (6)
=
K
k=1
P(Ak|Sk)
NT
i=1
p(xi,k|xi,k−1)P(si,k|si,k−1)
NZ (k)
j=1
p0(zj,k)Ak
0j
NT
i=1
p(zj,k|xi,k, Ak
ij = 1)Ak
ij ,
3

4. 162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
Internal Review Copy of May 19, 2014: Do not distribute. (ryan.turner@ngc.com)
where p0 is the clutter distribution, which is often a uniform distribution. The goal in tracking is
to compute p(Xk|Z1:k), the exact computation of which is intractable due to the “combinatorial
explosion” in summing out the assignments A1:k. The MHT MAP-based approach tackles this with
P(A1:k|Z1:k) ≈ I{A1:k = ˆA1:k}. Clearly an approximation is needed, but we show how to do
much better than the (sliding window) MAP estimate of MHT. This motivates the next section
where we derive a conjugate prior on the assignments A1:k, which is useful for improving upon
MAP inference; and we cast (2) as a special case of this distribution.
3 The Conjugate Assignment Prior
Given that we aim to compute the posterior P(A|Z),1
it is natural to ask what conjugate priors on
A are possible. Deriving approximate inference procedures is often greatly simplified if the prior on
the parameters is conjugate to the complete data likelihood: p(Z, X|A) [2]. We follow the standard
procedure for deriving the conjugate prior for an exponential family (EF) complete likelihood:
p(Z, X|A) =
NZ
j=1
p0(zj)A0j
NT
i=1
p(zj|xi, Aij = 1)Aij
NT
i=1
p(xi) =
NT
i=1
p(xi) exp(1 (A L)1) ,
Lij := log p(zj|xi, Aij = 1) , Li0 := 0 , L0j := log p0(zj) , (7)
where we have introduced the matrix L ∈ RNT +1×NZ +1
to represent log likelihood contribu-
tions from various assignments. Therefore, we have the following EF quantities: base measure
h(Z, X) =
NT
i=1 p(xi), partition function g(A) = 1, natural parameters η(A) = vec A, and suffi-
cient statistics S(Z, X) = vec L. This implies the conjugate assignments prior (CAP) for P(A|χ):
CAP(A|χ) := S(χ)−1
I{A ∈ A} exp(1 (χ A)1) , S(χ) :=
A∈A
exp(1 (χ A)1) , (8)
where A is the set of all assignment matrices that obey the one-to-one constraints (1). Note that χ
is a function of the the track meta-state S. We recover the assignment prior of (2) in the form of the
CAP distribution (8) via the following parameter settings:
χij = log
PDi
(1 − PDi)λ
= σ−1
(PDi) − log λ , i ∈ 1:NT , j ∈ 1:NZ , χ0j = χi0 = 0 . (9)
Due to the symmetries in the prior of (9) we can analytically normalize (8) in this case:
S(χ)−1
= P(A1:NT ,1:NZ
= 0) = Poisson(NZ|λ)
NT
i=1
(1 − PDi) . (10)
Given that the dummy row and columns of χ are zero in (9), equation (10) is clearly the only way
to get (8) to match (2) for the 0 assignment case.
Given this conjugate prior, finding the hyper-parameters of the posterior is as simple as:
χposterior = χprior + L, which at first glance may seem trivially easy. However, difficulties remain:
We still have not shown how to compute S(χ) or E[A] when |A| is large. In Section 5 we carefully
derive alternate variational updates that do not require E[A] to be computed exactly.
One insight into the partition function S(χ) is that if we slightly change the constraints in A so that
all the rows and columns must sum to one, i.e. we do not use a dummy row or column and A becomes
the set of permutation matrices, then S(χ) is equal to the matrix permanent of exp(χ), which is
#P-complete to compute [19]. Although the matrix permanent is #P-complete many accurate and
computationally efficient approximations exist, some based on belief propagation [22; 21].
4 Variational Formulation
As explained in Section 2, exact inference on the full model in (6) is intractable, and as promised we
show how to perform better inference than the existing solution of sliding window MAP. Our vari-
ational tracker enforces the factorization constraint that the posterior factorizes across assignment
1
In this section we drop the frame index k for brevity.
4

5. 216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
Internal Review Copy of May 19, 2014: Do not distribute. (ryan.turner@ngc.com)
matrices and latent track states:
p(A1:K, X1:K, S1:K|Z1:K) ≈ q(A1:K, X1:K, S1:K) = q(A1:K)q(X1:K, S1:K) . (11)
In some sense we can think of A as the “parameters” with X and S as the “latent variables” and
use the common variational practice of factorizing these two groups of variables. This gives the
variational lower bound L(q):
L(q) = Eq[log p(Z1:K, X1:K, A1:K, S1:K)] + H[q(X1:K, S1:K)] + H[q(A1:K)] , (12)
where H[·] denotes the Shannon entropy. From inspecting the VB lower bound (12) and (6) we
arrive at the following induced factorizations without further approximation
q(A1:K) =
K
k=1
q(Ak) , q(X1:K, S1:K) =
NT
i=1
q(xi,·)q(si,·) . (13)
In other words, the approximate posterior on assignment matrices factorizes across time; and the
approximate posterior on latent states factorizes across tracks.
State Posterior Update Based on the induced factorizations in (13) we can derive the updates for
the track states xi,· and meta-states si,· separately. Additionally, we can derive the updates for each
track separately. We begin with the variational updates for q(xi) using the standard VB update rules
[3, Ch. 10] and (6):
log q(xi,·)
c
= log p(xi,·) +
K
k=1
NZ (k)
j=1
E[Ak
ij] log N(zj,k|Hxi,k, R) (14)
=⇒ q(xi,·) ∝ p(xi,·)
K
k=1
NZ (k)
j=1
N(zj,k|Hxi,k, R/E[Ak
ij]) . (15)
Using the standard product of Gaussians formula [4] this is proportional to
q(xi,·) ∝ p(xi,·)
K
k=1
N(˜zi,k|Hxi,k, R/E[dik]) , ˜zi,k :=
1
E[dik]
NZ
j=1
E[Ak
ij]zj,k , (16)
and recall that E[dik] = 1 − E[Ak
i0] =
NZ
j=1 E[Ak
ij]. The form of the posterior q(xi,·) is equiva-
lent to a linear dynamical system with pseudo-measurements ˜zi,k and non-stationary measurement
covariance R/E[dik]. Therefore, q(xi,·) is simply implemented using a Kalman smoother [16].
Meta-State Posterior Update We next consider the posterior on the track meta-states:
log q(si,·)
c
= log p(si,·) +
K
k=1
Eq(Ak)[log p(Ak|Sk)]
c
= log p(si,·) +
K
k=1
sik ik , (17)
ik(s) := E[dik] log(PD(s)) + (1 − E[dik]) log(1 − PD(s)) , s ∈ 1:NS (18)
=⇒ q(si,·) ∝ p(si,·)
K
k=1
exp(sik ik) . (19)
If p(si,·) follows a Markov chain then the form for q(si,·) is the same as a hidden Markov model
(HMM) with emission log likelihoods ik ∈ [R−
]NZ
. Therefore, the meta-state posterior q(si,·)
update is implemented using the forward-backward algorithm [15].
Assignment Matrix Update The reader can verify that the exact updates under the lower bound
L(q) (12) yields a product of CAP distributions:
q(A1:K) =
K
k=1
CAP(Ak|E[Lk] + E[χk]) . (20)
This poses a problem as the state posterior updates of (16) and (19) require Eq(Ak)[Ak]; since q(Ak)
is a CAP distribution we know from Section 3 its expectation is difficult to compute. Therefore, in
the next section we modify the variational lower bound in (12) to obtain a tractable algorithm.
5

6. 270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
Internal Review Copy of May 19, 2014: Do not distribute. (ryan.turner@ngc.com)
5 The Assignment Matrix Update Equations
In this section we modify the variational lower bound (12) to obtain a tractable algorithm. The
resulting algorithm uses loopy belief propagation to compute Eq(Ak)[Ak] for use in (16) and (19).
We first note that the CAP distribution (8) can naturally be represented as a factor graph:
CAP(A|χ) ∝
NT
i=1
fR
i (Ai,·)
NZ
j=1
fC
j (A·,j)
NT
i=0
NZ
j=0
fS
ij(Aij) , (21)
with fR
i (v) := I{
NZ
j=0 vj = 1}, fC
j (v) := I{
NT
i=0 vi = 1}, and fS
ij(v) := exp(χijv). We use
the reparametrization methods described in Eaton and Ghahramani [6] to convert (21) to a pairwise
factor graph, where it is easier to compute the Bethe free energy. The Bethe entropy is:
Hβ[q(Ak)] :=
NT
i=1
NZ
j=0
H[q(Ri, Aij)] +
NZ
j=1
NT
i=0
H[q(Cj, Aij)]
−
NT
i=1
NZH[q(Ri)] −
NZ
j=1
NT H[q(Cj)] −
NT
i=1
NZ
j=1
H[q(Aij)] (22)
=
NT
i=1
H[q(Ak
i,·)] +
NZ (k)
j=1
H[q(Ak
·,j)] −
NT
i=1
NZ (k)
j=1
H[q(Ak
i,j)] (23)
where we introduced the constrained auxiliary variables Ri = Ai,· and Cj = A·,j and used the
implied relation H[q(Ri, Aij)] = H[q(Ri)] + H[q(Aij|Ri)] = H[q(Ri)] = H[q(Ai,·)].
We define a new variational lower bound Lβ(q), which merely replaces H[q(Ak)] with Hβ[q(Ak)].
Note that Lβ(q)
c
= L(q) with respect to q(X1:K, S1:K), which implies the state posterior updates
under the old bound L(q) in (16) and (19) remain unchanged with the new bound Lβ(q). In order to
get the new update equations for q(Ak) we examine Lβ(q) in terms of q(A1:K):
Lβ(q)
c
= E[log p(Z1:K|X1:K, A1:K)] + E[log p(A1:K|S1:K)] +
K
k=1
Hβ[q(Ak)] (24)
c
=
K
k=1
Eq(Ak)[1 (Ak (E[Lk] + E[χk]))1] +
K
k=1
Hβ[q(Ak)] (25)
c
=
K
k=1
Eq(Ak)[log CAP(Ak|E[Lk] + E[χk])] + Hβ[q(Ak)] . (26)
This corresponds to the Bethe free energy of the factor graph described in (21) [23; 9]. Therefore,
we can compute E[Ak] using loopy belief propagation.
Loopy BP Derivation We define the key quantities for the belief propagation:
µR
ij := msgfR
i →Aij
, µC
ij := msgfC
j →Aij
, νR
ij := msgAij →fR
i
, νC
ij := msgAij →fC
j
,
where all messages form functions in {0, 1} → R+
. Using the standard rules of BP we derive:
νR
ij (x) = µC
ij(x)fS
ij(x) , µR
ij(1) =
k=j
νR
ik(0) , µR
ij(0) =
l=j
νR
il (1)
k=j,l
νR
ik(0) , (27)
where we have exploited that there is only one non-zero value in Ai,·. Notice that
µR
ij(1) =
NZ
k=0
νR
ik(0) νR
ij (0) =⇒ ˜µR
ij :=
µR
ij(0)
µR
ij(1)
=
NZ
l=0
νR
il (1)
νR
il (0)
−
νR
ij (1)
νR
ij (0)
∈ R+
. (28)
We can write the ratios of νR
as
˜νR
ij :=
νR
ij (1)
νR
ij (0)
=
µC
ij(1)
µC
ij(0)
exp(χij) ∈ R+
. (29)
6

7. 324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
Internal Review Copy of May 19, 2014: Do not distribute. (ryan.turner@ngc.com)
1 2 3 4 5
0
0.1
0.2
0.3
0.4
Tracks
Error(RMSE)
(a) Loopy BP Results
0 500 1000
−400
−200
0
200
400
600
Easting
Northing
(b) Radar Scenario (VB)
0 500 1000
−400
−200
0
200
400
600
Easting
Northing
(c) Radar Scenario (MHT)
Figure 1: Left: Accuracy of loopy BP against the number of tracks NT in χ. The RMSE is between E[A] when
calculated by loopy BP (red ) and exact marginalization. The error of using the prior is shown as a reference
(blue ◦). The (hardly visible) 95% confidence intervals are shown as dashed lines. Center: Illustration of radar
tracking scenario using VB tracker. We show the true trajectories (black ·) and filtered state estimates (color
coded by track ∗) when tracks have ≥ 90% probability of being active. Right: Same as center but using MHT.
We can symmetrically apply (28) and (29) to the column messages ˜µC
ij and ˜νC
ij . As is common in
binary graphs, we summarize the entire message passing update scheme in terms of message ratios:
˜µR
ij =
NZ
l=0
˜νR
il − ˜νR
ij , ˜νR
ij =
exp(χij)
˜µC
ij
, ˜µC
ij =
NT
l=0
˜νC
lj − ˜νC
ij , ˜νC
ij =
exp(χij)
˜µR
ij
. (30)
Finally we compute the marginal distributions E[Aij] by normalizing the product of the incoming
messages to each variable:
E[Aij] = P(Aij = 1) = σ(χij − log ˜µR
ij − log ˜µC
ij) . (31)
6 Results
6.1 Loopy BP Experiments
We first demonstrate that the modified lower bound Lβ(q) does not yield assignment matrix updates
that are considerably different than the original lower bound L(q). Therefore, in this experiment we
only consider the accuracy of the loopy BP results from Section 5. To do this we generate matrices
χ to use in loopy BP by sampling data from a synthetic scenario and running the VB algorithm.
We take the χ parameters from the resulting posteriors and compare them with the E[A] by exact
marginalization. Since we are interested in calculating an expectation a natural performance metric
is the RMSE. As a reference point we also show the error of using the prior as a posterior.
In Figure 1(a) we show the average RMSE (of 3000 runs) from using loopy BP against the number
of tracks NT . We only consider NT ≤ 5 as the cost of exact marginalization grows exponentially.
We see that the error does not increase with the number of tracks, which is encouraging. We also
see that the error is zero when NT = 1; this is because in that case q(A) is a categorical distribution
and the factor graph is loop free, meaning the inference is exact.
6.2 Radar Tracking Example
We next demonstrate our algorithm on a tracking scenario that illustrates the necessity of being
able to handle a long data association window. We consider the case where a radar cross section
(RCS) measurement is available in addition to position (kinematic) measurements. As different
aircraft will have different RCS this provides another feature to aid in data association [18]. We
consider a scenario where a smaller aircraft approaches and temporarily closely tracks a larger one,
as illustrated in Figure 1(b) and 1(c). The smaller aircraft is shadowed as it is within the minimum
range resolution and only appears as one measurement. The long association window allows the
VB tracker to maintain a single track over the smaller aircraft as it approaches and later “peels
away” from the larger; unlike a short window approach like MHT it does not get tricked by clutter
measurements in the interim shadowed period; the MHT shows many more spurious tracks as a
result.
7

8. 378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
Internal Review Copy of May 19, 2014: Do not distribute. (ryan.turner@ngc.com)
(a) Soccer Tracking Problem
0 20 40 60 80 100
0
50
100
150
200
250
300
350
Frame
ErrorChange(%)
(b) Soccer Tracking Metrics
Figure 2: Left: Example from soccer player tracking. We show the filtered state estimates of the MHT
(magenta ·) and VB tracker (cyan ◦) for the last 25 points as well as the true positions (black). The green
boxes show the detection of the HOG-SVM for the current frame. Right: Percentage change in error metrics
by going from MHT to a VB tracker on a 100 frame sequence. The metrics are positional accuracy (green),
velocity accuracy (red), and spurious tracks (blue).
6.3 Real Data: Video Tracking in Sports
We demonstrate the efficacy of our tracking algorithm on the VS-PETS 2003 soccer player data set.
This data set consists of three synchronized views of players moving around a soccer field, and the
variety of player interactions presents a number of challenges to multi-object tracking which make
it an appropriate choice for testing our algorithm. Histogram of oriented gradients (HOG) features
have been shown to work particularly well for pedestrian detection on the Caltech and INRIA data
sets, and were thus used for this data set as well. To demonstrate the robustness of our tracker
to correct a detector provided minimal training examples, we used multi-scale HOG features from
50 positive and 50 negative examples of soccer players to train a sliding window support vector
machine (SVM) [17]. For each frame, the center position of each bounding box is provided as input
to our multi-object tracker. We show that given even modest detection rates from HOG-SVM, our
tracking algorithm still does well to separate clutter and deal with missed detections.
In Figure 2(a) we plot filtered state estimates for the VB tracker, where we threshold tracks to have ≥
90% posterior probability of being active for plotting; this replaces the notion of a “confirmed” track
in the standard tracking literature with a model-based method. We see that the MHT is tricked by
the various false detections from HOG-SVM and has spurious tracks across the field; the VB tracker
“introspectively” knows when a track is unlikely to be real. While both the MHT and VB detect the
referee in the upper-right of the frame, the VB tracker quickly sets this track to dormant when he
leaves the frame. The MHT temporarily extrapolates the track into the field before destroying it. In
Figure 2(b) we show that the spurious tracks go to nearly zero for the VB tracker. We also see that
its position and velocity errors are typically less than a half of those from the MHT.
7 Conclusions
The model-based manner of handling the track management problem shows clear advantages and
may be the path forward for the field, which can clearly benefit from algorithms that eliminate
arbitrary tuning parameters. The somewhat surprising result of reducing the exponential cost of the
MAP approach to linear using a variational approximation means our methodology may be desirable
even in tracking scenarios under which a full posterior does not confer advantages over a point
estimate. We have also incorporated the often neglected framing constraints into our variational
algorithm, which fits nicely with loopy belief propagation methods; there are close relations between
this work and prior art on approximating the permanent. Others areas, such as more sophisticated
meta-state models, confer opportunities to extend this work into more applications of tracking and
demonstrate it as a general method and alternative to dominant approaches such as the MHT.
8

9. 432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
Internal Review Copy of May 19, 2014: Do not distribute. (ryan.turner@ngc.com)
References
[1] Bar-Shalom, Y., Willett, P., and Tian, X. (2011). Tracking and Data Fusion: A Handbook of Algorithms.
YBS Publishing.
[2] Beal, M. and Ghahramani, Z. (2003). The variational Bayesian EM algorithm for incomplete data: with
application to scoring graphical model structures. In Bayesian Statistics, volume 7, pages 453–464.
[3] Bishop, C. M. (2007). Pattern Recognition and Machine Learning. Springer.
[4] Bromiley, P. (2013). Products and convolutions of Gaussian probability density functions. Tina-Vision
Memo 2003-003, University of Manchester.
[5] Davey, S. J. (2003). Extensions to the probabilistic multi-hypothesis tracker for improved data association.
PhD thesis, The University of Adelaide.
[6] Eaton, F. and Ghahramani, Z. (2013). Model reductions for inference: Generality of pairwise, binary, and
planar factor graphs. Neural Computation, 25(5):1213–1260.
[7] Griffiths, T. L. and Ghahramani, Z. (2006). Infinite latent feature models and the Indian buffet process.
In Weiss, Y., Sch¨olkopf, B., and Platt, J., editors, Advances in Neural Information Processing Systems 18,
pages 475–482, Cambridge, MA, USA. The MIT Press.
[8] Hartikainen, J. and S¨arkk¨a, S. (2010). Kalman filtering and smoothing solutions to temporal Gaussian
process regression models. In Machine Learning for Signal Processing (MLSP), pages 379–384, Kittil¨a,
Finland. IEEE.
[9] Heskes, T. (2003). Stable fixed points of loopy belief propagation are minima of the Bethe free energy. In
Advances in Neural Information Processing Systems 15, volume 15, pages 359–366. MIT Press.
[10] Jonker, R. and Volgenant, A. (1987). A shortest augmenting path algorithm for dense and sparse linear
assignment problems. Computing, 38(4):325–340.
[11] Kalman, R. E. (1960). A new approach to linear filtering and prediction problems. Transactions of the
ASME — Journal of Basic Engineering, 82(Series D):35–45.
[12] L´azaro-Gredilla, M., Van Vaerenbergh, S., and Lawrence, N. D. (2012). Overlapping mixtures of Gaus-
sian processes for the data association problem. Pattern Recognition, 45(4):1386–1395.
[13] Mahler, R. (2003). Multi-target Bayes filtering via first-order multi-target moments. IEEE Trans. AES,
39(4):1152–1178.
[14] Poore, A. P., Rijavec, N., Barker, T. N., and Munger, M. L. (1993). Data association problems posed as
multidimensional assignment problems: algorithm development. In Optical Engineering and Photonics in
Aerospace Sensing, pages 172–182. International Society for Optics and Photonics.
[15] Rabiner, L. (1989). A tutorial on hidden Markov models and selected applications in speech recognition.
Proceedings of the IEEE, 77(2):257–286.
[16] Rauch, H. E., Tung, F., and Striebel, C. T. (1965). Maximum likelihood estimates of linear dynamical
systems. AIAA Journal, 3(8):1445–1450.
[17] Sch¨olkopf, B. and Smola, A. J. (2001). Learning with Kernels: Support Vector Machines, Regularization,
Optimization, and Beyond. The MIT Press, Cambridge, MA, USA.
[18] Turner, R. D., Bottone, S., and Stanek, C. J. (2013). Online variational approximations to non-exponential
family change point models: With application to radar tracking. In Advances in Neural Information Pro-
cessing Systems 26, pages 306–314.
[19] Valiant, L. G. (1979). The complexity of computing the permanent. Theoretical computer science,
8(2):189–201.
[20] Van Gael, J., Teh, Y. W., and Ghahramani, Z. (2008). The infinite factorial hidden Markov model. In
NIPS, pages 1697–1704.
[21] Vontobel, P. O. (2013). The Bethe permanent of a nonnegative matrix. Information Theory, IEEE Trans-
actions on, 59(3):1866–1901.
[22] Watanabe, Y. and Chertkov, M. (2010). Belief propagation and loop calculus for the permanent of a
non-negative matrix. Journal of Physics A: Mathematical and Theoretical, 43(24):242002.
[23] Yedidia, J. S., Freeman, W. T., and Weiss, Y. (2001). Bethe free energy, Kikuchi approximations, and
belief propagation algorithms. Advances in neural information processing systems, 13.
9