1) The document presents a proof of an alternative expression for the log-likelihood ratio of the multivariate normal distribution. 2) It shows that the log-likelihood ratio, computed after observing data x for distributions i and j, can be expressed as x - μi + μj/2 * Σ^-1(μj - μi). 3) The proof involves expanding the definition of the joint probability density function of the multivariate normal distribution and applying properties of the covariance matrix Σ to simplify the resulting expression.

1. An Alternative Expression for the Log-Likelihood Ratio of the Multivariate Normal Distribution
Cole Arora
April 1, 2014
Theorem Suppose prior distribution P(i) = gi for all i ∈ {1, . . . , m} and multivariate normal model X|i ∼ Nk(µi
, Σ) for the k-vector of data X =
[ X1 ··· Xk ] , where the mean vector µi
= E[X|i] = [ E[X1|i] ··· E[Xk|i] ] = [ µi
1 ··· µi
k ] and the common, symmetric covariance matrix Σ is positive definite.
Then the log-likelihood ratio, computed posterior to making the observation x = [ x1 ··· xk ] , has the form
log
f(x|j)
f(x|i)
= x −
µi
+ µj
2
Σ−1
(µj
− µi
). (1)
Proof The left-hand side of Equation (1) can be expanded using the definition of the joint probability density function of the multivariate normal
distribution — which exists only because Σ is positive definite — as follows:
log
f(x|j)
f(x|i)
= log
1√
(2π)k|Σ|
exp −1
2 (x − µj
) Σ−1
(x − µj
)
1√
(2π)k|Σ|
exp −1
2 (x − µi) Σ−1
(x − µi)
= log exp
1
2
(x − µi
) Σ−1
(x − µi
) −
1
2
(x − µj
) Σ−1
(x − µj
)
=
1
2
(x − µi
) Σ−1
(x − µi
) −
1
2
(x − µj
) Σ−1
(x − µj
). (2)
For expediency, let
Σ−1
=





Σ−1
11 Σ−1
12 · · · Σ−1
1k
Σ−1
21 Σ−1
22 · · · Σ−1
2k
...
...
...
...
Σ−1
k1 Σ−1
k2 · · · Σ−1
kk





.
1

2. Substitution for x, µi
, µj
, and Σ−1
in Equation (2) gives
1
2
x1 − µi
1 · · · xk − µi
k





Σ−1
11 Σ−1
12 · · · Σ−1
1k
Σ−1
21 Σ−1
22 · · · Σ−1
2k
...
...
...
...
Σ−1
k1 Σ−1
k2 · · · Σ−1
kk








x1 − µi
1
...
xk − µi
k


 −
1
2
x1 − µj
1 · · · xk − µj
k





Σ−1
11 Σ−1
12 · · · Σ−1
1k
Σ−1
21 Σ−1
22 · · · Σ−1
2k
...
...
...
...
Σ−1
k1 Σ−1
k2 · · · Σ−1
kk








x1 − µj
1
...
xk − µj
k



=
1
2
k
a=1
(xa − µi
a)(Σ−1
a1 ) · · ·
k
a=1
(xa − µi
a)(Σ−1
ak )



x1 − µi
1
...
xk − µi
k


 −
1
2
k
a=1
(xa − µj
a)(Σ−1
a1 ) · · ·
k
a=1
(xa − µj
a)(Σ−1
ak )



x1 − µj
1
...
xk − µj
k



=
1
2
k
b=1
k
a=1
(xa − µi
a)(Σ−1
ab )(xb − µi
b) −
1
2
k
b=1
k
a=1
(xa − µj
a)(Σ−1
ab )(xb − µj
b)
=
1
2
k
b=1
k
a=1
(Σ−1
ab ) (xa − µi
a)(xb − µi
b) − (xa − µj
a)(xb − µj
b)
=
1
2
k
b=1
k
a=1
(Σ−1
ab )(xaxb − xaµi
b − xbµi
a + µi
aµi
b − xaxb + xaµj
b + xbµj
a − µj
aµj
b)
=
1
2
k
b=1
k
a=1
(Σ−1
ab )(−xaµi
b − xbµi
a + µi
aµi
b + xaµj
b + xbµj
a − µj
aµj
b)
=
1
2
k
b=1
k
a=1
(Σ−1
ab ) xa(µj
b − µi
b) + xb(µj
a − µi
a) + µi
aµi
b − µj
aµj
b . (3)
There are two key (and somewhat trivial) points that must be realized at this point about the terms under double summation; failure to do so makes
finishing the proof in this way impossible. The first point:
k
b=1
k
a=1
xa(µj
b − µi
b)(Σ−1
ab ) =
k
b=1
k
a=1
xb(µj
a − µi
a)(Σ−1
ba ) =
k
b=1
k
a=1
xb(µj
a − µi
a)(Σ−1
ab ),
where the first equality simply represents a renaming of the indices, and where the second equality holds because the matrix inverse of a symmetric matrix
2

3. is symmetric. So from this point forward, it is justified to replace xb(µj
a −µi
a) with xa(µj
b −µi
b) in the summand of Equation (3). The second point is similar:
k
b=1
k
a=1
(µi
bµj
a − µi
aµj
b)(Σ−1
ab ) =
k
b=1
k
a=1
(µi
bµj
a)(Σ−1
ab ) −
k
b=1
k
a=1
(µi
aµj
b)(Σ−1
ab )
=
k
b=1
k
a=1
(µi
bµj
a)(Σ−1
ab ) −
k
b=1
k
a=1
(µi
bµj
a)(Σ−1
ba )
=
k
b=1
k
a=1
(µi
bµj
a)(Σ−1
ab ) −
k
b=1
k
a=1
(µi
bµj
a)(Σ−1
ab )
= 0,
so the addition of the quantity (µi
bµj
a − µi
aµj
b) into the right-most term in the summand of Equation (3) would change absolutely nothing, which turns out
to be useful.
Using these two insights, Equation (3) can be rewritten as follows:
1
2
k
b=1
k
a=1
(Σ−1
ab ) 2xa(µj
b − µi
b) + µi
bµj
a − µi
aµj
b + µi
aµi
b − µj
aµj
b =
k
b=1
k
a=1
xa −
µi
a + µj
a
2
(Σ−1
ab )(µj
b − µi
b)
=
k
a=1
xa −
µi
a + µj
a
2
(Σ−1
a1 ) · · ·
k
a=1
xa −
µi
a + µj
a
2
(Σ−1
ak )



µj
1 − µi
1
...
µj
k − µi
k



= x1 −
µi
1 + µj
1
2
· · · xk −
µi
k + µj
k
2





Σ−1
11 Σ−1
12 · · · Σ−1
1k
Σ−1
21 Σ−1
22 · · · Σ−1
2k
...
...
...
...
Σ−1
k1 Σ−1
k2 · · · Σ−1
kk








µj
1 − µi
1
...
µj
k − µi
k



= x −
µi
+ µj
2
Σ−1
(µj
− µi
).
3