A Unifying Review of Gaussian Linear Models (Roweis 1999)

A Unifying Review of Linear Gaussian Models1
Sam Roweis, Zoubin Ghahramani
Feynman Liang
Application #: 10342444
November 11, 2014
1Roweis, Sam, and Zoubin Ghahramani. A Unifying Review of Linear Gaussian
Models." Neural Computation 11.2 (1999): 305{45. Print.
F. Liang Linear Gaussian Models Nov 2014 1 / 18

cially disparate models: : :
(a) Factor Analysis (b) PCA
(c) Mixture of Gaussians (d) Hidden Markov Models

Outline
Basic model
Inference and learning
problems
EM algorithm
Various specializations of
the basic model
Factor Analysis
R = lim!0 I
SPCA
PCA
Kalman Filter
Gaussian Mixture Model
1-NN
HMM
cts state A = 0 R diag
R = I
A6= 0
discrete state
A = 0 R = lim!0 R0
A6= 0

The Basic (Generative) Model
Goal: Model P(fxtgt
=1; fytgt
=1)
Assumptions:
Linear dynamics, additive Gaussian
noise
xt+1 = Axt + w; w N(0;Q)
yt = Cxt + v; v N(0; R)
wlog E[w] = E[v] = 0
Markov property
Time homogeneity
w
+
xt xt+1
yt
v
A
C
+
t
Figure: The Basic Model as a DBN
P(fxtgt
=1; fytgt
=1) = P(x1)
Y1
t=1
P(xt+1jxt )
Y
t=1
P(yt jxt )

Why Gaussians?
Gaussian family closed under ane transforms
x N(x ;x ); y N(y ;y ); a; b; c 2 R
=) ax + by + c N(ax + by + c; a2x + b2y )
Gaussian is conjugate prior for Gaussian likelihood
P(x) Normal; P(yjx) Normal =) P(xjy) Normal

The Inference Problem
Given the system model and initial distribution (fA; C;Q; R; 1;Q1g):
Filtering: P(xt jfyigti
=1)
Smoothing: P(xt jfyigi
=1) where t
If we had the partition function:
P(fyigi=1) =
Z
8fxi gi
=1
P(fxig; fyig)dfxig
Then
P(xt jfyigi
=1) =
P(fxig; fyig)
P(fyig)

The Learning Problem
Let = fA; C;Q; R; 1;Q1g, X = fxigi
=1, Y = fyigi
=1.
Given (several) observable sequences Y :
arg max L() = arg max log P(Y j)
Solved by expectation maximization.

Expectation Maximixation
For any distribution Q on Sx :
L() F(Q; ) =
Z
X
Q(X) log P(X; Y j)
Z
X
Q(X) logQ(X)dX
= L() + H(Q; P(jY ; )) H(Q)
= L() DKL(QjjP(jY ; ))
Monotonically increasing coordinate ascent on F(Q; ):
E step: Qk+1 arg maxQ F(Q; k ) = P(XjY ; k )
M step: k+1 arg max F(Qk+1; )

Continuous-State Static Modeling
Assumptions:
x is continuously supported
A = 0
x = w N(0;Q) =) y = Cx + v N(0;CQCT + R)
wlog Q = I
Ecient Inference Using Sucient Statistics: Gaussian is conjugate
prior for Gaussian likelihood, so
P(xjy) = N(

= CT (CCT + R)1
Learning: R must be constrained to avoid degenerate solution. . .

Continuous-State Static Modeling: Factor Analysis
y = Cx + v N(0; CCT + R)
Additional Assumption:
R diagonal =) observation noise v independent along basis for y
Interpretation:
R : variance along basis
C : correlation structure of latent factors
Properties:
Scale invariant
Not rotation invariant

Continuous-State Static Modeling: SPCA and PCA
y = Cx + v N(0; CCT + R)
Additional Assumptions:
R = I ; 2 R
For PCA: R = lim!0 I
Interpretation:
: global noise level
Columns of C : principal components
(optimizes three equivalent objectives)
Properties
Rotation invariant
Not scale invariant

Continuous-State Dynamic Modeling: Kalman Filters
Relax A = 0 assumptio.
Optimal Bayes

lter assuming linearity and normality (conjugate prior)

Discrete-State Modeling: Winner-Takes-All (WTA)
Non-linearity
Assume: x discretely supported,
R
7!
P
Winner-Takes-All Non-Linearity: WTA[x] = ei where i = arg maxj xj
xt+1 = WTA[Axt + w] w N(;Q)
yt = Cxt + v v N(0; R)
x WTA[N(; )] de

A Unifying Review of Gaussian Linear Models (Roweis 1999)

A Unifying Review of Gaussian Linear Models (Roweis 1999)

More Related Content

What's hot

Viewers also liked

Similar to A Unifying Review of Gaussian Linear Models (Roweis 1999)

More from Feynman Liang

Recently uploaded

A Unifying Review of Gaussian Linear Models (Roweis 1999)