Lecture 5: Variational Estimation and Inference

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Lecture'5
Varia%onal)Es%ma%on)and)Inference
Dahua%Lin
The$Chinese$University$of$Hong$Kong
1

Outline
• Es$ma$on)of)Models)in)Exponen$al)Families
• Es$ma$on)with)par$al)observa$ons)9)EM
• Mean)Field)Methods
2

Factorized+Exponen0al+Family
Consider)an)exponen-al)family)of)joint)distribu-ons)
over) :
Here,% %indicates%the%subset%of%...

With%Complete%Observa2ons
Given& ,&the&op,mal&es,mates&are&given&by
• With&canonical&parameteriza1on,&this&is&convex.
• Pa...

Example:)GMM
• GMM"involves:"observed*feature" "and"component*
indicator" .
• How%to%es)mate%if%both% %and% %are%observed?...

Es#mate(GMM
For$ ,$we$maximize
Using&Lagrange&mul.pliers,&we&get
6

Es#mate(GMM((cont'd)
For$ ,$we$minimize
where% ,%thus
7

Par$ally'Observed'Models
Consider)an)exponen-al)family)involving)observed(
variables) )and)latent(variables) :
Here,% %and...

Par$ally'Observed'Models'(cont'd)
Given&an&observa,on& ,&we&have
where% %is%called%condi&onal)log+par&&on:
This%also%belon...

MLE$with$Par,al$Observa,ons
The$maximum&likelihood&es.mate$is$obtained$by$
maximizing$the$marginal&likelihood$over$observe...

Issues
• The%condi&onal)log+par&&on% %as%below%is%
o-en%very%diﬃcult%to%evaluate:
• We$usually$resort$to$Expecta(on+Maximi...

Lower&Bound&of&
Let$ .$
By$conjugate$duality:
with%
!
12

Lower&Bound&of& &(cont'd)
Hence,&we&have&
Hence,& &is&a&lower&bound&of& &for&
any& .
13

Expecta(on+Maximiza(on
The$Expecta(on+Maximiza(on0(EM)$algorithm$is$
coordinate0ascent$on$ :
• E"step:
• M"step:
14

E"step
• Each&E"step&reduces&to&maximize& ,&
the&op4mal&solu4on&is&the&expecta*on&of& :
• By$conjugate*duality,$with$ ,$we...

M"step
• Each&M"step&reduces&to&maximize& ,&
the&op4mal&solu4on&is&a7ained&when
16

It#can#be#shown#that#EM#Op&mizes# .#Why?
17

log L(✓|x)
Q(✓; µ(t+1)
)
Q(✓; µ(t)
)
✓(t 1)
✓(t)
✓(t+1)
EM#Op&mizes#
Sta$onary)point)is)a-ained)when)
)and) )are)dually&co...

Info.&Geo.&Interpreta-on
• A#parameter# #indicates#a#condi0onal#
distribu0on#over# :# .
• A#mean# #is#realized#by#another#...

Info.&Geo.&Interpreta-on
• For%any% %and% :
• E#step:)minimize) )to)close)the)gap)
between) )and) .
• M#step:)M#projec;on)...

EM#with#iid#samples
Consider)a)common)problem:) )
are)generated)from)an)exponen5al)family)distribu5on,)
and)only) )is)obse...

EM#with#iid#samples#(cont'd)
Lower&bound& &is:
It#has:
22

• E#step:)
• M#step:)
op#ma&a'ained&when
23

EM#for#GMM
• The%condi&onal)expecta&on%is%determined%by%
.
• E1step%computes:
24

EM#for#GMM#(cont'd)
Given& ,&M)step:
25

What%if%it%is%intractable%to%compute%the%expected%
suﬃcient%sta6s6cs% ?
26

Varia%onal)EM
• Basic&idea:"Use"a"distribu-on" "from"a"tractable"
family" "to"approximate" ,"and"thus"
"to"approximate" .
...

Varia%onal)EM)(cont'd)
• Varia%onal)E+step:"with"restric+on"to" ,"
compu+ng" "is"tractable:
!
• M"step:"remains"the"same
28

Varia%onal)E+step
• "is"usually"chosen"to"be"an"exponen&al)family,"
parameterized"by" ."Then"the"varia&onal)E1step"
reduce...

Key$Problem
• With& &given,& &remains&an&exponen3al&
family&distribu3on:&
&with&
&and& .
• &plays&a&key&role&in&model&es3m...

Mean%Field%Methods
• Consider*an*exponen.al*family*distribu.on* *for*
which*it*is*intractable*to*compute*the*mean*given* ....

Product(Form
• We$say$a$joint$distribu1on$over$ $
is$of$the$product(form,$if$its$density$can$be$wri8en:
• An$exponen&al)fa...

Product(Form((cont'd)
• Log%par))on+func)on:
• Expecta)on:
• If$each$factor$is$tractable,$then$the$whole$
distribu5on$is$t...

Ising&Model&(formula2on)
It#is#intractable#to#compute# #exactly.
34

Ising&Model&(factorized&model)
Consider)a)factorized)model
where% .%Then
35

Ising&Model&(approxima2on)
To#ﬁnd# #that#approximates# ,#we#perform#I"
projec)on#of# #onto#the#factorized1family# :
with% ...

Ising&Model&(approxima2on)
The$best$approxima(on$can$be$solved$itera1vely:
Whereas' 'is'in'a'product(form,'the'parameters'...

Mean%Field%Theory
Consider)an)exponen&al)family:)
and$a$tractable(family$ .$Then$for$any$ :
!
!can!generally!be!factorized...

Mean%Field%Theory%(cont'd)
The$diﬀerence$between$ $and$the$tractable(lower(
bound$is$the$KL$divergence:
with% .%The%op+ma%...

Naive&Mean&Field
The$mean%ﬁeld%methods$are$called$naive%mean%ﬁeld$
when$ $is$of$product%form.$Consider:
and
40

Hence,&the&nega+ve&entropy&of& &can&be&factorized:
The$op'ma$ $can$be$solved$by$minimizing:
where% .
41

Naive&Mean&Field&(Op/ma)
• This&problem&can&be&solved&by&coordinate*descent.&
• When&op6ma&is&a7ained:&
• Hence,'the'op,ma...

Naive&Mean&Field&(Discussion)
• In$naive$mean$ﬁeld,$while$ $is$of$a$product$form,$
the$parameters$associated$with$diﬀerent...

Varia%onal)EM)(Recap)
• E#step((for(each(sample( ):
• M#step:
44

M
N
nd
↵
✓d
zdi
wdi
k
Latent&Dirichlet&
Alloca/on
• Variables
• Parameters:. ,.
• Observed:.
• Latent:. ,.
45

Condi&onal)Distribu&on
Let$ $and$ :
Two$latent&suﬀ.stats.:$ $and$ .$
46

Varia%onal)Distribu%on
• :#Dirichlet#with#
• :#Categorical#with# .
47

Varia%onal)E+Steps
• For% :
• For% :
48

M"Step
49

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

MLPI Lecture 0: Overview by Dahua Lin 1103 views
MLPI Lecture 1: Maths for Machine L... by Dahua Lin 2064 views
MLPI Lecture 4: Graphical Model and... by Dahua Lin 1064 views
MLPI Lecture 2: Monte Carlo Methods... by Dahua Lin 2203 views
MLPI Lecture 3: Advanced Sampling T... by Dahua Lin 1329 views

View on Slideshare

1 of 49 Ad

View on Slideshare

1 of 49 Ad