MLPI Lecture 4: Graphical Model and Exponential Family

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Lecture'4
Graphical)Model)and)Exponen1al)
Family
Dahua%Lin
The$Chinese$University$of$Hong$Kong
1

Outline
You$will$learn$the$basics$of$probabilis3c$modeling$in$
this$lecture:
• Graphical*Models
• Exponen3al*Families
• Co...

Graphical)Models
• The%key%idea%behind%graphical)models%is%
factoriza2on.
• A%graphical)model%generally%refers%to%a%family...

Graphical)Models
A"graphical)model"can"be"viewed"in"two"ways:
• A#data#structure#that#provides#the#skeleton#for#
represen3...

Distribu(ons+on+a+Graph
Consider)a)graph) ,)where)edges)can)be)
directed)or)undirected:
• A#ach'a'random'variable' 'to'eac...

Categories*of*Graphical*Models
• Bayesian)Networks)(Directed)Acyclic)Graphs)
• Markov)Random)Fields)(Undirected)Graphs)
• ...

Directed(Acyclic(Graphs
Consider)a)directed'graph) :
• "is"called"a"directed'acyclic'graph'(DAG)"if"it"has"no"
directed'ac...

Directed(Acyclic(Graphs((cont'd)
Consider)a)directed'acyclic'graph) :
• Given'an'edge' ,' 'is'called'a'parent'of' ,'
and' ...

Topological)Ordering
• A#topological)ordering#of#a#directed)graph#
#is#a#linear)ordering#of#ver,ces#such#that#
for#each#ed...

Bayesian(Networks
Given&a&DAG& ,&we&say&a&joint&distribu,on&
over& &factorizes&according&to& ,&if&its&density& &can&
be&ex...

Bayesian(Networks:(Example
!
11

Undirected)Graphs)and)Cliques
Consider)an)undirected)graph)
• A#clique#is#a#fully#connected#subset#of#ver4ces
• A#clique#i...

Undirected)Graphs)and)Cliques)
(cont'd)
13

Markov'Random'Fields
Consider)an)undirected)graph) ,)we)say)a)
joint)distribu2on)of) )factorizes)according)to) )if)its)
de...

Markov'Random'Fields'(cont'd)
• The%normalizing*constant% %is%usually%needed%to%
ensure%the%distribu2on%is%properly%normal...

a b
c
MRF$
Parameteriza-on
• All$MRFs$can$be$parameterized$
in$terms$of$maximal&cliques.$In$
prac9ce,$this$is$not$necessar...

The$graphical$structure$also$encodes$a$set$of$
condi3onal$independencies$among$the$variables.
17

Condi&onal)Independence
Consider)a)joint)distribu/on)over) ,) )and) )
are)called)condi&onally*independent)given) ,)denoted...

Condi&onal)Independence)(cont'd)
If#the#condi,onal#distribu,ons# #and# #have#
densi,es# #and# ,#then# ,#if#the#
following#...

I"map
• Let% %be%a%family%of%distribu2ons%(e.g.%a%graphical%
model).%We%deﬁne% %to%be%the%set%of%condi)onal,
independencie...

Condi&onal)Independencies)of)MRFs
• The%condi+onal%independencies%of%an%MRF%can%be%
characterized%in%three%ways:
• Local&i...

Local&
Independencies
• For%each% ,%the%Markov'
blanket%of% %w.r.t.% ,%denoted%
by% ,%is%the%set%of%all%
neighbors%of% .
•...

Pairwise(
Independencies
• Pairwise(independencies:#Given#
two#disjoint#sets# #
with#no#direct#edges#between#
them,# #is#i...

Global&
Independencies
• We$say$ $separates$ $and$ ,$
denoted$by$ ,$if$all$
paths$between$ $and$ $go$
through$ .
• Global+...

Rela%ons)between)Independencies
•
• Given'a'distribu/on'or'a'family'of'distribu/on' ,'
we'say' 'sa#sﬁes' 'if'it'sa/sﬁes'al...

Soundness
• Let% %be%a%distribu-on%that%factorizes%according%to%
an%undirected%graph% ,%then% ,%or%in%other%
words,% %is%a...

We#have#shown#that#if# #factorizes#according#to# ,#
then# #is#an#I5map#for# .#Is#the#converse#also#true?
27

Hammersley)Cliﬀord
• (Hammersley*Cliﬀord0Theorem)"Let" "be"a"posi5ve0
distribu5on"over" "and" "be"an"I.map"of"
,"then" "fa...

Condi&onal)Independencies)of)BN
• Condi'onal*independencies*of*a*Bayesian(network*
can*be*characterized*in*two*ways:
• loc...

b c
g
a
d
i
e f
h
Local&
Independencies
• Given' ,' 'is'
independent'of'its'non#
descendants'given'its'parents'
:
30

X
Z
Y
Y
Z
X
Z
X Y Z
X Y
indirect eﬀect
common cause common eﬀect
31

!separa'on
When%"influence"%can%flow%from% %to% %via% ,%we%say%
that%the%trail% %is%ac)ve:
• "is"ac#ve"iff" "is"not"observed....

!separa'on*(cont'd)
• A#trail# #is#called#ac#ve#when#all#
sub2trails# #are#ac#ve.
• Let# #be#three#sets#of#ver8ces#of# .# ...

b c
g
a
d
i
e f
h
Global&
Independencies
• Given' ,' 'is'
independent'of' 'given' 'if'
'and' 'is' 'separated'by' 'on'
the'...

Moralized*Graphs
• Given'a'directed'graph' ,'we'can'
construct'a'moralized'graph,'denoted'by' 'by'
adding'edges'between'ea...

Moralized*Graphs*(Illustra3on)
b c
g
a
d
i
e f
h
b c
g
a
d
i
e f
h
36

From%BN%to%MRF
If# #factorizes#according#to# #as
then% %factorizes%according%to% :
• .#Is#the#opposite#true?
37

From%BN%to%MRF%(cont'd)
• In$general,$moraliza(on$may$cause$the$loss$of$
condi(onal,independencies.
• $may$be$a$proper$sub...

Factor'Graphs
• An$MRF$does$not$always$fully$reveal$the$factorized.
structure$of$a$distribu8on.
• A$factor.graph$can$some8...

Factor'Graphs'(Illustra0on)
40

Study&of&Distribu.ons
• Graphical*models* *structure*of*(in)dependencies
• Exponen8al*families* *algebraic*characteris8cs
...

Exponen'al*Families
An#exponen&al)family# #over#a#measure#space# :
• suﬃcient)sta+s+cs:"
• canonical)parameter)func+on:"
•...

Par$$on'Func$on
• The%par$$on'func$on%is%given%by:
• The%log$par((on*func(on%given%by%
%is%o.en%used%instead%of% .
43

Parameter'Space
• An$exponen)al$family$is$essen)ally$determined$by$
the$domain$ $and$the$suﬃcient-sta.s.cs$ .
• The$set$of...

Many%important%families%of%distribu3ons%are%
exponen3al%families:
• Binomial(distribu/ons
• Poisson(distribu/on
• Normal(d...

Bernoulli)Distribu.on
Domain:(
Parameter:(
Density:)
Bernoulli)distribu.ons!describe!an!event!that!may!or!
may!not!happen....

Bernoulli)Distribu.on)(cont'd)
• suﬃcient)sta+s+cs:"
• canonical)parameter:"
• base)density:" "w.r.t."coun'ng
• par++on)fu...

Bernoulli)Distribu.on)(cont'd)
• suﬃcient)sta+s+cs:"
• canonical)parameters:"
• base)density:" "w.r.t."coun'ng
• par++on)f...

Poisson&Distribu,on
Domain:(
Parameter:(
Density:
Poisson&distribu,ons!characterize!
the!number!of!independent!
events!occ...

Poisson&Distribu,on&(cont'd)
• suﬃcient)sta+s+cs:"
• canonical)parameter:" "
• base)density:" "w.r.t."coun'ng
• par++on)fu...

Exponen'al*
Distribu'on
Domain:(
Parameter:(
Density:)
Exponen'al*distribu'ons!
characterize!the!*me!interval!
between!ind...

Exponen'al*Distribu'on*(cont'd)
• suﬃcient)sta+s+cs:" "or"
• canonical)parameter:" "or"
• base)density:" "w.r.t."Lebesgue
...

Normal'Distribu.on
Domain:(
Parameter:(
Density:
Normal'distribu.ons!are!probably!
the!most!widely!used!
distribu2ons!in!p...

Normal'Distribu.on'(cont'd)
• suﬃcient)sta+s+cs:"
• canonical)parameter:" "
• base)density:" "w.r.t."Lebesgue
• par++on)fu...

Normal'Distr.'in'Canonical'Form
The$normal$distribu1on$can$be$alterna1vely$
parameterized$in$the$canonical'form:
• poten&a...

Regular(Family
In#the#sequel,#we#focus#on#exponen2al#families#in#the#
canonical'form,#the#set#of#valid#canonical'parameter...

Iden%ﬁability
Let$ $be$a$parameterized$family:
• "is"called"iden%ﬁable"when"each"distribu1on"in" "
corresponds"to"a"unique...

Minimal'and'Overcomplete
Consider)an)exponen-al)family)with)suﬃcient)stats) ,
• If$there$exist$ $such$that$
$holds$almost$...

Minimal'and'Overcomplete'(cont'd)
• Consider*an*exponen.al*family*with*suﬃcient*stats*
*such*that* *is*constant,*then*for*...

Bernoulli)Revisited
Consider)two)representa.ons:
• [R1]" "
• [R2]"
For$each$representa-on:
• Is$it$minimal$or$overcomplete...

Mean%Parameters
• The%expecta'on%of%suﬃcient%sta0s0cs%as%below%are%
called%mean+parameters:
• Under'certain'condi+ons,'the...

Realizable(Mean(Parameters
• Given'a'suﬃcient'stats' ,'we'say'a'distribu4on' '
realizes'a'mean*parameter' 'if' .
• The'set...

Convex'Hulls
• Given'a'set' ,'the'convex'hull'of' ,'denoted'
by' ,'is'the'set'of'all'convex'combina/ons'of'
elements'in' ....

Probability*Simplex
Given&a&ﬁnite&space& ,&the&probability*simplex&over& :
When% ,% %reduces%to:%
and$ $is$an$ 'dimensiona...

Polytope(of(Mean(Parameters
When%the%sample%space% %is%ﬁnite,%given%any%
,%the%set% %is%a%convex'polytope:
Par$cularly,*ea...

Log$par((on*Func(on
The$log$par((on*func(on$given$by$
has$the$following$proper0es:
• "is"a"convex'func*on"and"thus"
"is"a"...

Log$par((on*Func(on*(cont'd)
• For%an%overcomplete*representa.on%with% ,%
%has% ,%because:%
• Conversely,,for,a,minimal&re...

Gradient)Map
The$gradient)map$ $is$a$mapping$
from$the$canonical)parameters$ $to$the$mean)
parameters$ .
• When&is& &injec...

Gradient)Map)(cont'd)
• The%gradient)map%is%injec.ve%if%and%only%if%the%
exponen2al%representa2on%is%minimal.
• An%exponen...

Gradient)Map)(cont'd)
• With&minimal&representa,on,& &is&onto& ,&the&
interior&of& .&
• Each&mean&parameter& &is&uniquely&...

Entropy
Given&an&exponen+al&family&distribu+on&
The$entropy$of$ $is$deﬁned$to$be:
71

Maximum'Entropy'(Problem)
Consider)a)ﬁnite)space) ,)we)want)to
!
What's'the'solu,on?
72

Maximum'Entropy'(Solu2on)
Using&the&method'of'Lagrange'mul0pliers,&we&get&the&
op.ma& :
The$solu)on$to$an$maximum&entropy&...

Kullback(Leibler-Divergence
The$Kullback(Leibler-divergence$(or$KL-divergence)$
between$two$probability$densi4es$ $and$ $(...

KL#Divergence#(cont'd)
• (Gibbs&inequality)" ,"where"the"equality"
holds"if"and"only"if" "almost"everywhere"
(w.r.t."the"b...

Projec'ons*of*Distribu'ons
Let$ $be$an$exponen&al)family$and$ $be$a$distribu-on,$
both$over$the$same$space$ :
• The%I"proj...

Maximum'Likelihood'Es1ma1on
Given&a&parameterized&family& &over& ,&
and& ,&the&log$likelihood&of& :
!is!called!an!maximum&...

MLE$(cont'd)
• Given' ,' 'is'called'an'
empirical)probability)measure,'which'has
• The%log$likelihood%of% %given% %can%be%...

MLE$(cont'd)
• We$have$
.
• Maximizing$ $is$equivalent$to$
minimizing$ .
• Maximum'likelihood'es/ma/on$is$equivalent$to$M1...

M"projec)ons
Given&an&exponen+al&family& &and&an&arbitrary&
distribu+on& &over& ,&then
Thus,&the&op#ma& &is
80

M"projec)ons,(cont'd)
• This&is&a&convex'problem.&The& &is&op-mal&iﬀ:
• M0projec-on&to&an&exponen1al&family& &is&to&ﬁnd&a&...

What%about%I"projec)ons%?
• We$will$see$their$u-lity$when$we$talk$about$mean%
ﬁeld%methods$and$varia0onal%inference.
82

Convex'Conjugate
Let$ $be$a$real)valued$func0on$ :
• The%convex'conjugate%of% %is%deﬁned%to%be
• "is"always"convex"no"ma,e...

Convex'Conjugate
• (Fenchel's*inequality)"
• (Fenchel2Moreau*theorem)" "iﬀ" "is"convex"and"
lower*semi2con:nuous:
84

Conjugate*Duality
• "is"called"dually&coupled"if"
.
• The%convex'conjugate%to%a%log+par..on%func.on% :
• Supreme(a*ained(a...

Conjugate*Duality*(cont'd)
• "has:
• "on" "determined"via"Cauchy"sequences.
86

Conjugate*Duality*(cont'd)
• With& ,&the&log$par((on*func(on& &has:
• Supreme(a*ained(at( (iﬀ( (is(dually&coupled,(
which(...

Prior%and%Posterior
• In$Bayesian(analysis,$we$usually$place$a$prior$with$
density$ $over$the$parameter$space$ .
• A$param...

Prior%and%Posterior%(cont'd)
• Compu'ng*the*posterior(distribu,on*is*in*generally*
very*diﬃcult.
• However,*under*certain(...

Conjugate*Prior
• A#prior#with#density# #is#called#a#conjugate*
prior#to#the#likelihood#model# ,#if#the#posterior#
distrib...

CP#for#Exponen,al#Families
Generally,)conjugate*pairs)in)exponen0al*families)are)in)
the)following)form:
• Prior:'
• Likel...

CP#for#Exponen,al#Families#(cont'd)
Hence,&the&posterior(update:
• with&a&single&observa1on:&
• with&mul*ple&observa*ons:&...

CP#for#Exponen,al#Families#(cont'd)
• The%family%of%conjugate*priors%is%largely%determined%
by%the%likelihood*model,%par6c...

Example:)Beta,Bernoulli
• Prior:'Beta'distribu0on
• Likelihood:+Bernoulli+distribu3on
• Posterior:*remains*a*Beta*distribu...

Example:)Normal-Normal
• Prior:'Normal'distribu1on
• Likelihood:+Normal+distribu4on+(ﬁxed+variance)
• Posterior:*remains*a...

Dirichlet)
Distribu-on
• Dirichlet)distribu.on"is"a"
distribu+on"over" .
• It"is"o2en"used"as"a"conjugate)
prior"to"the"Ca...

Dirichlet)Distribu-on)(cont'd)
• Mean:' 'with' .
• Covariance:'
• Mode:'
• Marginal:'
97

Dirichlet)Categorical
• Prior:'
• Likelihood:'
• Posterior:'remains'a'Dirichlet'distribu7on:'
• When' ,' 'reduces'to'a'uni...

Dirichlet)Distribu-on)(cont'd)
• Dirichlet*distribu/ons*are*an*exponen&al)family:
• Canonical*parameter:*
• Suﬃcient*stats...

Predic've)Distribu'on
Given& ,&the&distribu/on&of&a&new&sample& ?
With%exponen&al)family%and%conjugacy,%we%have
100

Important)Conjugate)Pairs
• Beta":"the"probability,parameter"of"Bernoulli,"
Binomial,"Geometric,"or"Nega4ve,Binomial
• Nor...

Important)Conjugate)Pairs)(cont'd)
• Dirichlet:#the#probability.vector#of#Categorical#or#
Mul4nomial
• Mul4variate.Normal:...

Examples)of)Graphical)Models
103

N
M
µk
2
zi
xi
⇡
GMM
A"Gaussian'Mixture'Model'(GMM)"
with"ﬁxed"variance:
• This&model&is&not$complete
• How&are& &and& &
g...

N
M
µk
2
zi
xi
⇡
µ0
2
0
↵
GMM#(with#Prior)
With%priors%placed%over%model*
parameters,%we%get%a%Hierarchical*
Bayesian*Mode...

N
M
µk
2
zi
xi
⇡
µ0
2
0
↵
GMM#(Joint#Model)
This%is%an%exponen&al)family.
106

Topic&Models
107

M
N
nd
✓d
zdi
wdi
k
PLSI
Probabilis)c+Latent+Seman)c+
Indexing+(PLSI):
• Each&topic&is&associated&with&
,&a&distribu2on&ov...

M
N
nd
✓d
zdi
wdi
k
↵
LDA
Latent&Dirichlet&Alloca/on&(LDA)!
completes!PLSI!by!placing!
Dirichlet&priors!over!latent!
varia...

M
N
nd
✓d
zdi
wdik
↵
LDA$(Joint$Model)
Again,'an'exponen&al)family.
110

Summary
• The%Basics%of%Graphical%Models
• Bayesian%Networks%and%Markov%Random%Fields.%
• How%the%joint%distribuBon%factor...

Summary'(cont'd)
• The%Basics%of%Exponen1al%Families
• The%form%of%exponen1al%families
• Minimal%and%overcomplete%represen...

Summary'(cont'd)
• Conjugate+Prior
• Posterior+distribu2ons+in+Bayesian+analysis
• Conjugate+prior,+especially+of+exponen2...

Summary'(cont'd)
• Prac&ce
• How+to+formulate+a+graphical+model+based+on+
intui&on
• Graphical+representa&on+of+a+model,+f...

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

MLPI Lecture 1: Maths for Machine L... by Dahua Lin 2064 views
MLPI Lecture 0: Overview by Dahua Lin 1103 views
Lecture 5: Variational Estimation a... by Dahua Lin 964 views
MLPI Lecture 2: Monte Carlo Methods... by Dahua Lin 2203 views
MLPI Lecture 3: Advanced Sampling T... by Dahua Lin 1329 views
Factor Graph & Sum-Product Algorithm by Xixiu Ouyang 1655 views

View on Slideshare

1 of 114 Ad

View on Slideshare

1 of 114 Ad