MLPI Lecture 2: Monte Carlo Methods (Basics)

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Lecture'2
Monte&Carlo&Methods
Dahua%Lin
The$Chinese$University$of$Hong$Kong
1

Monte&Carlo&
Methods
Monte&Carlo&methods!are!a!large!
family!of!computa0onal!
algorithms!that!rely!on!random&
sampling.!Th...

Expecta(ons,in,Sta(s(cal,Analysis
Compu&ng)expecta'on)is)perhaps)the)most)common)
opera&on)in)sta&s&cal)analysis.
• Compu'...

Expecta(ons,in,Sta(s(cal,Analysis,
(cont'd)
• Compu'ng*marginaliza'on:
• Compu'ng*expecta'on*of*func'ons:
4

Compu&ng)Expecta&on
• Generally,*expecta'on*can*be*wri/en*as*
• For*discrete*space:*
• For*con7nuous*space:*
5

Law$of$Large$Numbers
• (Strong))Law)of)Large)Numbers)(LLN):#Let#
#be#i.i.d#random#variables#and# #be#a#
measurable#func6on...

Monte&Carlo&Integra-on
• We$can$use$sample'mean$to$approximate$
expecta.on:
• How%many%samples%are%enough?
7

Variance(of(Sample(Mean
• By$the$Central(Limit(Theorem((CLT):
Here,$ $is$the$variance$of$ .
• The$variance$of$ $is$ .$The$...

Random'Number'Genera.on
• All$sampling$methods$rely$on$a$stream$of$random'
numbers$to$construct$random$samples.
• "True"$r...

Random'Number'Genera.on'(cont'd)
• The%sequence%of%pseudorandom+numbers%is%
determined%by%a%seed.
• If%a%randomized%simula...

RNGs
• Linear(Congruen-al(Generator((LCG):(
.
• C(and(Java's(buil-n.
• Useful(for(simple(randomized(program.
• Not(good(en...

RNGs
• Mersenne'Twister'(MT):'
• Passes'Die$hard'tests,'good'enough'for'most'
Monte'Carlo'experiments
• Provided'by'C++'11...

RNGs
• Xorshi(1024:.
• Proposed.in.year.2014
• Passes.BigCrush
• Incredibly.simple.(5*+*6.lines.of.C.code)
13

Sampling)a)Discrete)Distribu2on
• Linear(search:(
• Sorted(search:(
• but(much(faster(when(prob.(mass(concentrates(
on(a(f...

Sampling)a)Discrete)Distribu2on
• Huﬀman(coding:(
• (preprocessing(+( (per(sample
• Alias(methods((by(A.#J.#Walker):(
• (p...

Transform)Sampling
• Let% %be%the%cumula&ve)distribu&on)func&on)(cdf)%of%
a%distribu0on% .%Let% ,%then%
.
• Why% ?
• How%t...

Sampling)from)Mul/variate)Normal
How$to$draw$a$sample$from$a$mul$variate+normal+
distribu$on$ $?
(Algorithm)
1. Perform)Ch...

Rejec%on(Sampling
• (Rejec&on)sampling)algorithm):"To"sample"from"a"
distribu2on" ,"which"has" "for"
some" :
1. Sample"
2....

Rejec%on(Sampling((cont'd)
• What&are&the&acceptance&rate?
• What&are&the&problems&of&this&method?
19

Importance+Sampling
• Basic&idea:"generate"samples"from"an"easier&
distribu+on" ,"which"is"o4en"referred"to"as"the"
propos...

Importance+Sampling+(cont'd)
• Let% %be%the%target&distribu,on%and% %be%the%
proposal&distribu,on%with% ,%and%let%the%
imp...

Importance+Sampling+(cont'd)
• By$the$strong'law'of'large'numbers,$we$have$
• How%to%choose%a%good%proposal%distribu4on% ?...

Variance(of(Importance(Sampling
• The%variance%of% %is
• The%2nd%term%does%not%depend%on% ,%while%the%1st%
term%has
23

Op#mal'Proposal
• The%lower%bound%is%a1ained%when:
• The%op#mal'proposal'distribu#on%is%generally%diﬃcult%
to%sample%from....

Adap%ve(Importance(Sampling
• Basic'idea'**'Learn'to'do'sampling
• choose'the'proposal'from'a'tractable'family:'
.
• Objec...

Self%Normalized.Weights
• In$many$prac+cal$cases,$ $is$
known$only$upto$a$normalizing$constant.
• For$such$case,$we$may$ap...

Self%Normalized.Weights.(cont'd)
Note:
By#strong#law#of#large#numbers,#we#have#
27

MCMC:$Mo&va&on
Simple'strategies'like'transform)sampling,'rejec1on)
sampling,'and'importance)sampling'all'rely'on'drawing'...

MCMC:$Overview
• Markov'Chain'Monte'Carlo'(MCMC)"explores"the"
sample"space"through"an"ergodic"Markov"chain,"
whose"equili...

Markov'Processes
• A#Markov'process#is#a#stochas+c#process#where#the#
future#depends#only#on#the#present#and#not#on#the#
p...

Countable*Markov*Chain
We#ﬁrst#review#the#formula2on#and#proper2es#of#
Markov'chains#under#a#simple#se6ng,#where# #is#a#
c...

Homogeneous)Markov)Chains
A"homogeneous)Markov)chain"on"a"countable"space" ,"
denoted"by" "is"characterized"by"an"
ini5al"...

Stochas(c)Matrix
The$transi+on$probability$matrix$ $is$a$stochas'c(
matrix,$namely$it$is$nonnega've$and$has:$
."
33

Evolu&on(of(State(Distribu&ons
Let$ $be$the$distribu,on$of$ ,$then:
We#can#simply#write# .
34

Mul$%step*Transi$on*Probabili$es
• Consider*two*transi.on*steps:
• More&generally,& .
• Let& &be&the&distribu6on&of& ,&the...

Classes&of&States
• A#state# #is#said#to#be#accessible#from#state# ,#or# #
leads)to# ,#denoted#by# ,#if# #for#
some# .
• S...

Irreducibility
A"Markov"chain"is"said"to"be"irreducible"if"it"forms"a"
single"communica7ng"class,"or"in"other"words,"all"
...

Exercise(1
• Is$this$Markov$chain$irreducible$?
• Please$iden5fy$the$communica-ng/classes.
38

Periods(of(Markov(Chains
• The%period%of%a%state% %is%deﬁned%as
• A#state# #is#said#to#be#aperiodic#if# .
• Period#is#a#cl...

Aperiodic)Chains
• A#Markov#chain#is#called#aperiodic,#if#all#states#are#
aperiodic.
• An#irreducible,Markov,chain#is#aper...

First&Return&Time
• Suppose(the(chain(is(ini/ally(at(state( ,(the(ﬁrst%
return%)me(to(state( (is(deﬁned(to(be
Note(that( (...

Recurrence
A"state" "is"said"to"be"recurrent"if"it"is"guaranteed"to"
have"a"ﬁnite)hi+ng)-me,"as
Otherwise,* *is*said*to*be...

Recurrence'(cont'd)
• "is"recurrent"iﬀ"it"returns"to" "inﬁnitely+o-en:
• Recurrence"is"a"class"property:"if" "and" "is"
re...

Invariant(Distribu-ons
Consider)a)Markov)chain)with)TPM) )on) :
• A#distribu+on# #over# #is#called#an#invariant'
distribu,...

Exercise(2
• Is$this$chain$irreducible?
• Is$this$chain$periodic?
• Please$compute$the$invariant/distribu1on.
45

Posi%ve(Recurrence
• The%expected'return'+me%of%a%state% %is%deﬁned%to%
be% .
• When% %is%transient,% .%If% %is%recurrent,...

Existence)of)Invariant)Distribu3ons
For$an$irreducible$Markov$chain,$if$some$state$is$
posi,ve.recurrent,$then$all$states$...

Example:)1D)Random)Walk
• Under'what'condi/on'is'this'chain'recurrent?
• When'it'is'recurrent,'is'it'posi+ve-recurrent'or'...

Ergodic(Markov(Chains
• An$irreducible,$aperiodic,$and$posi-ve/recurrent$
Markov$chain$is$called$an$ergodic/Markov/chain,$...

Convergence)to)Equilibrium
Let$ $be$the$transi,on$probability$matrix$of$an$ergodic$
Markov$chain,$then$there$exists$a$uniq...

Ergodic(Theorem
• The%ergodic(theorem%relates%,me(mean%to%space(
mean:
• Let% %be%an%ergodic(Markov(chain%over%
%with%equi...

Ergodic(Theorem((cont'd)
• More&generally,&we&have&for&any&posi4ve&integer& :
• The%ergodic(theorem%is%the%theore+cal%foun...

Total&Varia*on&Distance
• The%total%varia)on%distance%between%two%measures%
%and% %is%deﬁned%as
• The%total%varia)on%dista...

Mixing&Time
The$%me$required$by$a$Markov$chain$to$get$close$to$
the$equilibrium$distribu%on$is$measured$by$the$mixing&
'me...

Spectral)Representa-on
• "be"a"ﬁnite"stochas'c(matrix"over" "with" .
• The"spectral(radius"of" ,"namely"the"maximum"
absol...

Spectral)Representa-on)(cont'd)
All#eigenvalues#of# #are#real#values,#given#by#
.#Let# #be#the#right#
eigenvector#associat...

Spectral)Gap
Let$ .$Then$the$spectral)
gap$is$deﬁned$to$be$ $and$the$absolute)
spectral)gap$is$deﬁned$to$be$ .$Then:
Here,...

Bounds'of'Mixing'Time
The$mixing&'me$can$be$bounded$by$the$inverse$of$
absolute&spectral&gap:
Generally,)the)goal)to)desig...

Ergodic(Flow
Consider)an)ergodic)Markov)chain)on)a)ﬁnite)space) )
with)transi5on)probability)matrix) )and)equilibrium)
dis...

Conductance
The$conductance$of$a$Markov$chain$is$deﬁned$as
60

Bounds'of'Spectral'Gap
(Jerrum'and'Sinclair'(1989))!The!spectral!gap!is!
bounded!by
61

Exercise(3
Consider)an)ergodic)ﬁnite)chain) )with) .)To)
improve)the)mixing)7me,)one)can)add)a)li:le)bit)
lazyness)as) .)
...

Exercise(4
Consider)a) )stochas.c)matrix) ,)given)by)
)when) .
• Please'specify'the'condi2on'under'which' 'is'
ergodic.
• ...

General'Markov'Chains
Next,&we&extend&the&formula2on&of&Markov'chain&from&
countable'space&to&general&measurable'space.
• ...

General'Markov'Chains'(cont'd)
• Generally,*a*homogeneous)Markov)chain,*denoted*by*
,*over*a*measurable*space* *is*
charac...

Stochas(c)Kernel
The$transi'on)probability)kernel$ $is$a$stochas'c)kernel:
• Given' ,' 'is'a'probability*
measure'over' .
...

Evolu&on(of(State(Distribu&ons
Suppose'the'distribu.on'of' 'is' ,'then
Again,'we'can'simply'write'this'as' .
67

Composi'on)of)Stochas'c)Kernels
• Composi(on*of*stochas(c*kernels* *and* *remains*
a*stochas(c*kernel,*denoted*by* ,*deﬁne...

Example:)Random)Walk)in)
Here,%the%stochas,c%kernel%is%given%by% .
69

Occupa&on)Time,)Return)Time,)and)
Hi4ng)Time
Let$ $be$a$measurable$set:
• The%occupa&on(&me:% .
• The%return(&me:% .
• The...

!irreducibility
• Deﬁne&
.
• Given&a&posi/ve&measure& &over& ,&a&markov&
chain&is&called& !irreducible&if& &
whenever& &is...

!irreducibility,(cont'd)
• A#Markov#chain#over# #is# 0irreducible#if#and#only#if#
either#of#the#following#statement#holds:...

!irreducibility,(cont'd)
• Typical)spaces)usually)come)with)natural'measure) :
• The)natural'measure)for)countable)space)i...

Transience)and)Recurrence
Given&a&Markov&chain& &over& ,&and& :
• "is"called"transient"if" "for"every" .
• "is"called"unif...

Transience)and)Recurrence)(cont'd)
• Consider*an* ,irreducible*chain,*then*either:
• Every* ,posi9ve*subset*is*recurrent,*...

Invariant(Measures
• A#measure# #is#called#an#invariant'measure#w.r.t.#the#
stochas2c#kernel# #if# ,#i.e.
• A#recurrent#Ma...

Posi%ve(Chains
• A#Markov#chain#is#called#posi%ve#if#it#is#irreducible#
and#admits#an#invariant,probability,measure# .
• T...

Subsampled+Chains
Let$ $be$a$stochas+c$kernel$and$ $be$a$probability$
vector$over$ .$Then$ $deﬁned$as$below$is$also$a$
sto...

Subsampled+Chains+(cont'd)
• When& ,& .
• If& &is&invariant&w.r.t.& ,&then& &is&also&invariant&w.r.t.&
.
79

Sta$onary)Process
• A#stochas*c#process# #is#called#sta$onary#if
• A#Markov#chain#is#sta$onary#if#it#has#an#invariant#
pro...

Birkhoﬀ(Ergodic(Theorem
• (Birkhoﬀ)Ergodic)Theorem)"Every"irreducible)
sta8onary"Markov"chain" "is"ergodic,"that"is,"for"
...

Markov'Chain'Monte'Carlo
(Markov(Chain(Monte(Carlo):"To"sample"from"a"target(
distribu6on" :
• We$ﬁrst$construct$a$Markov$...

Markov'Chain'Monte'Carlo'(cont'd)
• Then&we&simulate&the&chain,&usually&in&two&stages:
• (Burning(stage)&simulate&the&chai...

Detailed(Balance
Most%Markov%chains%in%MCMC%prac0ce%falls%in%a%
special%family:%reversible(chains
• A#distribu+on# #over#a...

Reversible)Chains
An#irreducible#Markov#chain# #with#transi5on#
probability#matrix# #and#an#invariant#distribu5on# :
• Thi...

Reversible)Chains)on)General)Spaces
Over%a%general%measurable%space% ,%a%stochas4c%
kernel% %is%called%reversible%w.r.t.%a...

Detailed(Balance(on(General(Spaces
Suppose'both' 'and' 'are'absolutely'con2nuous'
w.r.t.'a'base'measure' :'
Then%the%chain...

Detailed(Balance(on(General(Spaces
More%generally,%if%
where% ,%then%the%chain%is%reversible%
iﬀ
88

Metropolis*Has-ngs:1Overview
• In$MCMC$prac+ce,$the$target&distribu,on$ $is$
usually$known$up$to$an$unnormalized&density$ ...

Metropolis*Has-ngs:1How1it1Works
1. It%is%associated%with%a%proposal'kernel% .
2. At%each%itera2on,%a%candidate%is%generat...

Metropolis*Algorithm
• The%Metropolis*algorithm%is%a%precursor%(and%a%
special%case)%of%the%M6H%algorithm,%which%requires%...

Metropolis*Algorithm*(cont'd)
• At$each$itera+on,$with$current$state$ :
1. Generate$a$candidate$ $from$
2. Accept$the$cand...

Metropolis*Has-ngs0Algorithm
• The%Metropolis*Has-ngs0algorithm%requires%a%
proposal0kernel% ,
• %is%NOT%necessarily%symme...

Metropolis*Has-ngs0Algorithm
• At$each$itera+on,$with$current$status$ :
• Generate$a$candidate$ $from$
• Accept$the$candid...

Gibbs%Sampling
• The%Gibbs%sampler%was%introduced%by%Geman%and%
Geman%(1984)%from%sampling%from%a%Markov%
random%ﬁeld%over...

Gibbs%Sampling%(cont'd)
At#each#itera*on,#following#a#permuta*on# #over#
.#For# ,#let# ,#update# #by#
re;drawing# #condi*o...

Gibbs%Sampling%(cont'd)
• At$each$itera+on,$one$can$use$either$a$random'scan$
or$a$ﬁxed'scan.
• Diﬀerent$schedules$can$be$...

Example(1:(Gaussian(
Mixture(Model
98

Example(1:(Gibbs(Sampling
Given& ,&ini(alize& &and& .
Condi&oned(on( (and( :
99

Example(1:(Gibbs(Sampling((cont'd)
Condi&oned(on( (and( :
with
100

Example(2:(Ising(
Model
The$normalizing$constant$ $is$
usually$unknown$and$intractable$
to$compute.
101

Example(2:(Gibbs(Sampling
• Let% %denote%the%en*re% %vector%except%for%one%
entry% ,
• How%can%we%schedule%the%computa2on%...

Mixture(of(MCMC(Kernels
Let$ $be$stochas'c(kernels$with$the$same$
invariant$probability$measure$ :
• Let% %be%a%probabilit...

Composi'on)of)MCMC)Kernels
• "is"also"a"stochas'c(kernel"with"
invariant"probability"measure" .
• Note:" "is"generally"not...

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

Variance reduction techniques (vrt) by De La Salle Unive... 5996 views
MLPI Lecture 1: Maths for Machine L... by Dahua Lin 2064 views
MLPI Lecture 0: Overview by Dahua Lin 1103 views
Lecture 5: Variational Estimation a... by Dahua Lin 964 views
MLPI Lecture 4: Graphical Model and... by Dahua Lin 1064 views
MLPI Lecture 3: Advanced Sampling T... by Dahua Lin 1329 views

View on Slideshare

1 of 104 Ad

View on Slideshare

1 of 104 Ad