SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.
SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.
Successfully reported this slideshow.
Activate your 14 day free trial to unlock unlimited reading.
1.
.
.
The Chow-Liu algorithm based on the MDL with discreete
and continuous variables
Joe Suzuki
Osaka University
AIGM 2014, Paris
Joe Suzuki (Osaka University) The Chow-Liu algorithm based on the MDL with discreete aAnIGdMcon2t0i1n4u,ouPsarvisariable1s / 26
2.
The Chow-Liu Algorithm
Chow-Liu
P1; ;N: Probability of X(1); ; X(N) N ( 1)
G = (V; E): Undirected Graph
E := fg, V := f1; ;Ng (N 1), E := ffi ; jgji̸= j ; i ; j 2 Vg
do E̸= fg
1. choose fi ; jg 2 E that maximizes I (i ; j)
2. remove fi ; jg from E
3. if no loop is generated, add fi ; jg to E
Mutual Information of X(i); X(j):
I (i ; j) :=
Σ
x(i)
Σ
x(j)
Pi ;j (x(i); x(j)) log
Pi ;j (x(i); x(j))
Pi (x(j))Pi (x(i))
.
Tree E s.t.
Σ
fi ;jg2E I (i ; j) ! max
.
.D(P1; ;NjjQ) ! min
Joe Suzuki (Osaka University) The Chow-Liu algorithm based on the MDL with discreete aAnIGdMcon2t0i1n4u,ouPsarvisariable2s / 26
3.
The Chow-Liu Algorithm
Example
Q(x(1); x(2); x(3); x(4))
=
P1;2(x(1); x(2))P1;3(x(1); x(3))P1;4(x(1); x(4))
P1(x(1))P2(x(1)) P1(x(1))P3(x(1)) P1(x(1))P4(x(4))
P1(x(1))P2(x(2))P3(x(3))P4(x(4))
= P(x(1))P(x(2)jx(1))P(x(3)jx(1))P(x(4)jx(1))
i 1 1 2 1 2 3
j 2 3 3 4 4 4
I (i ; j) 12 10 8 6 4 2
j j
1 3
j j
2 4
j j
1 3
j j
2 4
j j
1 3
j j
2 4
j j
1 3
@@
j j
2 4
Joe Suzuki (Osaka University) The Chow-Liu algorithm based on the MDL with discreete aAnIGdMcon2t0i1n4u,ouPsarvisariable3s / 26
4.
The Chow-Liu Algorithm
Dendroid Distribution
X(1); ; X(N): Discrete Random Variables
V := f1; ;Ng
E ffi ; jgji̸= j ; i ; j 2 Vg
Q(x(1); ; x(N)jE) =
Π
fi ;jg2E
Pi ;j (x(i); x(j))
Pi (x(i))Pj (x(j))
Π
i2V
Pi (x(i)) ;
fPi (x(i))gi2V , fPi ;j (x(i); x(j))gi̸=j : from P1; ;N(x(1); ; x(N))
Joe Suzuki (Osaka University) The Chow-Liu algorithm based on the MDL with discreete aAnIGdMcon2t0i1n4u,ouPsarvisariable4s / 26
5.
The Chow-Liu Algorithm
Contribution
.
Starting from Data
.
.Learning rather than Approximation
distribution P1; ;N
data xn = f(x(1)
i ; ; x(N)
i )gni
=1
.
In any database,
..
.some
6.
elds are discrete and others continuous
Joe Suzuki: A Construction of Bayesian Networks from Databases
Based on an MDL Principle, UAI 1993
David Edwords, et. al: Selecting high-dimensional mixed graphical
models using minimal AIC or BIC forests, BMC Informatics 2010
Joe Suzuki: Learning Bayesian network structures when discrete and
continous variables are present, PGM 2014
Joe Suzuki (Osaka University) The Chow-Liu algorithm based on the MDL with discreete aAnIGdMcon2t0i1n4u,ouPsarvisariable5s / 26
7.
The Chow-Liu Algorithm
Maximum Likelihood (ML)
f^P
i (x(i))gi2V , f^P
i ;j (x(i); x(j))gi̸=j are obtained from xn
ML Estimation of MI:
^I
(i ; j) :=
Σ
x(i)
Σ
x(j)
^P
i ;j (x(i); x(j)) log
^P
i ;j (x(i); x(j))
^P
i (x(j))^P
i (x(i))
Empirical Entropy given E (minus Likelihood given E):
^H
n(xnjE) := n
Σ
i2V
^H
(i )