1.
.
.
The Chow-Liu algorithm based on the MDL with discreete
and continuous variables
Joe Suzuki
Osaka University
AIGM 2014, Paris
Joe Suzuki (Osaka University) The Chow-Liu algorithm based on the MDL with discreete aAnIGdMcon2t0i1n4u,ouPsarvisariable1s / 26

2.
The Chow-Liu Algorithm
Chow-Liu
P1; ;N: Probability of X(1); ; X(N) N ( 1)
G = (V; E): Undirected Graph
E := fg, V := f1; ;Ng (N 1), E := ffi ; jgji̸= j ; i ; j 2 Vg
do E̸= fg
1. choose fi ; jg 2 E that maximizes I (i ; j)
2. remove fi ; jg from E
3. if no loop is generated, add fi ; jg to E
Mutual Information of X(i); X(j):
I (i ; j) :=
Σ
x(i)
Σ
x(j)
Pi ;j (x(i); x(j)) log
Pi ;j (x(i); x(j))
Pi (x(j))Pi (x(i))
.
Tree E s.t.
Σ
fi ;jg2E I (i ; j) ! max
.
.D(P1; ;NjjQ) ! min
Joe Suzuki (Osaka University) The Chow-Liu algorithm based on the MDL with discreete aAnIGdMcon2t0i1n4u,ouPsarvisariable2s / 26

3.
The Chow-Liu Algorithm
Example
Q(x(1); x(2); x(3); x(4))
=
P1;2(x(1); x(2))P1;3(x(1); x(3))P1;4(x(1); x(4))
P1(x(1))P2(x(1)) P1(x(1))P3(x(1)) P1(x(1))P4(x(4))
P1(x(1))P2(x(2))P3(x(3))P4(x(4))
= P(x(1))P(x(2)jx(1))P(x(3)jx(1))P(x(4)jx(1))
i 1 1 2 1 2 3
j 2 3 3 4 4 4
I (i ; j) 12 10 8 6 4 2
j j
1 3
j j
2 4
j j
1 3
j j
2 4
j j
1 3
j j
2 4
j j
1 3
@@
j j
2 4
Joe Suzuki (Osaka University) The Chow-Liu algorithm based on the MDL with discreete aAnIGdMcon2t0i1n4u,ouPsarvisariable3s / 26

4.
The Chow-Liu Algorithm
Dendroid Distribution
X(1); ; X(N): Discrete Random Variables
V := f1; ;Ng
E ffi ; jgji̸= j ; i ; j 2 Vg
Q(x(1); ; x(N)jE) =
Π
fi ;jg2E
Pi ;j (x(i); x(j))
Pi (x(i))Pj (x(j))
Π
i2V
Pi (x(i)) ;
fPi (x(i))gi2V , fPi ;j (x(i); x(j))gi̸=j : from P1; ;N(x(1); ; x(N))
Joe Suzuki (Osaka University) The Chow-Liu algorithm based on the MDL with discreete aAnIGdMcon2t0i1n4u,ouPsarvisariable4s / 26

5.
The Chow-Liu Algorithm
Contribution
.
Starting from Data
.
.Learning rather than Approximation
distribution P1; ;N
data xn = f(x(1)
i ; ; x(N)
i )gni
=1
.
In any database,
..
.some

6.
elds are discrete and others continuous
Joe Suzuki: A Construction of Bayesian Networks from Databases
Based on an MDL Principle, UAI 1993
David Edwords, et. al: Selecting high-dimensional mixed graphical
models using minimal AIC or BIC forests, BMC Informatics 2010
Joe Suzuki: Learning Bayesian network structures when discrete and
continous variables are present, PGM 2014
Joe Suzuki (Osaka University) The Chow-Liu algorithm based on the MDL with discreete aAnIGdMcon2t0i1n4u,ouPsarvisariable5s / 26

7.
The Chow-Liu Algorithm
Maximum Likelihood (ML)
f^P
i (x(i))gi2V , f^P
i ;j (x(i); x(j))gi̸=j are obtained from xn
ML Estimation of MI:
^I
(i ; j) :=
Σ
x(i)
Σ
x(j)
^P
i ;j (x(i); x(j)) log
^P
i ;j (x(i); x(j))
^P
i (x(j))^P
i (x(i))
Empirical Entropy given E (minus Likelihood given E):
^H
n(xnjE) := n
Σ
i2V
^H
(i )