Upcoming SlideShare
Loading in...5
×

# A Quick Introduction to the Chow Liu Algorithm

857
-1

Published on

A very simple and quick introduction to the Chow-Liu algorithm.

Published in: Education, Technology, Business
0 Comments
5 Likes
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
Your message goes here
• Be the first to comment

No Downloads
Views
Total Views
857
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
16
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

### A Quick Introduction to the Chow Liu Algorithm

1. 1. A Quick Introduction to the Chow-Liu Algorithm Jee Vang, Ph.D. [email_address]
2. 2. What is the Chow-Liu Algorithm? <ul><li>Reported by Chow and Liu (1968). </li></ul><ul><ul><li>Also known as maximum weight spanning tree (MWST) algorithm or Kruskal’s algorithm. </li></ul></ul><ul><ul><li>Running time complexity is O(n 2 log(n)), where n is number of variables. </li></ul></ul><ul><li>The probability distribution associated to the tree constructed by the Chow-Liu algorithm is the one that is closest to the probability distribution associated to the data as measured by the Kullback-Leibler divergence (Pearl and Dechter 1989). </li></ul><ul><li>One of many uses include constructing graphical relationships between variables. </li></ul><ul><li>Assumptions </li></ul><ul><ul><li>No missing data </li></ul></ul><ul><ul><li>Variables are discrete </li></ul></ul><ul><ul><li>Data is independently and identically distributed </li></ul></ul>Copyright 2009 by Jee Vang
3. 3. Pseudo-code of Chow-Liu Algorithm <ul><li>Denote the following </li></ul><ul><ul><li>A(X i ,X j ) : a measure of association between two variables, X i and X j </li></ul></ul><ul><ul><li>U = {X 1 , X 2 , …, X n } : a set of n variables </li></ul></ul><ul><ul><li>D : a database of cases of the variables in U </li></ul></ul><ul><ul><li>T : a tree whose nodes have a 1-to-1 correspondence to the variables in U </li></ul></ul><ul><li>Inputs to Chow-Liu algorithm </li></ul><ul><ul><li>A, U, D </li></ul></ul><ul><li>Output of Chow-Liu algorithm </li></ul><ul><ul><li>T </li></ul></ul>Copyright 2009 by Jee Vang
4. 4. Pseudo-code of Chow-Liu Algorithm, Procedure <ul><li>Begin </li></ul><ul><ul><li>i <- 1 //a counter, set to one </li></ul></ul><ul><ul><li>L = {L 1 , L 2 , …, L ((n+1)/2)-n } <- all pairwise associations //L is a list of all pairwise associations in descending order; no self-pairing </li></ul></ul><ul><ul><li>P <- L i //P is a pointer to the first pair of variables in L </li></ul></ul><ul><ul><li>construct T //create a disconnected tree with nodes corresponding to variables in U </li></ul></ul><ul><ul><li>Do until n-1 edges have been added </li></ul></ul><ul><ul><ul><li>If a path does not exists between the pair of variables in P, add an undirected arc between them </li></ul></ul></ul><ul><ul><ul><li>P <- L (i+1) //set P to the next pair of variables in L </li></ul></ul></ul><ul><li>End </li></ul>Copyright 2009 by Jee Vang
5. 5. Measure of Association <ul><li>Mutual information was originally used as the measure of association (Chow and Liu 1968) </li></ul><ul><li>However, any measure of association satisfying the following property may be used </li></ul><ul><ul><li>Give three variables, X i , X j , and X k , such that X i and X k are conditionally independent given X j , I(X i ,X j ,X k ), any measure of association, A(X i ,X j ), such that, min(A(X i ,X j ), A(X j ,X k )) > A(X i ,X k ), can be used for the Chow-Liu algorithm (Acid and de Campos 1994). </li></ul></ul>Copyright 2009 by Jee Vang
6. 6. Example—Inputs <ul><li>A(X i ,X j ) : mutual information between two variables, X i and X j </li></ul><ul><li>U = {X 1 , X 2 , X 3 , X 4 , X 5 } : 5 variables </li></ul><ul><ul><ul><li>Domains of X 1 through X 5 are {true, false} </li></ul></ul></ul><ul><li>D : database of cases of X 1 through X 5 </li></ul>Copyright 2009 by Jee Vang X 1 X 2 X 3 X 4 X 5 true false false false false false true true false true false true false false true … … … … …
7. 7. Example—Procedure, Pairwise Associations <ul><li>Compute pairwise associations </li></ul><ul><ul><li>L 1,2 = 0.55 </li></ul></ul><ul><ul><li>L 1,3 = 0.34 </li></ul></ul><ul><ul><li>L 1,4 = 0.11 </li></ul></ul><ul><ul><li>L 1,5 = 0.59 </li></ul></ul><ul><ul><li>L 2,3 = 0.68 </li></ul></ul><ul><ul><li>L 2,4 = 0.03 </li></ul></ul><ul><ul><li>L 2,5 = 0.25 </li></ul></ul><ul><ul><li>L 3,4 = 0.01 </li></ul></ul><ul><ul><li>L 3,5 = 0.22 </li></ul></ul><ul><ul><li>L 4,5 = 0.10 </li></ul></ul><ul><li>Sort pairwise associations descendingly into list L </li></ul><ul><ul><li>L = {L 2,3 , L 1,5 , L 1,2 , L 1,3 , L 2,5 , L 3,5 ,L 1,4 ,L 4,5 , L 2,4 , L 3,4 } </li></ul></ul>Copyright 2009 by Jee Vang
8. 8. Example—Procedure, Tree Construction, Empty Tree <ul><li>L = {L 2,3 , L 1,5 , L 1,2 , L 1,3 , L 2,5 , L 3,5 ,L 1,4 ,L 4,5 , L 2,4 , L 3,4 } </li></ul><ul><li>Construct disconnected tree, T, with 5 nodes corresponding to variables in U </li></ul>X 2 X 3 X 5 X 4 X 1 Copyright 2009 by Jee Vang
9. 9. Example—Procedure, Tree Construction (cont…) <ul><li>L = { L 2,3 , L 1,5 , L 1,2 , L 1,3 , L 2,5 , L 3,5 ,L 1,4 ,L 4,5 , L 2,4 , L 3,4 } </li></ul><ul><li>Add arc between X 2 and X 3 in T, X 2 —X 3 </li></ul>X 3 X 2 X 5 X 4 X 1 Copyright 2009 by Jee Vang
10. 10. Example—Procedure, Tree Construction (cont…) <ul><li>L = {L 2,3 , L 1,5 , L 1,2 , L 1,3 , L 2,5 , L 3,5 ,L 1,4 ,L 4,5 , L 2,4 , L 3,4 } </li></ul><ul><li>Add arc between X 1 and X 5 in T, X 1 —X 5 </li></ul>X 3 X 2 X 5 X 4 X 1 Copyright 2009 by Jee Vang
11. 11. Example—Procedure, Tree Construction (cont…) <ul><li>L = {L 2,3 , L 1,5 , L 1,2 , L 1,3 , L 2,5 , L 3,5 ,L 1,4 ,L 4,5 , L 2,4 , L 3,4 } </li></ul><ul><li>Add arc between X 1 and X 2 in T, X 1 —X 2 </li></ul>X 3 X 2 X 5 X 4 X 1 Copyright 2009 by Jee Vang
12. 12. Example—Procedure, Tree Construction (cont…) <ul><li>L = {L 2,3 , L 1,5 , L 1,2 , L 1,3 , L 2,5 , L 3,5 ,L 1,4 ,L 4,5 , L 2,4 , L 3,4 } </li></ul><ul><li>Skip adding arc between X 1 and X 3 //path exists </li></ul><ul><li>Skip adding arc between X 2 and X 5 //path exists </li></ul><ul><li>Skip adding arc between X 3 and X 5 //path exists </li></ul>X 3 X 2 X 5 X 4 X 1 Copyright 2009 by Jee Vang
13. 13. Example—Procedure, Tree Construction (cont…) <ul><li>L = {L 2,3 , L 1,5 , L 1,2 , L 1,3 , L 2,5 , L 3,5 , L 1,4 ,L 4,5 , L 2,4 , L 3,4 } </li></ul><ul><li>Add arc between X 1 and X 4 in T, X 1 —X 4 </li></ul>X 3 X 2 X 5 X 4 X 1 Copyright 2009 by Jee Vang
14. 14. Example—Procedure, Tree Construction (cont…) <ul><li>L = {L 2,3 , L 1,5 , L 1,2 , L 1,3 , L 2,5 , L 3,5 ,L 1,4 , L 4,5 , L 2,4 , L 3,4 } </li></ul><ul><li>Final tree constructed </li></ul>X 3 X 2 X 5 X 4 X 1 Copyright 2009 by Jee Vang
15. 15. References <ul><li>Chow, C.K., and Liu, C.N. Approximating discrete probability distributions with dependence trees . IEEE Transactions on Information Theory , 14(3):462-467, 1968. </li></ul><ul><li>Pearl, J., and Dechter, R. Learning Structure from Data: A Survey . UCLA Cognitive Systems Laboratory, Technical Report CSD-910048 (R-132), June 1989. </li></ul><ul><li>Acid, S., and de Campos, L.M. Approximation of causal networks by polytrees. Proceedings of Information Processing and Management of Uncertainty in Knowledge-Based Systems , 1994. </li></ul>Copyright 2009 by Jee Vang
1. #### A particular slide catching your eye?

Clipping is a handy way to collect important slides you want to go back to later.