03 Uncertainty inference(discrete)

665 views

Published on

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
665
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
34
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

03 Uncertainty inference(discrete)

  1. 1. 1 Bayesian Networks Unit 3Uncertainty Inference:Discrete Wang, Yuan-Kai, 王元凱 ykwang@mails.fju.edu.tw http://www.ykwang.tw Department of Electrical Engineering, Fu Jen Univ. 輔仁大學電機工程系 2006~2011 Reference this document as: Wang, Yuan-Kai, “Uncertainty Inference - Discrete," Lecture Notes of Wang, Yuan-Kai, Fu Jen University, Taiwan, 2011.Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  2. 2. 王元凱 Unit - Uncertainty Inference (Discrete) p. Goal of this Unit • Review advanced concepts of statistics – Statistical Inference – Pattern recognition 2 Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  3. 3. 王元凱 Unit - Uncertainty Inference (Discrete) p. Related Units • Previous unit(s) – Probability Review – Statistics Review • Next units – Uncertainty Inference (Continuous) 3 Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  4. 4. 王元凱 Unit - Uncertainty Inference (Discrete) p. Self-Study • Artificial Intelligence: a modern approach – Russell & Norvig, 2nd, Prentice Hall, 2003. pp.462~474, – Chapter 13, Sec. 13.1~13.3 • 統計學的世界 – 墨爾著,鄭惟厚譯, 天下文化,2002 • 深入淺出統計學 – D. Grifiths, 楊仁和譯,2009, O’ Reilly 4 Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  5. 5. 王元凱 Unit - Uncertainty Inference (Discrete) p. 5 Contents 1. Acting Under Uncertainty …………………. 6 2. Basic Probability ..................………..……. 15 3. Marginal Probability ..…….......................... 27 4. Inference Using Full Joint Distribution ... 30 5. Independence ............................................ 43 6. Bayes Rule and Its Use ............................ 47 7. Summary ……………………………………. 62 5 Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  6. 6. 王元凱 Unit - Uncertainty Inference (Discrete) p. 6 1. Acting Under Uncertainty sensors ? ? environment agent ? actuators model Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  7. 7. 王元凱 Unit - Uncertainty Inference (Discrete) p. 7 Example 1-Localization (1/3) • Where is it – It is a robot – Sensor: camera, laser range finder, sonar – State: (x, y, orientation), Prob. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  8. 8. 王元凱 Unit - Uncertainty Inference (Discrete) p. 8 Example 1-Localization (2/3) • Where is it – It is a mobile station/robot – Sensor: Wireless LAN – State: (x, y), Prob. Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  9. 9. 王元凱 Unit - Uncertainty Inference (Discrete) p. 9 Example 1-Localization (3/3) • Where is it – It is a moving text – Sensor: computer vision techniques – State: (x, y, moving direction), Prob. t-3 t-2 t-1 t Output t Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  10. 10. 王元凱 Unit - Uncertainty Inference (Discrete) p. 10 Example 2-Correlation of Features and Words of Color • Word of color • Feature of color (Average) RGB=(255,0,0) – Red RGB=(220,10,10) RGB=(223,0,0) Uncertainty RGB=(180,20,20) – Light red RGB=(150,30,30) RGB=(147,25,25) ... Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  11. 11. 王元凱 Unit - Uncertainty Inference (Discrete) p. 11 Example 3-Target Tracking for Robot • The robot must keep the target in view • The target’s trajectory is not known in advance target robot • The environment may or may not be known Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  12. 12. 王元凱 Unit - Uncertainty Inference (Discrete) p. 12 Inaccuracy & Uncertainty  Sensor Inaccuracy • Movement Inaccuracy  Environmental Uncertainty Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  13. 13. 王元凱 Unit - Uncertainty Inference (Discrete) p. 13 Degree of Belief • Probability theory – Assigns a numerical degree of belief between 0 and 1 to an evidence – Provides a way of summarizing the uncertainty Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  14. 14. 王元凱 Unit - Uncertainty Inference (Discrete) p. 14 Techniques for Uncertainty • Bayes rule/Bayesian network with probability theory • Certainty factor in expert system • Fuzzy theory with possibility theory • Dempster-Shafer theory Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  15. 15. 王元凱 Unit - Uncertainty Inference (Discrete) p. 15 2. Basic Probability • Terms – Random variables – Full joint distribution (FJD) – Conditional probability table (CPT) Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  16. 16. 王元凱 Unit - Uncertainty Inference (Discrete) p. 16 Random Variable • Boolean random variable – Rain : true, false • Discrete random variable – Rain: cloudy, sunny, drizzle, drench • Continuous random variable – Rain: rainfall in millimeter We will focus on Boolean & discrete cases in most examples of this book Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  17. 17. 王元凱 Unit - Uncertainty Inference (Discrete) p. 17 For Boolean & Discrete R.V. • P(X) is a vector • Boolean R.V. – Rain: true, false – P(Rain) = <0.72, 0.28> • Discrete R.V. – Rain: cloudy, sunny, drizzle, drench – P(Rain) = <0.72, 0.1, 0.08, 0.1> – Normalized, i.e. sums to 1 Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  18. 18. 王元凱 Unit - Uncertainty Inference (Discrete) p. 18 Full Joint Distribution (1/3) • For a set of random variables { X1 , X2 ,  , Xn } • X1  X2    Xn are atomic events • P(X1  X2    Xn) is – A full joint probability distribution – A table of all joint prob. of all atomic events, if {X1,  , Xn} are discrete • All questions about probability of joint events can be answered by the table Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  19. 19. 王元凱 Unit - Uncertainty Inference (Discrete) p. 19 Full Joint Distribution (2/3) • Ex: X1: Rain, X2: Wind – X1: drizzle, drench, cloudy, – X2: strong, weak – X1  X2 are atomic events – P(X1  X2) is a 3x2 matrix of values Rain Drizzle Drench Cloudy Wind Strong 0.15 0.12 0.06 Weak 0.55 0.08 0.04 P(X1=drizzle  X2=strong) = 0.15, ... Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  20. 20. 王元凱 Unit - Uncertainty Inference (Discrete) p. 20 Full Joint Distribution (3/3) • All questions about probability of joint events can be answered by the table – P(Wind=Strong), P(Rain=Drizzle  Wind=Strong), ... Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  21. 21. 王元凱 Unit - Uncertainty Inference (Discrete) p. 21 Posterior v.s. Prior Probabilities • P(Cavity|Toothache) – Conditional probability – Posterior probability (after the fact/evidence) • P(Cavity) – Prior probability (the fact) Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  22. 22. 王元凱 Unit - Uncertainty Inference (Discrete) p. 22 An Example (1/2) • For a dental diagnosis – Let {Cavity,Toothache} be a set of Boolean random variables • Denotations for Boolean R.V. – P(Cavity=true) = P(cavity) – P(Cavity=false) = P(cavity) Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  23. 23. 王元凱 Unit - Uncertainty Inference (Discrete) p. 23 An Example (2/2) • The full joint probability distribution P(Toothache  Cavity) toothache toothache cavity 0.04 0.06 cavity 0.01 0.89 P(cavity  toothache)  0.04  0.01  0.06  0.11 P (cavity | toothache) P (cavity  toothache) 0.04    0.80 P (toothache) 0.04  0.01 Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  24. 24. 王元凱 Unit - Uncertainty Inference (Discrete) p. 24 Conditional Probability (Math) • P(X1=x1i| X2 =x2j) is a conditional probability • P(X1 | X2) is a conditional distribution function – All P(X1=x1i| X2 =x2j) for all possible i, j • For Boolean & discrete R.V.s, conditional distribution function is a table – Conditional distribution of continuous R.V. will not used in our discussions Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  25. 25. 王元凱 Unit - Uncertainty Inference (Discrete) p. 25 Conditional Probability Table (CPT) • For the dental diagnosis problem, – Toothache & Cavity are Boolean R.V.s – P(Toothache|Cavity) is a CPT Cavity P(toothache|Cavity) P(toothache|Cavity) T 0.90 0.1 F 0.05 0.95 Cavity P(toothache|Cavity) T 0.90 F 0.05 Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  26. 26. 王元凱 Unit - Uncertainty Inference (Discrete) p. 26 CPT v.s. FJD toothache toothache cavity 0.04 0.06 cavity 0.01 0.89 Sum of all atomic events = 1 Cavity P(toothache|Cavity) P(toothache|Cavity) T 0.90 0.1 F 0.05 0.95 Sum of a row = 1 Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  27. 27. 王元凱 Unit - Uncertainty Inference (Discrete) p. 27 3. Marginal Probability P( X i )   P(e ) e j E ( X i ) j Marginal probability • Probability of a random variable is the sum of the probabilities of the atomic events containing the random variable • Marginalization (summing out) P(toothache) toothache toothache =P(toothache  cavity)+cavity 0.04 0.06 P(toothache  cavity)cavity 0.01 0.89 =0.04 + 0.01 = 0.05 Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  28. 28. 王元凱 Unit - Uncertainty Inference (Discrete) p. 28 Marginal Probability (1/2) • Suppose a problem of a world contains only 3 random variables {X1, X2, X3}  P( X 1  X2  X3) 1 P( X 1  X 2 )   P( X x3 X 3 1  X 2  X 3  x3 ) P( X1  x1  X2  x2 )  P( X1  x1  X2  x2  X3  x3 ) x3X3 Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  29. 29. 王元凱 Unit - Uncertainty Inference (Discrete) p. 29 Marginal Probability (2/2) • Using higher order joint probability to calculate marginal and other lower order joint probability P( X 1  x1 )   P( X x 2  X 2 1  x1  X 2  x2 )    P( X x 2  X 2 x3 X 3 1  x1  X 2  x2  X 3  x3 ) Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  30. 30. 王元凱 Unit - Uncertainty Inference (Discrete) p. 30 4. Inference Using Full Joint Distributions • Probabilistic inference – Uses the full joint distribution as the "knowledge base" – Is the computation from observed evidence of posterior probabilities for query • Compute conditional probability • The most simple inference method: Inference by enumeration Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  31. 31. 王元凱 Unit - Uncertainty Inference (Discrete) p. 31 The Dental Diagnosis Example • The set of random variables: Toothache, Cavity, and Catch – All are Boolean random variables – Note: P(toothache)  P(Toothache=true) • The full joint distribution is a 2x2x2 table Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  32. 32. 王元凱 Unit - Uncertainty Inference (Discrete) p. 32 The Full Joint Distribution • 8 atomic events (sum=1) – P(toothachecatch cavity)=0.108 – P(toothachecatch cavity)=0.16 – P(toothachecatch cavity)=0.012 – P(toothachecatch cavity)=0.064 – ... Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  33. 33. 王元凱 Unit - Uncertainty Inference (Discrete) p. 33 Inference by Enumeration • P(cavity|toothache) P(cavity  toothache)  P(toothache) • We can answer the query by –Enumerating P(cavitytoothache) from the full joint distribution –Enumerating P(toothache) from the full joint distribution Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  34. 34. 王元凱 Unit - Uncertainty Inference (Discrete) p. 34 Joint Probability • P(cavity  toothache) Order-2 joint = 0.016+0.064 = 0.08 probability Marginal probability Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  35. 35. 王元凱 Unit - Uncertainty Inference (Discrete) p. 35 Marginal Probability • P(toothache) = 0.108+0.012+0.016+0.064 = 0.2 Marginal probability of Toothache Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  36. 36. 王元凱 Unit - Uncertainty Inference (Discrete) p. 36 Conditional Probability • P(cavity|toothache) P (cavity  toothache) 0.08    0 .4 P (toothache) 0.2 Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  37. 37. 王元凱 Unit - Uncertainty Inference (Discrete) p. 37 An Exercise • P(cavity  toothache)  P(Cavity=true  Toothache=true) = 0.108+0.012+0.072+0.008+0.016+0.064 = 0.28 Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  38. 38. 王元凱 Unit - Uncertainty Inference (Discrete) p. 38 Normalization (1/2) • P(cavity|toothache) and P(cavity|toothache) have the same denominator P(toothache) P(cavity  toothache) P(cavity | tootheache)  P(toothache) P(cavity  toothache) P(cavity | tootheache)  P(toothache) • 1/P(toothache) can be viewed as a normalization constant for probability calculation and derivation Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  39. 39. 王元凱 Unit - Uncertainty Inference (Discrete) p. 39 Normalization (2/2) • P(cavity|toothache) =  P(cavitytoothache) • P(cavity|toothache) =  P(cavitytoothache) =  [ P(cavitytoothache  catch) + P(cavitytoothache  catch) ] =  [ 0.108 + 0.012 ] =   0.12 = 0.6 Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  40. 40. 王元凱 Unit - Uncertainty Inference (Discrete) p. 40 A General Inference Procedure (1/2) • Let P(X|E=e) be the query – X be the query variable – E be the set of evidence variables – e be the observed values of E – H be the remaining unobserved variables (Hidden variables) • Inference of the query P(X|E=e) is P ( X | E  e)  P ( X  E  e ) 1    P ( X  E  e  H  h) 2 hH Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  41. 41. 王元凱 Unit - Uncertainty Inference (Discrete) p. 41 A General Inference Procedure (2/2) • E.x.: Query P(cavity|toothache) – X: Cavity, E/e: Toothache/true, H: Catch P (cavity | toothache ) P ( X | E  e)  P(cavity  toothache)  P ( X  E  e)   [ P(cavity  toothache  catch )    P( X  E  e  H  h)  P (cavity  toothache  catch )] hH Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  42. 42. 王元凱 Unit - Uncertainty Inference (Discrete) p. 42 Complexity of the Enumeration-based Inference Algorithm • For n Boolean R.V.s – Space complexity: O(2n) (Store the full joint distribution) – Time complexity: O(2n) (worst case) • For n discrete R.V.s, all have d discrete values – Time complexity: O(dn) (worst case) – Space complexity: O(dn) • It is not a practical algorithm, • More efficient algorithm is needed Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  43. 43. 王元凱 Unit - Uncertainty Inference (Discrete) p. 43 5. Independence • A and B are independent iff – P(A|B) = P(A), or – P(B|A) = P(B), or – P(AB) = P(A)P(B) A B  Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  44. 44. 王元凱 Unit - Uncertainty Inference (Discrete) p. 44 The Dental Diagnosis Example (1/2) • Original example: 3 random variables • Add a fourth variable: Weather – The Weather variable has 4 values – The full joint distribution: P(Toothache,Catch,Cavity,Weather) has 32 elements (2x2x2x4) • By commonsense, Weather is independent of the original 3 variables – P(ToothacheCatch  Cavity  Weather) = P(Toothache  Catch  Cavity)P(Weather) Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  45. 45. 王元凱 Unit - Uncertainty Inference (Discrete) p. 45 The Dental Diagnosis Example (2/2) • P(Toothache,Catch,Cavity,Weather) = P(Toothache,Catch,Cavity)P(Weather) – The 32-element table for 4 variables can be constructed from • One 8-element table, and • One 4-element table – Reduced from 32 elements to 12 elements Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  46. 46. 王元凱 Unit - Uncertainty Inference (Discrete) p. 46 Advantages of Independence • Independence can dramatically reduce the amount of elements of the full distribution table – i.e., independence can help in reducing • The size of the domain representation, and • The complexity of the inference problem • Independence are usually based on knowledge of the domain Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  47. 47. 王元凱 Unit - Uncertainty Inference (Discrete) p. 47 6. Bayes Rule and Its Use • Given two random variables X, Y • By product rule P( X  Y )  P( X | Y )P(Y ) P( X  Y )  P(Y | X )P( X ) P( X | Y ) P(Y )  P(Y | X )  P( X ) • Bayes rule underlies all modern AI systems for probabilistic inference Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  48. 48. 王元凱 Unit - Uncertainty Inference (Discrete) p. 48 Bayes’ Theorem Likelihood Prior P (e / h ) P ( h ) P (h / e)  P (e) Posterior Probability of Evidence Probability of an hypothesis, h, can be updated when evidence, e, has been obtained. Note: it is usually not necessary to calculate P(e) directly as it can be obtained by normalizing the posterior probabilities, P(hi | e). Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  49. 49. 王元凱 Unit - Uncertainty Inference (Discrete) p. 49 A Simple Example Consider two related variables: 1. Drug (D) with values y or n 2. Test (T) with values +ve or –ve And suppose we have the following probabilities: P(D = y) = 0.001 P(T = +ve | D = y) = 0.8 P(T = +ve | D = n) = 0.01 These probabilities are sufficient to define a joint probability distribution. Suppose an athlete tests positive. What is the probability that he has taken the drug? P (T   ve | D  y ) P ( D  y ) P(D  y|T   ve)  P (T   ve | D  y ) P ( D  y )  P (T   ve | D  n ) P ( D  n ) 0.8  0.001  0.8  0.001  0.01  0.999  0.074 Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  50. 50. 王元凱 Unit - Uncertainty Inference (Discrete) p. 50 A More Complex Case Suppose now that there is a similar link between Lung Cancer (L) and a chest X- ray (X) and that we also have the following relationships: History of smoking (S) has a direct influence on bronchitis (B) and lung cancer (L); L and B have a direct influence on fatigue (F). What is the probability that someone has bronchitis given that they smoke, have  P(b , s f , x , l ) fatigue and have received a positive X-ray result? 1 1 1 1 P(b , s , f , x ) P(b | s , f , x )  1  1 1 1 l  P(b, s , f , x , l ) 1 1 1 1 P( s , f , x ) 1 1 1 1 1 1 b ,l where, for example, the variable B takes on values b1 (has bronchitis) and b2 (does not have bronchitis). R.E. Neapolitan, Learning Bayesian Networks (2004) Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  51. 51. 王元凱 Unit - Uncertainty Inference (Discrete) p. 51 An Example • Prior knowledge – The probability of a patient has meningitis is 1/50,000: P(m)=1/50000 – The probability of a patient has stiff neck is 1/20: P(s)=1/20 – The meningitis causes the patient to have a stiff neck 50% of the time: P(s|m)=0.5 • The probability of a stiff-neck patient has meningitis P(s | m)P(m) 0.51/ 50000 P(m | s)    0.0002 P(s) 1/ 20 Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  52. 52. 王元凱 Unit - Uncertainty Inference (Discrete) p. 52 Generalization of Bayes’ Rule • Conditionalized on more evidences, say e P ( X | Y , e) P (Y | e) P(Y | X , e)  P ( X | e) • P(Cavity|toothachecatch) = (P(toothachecatch|Cavity)P(Cavity)) / P(toothachecatch) => P(toothache  catch  Cavity) / P(toothachecatch) Better way? Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  53. 53. 王元凱 Unit - Uncertainty Inference (Discrete) p. 53 Use Independence in Conditional (1/2) • Probe Catch and Toothache are independent, given the presence or the absence of a Cavity – Both Catch and Toothache are caused by Cavity, but neither has a direct effect on the other • P(toothachecatch|Cavity) = P(toothache|Cavity) P(catch|Cavity) • Conditional independence Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  54. 54. 王元凱 Unit - Uncertainty Inference (Discrete) p. 54 Use Independence in Conditional (2/2) • If Catch is conditionally independent of Toothache given Cavity • Equivalent statements – P(Catch  Toothache|Cavity) =P(Catch|Cavity) P(Toothache|Cavity) – P(Catch|ToothacheCavity) =P(Catch|Cavity) – P(Toothache|CatchCavity) =P(Toothache|Cavity) Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  55. 55. 王元凱 Unit - Uncertainty Inference (Discrete) p. 55 Conditional Independency for the Dentist Diagnosis Problem (1/2) • The conditional independence, like independence, can also decompose the full joint distribution into smaller pieces • P(Toothache  Catch  Cavity) • = P(Toothache  Catch|Cavity)P(Cavity) (by product rule) • = P(Toothache|Cavity) P(Catch|Cavity) P(Cavity) (by definition of conditional independence) Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  56. 56. 王元凱 Unit - Uncertainty Inference (Discrete) p. 56 Conditional Independency for the Dentist Diagnosis Problem (2/2) • P(Toothache  Catch  Cavity) has 23-1 elements • P(Toothache|Cavity)P(Catch|Cavity)P(Cavity) has 2+2+1=5 elements P(toothache|Cavity) P(toothache|Cavity) cavity 0.9 0.1 cavity 0.05 0.95 Conditional Probability Table (CPT) • Conditional independence can reduce the amount of probabilities Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  57. 57. 王元凱 Unit - Uncertainty Inference (Discrete) p. 57 Probabilistic Modeling of Problems (1/2) • Usually random variables have two semantics – Cause – Effect • Conditional probability P(Y|X) can be rewritten as P(Cause|Effect) or P(Effect|Cause) Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  58. 58. 王元凱 Unit - Uncertainty Inference (Discrete) p. 58 Probabilistic Modeling of Problems (2/2) • We usually assume some effects are conditionally independent given a cause – P(effect1  effect2 | cause) = P(effect1 | cause) P(effect2 | cause) The conditional independence can be drawn as a graphic model Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  59. 59. 王元凱 Unit - Uncertainty Inference (Discrete) p. 59 Advantages of Conditional Independence • The use of conditional independence reduces the size of the joint distribution from O(2n) to O(n) • Conditional independence is our most basic and robust form of knowledge about uncertain environments Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  60. 60. 王元凱 Unit - Uncertainty Inference (Discrete) p. 60 Conditional Independence (Math) • If P(Xi|Xj,X)=P(Xi|X), we say that variable Xi is conditional independent of Xj, given X – Denoted as I(Xi,Xj|X) – It means for Xi, if we know X, we can ignore Xj • If I(Xi,Xj|X), P(Xi,Xj|X) = P(Xi|X) P(Xj|X) – By chain rule, P(Xi,Xj|X) = P(Xi|Xj,X) p(Xj|X) – Since p(Xi|Xj,X)=p(Xi|X), – Then p(Xi,Xj|X)= p(Xi|X) p(Xj|X) Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  61. 61. 王元凱 Unit - Uncertainty Inference (Discrete) p. 61 Pairwise Independence • A generalization of pairwise independence – We say that the variables X1, …, Xn are mutually conditionally independent, given a n set X p( X 1 , X 2 , X n | X )   p( X i | X ) Why? i 1 n By chain rule P( X 1 , X 2 ,, X n )   P( X i | X i 1 ,, X 1 ) n i 1  P( X 1 , X 2 ,  , X n | X )   P( X i | X i 1 ,  , X 1 , X ) i 1 Since Xi is conditional independent of the others given X n  p( X 1 , X 2 , X n | X )   p( X i | X ) i 1 When X is empty, we have p(X1,X2, …,Xn)=p(X1)p(X2)…p(Xn) Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
  62. 62. 王元凱 Unit - Uncertainty Inference (Discrete) p. 62 7. Summary • The full joint distribution can answer any query of the domain – However, it is intractable • Independence and conditional independence is important for the reduction of the full joint distribution • We can now move on to Bayesian networks Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright

×