Fundamentals of Engineering Probability Visualization Techniques & MatLab Case Studies

4,531 views

Published on

This four-day course gives a solid practical and intuitive understanding of the fundamental concepts of discrete and continuous probability. It emphasizes visual aspects by using many graphical tools such as Venn diagrams, descriptive tables, trees, and a unique 3-dimensional plot to illustrate the behavior of probability densities under coordinate transformations. Many relevant engineering applications are used to crystallize crucial probability concepts that commonly arise in aerospace CONOPS and tradeoffs

Published in: Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
4,531
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
22
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Fundamentals of Engineering Probability Visualization Techniques & MatLab Case Studies

  1. 1. Course Sampler From ATI Professional Development Short Course Fundamentals of Engineering Probability Visualization Techniques & MatLab Case Studies Instructor: Dr. Ralph E. MorgansternATI Course Schedule: http://www.ATIcourses.com/schedule.htmATIs Engineering Probability: http://www.aticourses.com/Fundamentals_of_Engineering_Probability.htm
  2. 2. www.ATIcourses.comBoost Your Skills 349 Berkshire Drive Riva, Maryland 21140with On-Site Courses Telephone 1-888-501-2100 / (410) 965-8805Tailored to Your Needs Fax (410) 956-5785 Email: ATI@ATIcourses.comThe Applied Technology Institute specializes in training programs for technical professionals. Our courses keep youcurrent in the state-of-the-art technology that is essential to keep your company on the cutting edge in today’s highlycompetitive marketplace. Since 1984, ATI has earned the trust of training departments nationwide, and has presentedon-site training at the major Navy, Air Force and NASA centers, and for a large number of contractors. Our trainingincreases effectiveness and productivity. Learn from the proven best.For a Free On-Site Quote Visit Us At: http://www.ATIcourses.com/free_onsite_quote.aspFor Our Current Public Course Schedule Go To: http://www.ATIcourses.com/schedule.htm
  3. 3. Fundamental Probability Concepts • Probabilistic Interpretation of Random Experiments (P) – Outcomes: sample space – Events: collection of outcomes (set theoretic) – Probability Measure: assign number “probability” P ε [0,1] to event • Dfn#1-Sample Space (S): Fine-grained enumeration (atomic - parameters) – List all possible outcomes of a random experiment – ME - Mutually exclusive - Disjoint “atomic” – CE - Collectively exhaustive - Covers all outcomes • Dfn#2- Event Space (E): Coarse-grained enumeration (re-group into sets) – ME & CE List of Events S (all outcomes) Atomic Outcomes Events: A,B,C ME but not CE A D (Disjoint by dfn) Events: A,B,C ,D both ME & CE C B 14 INDEXDiscrete parameters uniquely define the coordinates of the Sample Space (S) and the collection of allparameter coordinate values defines all the atomic outcomes. As such atomic outcomes are mutuallyexclusive (ME) and collectively exhaustive (CE) and constitute a fundamental representation of the SampleSpace S.By taking ranges of the parameters such as A, B, C, and D, one can define a more useful Event Space whichshould consist of ME and CE events which cover all outcomes in S without overlap as shown in the figure. 14
  4. 4. Fair Dice Event Space Representations d2 • Coordinate Representation: 6 – Pair 6-sided dice 5 A: d1=3, d2 =arb. 4 – S={(d1,d2): d1,d2 = 1,2,…,6} 3 2 C: d1=d2 – 36 Outcomes Ordered pairs 1 d1 1 2 3 4 5 6 B: d1+d =7 • Matrix Representation: 1  [1 2 3 4 5 6]  (1,1) (1,2) (1,3) (1,4) (1,5) (1,2 )  6   (2,1) (2,2) (2,3) (2,4) (2,5) (2,6) – Cartesian Product: 2   3   = (3,1)  (3,2) (3,3) (3,4) (3,5) (3,6)  – {d1} x {d2} = d1 d2T 4 (4,1) (4,2) (4,3) (4,4) (4,5) (4,6)   (5,1) (5,2) (5,3) (5,4) (5,5) (5,6) 5     (6,1)  (6,2) (6,3) (6,4) (6,5) (6,6)  6 • Tree Representation: d2 d1 (1,1) (1,2) 1 (1,3) 36 Outcomes (1,4) Ordered Pairs 2 (1,5) 3 (1,6) • Polynomial Generator for Sum Start 4 2 Dice 5 (6,1) (6,2) ( x1 + x 2 + x3 + x 4 + x5 + x 6 ) 2 = 1x 2 + 2 x3 + 3 x 4 + 4 x5 + 5 x 6 + 6 x 7 6 (6,3) (6,4) Exponents represent + 5 x8 + 4 x9 + 3 x10 + 2 x11 + 1x12 (6,5) (6,6) 6-sided die face numbers Exponents represent pair sums Coefficients represent #ways 16It is helpful to have simple visual representations of Sample and Event SpacesFor a pair of 6-sided dice, coordinate, matrix, and tree representations are all useful representations. Alsothe polynomial generator for the sum of a pair of 6-sided dice immediately gives probabilities for each sum.Squaring the polynomial (x1+x2+x3+x4+x5 +x6)2 yields a generator polynomial whose exponents representall possible sums for a pair of 6-sided dice S={2,3,4,5,6,7,8,9,10,11,12}and whose coefficients C={1,2,3,4,5,6,5,4,3,2,1} represent the number of ways each sum can occur. Dividing by the coefficients C bythe total #outcomes 62 = 36 yields the probability “distribution” for the pair of dice.Venn diagrams for two or three events are useful; for example, the coordinate representation in the topfigure can be used to visualize the following events A: {d1 = 3 and d2 = arbitrary, B= {d1 + d2 = 7}, and C= {d1 = d2}Once we display these two events on the coordinate diagram their intersection properties are obvious, viz.,both A & B and A & C intersect, albeit at different points, while B & C do not intersect (no pointcorresponding to sum=7 and equal dice values). More than three intersecting sets, become problematic forVenn diagrams as the advantage of visualization is muddled somewhat by the increasing number ofoverlapping regions in theses cases (see next two slides). 16
  5. 5. Venn Diagram for 4 Sets 4C = (4C1 4-Singles) – (4C2 6-Pairs) + (4C3 4-Triples ) - ( 4C4 1-Quadruple) 0 A B AB BD AC ABD ABC AD ABCD BC ACD BCD CD D C 17As we go to Venn diagrams with more than 3 sets the labeling of regions becomes a practical limitation totheir use. In this case of 4 sets A,B,C, D, the labeling is still pretty straightforward and usable.The 4 singles A,B,C,D are labeled in an obvious manner at the edge of each circle.The 6 pairs AB,AC,AD,BC,BD,CD are labeled at the intersection of two circles. The 4 triples ABC, ABD,BCD, ACD are labeled within “curved triangular areas” corresponding to the intersections of three circles.The 1 quadruple ABCD is labeled within the unique “curved quadrilateral area” corresponding to theintersection of all four circles. 17
  6. 6. Trivial Computation of Probabilities of Events sum = d1 + d2 d2 Ex#1 Pair of Dice E1 S={(d1,d2): d1,d2 = 1,2,…,6} 6 12 5 11 E2 10 E1={(d1,d2): d1+d2 ¥ 10} 4 9 8 P(E1)=6/36=1/6 3 7 6 2 5 E2={(d1,d2): d1+ d2 = 7} 4 P(E2)=6/36=1/6 1 3 2 d1 1 2 3 4 5 6 Ex#2 Two Spins on Calibrated Wheel S={(s1,s2): s1,s2 ε [0,1]} s2 E1={(s1,s2): s1+s2 ¥ 1.5}--> P(E1) = ----- =.52/2=1/8 1 1 E1 0.5 E3 E2={(s1,s2): s2 § .25} --> P(E2)=1(.25)/1=.25 E2 0 s1 E3={(s1,s2): s1= .85; s2= .35}--> P(E3)=0/1=0 0 0.5 1 20For equally likely atomic events the probability of any outcome Event is easily computed as the (#atomicoutcomes in Event)/(total # outcomes). For a pair of dice, the total # of outcomes is 6*6=36 and hencesimple counting of the # points in E /36 yields P(E), etc.Two spins on a calibrated wheel [0, 1) can be represented by the unit square in the (s1 , s2)-plane and ananalogous calculation can be performed to obtain the probability for the event E by dividing the areacovered by the event by the area of the event space (“1”): P(E)= area(E)/ 1. 20
  7. 7. DeMorgans’ Formulas - Finite Unions and Intersections i) Compl(Union) = Intersec(Compls): ( E1 ∪ E2 ∪ c ∪ En ) c = E1 ∩ E2 ∩ c ∩ En c c c c ii) Compl(Intersec) = Union(Compls): ( E1 ∩ E2 ∩ ∩ En ) c = E1 ∪ E2 ∪ ∪ En Useful Forms: A∪ B i’) Union expressed ( A ∪ B) c = Ac B c Visualization Compl(Union) Intersec(Compl) ( A ∪ B)c as an Intersection (( A ∪ B) ) c c = A ∪ B = ( Ac B c ) c A Ac Intersect grey areas B Bc Ac & B c ii’) Intersection ( AB) c = Ac ∪ B c Ac B c Yields one Union(Compl) grey area Ac B c expressed as a Union Compl(Intersec) with A and B excluded (( AB) ) c c ( = AB = Ac ∪ B c )c Taking its complement ( Ac B c )c yields white area, i.e., A ∪ B 24 INDEXDeMorgan’s Laws for the complement of finite unions and intersections states thati) The complement of unions equals the intersections of the complements, andii) The complement of intersections equals the union of complementsThe alternate forms obtained by taking the complements of the original equations are often more useful because they give a direct decomposition of the union and the intersection of two or more setsi’) The union equals the complement of the (intersection of complements)ii’) The intersection equals the complement of the (union of complements)A graphical construction of A U B = (Ac Bc)c is also shown in the figure..Ac and Bc are the two shaded areas in the middle planes which exclude A and B respectively (white) ovalsIntersecting these two shaded areas and taking the complement leaves the white oval areas which is A U B 24
  8. 8. Set Algebra Summary Graphic Union A ∪ B = A ∪ Ac B = B ∪ Bc A Union AUB “A-B” “B-A” Intersection A ∩ B = A ⋅ B = AB A Bc A AB B Ac B x ∈ AB iff x ∈ A & x ∈ B Intersection Difference A − B ≡ A ∩ B c = AB c x ∈ A − B iff x ∈ A and x ∉ B Differences DeMorgans A ∪ B = ( Ac B c )c ( A ∪ B )c = Ac B c means ( ) c AB = Ac ∪ B c complement of (At least one) = (not any) 27This summary graphic illustrates the set algebra for two sets A , B and their union intersection anddifference.DeMorgans Law can be interpreted as saying “the complement of (“at least one”) is “not any”Associativity and commutivity of the two operations allows extension to more than two sets. 27
  9. 9. Basic Counting Principles Principle #0: Take Case n=3-4; generalize to n Binomial Expansion: (a+b)3 (a+b)n Repetitions Allowed Principle #1: Product Rule for Sub-experiments: 6- Bins = 263 ⋅103 m Num Suit Licenses ⋅ nm = ∏ nk 26 26 26 10 10 10 n = n1 ⋅ n2 H 1 D S C H 5 16- Bins k =1 Start 2 D Binary S 2 216 = 65,536 C H 13 D 2 2 2 2 ... 2 Generate “tree” of outcomes S C Digits #ways: 13 * 4 = 52 No Repetitions Principle #2: Perm n distinguish-obj take k k=n Arrange 11 Travel 5 Cooking 4 Garden All Books n! “Fill k-bins” 11! 5! 4! n Pk = (n) k = 3! Permute Groups (n − k )! k<n 11 Travel Books in 5 bins 11| 10| 9 |8 |7 Principle #3:Perm n-obj take n with r - Arrange 4! groups of indistinguishable objects Letters “TOOL” = 12 2!⋅1!⋅1! hable  # Distinguis n! 10!  Sequences  = n !⋅n !⋅ ⋅ n !   r − groups {4”r”, 3”s”, 2”o”, 1 “t”} 4!⋅3!⋅2!⋅1! = 12,600   1 2 r Principle #4: Combination of n-objects take k Committee of 4 22! 22! C4 = 22 = = 7315 from 22 people (22 − 4)!4! 18!4! n n! n Ck =   =  k  k ! ( n − k )! k ≤ n Order not   Committee of 3 {2M, 1F} 6⋅5 important! from {6M, 3F} 6 C2 ⋅3 C1 = ⋅ 3 = 45 2! = Principle #3 with {taken , not taken} not counted 28 INDEXOutcomes must be distinguished by labels. They are characterized by either i) distinct orderings or ii)distinct groupings. A grouping consists of objects with distinct labels; changing order within a group is nota new group, but is a new permutation. The four basic counting principles for groups of distinguishableobjects are summarized and examples of each are displayed in the table.Principle#0: This is practical advice to solve a problem with n= 2,3,4 objects first and then generalize the“solution pattern” to general n.Principle#1: This product rule is best understood in terms of the multiplicative nature of outcomes as we“branch out” on a tree. For a a single draw from a deck of cards there are 13 “number” branches and, inturn, each of these has 4 “suit” branches yielding 13*4 =52 distinguishable cards or outcomes.Principle#2: Permutation (ordering) of n objects take k at a time is best understood by setting up “k-containers” putting one of “n” in the first, one of “n-1” , ... and finally one of “n-k+1” in the kth container.The total #ways is obtained by the product rule as n*(n-1)*...*(n-k+1) = n!/(n-k)!Principle#3: Permutation of all ”n” objects consisting of “r “ groups of indistinguishable objects {3 t , 4s 5 u}. If all objects were distinguishable then the result would be n! permutations; however permutationswithin the r groups does not create new outcomes and therefore we divide by factorials of the numbers ineach group to obtain n!/(n1! n2! ... nr!)Principle#4: Combination of n objects take k is related to Principles#2, #3. There are n! permutations;ignoring permutations within r= 2 groups {“taken” , “not taken”} yields n!/(n! (n-k)!) 28
  10. 10. Counting with Replacement Refills Drop Down Select “B” from Alphabet and Replace A A B B ... Y Y Z Z Always have 26 letters to choose from A A B B Y Y Z Z 23 =8 distinct 4 distinct Permutation of “n” obj with (# drws) orderings groupings replacement taken “k” at a time n Pk = # replaceable objects = nk A {AAA} 3 “A” B {AAB} 2 “A”& 1”B” A n n n n n…n A {ABA} 2 “A”& 1”B” A B B {ABB} 2 “B”& 1”A” Bin# 1 2 3 …k S A A {BAA} 2 “A”& 1”B” B n=2 , k=3 B B {BAB} 2 “B”& 1”A” A {BBA} 2 “B”& 1”A” B {BBB} 3 “B” Combination of “n” obj with replacement taken “k” at a time effective # objects  n + k − 1  n + k − 1 n Ck = / n + (k-1) = n + k −1 Ck =  =  Note: “k” can be larger than “n” (draw k)  k   n −1  Example: From 2 objects {A, B} choose 3 with replacement (Only Way!) After each draw of an A or B “drop 4 Outcomes down a replacement” add 1 after each A B A/B A/B {AAA},{BBB} draw except last 4! {ABB},{AAB} (effective # objects) = 2 +(3-1)=4 2 C3 = 2+3−1C3 = 4C3 = / =4 3! 1! 41 INDEXCounting permutations and combinations with replacement is analogous to a candy machine purchase inwhich a new object drops down to replace the one that has been drawn, thus giving the same number ofchoices in each draw.Permutation of n obj taken k at a time with replacement: Each of the k draws has the same number ofoutcomes n because of replacement, the result is n*n*n... *n = nk and is written nPk with an “over-slash” onthe permutation symbol. The case n=2, k=3 of 3 draws with 2 replaceable objects {A,B} shows the slash-2 P3 =23 = 8 permutations that result.Combination of n obj taken k at a time with replacement: For n=2, k=3, 2 take 3 does not make anysense. However, with replacement, it does since each draw except the last drops down an identical item andhence the number of items to choose from becomes n +(k-1) and slash-nCk = n+(k-1)Ck. The tree verifies thisformula and explicitly shows that there are 4 distinct groupings {3A, 3B, 2A1B, 1A2B} exactly the numberof combinations with replacement given by the general formula slash-2C3 = 2+(3-1)C3 = 4C3 =4 41
  11. 11. II) Fundamentals of Probability 1. Axioms 2. Formulations: Classical, Frequentist, Bayesian, Ad Hoc 3. Adding Probabilities: Inclusion / Exclusion, CE & ME 4. Application of Venn Diagrams & Trees 5. Conditional Probability & Bayes’ “Inverse Probability” 6. Independent versus Disjoint Events 7. System Reliability Analysis 47As a theory, Probability is based on a small set of axioms which set forth fundamental properties ofconstruction.In practice, probability may be formulated theoretically, experimentally, or subjectively, but must alwaysobey the basic Axioms.Evaluating probabilities for events, is naturally developed in terms of their unions and intersections usingVenn Diagrams, Trees and Inclusion/Exclusion techniques.Conditional probabilities, their inverses (Bayes’ theorem), and the dependence between two or more eventsflow naturally from the basic axioms of probability.System reliability analysis utilizes all these fundamental concepts 47
  12. 12. Inclusion / Exclusion Ideas ME Events A,B - Disjoint AB= φ A B P(A∩B) = P(A) + P(B) No intersections ”Add Prob” No intersections Intersect: “CE, not ME” “Recast” as Disjoint Union “CE & ME” Not Disjoint AB∫φ A A B-A B ∫ AB P(A∩B) = P(A) + P(B-A) = P(A) + P(BAc) Intersection “AB” Counted Twice!! P(A∩B) ∫ P(A) + P(B) B = B ⋅ S = B ⋅ ( A ∪ Ac ) = BA ∪ BAc Subtract “P(AB)” from sum; count only once A BAC B P ( A ∪ B ) = P ( A) + P ( B ) − P ( AB ) AB P( BAc ) = P( B) − P( AB) Generalization by Induction: let D = B ∪ C P ( A ∪ B ∪ C ) = P ( A ∪ D ) = P ( A) + P ( D) − P ( AD ) = P ( A) + P ( B ∪ C ) − P( A ⋅ ( B ∪ C )) = P ( A) + {P ( B ) + P (C ) − P ( BC )} − {P ( AB ) + P ( AC ) − P ( ABAC )} Inclusion / P ( A ∪ B ∪ C ) = P ( A) + P ( B ) + P (C ) − P ( AB ) − P ( AC ) − P ( BC ) + P ( ABC ) Exclusion add singles subtract pairs add triples 54 INDEXIt is important to realize that although probabilities are simply numbers that add, the probability of theunion of two events P(A U B) is not equal to the sum of individual probabilities for the two events P(A) +P(B).This is because points in this overlap region AB are counted twice; to correct for this one needs to subtractout “once” the double counted points in the overlap yielding P(A U B) = P(A) + P(B)-P(AB).Only in the case of non-intersection AB = φ does the simple sum of probabilities hold.The generalization for a union of three or more sets alternates inclusion and exclusion; for A,B,C theprobability P(AUBUC) adds the singles, subtracts the doubles and adds the triple as shown. 54
  13. 13. Venn Diagram Application: Inclusion/Exclusion Given following information find how many club members play at least one sport T or S or B T (36) TS (22) S (28) Club: 36 T , 28 S, 18 B TSB (4) SB (9) Let N= Total # members (unknown) TB (12) 36 28 18 B (18) Write Probabilities as P(T) = ; P(S) = ; P(B) = ; etc. N N N CLUB Method 1: Subs into Formula for Union P ( T ∪ S ∪ B) = P (T ) + P( S ) + P( B ) − P (TS ) − P(TB ) − P ( BS ) + P (TBS ) 36 28 18 22 12 9 4 TS (22) = + + − − − + T (36) STc (6) N N N N N N N 43 18 1 = Thus 43 of “N” Club Members play 6 TSB N at least one sport. (N is irrelevant) (4) 5 8 SB (9) Method 2: Disjoint Union - Graphical TB (12) 1 T ∪ S ∪ B = T ∪ ST ∪ BT Sc c c BTcSc (1) CLUB P(T ∪ S ∪ B) = P(T ) + P( ST c ) + P( BT c S c ) 36 6 1 43 = + + = N N N N 68 INDEXThis example illustrates the ease by which a Venn diagram can display the probabilities associated with thevarious intersections of 3 sets T, S, and B.The number of elements in each of the 7 distinct regions is easily read off the figure; they are required toestablish the total number in their union T U S U B via the inclusion/exclusion formula.Another method of finding P(T U S U B ) is to decompose the union T U S U B into a union of disjoint setsT* U S* U B* for which the probability is additive, i.e., P(T* U S* U B* ) = P(T*) + P(U*) + P(B*). 68
  14. 14. Matching Problem – 1 “N” men throw hats onto floor; Each man in turn randomly draws a hat a) No Matches - Find Probability None draw own hat. Let Event Ei = ith man chooses his own hat ; compute: P(0 − matches) = 1 − P( E1 ∪ E2 ∪ ∪ EN ) 1|2|3|… | k | k+1 | … |N Hats i1 |i2 | i3 | … | in in+1 | in+2 | in+3 | … | iN Men Probability that M1 & M2 &...&Mn irrespective of what n “Ei s” choose own hats (N-n) Does not Matter draw own hats other men draw (Matched or Not Matched ) Total # of“n-tuple” N # perms ( N − n)!   P( Ei1 Ei2 Ein ) = = selections from N n Total# perms N!    N  ( N − n)! N! ( N − n)! 1 Sum Joint Probabilities ∑ P( Ei1 Ei2 Ein ) =   ⋅ = n !( N − n)! N ! = over all “n-tuples” n −tuples All n-tuples Eq. Likely n N! n!   P (0 − Matches ) = 1 − P ( E1 ∪ E2 ∪ E3 ) = 1 −  ∑ P ( Ei1 ) − ∑ P ( Ei1 Ei2 ) + ∑ P( Ei1 Ei2 Ei3 ) = 1 − {1 − 2! + 3!} = 1 1 1 3 1− tuples pairs triples  P(0 − matches) = 1 − P( E1 ∪ E2 ∪ ∪ EN ) = 1 − 1 + 1 − 2! 3! 4! 5! 1 + + ( −1) N 1 N!  e−1 N →∞ → b) k- Matches Poisson with success rate λ=1/N & “time  k! ⋅ e−1 →1 1 1 1 N −k 1   − + + + ( −1)  ( N − k )!  P(k matches) =  2! 3! 4! N→∞ intvl” t = N samples; a=λ *t =(1/N)*N =1 k! 69 INDEXHere is an example that requires the inclusion/exclusion expansion for a large number of intersecting sets.Since it becomes increasingly difficult to use Venn diagrams for a large number of intersecting sets, wemust use the set theoretic expansion to compute the probability. We shall spend some time on this problemas it is very rich in probability concepts.The problem statement is simple enough: “N men throw their hats onto the floor; each man in turnrandomly draws a hat. “a) What is the probability that no man draws his own hat?b) What is the probability of exactly k-matches?Key ideas: define Event Ei = ith man selects his own hat then take union of N sets E1 U E2 U ... U EN and P(no-matches)=1- P(E1 U E2 U ... U EN)The expansion of the P(E1 U E2 U ... U EN) involves addition and subtraction of P(singles), P(pairs),P(triples), etc. ( The events Ei are CE but not ME so you cannot simply sum up the P(Ei ) for k singles toobtain an answer to part b)) .This slide shows a key part of the proof which establishes the very simple result that the sum over singles,P(singles) = 1/(1!); sum over pairs is P(pairs)= 1/(2!) ; sum over triples is P(triples)=1/(3!); sum over 4-tuples, P(4-tuples) = 1/(4!); ... sum over N-tuples, P(N-tuple) = 1/(N!).Limit as N large approaches a Poisson Distribution with success rate for each draw λ=1/N and data lengtht =N i.e., parameter a =λ t =1 69
  15. 15. Man Hat Problem n =3 Tree/Table Counting M#1 M#2 M#3 M.E. Match Tree#1 Drw#1 Drw#2 Drw#3 Outcomes Outcomes M#1 M#2 M#3 #Matches E2 1 2 3 E3 {E1 E2 E3 } triple 1 2 3 3 1/2 Br#1 1 E1 1/2 3 2 c {E1 E2 E3 } c single 1 3 2 1 1/3 1 1/2 1 1 3 E3 {E1c E2 c E3 } single 2 1 3 1 E1C Br#2 Start 1/3 2 1/2 1 1 c c {E1 E2 E3 } c No-match 2 3 1 0 3 1/3 1/2 3 1 1 2 c c {E1 E2 E3 } c No-match 3 1 2 0 Br#3 E1C 1/2 2 E2 1 1 c {E1 E2 E3 } c single 3 2 1 1 P(Ei) = 1/3 2/6 2/6 From Table: From Tree: Connection: Matches & Events Prob[0-matches]=2/6 Prob[0-matches]=1-Pr[E1 U E2 U E3] Prob[1-matches]=3/6 Prob[Sgls]=P[E1]=P[E2]=P[E3]=1/3 =1-{Sum[Sngls]-Sum[Dbls]+Sum[Trpls]} Prob[2-matches]=0/6=0 Prob[Dbls] = P[E1E2]=(1/3)(1/2)=1/6 =1-{3(1/3) -3(1/6)+1(1/6)}=2/6 Prob[3-matches]=1/6 Prob[Trpls] = P[E1E2E3]=(1/3)(1/2)=1/6 Alternate Trees Yield: P[E1E3]= P[E2E3]=1/6 75This slide shows the complete the tree and associated table for the Man - Hat problem in which n=3 menthrow their hats in the center of a room and then randomly select a hat. The drawing order is fixed asMan#1, Man#2, Man #3, and the 1st column of nodes labeled as circled 1, 2, 3 shows the event E1 in whichthe Man#1draws his own hat, and the complementary event E1c i.e., Man#1 does not draw his own hat . The2nd column of nodes corresponds to the remaining two hats in each branch shows the event E2 in which theMan#2 draws his own hat; note that E2 has two contributions of 1/6 summing to 1/3. Similarly, the 3rd drawresults in the event E3 in two positions shown again summing to 1/3.The tree yields ME & CE outcomes expressed as composite states such as {E1E2E3}, {E1E2cE3c, etc., orequivalently in terms of the number of matches in the next column. The nodal sequence in the tree can betranslated into the table on the right which is analogous to the table we used on the previous slide. Thenumber of matches can be counted directly from the table as shown.The lower half of the slide compares the “ # of matches” events with the “compound events” formed fromthe “Ei”s{ no-matches, singles, pairs, and triples }. The connection between these two types of events isbased on the common event “no-matches,” i.e., the inclusion/exclusion expansion of the expression [1-P(E1U E2U E3) ] in terms of singles doubles and triples yields P(0-matches). 75
  16. 16. Conditional Probability - Definition & Properties ˆ P ( AS )  2 • Definition of Conditional Probability ˆ P( A | S ) ≡ =  ˆ P( S )  3 • In terms of atomic events si we can formally write ˆ ˆ P( ∪ si S ) ∑ P( s S ) ˆ i (# pts in Sˆ & A) A = ∪ si ˆ ) = P ( A S ) = si ∈ A = si ∈ A = si ∈ A P( A | S ˆ P( S ) ˆ P( S ) ˆ P( S ) (# pts in Sˆ ) ˆ • Note in case S = S it reduces to P(A) as it must A B •Asymmetry of Conditional Probability BA P(BA) P ( BA)  fraction  BA P ( B | A) = = = P ( A)  BA over A    A Given A Not Symmetrical! P( BA)  fraction  BA P( A | B) = = = P( B)  BA over B    Given B B 82 INDEXThe formal definition of conditional probability follows directly from the renormalization concept discussedon the previous slide. It is simply the joint probability defined on the intersection of the set A and S-cap,P(AS-cap) divided by the normalizing probability P(S-cap).It can also be written explicitly in terms of a sum over atomic events given in the second equation.Conditional probability is not symmetric because the joint probability on the intersection of A and B isdivided by probability of the conditioning set which is P(A) in one case and P(B) in the other. This is alsoeasily visualized using Venn diagrams where the “shape division” are obviously different in the two cases. 82
  17. 17. Examples - Coin Flips, 3-Sided Dice nH > nT Flip#3 Example#1: Three Coin Flips Flip#2 H {HHH} Given the first flip is H, Find Flip#1 H T {HHT} ˆ S Prob #H > #T H {HTH} H T T {HTT} #H > #T S 4 1 1 1 ˆ P ( S ) = ; P( HHH ) = ; P ( HHT ) = ; P( HTH ) = S H H {THH} 8 8 8 8 T T T {THT} 3 P ( HHH ) + P ( HHT ) + P ( HTH ) 3 = 8= H {TTH} P (nH > nT | H ) = ˆ) P( S 4 4 T 8 {TTT} Example#2: 4-Sided Dice Given the first “die” d1= 4” d1 d2 Find Prob of Event A: “d2= 4” 1 P(d2=4| d1= 4)=? 2 S S 3 (4,1) ˆ 4 1 4 (4,2) ˆ S P ( S ) = P( d1 = 4) = ; P( 4,4) = (4,3) 16 16 d2 (4,4) A 1 4 P(4,4) 1 P (d 2 = 4 | d1 = 4) = = 16 = ˆ P( S ) 4 4 3 ˆ S Reduced 16 2 Sample space 1 d1 1 2 3 4 83Here are two examples illustrating conditional probability.The first involves a series of three coin flips and a tree shows all possible outcomes for the original space S.The reduced set of outcomes conditions on the statement “ 1st draw is a head (red circle)” and S-cap onlytakes the upper branch of the tree and leads to a reduced set of outcomes. The conditional probability iscomputed either by considering outcomes in this conditioning space S-cap or by computing the probabilityfor S (the whole tree) and then renormalizing by the probability for S-cap ( upper branch). The second example involves the throw of a pair 4-sided dice and asks for the probability that d2 =4 giventhat d1=4, P(d2 =4 | d1 =4 ). The answer is obtained directly from the definition of conditional probabilityand is illustrated using a tree and a coordinate representation of the dice sample space with a Venn diagramoverlay for the event (d1, d2) = (4,4) (green) and the subspace S-cap {d1=4} (red rectangle). 83
  18. 18. Probability of Winning in the “Game of Craps” Rules for the “Game of Craps” First Throw - dice sum=(d1+d2) Subsequent Throws - dice sum=(d1+d2) 2, 3, 12 - “Lose” (L) “Point” - “Win” (W) 7, 11 - “Win” (W) 7 “Lose” (L) Other (O) - first time defines your “Point” = “5” say Other (O) “Throw Again” Thr#1 2 L Thr#2 Thr#3 Thr#4 4 S=d1+d2 #Ways #Prob 3 L 36 5 2, 12 1 1/36 4 W 4 6 5 Point L 3, 11 2 2/36 36 7 36 5 W 6 26 4 P Start O 6 o 4, 10 3 3/36 7 W 7 L 36 36 36 5 W i 8 26 6 n 5, 9 4 4/36 O 9 7 L t 36 36 s 6, 8 5 5/36 10 26 O 11 7 6 6/36 W 36 12 L   4  1  2 2 3 4 4  26  4  26  4  26  P (W | 5) = +  +   +   + =  = 36 36  36  36  36  36  36  36  1 − 26  5    36  P(W ) = P(7) + P(11) + ∑ P(W | Point )P(Point ) Points 6 2   = + + 2  P(W | 4) P (4) + P(W | 5) P (5) + P (W | 6) P(6) = .4929 36 36  1/ 3  3 / 36 2/5 4 / 36 5 / 11 5 / 36   85 INDEXHere we compute the probability of winning the game of craps previously described by the rules for the 1stand subsequent throws given in the box and illustrated by the tree. Since there are 36 equally likelyoutcomes the #ways for the two dice summing to either 2 or 12 is obviously 1/36, for 3 or 11 it is 2/36, andthe remaining sums of two dice can be read directly off the sum axis coordinate representation and aredisplayed in the table on the right.We have labeled the partial tree “given the point 5” by their conditional probabilities derived from the table.The probability for the three outcomes W(“5”), L (“7”), “Other (not “5 or 7”) can be read off the table asP(5)= 4/36, P(7)=6/36, P(Other)= 1-(4+6)/36 =26/36. Note that these are actually conditional probabilities;but since the throws are independent the conditionals are the same as the a prioris as taken from the table.The P(W|5) is obtained by summing all paths that lead to a win on this “infinite tree”. Thus the 2nd throwyields W with probability 4/36 and the 3rd throw yields W with probability P(5|Other)P(5)=(26/36)(4/36),and the 4th throw yields W with probability P(5|Other,Other)P(5)=(26/36)2 (4/36), ... leading to an infinitegeometric series which sums to (4/36)*1/(1-26/36)=2/5.The total probability of winning is the sum of winning on the 1st throw (“7” or “11”) plus winning on thesubsequent throws for each possible “point.” The infinite sum for the other points is obtained in a similarmanner to that for “5” and (taking points by pairs in the table leads to the factor of two) the final result isshown to be .4929, i.e., a 49.3% chance of winning! 85
  19. 19. Visualization of Joint, Conditional, & Total Probability Binary Comm Signal - 2 Levels {0,1} Binary Decision - {R0, R1}={(“0” rcvd , “1” rcvd} x = 0,1 Joint Probability (Symmetric) 0 1 sent P(0,R0) = P(R0,0) ovly R1 “0” sent & R0 (“0” rcvd ) & y =R0 ,R1 R0 rcvd R0 (“0” rcvd ) “0” sent Conditional Probability 0R1 (Non-Symmetric) R0 ,R1 1R1 Joint P(0|R0) ∫ P(R0|0) 0R0 1R0 “0” sent given R0 (“0” rcvd ) x = 0 ,1 P(0) = P(0, R0 ) + P(0, R1 ) P(R0 ) = P(R0 ,0) + P(R0 ,1) R0 (“0” rcvd ) given “0” sent Total Probability P(0) Total Probability P(R0) sum up joint on R0,R1 sum across joint on 0,1 Conditional Probability P( R0 ,0) P( R0 ,0) P ( R0 | 0) ≡ = Requires Total Probability P ( 0) P( R0 ,0) + P( R0 ,1) Re-normalize Joint Probability P(0), P(R0), etc. P( R0 ,0) P ( R0 ,0) P (0 | R0 ) ≡ = P ( R0 ) P ( R0 ,0) + P ( R0 ,1) 88 INDEXAnother way to visualize the communication channel is in terms of an overlay of a Signal Plane divided(equally) into “0”s and “1”s and a Detection Plane which characterizes how the “0”s and “1”s are detectedand is structured as shown so that when we overlay the two planes we obtain an Outcome Plane with fourdistinct regions whose areas represent probabilities of the four product (joint) states { 0R0, 0R1, 1R0, 1R1}(similar to the tree outputs).In this representation the total probability of a “0” P(0) can be thought of as decomposed into two partssummed vertically over the “0”-half of the bottom plane shown by the break arrow P(0) = P(0,R0) + P(0,R1)[Note: summing on the “1”-half of the bottom plane yields P(1) = P(1,R0) + P(1,R1).]Similarly the total probability P(R0) can be thought of as decomposed into two parts summed horizontallyover the “R0”-portion of the bottom plane shown by the break arrow P(R0) = P(R0,0) + P(R0,1); similarlywe have P(R1) = P(R1,0) + P(R1,1).The Total Probability of a given state is obtained by performing such sums over all joint states. 88
  20. 20. Log-Odds Ratio - Add & Subtract Measurement Information Note: Revisit Binary Comm Channel P( R0 | 0) = .95 P ( R1 | 1) = .90 P(0)=.5 E = “1” P( R1 | 0) = .05 P ( R0 | 1) = .10 P(1)=.5 Ec = “0” Relation between  P (1 | R1 )  P (1 | R1 ) e L1 L1 ≡ ln 1 − P(1 | R )  ⇒ e = 1 − P(1 | R ) ⇒  L1 P(1 | R1 ) = L1 and P(1|R1)  1  1 1 + e L1  P(1 | R1 )   P (1)   P ( R1 | 1)   P(1)   P( R1 | 1)  L1 ≡ ln 1 − P(1 | R )  = ln 1 − P(1)  + ln 1 − P( R | 1)  = ln P(0)  + ln P ( R | 0)            1     1     1  ≡ L0 ≡ ∆L1  P( R1 | 1)  Additive Meas Updates for L Lnew = Lold + ∆LR1  P (1)   P(0)  ; ∆LR1 = ln P( R | 0)  Lold = ln       1  Updates Meas#1: R1 Meas#2: R0 Alternate Meas#2: R1  .5   P( R0 | 1)   .10   P( R1 |1)  Lold = ln  = 0 ∆LR0 = ln  .90   .5   P( R | 0)  = ln .95     ∆LR1 = ln   = ln    0   P( R1 | 0)   .05   .9  = −2.25129 ∆LR1 = ln  = +2.8903  .05  Lnew = Lold + ∆LR0 Lnew = Lold + ∆LR1 = 2.8903 = 2.8903 + (−2.25129) = .63901 = 2.8903 + 2.8903 = 5.7806 Lnew = 0 + 2.8903 e 2.8903 e.63901 e 5.7806 P(1 | R1 ) = = .947 P(1 | R1 R0 ) = = .655 P (1 | R1 R0 ) = = .997 1 + e 2.8903 1 + e.63901 1 + e 5.7806 96 INDEXRevisiting the binary communication channel we now compute updates using the log odds ratio which areadditive updates. The update equation simply starts from the initial log odds ratio which isLold=ln[P(1)/P(1c)] =ln(.5/.5)=0 for the communication channel. There are two measurement types R1 andR0 and each adds an increment ∆L determined by its measurement statistics, viz.,R1: ∆LR1 =ln[(P(R1|1)/P(R1|1c)]=ln(.90/.05) = +2.8903 (positive “confirming”)R0: ∆LR0 = ln[(P(R0|1)/P(R0|1c)]=ln(.10/.95)= -2.25129. (negative “refuting”)The table illustrates how easy it is to accumulate the results of two measurements R1 followed by R0 by justadding the two ∆Ls to obtainLnew= 0+2.8903-2.25129=.63901,or alternately R1 followed by R1 to obtainLnew=0+2.8903+2.8903=5.7806.These log odds ratios are converted to actual probabilities by computing P= eLnew / (1+ eLnew ) yielding .655and .997 for the above two cases.If we want to find the number of R1 measurements needed to give .99999 probability of “1” we need onlyconvert .99999 to an L =ln[(.99999)/(1-.99999)] =11.51 and divide the result by 2.8903 to find 3.98 so that4 R1 measurements are sufficient. 96
  21. 21. Discrete Random Variables (RV) –Key Concepts • Discrete RVs: A series of measurements of random events • Characteristics: “Moments:” Mean and Std Deviation • Prob Mass Fcn: (PMF), Joint, Marginal, Conditional PMFs • Cumulative Distr Fcn: (CDF) i) Btwn 0 and 1, ii) Non-decreasing • Independence of two RVs • Transformations - Derived RVs • Expected Values (for given PMF) • Relationships Btwn two RVs: Correlations • Common PMFs Table • Applications of Common PMFs • Sums & Convolution: Polynomial Multiplication • Generating Function: Concept & Examples 122 INDEXThis slide gives a glossary of some of the key concepts involving random variables (RVs) which we shalldiscuss in detail in this section. Physical phenomena are always subject to some random components sothat RVs must appear in any realistic model and hence their statistical properties provide a framework foranalysis of multiple experiments using the same model. These concepts provide the rich environment thatallows analysis of complex random systems with several RVs by defining the distributions associated withtheir sums and transformations of these distributions inherent in the mathematical equations that are used tomodel the system.At any instant, a RV takes on a single random value and represents one sample from the underlying RVdistribution defined by its probability mass function (PMF). Often we need to know the probability for somerange of values of a RV and this is found by summing the individual probability values of the PMF; thus acumulative distribution function (CDF) is defined to handle such sums. The CDF formally characterizes thediscrete RV in terms of a quasi-continuous function that ranges between [0,1] and which has a uniqueinverse.Distributions can also be characterized by single numbers rather than PMFs or CDFs and this leads toconcepts of mean values, standard deviations, correlations between pairs of RVs and expected values.There are a number of fundamental PMFs used to describe physical phenomena and these common PMFswill be compared and illustrated through examples. Finally, the relationship between the sum of two RVsand the concept of convolution and the generating function for RVs will be discussed. 122

×