Fu Jen University Probability Review Document

Bayesian Networks
Unit 1
Probability Review
Wang, Yuan-Kai, 王元凱
ykwang@mails.fju.edu.tw
http://www.ykwang.tw

Department of Electrical Engineering, Fu Jen Univ.
輔仁大學電機工程系

2006~2011

Reference this document as:
Wang, Yuan-Kai, “Probability Review," Lecture Notes of Wang, Yuan-Kai,
Fu Jen University, Taiwan, 2008.
Fu Jen University Department of Electronic Engineering Yuan-Kai Wang Copyright

王元凱 Unit - Probability Review p. 2

Goal of this Unit
 Review basic concepts of
probability in terms of
 Image processing terminologies

Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright


Related Units
 Next units
 Statistics Review
 Uncertainty Inference (Discrete)
 Uncertainty Inference (Continuous)



Self-Study
 Probability Theory textbook
 Artificial Intelligence: a modern
approach
 Russell & Norvig, 2nd, Prentice Hall,
2003. pp.462~474,
 Chapter 13, Sec. 13.1~13.3



Contents
1. Probability ......................................... 6
2. Random Variable .............................. 9
3. Probability Distribution .................... 19
4. Joint Distribution .............................. 26
5. Conditional Probability .................... 46
6. Bayes Theorem ................................. 59
7. Summary …………………………….. 64
8. References ………………………….. 67



1. Probability
 We write P(A) as “the fraction of
possible worlds in which A is true”
Event space
of all possible
worlds Worlds in P(A) = Area of
which A is
true reddish oval

Its area is 1
Worlds in which A is False



The Axioms of Probability
0  P(A)  1
 P(True) = 1
 P(False) = 0
 P(A  B) = P(A) + P(B) - P(A  B)
Where do these axioms come
from?
Were they “discovered”?
Answers coming up later.


Theorems from the Axioms
0  P(A)  1,
P(True) = 1,
P(False) = 0
P(A  B) = P(A) + P(B) - P(A  B)
From these we can prove:
P(not A) = P(~A) = 1-P(A)
P(A) = P(A  B) + P(A  ~B)
How?


2. Random Variable
A variable with randomness
probability, degree of belief
An example
Rain : a random variable
Its possible values: true, false
Its randomness
P(Rain=true)=0.8,
P(Rain=false)=0.2


Types of Random Variable
Boolean random variable
Rain : true, false
Discrete random variable
(Multivalued R.V.)
Rain: cloudy, sunny, drizzle,
drench
Continuous random variable
Rain: rainfall in millimeter



Discrete R.V. (1/4)
Let A=Rain
A can take on more than 2 values
A is a random variable with arity k
if it can take on exactly one value
out of {v1,v2, .. vk}
Thus
P ( A  vi  A  v j )  0 if i  j
P ( A  v1  A  v2    A  vk )  1



Discrete R.V. (2/4)
i
P( A  v1  A  v2    A  vi )   P( A  v j )
j 1
Prove it?
Using the axioms of probability…
0  P(A)  1, P(True) = 1, P(False) = 0
P(A  B) = P(A) + P(B) - P(A  B)
 And assume that A obeys

P ( A  vi  A  v j )  0 if i  j
P( A  v1  A  v2    A  vk )  1


Discrete R.V. (3/4)
i
P( A  v1  A  v2    A  vi )   P( A  v j )
k j 1

 P( A  v )  1
j 1
j

0  P(A)  1, P(True) = 1, P(False) = 0
P(A or B) = P(A) + P(B) - P(A and B)
 And assuming that A obeys…
P( A  vi  A  v j )  0 if i  j
P( A  v1  A  v2    A  vk )  1


Discrete R.V. (4/4)
i
P( B  [ A  v1  A  v2    A  vi ])   P( B  A  v j )
j 1
k
P( B)   P( B  A  v j )
j 1
0  P(A)  1, P(True) = 1, P(False) = 0
P(A or B) = P(A) + P(B) - P(A and B)
 And assuming that A obeys…
P( A  vi  A  v j )  0 if i  j
P( A  v1  A  v2    A  vk )  1


Random Vector
 A random vector is a vector of
random variables
 X = (X1, X2, ..., XN)
 X1 ... XN are random variables



Example in Image Processing
 For a gray-level image,
random variable is “gray level”
• Random variable X
(Gray level) has n
possible values
{x1, x2, ..., xn}, n=256
• N random data
x1, x2, .., xN of X,
N=Width*Height
Discrete v.s. continuous ?


 For a color image,
random vector is (r,g,b)



Example in Computer Vision
 Random vector is a vector of
“features”
 Face recognition



3. Probability Distribution
 Probability distribution is a set of
probabilities
 Boolean R.V.
 P(A): P(A=true)=0.2, P(A=false)=0.8
 Discrete R.V.
 P(A): P(A=v1)=0.2, P(A=v2)=0.4,
P(A=v3)=0.4, P(A=v4)=0.1,
 Continuous R.V.
 Usually we plot the probability
distribution as a function
(Probability Distribution Function,
pdf)


Probability Distribution (1/3)
Continuous R.V.
0.8
0.6
0.4
0.2
0 Rain (mm)
0 100 200 300 400 500 600

• P(Rain) is a probability
density function (pdf)
• P(Rain=200) is a probability


Probability Distribution(2/3)
Boolean R.V.
P(Rain) is a probability distribution
P(Rain=false) is a probability
P(Rain)
0.8

0.2
Rain
false true



Probability Distribution(3/3)
Discrete R.V.
P(Rain) is a probability distribution
P(Rain=drizzle) is a probability
P(Rain)
0.8

0.2
Rain
drench
cloudy

drizzle
sunny



Common Probability
Distributions
Normal distribution (Gaussian)
Parameters: , 
Discussed in Section 3
Uniform distribution
Exponential distribution
Poisson distribution
...



The Uniform Distribution
1 w
 w if | x |
P(x) p( x)   2
w
 0 if | x |
 2

1/w

X
-w/2 0 w/2
2
w
E[ X ]  0 Var[ X ] 
12



• Random variable X (Gray level) has n possible
values {x1, x2, ..., xn}, n=256
• Distribution P(xi) of the image
•Is not a usual p.d.f.


4. Joint Distribution
 When there are more than one
random variables: X, Y
 We want to know "the probabilities
of both random variables"
 Joint probability distribution
 P(X  Y)
 P(X  Y  ...)



An Example
A set of random variables for the
typhoon world
 Rain, Wind, Speed, ...
 Every random variable
Describes one facet of typhoon
Has a probability distribution
 A typhoon is a situation of joint
probability
P(Rain=200mm  Wind=South 
Speed=100km/hr) = 0.4



P(X,Y) with X Y are Discrete
 P(X,Y) : (M x N) entries
y1 P(X=x1,Y=y1) P(X=x2,Y=y1) P(X=x3,Y=y1)
Y
y2 P(X=x1,Y=y2) P(X=x2,Y=y2) P(X=x3,Y=y2)
x1 x2 x3
X

y1 0.2 0.1 0.1
Y
y2 0.1 0.2 0.3
x1 x2 x3
X


Atomic Event
Atomic event is a
Combination of values of all random
variables
Situation (state) of typhoon
(statistical world)
For the typhoon example
(Rain=200mm  Wind=South 
Speed=100km/hr) is an atomic event
(Rain=200mm  Wind=South) is not


Joint Probability Distribution
 P(Rain=200mm  Wind=South 
Speed=100km/hr) is a joint probability
 P(Rain  Wind  Speed) is a probability
of all joint probabilities
 Joint probability function
A set of random variables can have
mixed types
 Ex: Rain: Boolean, Wind: Discrete,
Speed: Continuous
 We will consider all-discrete case


Prior and Joint Probability
(Math)
Let {X1,...,Xn} be a set of
random variables
P(Xi) is a probability function
P(Xi=xi) is a prior probability
P(X1  X2    Xn) is a joint
probability function
P(X1=x1  X2=x2    Xn=xn) is
a joint probability


For Boolean & Discrete R.V.
Let X be a single R.V.
P(X) is a vector
Boolean R.V.
Rain: true, false
P(Rain) = <0.72, 0.28>
Discrete R.V.
Rain: cloudy, sunny, drizzle, drench
P(Rain) = <0.72, 0.1, 0.08, 0.1>
Normalized, i.e. sums to 1


Full Joint Distribution (1/3)
For a set of random variables
{ X1 , X2 ,  , Xn }
X1  X2    Xn are atomic events
P(X1  X2    Xn) is
A full joint probability distribution
(FJD)
A table of all joint prob. of all atomic
events, if {X1,  , Xn} are discrete


Ex: X1: Rain, X2: Wind
X1: drizzle, drench, cloudy,
X2: strong, weak
X1  X2 are atomic events
P(X1  X2) is a 3x2 matrix of values
Rain Drizzle Drench Cloudy
Wind
Strong 0.15 0.12 0.06
Weak 0.55 0.08 0.04
P(X1=drizzle  X2=strong) = 0.15, ...


All questions about probability
of joint events can be answered
by the table
P(Wind=Strong),
P(Rain=Drizzle  Wind=Strong),
...



An Example with 3 Boolean
R.V. (1/2)
Boolean variables A, B, C
1. Make a truth table
listing all A B C
combinations of 0 0 0
values of your 0 0 1
variables 0 1 0
• if there are M 0 1 1
Boolean variables 1 0 0
then the table will 1 0 1
1 1 0
have 2M rows
1 1 1


An Example with 3 Boolean
R.V. (2/2)
2. For each A B C Prob
combination of 0 0 0 0.30
values, say how 0 0 1 0.05
probable it is 0 1 0 0.10
0 1 1 0.05
A 0.05 0.10 0.05 1 0 0 0.05
0.10 1 0 1 0.10
0.25
0.05 C 1 1 0 0.25
0.10 1 1 1 0.10
0.30 B
Sum 1.0


Using the FJD
• Once you have the FJD
• You can ask for the probability of
any logical expression: Inference

P( E )   P(row )
rows matching E



Examples of Inference (1/3)
P(Poor Male) = 0.4654 P( E )   P(row )
rows matching E



P(Poor) = 0.7604 P( E )   P(row )
rows matching E




P ( E1  E2 )
 P(row )
P ( E1 | E2 )  
rows matching E1 and E2

P ( E2 )  P(row )
rows matching E2

P(Male | Poor) = 0.4654 / 0.7604 = 0.612


Inference is a Big Deal
I’ve got this evidence
What’s the chance that this
conclusion is true?
I’ve got a sore neck: how likely am
I to have meningitis?
I see my lights are out and it’s
9pm. What’s the chance my
spouse is already asleep?



Where do Full Joint
Distributions Come from?
(1/2)
 Idea One: Expert Humans
 Idea Two: Learn them from data!



Where do FJD Come from?
(2/2)
Build a JD table for your Then fill in each row with
attributes in which the
probabilities are unspecified ˆ (row )  records matching row
P
total number of records
A B C Prob
0 0 0 ? A B C Prob
0 0 1 ? 0 0 0 0.30
0 1 0 ? 0 0 1 0.05
0 1 1 ? 0 1 0 0.10
1 0 0 ? 0 1 1 0.05
1 0 1 ? 1 0 0 0.05
1 1 0 ? 1 0 1 0.10
1 1 1 ? 1 1 0 0.25
1 1 1 0.10
Fraction of all records in which
A and B are True but C is False


Example of Learning a FJD
 This Joint was obtained by
learning from three attributes in
the UCI “Adult” Census Database
[Kohavi 1995]



5. Conditional Probability
 P(A|B) = Fraction of worlds in which B
is true that also have A true
H = “Have a headache”
F
F = “Coming down with Flu”
H
P(H) = 1/10
P(F) = 1/40
P(H|F) = 1/2
“Headaches are rare and flu is rarer, but if
you’re coming down with ‘flu there’s a 50-
50 chance you’ll have a headache”


Formula
H = “Have a headache”
F
F = “Coming down with Flu”
H

P(H) = 1/10
P(F) = 1/40
P(H|F) = 1/2
P( H  F )
P( H | F ) 
P( F )
P(H|F) v.s. P(HF)


Conditional, Joint, Prior
 Relationship among conditional,
joint, and prior probabilities
P(X1  X2)P(X2)
P(X1) P( X 1  X 2 )
P( X 1 | X 2 ) 
P( X 2 )
P( X1  X 2 )  P( X1 | X 2 )P( X 2 )



Posterior v.s. Prior
Probabilities
P(Cavity|Toothache)
Conditional probability
Posterior probability
(after the fact/evidence)
P(Cavity)
Prior probability (the fact)



An Example (1/2)
For a dental diagnosis
Let {Cavity,Toothache} be a set
of Boolean random variables
Denotations for Boolean R.V.
P(Cavity=true) = P(cavity)
P(Cavity=false) = P(cavity)



An Example (2/2)
The full joint probability
distribution P(Toothache  Cavity)
toothache toothache
cavity 0.04 0.06
cavity 0.01 0.89
P(cavity  toothache)  0.04  0.01  0.06  0.11
P(cavity| toothache )
P(cavity toothache ) 0.04
   0.80
P(toothache ) 0.04  0.01


Conditional Probability (Math)
P(X1=x1i| X2 =x2j) is a conditional
probability
P(X1 | X2) is a conditional
distribution function
All P(X1=x1i| X2 =x2j) for all possible i, j
For Boolean & discrete R.V.s,
conditional distribution function is a
conditional probability table
Conditional distribution of continuous
R.V. will not used in our discussions


Conditional Probability Table
(CPT)
For the dental diagnosis problem,
Toothache & Cavity are Boolean R.V.s
P(Toothache|Cavity) is a CPT
Cavity P(toothache|Cavity) P(toothache|Cavity)
T 0.90 0.1
F 0.05 0.95
Cavity P(toothache|Cavity)
T 0.90
F 0.05


CPT v.s. FJD
toothache toothache
cavity 0.04 0.06
cavity 0.01 0.89
Sum of all atomic events = 1
Cavity P(toothache|Cavity) P(toothache|Cavity)
T 0.90 0.1
F 0.05 0.95
Sum of a row = 1



More Than 2 Random
Variables in
Conditional Probability
Joint conditional probability
P( X1  X 2 | X 3  X 4 )
P( X1  X 2  X 3  X 4 )

P( X 3  X 4 )
 P( X1 | X 2  X 3  X 4 )P( X 2 | X 3  X 4 )



Chain Rule
• Joint probability can be calculated
from a chain of conditional
probability P( X | X )  P( X 1  X 2 )
1 2
P( X 2 )
P( X 1  X 2 )  P( X 1 | X 2 ) P( X 2 )
P( X1  X 2  X 3  X 4 )
 P( X1  ( X 2  X 3  X 4 ))
 P( X1 | X 2  X 3  X 4 )P( X 2  X 3  X 4 )
 P( X1 | X 2  X 3  X 4 )P( X 2 | X 3  X 4 )P( X 3 | X 4 )P( X 4 )
n
P( X1  X 2  X n )   P( X i | X i 1  X n )
i 1


Useful Easy-to-prove Facts

P( A | B)P(A | B )  1
nA

 P( A  v
k 1
k | B)  1



Where Are We?
 We have recalled the fundamentals
of probability
 We know what JDs are and how to
use them
 Next two sections
 Bayes rule
 Gaussian distribution



6. Bayes Theorem
P ( A | B ) P ( A)

P ( B | A) P ( B )
P( A  B)  P( A | B) P( B)
 P ( B  A)  P ( B | A) P ( A)
 P ( A | B ) P ( B )  P ( B | A) P ( A)
Bayes, Thomas (1763) An essay
towards solving a problem in the
doctrine of chances. Philosophical
Transactions of the Royal Society of
London, 53:370-418



More General Forms
of Bayes Rule
If A is a Boolean R.V.
P( A  B) P( A  B)
P( A |B)  
P( B) P ( A  B )  P ( A  B )
If A is a discrete R.V.
P( A  vi  B) P(B | A  vi )P( A  vi )
P( A  vi |B)  nA
 nA

P(A  v  B) P(B | A  v )P(A  v )
k 1
k
k 1
k k



Example –
Bayesian Detection (1/2)
 2-class recognition
 Ex.: Skin color detection
 Let
 A be skin color pixel
 A be non-skin color pixel
 c be the color (R,G,B) of a pixel
 We can get : P(c|A), P(c|A)
 For a pixel in a new image, is it a skin
color pixel?
P(c | A) P( A) P(c | A) P(A)
P( A | c)  > P ( A | c ) 
P (c ) P (c )


Example –
Bayesian Detection (2/2)
 2-class recognition
 Ex.: Background detection
 Let
 A be background pixel
 A be non-background pixel
 c be the color (R,G,B) of a pixel
 We can get : P(c|A), P(c|A)
 For a pixel in a new image, is it a
background pixel?
P(c | A) P( A) P(c | A) P(A)
P( A | c)  > P ( A | c ) 
P (c ) P (c )


7. Summary
Continuous variable Discrete variable
(X has M values, Y has N values)
P(X) Function of one variable M vector
P(X=x) Scalar* Scalar
P(X,Y) Function of two variables MxN matrix
P(X|Y) Function of two variables MxN matrix
P(X|Y=y) Function of one variable M vector
P(X=x|Y) Function of one variable N vector
P(X=x|Y=y) Scalar* Scalar


Ex (1/2)



Ex (2/2)



8. Reference
 Artificial Intelligence: a modern
approach
 Russell & Norvig, 2nd, Prentice Hall, 2003.
 Sections 13.1~13.3, pp.462~474.
 統計學的世界
 墨爾著，鄭惟厚譯
 天下文化，2002
 深入淺出統計學
 D. Grifiths, 楊仁和譯，2009
 O’ Reilly


Fu Jen University Probability Review Document

Recommended

Recommended

More Related Content

More from IEEE International Conference on Intelligent Information Hiding and Multimedia Signal Processing

More from IEEE International Conference on Intelligent Information Hiding and Multimedia Signal Processing (11)

Recently uploaded

Recently uploaded (20)

Fu Jen University Probability Review Document