AI 8 | Probability Basics, Bayes' Rule, Probability Distribution

Quantifying Uncertainty
CSI 341 Mohammad Imam Hossain | Lecturer, Dept. of CSE | UIU

Uncertainty
Agents may need to handle uncertainty, whether due to
▸Partial observability – partial sensor information what’s my current state!
▸Nondeterminism – what will be the state after performing a sequence of action!
▸Or a combination of both.
Our previous agents (problem solving, logical) handle uncertainty by
▸Keeping track of a belief state - a representation of the set of all possible world states that it might be in.
▸Generating a contingency plan that handles every possible eventuality that its sensors may report during execution.
Problems:
▸Impossibly large and complex belief-state representation due to partial observability
▸Contingent plan can grow arbitrarily large and unlikely contingencies
▸How to compare the merits of one plan with another if no guarantee to reach the goal can be assured.
2
Mohammad Imam Hossain | Lecturer, Dept. of CSE | UIU

Uncertainty >> Diagnosis problem
Problem: A dental patient’s toothache
Way 1:
Toothache ⇒ Cavity
But there exists a lot more reasons for Toothache … …
Toothache ⇒ Cavity v GumProblem v Abscess … …
Way 2(reverse):
Cavity ⇒ Toothache
But not all cavities cause pain.
Diagnosis Problem Complexities:
▸Laziness
▸Theoretical ignorance
▸Practical ignorance
3

Way 1:
Way 2(reverse):
Complexities:
▸Laziness
4
Can we reach a
logical decision from
either direction!!!!

Way 1:
Way 2(reverse):
Complexities:
▸Laziness
5
Can we reach a
logical decision from
either direction!!!!
An agent can at best
provide a degree of
belief that is
probability

Decision Theory
Logical Agent
– believes each sentence to be true/false/has no opinion.
Probabilistic Agent
– determines the numerical degree of belief for each sentence between 0 and 1.
– provides a way of summarizing the uncertainty that comes from our laziness and ignorance.
– probability statements are made with respect to a knowledge state, not with respect to the real world.
Decision Theory
– An agent is rational if and only if it chooses the action that yields the highest expected utility, averaged over
all the possible outcomes of the action.
Decision Theory = Probability theory + Utility theory
Probability Theory – agent can make probabilistic predictions of action outcomes
Utility Theory – select the action with highest expected utility.
6

Probability Theory >> Sample Space
Sample space(𝛀) :
The set of all possible worlds(𝝎) is called the sample space.
The possible worlds i.e. 𝝎’s are
 Mutually exclusive – two possible worlds can’t both be the case
 Exhaustive – one possible world must be the case
For example,
7
Experiment Sample Space
Flipping a coin {H, T}
Tossing a die {1, 2, 3, 4, 5, 6}
Flipping a coin and then flipping it a second time if a head occurs. If a tail
occurs on the first flip, then a die is tossed once.
{HH, HT, T1, T2, T3, T4, T5, T6}
Three items are selected at random from a manufacturing process. Each item
is inspected and classified defective, D, or, non-defective, N
{DDD, DDN, DND, DNN,
NDD, NDN, NND, NNN}

Probability Theory >> Event
Proposition/Event(𝝓) :
A subset of possible worlds that holds the proposition is called a proposition/event.
Sample space for tossing two dies,
Ω = { (1, 1), (1, 2), (1, 3), (1,4), (1, 5), (1,6),
(2, 1), (2, 2), (2, 3), (2, 4), (2, 5), (2, 6),
(3, 1), (3, 2), (3, 3), (3, 4), (3, 5), (3, 6),
(4, 1), (4, 2), (4, 3), (4, 4), (4, 5), (4, 6),
(5, 1), (5, 2), (5, 3), (5, 4), (5, 5), (5, 6),
(6, 1), (6, 2), (6, 3), (6, 4), (6, 5), (6, 6) }
Then the proposition that two dice add up to 11,
𝜙 = { (5, 6), (6, 5) }
8
Ω
𝜙
𝜙′

Probability Theory >> Event Types
▹ The subset of all possible worlds of Ω that are not in 𝜙 is called complement event 𝜙’
▹ Intersection of two events 𝜙1 and 𝜙2 is denoted as 𝜙1 ⋂ 𝜙2 , is the event containing all elements that are common to 𝜙1
and 𝜙2
▹ Union of two events 𝜙1 and 𝜙2 is denoted as 𝜙1 ∪ 𝜙2 , is the event containing all elements that belong to 𝜙1 or 𝜙2 or both.
▹ Two events 𝜙1 and 𝜙2 are mutually exclusive, or disjoint, if 𝜙1 ⋂ 𝜙2 = 0,
that is no elements in common.
For example,
A U C = regions 1, 2, 3, 4, 5, and 7
B’ ⋂ A = regions 4 and 7
A ⋂ B ⋂ C = region 1
(A U B) ⋂ C’ = region 2, 6, 7
9
𝜙

Probability Theory >> Event Probability
To every point in the sample space we assign a probability/weights such that the sum of all probabilities is 1.
So,
▸0 ≤ P(𝜔) ≤ 1 for every 𝜔
▸σ 𝜔∈Ω 𝑃(𝜔) = 1
Event Probability:
- The probability of an event/proposition 𝜙 is the sum of the weights of all possible worlds in 𝜙.
- For any event/proposition 𝜙, P(𝜙) = σ 𝜔∈𝜙 𝑃(𝜔)
Example 1:
If a coin is tossed twice, then what is the probability that at least 1 head occurs?
Soln:
Sample Space, 𝛺 ={ HH, HT, TH, TT } and proposition, 𝜙 = { HH, HT, TH }
if each possible worlds are equally likely to occur let, w
then w+w+w+w=1 ⟹ w=1/4
P(𝜙) = ¼ + ¼ + ¼ = ¾
10

Probability Theory >> Event Probability
Example 2:
A die is loaded in such a way that an event number is twice as likely to occur as an odd number. If 𝜙 is the event
that a number less than 4 occurs on a single toss of the die, find P(𝜙)
Soln:
Sample Space, Ω ={ 1, 2, 3, 4, 5, 6 } and proposition, 𝜙 = { 1, 2, 3 }
if the probability of each odd number is w then the probability of each even number is 2w
So, w+2w+w+2w+w+2w = 1 ⟹ w = 1/9
Odd number probability w = 1/9 and even number probability 2w = 2/9
P(𝜙) = 1/9 + 2/9 + 1/9 = 4/9
11

Probability Theory >> Additive Rule
If a and b are two events, then
P(a U b) = P(a) + P(b) – P(a ∩ b)
For 3 events a, b, c
P(a ∪ b ∪ c) = P(a) + P(b) + P(c) – P(a ∩ b) – P(b ∩ c) – P(c ∩ a) + P( a ∩ b ∩ c )
For mutually exclusive event P(a ∩ b) = 0,
P(a ∪ b) = P(a) + P(b)
For more than 2 events,
P(a1 ∪ a2 ∪ … … ∪ an ) = P(a1) + P(a2) + … … + P(an)
12

Example 1:
John is going to graduate from an industrial engineering department in a university by the end of the semester.
After being interviewed at two companies he likes, he assesses that his probability of getting an offer from company A is 0.8,
and his probability of getting an offer from company B is 0.6. If he believes that the probability that he will get offers from
both companies is 0.5, what is the probability that he will get at least one offer from these two companies?
Soln :
Event a = getting job offer from company 1
Event b = getting job offer from company 2
So,
P(a U b) = P(a) + P(b) – P(a ∩ b) = 0.8 + 0.6 – 0.5 = 0.9
13

Example 2:
If the probabilities are, respectively, 0.09, 0.15, 0.21, and 0.23 that a person purchasing a new automobile will
choose the color green, white, red, or blue, what is the probability that a given buyer will purchase a new automobile that
comes in one of those colors?
Soln :
Event a = purchasing a green automobile
Event b = purchasing a white automobile
Event c = purchasing a red automobile
Event d = purchasing a blue automobile
So,
P(a U b U c U d) = P(a) + P(b) + P(c) + P(d) = 0.09 + 0.15 + 0.21 + 0.23 = 0.68
14

Probability Theory >> Complement Rule
If 𝜙 and 𝜙’ are complementary events then,
P(𝜙) + P(𝜙’) = 1
Example 1:
If the probabilities that an automobile mechanic will service 3, 4, 5, 6, 7, or 8 or more cars on any given workday
are, respectively, 0.12, 0.19, 0.28, 0.24, 0.10, and 0.07, what is the probability that he will service at least 5 cars on his next day
at work?
Soln :
Event a = servicing at least 5 cars
Event a’ = servicing less than 5 cars
Here,
P(a’) = 0.12 + 0.19 = 0.31
So P(a) = 1 – P(a’) = 1 – 0.31 = 0.69
15

Probability Theory >> Prior vs Posterior Probabilities
Two kinds of probabilities:
▸Unconditional/prior probabilities
- refer to the degrees of belief in propositions in the absence of any other information.
- the prior probability of an event a is represented as P(a)
- for example, P(cavity) = 0.2 meaning cavity is true with probability 0.2 when you have no other information.
▸Conditional/posterior probabilities
- refer to the degrees of belief in propositions with some evidence and in the absence of any further information.
- the posterior probability of an event b when it is known that event a has occurred is represented as P(b | a)
- for example, P(cavity | toothache) = 0.6 meaning whenever toothache is true and we have no other information
conclude that cavity is true with probability 0.6
- conditional probability can be defined in terms of prior probabilities as follow:
P(b | a) =
P(a ∩ b)
P(a)
16

Probability Theory >> Conditional Probability
Example 1:
The probability that a regularly scheduled flight departs on time is P(D) = 0.83; the probability that it arrives on
time is P(A) = 0.82; and the probability that it departs and arrives on time is P(D ∩ A) = 0.78
Find the probability that a plane
- arrives on time, given that it departed on time, and
- departed on time, given that it has arrived on time.
Soln:
P(A | D) =
𝑃 𝐷 ∩𝐴
𝑃(𝐷)
=
0.78
0.83
= 0.94
P(D | A) =
𝑃 𝐷 ∩𝐴
𝑃(𝐴)
=
0.78
0.82
= 0.95
17

Probability Theory >> Conditional Probability
Example 2:
Consider an industrial process in the textile industry in which strips of a particular type of cloth are being
produced. These strips can be defective in two ways, length and nature of texture. For the case of the latter, the process of
identification is very complicated. It is known from historical information on the process that 10% of strips fail the length
test, 5% fail the texture test, and only 0.8% fail both tests. If a strip is selected randomly from the process and a quick
measurement identifies it as failing the length test, what is the probability that it is texture defective?.
Soln:
P(T | L) =
𝑃 𝑇 ∩ 𝐿
𝑃(𝐿)
=
0.008
0.1
= 0.08
18

Probability Theory >> Independent Event
Product Rule:
P(a ∩ b) = P(b ∩ a) = P(a) P(b | a) = P(b) P(a | b)
P(a1 ∩ a2 ∩ … … ∩ ak ) = P(a1) P(a2 | a1) P(a3 | a2 ∩ a1) … … P(ak | a1 ∩ a2 ∩ … … ∩ ak-1)
Independent Event:
Two events a and b are independent if and only if P(b | a) = P(b) or, P(a | b) = P(a)
For two independent events a, b product rule becomes,
P(a ∩ b) = P(a) P(b)
P(a1 ∩ a2 ∩ … … ∩ ak ) = P(a1) P(a2) P(a3) … … P(ak)
19

Example 1:
A small town has one fire engine and one ambulance available for emergencies. The probability that the fire
engine is available when needed is 0.98, and the probability that the ambulance is available when called is 0.92. In the event
of an injury resulting from a burning building, find the probability that both the ambulance and the fire engine will be
available, assuming they operate independently.
Soln:
Event a = ambulance is available
Event b = fire engine is available
P(a ∩ b) = P(a) P(b) = 0.98 * 0.92 = 0.9016
20

Example 2:
Three cards are drawn in succession, without replacement, from an ordinary deck of playing cards. Find the
probability that the event A1 ∩ A2 ∩ A3 occurs, where A1 is the event that the first card is a red ace, A2 is the event that the
second card is a 10 or a jack, and A3 is the event that the third card is greater than 3 but less than 7.
Soln:
Event a = first card is a red ace
Event b = the second card is a 10 or, a jack
Event c = the third card is greater than 3 but less than 7
P(a ∩ b ∩ c) = P(a) P(b | a) P(c | a ∩ b) = (2/52) (8/51) (12/50) = 8/5525
21

Probability Theory >> Total Probability
If the events b1, b2, b3, … … , bk constitute a partition of the sample space such that P(bi) ≠ 0 for i=1, 2, 3, … … , k,
then for any event a of sample space,
P(a) = σ𝑖=1
𝑘
𝑃(𝑏𝑖 ∩ 𝑎) = σ𝑖=1
𝑘
𝑃 𝑏𝑖 𝑃 𝑎 𝑏𝑖)
22

Probability Theory >> Total Probability
Example 1:
In a certain assembly plant, three machines, B1, B2, and B3, make 30%, 45%, and 25%, respectively, of the
products. It is known from past experience that 2%, 3%, and 2% of the products made by each machine, respectively, are
defective. Now, suppose that a finished product is randomly selected. What is the probability that it is defective?
Soln:
Event a = product is defective
Event b = product made by machine B1
Event c = product made by machine B2
Event d = product made by machine B3
P(a) = P(a ∩ b) + P(a ∩ c) + P(a ∩ d) = P(b) P(a | b) + p(c) P(a | c) + P(d) P(a | d) = 0.3*0.02 + 0.45*0.03 + 0.25*0.02 = 0.0245
23

Probability Theory >> Bayes’ Rule
If the events b1, b2, … … , bk constitute a partition of the sample space such that P(bi) ≠ 0 for i = 1, 2, 3, … …, k
then for any event a in sample space that P(a) ≠ 0,
P(br | a) =
𝑃(𝑏𝑟 ∩ 𝑎)
𝑃(𝑎)
=
𝑃 𝑏 𝑟 𝑃 𝑎 𝑏 𝑟)
𝑃(𝑎)
This rule also expressed as,
P(cause | effect) =
𝑃 𝑒𝑓𝑓𝑒𝑐𝑡 𝑐𝑎𝑢𝑠𝑒) 𝑃(𝑐𝑎𝑢𝑠𝑒)
𝑃(𝑒𝑓𝑓𝑒𝑐𝑡)
This rule is very important for diagnosis problems cause there are many cases where we do have good probability estimates
for these other 3 numbers and need to compute the 4th.
For example, the doctors knows P(symptoms | disease) and want to derive a diagnosis P(disease | symptoms).
Example 1:
Meningitis causes stiff neck 50% of the time. The probability of a patient having meningitis(m) is 1/50000. The
probability of a patient having stiff neck(s) is 1/20. What is the probability of meningitis given stiff neck?
Soln:
Here, P(s | m) = 0.5, P(m) = 1/50000, P(s) = 1/20
So, P(m | s) =
𝑃 𝑠 𝑚) 𝑃(𝑚)
𝑃(𝑠)
= 0.0002
24

Probability Theory >> Bayes’ Rule
Example 2:
A manufacturing firm employs three analytical plans for the design and development of a particular product.
For cost reasons, all three are used at varying times. In fact, plans 1, 2, and 3 are used for 30%, 20%, and 50% of the products,
respectively. The defect rate is different for the three procedures as follows: P(D|P1) = 0.01, P(D|P2) = 0.03, P(D|P3) = 0.02,
where P(D|Pj) is the probability of a defective product, given plan j. If a random product was observed and found to be
defective, which plan was most likely used and thus responsible?
Soln:
P(p1 | D) =
𝑃 𝑃1 𝑃(𝐷|𝑃1)
𝑃 𝑃1 𝑃(𝐷|𝑃1)+𝑃 𝑃2 𝑃(𝐷|𝑃2)+𝑃 𝑃3 𝑃(𝐷|𝑃3)
= 0.158
P(p2 | D) =
= 0.316
P(p3 | D) =
= 0.526
Plan 3 is most likely as the probability is higher.
25

Probability Theory >> Bayes’ Rule - Practices
Problem 1 >>
A diagnostic test has a probability 0.95 of giving a positive result when applied to a person suffering from a
certain disease, and a probability 0.10 of giving a (false) positive when applied to a non-sufferer. It is estimated that 0.5% of
the population are sufferers. Suppose that the test is now administered to a person about whom we have no relevant
information relating to the disease (apart from the fact that he/she comes from this population). Calculate the following
probabilities:
(a) that the test result will be positive;
(b) that, given a positive result, the person is a sufferer;
(c) that, given a negative result, the person is a non-sufferer;
(d) that the person will be misclassified.
Answer: 0.10425, 0.0455, 0.9997, 0.09975
26

Problem 2 >>
There are two boxes containing coins. The first box contains 60 gold coins and 40 silver coins. The second box
contains 30 gold coins and 70 silver coins. One of the two boxes is randomly chosen (both boxes have probability 0.5 of
being chosen) and then a coin is picked up at random from the chosen box. If a silver coin is picked up, what is the
probability that it comes from the first box?
Answer: 4/11
27

Problem 3 >>
Alice has two coins in her pocket, a fair coin (head on one side and tail on the other side) and a two-headed
coin (head on both sides). She picks one at random from her pocket, tosses it and obtains head. What is the probability that
she flipped the fair coin?
Answer: 1/3
28

Probability Theory >> Random Variable
A random variable has a domain of possible values. Each value has an assigned probability between 0 and 1. Random
variable names start with Uppercase letter and values are all Lowercase. The values are:
▸Mutually Exclusive (disjoint) – only one of them are true
▸Complete – there is always one that is true
Discrete random variable – set of possible outcomes is countable.
For example, The random variable Weather has domain of possible values {sunny, rain, cloudy, snow}. The assigned
probabilities are as follow:
P(Weather=sunny) = 0.7
P(Weather=rain) = 0.2
P(Weather=cloudy) = 0.08
P(Weather=snow) = 0.02
Probability of all the possible values a random variable is represented as a vector of numbers and is called Probability
Distribution.
P(Weather) = 0.7, 0.2, 0.08, 0.02
29

Probability Theory >> Probability Distribution - Discrete
Discrete Probability Distribution:
- A vector of numbers representing the probabilities of all the possible values of a random variable.
- For example, P(Weather) = 0.6, 0.1, 0.29, 0.01 assuming predefined order 𝑠𝑢𝑛𝑛𝑦, 𝑟𝑎𝑖𝑛, 𝑐𝑙𝑜𝑢𝑑𝑦, 𝑠𝑛𝑜𝑤
- Tabular representation:
- If X represents a random variable, then for each possible outcome x of X,
෍
𝑥
P(X = x) = 1
30
w sunny rain cloudy Snow
P(Weather = w) 0.6 0.1 0.29 0.01

Probability Theory >> Probability Distribution - Conditional
Conditional Probability Distribution:
- P(X | Y) = vector of numbers representing the probabilities of P(X=xi | Y=yj) for each possible i, j
- Tabular representation of the conditional probability distribution of random variable A, given the evidence random
variables B and C
- Sum of the probabilities for each row is 1
31
B C P(A=+a|B,C) P(A= -a|B,C)
+b +c 0.95 0.05
+b -c 0.94 0.06
-b +c 0.29 0.71
-b -c 0.001 0.999

Probability Theory >> Probability Distribution - Joint
Joint Probability Distribution:
- Probability distributions of multiple random variables.
- Consider all possible combinations of values of the variables.
- Tabular representation of the joint distribution of two random variables Weather and Season is as follow:
P(Weather, Season)
- If X and Y represents two random variables then,
σ 𝑥 σ 𝑦 𝑃(𝑋 = 𝑥, 𝑌 = 𝑦) = 1
32
Weather
Season
Sunny Rain Cloud Snow
Spring 0.07 0.03 0.10 0.06
Summer 0.13 0.01 0.05 0.01
Autumn 0.05 0.05 0.15 0.03
Winter 0.05 0.01 0.10 0.10

Probability Theory >> Probability Distribution – Full Joint
Full Joint Probability Distribution:
- Full joint probability distribution is a joint distribution for all of the random variables.
- It can be considered as the “knowledge base” from which answers to all questions may be derived.
- For example, full joint distribution for the Toothache, Cavity, Catch world is as follow,
- For example,
- P(cavity ∨ toothache) = 0.108 + 0.012 + 0.072 + 0.008 + 0.016 + 0.064 = 0.28
- P(cavity) = 0.108 + 0.012 + 0.072 + 0.008 = 0.2
33

Probability Theory >> Marginalization
Marginalization/Summing out:
- The process of sum up the probabilities for each possible value of the other variables.
- General Marginalization Formula:
P(Y) = σ 𝑧∈𝑍 𝑷(𝑌, 𝑧)
P(Cavity=cavity)
= σ 𝑧∈{𝐶𝑎𝑡𝑐ℎ,𝑇𝑜𝑜𝑡ℎ𝑎𝑐ℎ𝑒} 𝑷(𝑐𝑎𝑣𝑖𝑡𝑦, 𝑧)
= P(cavity, catch, toothache) + P(cavity, ¬catch, toothache) + P(cavity, catch, ¬ toothache) + P(cavity, ¬ catch, ¬ toothache)
= 0.108 + 0.012 + 0.072 + 0.008
= 0.2
34

Probability Theory >> Practices
Problem 1 >>
A survey has been done on UIU students to assess their interest in hostel accommodation. The data obtained is
as follows:
200 students participated in the survey, half of them male students. Among the male students, 50 were
juniors(first and second year) with 70% interested in hostel accommodation and the rest were seniors with 60% interested in
hostels. Among the females, 60 were juniors with 80% interested in hostels and the rest were seniors with 50% interested in
hostels.
▸Based on this data, construct a full joint distribution among the three random variables Gender(G), Category(C) and
Interest in hostel accommodation(H).
▸Calculate the following probabilities from your table:
i. Probability of a student being a junior.
ii. Probability of a female student not being interested in hostels.
35

Solution >>
P(J) = 35/200 + 48/200 + 15/200 + 12/200 = 110/200
P(F ⋂ ¬ H) = 12/200 + 20/200 = 32/200
36
M F
J H 35/200 48/200
¬H 15/200 12/200
S H 30/200 20/200
¬H 20/200 20/200

Problem 2 >>
A survey has been done in a furniture shop that sells both furniture and electronics items to assess the
probability of customers being interested in each type of product. The data obtained is as follows:
200 customers participated in the survey, 60% of them male. Among the male customers 40 are young, 30 are
middle-aged and the rest are old. Among the young males, 20% are interested in buying furniture and the rest in electronics.
The middle-aged male group is divided equally in both sections. For the old males, 30 are interested in furniture and the rest
in electronics. The women group has 20 young, 30 middle-aged and 30 old customers. Among the young women, half are
interested in buying electronics and the rest in furniture. In both the middle-aged and the old women’s group one-third are
interested in buying electronics and the rest in furniture.
▸Based on this data, construct a full joint distribution among the three random variables Gender(G), Age(A) and Type of
Product(T).
i. Probability of a customer being interested in Electronics.
ii. Probability of an old customer not being interested in Furniture.
37

Problem 3 >>
A survey has been done on final year students of a university to assess their interest in final year project/thesis.
The data obtained is as follows:
100 students participated in the survey, half of them male students. Among the male students 20 are interested
in project, others in thesis. In the project group, 5 like software engineering, 10 like AI and the rest like networking. In the
thesis group, 10 like software engineering, 15 like AI and the rest like networking. Among the female students 30 are
interested in project. Among these 30 students, 12 like software engineering, 10 like AI and United International University
(UIU) Dept. of Computer Science & Engineering (CSE) the rest like networking. Among the female students interested in
thesis, 10 like software engineering, 5 like AI and the rest like networking.
▸Based on this data, construct a full joint distribution among the three random variables Gender(G), Subject(S) and Type of
work(T).
i. Probability of a student being interested in thesis.
ii. Probability of a male student not liking AI.
38

39
THANKS!
Any questions?
You can find me at imam@cse.uiu.ac.bd

AI 8 | Probability Basics, Bayes' Rule, Probability Distribution

More Related Content

What's hot

Similar to AI 8 | Probability Basics, Bayes' Rule, Probability Distribution

More from Mohammad Imam Hossain

Recently uploaded

AI 8 | Probability Basics, Bayes' Rule, Probability Distribution