Information Sources and Entropy

1. Sanjivani College of Engineering, Kopargaon
Department of Electronics & Computer Engineering
(An Autonomous Institute)
Affiliated to Savitribai Phule Pune University
Accredited ‘A’ Grade by NAAC
________________________________________________________________________________________
Subject: Discrete Mathematics and Information Theory (EC 201)
UNIT-6
Topic: Information Sources and Entropy
Dipak Mahurkar
Assistant Professor, ECE Department
Dipak Mahurkar ECE Department 1

2. What is Information?
Dipak Mahurkar ECE Department 2
• Information is the source of a communication system, whether it is analog
or digital.
• Information theory is a mathematical approach to the study of coding of
information along with the quantification, storage, and communication of
information.
• A measure of uncertainty.
• Can we really analyze it quantitatively?
• What do the numerical values mean?
• Is it tied to “knowledge”?
• Is it subjective?

3. Conditions of Occurrence of Events
Dipak Mahurkar ECE Department 3
• If we consider an event, there are three conditions of occurrence.
• If the event has not occurred, there is a condition of uncertainty.
• If the event has just occurred, there is a condition of surprise.
• If the event has occurred, a time back, there is a condition of having some
information.
• These three events occur at different times.
• The difference in these conditions help us gain knowledge on the probabilities
of the occurrence of events.

4. Uncertainty and Probability
Dipak Mahurkar ECE Department 4

5. Rate of Information
Dipak Mahurkar ECE Department 5
The average number of bits of information per second.
R = r. H bits / second
Where r=generated message per second.

6. Entropy
Dipak Mahurkar ECE Department 6
When we observe the possibilities of the occurrence of an event, how
surprising or uncertain it would be, it means that we are trying to have an
idea on the average content of the information from the source of the event.
Entropy can be defined as a measure of the average information content per
source symbol.
Where pi is the probability of the occurrence of character number i from a
given stream of characters and b is the base of the logarithm used. This is
also called as Shannon’s Entropy.
The amount of uncertainty remaining about the channel input after
observing the channel output, is called as Conditional Entropy. It is
denoted by H(x∣y)

7. Entropy Contd…
Dipak Mahurkar ECE Department 7
Consider that there are M={m1,m2,….} different message with
probabilities P={p1,p2…..}
Suppose that a sequence of L messages is transmitted,
p1L message of m1 are transmitted
p2L message of m2 are transmitted
..
pmL message of mm are transmitted
Info I(m1) = log2(1/p1)
If (p, L) message at m1 are transmitted
I1(total) = p1L log2(1/p1)
I2(total) = p2L log2(1/p2)

8. Entropy Contd…
Dipak Mahurkar ECE Department 8
If (p, L) message at m1 are transmitted
I1(total) = p1L log2(1/p1)
I2(total) = p2L log2(1/p2)
.
.
.
Im (total) = pm L log2(1/pm)
I(total) = p1L log2(1/p1) + p2L log2(1/p2) + ….+ pmL log2(1/pm)
Average Info = Total Info / No. of messages
= I(total)/L
= [p1L log2(1/p1) + p2L log2(1/p2) + ….+ pmL log2(1/pm)] / L
= p1 log2(1/p1) + p2 log2(1/p2) + ….+ pm log2(1/pm)
Hence, we can write,
Entropy = 𝑘=1
𝑚
𝑝𝑘𝑙𝑜𝑔2(
1
𝑝𝑘
)

9. Properties of Entropy
• Entropy is zero if the event is sure, then H = 0
• When pk = 1/m for all m symbols, then symbols are equally
likely H = log2m
• Upper bound on entropy is given as Hmax = log2m
Dipak Mahurkar ECE Department 9

10. Properties of Entropy Contd…
• H = 0 if pk = 1 or pk = 0
For pk = 1
H = 𝑘=1
𝑚
𝑝𝑘𝑙𝑜𝑔2(
1
𝑝𝑘
)
= 𝑘=1
𝑚
𝑙𝑜𝑔2(
1
1
)
= 𝑘=1
𝑚
(
𝑙𝑜𝑔10(1)
𝑙𝑜𝑔2(10)
) = 0
Dipak Mahurkar ECE Department 10

11. Properties of Entropy Contd…
For pk = 0
H = 𝑘=1
𝑚
𝑝𝑘𝑙𝑜𝑔2(
1
𝑝𝑘
) = 0
Dipak Mahurkar ECE Department 11

12. Properties of Entropy Contd…
For pk = 1/m
H = 𝑘=1
𝑚
𝑝𝑘𝑙𝑜𝑔2(
1
𝑝𝑘
)
= 𝑘=1
𝑚
(
1
𝑚
)𝑙𝑜𝑔2(𝑚)
= log2(m)
Dipak Mahurkar ECE Department 12

13. Source Efficiency and Redundancy
Dipak Mahurkar ECE Department 13

14. Self Information and Mutual Information
Dipak Mahurkar ECE Department 14
• Self information is always non negative.
• The unit of average mutual information is bits
• When the base of the logarithm is 2 then the unit of measure of
information is bits.
• Entropy of a random variable is also infinity.
• The self information of a random variable is infinity.
• Smaller the code rate, more are the redundant bits.
• When probability of error during transmission is 0.5 then the
channel is very noisy and thus no information is received.

15. Example
Dipak Mahurkar ECE Department 15
For the discrete memoryless source there are three symbols with
p1 = α and p2 = p3. Find the entropy of the source.
Solution:
Given p1 = α , p2 = p3
p1+p2+p3 = 1
Since, p2=p3;
p1+p2+p2 = 1 => α+2p2 = 1
Hence, p2 = (1- α)/2 = p3

16. Example
Dipak Mahurkar ECE Department 16
H = 𝑘=1
𝑚
𝑝𝑘𝑙𝑜𝑔2(
1
𝑝𝑘
)
=𝑝1 𝑙𝑜𝑔2(
1
𝑝1
)+𝑝2 𝑙𝑜𝑔2(
1
𝑝2
)+𝑝3 𝑙𝑜𝑔2(
1
𝑝3
)
= α𝑙𝑜𝑔2(
1
α
)+(1- α )𝑙𝑜𝑔2(
1
(1−α)
)

17. Example 2
Dipak Mahurkar ECE Department 17

18. Example 3
Dipak Mahurkar ECE Department 18
H = 𝑘=1
𝑚
𝑝𝑘𝑙𝑜𝑔2(
1
𝑝𝑘
)
=(
1
2
) 𝑙𝑜𝑔2(2)+(
1
4
) 𝑙𝑜𝑔2(4)+
1
8
𝑙𝑜𝑔2 8 +……… +
1
2𝑛 𝑙𝑜𝑔2 2𝑛
= (1/2)+(2/4)+(3/8)+…….+(n/2n)
= 2 – [ (n+2)/2n]

19. Example 3
Dipak Mahurkar ECE Department 19
The source emits three message with probabilities p1 = 0.7, p2 = 0.2, p3 =
0.1.
Calculate:
1. Source of Entropy
2. Maximum Entropy
3. Source Efficiency
4. Redundancy

20. Example 3
Dipak Mahurkar ECE Department 20
H = 𝑘=1
𝑚
𝑝𝑘𝑙𝑜𝑔2(
1
𝑝𝑘
)
= 0.7𝑙𝑜𝑔2(1/0.7)+0.2 𝑙𝑜𝑔2(1/0.2)+0.1𝑙𝑜𝑔2 1/0.1
= 0.3602+ 0.4643 + 0.3321
= 1.1566 bits/message
Hmax = log2m = log23 = 1.5849 bits/message
ηsource = H/Hmax = 1.1566/1.5849 = 0.7297
γsource = 1 – ηsource = 1 – 0.7297= 0.27

21. Example
Dipak Mahurkar ECE Department 21
A discrete source emits one of six symbols once every milli sec.
The symbol probabilities are ½, ¼, 1/8, 1/16, 1/32 and 1/32
respectively.
Find the source Entropy and Information rate
Solution:
R = r*H M = 6, r = 103
H = 1.9375 bits/message
R = 103 x 1.9375
R = 1937.5 bits/sec

22. Dipak Mahurkar ECE Department 22
Thank You!