This document provides an introduction to information channels. It defines an information channel as having an input alphabet, output alphabet, and conditional probabilities relating input and output symbols. It discusses how to represent channels in matrix form and calculates various probabilities. It also covers zero-memory channels, extensions of channels to multiple inputs/outputs, entropy, mutual information, and uses the binary symmetric channel as an example.

1. Introduction to Information
Channel
Akhil Nadh PC
17203101
MTech ( IS )
Submitted To
Mr. D K Gupta

2. Topics Discussed
q Definition Of Information Channel
q How to Represent Information Channel
qCalculation Of Various Probabilities associated with Information Channel

3. Information Channel
v Defined by
§ Input Alphabet A , defined as A= { 𝑎"} for i ={1,2,3….., r } , where r is the size of input
alphabet
§ Output Alphabet B , defined as B= { 𝑏% } for j ={1,2,3….., s } , where s is the size of output
alphabet
§ r ≠ s ( need not be)
§ 𝑃(
()
*+
) ∀𝑖 , 𝑗 Probability of output symbol 𝑏% is received if 𝑎" is send

4. Information Channel
q Information Channel does not produce information
q Used to transit information from source to destination

5. Zero Memory Information Channel
q Output symbol 𝑏% is depended upon the input symbol 𝑎"
qIf output symbol 𝑏% depended upon several preceding input symbols then the channel is called
Channel with Memory

6. Conventional ways of describing
Informational Channel
qArranging conditional probability of output in matrix form
q
𝑏0 𝑏1
𝑎0 𝑃(
(2
*2
)
𝑎1 𝑃(
(2
*3
)
𝑏4
𝑃(
(5
*2
) ⃗ corresponds to probability for fixed input is send
𝑎7 𝑃(
(2
*8
)
𝑎9 𝑃(
(2
*:
) 𝑃(
(5
*:
)

7. Concise Representation
q 𝑃"% = 𝑃
()
*+
→ 1
q𝑃 =
𝑃00
𝑃10
𝑃90
𝑃04
𝑃04
𝑃94
qIf sum =1 => Markov’s Matrix
q∑ 𝑃"% = 1 ∀ 𝑖 = 1,2,…, 4 4
%C0
Sum =1

8. Nth Extension of the Channel Information
qInput Symbol 𝐴E = {𝛼"} ∀ 𝑖 = 1,2,3… , 𝑟E
qOutput Symbol 𝐵E = {𝛽%} ∀ 𝑗 = 1,2,3… , 𝑠E
qChannel Matrix
∏ =
∏00
∏10
∏90
∏04N
∏04N
∏9N4N
q∏"% = 𝑃(
P)
Q+
)

9. Amount Of Information A channel can
Transmitted
qConsider channel with
q 𝑟 input symbols
q 𝑠 output symbols
q Defined by conditional probability P
𝑃00 −> probability of occurrence of 𝑏0 when 𝑎0 is sent
qProbability of various output symbols given probability of some input symbol is given by
𝑃 𝑎0 𝑃00 + 𝑃 𝑎1 𝑃10 + ⋯+ 𝑃 𝑎9 𝑃90 = 𝑃 𝑏0
𝑃 𝑎0 𝑃01 + 𝑃 𝑎1 𝑃11 + ⋯+ 𝑃 𝑎9 𝑃91 = 𝑃 𝑏1
𝑃 𝑎0 𝑃04 + 𝑃 𝑎1 𝑃14 + ⋯+ 𝑃 𝑎9 𝑃94 = 𝑃 𝑏4
𝑃00

10. Amount Of Information A channel can
Transmitted
q𝑃00 → 𝑃 𝑎" = 𝑃
()
*+
qAccording to Bayes theorem
q𝑃
*)
(+
=
V
W)
X+
∗V *+
V ()
=
V
W)
X+
∗V *+
∑ V
W)
X+
∗V *+
:
+Z2
Input probability
Channel matrix probability
Backward probability

11. Amount Of Information A channel can
Transmitted
qJoint probability is given by
𝑃 𝑎" , 𝑏" = 𝑃
𝑏%
𝑎"
∗ 𝑃 𝑎"
= 𝑃
*+
()
∗ 𝑃 𝑏%

12. Binary Symmetric Channel
qSymmetric iff:
𝑃
0
[
= 𝑃
[
0
Receiving
Sending

13. Binary Symmetric Channel
q input size = output size = 2
q 𝑝 = 1 − 𝑝
q p = probability that error will occur
qSymmetric if probability of error is same while
transmitting the symbols

14. Binary Symmetric Channel
qChannel Matrix Form :
𝑃]^_ =
𝑝⃑ 𝑝
𝑝 𝑝⃑
q 2nd – Extension of Binary Symmetric Channel
∏=
𝑝⃑1
𝑝⃑ 𝑝
𝑝𝑝⃑
𝑝1
𝑝⃑ 𝑝
𝑝⃑1
𝑝1
𝑝𝑝⃑
𝑝𝑝⃑
𝑝1
𝑝⃑1
𝑝⃑ 𝑝
𝑝1
𝑝𝑝⃑
𝑝⃑ 𝑝
𝑝⃑1

15. Binary Symmetric Channel
qConcise Form
∏ =
𝑝⃑ 𝑃 𝑝𝑃
𝑝𝑃 𝑝⃑ 𝑃
qP be channel matrix of Binary Symmetric Channel

16. Various output symbols of our channel occur
according to set of probabilities given by 𝑏%
qP( given output symbol𝑏%)
= P(𝑏%) ->If we do not know which input symbol is sent
= P(
()
*+
) -> if input symbol 𝑎" is sent
q P( choosing input symbol 𝑎0)
=𝑃(𝑎0) ->if we do not know which output symbol it will produce
= P(
*+
()
) -> given output symbol 𝑏% is produced
Change in Probability
Change in Probability
A priory probability
A posteriorly probability

17. Entropy of source A
qA priory Entropy of source A
𝐻 𝐴 = ∑ 𝑃 𝑎" 𝑙𝑜𝑔
0
V(*+)f -> avg # binits need to represent a symbol from
source with a priority probability i=1,2,…,r
qA posteriorly Entropy of Source A when 𝑏% is received
H(f
()h ) = ∑ 𝑃 *+
()h 𝑙𝑜𝑔
0
V
X+
W)
h
f

18. Example - Question
qA={0,1}
qB={0,1}
q𝑃{𝑎0 = 0} =
7
j
𝑃{𝑎0 = 1} =
0
j
q𝑃
()
*+
=
1
7
0
7
0
0[
k
0[
qFind 𝑃{𝑏0 = 0} , 𝑃{𝑏0 = 1} , 𝑃{
*2C0
(2C0
} P(𝑎0 = 0, 𝑏0 = 0) H A ,H
m
[
,H
m
0

19. Example - Solution
qP(b=0)=
7
j
∗
1
7
+ ¼ ∗
0
0[
=
10
j[
qP(b=1)=
7
j
∗
0
7
+ ¼ ∗
k
0[
=
0k
j[
q P(a=1/b=1)
2
o
∗
p
2q
2p
oq
=
k
0k
q P(a=0/b=0)
8
o
∗
3
8
32
oq
=
1[
0k
qP(a=0 , b=0 ) = P(a=0/b=0)P(b=0) =
1[
10
∗
10
j[
=
0
1

20. Example - Solution
qEntropy of source
𝐻 𝐴 =
3
4
log
4
3
+
1
4
log
4
1
= 0.811 bits/symbol
qA posteriorly Entropy when 0 is received
𝐻
𝐴
0
=
20
21
log
21
20
+
1
21
log
21
1
= 0.267 bits/symbol
qA posteriorly Entropy when 1is received
𝐻
𝐴
1
=
9
19
log
19
9
+
10
19
log
19
10
= 0.998 bits/symbol
More Uncertainty

21. Shannon’s First Theorem
qEntropy of alphabet may be interpreted as the average number of
binary digits (binits) necessary to represent one symbol of that alphabet
qDeals only with coding for a source with fixed set of source probabilities
qNot with coding for a source which select new probabilities after each output
symbol

22. Generalization of Shannon’s First
Theorem
qWhat is the most efficient method of coding from a source ?
q here , source is input alphabet A
statistics of code change from symbol to symbol
which source we need to pick is given by P(bj)
compact code of one set of statistics will not be in general compact code of other statistics
q For each transmitted symbol we construct s binary codes ( s -> size of output alphabet )
qBuild s codes one for each of possible symbol received for bj

23. input symbol code1 code 2 code3 … … code s
a1 l11 l12 l1s
a2 l21 l23 l2s
a3
…
…
ar lr1 lr2 lrs
l11 -> length of code word 1 for input symbol a1
Generalization of Shannon’s First
Theorem
Depending on what we receive

24. q-> We need each code words to be instantaneous , we apply Shannon’s first theorem
𝐻
f
()
≤ 𝑃
*+
()
∗ 𝑙"% -> avg length of jth code
Generalization of Shannon’s First
Theorem
j th code is employed only when 𝑏% is the received symbol

25. Generalization of Shannon’s First
Theorem
qGeneralized Shannon’s First Theorem
∑ 𝑃 𝑏 𝐻
f
(
≤
Š
E
< ∑ 𝑃 𝑏 𝐻
f
(
+
0
E]f n-> Nth extension summed over input and
output symbols
𝐻 𝑆 ≤
Š
E
< 𝐻 𝑆 +
0
E
-> for single source
-> valid for only instantaneous codes which are uniquely decodable *
Avg #binits need to encoded from input symbol A
if we already have corresponding out put symbol b
Equivocation of A wrt to B

26. Channel Equivocation
q𝐻
f
]
= ∑ ∑ 𝑃 𝑎, 𝑏 log
0
V
X
W
]f

27. Mutual Information Of Channel
q 𝐼 𝐴, 𝐵 = H A − H
m
Ž
On the average observation of single output symbol provides us with these many bits of information
also called uncertainty resolved
q 𝐼 𝐴, 𝐵 = ∑ ∑ 𝑃 𝑎, 𝑏 log (
V *,(
V * V(()] )f *
q𝐼 𝐴E
, 𝐵E
= 𝑛 𝐼 𝐴, 𝐵

28. Mutual Information Of Channel
qCan the Mutual Information be negative ?
H A − H
m
()
this can be negative as we have seen already
q 𝐼 𝐴, 𝐵 ≥ 0 iff 𝑃 𝑎", 𝑏% = 𝑃 𝑎" 𝑃(𝑏%) for all i,j proof
We cannot loose information on average on observing output of the channel

29. Joint Entropy
q 𝐻 𝐴, 𝐵 = ∑ ∑ 𝑃 𝑎, 𝑏 log (
0
V *,(] )f
q𝐻 𝐴, 𝐵 = H(A) + H(B) – I ( A,B)
q𝐻 𝐴, 𝐵 = 𝐻 𝐴 + 𝐻
]
f
q𝐻 𝐴, 𝐵 = 𝐻 𝐵 + 𝐻
f
]

30. Example – Binary Symmetric Channel
Given Data
qChannel Matrix
𝑃]^_ =
𝑝⃑ 𝑝
𝑝 𝑝⃑
where 𝑝⃑=1- 𝑝
qProbability of transmitting input symbols
1 → 𝜔
0 → 𝜔’

31. Example – Binary Symmetric Channel
𝐼 𝐴, 𝐵 = H B − H
B
A
= H B − ( p log
0
”
+ 𝑝⃑log
0
•⃑
)
P( 𝑏% =0) = 𝜔𝑝⃑ + 𝜔’𝑝
P( 𝑏% =1) = 𝜔𝑝 + 𝜔’𝑝⃑
𝐼 𝐴, 𝐵 =H( 𝜔𝑝 + 𝜔’𝑝⃑)-H(p)

32. Thank You