Course: BEng (Hons) in Electrical and Electronic Engineering

1. EEEC6440315 COMMUNICATION SYSTEMS
Information Theory
FACULTY OF ENGINEERING AND COMPUTER TECHNOLOGY
BENG (HONS) IN ELECTRICALAND ELECTRONIC ENGINEERING
Ravandran Muttiah BEng (Hons) MSc MIET

2. 1
Shannon Limit For Information Capacity
The information capacity of communication systems represents the number
of independent symbols that can be carried through the system in a given
unit of time. The most basic symbol is the binary digit (bit). Therefore, it is
often convenient to express the information capacity of a system in bits per
second (bps). The relationship among bandwidth, transmission time and
information capacity. Simply stated, Hartley’s law is,
𝐼 ∝ 𝐵 × 𝑇
where,
𝐼 = information capacity (bits per second)
𝐵 = bandwidth (Hertz)
𝑇 = transmission time (seconds)
It can be seen that the information capacity is a linear function of bandwidth
and transmission time and is directly proportional to both. If either the
bandwidth or the transmission time is changed, a directly proportional
change in information capacity will occur. Shannon limit is relating the
information capacity of a communications channel to bandwidth and signal
to noise ratio.

3. 2
Digital
Terminal
Interface
Digital
Terminal
Interface
Analogue to
Digital
Converter
Digital to
Analogue
Converter
Digital
Modulation
Free Space
Propagation
Physical
Facility
Digital
Destination
Output
Analogue
Destination
Output
Analogue
Source
Input
Digital
Source
Input
Figure 1: Digital Transmission and Digital Radio Interface.

4. Concepts Related To Channel Capacity
Mathematically stated, the Shannon limit for information capacity is,
𝐼 = 𝐵log2 1 + 𝑆
𝑁
𝐼 = 3.32log10 1 + 𝑆
𝑁
where,
𝑆
𝑁
= signal to noise power ratio (unitless).
Nyquist Bandwidth;
For binary signals (two voltage levels),
𝐼 = 2𝐵
With multilevel signaling (𝑀-ary signaling),
𝐼 = 2𝐵log2 𝑀
𝑀 = number of discrete signal or voltage levels and 𝑁 = number of bits.
∴ 𝑀 = 2𝑁
3

5. 4
Example 1:
For a standard voice-band communications channel with a signal-to-
noise power ratio of 1000 30 dB and a bandwidth of 2.7 KHz. Find the
Shannon limit for information capacity.
𝐼 = 2700 log2 1 + 1000
= 26.9 Kbps

6. Example Of Nyquist And Shannon Formulations
Spectrum of a channel between 3 MHz and 4 MHz: SNRdB = 24 dB
𝐵 = 4 MHz − 3 MHz = 1 MHz
SNRdB = 24 dB = 10log10 SNR
SNR = 251
Using Shannon’s formula,
𝐼 = 106
× log2 1 + 251 ≈ 106
× 8 = 8 Mbps
How many signaling levels are required?
𝐼 = 2𝐵log2𝑀
8 × 106
= 2 × 106
× log2𝑀
4 = log2𝑀
𝑀 = 16
5

7. Linear Block Codes
We assume that the output of an information source is a sequence of binary
digits "0" or "1". In block coding, this binary information sequence is
segmented into message blocks of fixed length; each message block,
denoted by 𝐮, consists of k information digits.
There are a total of 2𝑘
distinct messages. The encoder, according to certain
rules transforms each input message 𝐮 into a binary 𝑛-tuple 𝐯 with 𝑛 > 𝑘
as shown in figure 1. This binary 𝑛-tuple 𝐯 is referred to as the code word
(or code vector) of the message 𝐮 , as shown in figure 2.
6
Block Coder
Figure 1: Block Code.
k information
digits
n encoded
digits

8. 7
Therefore, corresponding to the 2𝑘
possible messages, there are 2𝑘
code words. This set of 2𝑘
code words is called a block code. For a
block code to be useful, the 2𝑘
code words must be distinct. Therefore,
there should be a one-to-one correspondence between a message 𝐮 and
its code word 𝐯.
Information Digits
Parity
Digits
Rate, 𝑅 =
𝑘
𝑛
𝑘 𝑛 − 𝑘
𝑛 digit codeword
Figure 2: Code word.

9. 8
A block code of length 𝑛 and 2𝑘
code words is called a linear 𝑛, 𝑘 code if
and only if its 2𝑘
code words form a 𝑘-dimensional subspace of the vector
space of all the 𝑛-tuples over the field GF(2). In fact, a binary block code is
linear if and only if the modulo-2 sum of two code words is also a code
word. The block code given in table 1 is a 7, 4 linear code. One can easily
check that the sum of any two code words in this code is also a code word.
Definition
(0 0 0 0) (0 0 0 0 0 0 0)
(1 0 0 0) (1 1 0 1 0 0 0)
(0 1 0 0) (0 1 1 0 1 0 0)
(1 1 0 0) (1 0 1 1 1 0 0)
(0 0 1 0) (1 1 1 0 0 1 0)
(1 0 1 0) (0 0 1 1 0 1 0)
(0 1 1 0) (1 0 0 0 1 1 0)
(1 1 1 0) (0 1 0 1 1 1 0)
(0 0 0 1) (1 0 1 0 0 0 1)
(1 0 0 1) (0 1 1 1 0 0 1)
(0 1 0 1) (1 1 0 0 1 0 1)
(1 1 0 1) (0 0 0 1 1 0 1)
(0 0 1 1) (0 1 0 0 0 1 1)
(1 0 1 1) (1 0 0 1 0 1 1)
(0 1 1 1) (0 0 1 0 1 1 1)
(1 1 1 1) (1 1 1 1 1 1 1)
Message Code Words
Table 1: Linear Block Code with 𝑘 = 4 and 𝑛 = 7

10. 9
Since an 𝑛, 𝑘 linear code 𝐶 is a 𝑘-dimensional subspace of the vector space
𝑉
𝑛 of all the binary 𝑛-tuples, it is possible to find 𝑘 linearly independent code
words, 𝑔0, 𝑔1, … , 𝑔𝑘−1 in 𝐶 such that every code word 𝐯 in 𝐶 is a linear
combination of these 𝑘 code words, that is,
𝐯 = 𝑢0𝐠0 + 𝑢1𝐠1 + ⋯ + 𝑢𝑘−1𝐠𝑘−1 (1)
Where 𝑢𝑖 = 0 or 1 for 0 ≤ 𝑖 ≤ 𝑘. Let us arrange these linearly independent
code words as the row of a 𝑘 × 𝑛 matrix as follows:
𝐆 =
𝐠0
𝐠1
⋮
𝐠𝑘−1
=
𝑔0,0 𝑔0,1 𝑔0,2 ⋯ 𝑔0,𝑛−1
𝑔1,0 𝑔1,1 𝑔1,2 ⋯ 𝑔1,𝑛−1
⋮ ⋮ ⋮ ⋮ ⋮
𝑔𝑘−1,0 𝑔𝑘−1,1 𝑔𝑘−1,2 ⋯ 𝑔𝑘−1,𝑛−1
(2)
Where 𝐠𝑖 = 𝑔𝑖,0, 𝑔𝑖,1, … , 𝑔𝑖,𝑛−1 for 0 ≤ 𝑖 ≤ 𝑘.
If 𝐮 = 𝑢0, 𝑢1, ⋯ 𝑢𝑘−1 is the message to be encoded, the corresponding code
word be given as follows:
𝐯 = 𝐮 ∙ 𝐆

11. 10
= 𝑢0, 𝑢1, ⋯ , 𝑢𝑘−1 ∙
𝐠0
𝐠1
⋮
𝐠𝑘−1
= 𝑢0𝐠0 + 𝑢1𝐠1 + ⋯ + 𝑢𝑘−1𝐠𝑘−1 (3)
Clearly, the rows of 𝐆 generate (or span) the 𝑛, 𝑘 linear code 𝐶. For this
reason, the matrix 𝐆 is called a generator matrix for 𝐶. It follows from (3)
that an 𝑛, 𝑘 linear code is completely specified by the 𝑘 rows of a
generator matrix 𝐆. Therefore, the encoder has only to store the 𝑘 row of 𝐆
and to form a linear combination of these 𝑘 rows based on the input
message.

12. 11
Example 2:
The 7, 4 linear code given in table 1 has the following matrix as a
generator matrix.
𝐆 =
𝐠0
𝐠1
𝐠2
𝐠3
=
1 1 0 1 0 0 0
0 1 1 0 1 0 0
1 1 1 0 0 1 0
1 0 1 0 0 0 1
If 𝐮 = 1 1 0 1 is the message to be encoded, its corresponding code word,
according to (3), would be
𝐯 = 1 ∙ 𝐠0 + 1 ∙ 𝐠1 + 0 ∙ 𝐠2 + 1 ∙ 𝐠3
= 1 1 0 1 0 0 0 + 0 1 1 0 1 0 0 + 1 0 1 0 0 0 1
= 0 0 0 1 1 0 1

13. 12
A desirable property for a linear block code to possess is the systematic
structure of the code words as shown in figure 3, where a code word is
divided into two parts, the message part and the redundant checking part.
The message part consists of unaltered information (or message) digits and
the redundant checking part consists 𝑛 − 𝑘 parity-check digits, which are
linear sums of the information digits. A linear block code with this
structure is referred to as a linear systematic block code. The 7, 4 code
given in table is a linear systematic block code; the rightmost four digits of
each code are identical to the corresponding information digits.
Redundant
Checking Part
Message
Part
𝑛 − 𝑘 digits 𝑘 digits
Figure 3: Systematic format of a code word

14. 13
A linear systematic 𝑛, 𝑘 code is completely specified by a 𝑘 × 𝑛 matrix 𝐆 of the
following form:
𝐆 =
𝐠0
𝐠1
𝐠2
⋮
𝐠𝑘−1
=
𝑝0,0 𝑝0,1 ⋯ 𝑝0,𝑛−𝑘−1 1 0 0 ⋯ 0
𝑝1,0 𝑝1,1 ⋯ 𝑝1,𝑛−𝑘−1 0 1 0 ⋯ 0
𝑝2,0 𝑝2,1 ⋯ 𝑝2,𝑛−𝑘−1 0 0 1 ⋯ 0
⋮
𝑝𝑘−1,0 𝑝𝑘−1,1 ⋯ 𝑝𝑘−1,𝑛−𝑘−1 0 0 0 ⋯ 1
(4)
where 𝑝𝑖𝑗 = 0 or 1. Let 𝐼𝑘 denote the 𝑘 × 𝑘 identity matrix. Then 𝐆 = 𝐏 𝐈𝑘 .
Let 𝑢 = 𝑢0, 𝑢1, … , 𝑢𝑘−1 be the message to be encoded. The corresponding code
word is,
𝐯 = 𝑣0, 𝑣1, 𝑣2, … , 𝑢𝑘−1
= 𝑢0, 𝑢1, … , 𝑢𝑘−1 ∙ 𝐆 (5)
P matrix 𝑘 × 𝑘 identity matrix

15. 14
Example 3:
The matirx 𝐆 given in example 2 is in systematic form. Let 𝐮 = 𝑢0, 𝑢1, 𝑢2, 𝑢3
be the message to be encoded and let 𝐯 = 𝑣0, 𝑣1, 𝑣2, 𝑣3, 𝑣4, 𝑣5, 𝑣6 be the
corresponding code word. Then,
𝐯 = 𝑢0, 𝑢1, 𝑢2, 𝑢3 ∙
1 1 0 1 0 0 0
0 1 1 0 1 0 0
1 1 1 0 0 1 0
1 0 1 0 0 0 1
By matrix multiplication, we obtain the following digits of the code word 𝐯,
𝑣6 = 𝑢3
𝑣5 = 𝑢2
𝑣4 = 𝑢1
𝑣3 = 𝑢0
𝑣2 = 𝑢1 + 𝑢2 + 𝑢3
𝑣1 = 𝑢0 + 𝑢1 + 𝑢2
𝑣0 = 𝑢0 + 𝑢2 + 𝑢3

16. 15
The code word corresponding to the message 1 0 1 1 is 1 0 0 1 0 1 1 .
There is another useful matrix associated with every linear block code. For any
𝑘 × 𝑛 matrix 𝐆 with 𝑘 linearly independent rows, there exists an 𝑛 − 𝑘 × 𝑛
matrix 𝐇 with 𝑛 − 𝑘 linearly independent rows such that any vector in the row
space of 𝐆 is orthogonal to the rows of 𝐇 and any vector that is orthogonal to the
rows of 𝐇 is in the row space of 𝐆. Hence, we can describe the 𝑛, 𝑘 linear
code generated by 𝐆 in an alternative way as follows:
An 𝑛-tuple 𝐯 is a code word in the code generated by 𝐆 if and only if 𝐯 ∙ 𝐇𝐓
=
0. This matrix 𝐇 is called a parity-check matrix of the code. If the generator
matrix of an 𝑛, 𝑘 linear code is in the systematic form of (4), the parity-check
matrix may take the following form:
𝐇 = 𝐈𝑛−𝑘 𝐏𝐓
=
1 0 0 ⋯ 0 𝑝0,0 𝑝1,0 ⋯ 𝑝𝑘−1,0
0 1 0 ⋯ 0 𝑝0,1 𝑝1,1 ⋯ 𝑝𝑘−1,1
0 0 1 ⋯ 0 𝑝0,2 𝑝1,2 ⋯ 𝑝𝑘−1,2
⋮
0 0 0 ⋯ 1 𝑝0,𝑛−𝑘−1 𝑝1,𝑛−𝑘−1 ⋯ 𝑝𝑘−1,𝑛−𝑘−1
(6)

17. 16
Example 4:
Consider the generator matrix of a 7, 4 linear code given in example 2. The
corresponding parity-check matrix is,
𝐇 =
1 0 0 1 0 1 1
0 1 0 1 1 1 0
0 0 1 0 1 1 1
Syndrome And Error Detection
Consider an 𝑛, 𝑘 linear code with generator matrix 𝐆 and parity-check matrix
𝐇. Let 𝐯 = 𝑣0, 𝑣1, … , 𝑣𝑛−1 be a code word that was transmitted over a noisy
channel. Let 𝐫 = 𝑟0, 𝑟1, … , 𝑟𝑛−1 be the received vector at the output of the
channel. Because of the channel noise, 𝐫 may be different from 𝐯. The vector
sum,
𝐯 = 𝐫 + 𝐯
= 𝑒0, 𝑒1, … , 𝑒𝑛−1 (7)

18. 17
Is an 𝑛-tuple where 𝑒𝑖 = 1 for 𝑟𝑖 ≠ 𝑣𝑖 and 𝑒𝑖 = 0 for 𝑟𝑖 = 𝑣𝑖. This 𝑛-tuple is
called the error vector (or error pattern). The 1’s in 𝐞 are the transmission errors
caused by the channel noise. It follows from (7) that the received vector 𝐫 is the
vector sum of the transmitted code word and the error vector, that is,
𝐫 = 𝐯 + 𝐞 (8)
Of course, the receiver does not allow either 𝐯 or 𝐞. Upon receiving 𝐫, the
decoder must first determine whether 𝐫 contains transmission errors. If the
presence of errors is detected, the decoder will either take actions to locate the
errors and correct them (FEC) or request for a transmission of (ARQ). When 𝐫 is
received, the decoder computes the following 𝑛 − 𝑘 -tuple:
𝐬 = 𝐫 ∙ 𝐇𝐓
= 𝑠0, 𝑠1, … , 𝑠𝑛−𝑘−1 (9)
Which is called the syndrome of 𝐫. Then 𝐬 = 0 if and only if 𝐫 is a code word,
and 𝐬 ≠ 0 if and only if 𝐫 is not a code word. Therefore, when 𝐬 ≠ 0, we know
that 𝐫 is not a code word and the presence or errors has been detected.

19. 18
When 𝐬 = 0, 𝐫 is a code word and the receiver accepts 𝐫 as the transmitted code
word. It is possible that the errors in certain error vectors are not detectable (i.e.,
𝐫 contains errors but 𝐬 = 𝐫 ∙ 𝐇𝐓
= 0). This happens when the error pattern 𝐞 is
identical to a nonzero code word. In this event, 𝐫 is the sum of two code words
which is a code word, and consequently 𝐫 ∙ 𝐇𝐓
= 0.
Error patterns of this kind are called undetectable error patterns. Since there are
2𝑘
− 1 nonzero code words, there are 2𝑘
− 1 undetectable error patterns. When
an undetectable error pattern occurs, the decoder makes a decoding error.
Based on (6) and (9), the syndrome digits are as follows:
𝑠0 = 𝑟0 + 𝑟𝑛−𝑘𝑃00 + 𝑟𝑛−𝑘+1𝑃10 + ⋯ + 𝑟𝑛−1𝑃𝑘−1, 0
𝑠1 = 𝑟1 + 𝑟𝑛−𝑘𝑃01 + 𝑟𝑛−𝑘+1𝑃11 + ⋯ + 𝑟𝑛−1𝑃𝑘−1, 1
⋮
𝑠𝑛−𝑘−1 = 𝑟𝑛−𝑘−1 + 𝑟𝑛−𝑘𝑃0, 𝑛−𝑘−1 + 𝑟𝑛−𝑘+1𝑃1, 𝑛−𝑘−1 + ⋯ + 𝑟𝑛−1𝑃𝑘−1, 𝑛−𝑘−1
(10)

20. (1) Thamer Muhammed Jamel, Information Theory, University of
Technology Iraq, 2012.
References
19