Information theory

1. INFORMATION
THEORY
© 2020 APARNA LAL

2. Information is only useful when it can be stored
and/or communicated. We have all learned this
lesson the hard way when we have forgotten to save
a document we were working on. In a digital form,
information is stored in ‘bits’, or a series of numbers
that can either be 0 or 1. The letters in your
keyboard are stores in a ‘byte’, which is 8 bits, which
allows for 2⁸ =256 combinations. It is important to
know that information storage and communication
are almost the same thing, as you can think of
storage as communication with a hard disk.
When is information useful?
© 2020 APARNA LAL

3. 1. RATE OF INFORMATION
The information rate is represented by R and it is given as,
Information Rate : R = rH
Here R is the information rate.
H is the Entropy or average information.
And r is the rate at which messages are generated.
Information rate R is represented in average number of bits of
information per second.
© 2020 APARNA LAL

4. 2.Bit Rate
It is defined as no of bits transmitted or sent in one second. It is expressed in bits per second.
Bit rate = 1/ Bit interval
4. Baud Rate
The baud rate is the rate at which information is transferred in a communication channel. Baud rate
is number of signal transferred units per second. It is expressed in baud per second.
3. Data Rate :
Data Rate is defined as the amount of data transmitted during a specified time period over a network. It is the
speed at which data is transferred from one device to another or between a peripheral device and the
computer. It is generally measured in Mega bits per second(Mbps) or Mega bytes per second(MBps).
For example, if bandwidth is 100 Mbps but data rate is 50 Mbps, it means maximum 100 Mb data can be
transferred but channel is transmitting only 50 Mb data per second.
© 2020 APARNA LAL

5. Signal to Noise Ratio
In analog and digital communications, signal-to-noise ratio, often written
S/N or SNR, is a measure of signal strength relative to background noise.
The ratio is usually measured in decibels (dB) using a signal-to-noise ratio
formula. It is defined as the ratio of signal power to noise power at the
same point.
𝑆
𝑁
=
𝑋𝑠
𝑋𝑛
=
Vs2
/R
𝑉𝑛2
/𝑅
=
𝑉𝑠
𝑉𝑛
Xs= Signal Power Xn=Noise Power Vs =signal voltage, Vn= Noise Voltage
Noise figure
To express the noise quality of the receiver noise figure is used
It is the ratio of S/N power at the input to S/N power at the output
𝐹 =
𝑆/𝑁𝑖𝑛𝑝𝑢𝑡
𝑆/𝑁𝑜𝑢𝑡𝑝𝑢𝑡
© 2020 APARNA LAL

6. Channel Capacity
It is basically measure of how much information can be propagated through a
communication channel.
It is function of:-
• Bandwidth of the communication channel
• Time of transmission
Hartley’s Law
It is stated as “ the total amount of information that can be transmitted is
proportional to frequency range transmitted and the time of the transmission.”
C α B× t
Where, C= Information capacity (bits per second)
B= Bandwidth (hertz)
T=transmission time (seconds)
© 2020 APARNA LAL

7. If bandwidth of channel is greater then more information can be transmitted in a
given time. Also same signal can be transmitted using narrow channel but the time
taken will be more.
Transmission of music (bandwidth =15-20 kHz) requires larger time as compared to
voice (bandwidth=3kHz).
Channel Capacity formula can be modified as
C=2 B log2N
Where , B= channel bandwidth
N=no of bits per symbol
Then for binary data transmission
C=2B
© 2020 APARNA LAL

8. Sampling
In digital transmission, continuous wave is not transmitted. Only discrete value are
transmitted. If information is analog it is converted to discrete form by using a
method called sampling.
Sample is a piece of data taken from the whole data which is continuous in the time
domain.
© 2020 APARNA LAL

9. In other word, the process of taking only a few values of an analog signal at regular
interval of time is referred to as sampling. Consider the waveform. If we divide the
waveform at discrete intervals of time say 100µsec, the sample waveform looks like
figure below. The sampled signal is discrete in time and continuous in amplitude.
Thus sampling converts the continuous value continuous time signal to a continuous
value discrete time signal.
© 2020 APARNA LAL

10. Sampling Rate
To discretize the signals, the gap between the samples should be fixed. That gap can
be termed as a sampling period Ts.
Sampling Frequency=1/Ts=Fs
Ts is the sampling time
fs is the sampling frequency or the sampling rate.
© 2020 APARNA LAL

11. Sampling Theorem/Nyquist Theorem
You can perfectly reconstruct the original signal from the samples provided you
follow a simple rule called as sampling theorem which states that :
“For a sample to be reproduce accurately at a receiver each cycle of the analog input
signal must be sampled at least twice”.
For effective reproduction of the original signal, the sampling rate should be
twice the highest frequency.
Which means
Fs= 2 Fm
Ts = 1/ Fs= 1/2Fm
If sampling rate is less than 2 Fm distortion occurs while reconstructing the information.
© 2020 APARNA LAL

12. Shannon Theorem
• A given communication system has a maximum rate of information C known as the
channel capacity.
• If the information rate R is less than C, then one can approach arbitrarily small error
probabilities by using intelligent coding techniques.
• To get lower error probabilities, the encoder has to work on longer blocks of signal
data. This entails longer delays and higher computational requirements.
• Thus, if R ≤ C then transmission may be accomplished without error in the presence of
noise.
Unfortunately, Shannon’s theorem is not a constructive proof — it merely states that such a coding
method exists. The proof can therefore not be used to develop a coding method that reaches the
channel capacity.
© 2020 APARNA LAL

13. Shannon-Hartleys Theorem
• Noise puts limitations on increase in symbol levels. Greater the amount of noise, lower are the
symbol levels and hence lower is the channel capacity.
• Thus presence of noise reduces the amount of information that can be transmitted in a given
bandwidth.
• Increasing bandwidth increases information rate but wider bandwidth means more noise signals.
• Therefore in communication system bandwidth is a compromise between information
transmission rate and amount of noise.
• Shannon theorem relates the channel capacity to bandwidth and S/N ratio
• Mathematically stated
C = B log2 (1+
𝑆
𝑁
)= 3.32Blog10 (1+
𝑆
𝑁
)
• Where C= Channel Capacity, bits/s(bps)
• B= bandwidth, Hz
•
𝑆
𝑁
= signal to noise power ratio.
• the above equation is Shannon’s Information Capacity theorem.
© 2020 APARNA LAL