This document provides an overview of information theory and its applications in communication engineering. It discusses key concepts such as source coding techniques like lossless and lossy data compression, algorithms like Huffman coding and arithmetic coding. It also covers channel coding techniques including error detection codes, forward error correction, block codes, convolutional codes, turbo codes and their applications in areas like digital television, DVDs, wireless communications. The document concludes that information theory provides fundamental limits for efficient digital communication and coding applications have improved reliability and efficiency of modern communication systems.

1. Application of information
theory in Communication
engineering
Presented by: Abdul Razaque
NEDUET KARACHI
abdulrazaque15tc@gmail.com
10/4/2020 1

2. Contents
• Introduction
• Source coding
• Huffman Algorithm
• Image/Video Compression
• Channel coding
• Techniques
• Transmission errors
• Block codes
• Convolution codes
• Turbo codes
• Conclusion
• Reference
10/4/2020 2

3. Introduction
• Information theory studies the quantification, storage, and
communication of information.
• It was proposed by Claude Shannon in 1948 to find fundamental
limits on signal processing and communication operations such
as data compression.
• Coding theory is one of the most important and direct applications of
information theory, which is subdivided into source coding theory and
channel coding theory .
10/4/2020
3

4. Source coding
• Source coding remove redundancy in the information to make the message smaller.
1. Loss less Data Compression: it is used to compress the files without losing an original file's
quality and data. we can say that in this technique, file size is reduced, but the quality of data
remains the same.
2. Lossy Data compression: it is used to compress larger files into smaller files. In this
compression technique, some specific amount of data and quality are removed (loss) from
the original file.it is useful for us when the quality of data is not our first priority.
10/4/2020 4

5. Huffman algorithm
• It is basically used for encoding entropy and to compress data without loss. In order to
choose a particular representation for each symbol, Huffman coding makes use of a
particular method that leads to a prefix code.
• This method uses a minimum number of bits in the form of strings to represent mostly
used and common source symbols.
• Huffman coding used to encode run length. For JPEG, PNG, MPEG, MP3, AAC and also
used in steganography for JPEG carrier compression.
10/4/2020 5

6. Arithmetic algorithm
• Arithmetic coding does not assign particular bit patterns to the source symbols.
Instead, a single arithmetic codeword is assigned to the complete sequence of
symbols. Arithmetic codewords consist of sub-intervals of the unit interval [0, 1).
• Supports to general purpose applications like PPM, PAQ.
For image JPEG2000, JBIG, MPEG for teleconferencing SKYPE, Flash
Arithmetic coding is used.
10/4/2020 6

7. Image/Video compression
• Intra frame coding and inter frame coding are two important concepts
involved in video coding.
• Intra frame coding In this, Data Reduction takes place within the frames. It
exploits redundancy within the frames. This effectively means using the spatial
correlations.
• inter frame coding Data Reduction takes place between the frames. It Exploits
redundancy between successive frames
10/4/2020 7

8. JPEG
• JPEG is a commonly used method of lossy compression for digital
images, particularly for those images produced by digital photography.
• JPEG compression reduces the file size with minimum image
degradation by removing the smallest possible amount of information.
• The degree of compression can be adjusted, allowing a selectable
tradeoff between storage size and image quality.
10/4/2020 8

9. MPEG
• It set standards for audio and video compression and transmission
• It has produced a number of digital media standards. Examples include:
• MPEG-1 – Audio/video standards designed for digital storage media (such as
an MP3 file)
• MPEG-2 – Standards for digital television and DVD video
• MPEG-4 – Multimedia standards for the computers, mobile devices, and the
web
10/4/2020 9

10. Channel Coding
• Channel coding deals with error control techniques. If the data at the
output of a communications system has errors that are too frequent for
the desired use, the errors can be reduced by the use of a number of
technique.
• Coding allows an increased rate of information transfer at a fixed error
rate for a fixed transfer rate.
10/4/2020 10

11. Automatic Repeat Request
• Automatic Repeat Request (ARQ) when a receiver circuit detects errors in
a block of data, it requests that the data is retransmitted.
• A feedback channel is necessary for this retransmission.
• ARQ is often used where there is a full duplex (2-
way) channel because it is relatively inexpensive to implement.
10/4/2020 11

12. Forward error correction
• the transmitted data is encoded so that the data can correct as well as detect
errors caused by channel noise.
• The channel encoder accepts these message bits and add redundant bits to
them leaving higher bit rate for transmission.
• FEC is used where the channel is not full duplex or where ARQ is not desirable
because real time is required.
10/4/2020 12

13. Transmission errors
• There are two types of transmission errors
(i) random error (ii) burst error.
1. Random errors are those that occur in a purely random manner.
BCH codes are useful in dealing with this sort of error.
2. Burst errors occur in forms of bunches and are not independent.
Convolution codes are not effective for this sort of errors.
10/4/2020 13

14. Block codes
• Block Codes are used to detect or correct errors.
• It accepts block of k information bits and produce block of n coded bits.
the n-k redundant bits are added to the k information bit to get n coded
bits, these codes are also related to (n, k) block codes.
10/4/2020 14

15. Convolution codes
• Convolution codes are error detecting code used to reliably transmit digital data
over unreliable communication system to channel noise.
• The code words are generated by discrete – time convolution of the input
sequence with impulse response of the encoder. Unlike block codes, channel
encoder accepts messages as a continuous sequence and generates a continuous
sequence of encoded bits at the output.
10/4/2020 15

16. Hamming codes
• Hamming codes can detect up to two simultaneous bit errors, and correct single-
bit errors; thus reliable communication is possible when the Hamming distance
between the transmitted and received bit patterns is less than or equal to one.
• It is suitable where burst error does not occur like transmission medium.
• It is widely used in computer memory because of simplicity of hamming codes.
10/4/2020 16

17. Turbo codes
• Turbo coding is an repeated soft-decoding scheme that combines two or
more relatively simple convolutional codes and an interleaver to produce a
block code that can perform to within a fraction of a decibel of the
Shannon limit.
• Turbo codes are used extensively in 3G and 4G mobile telephony
standards e.g. in HSPA, EV-DO and LTE. MediaFLO, terrestrial mobile
television system.
10/4/2020 17

18. Reed Solomon codes
• Reed Solomon codes designed this for the purpose to detect and correct error
multiple random symbol errors. Red Solomon add t to check symbols to the data
and detect combination of up to t incorrect symbols, and correct half of them as
t/2 symbols. It is reliable where burst error occurs.
• There are various applications such as CDs, DVDs and DSL & WiMAX in data
transmissions.
10/4/2020 18

19. Conclusion
• Information theory was created to find practical ways to make better, more
efficient codes and find the limits on how fast computers could process digital
signals. Every piece of digital information is the result of codes that have been
examined and improved using Shannon's equation.
• Coding applications have grown rapidly in the past several years with the cost
effective performance demonstrated to increase the efficiency and reliability
of communication.
10/4/2020 19

20. Reference
1. Sayood, K.,” Introduction to data compression”, Morgan Kaufmann Publishers, San
Francisco, CA. (Fourth Edition in 2013).
2. Muzhir Shaban AL-Ani , Fouad Hammadi Awad,” The Jpeg Image Compression A
Lgorithm”, International Journal of Advances in Engineering & Technology, May
2013.
3. Rubal Chaudhary1, Vrinda Gupta,” Error Control Techniques and Their
Applications”,IJCAES, Vol 1,issue 2, June2011.
4. C. E. Shannon, “A mathematical theory of communication,” Bell Syst. Tech. J. , vol.
27, pp. 379–423, 623–656, July–Oct-1948.
5. https://www.javatpoint.com/lossless-vs-lossy-data-compression
10/4/2020 20

21. THANK YOU
10/4/2020 21