This document contains formulas and definitions related to information theory and source coding. It defines key terms like entropy, average length, source/code efficiency, and redundancy. It also provides formulas to calculate these metrics for sources with different probabilities, and extended sources. Formulas are given for encoding sources using Huffman coding in ternary and quaternary systems.

1. INFORMATION THEORY AND
CODING
5th SEM E&C
JAYANTHDWIJESH H P M.tech (DECS)
Assistant Professor – Dept of E&C
B.G.S INSTITUTE OF TECHNOLOGY (B.G.S.I.T)
B.G Nagara, Nagamangala Tq, Mandya District- 571448

2. FORMULAS FOR REFERENCE
MODULE – 2 (source coding)
 Entropy of source or Average information content of the source.
H(S) = 𝑷𝒊 𝐥𝐨𝐠(
𝟏
𝑷 𝒊
𝒒
𝒊=𝟏 ) bits/symbol or H(S) = 𝑷 𝑲 𝐥𝐨𝐠 𝟐 𝟏 (
𝟏
𝑷 𝑲
𝑵
𝑲=𝟏 ) bits/symbol
 Average length
L = 𝑷𝒊
𝒒
𝒊=𝟏 𝒍𝒊 bits/symbol or L = 𝑷𝒊
𝑵
𝒊=𝟏 𝒍𝒊 bits/symbol
 Source or code efficiency
𝜼 𝑺=
𝑯(𝑺)
𝑳
X 𝟏𝟎𝟎% or 𝜼 𝑪=
𝑯(𝑺)
𝑳
X 𝟏𝟎𝟎%
 Source or code redundancy
𝑹 𝜼 𝑺
= 1- 𝜼 𝑺 = (1 -
𝑯(𝑺)
𝑳
) X 𝟏𝟎𝟎% or 𝑹 𝜼 𝑪
= 1- 𝜼 𝑪 = (1 -
𝑯(𝑺)
𝑳
) X 𝟏𝟎𝟎%
 Compute the number of stages required for the encoding operation, which is
given by
𝒏 =
𝑵−𝒓
𝒓−𝟏
or  =
𝒒−𝒓
𝒓−𝟏
 The probability of “0”s and “1”s and “2” s in the code are found using the
formulas
P (0) =
𝟏
𝑳
[𝐍𝐮𝐦𝐛𝐞𝐫 𝐨𝐟 "0" s in the code for
𝒒
𝒊=𝟏 𝑿𝒊 ] [𝒑𝒊 ] or
P (0) =
𝟏
𝑳
[𝐍𝐮𝐦𝐛𝐞𝐫 𝐨𝐟 "0" s in the code for𝑵
𝒊=𝟏 𝑿𝒊 ] [𝒑𝒊 ] .
P (1) =
𝟏
𝑳
[𝐍𝐮𝐦𝐛𝐞𝐫 𝐨𝐟 "1" s in the code for
𝒒
𝒊=𝟏 𝑿𝒊 ] [𝒑𝒊 ] or
P (1) =
𝟏
𝑳
[𝐍𝐮𝐦𝐛𝐞𝐫 𝐨𝐟 "1" s in the code for𝑵
𝒊=𝟏 𝑿𝒊 ] [𝒑𝒊 ] .
P (2) =
𝟏
𝑳
[𝐍𝐮𝐦𝐛𝐞𝐫 𝐨𝐟 "2" s in the code for
𝒒
𝒊=𝟏 𝑿𝒊 ] [𝒑𝒊 ] or
P (2) =
𝟏
𝑳
[𝐍𝐮𝐦𝐛𝐞𝐫 𝐨𝐟 "2" s in the code for𝑵
𝒊=𝟏 𝑿𝒊 ] [𝒑𝒊 ] .
 The variance of the word length is calculated from
Var ( 𝒍𝒊 ) = E [( 𝒍𝒊 − 𝐋 ) 𝟐
= 𝑷𝒊
𝒒
𝒊=𝟏 ( 𝒍𝒊 − 𝐋 ) 𝟐
 The Smallest integer value of 𝒍𝒊 if found using
𝟐 𝒍 𝒊 
𝟏
𝑷𝒊
or 𝒍𝒊  𝒍𝒐𝒈 𝟐
𝟏
𝑷𝒊
 The average length 𝑳 𝟐 of the 2nd
extension is given by
𝑳 𝟐 = 𝑷𝒊
𝒒
𝒊=𝟏 𝒍𝒊 bits/symbol or 𝑳 𝟐 = 𝑷𝒊
𝑵
𝒊=𝟏 𝒍𝒊 bits/symbol

3.  The average length 𝑳 𝟑 of the 3rd
extension is given by
𝑳 𝟑 = 𝑷𝒊
𝒒
𝒊=𝟏 𝒍𝒊 bits/symbol or 𝑳 𝟑 = 𝑷𝒊
𝑵
𝒊=𝟏 𝒍𝒊 bits/symbol
 The entropy of the 2nd extended
source is calculated as
H (𝑺 𝟐
) = 2 H(S)
 The entropy of the 3rd extended
source is calculated as
H (𝑺 𝟑
) = 3H(S)
 Source or code efficiency of the 2nd
extended source is
𝜼(𝟐)
𝑺
=
𝐇 (𝑺 𝟐
)
𝑳 𝟐
X 𝟏𝟎𝟎% or 𝜼(𝟐)
𝑪
=
𝐇 (𝑺 𝟐
)
𝑳 𝟐
X 𝟏𝟎𝟎%
 Source or code redundancy of the 2nd
extended source is
𝑹(𝟐)
𝜼 𝑺
= 1- 𝜼(𝟐)
𝑺
= (1 -
𝐇 (𝑺 𝟐)
𝑳 𝟐
) X 𝟏𝟎𝟎% or 𝑹(𝟐)
𝜼 𝑪
= 𝟏 − 𝜼(𝟐)
𝑪
= (1 -
𝐇 (𝑺 𝟐)
𝑳 𝟐
)
𝐗 𝟏𝟎𝟎%
 Source or code efficiency of the 3rd
extended source is
𝜼(𝟑)
𝑺
=
𝐇 (𝑺 𝟑
)
𝑳 𝟑
X 𝟏𝟎𝟎% or 𝜼(𝟑)
𝑪
=
𝐇 (𝑺 𝟑
)
𝟑
X 𝟏𝟎𝟎%
 Source or code redundancy of the 3rd
extended source is
𝑹(𝟑)
𝜼 𝑺
= 1- 𝜼(𝟑)
𝑺
= (1 -
𝐇 (𝑺 𝟑)
𝑳 𝟐
) X 𝟏𝟎𝟎% or 𝑹(𝟑)
𝜼 𝑪
= 𝟏 − 𝜼(𝟑)
𝑪
= (1 -
𝐇 (𝑺 𝟑)
𝑳 𝟑
)
𝐗 𝟏𝟎𝟎%
 The average length 𝑳 𝟑
of the Huffman ternary code is given by
𝑳(𝟑)
= 𝑷𝒊
𝒒
𝒊=𝟏 𝒍𝒊 trinits /Msg- symbol or 𝑳(𝟑)
= 𝑷𝒊
𝑵
𝒊=𝟏 𝒍𝒊 trinits / Msg- symbol
 The average length 𝑳 𝟒
of the Huffman quaternary code is given by
𝑳(𝟒)
= 𝑷𝒊
𝒒
𝒊=𝟏 𝒍𝒊 quaternary digits /Msg- symbol or
𝑳(𝟒)
= 𝑷𝒊
𝑵
𝒊=𝟏 𝒍𝒊 quaternary digits / Msg- symbol
 The entropy in ternary units/ message symbol is found by using equation
𝐇 𝟑(S) =
𝐇(𝐒)
𝒍𝒐𝒈 𝟐 𝟑
ternary units/ message symbol or
𝐇 𝟑(S) = 𝑷 𝑲 𝐥𝐨𝐠 𝟑 𝟏 (
𝟏
𝑷 𝑲
𝑵
𝑲=𝟏 ) ternary units/ message symbol or
𝐇 𝟑(S) = 𝑷𝒊 𝐥𝐨𝐠 𝟑 𝟏 (
𝟏
𝑷 𝒊
𝒒
𝒊=𝟏 ) ternary units/ message symbol
 The entropy in quaternary units/ message symbol is found by using equation
𝐇 𝟒(S) =
𝐇(𝐒)
𝒍𝒐𝒈 𝟐 𝟒
quaternary units/ message symbol or

4. 𝐇 𝟒(S) = 𝑷 𝑲 𝐥𝐨𝐠 𝟒 𝟏 (
𝟏
𝑷 𝑲
𝑵
𝑲=𝟏 ) quaternary units/ message symbol or
𝐇 𝟒(S) = 𝑷𝒊 𝐥𝐨𝐠 𝟒 𝟏 (
𝟏
𝑷 𝒊
𝒒
𝒊=𝟏 ) quaternary units/ message symbol
 Source or code efficiency of the ternary is given by
𝜼 𝒔(𝟑)
=
𝐇 𝟑(𝐒)
𝑳(𝟑)
X 𝟏𝟎𝟎% or 𝜼 𝒄(𝟑)
=
𝐇 𝟑(𝐒)
𝑳(𝟑)
X 𝟏𝟎𝟎% or
𝜼 𝑺=
𝐇 𝟑(𝐒)
𝑳
X 𝟏𝟎𝟎% or 𝜼 𝑪=
𝐇 𝟑(𝐒)
𝑳
X 𝟏𝟎𝟎%
 Source or code efficiency of the quaternary is given by
𝜼 𝒔(𝟒)
=
𝐇 𝟒(𝐒)
𝑳(𝟒)
X 𝟏𝟎𝟎% or 𝜼 𝒄(𝟒)
=
𝐇 𝟒(𝐒)
𝑳(𝟒)
X 𝟏𝟎𝟎% or
𝜼 𝑺=
𝐇 𝟒(𝐒)
𝑳
X 𝟏𝟎𝟎% or 𝜼 𝑪=
𝐇 𝟒(𝐒)
𝑳
X 𝟏𝟎𝟎%
 Source or code redundancy of the ternary is given by
𝑹 𝜼 𝒔(𝟑)
= 1- 𝜼 𝒔(𝟑)
= (1 -
𝐇 𝟑(𝐒)
𝑳(𝟑) ) X 𝟏𝟎𝟎% or
𝑹 𝜼 𝒄(𝟑)
= 𝟏 − 𝜼 𝒄(𝟑)
= (1 -
𝐇 𝟑(𝐒)
𝑳(𝟑) ) 𝐗 𝟏𝟎𝟎% or
𝑹 𝜼 𝑺
= 1- 𝜼 𝑺 = (1 -
𝐇 𝟑(𝐒)
𝑳
) X 𝟏𝟎𝟎% or 𝑹 𝜼 𝑪
= 1- 𝜼 𝑪 = (1 -
𝐇 𝟑(𝐒)
𝑳
) X 𝟏𝟎𝟎%
 Source or code redundancy of the quaternary is given by
𝑹 𝜼 𝒔(𝟒)
= 1- 𝜼 𝒔(𝟒)
= (1 -
𝐇 𝟒(𝐒)
𝑳(𝟒) ) X 𝟏𝟎𝟎% or
𝑹 𝜼 𝒄(𝟒)
= 𝟏 − 𝜼 𝒄(𝟒)
= (1 -
𝐇 𝟒(𝐒)
𝑳(𝟒) ) 𝐗 𝟏𝟎𝟎% or
𝑹 𝜼 𝑺
= 1- 𝜼 𝑺 = (1 -
𝐇 𝟒(𝐒)
𝑳
) X 𝟏𝟎𝟎% or 𝑹 𝜼 𝑪
= 1- 𝜼 𝑪 = (1 -
𝐇 𝟒(𝐒)
𝑳
) X 𝟏𝟎𝟎%