Jayanth Dwijesh H P, profile picture
Uploaded byJayanth Dwijesh H P
447 views

VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).

This document provides formulas and concepts related to information theory and coding. It defines key information theory terms like self-information, entropy, information rate, bits, Hartley's, nats, extremal entropy, source efficiency, and source redundancy. Formulas are given for calculating each of these terms. The document also provides formulas for calculating the average information content of symbols from different states. Finally, it lists some basic log properties that are important in information theory calculations.

Embed presentation

Downloaded 20 times
1 / 158
2 / 158
3 / 158
4 / 158
5 / 158
6 / 158
7 / 158
8 / 158
9 / 158
10 / 158
11 / 158
12 / 158
13 / 158
14 / 158
15 / 158
16 / 158
17 / 158
18 / 158
19 / 158
20 / 158
21 / 158
22 / 158
23 / 158
24 / 158
25 / 158
26 / 158
27 / 158
28 / 158
29 / 158
30 / 158
31 / 158
32 / 158
33 / 158
34 / 158
35 / 158
36 / 158
37 / 158
38 / 158
39 / 158
40 / 158
41 / 158
42 / 158
43 / 158
44 / 158
45 / 158
46 / 158
47 / 158
48 / 158
49 / 158
50 / 158
51 / 158
52 / 158
53 / 158
54 / 158
55 / 158
56 / 158
57 / 158
58 / 158
59 / 158
60 / 158
61 / 158
62 / 158
63 / 158
64 / 158
65 / 158
66 / 158
67 / 158
68 / 158
69 / 158
70 / 158
71 / 158
72 / 158
73 / 158
74 / 158
75 / 158
76 / 158
77 / 158
78 / 158
79 / 158
80 / 158
81 / 158
82 / 158
83 / 158
84 / 158
85 / 158
86 / 158
87 / 158
88 / 158
89 / 158
90 / 158
91 / 158
92 / 158
93 / 158
94 / 158
95 / 158
96 / 158
97 / 158
98 / 158
99 / 158
100 / 158
101 / 158
102 / 158
103 / 158
104 / 158
105 / 158
106 / 158
107 / 158
108 / 158
109 / 158
110 / 158
111 / 158
112 / 158
113 / 158
114 / 158
115 / 158
116 / 158
117 / 158
118 / 158
119 / 158
120 / 158
121 / 158
122 / 158
123 / 158
124 / 158
125 / 158
126 / 158
127 / 158
128 / 158
129 / 158
130 / 158
131 / 158
132 / 158
133 / 158
134 / 158
135 / 158
136 / 158
137 / 158
138 / 158
139 / 158
140 / 158
141 / 158
142 / 158
143 / 158
144 / 158
145 / 158
146 / 158
147 / 158
148 / 158
149 / 158
150 / 158
151 / 158
152 / 158
153 / 158
154 / 158
155 / 158
156 / 158
157 / 158
158 / 158
INFORMATION THEORY AND CODING 5th SEM E&C JAYANTHDWIJESH H P M.tech (DECS) Assistant Professor – Dept of E&C B.G.S INSTITUTE OF TECHNOLOGY (B.G.S.I.T) B.G Nagara, Nagamangala Tq, Mandya District- 571448
FORMULAS FOR REFERENCE MODULE –1(INFORMATION THEORY)  Amount of information or Self information. 𝑰 𝑲 = log ( 𝟏 𝑷 𝑲 ) or 𝑰 𝑲 = 𝐥𝐨𝐠 𝟐 𝟏( 𝟏 𝑷 𝑲 ) or I (𝒎 𝑲) = log ( 𝟏 𝑷 𝑲 )  Entropy of source or Average information content of the source. H = 𝑷𝒊 𝐥𝐨𝐠 𝟐 𝟏 ( 𝟏 𝑷 𝒊 𝑴 𝒊=𝟏 ) bits/symbol or H = 𝑷 𝑲 𝐥𝐨𝐠 𝟐 𝟏 ( 𝟏 𝑷 𝑲 𝑴 𝑲=𝟏 ) bits/symbol or H(S) = 𝑷𝒊 𝐥𝐨𝐠 𝟐 𝟏 ( 𝟏 𝑷 𝒊 𝒒 𝒊=𝟏 ) bits/symbol or H(S) = 𝑷𝒊 𝐥𝐨𝐠( 𝟏 𝑷 𝒊 𝒒 𝒊=𝟏 ) bits/symbol or H(S) = 𝑷 𝑲 𝐥𝐨𝐠 𝟐 𝟏 ( 𝟏 𝑷 𝑲 𝑵 𝑲=𝟏 ) bits/symbol  Information rate or average information rate. 𝑹 𝑺 = 𝒓 𝒔H(S) bits/sec or R= 𝒓 𝒔H bits/sec or R=r H bits/sec  Bits. 𝑰 𝑲 = 𝐥𝐨𝐠 𝟐 𝟏( 𝟏 𝑷 𝑲 ) bits  Hartley’s or Decits. 𝑰 𝑲 = 𝐥𝐨𝐠 𝟏𝟎 𝟏( 𝟏 𝑷 𝑲 ) Hartley’s or Decits  Nats or Neper. 𝑰 𝑲 = 𝐥𝐨𝐠 𝒆 𝟏( 𝟏 𝑷 𝑲 ) Nats or Neper.  Extremal or Upper bound or Maximum entropy 𝑯(𝑺) 𝒎𝒂𝒙 = 𝐥𝐨𝐠 𝟐 𝒒 bits/message-symbol or 𝑯(𝑺) 𝒎𝒂𝒙 = 𝐥𝐨𝐠 𝟐 𝑵 bits/message-symbol.  Source efficiency 𝜼 𝑺= 𝑯(𝑺) 𝑯(𝑺) 𝒎𝒂𝒙 or 𝜼 𝑺= 𝑯(𝑺) 𝑯(𝑺) 𝒎𝒂𝒙 X 𝟏𝟎𝟎%  Source redundancy 𝑹 𝜼 𝑺 = 1- 𝜼 𝑺 = (1 - 𝑯(𝑺) 𝑯(𝑺) 𝒎𝒂𝒙 ) X 𝟏𝟎𝟎%  The average information content of the symbols emitted from the i th state. 𝑯𝒊= 𝑷𝒊𝒋 𝐥𝐨𝐠 𝟐 𝟏 ( 𝟏 𝑷𝒊𝒋 𝒏 𝒋=𝟏 ) bits/symbol or 𝑯𝒊= 𝑷𝒊𝒋 𝐥𝐨𝐠 𝒐 𝟏 ( 𝟏 𝑷𝒊𝒋 𝒏 𝒋=𝟏 ) bits/symbol  The average information content of the symbols emitted from the k th state.
𝑯 𝒌= 𝑷𝒍𝑲 𝐥𝐨𝐠 𝟐 𝟏 ( 𝟏 𝑷𝒍𝑲 𝑴 𝒍=𝟏 ) bits/symbol  The average information content per symbol in a message of length N. 𝑮 𝑵= 𝟏 𝑵 𝑷(𝒊 𝒎𝒊)log 𝟏 𝑷(𝒎 𝒊) or 𝑮 𝑵= − 𝟏 𝑵 𝑷(𝒊 𝒎𝒊)log P (𝒎𝒊) = 𝟏 𝑵 H ( 𝒔)  The entropy of the second order symbols. 𝑮 𝑵 = 𝟏 𝑵 H ( 𝒔 𝒙 𝑵 ) where N=2.  The entropy of the third order symbols. 𝑮 𝑵 = 𝟏 𝑵 H ( 𝒔 𝒙 𝑵 ) where N=3.  Log properties 1. 𝐥𝐨𝐠 𝒂 𝒃 = 𝟏 𝐥𝐨𝐠 𝒃 𝒂 𝟐. 𝐥𝐨𝐠 𝒙 𝒃 𝐥𝐨𝐠 𝒙 𝒂 = 𝐥𝐨𝐠 𝒂 𝒃 𝟑. 𝐥𝐨𝐠 𝒆 𝟏𝟎 = ln (10)
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).

More Related Content

PDF
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-2 notes(15...
byJayanth Dwijesh H P
 
PPT
Channel Coding (Error Control Coding)
byOla Mashaqi @ an-najah national university
 
PDF
Neural Network
byAshish Kumar
 
PPT
Introduction to Genetic Algorithms
byPremsankar Chakkingal
 
PPTX
Convolutional Error Control Coding
byMohammed Abuibaid
 
PPTX
Shannon Fano
byanithabalaprabhu
 
PPTX
Presentation of daa on approximation algorithm and vertex cover problem
bysumit gyawali
 
PDF
4.Sampling and Hilbert Transform
byINDIAN NAVY
 
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-2 notes(15...
byJayanth Dwijesh H P
 
Channel Coding (Error Control Coding)
byOla Mashaqi @ an-najah national university
 
Neural Network
byAshish Kumar
 
Introduction to Genetic Algorithms
byPremsankar Chakkingal
 
Convolutional Error Control Coding
byMohammed Abuibaid
 
Shannon Fano
byanithabalaprabhu
 
Presentation of daa on approximation algorithm and vertex cover problem
bysumit gyawali
 
4.Sampling and Hilbert Transform
byINDIAN NAVY
 

What's hot

PDF
[기초개념] Graph Convolutional Network (GCN)
byDonghyeon Kim
 
PPTX
15934 product modulator
byManish Kumar
 
PPTX
Introduction to Graph neural networks @ Vienna Deep Learning meetup
byLiad Magen
 
PDF
Information theory
byMadhumita Tamhane
 
PDF
Deep learning
byKuppusamy P
 
PDF
Gradient-based optimization for Deep Learning: a short introduction
byChristian Perone
 
PPT
Learning sets of rules, Sequential Learning Algorithm,FOIL
byPavithra Thippanaik
 
PDF
Error control coding bch, reed-solomon etc..
byMadhumita Tamhane
 
PPTX
Applications of Information Theory
byDarshan Bhatt
 
PDF
Communication Theory - Random Process.pdf
byRajaSekaran923497
 
PPTX
Red Deer Algorithm (RDA)
byAmir Mohammad Fathollahi Fard
 
PDF
Cnn backpropagation derivation
byPunnoose A.K
 
PPT
RESEAUX DE DISTRIBUTION PUBLIQUE HTA.ppt
byOubella M'hand
 
PPT
Discrete Signal Processing
bymargretrosy
 
PDF
Digital Signal Processing[ECEG-3171]-Ch1_L02
byRediet Moges
 
PDF
Cours Transformée de Laplace.pdf
bySohaMoussaoui
 
PDF
Artificial Neural Networks Lect5: Multi-Layer Perceptron & Backpropagation
byMohammed Bennamoun
 
PPTX
CNN
bychs71
 
PPTX
Longest Common Subsequence
byKrishma Parekh
 
PPT
Basics & asymptotic notations
byRajendran
 
[기초개념] Graph Convolutional Network (GCN)
byDonghyeon Kim
 
15934 product modulator
byManish Kumar
 
Introduction to Graph neural networks @ Vienna Deep Learning meetup
byLiad Magen
 
Information theory
byMadhumita Tamhane
 
Deep learning
byKuppusamy P
 
Gradient-based optimization for Deep Learning: a short introduction
byChristian Perone
 
Learning sets of rules, Sequential Learning Algorithm,FOIL
byPavithra Thippanaik
 
Error control coding bch, reed-solomon etc..
byMadhumita Tamhane
 
Applications of Information Theory
byDarshan Bhatt
 
Communication Theory - Random Process.pdf
byRajaSekaran923497
 
Red Deer Algorithm (RDA)
byAmir Mohammad Fathollahi Fard
 
Cnn backpropagation derivation
byPunnoose A.K
 
RESEAUX DE DISTRIBUTION PUBLIQUE HTA.ppt
byOubella M'hand
 
Discrete Signal Processing
bymargretrosy
 
Digital Signal Processing[ECEG-3171]-Ch1_L02
byRediet Moges
 
Cours Transformée de Laplace.pdf
bySohaMoussaoui
 
Artificial Neural Networks Lect5: Multi-Layer Perceptron & Backpropagation
byMohammed Bennamoun
 
CNN
bychs71
 
Longest Common Subsequence
byKrishma Parekh
 
Basics & asymptotic notations
byRajendran
 

Similar to VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).

PDF
Information theory
byMadhumita Tamhane
 
PPT
Digital Communication: Information Theory
byDr. Sanjay M. Gulhane
 
PPTX
6.1-Information Sources and Entropy.pptx
byDipakMahurkar1
 
PDF
Information theory
byAMIE(I) Study Circle
 
PDF
Tele4653 l9
byVin Voro
 
PDF
UNIT-2.pdf
byMatrusri Engineering College
 
PPTX
Information Theory and coding - Lecture 2
byAref35
 
PPTX
Information Theory Coding 1
byMahafuz Aveek
 
PPTX
1.Introduction on information measures and entropy
byIRTSDRDARMAWATYMOHDA
 
PDF
Lecture Information Theory and CodingPart-1.pdf
bymsc9219
 
PDF
Information_Theory_and_Coding_ITC_CSE.pdf
bykillerdew00
 
PDF
VTU CBCS E&C 5th sem Information theory and coding(15EC54) Module -2notes
byJayanth Dwijesh H P
 
PPTX
Information theory & coding PPT Full Syllabus.pptx
byprernaguptaec
 
PPTX
UNIT-1.pptx for Information Theory and Coding
byBala Vishnu J
 
PDF
VTU CBCS E&C 5th sem Information theory and coding(15EC54) Module -3 notes
byJayanth Dwijesh H P
 
PDF
VTU E&C,TCE CBCS[NEW]5th Sem Information Theory and Coding Module-3 notes(15&...
byJayanth Dwijesh H P
 
PDF
Dcs unit 2
byAnil Nigam
 
PPT
DC Lecture Slides 1 - Information Theory.ppt
byshortstime400
 
PDF
Information Theory Mike Brookes E4.40, ISE4.51, SO20.pdf
bySaad Falh
 
PDF
sssssssssssssssssssssssssssssssssssssssssssssssssssssssss
bytejaravi1611
 
Information theory
byMadhumita Tamhane
 
Digital Communication: Information Theory
byDr. Sanjay M. Gulhane
 
6.1-Information Sources and Entropy.pptx
byDipakMahurkar1
 
Information theory
byAMIE(I) Study Circle
 
Tele4653 l9
byVin Voro
 
UNIT-2.pdf
byMatrusri Engineering College
 
Information Theory and coding - Lecture 2
byAref35
 
Information Theory Coding 1
byMahafuz Aveek
 
1.Introduction on information measures and entropy
byIRTSDRDARMAWATYMOHDA
 
Lecture Information Theory and CodingPart-1.pdf
bymsc9219
 
Information_Theory_and_Coding_ITC_CSE.pdf
bykillerdew00
 
VTU CBCS E&C 5th sem Information theory and coding(15EC54) Module -2notes
byJayanth Dwijesh H P
 
Information theory & coding PPT Full Syllabus.pptx
byprernaguptaec
 
UNIT-1.pptx for Information Theory and Coding
byBala Vishnu J
 
VTU CBCS E&C 5th sem Information theory and coding(15EC54) Module -3 notes
byJayanth Dwijesh H P
 
VTU E&C,TCE CBCS[NEW]5th Sem Information Theory and Coding Module-3 notes(15&...
byJayanth Dwijesh H P
 
Dcs unit 2
byAnil Nigam
 
DC Lecture Slides 1 - Information Theory.ppt
byshortstime400
 
Information Theory Mike Brookes E4.40, ISE4.51, SO20.pdf
bySaad Falh
 
sssssssssssssssssssssssssssssssssssssssssssssssssssssssss
bytejaravi1611
 

More from Jayanth Dwijesh H P

PDF
Network and cyber security module(15ec835, 17ec835)
byJayanth Dwijesh H P
 
PDF
VTU Network & cyber security (1-5 Module) Full notes
byJayanth Dwijesh H P
 
PDF
Vtu network security(10 ec832) unit 3 notes.
byJayanth Dwijesh H P
 
PDF
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-5 notes(15...
byJayanth Dwijesh H P
 
PDF
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-4 notes(15...
byJayanth Dwijesh H P
 
PDF
Network security 10EC832 vtu notes
byJayanth Dwijesh H P
 
PDF
VTU network security(10 ec832) unit 6 notes
byJayanth Dwijesh H P
 
PDF
Vtu network security(10 ec832) unit 5 notes.
byJayanth Dwijesh H P
 
PDF
Vtu network security(10 ec832) unit 2 notes..
byJayanth Dwijesh H P
 
PDF
Vtu network security(10 ec832) unit 1 notes
byJayanth Dwijesh H P
 
PDF
VTU CBCS E&C 5th sem Information theory and coding(15EC54) Module -1notes
byJayanth Dwijesh H P
 
PDF
VTU CBCS E&C 5th sem Information theory and coding(15EC54) Module -5 notes
byJayanth Dwijesh H P
 
PDF
VTU CBCS E&C 5th sem Information theory and coding(15EC54) Module -4 notes
byJayanth Dwijesh H P
 
Network and cyber security module(15ec835, 17ec835)
byJayanth Dwijesh H P
 
VTU Network & cyber security (1-5 Module) Full notes
byJayanth Dwijesh H P
 
Vtu network security(10 ec832) unit 3 notes.
byJayanth Dwijesh H P
 
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-5 notes(15...
byJayanth Dwijesh H P
 
VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-4 notes(15...
byJayanth Dwijesh H P
 
Network security 10EC832 vtu notes
byJayanth Dwijesh H P
 
VTU network security(10 ec832) unit 6 notes
byJayanth Dwijesh H P
 
Vtu network security(10 ec832) unit 5 notes.
byJayanth Dwijesh H P
 
Vtu network security(10 ec832) unit 2 notes..
byJayanth Dwijesh H P
 
Vtu network security(10 ec832) unit 1 notes
byJayanth Dwijesh H P
 
VTU CBCS E&C 5th sem Information theory and coding(15EC54) Module -1notes
byJayanth Dwijesh H P
 
VTU CBCS E&C 5th sem Information theory and coding(15EC54) Module -5 notes
byJayanth Dwijesh H P
 
VTU CBCS E&C 5th sem Information theory and coding(15EC54) Module -4 notes
byJayanth Dwijesh H P
 

Recently uploaded

PDF
RVV Basic Terms and Analogy - (en/zhTW) - DannyJiang
byDanny Jiang
 
PDF
Taykon Proses -ENG- Process Equipment Manufacturer
byTAYKON
 
PDF
Requirement Engineering : Capturing the Requirements
byGunjalSanjay
 
PPTX
Industry 4.0- fourth industrial revolution Presentation.pptx
byrameswari15491
 
PDF
Chip Designer's Code - Linux Terminal Part 7.1 - Text Processing
byAmr Adel
 
PPTX
Next-Gen Satellite Imaging: Optimizing High-Resolution Remote Sensing with Ad...
byraghabBsingh
 
PPTX
Why Air Droppable Containers Are Critical for Modern Aerial Logistics
byNeometrix_Engineering_Pvt_Ltd
 
PDF
Formality - Logic Equivalence Checking - Part 2
byAmr Adel
 
PPTX
FROM MEASUREMENT TO MEANING : Metrology-Driven Battery Testing for Reliable ...
byRunjhunDutta
 
PPTX
24ME402-ENGINEERING MATERIALS AND METALLURGY --- UNIT-I
byDJAGADEESH1
 
PPTX
TechSprint WinterHack || After an intense hackathon filled with creativity, ...
bybajpaitusharoffon678
 
PDF
LS029B3SX06A 2.9 Inch TFT LCD 2K Resolution 2160X2160 For AR VR
bySyluxdisplay
 
PDF
Formality - Logic Equivalence Checking - Part 1
byAmr Adel
 
PDF
Introduction to Software , Product and Process
byGunjalSanjay
 
PPTX
battery energy storage system (BESS).pptx
bymanish99skp
 
PPTX
Filtration-slow sand filtration and rapid sand filtration.pptx
byChemical Engineering Dept. NIT Rourkela-769008, Odisha, India
 
PDF
Fundamentals of AI - Problem Solving Approach.pdf
byVyankateshBhaurkar1
 
PDF
Software Engineering Perspective Process Models
byGunjalSanjay
 
PPTX
1.-Static-sensor-characteristics,sensors and actuators ;SDU,denmark.pptx
byMohammadTabish20
 
PPTX
Energy system modelling to support national energy and climate policy making ...
byIEA-ETSAP
 
RVV Basic Terms and Analogy - (en/zhTW) - DannyJiang
byDanny Jiang
 
Taykon Proses -ENG- Process Equipment Manufacturer
byTAYKON
 
Requirement Engineering : Capturing the Requirements
byGunjalSanjay
 
Industry 4.0- fourth industrial revolution Presentation.pptx
byrameswari15491
 
Chip Designer's Code - Linux Terminal Part 7.1 - Text Processing
byAmr Adel
 
Next-Gen Satellite Imaging: Optimizing High-Resolution Remote Sensing with Ad...
byraghabBsingh
 
Why Air Droppable Containers Are Critical for Modern Aerial Logistics
byNeometrix_Engineering_Pvt_Ltd
 
Formality - Logic Equivalence Checking - Part 2
byAmr Adel
 
FROM MEASUREMENT TO MEANING : Metrology-Driven Battery Testing for Reliable ...
byRunjhunDutta
 
24ME402-ENGINEERING MATERIALS AND METALLURGY --- UNIT-I
byDJAGADEESH1
 
TechSprint WinterHack || After an intense hackathon filled with creativity, ...
bybajpaitusharoffon678
 
LS029B3SX06A 2.9 Inch TFT LCD 2K Resolution 2160X2160 For AR VR
bySyluxdisplay
 
Formality - Logic Equivalence Checking - Part 1
byAmr Adel
 
Introduction to Software , Product and Process
byGunjalSanjay
 
battery energy storage system (BESS).pptx
bymanish99skp
 
Filtration-slow sand filtration and rapid sand filtration.pptx
byChemical Engineering Dept. NIT Rourkela-769008, Odisha, India
 
Fundamentals of AI - Problem Solving Approach.pdf
byVyankateshBhaurkar1
 
Software Engineering Perspective Process Models
byGunjalSanjay
 
1.-Static-sensor-characteristics,sensors and actuators ;SDU,denmark.pptx
byMohammadTabish20
 
Energy system modelling to support national energy and climate policy making ...
byIEA-ETSAP
 

VTU E&C,TCE CBCS[NEW] 5th Sem Information Theory and Coding Module-1 notes(15&17scheme).

  • 1.
    INFORMATION THEORY AND CODING 5thSEM E&C JAYANTHDWIJESH H P M.tech (DECS) Assistant Professor – Dept of E&C B.G.S INSTITUTE OF TECHNOLOGY (B.G.S.I.T) B.G Nagara, Nagamangala Tq, Mandya District- 571448
  • 2.
    FORMULAS FOR REFERENCE MODULE–1(INFORMATION THEORY)  Amount of information or Self information. 𝑰 𝑲 = log ( 𝟏 𝑷 𝑲 ) or 𝑰 𝑲 = 𝐥𝐨𝐠 𝟐 𝟏( 𝟏 𝑷 𝑲 ) or I (𝒎 𝑲) = log ( 𝟏 𝑷 𝑲 )  Entropy of source or Average information content of the source. H = 𝑷𝒊 𝐥𝐨𝐠 𝟐 𝟏 ( 𝟏 𝑷 𝒊 𝑴 𝒊=𝟏 ) bits/symbol or H = 𝑷 𝑲 𝐥𝐨𝐠 𝟐 𝟏 ( 𝟏 𝑷 𝑲 𝑴 𝑲=𝟏 ) bits/symbol or H(S) = 𝑷𝒊 𝐥𝐨𝐠 𝟐 𝟏 ( 𝟏 𝑷 𝒊 𝒒 𝒊=𝟏 ) bits/symbol or H(S) = 𝑷𝒊 𝐥𝐨𝐠( 𝟏 𝑷 𝒊 𝒒 𝒊=𝟏 ) bits/symbol or H(S) = 𝑷 𝑲 𝐥𝐨𝐠 𝟐 𝟏 ( 𝟏 𝑷 𝑲 𝑵 𝑲=𝟏 ) bits/symbol  Information rate or average information rate. 𝑹 𝑺 = 𝒓 𝒔H(S) bits/sec or R= 𝒓 𝒔H bits/sec or R=r H bits/sec  Bits. 𝑰 𝑲 = 𝐥𝐨𝐠 𝟐 𝟏( 𝟏 𝑷 𝑲 ) bits  Hartley’s or Decits. 𝑰 𝑲 = 𝐥𝐨𝐠 𝟏𝟎 𝟏( 𝟏 𝑷 𝑲 ) Hartley’s or Decits  Nats or Neper. 𝑰 𝑲 = 𝐥𝐨𝐠 𝒆 𝟏( 𝟏 𝑷 𝑲 ) Nats or Neper.  Extremal or Upper bound or Maximum entropy 𝑯(𝑺) 𝒎𝒂𝒙 = 𝐥𝐨𝐠 𝟐 𝒒 bits/message-symbol or 𝑯(𝑺) 𝒎𝒂𝒙 = 𝐥𝐨𝐠 𝟐 𝑵 bits/message-symbol.  Source efficiency 𝜼 𝑺= 𝑯(𝑺) 𝑯(𝑺) 𝒎𝒂𝒙 or 𝜼 𝑺= 𝑯(𝑺) 𝑯(𝑺) 𝒎𝒂𝒙 X 𝟏𝟎𝟎%  Source redundancy 𝑹 𝜼 𝑺 = 1- 𝜼 𝑺 = (1 - 𝑯(𝑺) 𝑯(𝑺) 𝒎𝒂𝒙 ) X 𝟏𝟎𝟎%  The average information content of the symbols emitted from the i th state. 𝑯𝒊= 𝑷𝒊𝒋 𝐥𝐨𝐠 𝟐 𝟏 ( 𝟏 𝑷𝒊𝒋 𝒏 𝒋=𝟏 ) bits/symbol or 𝑯𝒊= 𝑷𝒊𝒋 𝐥𝐨𝐠 𝒐 𝟏 ( 𝟏 𝑷𝒊𝒋 𝒏 𝒋=𝟏 ) bits/symbol  The average information content of the symbols emitted from the k th state.
  • 3.
    𝑯 𝒌= 𝑷𝒍𝑲𝐥𝐨𝐠 𝟐 𝟏 ( 𝟏 𝑷𝒍𝑲 𝑴 𝒍=𝟏 ) bits/symbol  The average information content per symbol in a message of length N. 𝑮 𝑵= 𝟏 𝑵 𝑷(𝒊 𝒎𝒊)log 𝟏 𝑷(𝒎 𝒊) or 𝑮 𝑵= − 𝟏 𝑵 𝑷(𝒊 𝒎𝒊)log P (𝒎𝒊) = 𝟏 𝑵 H ( 𝒔)  The entropy of the second order symbols. 𝑮 𝑵 = 𝟏 𝑵 H ( 𝒔 𝒙 𝑵 ) where N=2.  The entropy of the third order symbols. 𝑮 𝑵 = 𝟏 𝑵 H ( 𝒔 𝒙 𝑵 ) where N=3.  Log properties 1. 𝐥𝐨𝐠 𝒂 𝒃 = 𝟏 𝐥𝐨𝐠 𝒃 𝒂 𝟐. 𝐥𝐨𝐠 𝒙 𝒃 𝐥𝐨𝐠 𝒙 𝒂 = 𝐥𝐨𝐠 𝒂 𝒃 𝟑. 𝐥𝐨𝐠 𝒆 𝟏𝟎 = ln (10)