The document discusses various topics related to source and error control coding including information theory, entropy, channel capacity, Shannon-Fano coding, Huffman coding, mutual information, linear block codes, cyclic codes, and ARQ techniques. Specific examples are provided on calculating entropy, mutual information, channel capacity, and comparing the efficiency of Shannon-Fano and Huffman codes for a given data source. Trellis and state diagrams are shown as examples of error coding techniques.

1. UNIT IV
SOURCE AND ERROR CONTROL CODING
Entropy, Source encoding theorem,
Shannon fano coding, Huffman coding,
mutual information, channel capacity,
Error Control Coding, linear block codes,
cyclic codes - ARQ Techniques.

2. UNIT IV
SOURCE AND ERROR CONTROL CODING
Information Theory
Mathematical modeling and analysis of a communication
system
Signal Compression and data rate
Entropy and channel capacity
Entropy – Probabilistic behavior of source
Capacity – Ability to convey information

3. Mutual Information
I(x,y) = H(x)-H(x/y)
I(y,x) = H(y)-H(y/x)
Properties
1. Symmetric ie I(x,y) = I(y,x)
2. Always non negative
3. I(x,y) = H(x)+H(y)-H(x,y)
UNIT IV
SOURCE AND ERROR CONTROL CODING

4. Channel Capacity
Maximum mutual information in any single use of a channel
C= maxP(xj) I(x,y)
UNIT IV
SOURCE AND ERROR CONTROL CODING

5. Automatic Repeat Request
Stop and Wait
Continuous ARQ with pullback
Continuous ARQ with selective repeat
UNIT IV
SOURCE AND ERROR CONTROL CODING

21. Trellis diagram
a a
b b
c c
d d
m2,m
1
m X1=m X2=
m+m1+m2
X3=
m+m2
m1,
m
00 a
0 0 0+0+0= 0 0+0=0 00 a
1 1 1+0+0=1 1+0=1 01 b
01 b
0 0 1 0 10 c
1 1 0 1 11 d
10 c
0 0 1 1 00 a
1 1 0 0 01 b
11 d
0 0 0 1 10 c
1 1 1 0 11 d

22. State Diagram
b
a d
c
000
001
010
011
100
101
110
111

23. a
a
b
0
1
c
d
0
1
a
b
0
1
a
b
0
1
c
d
0
1
a
b
0
1
c
d
0
1

24. Entropy = pklog2(1/pk) where k ranges 1 to 6
= p1log2(1/p1) + p2log2(1/p2) + p3log2(1/p3) +….
p6log2(1/p6)
= 0.3log2(1/0.3) + 0.1log2(1/0.1) +
I R = H* No of symbols
= H*500
= H*9600

25. Find code efficiency using SF and HM code for the source messages m1= 0.4,
m2= 0.15, m3=0.15, m4=0.15, m5 = 0.15
SF CODE
STEP 1 STEP 2 STEP 3 STEP 4
M1 0.4 M1 0.4 0.4 0 0
M2 0.15 M2 0.15 0.15 1 0.15 0 0.15 0 100
M3 0.15 M3 0.15 0.15 1 0.15 0 0.15 1 101
M4 0.15 M4 0.15 0.15 1 0.15 1 0.15 0 110
M5 0.15 M5 0.15 0.15 1 0.15 1 0.15 1 111
HM CODE
STEP 1 STEP 2 STEP 3 STEP 4 STEP 4
M1 0.4 M1 0.4 0.4 0.4 0.4 0.6 0 1 1
M2 0.15 M2 0.15 0.15 0.3 0.3 0 0.4 1 010 010
M3 0.15 M3 0.15 0.15 0.15 0 0.3 1 110 011
M4 0.15 M4 0.15 0.15 0 0.15 1 000 000
M5 0.15 M5 0.15 0.15 1 100 001

26. Code Efficiency = Entropy/N
Entropy = pklog2(1/pk)
= p1log2(1/p1)+ p2log2(1/p2)+ p3log2(1/p3)+ p4log2(1/p4)+
p5log2(1/p5)
= 0.4log2(1/0.4)+ 0.15log2(1/0.15)+ 0.15log2(1/0.15)+
0.15log2(1/0.15)+ 0.15log2(1/0.15)
= 2.171
SF Code
N = pknk
= p1n1+ p2n2+ p3n3+ p4n4+ p5n5
= 0.4*1+0.15*3+0.15*3+0.15*3+0.15*3
= 2.2

27. HM Code
N = pknk
= p1n1+ p2n2+ p3n3+ p4n4+ p5n5
= = 0.4*1+0.15*3+0.15*3+0.15*3+0.15*3
= 2.2
Code efficiency for SF = 0.9868 = 0.9868*100=
98.68 %
Code efficiency for HM = 0.9868 = 98.68 %

28. M1 M2 M3
G1 = 100, G2 = 111, G3 = 101
G1
G2 G3
Output

29. Present
State
M3M2
M1 X1 = M1 X2 =
m1+m2+
m3
X3 =
m1+m3
Next
State
M2M1
00 0 0 0 0 00
1 1 1 1 01
01 0 0 1 0 10
1 1 0 1 11
10 0 0 1 1 00
1 1 0 0 01
11 0 0 0 1 10
1 1 1 0 11

30. Trellis Diagram
Present State Next State
A=00
B=01
C=10
D=11
A=00
B=01
C=10
D=11
0/000
1/111
0/010
1/101
0/011
1/100
0/001
1/110

31. State diagram
A
B
C
D
0/000
1/111
0/010
1/101
0/011
1/100
0/001
1/110

32. Code Tree
a
0
1
a
b
0
a
b
1 0
1
c
d