unit2.ppt Information Theory and Coding PPT

Definitions
• A word is a sequence of symbols.
• A code is a set of vectors called codewords.
• The Hamming weight (w(c)) of a codeword
(or any vector) is equal to the number of non-
zero elements in the codeword.
• Hamming distance between two codewords
is the number of places by which the
codewords differ. d(c1,c2)
• d(c1, c2) = w (c1 – c2).

Example
• Code C = {00000, 10100, 11110, 11001} is a
block code of block length equal to 5.
• M = 4, k = 2 and n = 5
• The sequence to be encoded is 1 0 0 1 0 1 0
0 1 1.…
• Partitioned: 10 01 01 00 11…
• Codeword: 11110 10100 10100 00000
11001…

Parity-check codes
minimum weight w*= 2
d* = 2.

Galois Fields – Finite Field

Error control techniques
• Automatic Repeat reQuest (ARQ)
– Full-duplex connection, error detection codes
– The receiver sends a feedback to the transmitter, saying that
if any error is detected in the received packet or not (Not-
Acknowledgement (NACK) and Acknowledgement (ACK),
respectively).
– The transmitter retransmits the previously sent packet if it
receives NACK.
• Forward Error Correction (FEC)
– Simplex connection, error correction codes
– The receiver tries to correct some errors
• Hybrid ARQ (ARQ+FEC)
– Full-duplex, error detection and correction codes

Linear block codes
• The information bit stream is chopped into blocks of k bits.
• Each block is encoded to a larger block of n bits.
• The coded bits are modulated and sent over channel.
• The reverse procedure is done at the receiver.
Data block
Channel
encoder
Codeword
k bits n bits
rate
Code
bits
Redundant
n
k
R
n-k
c 

Linear block codes – cont’d
• The Hamming weight of vector U, denoted by w(U),
is the number of non-zero elements in U.
• The Hamming distance between two vectors U and
V, is the number of elements in which they differ.
• The minimum distance of a block code is
)
(
)
( V
U
V
U, 
w
d
)
(
min
)
,
(
min
min i
i
j
i
j
i
w
d
d U
U
U 



• Error detection capability is given by
• Error correcting-capability t of a code, which is defined
as the maximum number of guaranteed correctable
errors per codeword, is





 

2
1
min
d
t
1
min 
d
e

• For memory less channels, the probability that the decoder commits an
erroneous decoding is
– is the transition probability or bit error probability over channel.
• The decoded bit error probability is
j
n
j
n
t
j
M p
p
j
n
P 











  )
1
(
1
j
n
j
n
t
j
B p
p
j
n
j
n
P 











  )
1
(
1
1
p

• Encoding in (n,k) block code
– The rows of G, are linearly independent.
– The sum of any two codewords is also a codeword.
mG
U 
k
n
k
k
n
m
m
m
u
u
u
m
m
m
u
u
u
V
V
V
V
V
V





















2
2
2
1
1
2
1
2
1
2
1
2
1
)
,
,
,
(
)
,
,
,
(
)
,
,
,
(






• Systematic block code (n,k)
– For a systematic code, the first (or last) k elements in
the codeword are information bits.
matrix
)
(
matrix
identity
]
[
k
n
k
k
k
k
k
k






P
I
I
P
G
)
,...,
,
,
,...,
,
(
)
,...,
,
(
bits
message
2
1
bits
parity
2
1
2
1 k
k
n
n m
m
m
p
p
p
u
u
u 


U
Coefficient
Matrix  gives
dependency of
parity bits on the
message bits

• For any linear code we can find an matrix ,
whose rows are orthogonal to rows of :
• H is called the parity check matrix and its rows are
linearly independent.
• For systematic linear block codes:
n
k
n 
 )
(
H
G
0
GH 
T
]
[ T
k
n P
I
H 


• Example: Block code (6,3)






















1
0
0
0
1
0
0
0
1
1
1
0
0
1
1
1
0
1
3
2
1
V
V
V
G
1
1
1
1
1
0
0
0
0
1
0
1
1
1
1
1
1
0
1
1
0
0
0
1
1
0
1
1
1
1
1
0
0
0
1
1
1
1
0
0
0
1
1
0
1
0
0
0
1
0
0
0
1
0
1
0
0
1
1
0
0
1
0
0
0
0
1
0
0
0
1
0
Message vector Codeword

• Syndrome testing:
– S is syndrome of r, corresponding to the error pattern e.
Format
Channel
encoding
Modulation
Channel
decoding
Format
Demodulation
Detection
Data source
Data sink
U
r
m
m̂
channel
or vector
pattern
error
)
,....,
,
(
or vector
codeword
received
)
,....,
,
(
2
1
2
1
n
n
e
e
e
r
r
r


e
r
e
U
r 

T
T
eH
rH
S 

mGHT
+ eHT

• Standard array and syndrome table decoding
1. Calculate
2. Find the coset leader, , corresponding to .
3. Calculate and corresponding .
– Note that
• If , error is corrected.
• If , undetectable decoding error occurs.
T
rH
S 
i
e
e 
ˆ S
e
r
U ˆ
ˆ 
 m̂
)
ˆ
ˆ
(
ˆ
ˆ e
(e
U
e
e)
U
e
r
U 







e
e 
ˆ
e
e 
ˆ

(101110)
(100000)
(001110)
ˆ
ˆ
estimated
is
vector
corrected
The
(100000)
ˆ
is
syndrome
this
to
ing
correspond
pattern
Error
(100)
(001110)
:
computed
is
of
syndrome
The
received.
is
(001110)
ted.
transmit
(101110)











e
r
U
e
H
rH
S
r
r
U
T
T
111
010001
100
100000
010
010000
001
001000
110
000100
011
000010
101
000001
000
000000
Error pattern Syndrome

Repetition Codes
• (n,1) code, eg. (5,1)
• G = [ 1 1 1 1 1 ]
• H = 1 0 0 0 1
0 1 0 0 1
0 0 1 0 1
0 0 0 1 1

 Hamming codes are a subclass of linear block codes and belong to
the category of perfect codes.
 Hamming codes are expressed as a function of a single integer
 The columns of the parity-check matrix, H, consist of all non-zero
binary m-tuples.
Hamming codes
2

m
t
m
n-k
m
k
n
m
m
1
:
capability
correction
Error
:
bits
parity
of
Number
1
2
:
bits
n
informatio
of
Number
1
2
:
length
Code








Hamming codes
• Example: Systematic Hamming code (7,4)
]
[
1
0
1
1
1
0
0
1
1
0
1
0
1
0
1
1
1
0
0
0
1
3
3
T
P
I
H 












]
[
1
0
0
0
1
1
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
0
0
1
1
1
0
4
4













 I
P
G

Cyclic block codes
• Cyclic codes are a subclass of linear block
codes.
• Encoding and syndrome calculation are easily
performed using feedback shift-registers.
– Hence, relatively long block codes can be
implemented with reasonable complexity.
• BCH and Reed-Solomon codes are cyclic codes.

Cyclic block codes
• A linear (n,k) code is called a Cyclic code if all cyclic shifts of a
codeword are also a codeword.
– Linearity and Cyclic property
– Example:
)
,...,
,
,
,
,...,
,
(
)
,...,
,
,
(
1
2
1
0
1
1
)
(
1
2
1
0









i
n
n
i
n
i
n
i
n
u
u
u
u
u
u
u
u
u
u
u
U
U “i” cyclic shifts of U
U
U
U
U
U
U






)
1101
(
)
1011
(
)
0111
(
)
1110
(
)
1101
(
)
4
(
)
3
(
)
2
(
)
1
(

Cyclic block codes
• Algebraic structure of Cyclic codes, implies expressing
codewords in polynomial form
• Relationship between a codeword and its cyclic shifts:
)
1
(
degree
...
)
( 1
1
2
2
1
0 n-
X
u
X
u
X
u
u
X n
n







U
X i
U(X) = X i
( u0 + u1X + ….. un-i-1X n-i-1
+ un-iX n-i
….. + un-1X n-1
)
= u0 X i
+ u1 X i+1
+ ….. un-i-1X n-1
+ un-iX n
….. + un-1X n+i-1
= un-iX n
….. + un-1X n+i-1
+ u0 X i
+ u1 X i+1
+ ….. un-i-1X n-1

Cyclic block codes
X i
U(X) = un-iX n
….. + un-1X n+i-1
+ u0 X i
+ u1 X i+1
+ ….. un-i-1X n-1
= un-i +un-i + un-iX n
….. + un-1 X i-1
+ un-1 X i-1
+ un-1 X n+i-1
+ u0 X i
+ u1 X i+1
+ ….. un-i-1X n-1
= un-i +un-i (Xn
+ 1)….. + un-1 X i-1
+ un-1 X i-1
(Xn
+ 1)
+ u0 X i
+ u1 X i+1
+ ….. un-i-1X n-1
= un-i +….. + un-1 X i-1
+ u0 X i
+ u1 X i+1
+ ….. un-i-1X n-1

Cyclic block codes
Let ,
U(i)
(X) = un-i +….. + un-1 X i-1
+ u0 X i
+ u1 X i+1
+ ….. un-i-1X n-1
q(X) = un-i + un-i-1 X + ………+ un-1 X i-1
X i
U(X) = q(X) (Xn
+ 1)+ U(i)
(X)
)
1
(
modulo
)
(
)
(
)
(

 n
i
i
X
X
X
X U
U
By extension
)
1
(
modulo
)
(
)
(
)
1
(

 n
X
X
X
X U
U

Cyclic block codes
Let C be a binary (n,k) linear cyclic code
 Within the set of code polynomials in C, there is a
unique monic polynomial g(X) with minimal degree r, called as the
generator polynomial.
 g(X) is a polynomial of degree n-k and is a factor of (Xn
+ 1)
 Every code polynomial U(X) in C, can be expressed uniquely as
r
r X
g
X
g
g
X 


 ...
)
( 1
0
g
U(X) = m(X) g(X)

Cyclic block codes
m(X) = m0 + m1X + ….. mk-1X k-1
In Systematic form,
U(X) = p(X) + X n-k
m(X)
p(X) = p0 + p1X + ….. pn-k-1X n-k-1
m(X) g(X) = p(X) + X n-k
m(X)
X n-k
m(X) = m(X) + p(X)
g(X) g(X)
1
1
1
1
1
0
0
0
0
1
0
1
1
1
1
1
1
0
1
1
0
0
0
1
1
0
1
1
1
1
1
0
0
0
1
1
1
1
0
0
0
1
1
0
1
0
0
0
1
0
0
0
1
0
1
0
0
1
1
0
0
1
0
0
0
0
1
0
0
0
1
0
Message vector Codeword

Cyclic block codes
• Systematic encoding algorithm for an
(n,k) Cyclic code:
1. Multiply the message polynomial by
2. Divide the result of Step 1 by the generator
polynomial . Let be the reminder.
3. Add to to form the codeword
)
(X
m k
n
X 
)
(X
g )
(X
p
)
(X
p )
(X
X k
n
m

)
(X
U

Cyclic block codes
 The orthogonality of G and H in polynomial form is
expressed as
 This means is also a factor of
 The row , of generator matrix is formed by the
coefficients of the cyclic shift of the generator
polynomial.































r
r
r
r
k
g
g
g
g
g
g
g
g
g
g
g
g
X
X
X
X
X









1
0
1
0
1
0
1
0
1
)
(
)
(
)
(
0
0
g
g
g
G
1
)
(
)
( 
 n
X
X
X h
g
1

n
X
)
(X
h
k
i
i ,...,
1
, 
"
1 "

k

Cyclic block codes
• Example: For the systematic (7,4) Cyclic code
with generator polynomial
1. Find the codeword for the message )
1011
(

m
3
1
)
( X
X
X 


g
1
)
(
)
(
)
(
:
polynomial
codeword
the
Form
1
)
1
(
)
1
(
:
(
by
)
(
Divide
)
1
(
)
(
)
(
1
)
(
)
1011
(
3
,
4
,
7
6
5
3
3
remainder
generator
3
quotient
3
2
6
5
3
6
5
3
3
2
3
3
3
2

































)
1
1
0
1
0
0
1
(
bit
s
message
bits
parity

U
m
p
U
g
m
m
m
m
m
p
g
q
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X)
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
k
n
k
n
(X)
(X)
(X)
k
n
k
n

Cyclic block codes
2. Find the generator and parity check matrices, G and H,
respectively.






















1
0
1
1
0
0
0
0
1
0
1
1
0
0
0
0
1
0
1
1
0
0
0
0
1
0
1
1
)
1101
(
)
,
,
,
(
1
0
1
1
)
( 3
2
1
0
3
2
G
g g
g
g
g
X
X
X
X
Not in systematic form.
We do the following:
row(4)
row(4)
row(2)
row(1)
row(3)
row(3)
row(1)


















1
0
0
0
1
0
1
0
1
0
0
1
1
1
0
0
1
0
1
1
0
0
0
0
1
0
1
1
G











1
1
1
0
1
0
0
0
1
1
1
0
1
0
1
1
0
1
0
0
1
H
4
4
I
3
3
I T
P
P

Cyclic block codes
• Syndrome decoding for Cyclic codes:
– Received codeword in polynomial form is given by
– The syndrome is the reminder obtained by dividing the
received polynomial by the generator polynomial.
– With syndrome and Standard array, error is estimated.
• In Cyclic codes, the size of standard array is considerably reduced.
)
(
)
(
)
( X
X
X e
U
r 

Received
codeword
Error
pattern
)
(
)
(
)
(
)
( X
X
X
X S
g
q
r 
 Syndrome

Circuit for encoding systematic cyclic codes
• In the circuit, first the message flows to the shift register, and
feedback switch is set to ‘1’, where after check-bit-switch is
turned on, and the feedback switch to ‘0’, enabling the check
bits to be outputted .
1
0

Decoding cyclic codes:
syndrome table
Using denotation of this example:
 
16.20 ( ) mod ( )/ ( )
s x e x g x


 
( ) mod ( )/ ( )
s x r x g x

Table 16.6
Decoding cyclic codes: error
correction

Decoding circuit for (7,4) code
syndrome computation
• To start with, the switch is at “0” position
• Then shift register is stepped until all the received code bits
have entered the register
• This results is a 3-bit syndrome (n - k = 3 ):
that is then left to the register
• Then the switch is turned to the position “1” that drives the
syndrome out of the register.
received code syndrome
 
( ) mod ( )/ ( )
p p p

S R G
3
( ) 1
p p p
  
G
1
0
received code syndrome
x0 x1
xn-1

Some block codes that can be
realized by cyclic codes
• (n,1) Repetition codes. High coding gain (minimum distance always n-1), but
very low rate: 1/n
• (n,k) Hamming codes. Minimum distance always 3. Thus can detect 2 errors and
correct one error. n=2m
-1, k = n - m,
• Maximum-length codes. For every integer there exists a maximum length
code (n,k) with n = 2k
- 1,dmin = 2k-1
.
• BCH-codes. For every integer there exist a code with n = 2m-1,
and where t is the error correction capability
• (n,k) Reed-Solomon (RS) codes. Works with k symbols that consists of m bits
that are encoded to yield code words of n symbols. For these codes
and
• Nowadays BCH and RS are very popular due to large dmin, large number of codes,
and easy generation
• Code selection criteria: number of codes, correlation properties, code gain,
code rate, error correction/detection properties.
3
k 
3

m
 
k n mt min
2 1
 
d t
2 1,number of check symbols 2
   
m
n n k t
min
2 1
 
d t
1: Task: find out from literature what is meant by dual codes!
3
m 

Probability of Error Correction

Hamming Bound or Sphere
Packing Bound

unit2.ppt Information Theory and Coding PPT

More Related Content

Similar to unit2.ppt Information Theory and Coding PPT

Recently uploaded

unit2.ppt Information Theory and Coding PPT