Introduction to adaptive signal processing

EECS0712 Adaptive Signal Processing
1
Introduction to Adaptive Signal
Processing
1
Introduction to Adaptive Signal
Processing
Assoc. Prof. Dr. Peerapol Yuvapoositanon
Dept. of Electronic Engineering
CESdSP ASP1-1
http://embedsigproc.wordpress.com/eecs0712
Assoc. Prof. Dr. P.Yuvapoositanon

Course Outline
• Introduction to Adaptive Signal Processing
• Adaptive Algorithms Families:
• Newton’s Method and Steepest Descent
• Least Mean Squared (LMS)
• Recursive Least Squares (RLS)
• Kalman Filtering
• Applications of Adaptive Signal Processing in
Communications and Blind Equalization
• Introduction to Adaptive Signal Processing
• Adaptive Algorithms Families:
• Newton’s Method and Steepest Descent
• Least Mean Squared (LMS)
• Recursive Least Squares (RLS)
• Kalman Filtering
• Applications of Adaptive Signal Processing in
Communications and Blind Equalization
CESdSP ASP1-2

Evaluation
• Assignment= 20 %
• Midterm = 30 %
• Final = 50 %
CESdSP
ASP1-3

Textbooks
CESdSP
ASP1-4

http://embedsigproc.wordpress.com
/eecs0712-adaptive-signal-processing/
CESdSP
ASP1-5

QR code
CESdSP
ASP1-6

Adaptive Signal Processing
• Definition: Adaptive signal processing is the
design of adaptive systems for signal-
processing applications.
[http://encyclopedia2.thefreedictionary.com/adaptive+signal+pr
ocessing]
• Definition: Adaptive signal processing is the
design of adaptive systems for signal-
processing applications.
[http://encyclopedia2.thefreedictionary.com/adaptive+signal+pr
ocessing]
CESdSP
ASP1-7

System Identification
• Let’s consider a system called “plant”
• We need to know its characteristics, i.e., The
impulse response of the system
CESdSP ASP1-8

Plant Comparison
CESdSP ASP1-9

Error of Plant Outputs
CESdSP ASP1-10

Error of Estimation
• Error of estimation is represented by the
signal energy of error
2 2
2 2
( )
2
e d y
d dy y
 
  
CESdSP ASP1-11
2 2
2 2
( )
2
e d y
d dy y
 
  

Adaptive System
• We can do it adaptively
CESdSP ASP1-12

• Adjust the weight for minimum error e
One-weight
CESdSP ASP1-13

2 2
2 2
2 2
0 0 0 0
( )
2
( ) 2( )( ) ( )I I
e d y
d dy y
w x w x w x w x
 
  
  
CESdSP
2 2
2 2
2 2
0 0 0 0
( )
2
( ) 2( )( ) ( )I I
e d y
d dy y
w x w x w x w x
 
  
  
ASP1-14

Error Curve
• Parabola equation
CESdSP ASP1-15

Partial diff. and set to zero
• Partial differentiation
• Set to zero
• Result:
2
2 2
0 0 0 0
0 0
2 2
0 0
( ) 2( )( ) ( )
2 2
I I
I I
I
e
w x w x w x w x
w w
w x w x
 
  
 
  
• Partial differentiation
• Set to zero
• Result:
CESdSP
2
2 2
0 0 0 0
0 0
2 2
0 0
( ) 2( )( ) ( )
2 2
I I
I I
I
e
w x w x w x w x
w w
w x w x
 
  
 
  
2 2
0 00 2 2 I
w x w x  
0 0
I
w w
ASP1-16

Multiple Weight Plants
• We calculate the weight adaptively
• Questions:
– What is the type of signal “x” to be used, e.g.
Sine, Cosine or Random signals ?
– If there is more than one weight w0 , i.e., w0….wN-
1, how do we calculate the solution?
• We calculate the weight adaptively
• Questions:
– What is the type of signal “x” to be used, e.g.
Sine, Cosine or Random signals ?
– If there is more than one weight w0 , i.e., w0….wN-
1, how do we calculate the solution?
CESdSP ASP1-17

Plants with Multiple Weight
• If we have multiple weights
CESdSP
1
0 1w w z
 w
ASP1-18

• In the case of two-weight
Two-weight
CESdSP ASP1-19

Input
• From
• We construct the x as vector with first
element is the most recent
(3), (2), (1), (0), ( 1), ( 2),...x x x x x x 
• From
• We construct the x as vector with first
element is the most recent
CESdSP
[ (3) (2) (1) (0)...]T
x x x xx
ASP1-20

Plants with Multiple Weight
(aka “Transversal Filter”)
• If we have multiple weights
( )x n ( 1)x n 
CESdSP
0 ( )w x n
0 ( 1)w x n 
0 0( ) ( ) ( 1)y n w x n w x n  
ASP1-21

Regression input signal vector
• If the current time is n, we have “Regression
input signal vector”
[ ( ) ( 1) ( 2) ( 3)...]T
x n x n x n x n   x
CESdSP
[ ( ) ( 1) ( 2) ( 3)...]T
x n x n x n x n   x
ASP1-22

0
0 1
1
[ ]T
w
w ww
 
  
  
w
CESdSP
0
0 1
1
[ ]T
w
w ww
 
  
  
w
0
0 1
1
ˆ [ ]
I
I I T
I
w
w w
w
 
 
  
 
 
w
ASP1-23

Convolution
• Output of plant is a convolution
• Ex For N=2
1
1
( ) ( )
N
k
k
y n w x n k


 
• Output of plant is a convolution
• Ex For N=2
CESdSP
1
1
( ) ( )
N
k
k
y n w x n k


 
0 0( ) ( 0) ( 1)y n w x n w x n   
ASP1-24

0 1
0 1
0 1
0 1
0 1
(3) (3) (2)
(2) (2) (1)
(1) (1) (0)
(0) (0) ( 1)
( 1) ( 1) ( 2)
y w x w x
y w x w x
y w x w x
y w x w x
y w x w x
 
 
 
  
    
CESdSP
0 1
0 1
0 1
0 1
0 1
(3) (3) (2)
(2) (2) (1)
(1) (1) (0)
(0) (0) ( 1)
( 1) ( 1) ( 2)
y w x w x
y w x w x
y w x w x
y w x w x
y w x w x
 
 
 
  
    
ASP1-25

• We can use a vector-matrix multiplication
• For example, for n=3 we construct y(3) as
0 1 0 1
(3)
(3) (3) (2) [ ] (3)
(2)
T
x
y w x w x w w
x
 
     
  
w x
• We can use a vector-matrix multiplication
CESdSP
0 1 0 1
(3)
(3) (3) (2) [ ] (3)
(2)
T
x
y w x w x w w
x
 
     
  
w x
0 1 0 1
(1)
(1) (1) (0) [ ] (1)
(0)
T
x
y w x w x w w
x
 
     
  
w x
ASP1-26

0 1 0 1
0 1 0 1
0 1 0 1
0 1 0 1
(3)
(3) (3) (2) [ ] (3)
(2)
(2)
(2) (2) (1) [ ] (2)
(1)
(1)
(1) (1) (0) [ ] (1)
(0)
(2)
(0) (0) ( 1) [ ] (0
(1)
T
T
T
T
x
y w x w x w w
x
x
y w x w x w w
x
x
y w x w x w w
x
x
y w x w x w w
x
 
     
  
 
     
  
 
     
  
 
      
  
w x
w x
w x
w x )
CESdSP
0 1 0 1
0 1 0 1
0 1 0 1
0 1 0 1
(3)
(3) (3) (2) [ ] (3)
(2)
(2)
(2) (2) (1) [ ] (2)
(1)
(1)
(1) (1) (0) [ ] (1)
(0)
(2)
(0) (0) ( 1) [ ] (0
(1)
T
T
T
T
x
y w x w x w w
x
x
y w x w x w w
x
x
y w x w x w w
x
x
y w x w x w w
x
 
     
  
 
     
  
 
     
  
 
      
  
w x
w x
w x
w x )
ASP1-27

• The error squared is
• Let us stop there to consider Random signal
theory first.
2 2
2 2
2 2
( )
2
ˆ ˆ( ) 2( )( ) ( )T T T T
e d y
d dy y
 
  
  w x w x w x w x
• The error squared is
• Let us stop there to consider Random signal
theory first.
CESdSP
2 2
2 2
2 2
( )
2
ˆ ˆ( ) 2( )( ) ( )T T T T
e d y
d dy y
 
  
  w x w x w x w x
ASP1-28

Review of Random Signals
CESdSP ASP1-29

Wireless Transmissions
• Ideal signal transmission
11 00 11 00 11 0011 11 11 000011
CESdSP
ASP2-30
11 00 11 00 11 0011 11 11 000011
Information
Information is Random

Random variable
CESdSP ASP1-31

Random Variable
• Random variable is a function
• For a single time Coin Tossing
1,
( )
-1,
x H
X x
x T
 
 
• Random variable is a function
• For a single time Coin Tossing
CESdSP
1,
( )
-1,
x H
X x
x T
 
 
ASP1-32

Our signal x(n) is a Random
Variable
• For a series of Coin Tossing
1,
( )
-1,
i
i
i
x H
X x
x T
 
 
• For a series of Coin Tossing
CESdSP
1,
( )
-1,
i
i
i
x H
X x
x T
 
 
0 1 2 3 4{ , , , , ,....}x x x x x x
ASP1-33

Coin tossing and Random Variable
• If random
• We have random variable X
0 1 2 3 4
{ , , , , }
{ , , , , }
x H H T H T
x x x x x


CESdSP
• If random
• We have random variable X
0 1 2 3 4( ) { ( ), ( ), ( ), ( ), ( )}
{ ( ), ( ), ( ), ( ), ( )}
{1,1, 1,1, 1}
iX x X x X x X x X x X x
X H X H X T X H X T


  
ASP1-34

Random Digital Signal
• If the random variable is a function of time, it
is called a stochastic process
CESdSP ASP1-35

Probability Mass Function
• We need also to define the probability of each
random variable
( ) { ( ), ( ), ( ), ( ), ( )}
{1,1, 1,1, 1}
X x X H X H X T X H X T
  
CESdSP
( ) { ( ), ( ), ( ), ( ), ( )}
{1,1, 1,1, 1}
X x X H X H X T X H X T
  
ASP1-36

Probability Mass Function
• PMF is for Discrete distribution function
CESdSP ASP1-37

Time and Emsemble
CESdSP ASP1-38

Probability of X(2)
CESdSP ASP1-39

Probability Density Function
• PDF is for Continuous Distribution Function
CESdSP ASP1-40

CESdSP ASP1-41

Probability Density Function
• PDF values can be > 1 as long as its area under
curve is 1
2
CESdSP
1/2
2
1
1
ASP1-42

Cumulative Distribution Function
CESdSP
( ( )) Pr[ ( )]P x n X x n x
ASP1-43

( )
( ( )) ( )
x n
P x n p z dz

 x x
CESdSP
( )
( ( )) ( )
x n
P x n p z dz

 x x
ASP1-44

Expectation Operator
{}E 
CESdSP
{}E 
ASP1-45

Expected Value
• Expected value is known as the “Mean”
{ } ( )X XE x xp x dx


 
CESdSP
{ } ( )X XE x xp x dx


 
ASP1-46

Example of Expected Value
(Discrete)
• We toss a die N times and get a set of
outcomes
• Suppose we roll a die with N=6, we might get
{ ( )} { (1), (2), (3),..., ( )}X i X X X X N
• We toss a die N times and get a set of
outcomes
• Suppose we roll a die with N=6, we might get
CESdSP
{ ( )} { (1), (2), (3),..., ( )}X i X X X X N
{ ( )} {2,3,6,3,1,1}X i 
ASP1-47

Example of Expected Value
(Discrete)
• But, empirically we have Empirical (Monte
Carlo) estimate as Expected Value
6
1
{ } ( )Pr( ( ))
1 1 1 1
1 2 3 6
3 6 3 6
2.67
X
i
E x X i X X i

 
       


CESdSP
6
1
{ } ( )Pr( ( ))
1 1 1 1
1 2 3 6
3 6 3 6
2.67
X
i
E x X i X X i

 
       


ASP1-48

Theoretical Expected Value
• But in theory, for a die
6
1
{ } ( )Pr( ( ))
1 1 1 1 1 1
1 2 3 4 5 6
6 6 6 6 6 6
3.5
X
i
E X X i X X i

 
           


1
Pr( ( ))
6
X X i 
CESdSP
6
1
{ } ( )Pr( ( ))
1 1 1 1 1 1
1 2 3 4 5 6
6 6 6 6 6 6
3.5
X
i
E X X i X X i

 
           


ASP1-49

Ensemble Average
i ensembles
1 1 2 2Ensemble Average of (1) (1)Pr[ (1)] (1)Pr[ (1)]
(1)Pr[ (1)]N N
x x x x x
x x
  


1 ensemble
CESdSP
ASP1-50
i ensembles

Ensemble Average
{ ( )}E x n 
CESdSP
ASP1-51
{ ( )} ( ) ( ( )) ( )E x n x n p x n dx n


  x
{ ( )}E x n 

• I) Linearity
CESdSP
ASP1-52
{ ( ) ( )} { ( )} { ( )}E ax n by n aE x n bE y n  

• II)
{ ( ) ( )} { ( )} { ( )}E x n y n E x n E y n
CESdSP
ASP1-53
{ ( ) ( )} { ( )} { ( )}E x n y n E x n E y n

• III)
{ ( )} ( ( )) ( ( )) ( )E y n g x n p x n dx n


  x
CESdSP
ASP1-54
{ ( )} ( ( )) ( ( )) ( )E y n g x n p x n dx n


  x

Autocorrelation
1 1( , ) { ( ) ( )}r n m E x n x mxx
CESdSP
ASP1-55
1 11 1 1 1 1 1( , ) ( ) ( ) ( ( ), ( )) ( ) ( )r n m x n x m p x n x m dx n x m
 
 
  xx x x

1 1(1,4) { (1) (4)}r E x xxx
CESdSP
ASP1-56

Autocorrelation
• n=m
2
( , ) ( , ) { ( )}r n m r n n E x n xx xx
CESdSP
ASP1-57
2
( , ) ( , ) { ( )}r n m r n n E x n xx xx

Autocorrelation Matrix
(0,0) (0,1) (0, 1)
(1,0) (1,1) (1, 1)
( 1,0) ( 1,1) ( 1, 1)
r r r N
r r r N
r N r N r N N
  
 
  
  
 
 
      
xx xx xx
xx xx xx
xx
xx xx xx
R

  

CESdSP
ASP1-58
(0,0) (0,1) (0, 1)
(1,0) (1,1) (1, 1)
( 1,0) ( 1,1) ( 1, 1)
r r r N
r r r N
r N r N r N N
  
 
  
  
 
 
      
xx xx xx
xx xx xx
xx
xx xx xx
R

  


Covariance
( , ) {[ ( ) ( )][ ( ) ( )]}c n m E x n n x m m   xx
CESdSP
ASP1-59
( , ) {[ ( ) ( )][ ( ) ( )]}c n m E x n n x m m   xx

Stationarity (I)
• I)
{ ( )} { ( )}E x n E x m  
n1
CESdSP
ASP1-60
n2

Stationarity (II)
• II)
( , ) { ( ) ( )}r n n m E x n x n m  xx
1 1 1 1( , ) { ( ) ( )}r n n m E x n x n m  xx
CESdSP
ASP1-61
1 1 1 1( , ) { ( ) ( )}r n n m E x n x n m  xx

Expected Value of Error Energy
• Let’s take the expected value of error energy
2 2 2
ˆ ˆ{ } {( ) 2( )( ) ( ) }
ˆ ˆ ˆ{( )( )} 2 {( )( )} {( )( )}
ˆ ˆ ˆ{ } 2 {( )( )} { }
ˆ ˆ ˆ2 {( )( )}
T T T T
T T T T T T
T T T T T T
T T T T
E e E
E E E
E E E
E
  
  
  
  
w x w x w x w x
w x x w x w w x w x x w
w xx w x w x w w xx w
w Rw x w x w w Rw
CESdSP
ASP1-62
2 2 2
ˆ ˆ{ } {( ) 2( )( ) ( ) }
ˆ ˆ ˆ{( )( )} 2 {( )( )} {( )( )}
ˆ ˆ ˆ{ } 2 {( )( )} { }
ˆ ˆ ˆ2 {( )( )}
T T T T
T T T T T T
T T T T T T
T T T T
E e E
E E E
E E E
E
  
  
  
  
w x w x w x w x
w x x w x w w x w x x w
w xx w x w x w w xx w
w Rw x w x w w Rw

Vector-Matrix Differentiation
Î)
ˆ
ˆ ˆ ÎI) 2
ˆ
T
T T






w x x
w
w xx w Rw
w
CESdSP
Î)
ˆ
ˆ ˆ ÎI) 2
ˆ
T
T T






w x x
w
w xx w Rw
w
ASP1-63

Partial diff. and set to zero
• Differentiation
• Result:
ˆ0 2 {( ) } 2
ˆ
ˆ2 { } 2
ˆ2 2
T
E
E d

  

  
  
w x x Rw
w
x Rw
r Rw
• Differentiation
• Result:
CESdSP
ˆ0 2 {( ) } 2
ˆ
ˆ2 { } 2
ˆ2 2
T
E
E d

  

  
  
w x x Rw
w
x Rw
r Rw
1
ˆ 
w R r
ASP1-64

2-D Error surface
CESdSP
1
ˆ 
w R r
ASP1-65

Four Basic Classes of Adaptive
Signal Processing
• I) Identification
• II) Inverse Modelling
• III) Prediction
• IV) Interference Cancelling
CESdSP
ASP1-66
• I) Identification
• II) Inverse Modelling
• III) Prediction
• IV) Interference Cancelling

The Four Classes of Adaptive
Filtering
CESdSP
ASP1-67

System Identification
CESdSP
ASP2-68

Inverse Modelling
CESdSP
ASP2-69

Prediction
CESdSP
ASP2-70

Interference Canceller
CESdSP
ASP2-71

What are we looking for in
Adaptive Systems?
• Rate of Convergence
• Misadjustment
• Tracking
• Robustness
• Computational Complexity
• Numerical Properties
• Rate of Convergence
• Misadjustment
• Tracking
• Robustness
• Computational Complexity
• Numerical Properties
CESdSP
ASP1-72

Introduction to adaptive signal processing

More Related Content

What's hot

Similar to Introduction to adaptive signal processing

Recently uploaded

Introduction to adaptive signal processing