Introduction to adaptive signal processing

EECS0712 Adaptive Signal Processing
1
Introduction to Adaptive Signal
Processing
1
Introduction to Adaptive Signal
Processing
Assoc. Prof. Dr. Peerapol Yuvapoositanon
Dept. of Electronic Engineering
CESdSP ASP1-1
http://embedsigproc.wordpress.com/eecs0712
Assoc. Prof. Dr. P.Yuvapoositanon

Course Outline
• Introduction to Adaptive Signal Processing
• Adaptive Algorithms Families:
• Newton’s Method and Steepest Descent
• Least Mean Squared (LMS)
• Recursive Least Squares (RLS)
• Kalman Filtering
• Applications of Adaptive Signal Processing in
Communications and Blind Equalization
• Introduction to Adaptive Signal Processing
• Adaptive Algorithms Families:
• Newton’s Method and Steepest Descent
• Least Mean Squared (LMS)
• Recursive Least Squares (RLS)
• Kalman Filtering
• Applications of Adaptive Signal Processing in
Communications and Blind Equalization
CESdSP ASP1-2

Evaluation
• Assignment= 20 %
• Midterm = 30 %
• Final = 50 %
CESdSP
ASP1-3

Textbooks
CESdSP
ASP1-4

http://embedsigproc.wordpress.com
/eecs0712-adaptive-signal-processing/
CESdSP
ASP1-5

QR code
CESdSP
ASP1-6

Adaptive Signal Processing
• Definition: Adaptive signal processing is the
design of adaptive systems for signal-
processing applications.
[http://encyclopedia2.thefreedictionary.com/adaptive+signal+pr
ocessing]
• Definition: Adaptive signal processing is the
design of adaptive systems for signal-
processing applications.
[http://encyclopedia2.thefreedictionary.com/adaptive+signal+pr
ocessing]
CESdSP
ASP1-7

System Identification
• Let’s consider a system called “plant”
• We need to know its characteristics, i.e., The
impulse response of the system
CESdSP ASP1-8

Plant Comparison
CESdSP ASP1-9

Error of Plant Outputs
CESdSP ASP1-10

Error of Estimation
• Error of estimation is represented by the
signal energy of error
2 2
2 2
( )
2
e d y
d dy y
 
  
CESdSP ASP1-11
2 2
2 2
( )
2
e d y
d dy y
 
  

Adaptive System
• We can do it adaptively
CESdSP ASP1-12

• Adjust the weight for minimum error e
One-weight
CESdSP ASP1-13

2 2
2 2
2 2
0 0 0 0
( )
2
( ) 2( )( ) ( )I I
e d y
d dy y
w x w x w x w x
 
  
  
CESdSP
2 2
2 2
2 2
0 0 0 0
( )
2
( ) 2( )( ) ( )I I
e d y
d dy y
w x w x w x w x
 
  
  
ASP1-14

Error Curve
• Parabola equation
CESdSP ASP1-15

Partial diff. and set to zero
• Partial differentiation
• Set to zero
• Result:
2
2 2
0 0 0 0
0 0
2 2
0 0
( ) 2( )( ) ( )
2 2
I I
I I
I
e
w x w x w x w x
w w
w x w x
 
  
 
  
• Partial differentiation
• Set to zero
• Result:
CESdSP
2
2 2
0 0 0 0
0 0
2 2
0 0
( ) 2( )( ) ( )
2 2
I I
I I
I
e
w x w x w x w x
w w
w x w x
 
  
 
  
2 2
0 00 2 2 I
w x w x  
0 0
I
w w
ASP1-16

Multiple Weight Plants
• We calculate the weight adaptively
• Questions:
– What is the type of signal “x” to be used, e.g.
Sine, Cosine or Random signals ?
– If there is more than one weight w0 , i.e., w0….wN-
1, how do we calculate the solution?
• We calculate the weight adaptively
• Questions:
– What is the type of signal “x” to be used, e.g.
Sine, Cosine or Random signals ?
– If there is more than one weight w0 , i.e., w0….wN-
1, how do we calculate the solution?
CESdSP ASP1-17

Plants with Multiple Weight
• If we have multiple weights
CESdSP
1
0 1w w z
 w
ASP1-18

• In the case of two-weight
Two-weight
CESdSP ASP1-19

Input
• From
• We construct the x as vector with first
element is the most recent
(3), (2), (1), (0), ( 1), ( 2),...x x x x x x 
• From
• We construct the x as vector with first
element is the most recent
CESdSP
[ (3) (2) (1) (0)...]T
x x x xx
ASP1-20

Plants with Multiple Weight
(aka “Transversal Filter”)
• If we have multiple weights
( )x n ( 1)x n 
CESdSP
0 ( )w x n
0 ( 1)w x n 
0 0( ) ( ) ( 1)y n w x n w x n  
ASP1-21

Regression input signal vector
• If the current time is n, we have “Regression
input signal vector”
[ ( ) ( 1) ( 2) ( 3)...]T
x n x n x n x n   x
CESdSP
[ ( ) ( 1) ( 2) ( 3)...]T
x n x n x n x n   x
ASP1-22

0
0 1
1
[ ]T
w
w ww
 
  
  
w
CESdSP
0
0 1
1
[ ]T
w
w ww
 
  
  
w
0
0 1
1
ˆ [ ]
I
I I T
I
w
w w
w
 
 
  
 
 
w
ASP1-23

Convolution
• Output of plant is a convolution
• Ex For N=2
1
1
( ) ( )
N
k
k
y n w x n k


 
• Output of plant is a convolution
• Ex For N=2
CESdSP
1
1
( ) ( )
N
k
k
y n w x n k


 
0 0( ) ( 0) ( 1)y n w x n w x n   
ASP1-24

0 1
0 1
0 1
0 1
0 1
(3) (3) (2)
(2) (2) (1)
(1) (1) (0)
(0) (0) ( 1)
( 1) ( 1) ( 2)
y w x w x
y w x w x
y w x w x
y w x w x
y w x w x
 
 
 
  
    
CESdSP
0 1
0 1
0 1
0 1
0 1
(3) (3) (2)
(2) (2) (1)
(1) (1) (0)
(0) (0) ( 1)
( 1) ( 1) ( 2)
y w x w x
y w x w x
y w x w x
y w x w x
y w x w x
 
 
 
  
    
ASP1-25

• We can use a vector-matrix multiplication
• For example, for n=3 we construct y(3) as
0 1 0 1
(3)
(3) (3) (2) [ ] (3)
(2)
T
x
y w x w x w w
x
 
     
  
w x
• We can use a vector-matrix multiplication
CESdSP
0 1 0 1
(3)
(3) (3) (2) [ ] (3)
(2)
T
x
y w x w x w w
x
 
     
  
w x
0 1 0 1
(1)
(1) (1) (0) [ ] (1)
(0)
T
x
y w x w x w w
x
 
     
  
w x
ASP1-26

0 1 0 1
0 1 0 1
0 1 0 1
0 1 0 1
(3)
(3) (3) (2) [ ] (3)
(2)
(2)
(2) (2) (1) [ ] (2)
(1)
(1)
(1) (1) (0) [ ] (1)
(0)
(2)
(0) (0) ( 1) [ ] (0
(1)
T
T
T
T
x
y w x w x w w
x
x
y w x w x w w
x
x
y w x w x w w
x
x
y w x w x w w
x
 
     
  
 
     
  
 
     
  
 
      
  
w x
w x
w x
w x )
CESdSP
0 1 0 1
0 1 0 1
0 1 0 1
0 1 0 1
(3)
(3) (3) (2) [ ] (3)
(2)
(2)
(2) (2) (1) [ ] (2)
(1)
(1)
(1) (1) (0) [ ] (1)
(0)
(2)
(0) (0) ( 1) [ ] (0
(1)
T
T
T
T
x
y w x w x w w
x
x
y w x w x w w
x
x
y w x w x w w
x
x
y w x w x w w
x
 
     
  
 
     
  
 
     
  
 
      
  
w x
w x
w x
w x )
ASP1-27

• The error squared is
• Let us stop there to consider Random signal
theory first.
2 2
2 2
2 2
( )
2
ˆ ˆ( ) 2( )( ) ( )T T T T
e d y
d dy y
 
  
  w x w x w x w x
• The error squared is
• Let us stop there to consider Random signal
theory first.
CESdSP
2 2
2 2
2 2
( )
2
ˆ ˆ( ) 2( )( ) ( )T T T T
e d y
d dy y
 
  
  w x w x w x w x
ASP1-28

Review of Random Signals
CESdSP ASP1-29

Wireless Transmissions
• Ideal signal transmission
11 00 11 00 11 0011 11 11 000011
CESdSP
ASP2-30
11 00 11 00 11 0011 11 11 000011
Information
Information is Random

Random variable
CESdSP ASP1-31

Random Variable
• Random variable is a function
• For a single time Coin Tossing
1,
( )
-1,
x H
X x
x T
 
 
• Random variable is a function
• For a single time Coin Tossing
CESdSP
1,
( )
-1,
x H
X x
x T
 
 
ASP1-32

Our signal x(n) is a Random
Variable
• For a series of Coin Tossing
1,
( )
-1,
i
i
i
x H
X x
x T
 
 
• For a series of Coin Tossing
CESdSP
1,
( )
-1,
i
i
i
x H
X x
x T
 
 
0 1 2 3 4{ , , , , ,....}x x x x x x
ASP1-33

Coin tossing and Random Variable
• If random
• We have random variable X
0 1 2 3 4
{ , , , , }
{ , , , , }
x H H T H T
x x x x x


CESdSP
• If random
• We have random variable X
0 1 2 3 4( ) { ( ), ( ), ( ), ( ), ( )}
{ ( ), ( ), ( ), ( ), ( )}
{1,1, 1,1, 1}
iX x X x X x X x X x X x
X H X H X T X H X T


  
ASP1-34

Random Digital Signal
• If the random variable is a function of time, it
is called a stochastic process
CESdSP ASP1-35

Probability Mass Function
• We need also to define the probability of each
random variable
( ) { ( ), ( ), ( ), ( ), ( )}
{1,1, 1,1, 1}
X x X H X H X T X H X T
  
CESdSP
( ) { ( ), ( ), ( ), ( ), ( )}
{1,1, 1,1, 1}
X x X H X H X T X H X T
  
ASP1-36

Probability Mass Function
• PMF is for Discrete distribution function
CESdSP ASP1-37

Time and Emsemble
CESdSP ASP1-38

Probability of X(2)
CESdSP ASP1-39

Probability Density Function
• PDF is for Continuous Distribution Function
CESdSP ASP1-40

CESdSP ASP1-41

Probability Density Function
• PDF values can be > 1 as long as its area under
curve is 1
2
CESdSP
1/2
2
1
1
ASP1-42

Cumulative Distribution Function
CESdSP
( ( )) Pr[ ( )]P x n X x n x
ASP1-43

( )
( ( )) ( )
x n
P x n p z dz

 x x
CESdSP
( )
( ( )) ( )
x n
P x n p z dz

 x x
ASP1-44

Expectation Operator
{}E 
CESdSP
{}E 
ASP1-45

Expected Value
• Expected value is known as the “Mean”
{ } ( )X XE x xp x dx


 
CESdSP
{ } ( )X XE x xp x dx


 
ASP1-46

Example of Expected Value
(Discrete)
• We toss a die N times and get a set of
outcomes
• Suppose we roll a die with N=6, we might get
{ ( )} { (1), (2), (3),..., ( )}X i X X X X N
• We toss a die N times and get a set of
outcomes
• Suppose we roll a die with N=6, we might get
CESdSP
{ ( )} { (1), (2), (3),..., ( )}X i X X X X N
{ ( )} {2,3,6,3,1,1}X i 
ASP1-47

Example of Expected Value
(Discrete)
• But, empirically we have Empirical (Monte
Carlo) estimate as Expected Value
6
1
{ } ( )Pr( ( ))
1 1 1 1
1 2 3 6
3 6 3 6
2.67
X
i
E x X i X X i

 
       


CESdSP
6
1
{ } ( )Pr( ( ))
1 1 1 1
1 2 3 6
3 6 3 6
2.67
X
i
E x X i X X i

 
       


ASP1-48

Theoretical Expected Value
• But in theory, for a die
6
1
{ } ( )Pr( ( ))
1 1 1 1 1 1
1 2 3 4 5 6
6 6 6 6 6 6
3.5
X
i
E X X i X X i

 
           


1
Pr( ( ))
6
X X i 
CESdSP
6
1
{ } ( )Pr( ( ))
1 1 1 1 1 1
1 2 3 4 5 6
6 6 6 6 6 6
3.5
X
i
E X X i X X i

 
           


ASP1-49

Ensemble Average
i ensembles
1 1 2 2Ensemble Average of (1) (1)Pr[ (1)] (1)Pr[ (1)]
(1)Pr[ (1)]N N
x x x x x
x x
  


1 ensemble
CESdSP
ASP1-50
i ensembles

Ensemble Average
{ ( )}E x n 
CESdSP
ASP1-51
{ ( )} ( ) ( ( )) ( )E x n x n p x n dx n


  x
{ ( )}E x n 

• I) Linearity
CESdSP
ASP1-52
{ ( ) ( )} { ( )} { ( )}E ax n by n aE x n bE y n  

• II)
{ ( ) ( )} { ( )} { ( )}E x n y n E x n E y n
CESdSP
ASP1-53
{ ( ) ( )} { ( )} { ( )}E x n y n E x n E y n

• III)
{ ( )} ( ( )) ( ( )) ( )E y n g x n p x n dx n


  x
CESdSP
ASP1-54
{ ( )} ( ( )) ( ( )) ( )E y n g x n p x n dx n


  x

Autocorrelation
1 1( , ) { ( ) ( )}r n m E x n x mxx
CESdSP
ASP1-55
1 11 1 1 1 1 1( , ) ( ) ( ) ( ( ), ( )) ( ) ( )r n m x n x m p x n x m dx n x m
 
 
  xx x x

1 1(1,4) { (1) (4)}r E x xxx
CESdSP
ASP1-56

Autocorrelation
• n=m
2
( , ) ( , ) { ( )}r n m r n n E x n xx xx
CESdSP
ASP1-57
2
( , ) ( , ) { ( )}r n m r n n E x n xx xx

Autocorrelation Matrix
(0,0) (0,1) (0, 1)
(1,0) (1,1) (1, 1)
( 1,0) ( 1,1) ( 1, 1)
r r r N
r r r N
r N r N r N N
  
 
  
  
 
 
      
xx xx xx
xx xx xx
xx
xx xx xx
R

  

CESdSP
ASP1-58
(0,0) (0,1) (0, 1)
(1,0) (1,1) (1, 1)
( 1,0) ( 1,1) ( 1, 1)
r r r N
r r r N
r N r N r N N
  
 
  
  
 
 
      
xx xx xx
xx xx xx
xx
xx xx xx
R

  


Covariance
( , ) {[ ( ) ( )][ ( ) ( )]}c n m E x n n x m m   xx
CESdSP
ASP1-59
( , ) {[ ( ) ( )][ ( ) ( )]}c n m E x n n x m m   xx

Stationarity (I)
• I)
{ ( )} { ( )}E x n E x m  
n1
CESdSP
ASP1-60
n2

Stationarity (II)
• II)
( , ) { ( ) ( )}r n n m E x n x n m  xx
1 1 1 1( , ) { ( ) ( )}r n n m E x n x n m  xx
CESdSP
ASP1-61
1 1 1 1( , ) { ( ) ( )}r n n m E x n x n m  xx

Expected Value of Error Energy
• Let’s take the expected value of error energy
2 2 2
ˆ ˆ{ } {( ) 2( )( ) ( ) }
ˆ ˆ ˆ{( )( )} 2 {( )( )} {( )( )}
ˆ ˆ ˆ{ } 2 {( )( )} { }
ˆ ˆ ˆ2 {( )( )}
T T T T
T T T T T T
T T T T T T
T T T T
E e E
E E E
E E E
E
  
  
  
  
w x w x w x w x
w x x w x w w x w x x w
w xx w x w x w w xx w
w Rw x w x w w Rw
CESdSP
ASP1-62
2 2 2
ˆ ˆ{ } {( ) 2( )( ) ( ) }
ˆ ˆ ˆ{( )( )} 2 {( )( )} {( )( )}
ˆ ˆ ˆ{ } 2 {( )( )} { }
ˆ ˆ ˆ2 {( )( )}
T T T T
T T T T T T
T T T T T T
T T T T
E e E
E E E
E E E
E
  
  
  
  
w x w x w x w x
w x x w x w w x w x x w
w xx w x w x w w xx w
w Rw x w x w w Rw

Vector-Matrix Differentiation
Î)
ˆ
ˆ ˆ ÎI) 2
ˆ
T
T T






w x x
w
w xx w Rw
w
CESdSP
Î)
ˆ
ˆ ˆ ÎI) 2
ˆ
T
T T






w x x
w
w xx w Rw
w
ASP1-63

Partial diff. and set to zero
• Differentiation
• Result:
ˆ0 2 {( ) } 2
ˆ
ˆ2 { } 2
ˆ2 2
T
E
E d

  

  
  
w x x Rw
w
x Rw
r Rw
• Differentiation
• Result:
CESdSP
ˆ0 2 {( ) } 2
ˆ
ˆ2 { } 2
ˆ2 2
T
E
E d

  

  
  
w x x Rw
w
x Rw
r Rw
1
ˆ 
w R r
ASP1-64

2-D Error surface
CESdSP
1
ˆ 
w R r
ASP1-65

Four Basic Classes of Adaptive
Signal Processing
• I) Identification
• II) Inverse Modelling
• III) Prediction
• IV) Interference Cancelling
CESdSP
ASP1-66
• I) Identification
• II) Inverse Modelling
• III) Prediction
• IV) Interference Cancelling

The Four Classes of Adaptive
Filtering
CESdSP
ASP1-67

System Identification
CESdSP
ASP2-68

Inverse Modelling
CESdSP
ASP2-69

Prediction
CESdSP
ASP2-70

Interference Canceller
CESdSP
ASP2-71

What are we looking for in
Adaptive Systems?
• Rate of Convergence
• Misadjustment
• Tracking
• Robustness
• Computational Complexity
• Numerical Properties
• Rate of Convergence
• Misadjustment
• Tracking
• Robustness
• Computational Complexity
• Numerical Properties
CESdSP
ASP1-72

Introduction to adaptive signal processing

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Introduction to adaptive signal processing

Similar to Introduction to adaptive signal processing (20)

Recently uploaded

Recently uploaded (20)

Introduction to adaptive signal processing