Avionics 738 Adaptive Filtering at Air University PAC Campus by Dr. Bilal A. Siddiqui in Spring 2018. This lecture covers background material for the course.

1. Av-738
Adaptive Filter Theory
Lecture 2- Background Material
Dr. Bilal A. Siddiqui
Air University (PAC Campus)
Spring 2018

2. Stochastic Processes
• Stochastic or random process is time evolution of a statistical
phenomenon according to probabilistic laws
• Because it is a random process, we cannot predict exact values of
how the process will evolve in time.
• E.g. tv, radar, speech, earthquake signals and of course noise.
• However, this does not mean that we cannot predict the mean or
average behavior of the process and make a “good” guess of its
“expected” value.

3. Some moments
• The mean value function of this process is
𝜇 𝑛 = 𝐸 𝑢 𝑛
Where E is the mathematical expectation operator.
• Autocorrelation function is defined as (k=…,-2,-1,0,1,2,…)
𝑟 𝑛, 𝑛 − 𝑘 = 𝐸 𝑢 𝑛 ∗
𝑢 𝑛 − 𝑘
• Autocovariance function is defined as (k=…,-2,-1,0,1,2,…)
𝑐 𝑛, 𝑛 − 𝑘 = 𝐸 𝑢 𝑛 − 𝜇(𝑛) 𝑢 𝑛 − 𝑘 − 𝜇(𝑛 − 𝑘)
Or we can write
𝑐 𝑛, 𝑛 − 𝑘 = 𝑟 𝑛, 𝑛 − 𝑘 − 𝜇 𝑛 𝜇 𝑛 − 𝑘 ∗

4. Stationarity
• A process is “stationary” if its statistical properties are invariant with
time.
• Let’s say we make the following discrete measurements of a random
variable u:
𝑢 𝑛 − 𝑀 , 𝑢 𝑛 − 𝑀 − 1 , … 𝑢 𝑛 − 1 , 𝑢(𝑛)
• The process is strictly stationary if the joint pdf of u(n-M)…u(n)
remains the same regardless of value of sample number (n…n-M).
• However, it is not always possible to find the joint pdf of an arbitrary
set of measurements of a stochastic process.
• Therefore, we limit ourselves to partial characterization of the
random variable by looking only at the first two moments.

5. Wide Sense Stationary
• This form of partial characterization is useful because:
• It is easy to implement
• Well suited to linear operations on stochastic processes
• For a stationary process, obviously
𝜇 𝑛 = 𝜇
• Also autocorrelation and autocovariance depend only on time difference
𝑟 𝑛, 𝑛 − 𝑘 = 𝑟 𝑘 , 𝑐 𝑛, 𝑛 − 𝑘 = 𝑐 𝑘
• When k=0,
𝑟 0 = 𝐸 𝑢 𝑛 2 , 𝑐 0 = 𝜎 𝑢
2
• These two conditions are not necessary for stationarity
• However, if above conditions hold, the stochastic system is known as wide sense
stationary (wss) or stationary to the second order.
• A process is WSS iff 𝐸 𝑢 𝑛 2
< ∞. Most stochastic processes satisfy this
condition!

6. Ergodicity
• Expectations are ensemble averages “across” the process”. But sample (or time averages)
are averages along the process.
• For example, Take N resistors (N should be very large) and plot the voltage across those
resistors for a long period. For each resistor you will have a waveform. Calculate the
average value of that waveform; this gives you the time average. Note also that you
have N waveforms as we have N resistors. These N plots are known as an ensemble. Now
take a particular instant of time in all those plots and find the average value of the
voltage. That gives you the ensemble average for each plot.
• Usually, ensemble averages are not available. However, we wish to use time averages to
build stochastic model of the system. For this approach to work, we need to shown time
averages converge to ensemble averages in some statistical sense.
• A random process is ergodic if its time average is the same as its ensemble average.
• The state of an ergodic process after a long time is nearly independent of its initial state.

7. Correlation Ergodicity
• Let 𝜇 be the mean of the wide sense stationary process u(n).
• An estimate of 𝜇 may be made by using the sample mean 𝜇 𝑁 =
1
𝑁 𝑛=0
𝑁−1
𝑢(𝑛).
Note that 𝜇 𝑁 is a random number.
• A process is said to be ergodic in the mean square error sense if the mean square of
the error between 𝜇 𝑁 and 𝜇 approaches zero as 𝑁 → ∞
lim
𝑁→∞
𝜇 − 𝜇 2 = 0
• For a wss ergodic process, we can write can estimate of the autocorrelation
𝑟 𝑘, 𝑁 =
1
𝑁 𝑛=0
𝑁−1
𝑢 𝑛 𝑢 𝑛 − 𝑘
The process is called correlation ergodic if error between r(k) and 𝑟 𝑘, 𝑁 approaches
zero as 𝑁 → ∞

8. Correlation Matrix
• Let the Mx1 observation matrix be 𝒖 𝒏 = 𝑢 𝑛 , … 𝑢 𝑛 − 𝑀 + 1 𝑇
• Correlation matrix of a wss process is defined as expectation of outer
product of observation matrix. 𝑅 = 𝐸 𝒖 𝒏 𝒖 𝒏 𝑯
• H is the Hermitian (transpose combined with complex conjugation)
• Since 𝑟 𝑘 = 𝐸 𝑢 𝑛 ∗ 𝑢 𝑛 − 𝑘

9. Important Properties of Correlation Matrix
Property 1: Correlation matrix of wss process is Hermitian, i.e. 𝑅 𝐻 = 𝑅. For real valued data R is
symmetric i.e. RT=R
A matrix is Hermitian if it is equal to its complex conjugate. In other words 𝑟 −𝑘 = 𝑟∗
(𝑘)
Property 2: Correlation matrix of wss process is Toeplitz.
A matrix is Toeplitz if all elements on the main diagonal are equal and all elements on diagonals
parallel to the main diagonal are also equal.
This gives us the important result that if R is Toeplitz, the process must be wss!
Property 3: Correlation matrix of a wss process is positive semi-definite, i.e. 𝑅 ≥ 0
Positive semi-definiteness means, (a) the eigenvalues of the matrix have non-negative real parts, and (b) for
any vector 𝑥, the following holds 𝑥 𝐻 𝑅𝑥 ≥ 0 or 𝑥 𝑇 𝑅𝑥 ≥ 0 for real 𝑥.

10. Correlation Matrix of a Sine Wave plus Noise
• A signal with important applications is sinusoid corrupted with zero mean white noise (v)
𝑢 𝑛 = 𝛼𝑒 𝑗𝜔𝑛 + 𝑣(𝑛)
• Think of the sinusoid as the target signature in a radar, corrupted with thermal noise
• Since v is zero mean, the mean of u(n) is obviously 𝛼𝑒 𝑗𝜔𝑛
• Autocorrelation of white noise, by definition is:
• Obviously, the source and noise are independent and uncorrelated. Therefore the
autocovariance of u is simply the sum of autocorrelation of the two components.
• Quiz: prove this! (5 minutes)

11. Correlation Matrix of {Sine}+{white noise}
• Therefore, we can have a two step practical procedure for estimating
parameters of a complex sinusoid in presence of white noise.

12. Correlation Matrix of a Pure Sinusoid
• Consider a pure sinusoid with no added noise.
• Let’s use only three samples, M=3
• It can easily be seen that this matrix is singular (prove at home).
• In general, if there is no additive noise, the correlation matrix of sums
of sinusoids is singular. Therefore, noise can be helpful!

13. z-Transform
• The two-sided z-transform is defined as
• where z is a complex variable. For more info, see lectures by Brian Douglas on YouTube.
• Important properties
1. Linear transform
2. Time-shift
3. Convolution

14. z-Transform of Linear Filter
• Applying z-transform
• 𝐻 𝑧 = 𝑈(𝑧)/𝑉(𝑧) is known as the transfer function of the linear filter

15. A subclass of Linear Filters
• If the input sequence v(n) and output sequence u(n) can be relates as
• Applying z-transform
𝐻 𝑧 =
𝑈 𝑧
𝑉 𝑧
=
𝑗=0
𝑁
𝑏𝑗 𝑧−𝑗
𝑗=0
𝑁
𝑎𝑗 𝑧−𝑗
=
𝑏0
𝑎0
𝑘=1
𝑁
1 − 𝑐 𝑘 𝑧−1
𝑘=1
𝑁
1 − 𝑑 𝑘 𝑧−1
• Roots of the numerator (ck) are zeros, and roots of denominator (dk)
are the poles of transfer function H(z).
• Poles and zeros are real or complex conjugate pairs.

16. Another way to look at FIR and IIR filters
• Based on 𝐻 𝑧 =
𝑏0
𝑎0
𝑘=1
𝑁
1−𝑐 𝑘 𝑧−1
𝑘=1
𝑁 1−𝑑 𝑘 𝑧−1 , we may
define 2 classes of linear filters
1. FIR filters in which all dk=0. This is an all zero filter.
The impulse response has a finite duration (no
integration, i.e. no memory of previous outputs)
2. IIR filters in which all ck=0. This is an all pole filter.
The impulse response has infinite duration
(integration of output lends memory of past which
persists)
• For a filter to be stable (BIBO), we require all the
poles of transfer function to lie within the unit
circle, i.e. 𝑑 𝑘 < 1 for all k.

17. Stochastic Models
• Stochastic model is a mathematical expression which tries to explain the
law governing input-output behavior of a stochastic process.
• The model proposed by Yule (1927) was that an observation vector u
may be generated by applying impulses (“shocks”) to an appropriate
linear filter (remember FIR and IIR filters from previous lecture).
• The “shocks” are random variables drawn from a white Gaussian noise.

18. Linear Filter
• A stochastic process may be called a linear process if
• Structure of the linear filter depends on how the two linear
combinations above are formulated
1. Autoregressive (AR) models, in which no past values of input are used.
2. Moving average (MA) models, in which no past values of output are used.
3. ARMA models, in which past values of both inputs and outputs are used.

19. AR Models
• It has the structure
𝑢 𝑛 + 𝑎1
∗
𝑢 𝑛 − 1 + ⋯ 𝑎 𝑀
∗
𝑢 𝑛 − 𝑀 = 𝑣(𝑛)
• To see why it is referred to as autoregressive, rearrange the above
𝑢 𝑛 = 𝑤1
∗
𝑢 𝑛 − 1 + ⋯ 𝑤 𝑀
∗
𝑢 𝑛 − 𝑀 + 𝑣 𝑛 , where wk=-ak
• We can also write it as
𝑢 𝑛 =
𝑘=1
𝑀
𝑤 𝑘
∗
𝑢 𝑛 − 𝑘 + 𝑣 𝑛
• Therefore, the present output is the finite linear combination of model output, plus an error term v(n). This is a
regression equation
• We can also write
𝑘=0
𝑀
𝑎 𝑘
∗
𝑢 𝑛 − 𝑘 = 𝑣 𝑛
• Taking the z-transform
𝐻𝐴 𝑧 𝑈(𝑧) = 𝑉(𝑧)
𝐻𝐴 𝑧 =
𝑛=0
𝑀
𝑎 𝑛
∗ 𝑧−𝑛

20. Interpretation of AR Filter
• Depending on whether u(n) is viewed as input or output, we can
interpret

21. MA Models
• It has the structure
𝑢 𝑛 = 𝑣 𝑛 + 𝑏1
∗
𝑣 𝑛 − 1 + ⋯ 𝑏 𝑀
∗
𝑣 𝑛 − 𝐾
• MA model is an all pole filter, the opposite of an AR filter

22. ARMA Models
• It has the structure
𝑢 𝑛 + 𝑎1
∗
𝑢 𝑛 − 1 + ⋯ 𝑎 𝑀
∗
𝑢 𝑛 − 𝑀 = 𝑣 𝑛 + 𝑏1
∗
𝑣 𝑛 − 1 + ⋯ 𝑏 𝑀
∗
𝑣 𝑛 − 𝑀
• ARMA model transfer function of course has both poles and zeros
• Therefore, it is important to make sure all poles are within unit circle whether
used as a process analyzer or process generator.

23. Comparison between AR, MA and ARMA
models
• From a computational point of view, AR model is preferable to
MA/ARMA
• Computation of coefficients of AR model requires solution of a system
of linear equations. These are called the Yule Walker equations.
• Computation of coefficients of MA or ARMA model requires solution
of a system of nonlinear equations.
• Therefore, in practice AR models are much more widely used.

24. Find the coefficients of AR model
• AR model has the structure 𝑘=0
𝑀
𝑎 𝑘
∗
𝑢 𝑛 − 𝑘 = 𝑣 𝑛
• This is a linear, constant coefficient difference equation with M unknowns.
• Multiply both sides by 𝑢∗ 𝑛 − 𝑙 and take expectation.
• We can interchange expectation with summation, and realize that
𝐸 𝑢 𝑛 − 𝑘 ∗
𝑢 𝑛 − 𝑙 = 𝑟 𝑙 − 𝑘 and 𝐸 𝑣 𝑛 ∗
𝑢 𝑛 − 𝑙 = 0 as future realization
of white noise v does not depend on previous filter outputs u.
• Therefore, we can write
which can be written as the difference equation

25. Yule-Waker Equations
• We need to find AR filter coefficients 𝑎1, 𝑎2, … 𝑎 𝑀 and input var 𝜎𝑣
2
• Re-writing the previous equations for 𝑙 = 1,2, … 𝑀, we can write
• These are known as the Yule-Walker equations
• We can also show that
𝑤 𝑘 = −𝑎 𝑘

26. Solve Yule Walker Equations using computer

27. Solution via Matlab
• In this example we will use input v(n) as WGN with zero mean and 𝜎𝑣
2
= 1
v=(randn(1,500));
• Let us generate the data using a filter 𝑢 𝑛 − 0.9𝑢 𝑛 − 1 + 0.5𝑢 𝑛 − 2 = 𝑣 𝑛
u=filter(1, [1 -0.9 0.5],v);
• To build the autocorrelation matrix, we first find the autocorrelations
[ru, lags]=xcorr(u,2,'unbiased'); % only first two lags
ru(1:2)=[]; %delete the first element corresponding to negative lags
• Build the correlation matrix
R=toeplitz(ru(1:2));
• Solve the Yule Walker equations
r=[ru(2) ru(3)]’;
a=-inv(R)*r;
var_v=(ru(1)+a(1)*ru(2)+a(2)*ru(3));
𝑤 𝑘 = −𝑎 𝑘

28. aryule – Matlab function
• We can also use the aryule function of Matlab to directly solve the
Yule Walker equations.
• a = aryule(x,p) returns the normalized autoregressive (AR)
parameters corresponding to a model of order p for the input
array, x. If x is a vector, then the output array, a, is a row vector.
Ifx is a matrix, then the parameters along the nth row
of a model the nth column of x. a has p + 1 columns. p must be
less than the number of elements (or rows) of x.
• [a,e] = aryule(x,p) returns the estimated variance, e, of the
white noise input.

29. Homework No 1
1. Find the autocorrelation of the 𝑥 𝑛 = 𝑎 cos(𝑛𝜔 + 𝜃) , where 𝑎 and 𝜔 are
constants and 𝜃 is normally distributed over the interval −𝜋 to 𝜋.
2. Take a=2, 𝜔=0.01 and 𝜃 is zero mean with variance of 0.5. Verify your analytic
solution in part (1) with Matlab.
3. Calculate the 5x5 autocorrelation matrix for a realization of 5000 samples
(calculate each of the 25 entries using 𝑟 𝑘, 𝑁 =
1
𝑁 𝑛=0
𝑁−1
𝑢 𝑛 𝑢 𝑛 − 𝑘 ).
4. Can you confirm if the realization is wide-sense stationary?
5. Approximate this process using an AR model of order 2,3,…10. What is the
best order of the model? The best model is the one which gives the best
approximation to 𝜎𝑣
2
.