Introductory lecture to Avionics 738 Spring 2018 at Air University PAC Campus by Dr. Bilal A. Siddiqui

1. Av-738
Adaptive Filter Theory
Lecture 1- Introduction
Dr. Bilal A. Siddiqui
Air University (PAC Campus)
Spring 2018

2. Course Outline
• Introduction/Background material
• Linear Optimum Filtering
• Weiner filter
• Kalman filter
• Linear Adaptive Filtering
• Method of steepest descent
• Least Mean Square (LMS) filter
• Recursive Least Squares (RLS) filter
• Tracking time varying systems
• Nonlinear Adaptive Filtering
• Back-propagation learning (BP) neural networks
• Radial basis function (RBF) neural networks
• Applications to student voted areas of interest

3. Grading and then some
• Grades and absentee policies as in vogue at AU.
• One homework every week.
• Difficulty level will increase with each week.
• Quiz of 05-10 minutes each week, so be prepared.
• Term project for the course will also carry a major chunk of the grade.
• We will review some background material but basic knowledge of random
processes, calculus, discrete transforms and complex algebra is assumed.
• I don’t fail people. I abhor it. But getting caught cheating is an automatic F.
• Be regular. The course is fast paced; there is much to loose if you don’t
attend.
• Try to tailor the course project with your thesis. This keeps your interested.
• Try to publish from your course project. This keeps me interested!

4. Text Books
• Simon Haykin, Adaptive Filter Theory, 4th ed., Prentice Hall, 2002.
• Ali H. Syed, “Fundamentals of Adaptive Filtering”, J.W. & Sons, 2003.
• Behrouz Farhang-Boroujeny, “Adaptive Filters – Theory and
Applications”, 2nd Ed., J.W. & Sons, 2013
• Paulo S.R. Diniz, “Adaptive Filtering - Algorithms and Practical
Implementation”, 3rd Ed., Springer, 2008
• Poularikas and Ramadan, “Adaptive Filtering Primer with Matlab”,
CRC Press, 2006.

5. What is this course about?
• Data is often corrupted and coupled with noise.
• Sources of noise are
• Sensor noise and bias
• Quantization error due to digital conversion
• Corruption due to communication channel
• To make sense of it all, one needs to filter the chaff from the grain.
• Filter is a device (hard or soft) which is applied to some noisy and
corrupt data to extract information we desire.
• There is a difference between data and information!

6. What does a filter do?
• A filter does one of the following 3 basic info-processing tasks
• Filtering: extraction of desired information at current time ‘t’ from data
collected upto (and including) ‘t’
• Smoothing: information is desired at some time t0 by using data acquired
before and after some past time 𝑡0 = 𝑡 − 𝜏 including current time t. Since t<t0,
there is a delay in obtaining the desired information. Since data before and
after t0 is used, we expect smoothing to be more accurate than filtering
• Predicting: derive what the desired information will be like in some future time
𝑡 + 𝜏, by using data measured up to (and including) current time t.
• A filter may be linear if its output is a linear function of data
measured.
• It is a nonlinear filter if it is not linear 
• Linear filters are easier to implement, understand and design, but
nonlinear filters are more accurate (but considerably more “difficult”)

7. An example (communications)
• Transmitter converts digital (1-0) signal into a waveform suitable for transmission
• Channel suffers from two types of impairments:
• Intersymbol interference. Channel transmission is “good” in some frequencies and not in
others, which causes some messages to be “smeared”
• Noise. This can be any internal or external (usually additive) interference. Can be electronic
noise due to thermal variations.
• The net result is a received signal which is both a noisy and distorted version of
transmitted message.
• The receiver therefore needs to be equipped with a filter to remove the noise and
recover the original message. Think of how clear videos on YouTube are, for
example.
Digital source of
info
Transmitter Channel Receiver User of info

8. Another example (aerospace)
• Generally the equation describing system evolution (F=ma) is not entirely accurate. This is
called “process noise”. Wind and turbulence maybe another source of system error.
• Errors in measurement may be additive, zero mean noise, or constant (or slow varying)
bias.
• State vector x(t) consists of positions and velocities.
• It may not be possible to measure each state individually.
• In other words, measurements may be functions of various states (e.g. 𝑉𝑣𝑖 =
𝑉 sin(𝜃 − 𝛼))……and it is a pain in the neck measuring 𝛼
• Measurement maybe through some radar which adds noise
• Prior information is assumed knowledge of statistical parameters (mean and correlation)
of process and noise. This again is a problem. How do we know signal statistics?
Aircraft Sensors Filter
observations
y(t)
state
x(t)
state estimate
𝑥(𝑡)
system errors
and disturbances
measurement
errors
Prior
information

9. Optimum Filters
• Optimal means “best”. Here it means best in some statistical sense.
• The requirement to design the best (optimum) filter which takes noisy data as
input and minimizes the effect of noise according to some statistical criterion.
• A widely used statistical criterion is minimization of mean square of the error
signal. Squaring makes the “error surface” smooth, something which makes
optimization easy.
• If the process is “stationary” (mean and variance do not vary with time), the
resulting filter is called the Wiener Filter. The filter parameters will also be fixed
(not varying with time).
• In the more general case of non-stationary process or noise, the filter parameters
will also vary with time. The optimum solution for dynamic process and non-
stationary noise is called Kalman Filter.

10. A disadvantage of Wiener filter
• Wiener filter is time-invariant since the noise and process statistics is stationary (or very slowly varying).
• For time-invariant filters, parameters and the structure of the filter are fixed. Once prescribed specifications are
given, the design of time-invariant linear filters entails three basic steps:
1. the approximation of signal statistics
2. the choice of an appropriate structure defining the filter
3. The choice of the filter parameter, depending on outcomes of steps 1 and 2.
• Weiner filter requires a priori knowledge of statistics of the data to be processed! The filter is optimum only when
actual statistics of input data matches a priori information on which the filter was designed.
• If a priori info is not available, it is not possible to design the Wiener filter.
• One way to do this is to wait for sufficient data to be collected before designing the Wiener filter. This is inefficient
and cannot be applied in real time
• By real time, we mean an operation in which the filter estimate is based on data available now!
• To mitigate this disadvantage, we design “adaptive filters”. Adaptive filter perform steps 1 and 3 above online (real
time). Sometimes, step 2 is also automated for online design.

11. Adaptive Filters
• An adaptive filter is “self-designing in the sense that the algorithm is recursive
(updates filter parameters or structure) every time new data is available.
• Filter parameters are therefore “data dependent” and time varying.
• This makes it possible for filter to perform satisfactorily when complete signal
stats are not available.
• An adaptive filter is also required when performance specifications cannot be
satisfied by time-invariant filters.
• The algorithm starts from initial conditions representing the “best guess” of signal
statistics we think represent the environment we operate the filter in.
• In a stationary environment (process in not dynamic and noise stats are time-
invariant), the adaptive filter eventually converges to the Wiener filter after
“some” iterations!
• In other words, instead of finding the optimum filter parameters and performance
in one shot, we hit the optimum by “learning” in a “trial and error” manner.
• In a non-stationary environment, adaptive filters can “track” sufficiently slow
variations in signal statistics (depending on “filter bandwidth”).

12. Is the adaptive filter linear?
• What does linearity mean?
• Superposition holds (sum of two inputs produce the sum of their individual
outputs, i.e. 𝑦 𝑥1 + 𝑥2 = 𝑦 𝑥1 + 𝑦 𝑥2
• Homogeneity holds (scaling the input by a factor scales the output by the
same factor, i.e. 𝑦 𝑎. 𝑥 = 𝑎. 𝑦(𝑥)
• Adaptive filters are time-varying since their parameters are
continually changing in order to meet a performance requirement.
• Strictly speaking, an adaptive filter is a nonlinear filter since its
characteristics are dependent on the input signal and consequently
the homogeneity and superposition conditions are not satisfied.
• However, if we freeze the filter parameters at a given instant
of time, most adaptive filters considered in this course are linear in
the sense that their output signals are linear functions of their input
signals.
• Are neural networks linear filters?

13. Choice of Adaptive Filter algorithms
• A wide variety of adaptive filtering algorithms have been developed
• Choice of one algo over another depends on following factors
• Rate of convergence (in how many iterations does the algo converge “close”
to the performance of Wiener filter)
• Misadjustment (how “far” was the final estimate from the Wiener estimate)
• Tracking (ability to track variations in signal statistics)
• Robustness (disturbances should not produce large errors in the estimate)
• Computational Cost (number of FLOPs and memory requirements)
• Modularity (processes can be cascaded (in series). This favors VLSI
implementation)
• Parallelization (not all algorithms can be parallelized to use GPU)
• Numerical accuracy (robustness against word-length of ADC  quantization
errors)

14. Applications of Adaptive filtering
• A wide variety of applications in
• Digital signal processing (DSP)
• Control systems (particularly nonlinear control)
• Adaptive filters have received wide applications in
• Communications
• Radar/Sonar
• Seismology
• Biomedical engineering
• Aerospace and mechanical engineering etc.
• There is something common in all of the above, though.
• There is an input, a desired response and the error
between them to adjust filter parameters.
• There are four classes of adaptive filtering applications

15. Application Class I
System Identification
• In this class, the adaptive filter is used to identify a mathematical model
which represents the “best fit” to the behavior of the unknown system
(plant)
• Plant and filter are driven by the same input.
• Therefore, the input is also designed to excite all possible behaviors of
the plant. This is an important point: the input should be “sufficiently
exciting”.
• The resulting model can also be used for control design and prediction.
• Sys Id=Channel estimation in Communication Eng.

16. Application Class II
Inverse Modeling / Channel Equalization
• In this class, the adaptive filter is used to provide an inverse model that
represents the “best” cancellation of the unknown plant’s dynamics.
• Ideally, for a linear plant, the transfer function of the filter will be the
reciprocal of the plant’s transfer function.
• In Communications Eng., this is the task of Channel Equalization
(combination of plant (channel) and its inverse represent an ideal
transmission medium.
• A delayed version of system input serves as desired response. In some
cases no delay is used

17. Application Class III
Prediction
• In this class, the adaptive filter is used to provide the best prediction (in a
statistical sense) of the present (or future) value of a random signal (original
signal corrupted with noise)
• Past values of the signal serve as inputs to the filter
• Depending on the application, the filter output or prediction error serves as
system output
• For output 2, the system operates as a predictor.
• For output 1, it operates as a prediction-error filter. This error is used to drive
some other process. This is used in speech recognition.

18. Application Class IV
Interference Cancellation
• In this class, the adaptive filter is used to cancel unknown interference
in the primary signal
• Primary signal = sensor measurement = information carrying signal +
unknown interference
• A reference signal is used as input to excite the filter
• The purpose is to cancel noise and interferences (slow varying biases)

19. Applications in Brief
Class of Adaptive Filtering Application
1. Identification 1. System identification (control design)
2. Channel estimation
3. Layered earth modeling
2. Inverse 1. Channel equalization
2. Blind deconvolution
3. Prediction 1. Linear predictive coding (speech recognition)
2. Spectrum analysis
3. Signal detection
4. Interference cancellation 1. Noise and echo cancellation (ECG)
2. Adaptive beam-forming (radars)

20. Some historical notes
(Prof. Ali Syed, UCLA)

21. Adaptive Filter Structure
• Operation of linear adaptive filters requires two basic processes:
• A filtering process which produces an output response to a sequence of input
data
• An adaptation process which recursively adjusts parameters of the filter
• These two processes are interactive.
• Choice of structure of the filter obviously has a profound effect on the
operation of the algorithm
• Linear adaptive filters generally have two forms:
• Infinite impulse response (IIR) filters (infinite but fading memory)
• Finite impulse response (FIR) filters (finite memory)

22. Filter Structure:
Transversal FIR Filters
• Filters with finite memory (or those which discard measurements beyond a certain
point in history) are of two basic types.
• The first type is “transversal filter”, aka “tapped delay line”
• Consists of (1) unit element delay (2) multiplier and (3) adder
• Number of delay elements (M) determines filter length aka filter order
𝑦 𝑛 = 𝑘=0
𝑀
𝑤 𝑘 ∗ 𝑢(𝑛 − 𝑘)
* represents
complex
conjugate, since in
general signal may
be complex

23. Filter Structure:
Lattice FIR Filter
• Lattice predictors have a modular structure: number of cascaded stages.
• Each stage “looks” like a lattice cell, hence the name.
• Each stage is represented by a pair of input-output relations
𝑓𝑚 𝑛 = 𝑓 𝑚−1 𝑛 + 𝑘 𝑚
∗
𝑏 𝑚−1 𝑛 − 1
𝑏 𝑚 𝑛 = 𝑏 𝑚−1 𝑛 − 1 + 𝑘 𝑚 𝑓 𝑚−1(𝑛)
• Here, f is the forward prediction error and b is the backward prediction error.
• The filter is initialized as
𝑓0 𝑛 = 𝑏0 𝑛 = 𝑢0(𝑛)
Each stage is solved sequentially.
A linear combination of backward prediction errors can be used to provide prediction of
some desired response.

24. Filter Structure: FIR Lattice Predictor

25. Filter Structure:
Recursive or IIR filters
• FIR filters only had feed forward
elements. There is no danger of
instability!
• IIR filters on the other hand have
feedback elements. Feedback gives
infinite memory. But feedback can
also destabilize if it is “too much”.
• IIR can become unstable if the filter
parameters are not chosen correctly.
• Nevertheless IIR filters have their
uses and there are methods of
tuning them and keeping them
stable

26. Adaptation Techniques
• There is no unique way of adapting the filter parameters (weights)
• It is important to understand capabilities and limitations of all
available adaptation algorithms.
• This understanding will allow us to select the
• Two of the more popular techniques for adaptation are
• Stochastic gradient adaptation algorithms
• Least squares adaptation algorithms
• We discuss each briefly, next.

27. Adaptation Algorithms
Stochastic Gradient Approach
• Structure used for implementation is transversal (tapped-delay line)
• The cost function to be minimized is the mean-squared error.
• Cost function is shaped like a multidimensional paraboloid, with a
unique minimum (which corresponds to Wiener solution), which we
try to reach by the “method of steepest descent”
• The cost function is a second order function of filter weights
• The recursive algorithm which results is called the celebrated Least
Mean Squares (LMS) algorithm

28. Adaptation Algorithms
Stochastic Gradient Approach
• The tap-weights (filter-weights) are updated with the following law
• In a non-stationary environment the error-performance surface changes
continuously, so the LMS must continually track the bottom of this surface
• Changes in the input statistics must be slow compared with the LMS
learning rate for tracking to occur
• It also converges very slowly and is sensitive to the “quality” of input data.
• For lattice filter structure, the stochastic gradient approach produces the
“Gradient Adaptive Lattice” (GAL) algorithm.

29. Adaptation Algorithms
Least Squares Estimation Approach
• These algorithms adapt filter weights based on the method of least
squares developed by Gauss.
• Cost function consists of weighted sum of squared errors.
• One popular algorithm of this kind is the Recursive Least Squares (RLS)
algorithm which is a special form of Kalman filter.
• In a Kalman filter, a “state” is updated using new information coming into
the filter (called “innovation”)

30. Adaptation Algorithms
Least Squares Estimation Approach
• Stochastic gradient algorithms are model independent. They have good
tracking performance.
• Least squares algorithms are model dependent and may perform inferior to
LMS algorithms if the model structure is not accurate.
• However, RLS converges more rapidly than LMS.
• Three basic classes of recursive least squares algorithms exist:
• Standard Recursive least-squares (RLS): The algorithm relies on the matrix-inversion
lemma; the algorithm converges rapidly and has rather high computational
complexity; lack
of numerical robustness
– Square-root RLS: numerically robust form of the standard RLS
– Fast RLS: less computations (hence faster) version, but numerically less stable.

31. How to choose adaptive filtering algorithms?
• When choosing an adaptive filter practical issues that are important:
• computational cost
• Performance
• Robustness
• Adaptive filter algorithms generally assume the input data is in
baseband form, for bandpass signals this means complex baseband
following frequency translation
𝑢 𝑛 = 𝑢𝐼 𝑛 + 𝑗𝑢 𝑄 𝑛
Algorithms are thus typically developed in complex form, with
the real form being a special case
• The use of computer simulation is very useful as a first step in
the evaluation process

32. Term Projects
• Floor is open for discussion