In these two lectures, we’re looking at basic discrete time representations of linear, time invariant plants and models and seeing how their parameters can be estimated using the normal equations.
The key example is the first order, linear, stable RC electrical circuit which we met last week, and which has an exponential response.
MATHEMATICAL MODELING OF COMPLEX REDUNDANT SYSTEM UNDER HEAD-OF-LINE REPAIREditor IJMTER
Suppose a composite system consisting of two subsystems designated as ‘P’ and
‘Q’ connected in series. Subsystem ‘P’ consists of N non-identical units in series, while the
subsystem ‘Q’ consists of three identical components in parallel redundancy.
Exponential State Observer Design for a Class of Uncertain Chaotic and Non-Ch...ijtsrd
In this paper, a class of uncertain chaotic and non-chaotic systems is firstly introduced and the state observation problem of such systems is explored. Based on the time-domain approach with integral and differential equalities, an exponential state observer for a class of uncertain nonlinear systems is established to guarantee the global exponential stability of the resulting error system. Besides, the guaranteed exponential convergence rate can be calculated correctly. Finally, numerical simulations are presented to exhibit the feasibility and effectiveness of the obtained results. Yeong-Jeu Sun "Exponential State Observer Design for a Class of Uncertain Chaotic and Non-Chaotic Systems" Published in International Journal of Trend in Scientific Research and Development (ijtsrd), ISSN: 2456-6470, Volume-3 | Issue-1 , December 2018, URL: http://www.ijtsrd.com/papers/ijtsrd20219.pdf
http://www.ijtsrd.com/engineering/electrical-engineering/20219/exponential-state-observer-design-for-a-class-of-uncertain-chaotic-and-non-chaotic-systems/yeong-jeu-sun
I am Green J. I am a Signal Processing Assignment Expert at matlabassignmentexperts.com. I hold a Ph.D. in Matlab, University of Oxford, UK. I have been helping students with their assignments for the past 10 years. I solve assignments related to Signal Processing.
Visit matlabassignmentexperts.com or email info@matlabassignmentexperts.com. You can also call on +1 678 648 4277 for any assistance with Signal Processing Assignments.
An Exponential Observer Design for a Class of Chaotic Systems with Exponentia...ijtsrd
In this paper, a class of generalized chaotic systems with exponential nonlinearity is studied and the state observation problem of such systems is explored. Using differential inequality with time domain analysis, a practical state observer for such generalized chaotic systems is constructed to ensure the global exponential stability of the resulting error system. Besides, the guaranteed exponential decay rate can be correctly estimated. Finally, several numerical simulations are given to demonstrate the validity, effectiveness, and correctness of the obtained result. Yeong-Jeu Sun "An Exponential Observer Design for a Class of Chaotic Systems with Exponential Nonlinearity" Published in International Journal of Trend in Scientific Research and Development (ijtsrd), ISSN: 2456-6470, Volume-5 | Issue-1 , December 2020, URL: https://www.ijtsrd.com/papers/ijtsrd38233.pdf Paper URL : https://www.ijtsrd.com/engineering/electrical-engineering/38233/an-exponential-observer-design-for-a-class-of-chaotic-systems-with-exponential-nonlinearity/yeongjeu-sun
MATHEMATICAL MODELING OF COMPLEX REDUNDANT SYSTEM UNDER HEAD-OF-LINE REPAIREditor IJMTER
Suppose a composite system consisting of two subsystems designated as ‘P’ and
‘Q’ connected in series. Subsystem ‘P’ consists of N non-identical units in series, while the
subsystem ‘Q’ consists of three identical components in parallel redundancy.
Exponential State Observer Design for a Class of Uncertain Chaotic and Non-Ch...ijtsrd
In this paper, a class of uncertain chaotic and non-chaotic systems is firstly introduced and the state observation problem of such systems is explored. Based on the time-domain approach with integral and differential equalities, an exponential state observer for a class of uncertain nonlinear systems is established to guarantee the global exponential stability of the resulting error system. Besides, the guaranteed exponential convergence rate can be calculated correctly. Finally, numerical simulations are presented to exhibit the feasibility and effectiveness of the obtained results. Yeong-Jeu Sun "Exponential State Observer Design for a Class of Uncertain Chaotic and Non-Chaotic Systems" Published in International Journal of Trend in Scientific Research and Development (ijtsrd), ISSN: 2456-6470, Volume-3 | Issue-1 , December 2018, URL: http://www.ijtsrd.com/papers/ijtsrd20219.pdf
http://www.ijtsrd.com/engineering/electrical-engineering/20219/exponential-state-observer-design-for-a-class-of-uncertain-chaotic-and-non-chaotic-systems/yeong-jeu-sun
I am Green J. I am a Signal Processing Assignment Expert at matlabassignmentexperts.com. I hold a Ph.D. in Matlab, University of Oxford, UK. I have been helping students with their assignments for the past 10 years. I solve assignments related to Signal Processing.
Visit matlabassignmentexperts.com or email info@matlabassignmentexperts.com. You can also call on +1 678 648 4277 for any assistance with Signal Processing Assignments.
An Exponential Observer Design for a Class of Chaotic Systems with Exponentia...ijtsrd
In this paper, a class of generalized chaotic systems with exponential nonlinearity is studied and the state observation problem of such systems is explored. Using differential inequality with time domain analysis, a practical state observer for such generalized chaotic systems is constructed to ensure the global exponential stability of the resulting error system. Besides, the guaranteed exponential decay rate can be correctly estimated. Finally, several numerical simulations are given to demonstrate the validity, effectiveness, and correctness of the obtained result. Yeong-Jeu Sun "An Exponential Observer Design for a Class of Chaotic Systems with Exponential Nonlinearity" Published in International Journal of Trend in Scientific Research and Development (ijtsrd), ISSN: 2456-6470, Volume-5 | Issue-1 , December 2020, URL: https://www.ijtsrd.com/papers/ijtsrd38233.pdf Paper URL : https://www.ijtsrd.com/engineering/electrical-engineering/38233/an-exponential-observer-design-for-a-class-of-chaotic-systems-with-exponential-nonlinearity/yeongjeu-sun
The issues about maneuvering target track prediction were discussed in this paper. Firstly, using Kalman filter which based on current statistical model describes the state of maneuvering target motion, thereby analyzing time range of the target maneuvering occurred. Then, predict the target trajectory in real time by the improved gray prediction model. Finally, residual test and posterior variance test model accuracy, model accuracy is accurate.
Performance Comparison of Identification Methods Applied to Power Systems for...Reza Pourramezan
Authors: Reza Pourramezan, Sadegh Vaez-Zadeh, and Hamid Reza Nourzadeh
Published in 2006 IEEE International Conference on Industrial Technology (ICIT)
DOI: 10.1109/ICIT.2006.372551
http://ieeexplore.ieee.org/document/4237873/
2013.06.18 Time Series Analysis Workshop ..Applications in Physiology, Climat...NUI Galway
Professor Dimitris Kugiumtzis, Aristotle University of Thessaloniki, Greece, presented this workshop on nonlinear analysis of time series as part of the Summer School on Modern Statisitical Analysis and Computational Methods hosted by the Social Sciences Compuing Hub at the Whitaker Institute, NUI Galway on 17th-19th June 2013.
The generalized design principle of TS fuzzy observers for one class of continuous-time
nonlinear MIMO systems is presented in this paper. The problem addressed can be indicated as
a descriptor system approach to TS fuzzy observers design, implying the asymptotic convergence of the state observer error. A new structure of linear matrix inequalities is outlined to possess the observer asymptotic dynamic properties closest to the optimal.
The ability to recreate computational results with minimal effort and actionable metrics provides a solid foundation for scientific research and software development. When people can replicate an analysis at the touch of a button using open-source software, open data, and methods to assess and compare proposals, it significantly eases verification of results, engagement with a diverse range of contributors, and progress. However, we have yet to fully achieve this; there are still many sociotechnical frictions.
Inspired by David Donoho's vision, this talk aims to revisit the three crucial pillars of frictionless reproducibility (data sharing, code sharing, and competitive challenges) with the perspective of deep software variability.
Our observation is that multiple layers — hardware, operating systems, third-party libraries, software versions, input data, compile-time options, and parameters — are subject to variability that exacerbates frictions but is also essential for achieving robust, generalizable results and fostering innovation. I will first review the literature, providing evidence of how the complex variability interactions across these layers affect qualitative and quantitative software properties, thereby complicating the reproduction and replication of scientific studies in various fields.
I will then present some software engineering and AI techniques that can support the strategic exploration of variability spaces. These include the use of abstractions and models (e.g., feature models), sampling strategies (e.g., uniform, random), cost-effective measurements (e.g., incremental build of software configurations), and dimensionality reduction methods (e.g., transfer learning, feature selection, software debloating).
I will finally argue that deep variability is both the problem and solution of frictionless reproducibility, calling the software science community to develop new methods and tools to manage variability and foster reproducibility in software systems.
Exposé invité Journées Nationales du GDR GPL 2024
The issues about maneuvering target track prediction were discussed in this paper. Firstly, using Kalman filter which based on current statistical model describes the state of maneuvering target motion, thereby analyzing time range of the target maneuvering occurred. Then, predict the target trajectory in real time by the improved gray prediction model. Finally, residual test and posterior variance test model accuracy, model accuracy is accurate.
Performance Comparison of Identification Methods Applied to Power Systems for...Reza Pourramezan
Authors: Reza Pourramezan, Sadegh Vaez-Zadeh, and Hamid Reza Nourzadeh
Published in 2006 IEEE International Conference on Industrial Technology (ICIT)
DOI: 10.1109/ICIT.2006.372551
http://ieeexplore.ieee.org/document/4237873/
2013.06.18 Time Series Analysis Workshop ..Applications in Physiology, Climat...NUI Galway
Professor Dimitris Kugiumtzis, Aristotle University of Thessaloniki, Greece, presented this workshop on nonlinear analysis of time series as part of the Summer School on Modern Statisitical Analysis and Computational Methods hosted by the Social Sciences Compuing Hub at the Whitaker Institute, NUI Galway on 17th-19th June 2013.
The generalized design principle of TS fuzzy observers for one class of continuous-time
nonlinear MIMO systems is presented in this paper. The problem addressed can be indicated as
a descriptor system approach to TS fuzzy observers design, implying the asymptotic convergence of the state observer error. A new structure of linear matrix inequalities is outlined to possess the observer asymptotic dynamic properties closest to the optimal.
The ability to recreate computational results with minimal effort and actionable metrics provides a solid foundation for scientific research and software development. When people can replicate an analysis at the touch of a button using open-source software, open data, and methods to assess and compare proposals, it significantly eases verification of results, engagement with a diverse range of contributors, and progress. However, we have yet to fully achieve this; there are still many sociotechnical frictions.
Inspired by David Donoho's vision, this talk aims to revisit the three crucial pillars of frictionless reproducibility (data sharing, code sharing, and competitive challenges) with the perspective of deep software variability.
Our observation is that multiple layers — hardware, operating systems, third-party libraries, software versions, input data, compile-time options, and parameters — are subject to variability that exacerbates frictions but is also essential for achieving robust, generalizable results and fostering innovation. I will first review the literature, providing evidence of how the complex variability interactions across these layers affect qualitative and quantitative software properties, thereby complicating the reproduction and replication of scientific studies in various fields.
I will then present some software engineering and AI techniques that can support the strategic exploration of variability spaces. These include the use of abstractions and models (e.g., feature models), sampling strategies (e.g., uniform, random), cost-effective measurements (e.g., incremental build of software configurations), and dimensionality reduction methods (e.g., transfer learning, feature selection, software debloating).
I will finally argue that deep variability is both the problem and solution of frictionless reproducibility, calling the software science community to develop new methods and tools to manage variability and foster reproducibility in software systems.
Exposé invité Journées Nationales du GDR GPL 2024
Earliest Galaxies in the JADES Origins Field: Luminosity Function and Cosmic ...Sérgio Sacani
We characterize the earliest galaxy population in the JADES Origins Field (JOF), the deepest
imaging field observed with JWST. We make use of the ancillary Hubble optical images (5 filters
spanning 0.4−0.9µm) and novel JWST images with 14 filters spanning 0.8−5µm, including 7 mediumband filters, and reaching total exposure times of up to 46 hours per filter. We combine all our data
at > 2.3µm to construct an ultradeep image, reaching as deep as ≈ 31.4 AB mag in the stack and
30.3-31.0 AB mag (5σ, r = 0.1” circular aperture) in individual filters. We measure photometric
redshifts and use robust selection criteria to identify a sample of eight galaxy candidates at redshifts
z = 11.5 − 15. These objects show compact half-light radii of R1/2 ∼ 50 − 200pc, stellar masses of
M⋆ ∼ 107−108M⊙, and star-formation rates of SFR ∼ 0.1−1 M⊙ yr−1
. Our search finds no candidates
at 15 < z < 20, placing upper limits at these redshifts. We develop a forward modeling approach to
infer the properties of the evolving luminosity function without binning in redshift or luminosity that
marginalizes over the photometric redshift uncertainty of our candidate galaxies and incorporates the
impact of non-detections. We find a z = 12 luminosity function in good agreement with prior results,
and that the luminosity function normalization and UV luminosity density decline by a factor of ∼ 2.5
from z = 12 to z = 14. We discuss the possible implications of our results in the context of theoretical
models for evolution of the dark matter halo mass function.
What is greenhouse gasses and how many gasses are there to affect the Earth.moosaasad1975
What are greenhouse gasses how they affect the earth and its environment what is the future of the environment and earth how the weather and the climate effects.
Professional air quality monitoring systems provide immediate, on-site data for analysis, compliance, and decision-making.
Monitor common gases, weather parameters, particulates.
This presentation explores a brief idea about the structural and functional attributes of nucleotides, the structure and function of genetic materials along with the impact of UV rays and pH upon them.
THE IMPORTANCE OF MARTIAN ATMOSPHERE SAMPLE RETURN.Sérgio Sacani
The return of a sample of near-surface atmosphere from Mars would facilitate answers to several first-order science questions surrounding the formation and evolution of the planet. One of the important aspects of terrestrial planet formation in general is the role that primary atmospheres played in influencing the chemistry and structure of the planets and their antecedents. Studies of the martian atmosphere can be used to investigate the role of a primary atmosphere in its history. Atmosphere samples would also inform our understanding of the near-surface chemistry of the planet, and ultimately the prospects for life. High-precision isotopic analyses of constituent gases are needed to address these questions, requiring that the analyses are made on returned samples rather than in situ.
THE IMPORTANCE OF MARTIAN ATMOSPHERE SAMPLE RETURN.
LeastSquaresParameterEstimation.ppt
1. EE-M110 2006/7, EF L5&6 1/29, v2.0
Lectures 5 & 6:
Least Squares Parameter Estimation
q1
q2
f(q)
Dr Martin Brown
Room: E1k, Control Systems Centre
Email: martin.brown@manchester.ac.uk
Telephone: 0161 306 4672
http://www.eee.manchester.ac.uk/intranet/pg/coursematerial/
2. EE-M110 2006/7, EF L5&6 2/29, v2.0
L5&6: Resources
Core texts
• Ljung, Chapters 4&7
• Norton, Chapter 4
• On-line, Chapters 4&5
In these two lectures, we’re looking at basic discrete time
representations of linear, time invariant plants and
models and seeing how their parameters can be
estimated using the normal equations.
The key example is the first order, linear, stable RC
electrical circuit which we met last week, and which has
an exponential response.
3. EE-M110 2006/7, EF L5&6 3/29, v2.0
L5&6: Learning Objectives
L5 Linear models and quadratic performance criterion
– ARX & ARMAX discrete-time, linear systems
– Predictive models, regression and exemplar data
– Residual signal
– Performance criterion
L6 Normal equations, interpretation and properties
– Quadratic cost functions
– Derive the normal equations for parameter estimation
– Examples
We’re not too concerned with system dynamics today, we’re
concentrating on the general form of least squares parameter
estimation
4. EE-M110 2006/7, EF L5&6 4/29, v2.0
Introduction to Parametric System Identification
In a full, physical, linear model, the model’s structure and
coefficients can be determined from first principles
In most cases, we have to estimate/tune the parameters because
of an incomplete understanding about the full system (unknown
drag, …)
We can use exemplar data (input/output examples), {x(t), y(t)}, to
estimate the unknown parameters
Initially assume that the structure is known (unrealistic, but …), and
all that remains to be estimated are the parameter values.
Plant
q
Model
q
u(t)
y(t)
y(t)
^
^
w(t)
v(t)
)
(
)
(
)
(
t
w
t
u
t
x
e(t)
5. EE-M110 2006/7, EF L5&6 5/29, v2.0
Recursive Parameter Estimation Framework
where:
q, q(t-1) are the real and estimated parameter vectors, respectively.
u(t) is the control input sequence
y(t), y(t), are the real and estimated outputs, respectively
e(t) is a white noise sequence (output/measurement noise)
w(t) is the disturbances from measurable sources
Controller
Plant
q
Model
q(t-1)
u(t) y(t)
y(t)
^
^
e(t)
w(t)
+
+
+
-
^
^
v(t)
6. EE-M110 2006/7, EF L5&6 6/29, v2.0
Basic Assumptions in System Identification
1) It is assumed that the unobservable disturbances can be
aggregated and represented by a single additive noise e(t).
There may also be input noise. Generally, it is assumed to be
zero-mean, Gaussian
2) The system is assumed to be linear with time-invariant
parameters, so q is not time-varying. This is only
approximately true within certain limits
3) The input signal u(t) is assumed exactly known. Often there
is noise associated with reading/measuring it
4) The system noise e(t) is assumed to be uncorrelated with
the input process u(t). This is unlikely to be true for instance
to due feedback of y(t)
5) The input signals need to be sufficiently exciting, they need
to excite all relevant modes in the model for identification and
testing
7. EE-M110 2006/7, EF L5&6 7/29, v2.0
Discrete-Time Transfer Function Models
On this course, we’re primarily concerned with discrete time signals and
systems.
Real-world physical, mechanical, electrical systems are continuous
Consider the CT resistor-capacitor circuit:
So let q-1 denote the backward shift operator q-1y(t)=y(t-1), then we have
NB we can use the c2d() Matlab function to go from the continuous time
(transfer function, state space) domain to the discrete time, z-domain.
( ) )
1
(
)
1
(
1
)
(
)
(
)
(
)
(
)
1
(
)
(
)
(
)
(
t
u
t
y
t
y
t
u
t
y
t
y
t
y
RC
t
u
t
y
dt
t
dy
RC
RC
RC
( ) 1
1
)
(
1
1
)
(
)
(
)
(
)
(
)
(
q
q
B
q
q
A
t
u
q
B
t
y
q
A
RC
RC
8. EE-M110 2006/7, EF L5&6 8/29, v2.0
Transfer Function/ARX DT LTI Model
The previous model is an example of an AutoRegressive with eXogenous
input), which can be expressed more generally as:
Some comments about the form of this model.
1. The degree of the polynomials determines the complexity of the system’s
response and the number of parameters that have to be estimated. The
roots of A(q) determine system stability
2. a0=1, without loss of generality, so the model can be written as a predictive
model y(t) = y(t-1) + … + u(t-1) + …
3. b0=0, as it is assumed that an input cannot instantly affect the output, and
so there must be at least a delay of one time instant between u & y
(assumes a fast enough sample time, relative to the system dynamics).
4. Typically e~N(0,s2) – independent and identically distributed
5. Close relationship between the q-shift and z-transform
6. When n=0, this produces a finite impulse response
m
m
n
n
q
b
q
b
q
B
q
a
q
a
q
A
t
e
t
u
q
B
t
y
q
A
1
1
1
1
)
(
1
)
(
)
(
)
(
)
(
)
(
)
(
9. EE-M110 2006/7, EF L5&6 9/29, v2.0
Linear Regression
The ARX system’s prediction model can be expressed as
• Here the model’s parameters can be written as:
• Treat the model as a deterministic system
• This is natural if the error term is considered to be insignificant or difficult
to guess
• This denotes the model structure M (linear, time invariant, for example),
and a particular model with a parameter value q, is M(q).
This can be written as a linear regression structure:
where
Parameter vector:
Input vector:
The term regression comes from the statistics literature and provides a
powerful set of techniques for determining the parameters and
interpretating the models. Need access to previous outputs y(t-1) …
)
(
))
(
1
(
)
(
)
(
)
|
(
ˆ t
y
q
A
t
u
q
B
t
y
θ
θ
x
θ )
(
)
|
(
ˆ t
t
y T
T
m
n b
b
a
a ]
,
,
,
,
,
[ 1
1
θ
T
m
t
u
t
u
n
t
y
t
y
t )]
(
,
),
1
(
),
(
,
),
1
(
[
)
(
x
T
m
n b
b
a
a ]
,
,
,
,
,
[ 1
1
θ
10. EE-M110 2006/7, EF L5&6 10/29, v2.0
LTI DT ARMAX Model
A more general discrete time, linear time invariant model also includes
Moving Average terms on the error/residual signal
Here, we describe the equation error term, e(t), as a moving average of
white noise (non-iid measurement errors)
Simple example
y(t) = 0.5y(t-1) + 0.3y(t-2) + 1.2u(t-1) - 0.3u(t-2) + 0.5e(t) + 0.5e(t-1)
This can be written as a pseudolinear regression
c
c
b
b
a
a
n
n
n
n
n
n
q
c
q
c
q
C
q
b
q
b
q
B
q
a
q
a
q
A
t
e
q
C
t
u
q
B
t
y
q
A
1
1
1
1
1
1
1
)
(
)
(
1
)
(
)
(
)
(
)
(
)
(
)
(
)
(
)
(
)
(
)
(
1
)
(
)
(
)
(
)
|
(
ˆ t
y
q
C
q
A
t
u
q
C
q
B
t
y
θ
11. EE-M110 2006/7, EF L5&6 11/29, v2.0
Exemplar Training Data
To estimate the unknown parameters q, we need to collect some
exemplar input-output data, and system identification is then a process
of estimating the parameter values that best fit the data.
The data is generated by a system of noisy ARX linear equations of the
form
where
y is a column vector of measured plant outputs (T,1)
X is a matrix of input regressors (T,n+m)
q is the “true” parameter vector (n+m,1)
e is the error vector (T,1)
Each row of X represents a single input/output sample. Each column of
X represents a time delayed output or input.
Note that there is a “burn-in” period to measure the time-delayed outputs
y(1), y(2), … which are necessary to form the inputs to the time-delayed
vector
[y(1), …, y(t-n)]
e
Xθ
y
12. EE-M110 2006/7, EF L5&6 12/29, v2.0
Example: Data for 1st Order ARX Model
1st Order model representation
First order plant model (exponential decay) with no external disturbances
and the measurement noise is additive (Slide 7)
Input vector, output signal and parameters
At time t, the 1st order DT model is represented as
Output y(t)
Input x(t) = [y(t-1); u(t-1)]
Parameters q = [q1; q2]
Data
As there are two parameters, if the system is truly first order and there is
no measurement noise on any of the signals, we just need two (linearly
independent) samples to estimate q.
If there is measurement noise in y(t), we need to collect more data to
reduce the effect of the random noise.
Store X=[y(1) u(1); y(2) u(2); y(3) u(3); …], y=[y(2); y(3); y(4); …]
13. EE-M110 2006/7, EF L5&6 13/29, v2.0
Prediction Residual Signal
The residual signal (measured-predicted) is defined as:
and can be represented as:
A simple regression interpretation is
(each x represents an exemplar
sample from a single input,
single output system)
)
(t
x
Plant
q
Model
q
y(t)
y(t)
^
^
e(t)
+
+
+
-
x(t)
x(t) r(t)
output
measurement
“residual”
x
x
x
x
x
x
x
)
(
ˆ
)
(
)
( t
y
t
y
t
r
)
(
ˆ
)
(
t
y
t
y
)
(
ˆ t
y
)
(t
y
)
(t
r
14. EE-M110 2006/7, EF L5&6 14/29, v2.0
The model’s response can be expressed as
y(t) = xT(t)q
where q is the model’s estimated parameter vector and x(t) is
the input vector
If y(t)=y(t), the model’s response is correct for that single
time sample. The residual r(t)=y(t)-y(t) is zero. The
residual’s magnitude gives us an idea of the “goodness” of
the parameter vector estimate for that data point.
For a set of measured outputs and predictions {y(t),y(t)}t, the
“size” of the residual vector r=y-y, is an estimate of the
parameter goodness
We can determine the size by looking at the norm of r.
Measures of Model Goodness
^ ^
^
^
^
^
^
15. EE-M110 2006/7, EF L5&6 15/29, v2.0
Residual Norm Measures
A vector p-norm (of a vector r) is defined by:
The most common p-norm is the 2-norm:
The vector p-norm has the properties that:
• ||r|| 0
• ||r|| = 0 iff r = 0
• ||kr|| = k||r||
• ||r1+r2|| ||r1||+||r2||
For the residual vector, the norm is only zero if all the
residuals are zero. Otherwise, a small norm means that, on
average, the individual residuals are small in magnitude.
p
T
i
p
i
p
r
1
1
r
T
i
i
r
1
2
2
r
16. EE-M110 2006/7, EF L5&6 16/29, v2.0
Sum of Squared Residuals
The most common discrete time performance index is the
sum of squared residuals (2-norm squared):
For each data point, the model’s output is compared
against the plants and error is squared and summed
over the remaining points.
Any non-zero value for any of the residual values will mean
that the performance index is positive
The performance function f(q) is a function of the
parameter values, because some parameter values will
cause large residuals, others will cause small residuals.
We want the parameter values that minimize f(q) (0).
T
i
i
i y
y
f
1
2
^
2
2
)
( r
θ
17. EE-M110 2006/7, EF L5&6 17/29, v2.0
The aim of parameter estimation is to estimate the values of q that
minimize this performance index (sum squared residuals or errors
SSE).
When the model can predict the model exactly:
r(t) = e(t)
The residual signal is equal to the additive noise signal
Note that the SSE is often replaced by the mean squared error MSE
defined by
MSE = SSE/T s2 (the variance of the additive noise signal)
This is the variance of the residual signal.
This is simply represents the average squared error and ensures that the
performance function does not depend on the amount of data
Example, when we have 1000 repeated trials (step responses) of 9 data
points for the DT electric circuit, with additive noise N(0,0.01)
MSE = ||r||2
2/T = 0.0103 s2
RMSE = 0.1015 s`
Relationship between Noise & Residual
18. EE-M110 2006/7, EF L5&6 18/29, v2.0
Example: DT RC Electrical Circuit
Consider the DT, first order, LTI representation of the RC circuit which is an
ARX model (Slide 7 & 12)
Assume that /RC=0.5, then:
y(t) = 0.5*y(t-1) + 0.5*u(t-1)
Here the system is initially at rest y(0)=0. Note that u here refers to a step
signal which is switched on at t=1 & u(0)=0, rather than the control
signal
Assume that 10 steps are taken, we collect 9 data points for system
identification:
>> X=[y(1:end-1)’ u(1:end-1)];
>> y1 = y(2:end)’;
Gaussian random noise of standard error 0.05 was also added to y1
>> y1e = y1+0.05*randn(size(y1));
1
992
.
0
1
5
.
0
1
0
0
0
X
996
.
0
75
.
0
5
.
0
0
y
19. EE-M110 2006/7, EF L5&6 19/29, v2.0
Example: Noisy Electric Circuit
Note here, we’re cheating a bit by assuming the exact
measurement y(t-1) is available to the model’s input but only
the noisy measurement ye(t) is available to the model’s output.
NB, in these notes, y() generally denotes the noisy output
1
992
.
0
1
5
.
0
1
0
0
0
X
996
.
0
75
.
0
5
.
0
0
y
008
.
1
774
.
0
510
.
0
037
.
0
)
01
.
0
,
0
(
y
ye
090
.
0
ˆ
,
0081
.
0
ˆ
,
073
.
0
033
.
0
029
.
0
001
.
0
0367
.
0
9744
.
0
744
.
0
511
.
0
0
ˆ
]
5173
0
4756
0
[
ˆ
2
2
2
s
s
r
r
y
θ
T
T
T
.
.
NB
randn(‘state’, 123456)
20. EE-M110 2006/7, EF L5&6 20/29, v2.0
Parameter Estimation
An important part of system identification is being able to
estimate the parameters of a linear model, when a
quadratic performance function is used to measure the
model’s goodness.
This produces the well-known normal equations for least
squares estimation
• This is a closed form solution
• Efficiently and robustly solved (in Matlab)
• Permits a statistical interpretation
• Can be solved recursively
Investigated over the next 3-4 lectures
( ) y
X
X
X
θ T
T 1
ˆ
21. EE-M110 2006/7, EF L5&6 21/29, v2.0
Noise-free Parameter Determination
Parameter estimation works by assuming a plant/model
structure, which is taken to be exactly known.
If there are n+m parameters in the model, we can collect
n+m pieces of data (linearly independent – to ensure
that the input/data matrix, X, is invertible):
Xq = y
and invert the matrix to find the exact parameter values:
q =X-1y
In Matlab, both of the following forms are equivalent:
theta = inv(X)*y;
theta = Xy;
theta = [0.5 0.5] % Previous example
22. EE-M110 2006/7, EF L5&6 22/29, v2.0
Linear Model and Quadratic Performance
When the model is linear and the data is noisy (missing inputs,
unmeasurable disturbances), the Sum Squared Error (SSE)
performance index can be expressed as:
This expression is quadratic in q. Typically
size(X,1)>>size(X,2)
It is of the form (for 2 inputs/parameters):
The equivalent system of linear equations Xq=y+e is inconsistent
( )
( )
T
i
T
i
T
i
T
i
i
T
i
i
T
i
T
i
i
T
i
i
i
y
y
y
y
y
f
1
2
1
1
2
1
2
1
2
^
2
)
(
θ
x
θ
x
θ
x
θ
2
1
2
2
2
1
2
1 5
.
0
6
8
3
2
5
)
( q
q
q
q
q
q
θ
f
23. EE-M110 2006/7, EF L5&6 23/29, v2.0
Quadratic Matrix Representation
This can also be expressed in matrix form
The general form for a quadratic is:
where
( ) ( )
Xθ
X
θ
y
X
θ
y
y
Xθ
X
θ
y
X
θ
Xθ
y
y
y
Xθ
y
Xθ
y
θ
T
T
T
T
T
T
T
T
T
T
T
T
f
2
)
(
c
f T
T
θ
f
Hθ
θ
θ 2
1
)
(
)
(
.
.
j
i
k j
k
i
k
ij
T
x
x
E
x
x
H
D
X
X
H
)
(
.
y
x
E
y
x
f
i
k k
i
k
i
T
D
y
X
f
Hessian/covariance matrix Cross-correlation vector
24. EE-M110 2006/7, EF L5&6 24/29, v2.0
When the parameter vector is optimal:
For a quadratic MSE with a linear model:
At optimality:
In Matlab, the normal equations are:
thetaHat = inv(X’*X)*X’*y;
thetaHat = pinv(X)*y;
thetaHat = Xy;
Normal Equations for a Linear Model
0
θ
f
( )
y
X
Xθ
X
y
y
y
X
θ
Xθ
X
θ
θ
θ
T
T
T
T
T
T
T
f
2
2
2
f
^
q
q
( ) y
X
X
X
θ
0
y
X
θ
X
X
T
T
T
T
1
ˆ
ˆ
26. EE-M110 2006/7, EF L5&6 26/29, v2.0
Example 2: Electrical Circuit ARX Model
9 exemplars and 2 parameters.
Additive measurement noise
T
]
511
.
0
467
.
0
[
ˆ
θ
Inverse Hessian matrix
89
.
6
57
.
5
y
XT
( )
799
.
0
897
.
0
897
.
0
194
.
1
1
1
X
X
H T
Least squares solution
1
992
.
0
1
5
.
0
1
0
0
0
X
007
.
1
774
.
0
510
.
0
037
.
0
y
Hessian (variance/covariance)
matrix and correlation vector
8
0078
.
6
0078
.
6
3489
.
5
X
X
H T
NB
randn(‘state’, 123456)
See slides 7, 12, 18 & 19
27. EE-M110 2006/7, EF L5&6 27/29, v2.0
We can “plot” the performance index against different
parameter values in a model
As shown earlier, f() is a quadratic function in q
It is “centred” at q, I.e. f(q) = min f(q)
The shape (contours) depends on the Hessian matrix X, this
influences the ability to identify the plant. See next lectures
q1
q2
f
Investigation into the Performance Function
^ ^
28. EE-M110 2006/7, EF L5&6 28/29, v2.0
L5&6 Summary
ARX and ARMAX discrete time linear models are widely used
System identification is being considered simply as parameter estimation
The residual vector is used to assess the quality of the model (parameter
vector)
The sum, squared error/residual (2-norm) is commonly used to measure
the residual’s size because it can be interpreted as the noise variance
and because it is analytically convenient
For a linear model, the SSE is a quadratic function of the parameters,
which can be differentiated to estimate the optimal parameter via the
normal equations
29. EE-M110 2006/7, EF L5&6 29/29, v2.0
L5&6 Lab
Theory
• Make sure you can derive the normal equations S22-24
Matlab
1. Implement the DT RC circuit simulation, S18, so you can perform a least
squares parameter estimation given noisy data about the electrical circuit
2. Set the Gaussian random seed, as per S26 and check your estimates
are the same
3. Set different seed and note that the optimal parameter values are
different
4. Perform the step experiment 10, 100, 1000, … times and note that the
estimated optimal parameter values tend towards the true values of [0.5
0.5].
5. Load the data into the identification toolbox GUI and create a first order
parametric model with model orders [1 1 1]. NB you do not need to
remove the means from the data (why not?). Calculate the model and
view the value of the parameters and the model fit, as well as checking
the step response and validating the model.