Download to read offline
![1.1 Introduction 3
Fig.1.1 Contour plot of
f (x1, x2)
A
10
f (x , x ) = 0
1 2
f (x , x ) = 50
1 2
1
x
2
x
20
30
40
50
each other, and must be adjusted simultaneously to yield the optimum performance
criterion.
The most important general approach to optimization is based on numerical meth-
ods. In this approach, iterative numerical procedures are used to generate a series
of progressively improved solutions to the optimization problem, starting with an
initial estimate for the solution. The process is terminated when some convergence
criterion is satisfied. For example, when changes in the independent variables or the
performance criterion from iteration to iteration become insignificant.
Numerical methods can be used to solve highly complex optimization problems
of the type that cannot be solved analytically. Furthermore, they can be readily
programmed on the digital computer. Consequently, they have all but replaced most
other approaches to optimization.
The discipline encompassing the theory and practice of numerical optimization
methods has come to be known as mathematical programming [1–5]. During the past
40 years, several branches of mathematical programming have evolved, as follows:
1. Linear programming
2. Integer programming
3. Quadratic programming
4. Nonlinear programming
5. Dynamic programming
Each one of these branches of mathematical programming is concerned with a spe-
cific class of optimization problems. The differences among them will be examined
in Sect.1.6.](https://image.slidesharecdn.com/20986-250315065426-4aaea085/85/Practical-Optimization-Algorithms-and-Engineering-Applications-2nd-Edition-Andreas-Antoniou-30-320.jpg)
![4 1 The Optimization Problem
1.2 The Basic Optimization Problem
Before optimization is attempted, the problem at hand must be properly formulated.
A performance criterion F must be derived in terms of n parameters x1, x2, . . . , xn
as
F = f (x1, x2, . . . , xn)
F is a scalar quantity which can assume numerous forms. It can be the cost of a
product in a manufacturing environment or the difference between the desired per-
formance and the actual performance in a system. Variables x1, x2, . . . , xn are the
parameters that influence the product cost in the first case or the actual performance in
the second case. They can be independent variables, like time, or control parameters
that can be adjusted.
The most basic optimization problem is to adjust variables x1, x2, . . . , xn in
such a way as to minimize quantity F. This problem can be stated mathematically
as
minimize F = f (x1, x2, . . . , xn) (1.1)
Quantity F is usually referred to as the objective or cost function.
The objective function may depend on a large number of variables, sometimes as
many as 100 or more. To simplify the notation, matrix notation is usually employed.
If x is a column vector with elements x1, x2, . . . , xn, the transpose of x, namely,
xT , can be expressed as the row vector
xT
= [x1 x2 · · · xn]
In this notation, the basic optimization problem of Eq. (1.1) can be expressed as
minimize F = f (x) for x ∈ En
where En represents the n-dimensional Euclidean space.
On many occasions, the optimization problem consists of finding the maximum
of the objective function. Since
max[ f (x)] = −min[− f (x)]
the maximum of F can be readily obtained by finding the minimum of the negative of
F and then changing the sign of the minimum. Consequently, in this and subsequent
chapters we focus our attention on minimization without loss of generality.
In many applications, a number of distinct functions of x need to be optimized
simultaneously. For example, if the system of nonlinear simultaneous equations
fi (x) = 0 for i = 1, 2, . . . , m
needs to be solved, a vector x is sought which will reduce all fi (x) to zero simulta-
neously. In such a problem, the functions to be optimized can be used to construct a
vector
F(x) = [ f1(x) f2(x) · · · fm(x)]T
The problem can be solved by finding a point x = x∗ such that F(x∗) = 0. Very
frequently, a point x∗ that reduces all the fi (x) to zero simultaneously may not](https://image.slidesharecdn.com/20986-250315065426-4aaea085/85/Practical-Optimization-Algorithms-and-Engineering-Applications-2nd-Edition-Andreas-Antoniou-31-320.jpg)
![1.2 The Basic Optimization Problem 5
exist but an approximate solution, i.e., F(x∗) ≈ 0, may be available which could be
entirely satisfactory in practice.
A similar problem arises in scientific or engineering applications when the func-
tion of x that needs to be optimized is also a function of a continuous independent
parameter (e.g., time, position, speed, frequency) that can assume an infinite set
of values in a specified range. The optimization might entail adjusting variables
x1, x2, . . . , xn so as to optimize the function of interest over a given range of the
independent parameter. In such an application, the function of interest can be sampled
with respect to the independent parameter, and a vector of the form
F(x) = [ f (x, t1) f (x, t2) · · · f (x, tm)]T
can be constructed, where t is the independent parameter. Now if we let
fi (x) ≡ f (x, ti )
we can write
F(x) = [ f1(x) f2(x) · · · fm(x)]T
A solution of such a problem can be obtained by optimizing functions fi (x) for
i = 1, 2, . . . , m simultaneously. Such a solution would, of course, be approximate
because any variations in f (x, t) between sample points are ignored. Nevertheless,
reasonable solutions can be obtained in practice by using a sufficiently large number
of sample points. This approach is illustrated by the following example.
Example 1.1 The step response y(x, t) of an nth-order control system is required
to satisfy the specification
y0(x, t) =
⎧
⎪
⎪
⎨
⎪
⎪
⎩
t for 0 ≤ t < 2
2 for 2 ≤ t < 3
−t + 5 for 3 ≤ t < 4
1 for 4 ≤ t
as closely as possible. Construct a vector F(x) that can be used to obtain a function
f (x, t) such that
y(x, t) ≈ y0(x, t) for 0 ≤ t ≤ 5
Solution The difference between the actual and specified step responses, which
constitutes the approximation error, can be expressed as
f (x, t) = y(x, t) − y0(x, t)
and if f (x, t) is sampled at t = 0, 1, . . . , 5, we obtain
F(x) = [ f1(x) f2(x) · · · f6(x)]T](https://image.slidesharecdn.com/20986-250315065426-4aaea085/85/Practical-Optimization-Algorithms-and-Engineering-Applications-2nd-Edition-Andreas-Antoniou-32-320.jpg)
![1.2 The Basic Optimization Problem 7
Several special cases of the L p norm are of particular interest. If p = 1
F ≡ L1 =
m
i=1
| fi (x)|
and, therefore, in a minimization problem like that in Example1.1, the sum of the
magnitudes of the individual element functions is minimized. This is called an L1
problem.
If p = 2, the Euclidean norm
F ≡ L2 =
m
i=1
| fi (x)|2
1/2
is minimized, and if the square root is omitted, the sum of the squares is minimized.
Such a problem is commonly referred to as a least-squares problem.
In the case where p = ∞, if we assume that there is a unique maximum of | fi (x)|
designated F̂ such that
F̂ = max
1≤i≤m
| fi (x)|
then we can write
F ≡ L∞ = lim
p→∞
m
i=1
| fi (x)|p
1/p
= F̂ lim
p→∞
m
i=1
| fi (x)|
F̂
p
1/p
Since all the terms in the summation except one are less than unity, they tend to zero
when raised to a large positive power. Therefore, we obtain
F = F̂ = max
1≤i≤m
| fi (x)|
Evidently, if the L∞ norm is used in Example1.1, the maximum approximation error
is minimized and the problem is said to be a minimax problem.
Often the individual element functions of F(x) are modified by using constants
w1, w2, . . . , wm as weights. For example, the least-squares objective function can
be expressed as
F =
m
i=1
[wi fi (x)]2
so as to emphasize important or critical element functions and de-emphasize unim-
portant or uncritical ones. If F is minimized, the residual errors in wi fi (x) at the end
of the minimization would tend to be of the same order of magnitude, i.e.,
error in |wi fi (x)| ≈ ε
and so
error in | fi (x)| ≈
ε
|wi |
Consequently, if a large positive weight wi is used with fi (x), a small residual error
is achieved in | fi (x)|.](https://image.slidesharecdn.com/20986-250315065426-4aaea085/85/Practical-Optimization-Algorithms-and-Engineering-Applications-2nd-Edition-Andreas-Antoniou-34-320.jpg)
![8 1 The Optimization Problem
1.3 General Structure of Optimization Algorithms
Most of the available optimization algorithms entail a series of steps which are
executed sequentially. A typical pattern is as follows:
Algorithm 1.1 General optimization algorithm
Step 1
(a) Set k = 0 and initialize x0.
(b) Compute F0 = f (x0).
Step 2
(a) Set k = k + 1.
(b) Compute the changes in xk given by column vector Δxk where
ΔxT
k = [Δx1 Δx2 · · · Δxn]
by using an appropriate procedure.
(c) Set xk = xk−1 + Δxk
(d) Compute Fk = f (xk) and ΔFk = Fk−1 − Fk.
Step 3
Check if convergence has been achieved by using an appropriate criterion, e.g.,
by checking ΔFk and/or Δxk. If this is the case, continue to Step 4; otherwise,
go to Step 2.
Step 4
(a) Output x∗ = xk and F∗ = f (x∗).
(b) Stop.
In Step 1, vector x0 is initialized by estimating the solution using knowledge about
the problem at hand. Often the solution cannot be estimated and an arbitrary solution
may be assumed, say, x0 = 0. Steps 2 and 3 are then executed repeatedly until
convergence is achieved. Each execution of Steps 2 and 3 constitutes one iteration,
that is, k is the number of iterations.
When convergence is achieved, Step 4 is executed. In this step, column vector
x∗
= [x∗
1 x∗
2 · · · x∗
n ]T
= xk
and the corresponding value of F, namely,
F∗
= f (x∗
)
are output. The column vector x∗ is said to be the optimum, minimum, solution point,
or simply the minimizer, and F∗ is said to be the optimum or minimum value of the
objective function. The pair x∗ and F∗ constitute the solution of the optimization
problem.
Convergence can be checked in several ways, depending on the optimization
problem and the optimization technique used. For example, one might decide to
stop the algorithm when the reduction in Fk between any two iterations has become
insignificant, that is,
|ΔFk| = |Fk−1 − Fk| εF (1.2)](https://image.slidesharecdn.com/20986-250315065426-4aaea085/85/Practical-Optimization-Algorithms-and-Engineering-Applications-2nd-Edition-Andreas-Antoniou-35-320.jpg)
![10 1 The Optimization Problem
Fig.1.3 The double inverted
pendulum
θ 1
θ 2
Μ
u(t)
A problem that does not entail any equality or inequality constraints is said to be an
unconstrained optimization problem.
Constrained optimization is usually much more difficult than unconstrained opti-
mization, as might be expected. Consequently, the general strategy that has evolved
in recent years towards the solution of constrained optimization problems is to re-
formulate constrained problems as unconstrained optimization problems. This can
be done by redefining the objective function such that the constraints are simultane-
ously satisfied when the objective function is minimized. Some real-life constrained
optimization problems are given as Examples1.2 to 1.4 below.
Example 1.2 Consider a control system that comprises a double inverted pendulum
as depicted in Fig. 1.3. The objective of the system is to maintain the pendulum in the
upright position using the minimum amount of energy. This is achieved by applying
an appropriate control force to the car to damp out any displacements θ1(t) and θ2(t).
Formulate the problem as an optimization problem.
Solution The dynamic equations of the system are nonlinear and the standard practice
is to apply a linearization technique to these equations to obtain a small-signal linear
model of the system as [6]
ẋ(t) = Ax(t) + fu(t) (1.5)
where
x(t) =
⎡
⎢
⎢
⎣
θ1(t)
θ̇1(t)
θ2(t)
θ̇2(t)
⎤
⎥
⎥
⎦ , A =
⎡
⎢
⎢
⎣
0 1 0 0
α 0 −β 0
0 0 0 1
−α 0 α 0
⎤
⎥
⎥
⎦ , f =
⎡
⎢
⎢
⎣
0
−1
0
0
⎤
⎥
⎥
⎦
with α 0, β 0, and α = β. In the above equations, ẋ(t), θ̇1(t), and θ̇2(t)
represent the first derivatives of x(t), θ1(t), and θ2(t), respectively, with respect
to time, θ̈1(t) and θ̈2(t) would be the second derivatives of θ1(t) and θ2(t), and
parameters α and β depend on system parameters such as the length and weight of
each pendulum, the mass of the car, etc. Suppose that at instant t = 0 small nonzero](https://image.slidesharecdn.com/20986-250315065426-4aaea085/85/Practical-Optimization-Algorithms-and-Engineering-Applications-2nd-Edition-Andreas-Antoniou-37-320.jpg)
![1.4 Constraints 11
displacements θ1(t) and θ2(t) occur, which would call for immediate control action
in order to steer the system back to the equilibrium state x(t) = 0 at time t = T0. In
order to develop a digital controller, the system model in Eq. (1.5) is discretized to
become
x(k + 1) = x(k) + gu(k) (1.6)
where = I + ΔtA, g = Δtf, Δt is the sampling interval, and I is the identity
matrix. Let x(0) = 0 be given and assume that T0 is a multiple of Δt, i.e., T0 =
KΔt where K is an integer. We seek to find a sequence of control actions u(k) for
k = 0, 1, . . . , K − 1 such that the zero equilibrium state is achieved at t = T0, i.e.,
x(T0) = 0.
Let us assume that the energy consumed by these control actions, namely,
J =
K−1
k=0
u2
(k)
needs to be minimized. This optimal control problem can be formulated analytically
as
minimize J =
K−1
k=0
u2
(k) (1.7a)
subject to: x(K) = 0 (1.7b)
From Eq. (1.6), we know that the state of the system at t = KΔt is determined
by the initial value of the state and system model in Eq. (1.6) as
x(K) = K
x(0) +
K−1
k=0
K−k−1
gu(k)
≡ −h +
K−1
k=0
gku(k)
where h = −K x(0) and gk = K−k−1g. Hence the constraint in Eq. (1.7b) is
equivalent to
K−1
k=0
gku(k) = h (1.8)
If we define u = [u(0) u(1) · · · u(K − 1)]T and G = [g0 g1 · · · gK−1], then the
constraint in Eq. (1.8) can be expressed as Gu = h, and the optimal control problem
at hand can be formulated as the problem of finding a u that solves the minimization
problem](https://image.slidesharecdn.com/20986-250315065426-4aaea085/85/Practical-Optimization-Algorithms-and-Engineering-Applications-2nd-Edition-Andreas-Antoniou-38-320.jpg)
![12 1 The Optimization Problem
minimize uT
u (1.9a)
subject to: a(u) = 0 (1.9b)
where a(u) = Gu − h. In practice, the control actions cannot be made arbitrarily
large in magnitude. Consequently, additional constraints are often imposed on |u(i)|,
for instance,
|u(i)| ≤ m for i = 0, 1, . . . , K − 1
These constraints are equivalent to
m + u(i) ≥ 0
m − u(i) ≥ 0
Hence if we define
c(u) =
⎡
⎢
⎢
⎢
⎢
⎢
⎣
m + u(0)
m − u(0)
.
.
.
m + u(K − 1)
m − u(K − 1)
⎤
⎥
⎥
⎥
⎥
⎥
⎦
then the magnitude constraints can be expressed as
c(u) ≥ 0 (1.9c)
Obviously, the problem in Eqs. (1.9a)–(1.9c) fits nicely into the standard form of
optimization problems given by Eqs. (1.4a)–(1.4c).
Example 1.3 High performance in modern optical instruments depends on the qual-
ity of components like lenses, prisms, and mirrors. These components have reflecting
or partially reflecting surfaces, and their performance is limited by the reflectivities
of the materials of which they are made. The surface reflectivity can, however, be
altered by the deposition of a thin transparent film. In fact, this technique facilitates
the control of losses due to reflection in lenses and makes possible the construction
of mirrors with unique properties [7,8].
As is depicted in Fig. 1.4, a typical N-layer thin-film system consists of N layers of
thin films of certain transparent media deposited on a glass substrate. The thickness
and refractive index of the ith layer are denoted as xi and ni , respectively. The
refractive index of the medium above the first layer is denoted as n0. If φ0 is the
angle of incident light, then the transmitted ray in the (i − 1)th layer is refracted at
an angle φi which is given by Snell’s law, namely,
ni sin φi = n0 sin φ0
Given angle φ0 and the wavelength of light, λ, the energy of the light reflected
from the film surface and the energy of the light transmitted through the film surface
are usually measured by the reflectance R and transmittance T , which satisfy the
relation
R + T = 1](https://image.slidesharecdn.com/20986-250315065426-4aaea085/85/Practical-Optimization-Algorithms-and-Engineering-Applications-2nd-Edition-Andreas-Antoniou-39-320.jpg)
![1.4 Constraints 13
Fig.1.4 An N-layer
thin-film system
n0
n3
n2
nN
nN+1
layer 1
layer 2
layer 3
x2
x1 n1
Substrate
φ1
φ0
φ2
layer N xn
φN
For an N-layer system, R is given by (see [9] for details)
R(x1, . . . , xN , λ) =
η0 − y
η0 + y
2
y =
c
b
b
c
=
N
k=1
cos δk ( j sin δk)/ηk
jηk sin δk cos δk
1
ηN+1
where j =
√
−1 and
δk =
2πnk xk cos φk
λ
ηk =
⎧
⎪
⎪
⎪
⎪
⎪
⎪
⎨
⎪
⎪
⎪
⎪
⎪
⎪
⎩
nk/ cos φk for light polarized with the electric
vector lying in the plane of incidence
nk cos φk for light polarized with the electric
vector perpendicular to the
plane of incidence
The design of a multilayer thin-film system can now be accomplished as follows:
Given a range of wavelengths λl ≤ λ ≤ λu and an angle of incidence φ0, find
x1, x2, . . . , xN such that the reflectance R(x, λ) best approximates a desired re-
flectance Rd(λ) for λ ∈ [λl, λu]. Formulate the design problem as an optimization
problem.](https://image.slidesharecdn.com/20986-250315065426-4aaea085/85/Practical-Optimization-Algorithms-and-Engineering-Applications-2nd-Edition-Andreas-Antoniou-40-320.jpg)
![14 1 The Optimization Problem
Solution In practice, the desired reflectance is specified at grid points λ1, λ2, . . . , λK
in the interval [λl, λu]; hence the design may be carried out by selecting xi such
that the objective function
J =
K
i=1
wi [R(x, λi ) − Rd(λi )]2
(1.10)
is minimized, where
x = [x1 x2 · · · xN ]T
and wi 0 is a weight to reflect the importance of term [R(x, λi ) − Rd(λi )]2 in
Eq. (1.10). If we let η = [1 ηN+1]T , e+ = [η0 1]T , e− = [η0 −1]T , and
M(x, λ) =
N
k=1
cos δk ( j sin δk)/ηk
jηk sin δk cos δk
then R(x, λ) can be expressed as
R(x, λ) =
bη0 − c
bη0 + c
2
=
eT
−M(x, λ)η
eT
+M(x, λ)η
2
Finally, we note that the thickness of each layer cannot be made arbitrarily thin or
arbitrarily large and, therefore, constraints must be imposed on the elements of x as
dil ≤ xi ≤ diu for i = 1, 2, . . . , N
The design problem can now be formulated as the constrained minimization problem
minimize J =
K
i=1
wi
eT
−M(x, λi )η
eT
+M(x, λi )η
2
− Rd(λi )2
subject to: xi − dil ≥ 0 for i = 1, 2, . . . , N
diu − xi ≥ 0 for i = 1, 2, . . . , N
Example 1.4 Quantities q1, q2, . . . , qm of a certain product are produced by m
manufacturing divisions of a company, which are at distinct locations. The product
is to be shipped to n destinations that require quantities b1, b2, . . . , bn. Assume
that the cost of shipping a unit from manufacturing division i to destination j is ci j
with i = 1, 2, . . . , m and j = 1, 2, . . . , n. Find the quantity xi j to be shipped
from division i to destination j so as to minimize the total cost of transportation, i.e.,
minimize C =
m
i=1
n
j=1
ci j xi j
This is known as the transportation problem. Formulate the problem as an optimiza-
tion problem.](https://image.slidesharecdn.com/20986-250315065426-4aaea085/85/Practical-Optimization-Algorithms-and-Engineering-Applications-2nd-Edition-Andreas-Antoniou-41-320.jpg)
![1.4 Constraints 15
Solution Note that there are several constraints on variables xi j . First, each division
can provide only a fixed quantity of the product, hence
n
j=1
xi j = qi for i = 1, 2, . . . , m
Second, the quantity to be shipped to a specific destination has to meet the need of
that destination and so
m
i=1
xi j = bj for j = 1, 2, . . . , n
In addition, the variables xi j are nonnegative and thus, we have
xi j ≥ 0 for i = 1, 2, . . . , m and j = 1, 2, . . . , n
If we let
c = [c11 · · · c1n c21 · · · c2n · · · cm1 · · · cmn]T
x = [x11 · · · x1n x21 · · · x2n · · · xm1 · · · xmn]T
A =
⎡
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎣
1 1 · · · 1 0 0 · · · 0 · · · · · · · · · · · ·
0 0 · · · 0 1 1 · · · 1 · · · · · · · · · · · ·
· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
0 0 · · · 0 0 0 · · · 0 · · · 1 1 · · · 1
1 0 · · · 0 1 0 · · · 0 · · · 1 0 · · · 0
0 1 · · · 0 0 1 · · · 0 · · · 0 1 · · · 0
· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
0 0 · · · 1 0 0 · · · 1 · · · 0 0 · · · 1
⎤
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎦
b = [q1 · · · qm b1 · · · bn]T
then the minimization problem can be stated as
minimize C = cT
x (1.11a)
subject to: Ax = b (1.11b)
x ≥ 0 (1.11c)
where cT x is the inner product of c and x. The above problem, like those in Exam-
ples1.2 and 1.3, fits into the standard optimization problem in Eqs. (1.4a)–(1.4c).
Since both the objective function in Eq. (1.11a) and the constraints in Eqs. (1.11b)
and (1.11c) are linear, the problem is known as a linear programming (LP) problem
(see Sect.1.6.1).
Example 1.5 An investment portfolio is a collection of a variety of securities owned
by an individual whereby each security is associated with a unique possible return and
a corresponding risk. Design an optimal investment portfolio that would minimize
the risk involved subject to an acceptable return for a portfolio that comprises n
securities.](https://image.slidesharecdn.com/20986-250315065426-4aaea085/85/Practical-Optimization-Algorithms-and-Engineering-Applications-2nd-Edition-Andreas-Antoniou-42-320.jpg)
![16 1 The Optimization Problem
Solution Some background theory on portfolio selection can be found in [10]. As-
sume that the amount of resources to be invested is normalized to unity (e.g., 1
million dollars) and let xi represent the return of security i at some specified time in
the future, say, in one month’s time. The return xi can be assumed to be a random
variable and hence three quantities pertaining to security i can be evaluated, namely,
the expected return
μi = E[xi ]
the variance of the return
σ2
i = E[(xi − μi )2
]
and the correlation between the returns of the ith and jth securities
ρi, j =
E[(xi − μi )(x j − μj )]
σi σj
for i, j = 1, 2, . . . , n
With these quantities known, constructing a portfolio amounts to allocating a fraction
wi of the available resources to security i, for i = 1, 2, . . . , n. This leads to the
constraints
0 ≤ wi ≤ 1 for i = 1, 2, . . . , n and
n
i=1
wi = 1
Given a set of investment allocations {wi , i = 1, 2, . . . , n}, the expected return of
the portfolio can be deduced as
E
n
i=1
wi xi
=
n
i=1
wi μi
The variance for the portfolio, which measures the risk of the investment, can be
evaluated as
E
n
i=1
wi xi − E(
n
i=1
wi xi )
2
= E
⎧
⎨
⎩
n
i=1
wi (xi − μi )
⎡
⎣
n
j=1
wj (x j − μj )
⎤
⎦
⎫
⎬
⎭
= E
⎡
⎣
n
i=1
n
j=1
(xi − μi )(x j − μj )wi wj
⎤
⎦
=
n
i=1
n
j=1
E[(xi − μi )(x j − μj )]wi wj
=
n
i=1
n
j=1
(σi σj ρi j )wi wj
=
n
i=1
n
j=1
qi j wi wj
where
qi j = σi σj ρi j](https://image.slidesharecdn.com/20986-250315065426-4aaea085/85/Practical-Optimization-Algorithms-and-Engineering-Applications-2nd-Edition-Andreas-Antoniou-43-320.jpg)
Practical Optimization: Algorithms and Engineering Applications 2nd Edition Andreas Antoniou
- 1. Read Anytime AnywhereEasy Ebook Downloads at ebookmeta.com Practical Optimization: Algorithms and Engineering Applications 2nd Edition Andreas Antoniou https://ebookmeta.com/product/practical-optimization- algorithms-and-engineering-applications-2nd-edition-andreas- antoniou/ OR CLICK HERE DOWLOAD EBOOK Visit and Get More Ebook Downloads Instantly at https://ebookmeta.com
- 2. Recommended digital products(PDF, EPUB, MOBI) that you can download immediately if you are interested. Practical Optimization Algorithms and Engineering Applications 2nd Andreas Antoniou Wu Sheng Lu https://ebookmeta.com/product/practical-optimization-algorithms-and- engineering-applications-2nd-andreas-antoniou-wu-sheng-lu/ ebookmeta.com Optimization for Chemical and Biochemical Engineering: Theory, Algorithms, Modeling and Applications 1st Edition Vassilios S. Vassiliadis https://ebookmeta.com/product/optimization-for-chemical-and- biochemical-engineering-theory-algorithms-modeling-and- applications-1st-edition-vassilios-s-vassiliadis/ ebookmeta.com Lectures on Modern Convex Optimization 2020-2023: Analysis, Algorithms, Engineering Applications Aharon Ben- Tal https://ebookmeta.com/product/lectures-on-modern-convex- optimization-2020-2023-analysis-algorithms-engineering-applications- aharon-ben-tal/ ebookmeta.com Book of Love 1st Edition Aakash Shukla https://ebookmeta.com/product/book-of-love-1st-edition-aakash-shukla/ ebookmeta.com
- 3. Energy Efficiency inDeveloping Countries 1st Edition Suzana Tavares Da Silva Gabriela Prata Dias Eds https://ebookmeta.com/product/energy-efficiency-in-developing- countries-1st-edition-suzana-tavares-da-silva-gabriela-prata-dias-eds/ ebookmeta.com Romancing The President Girls On Top 1st Edition M K Moore https://ebookmeta.com/product/romancing-the-president-girls-on- top-1st-edition-m-k-moore/ ebookmeta.com Understanding the Many Faces of Human Security Perspectives of Northern Indigenous Peoples 1st Edition Kamrul Hossain https://ebookmeta.com/product/understanding-the-many-faces-of-human- security-perspectives-of-northern-indigenous-peoples-1st-edition- kamrul-hossain/ ebookmeta.com The Rom-Com Agenda Jayne Denker https://ebookmeta.com/product/the-rom-com-agenda-jayne-denker/ ebookmeta.com Group Survival in the Ancient Mediterranean Rethinking Material Conditions in the Landscape of Jews and Christians 2021st Edition Richard Last https://ebookmeta.com/product/group-survival-in-the-ancient- mediterranean-rethinking-material-conditions-in-the-landscape-of-jews- and-christians-2021st-edition-richard-last/ ebookmeta.com
- 4.
- 5. Texts in ComputerScience Practical Optimization Andreas Antoniou Wu-Sheng Lu Algorithms and Engineering Applications SecondEdition
- 6. Texts in ComputerScience Series Editors David Gries, Department of Computer Science, Cornell University, Ithaca, NY, USA Orit Hazzan , Faculty of Education in Technology and Science, Technion—Israel Institute of Technology, Haifa, Israel
- 7. More information aboutthis series at http://www.springer.com/series/3191
- 8. Andreas Antoniou •Wu-Sheng Lu Practical Optimization Algorithms and Engineering Applications Second Edition 123
- 9. Andreas Antoniou Department ofElectrical and Computer Engineering University of Victoria Victoria, BC, Canada Wu-Sheng Lu Department of Electrical and Computer Engineering University of Victoria Victoria, BC, Canada ISSN 1868-0941 ISSN 1868-095X (electronic) Texts in Computer Science ISBN 978-1-0716-0841-8 ISBN 978-1-0716-0843-2 (eBook) https://doi.org/10.1007/978-1-0716-0843-2 1st edition: © 2007 Springer Secience+Business Media, LLC 2nd edition: © Springer Science+Business Media, LLC, part of Springer Nature 2021 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Science+Business Media, LLC part of Springer Nature. The registered company address is: 1 New York Plaza, New York, NY 10004, U.S.A.
- 10.
- 11. Preface to theSecond Edition Optimization methods and algorithms continue to evolve at a tremendous rate and are providing solutions to many problems that could not be solved before in economics, finance, geophysics, molecular modeling, computational systems biology, operations research, and all branches of engineering (see the following link for details: https://en.wikipedia.org/wiki/Mathematical_optimization#Molecular_ modeling). The growing demand for optimization methods and algorithms has been addressed in the second edition by updating some material, adding more examples, and introducing some recent innovations, techniques, and methodologies. The emphasis continues to be on practical methods and efficient algorithms that work. Chapters 1–8 continue to deal with the basics of optimization. Chapter 5 now includes two increasingly popular line search techniques, namely, the so-called two-point and backtracking line searches. In Chap. 6, a new section has been added that deals with the application of the conjugate-gradient method for the solution of linear systems of equations. In Chap. 9, some state-of-the art applications of unconstrained optimization to machine learning and source localization are added. The first application is in the area of character recognition and it is a method for classifying handwritten digits using a regression technique known as softmax. The method is based on an accelerated gradient descent algorithm. The second application is in the area of communications and it deals of the problem formulation and solution methods for identifying the location of a radiating source given the distances between the source and several sensors. The contents of Chaps. 10–12 are largely unchanged except for some editorial changes whereas Chap. 13 combines the material found in Chaps. 13 and 14 of the first edition. Chapter 14 is a new chapter that presents additional concepts and properties of convex functions that are not covered in Chapter 2. It also describes several algorithms for the solution of general convex problems and includes a detailed exposition of the so-called alternating direction method of multipliers (ADMM). Chapter 15 is a new chapter that focuses on sequential convex programming, sequential quadratic programming, and convex-concave procedures for general vii
- 12. nonconvex problems. Italso includes a section on heuristic ADMM techniques for nonconvex problems. In Chap. 16, we have added some new state-of-the art applications of con- strained optimization for the design of Finite-Duration Impulse Response (FIR) and Infinite-Duration Impulse Response (IIR) digital filters, also known as nonrecursive and recursive filters, respectively, using second-order cone programming. Digital filters that would satisfy multiple specifications such as maximum passband gain, minimum stopband gain, maximum transition-band gain, and maximum pole radius, can be designed with these methods. The contents of Appendices A and B are largely unchanged except for some editorial changes. Many of our past students at the University of Victoria have helped a great deal in improving the first edition and some of them, namely, Drs. M. L. R. de Campos, Sunder Kidambi, Rajeev C. Nongpiur, Ana Maria Sevcenco, and Ioana Sevcenco have provided meaningful help in the evolution of the second edition as well. We would also like to thank Drs. Z. Dong, T. Hinamoto, Y. Q. Hu, and W. Xu for useful discussions on optimization theory and its applications, Catherine Chang for typesetting the first draft of the second edition, and to Lynne Barrett for checking the entire second edition for typographical errors. Victoria, Canada Andreas Antoniou Wu-Sheng Lu viii Preface to the Second Edition
- 13. Preface to theFirst Edition The rapid advancements in the efficiency of digital computers and the evolution of reliable software for numerical computation during the past three decades have led to an astonishing growth in the theory, methods, and algorithms of numerical optimization. This body of knowledge has, in turn, motivated widespread appli- cations of optimization methods in many disciplines, e.g., engineering, business, and science, and led to problem solutions that were considered intractable not too long ago. Although excellent books are available that treat the subject of optimization with great mathematical rigor and precision, there appears to be a need for a book that provides a practical treatment of the subject aimed at a broader audience ranging from college students to scientists and industry professionals. This book has been written to address this need. It treats unconstrained and constrained optimization in a unified manner and places special attention on the algorithmic aspects of opti- mization to enable readers to apply the various algorithms and methods to specific problems of interest. To facilitate this process, the book provides many solved examples that illustrate the principles involved, and includes, in addition, two chapters that deal exclusively with applications of unconstrained and constrained optimization methods to problems in the areas of pattern recognition, control sys- tems, robotics, communication systems, and the design of digital filters. For each application, enough background information is provided to promote the under- standing of the optimization algorithms used to obtain the desired solutions. Chapter 1 gives a brief introduction to optimization and the general structure of optimization algorithms. Chapters 2 to 9 are concerned with unconstrained opti- mization methods. The basic principles of interest are introduced in Chapter 2. These include the first-order and second-order necessary conditions for a point to be a local minimizer, the second-order sufficient conditions, and the optimization of convex functions. Chapter 3 deals with general properties of algorithms such as the concepts of descent function, global convergence, and rate of convergence. Chapter 4 presents several methods for one-dimensional optimization, which are commonly referred to as line searches. The chapter also deals with inexact line-search methods that have been found to increase the efficiency in many optimization algorithms. Chapter 5 presents several basic gradient methods that include the steepest-descent, Newton, and Gauss-Newton methods. Chapter 6 presents a class of methods based ix
- 14. on the conceptof conjugate directions such as the conjugate-gradient, Fletcher-Reeves, Powell, and Partan methods. An important class of unconstrained optimization methods known as quasi-Newton methods is presented in Chapter 7. Representative methods of this class such as the Davidon-Fletcher-Powell and Broydon-Fletcher-Goldfarb-Shanno methods and their properties are investigated. The chapter also includes a practical, efficient, and reliable quasi-Newton algorithm that eliminates some problems associated with the basic quasi-Newton method. Chapter 8 presents minimax methods that are used in many applications including the design of digital filters. Chapter 9 presents three case studies in which several of the unconstrained optimization methods described in Chapters 4 to 8 are applied to point pattern matching, inverse kinematics for robotic manipulators, and the design of digital filters. Chapters 10 to 16 are concerned with constrained optimization methods. Chapter 10 introduces the fundamentals of constrained optimization. The concept of Lagrange multipliers, the first-order necessary conditions known as Karush-Kuhn-Tucker conditions, and the duality principle of convex programming are addressed in detail and are illustrated by many examples. Chapters 11 and 12 are concerned with linear programming (LP) problems. The general properties of LP and the simplex method for standard LP problems are addressed in Chapter 11. Several interior-point methods including the primal affine-scaling, primal Newton-barrier, and primal-dual path-following methods are presented in Chapter 12. Chapter 13 deals with quadratic and general convex programming. The so-called active-set methods and several interior-point methods for convex quad- ratic programming are investigated. The chapter also includes the so-called cutting plane and ellipsoid algorithms for general convex programming problems. Chapter 14 presents two special classes of convex programming known as semidefinite and second-order cone programming, which have found interesting applications in a variety of disciplines. Chapter 15 treats general constrained optimization problems that do not belong to the class of convex programming; special emphasis is placed on several sequential quadratic programming methods that are enhanced through the use of efficient line searches and approximations of the Hessian matrix involved. Chapter 16, which concludes the book, examines several applications of con- strained optimization for the design of digital filters, for the control of dynamic systems, for evaluating the force distribution in robotic systems, and in multiuser detection for wireless communication systems. The book also includes two appendices, A and B, which provide additional support material. Appendix A deals in some detail with the relevant parts of linear algebra to consolidate the understanding of the underlying mathematical principles involved whereas Appendix B provides a concise treatment of the basics of digital filters to enhance the understanding of the design algorithms included in Chaps. 8, 9, and 16. The book can be used as a text for a sequence of two one-semester courses on optimization. The first course comprising Chaps. 1 to 7, 9, and part of Chap. 10 may be offered to senior undergraduate or first-year graduate students. The prerequisite knowledge is an undergraduate mathematics background of calculus and linear x Preface to the First Edition
- 15. algebra. The materialin Chaps. 8 and 10 to 16 may be used as a text for an advanced graduate course on minimax and constrained optimization. The prereq- uisite knowledge for this course is the contents of the first optimization course. The book is supported by online solutions of the end-of-chapter problems under password as well as by a collection of MATLAB programs for free access by the readers of the book, which can be used to solve a variety of optimization problems. These materials can be downloaded from book’s website: https://www.ece.uvic.ca/ optimization/. We are grateful to many of our past students at the University of Victoria, in particular, Drs. M. L. R. de Campos, S. Netto, S. Nokleby, D. Peters, and Mr. J. Wong who took our optimization courses and have helped improve the manuscript in one way or another; to Chi-Tang Catherine Chang for typesetting the first draft of the manuscript and for producing most of the illustrations; to R. Nongpiur for checking a large part of the index; and to P. Ramachandran for proofreading the entire manuscript. We would also like to thank Professors M. Ahmadi, C. Charalambous, P. S. R. Diniz, Z. Dong, T. Hinamoto, and P. P. Vaidyanathan for useful discussions on optimization theory and practice; Tony Antoniou of Psicraft Studios for designing the book cover; the Natural Sciences and Engineering Research Council of Canada for supporting the research that led to some of the new results described in Chapters 8, 9, and 16; and last but not least the University of Victoria for supporting the writing of this book over a number of years. Andreas Antoniou Wu-Sheng Lu Preface to the First Edition xi
- 16. Contents 1 The OptimizationProblem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 The Basic Optimization Problem . . . . . . . . . . . . . . . . . . . . . . 4 1.3 General Structure of Optimization Algorithms. . . . . . . . . . . . . 8 1.4 Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.5 The Feasible Region . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 1.6 Branches of Mathematical Programming. . . . . . . . . . . . . . . . . 20 1.6.1 Linear Programming . . . . . . . . . . . . . . . . . . . . . . . . 21 1.6.2 Integer Programming . . . . . . . . . . . . . . . . . . . . . . . 22 1.6.3 Quadratic Programming . . . . . . . . . . . . . . . . . . . . . 22 1.6.4 Nonlinear Programming . . . . . . . . . . . . . . . . . . . . . 23 1.6.5 Dynamic Programming . . . . . . . . . . . . . . . . . . . . . . 23 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2 Basic Principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 2.2 Gradient Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 2.3 The Taylor Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 2.4 Types of Extrema . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 2.5 Necessary and Sufficient Conditions For Local Minima and Maxima . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 2.5.1 First-Order Necessary Conditions. . . . . . . . . . . . . . . 34 2.5.2 Second-Order Necessary Conditions. . . . . . . . . . . . . 36 2.6 Classification of Stationary Points . . . . . . . . . . . . . . . . . . . . . 40 2.7 Convex and Concave Functions . . . . . . . . . . . . . . . . . . . . . . . 50 2.8 Optimization of Convex Functions . . . . . . . . . . . . . . . . . . . . . 57 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 3 General Properties of Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . 63 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 3.2 An Algorithm as a Point-to-Point Mapping . . . . . . . . . . . . . . . 63 xiii
- 17. 3.3 An Algorithmas a Point-to-Set Mapping . . . . . . . . . . . . . . . . 65 3.4 Closed Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 3.5 Descent Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 3.6 Global Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 3.7 Rates of Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 4 One-Dimensional Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 4.2 Dichotomous Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 4.3 Fibonacci Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 4.4 Golden-Section Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 4.5 Quadratic Interpolation Method . . . . . . . . . . . . . . . . . . . . . . . 91 4.5.1 Two-Point Interpolation . . . . . . . . . . . . . . . . . . . . . 94 4.6 Cubic Interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 4.7 Algorithm of Davies, Swann, and Campey . . . . . . . . . . . . . . . 97 4.8 Inexact Line Searches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 5 Basic Multidimensional Gradient Methods . . . . . . . . . . . . . . . . . . . 115 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 5.2 Steepest-Descent Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 5.2.1 Ascent and Descent Directions . . . . . . . . . . . . . . . . 116 5.2.2 Basic Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 5.2.3 Orthogonality of Directions . . . . . . . . . . . . . . . . . . . 119 5.2.4 Step-Size Estimation for Steepest-Descent Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 5.2.5 Step-Size Estimation Using the Barzilai–Borwein Two-Point Formulas . . . . . . . . . . . . . . . . . . . . . . . . 122 5.2.6 Convergence. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 5.2.7 Scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 5.3 Newton Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 5.3.1 Modification of the Hessian . . . . . . . . . . . . . . . . . . . 128 5.3.2 Computation of the Hessian . . . . . . . . . . . . . . . . . . 135 5.3.3 Newton Decrement . . . . . . . . . . . . . . . . . . . . . . . . . 135 5.3.4 Backtracking Line Search . . . . . . . . . . . . . . . . . . . . 135 5.3.5 Independence of Linear Changes in Variables . . . . . 136 5.4 Gauss–Newton Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 xiv Contents
- 18. 6 Conjugate-Direction Methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 6.2 Conjugate Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 6.3 Basic Conjugate-Directions Method . . . . . . . . . . . . . . . . . . . . 148 6.4 Conjugate-Gradient Method . . . . . . . . . . . . . . . . . . . . . . . . . . 152 6.5 Minimization of Nonquadratic Functions . . . . . . . . . . . . . . . . 157 6.6 Fletcher–Reeves Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 6.7 Powell’s Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 6.8 Partan Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 6.9 Solution of Systems of Linear Equations . . . . . . . . . . . . . . . . 173 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178 7 Quasi-Newton Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 7.2 The Basic Quasi-Newton Approach . . . . . . . . . . . . . . . . . . . . 180 7.3 Generation of Matrix Sk . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181 7.4 Rank-One Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 7.5 Davidon–Fletcher–Powell Method . . . . . . . . . . . . . . . . . . . . . 190 7.5.1 Alternative Form of DFP Formula . . . . . . . . . . . . . . 196 7.6 Broyden–Fletcher–Goldfarb–Shanno Method . . . . . . . . . . . . . 197 7.7 Hoshino Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 7.8 The Broyden Family . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200 7.8.1 Fletcher Switch Method . . . . . . . . . . . . . . . . . . . . . 200 7.9 The Huang Family . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 7.10 Practical Quasi-Newton Algorithm . . . . . . . . . . . . . . . . . . . . . 202 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208 8 Minimax Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 8.2 Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 8.3 Minimax Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213 8.4 Improved Minimax Algorithms . . . . . . . . . . . . . . . . . . . . . . . 219 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236 9 Applications of Unconstrained Optimization . . . . . . . . . . . . . . . . . . 239 9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239 9.2 Classification of Handwritten Digits . . . . . . . . . . . . . . . . . . . . 240 9.2.1 Handwritten-Digit Recognition Problem . . . . . . . . . . 240 9.2.2 Histogram of Oriented Gradients . . . . . . . . . . . . . . . 240 9.2.3 Softmax Regression for Use in Multiclass Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243 Contents xv
- 19. 9.2.4 Use ofSoftmax Regression for the Classification of Handwritten Digits . . . . . . . . . . . . . . . . . . . . . . . 249 9.3 Inverse Kinematics for Robotic Manipulators . . . . . . . . . . . . . 253 9.3.1 Position and Orientation of a Manipulator . . . . . . . . 253 9.3.2 Inverse Kinematics Problem . . . . . . . . . . . . . . . . . . 256 9.3.3 Solution of Inverse Kinematics Problem. . . . . . . . . . 257 9.4 Design of Digital Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261 9.4.1 Weighted Least-Squares Design of FIR Filters . . . . . 262 9.4.2 Minimax Design of FIR Filters . . . . . . . . . . . . . . . . 267 9.5 Source Localization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275 9.5.1 Source Localization Based on Range Measurements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275 9.5.2 Source Localization Based on Range-Difference Measurements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 10 Fundamentals of Constrained Optimization . . . . . . . . . . . . . . . . . . 285 10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285 10.2 Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286 10.2.1 Notation and Basic Assumptions . . . . . . . . . . . . . . . 286 10.2.2 Equality Constraints . . . . . . . . . . . . . . . . . . . . . . . . 286 10.2.3 Inequality Constraints . . . . . . . . . . . . . . . . . . . . . . . 290 10.3 Classification of Constrained Optimization Problems . . . . . . . . 292 10.3.1 Linear Programming . . . . . . . . . . . . . . . . . . . . . . . . 293 10.3.2 Quadratic Programming . . . . . . . . . . . . . . . . . . . . . 294 10.3.3 Convex Programming . . . . . . . . . . . . . . . . . . . . . . . 295 10.3.4 General Constrained Optimization Problem . . . . . . . 295 10.4 Simple Transformation Methods. . . . . . . . . . . . . . . . . . . . . . . 296 10.4.1 Variable Elimination . . . . . . . . . . . . . . . . . . . . . . . . 296 10.4.2 Variable Transformations . . . . . . . . . . . . . . . . . . . . 300 10.5 Lagrange Multipliers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303 10.5.1 Equality Constraints . . . . . . . . . . . . . . . . . . . . . . . . 305 10.5.2 Tangent Plane and Normal Plane . . . . . . . . . . . . . . . 308 10.5.3 Geometrical Interpretation . . . . . . . . . . . . . . . . . . . . 310 10.6 First-Order Necessary Conditions . . . . . . . . . . . . . . . . . . . . . . 312 10.6.1 Equality Constraints . . . . . . . . . . . . . . . . . . . . . . . . 312 10.6.2 Inequality Constraints . . . . . . . . . . . . . . . . . . . . . . . 314 10.7 Second-Order Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319 10.7.1 Second-Order Necessary Conditions. . . . . . . . . . . . . 320 10.7.2 Second-Order Sufficient Conditions . . . . . . . . . . . . . 323 10.8 Convexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326 10.9 Duality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328 xvi Contents
- 20. Problems . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338 11 Linear Programming Part I: The Simplex Method . . . . . . . . . . . . . 339 11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339 11.2 General Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339 11.2.1 Formulation of LP Problems . . . . . . . . . . . . . . . . . . 339 11.2.2 Optimality Conditions . . . . . . . . . . . . . . . . . . . . . . . 341 11.2.3 Geometry of an LP Problem . . . . . . . . . . . . . . . . . . 346 11.2.4 Vertex Minimizers . . . . . . . . . . . . . . . . . . . . . . . . . 360 11.3 Simplex Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363 11.3.1 Simplex Method for Alternative-Form LP Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363 11.3.2 Simplex Method for Standard-Form LP Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374 11.3.3 Tabular Form of the Simplex Method . . . . . . . . . . . 383 11.3.4 Computational Complexity . . . . . . . . . . . . . . . . . . . 385 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391 12 Linear Programming Part II: Interior-Point Methods . . . . . . . . . . 393 12.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393 12.2 Primal-Dual Solutions and Central Path . . . . . . . . . . . . . . . . . 394 12.2.1 Primal-Dual Solutions . . . . . . . . . . . . . . . . . . . . . . . 394 12.2.2 Central Path . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396 12.3 Primal Affine Scaling Method . . . . . . . . . . . . . . . . . . . . . . . . 398 12.4 Primal Newton Barrier Method . . . . . . . . . . . . . . . . . . . . . . . 402 12.4.1 Basic Idea . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402 12.4.2 Minimizers of Subproblem . . . . . . . . . . . . . . . . . . . 402 12.4.3 A Convergence Issue . . . . . . . . . . . . . . . . . . . . . . . 403 12.4.4 Computing a Minimizer of the Problem in Eqs. (12.26a) and (12.26b) . . . . . . . . . . . . . . . . . 404 12.5 Primal-Dual Interior-Point Methods . . . . . . . . . . . . . . . . . . . . 407 12.5.1 Primal-Dual Path-Following Method . . . . . . . . . . . . 407 12.5.2 A Nonfeasible-Initialization Primal-Dual Path-Following Method . . . . . . . . . . . . . . . . . . . . . . 412 12.5.3 Predictor-Corrector Method . . . . . . . . . . . . . . . . . . . 415 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424 13 Quadratic, Semidefinite, and Second-Order Cone Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425 13.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425 13.2 Convex QP Problems with Equality Constraints . . . . . . . . . . . 426 Contents xvii
- 21. 13.3 Active-Set Methodsfor Strictly Convex QP Problems . . . . . . . 429 13.3.1 Primal Active-Set Method . . . . . . . . . . . . . . . . . . . . 430 13.3.2 Dual Active-Set Method . . . . . . . . . . . . . . . . . . . . . 434 13.4 Interior-Point Methods for Convex QP Problems . . . . . . . . . . 435 13.4.1 Dual QP Problem, Duality Gap, and Central Path . . . 435 13.4.2 A Primal-Dual Path-Following Method for Convex QP Problems . . . . . . . . . . . . . . . . . . . . 437 13.4.3 Nonfeasible-initialization Primal-Dual Path-Following Method for Convex QP Problems. . . 439 13.4.4 Linear Complementarity Problems . . . . . . . . . . . . . . 442 13.5 Primal and Dual SDP Problems . . . . . . . . . . . . . . . . . . . . . . . 445 13.5.1 Notation and Definitions . . . . . . . . . . . . . . . . . . . . . 445 13.5.2 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 447 13.6 Basic Properties of SDP Problems . . . . . . . . . . . . . . . . . . . . . 450 13.6.1 Basic Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . 450 13.6.2 Karush-Kuhn-Tucker Conditions . . . . . . . . . . . . . . . 450 13.6.3 Central Path . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 451 13.6.4 Centering Condition . . . . . . . . . . . . . . . . . . . . . . . . 452 13.7 Primal-Dual Path-Following Method . . . . . . . . . . . . . . . . . . . 453 13.7.1 Reformulation of Centering Condition . . . . . . . . . . . 453 13.7.2 Symmetric Kronecker Product . . . . . . . . . . . . . . . . . 454 13.7.3 Reformulation of Eqs. (13.79a)–(13.79c) . . . . . . . . . 455 13.7.4 Primal-Dual Path-Following Algorithm . . . . . . . . . . 457 13.8 Predictor-Corrector Method . . . . . . . . . . . . . . . . . . . . . . . . . . 460 13.9 Second-Order Cone Programming . . . . . . . . . . . . . . . . . . . . . 465 13.9.1 Notation and Definitions . . . . . . . . . . . . . . . . . . . . . 465 13.9.2 Relations Among LP, QP, SDP, and SOCP Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 466 13.9.3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 468 13.10 A Primal-Dual Method for SOCP Problems . . . . . . . . . . . . . . 472 13.10.1 Assumptions and KKT Conditions . . . . . . . . . . . . . . 472 13.10.2 A Primal-Dual Interior-Point Algorithm . . . . . . . . . . 473 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 476 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 480 14 Algorithms for General Convex Problems . . . . . . . . . . . . . . . . . . . 483 14.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483 14.2 Concepts and Properties of Convex Functions . . . . . . . . . . . . 483 14.2.1 Subgradient . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 484 14.2.2 Convex Functions with Lipschitz-Continuous Gradients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 488 14.2.3 Strongly Convex Functions . . . . . . . . . . . . . . . . . . . 491 xviii Contents
- 22. 14.2.4 Conjugate Functions. . . . . . . . . . . . . . . . . . . . . . . . 494 14.2.5 Proximal Operators . . . . . . . . . . . . . . . . . . . . . . . . . 498 14.3 Extension of Newton Method to Convex Constrained and Unconstrained Problems . . . . . . . . . . . . . . . . . . . . . . . . . 500 14.3.1 Minimization of Smooth Convex Functions Without Constraints . . . . . . . . . . . . . . . . . . . . . . . . 500 14.3.2 Minimization of Smooth Convex Functions Subject to Equality Constraints . . . . . . . . . . . . . . . . . . . . . . 502 14.3.3 Newton Algorithm for Problem in Eq. (14.34) with a Nonfeasible x0 . . . . . . . . . . . . . . . . . . . . . . . 504 14.3.4 A Newton Barrier Method for General Convex Programming Problems . . . . . . . . . . . . . . . . . . . . . . 507 14.4 Minimization of Composite Convex Functions . . . . . . . . . . . . 512 14.4.1 Proximal-Point Algorithm . . . . . . . . . . . . . . . . . . . . 512 14.4.2 Fast Algorithm For Solving the Problem in Eq. (14.56) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514 14.5 Alternating Direction Methods . . . . . . . . . . . . . . . . . . . . . . . . 519 14.5.1 Alternating Direction Method of Multipliers . . . . . . . 519 14.5.2 Application of ADMM to General Constrained Convex Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . 526 14.5.3 Alternating Minimization Algorithm (AMA). . . . . . . 529 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 530 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537 15 Algorithms for General Nonconvex Problems . . . . . . . . . . . . . . . . . 539 15.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 539 15.2 Sequential Convex Programming . . . . . . . . . . . . . . . . . . . . . . 540 15.2.1 Principle of SCP . . . . . . . . . . . . . . . . . . . . . . . . . . . 540 15.2.2 Convex Approximations for fðxÞ and cjðxÞ and Affine Approximation of aiðxÞ . . . . . . . . . . . . . 541 15.2.3 Exact Penalty Formulation . . . . . . . . . . . . . . . . . . . 544 15.2.4 Alternating Convex Optimization . . . . . . . . . . . . . . . 546 15.3 Sequential Quadratic Programming. . . . . . . . . . . . . . . . . . . . . 550 15.3.1 Basic SQP Algorithm . . . . . . . . . . . . . . . . . . . . . . . 552 15.3.2 Positive Definite Approximation of Hessian . . . . . . . 554 15.3.3 Robustness and Solvability of QP Subproblem of Eqs. (15.16a)–(15.16c) . . . . . . . . . . . . . . . . . . . . 555 15.3.4 Practical SQP Algorithm for the Problem of Eq. (15.1). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 556 15.4 Convex-Concave Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . 557 15.4.1 Basic Convex-Concave Procedure . . . . . . . . . . . . . . 558 15.4.2 Penalty Convex-Concave Procedure . . . . . . . . . . . . . 559 15.5 ADMM Heuristic Technique for Nonconvex Problems . . . . . . 562 Contents xix
- 23. Problems . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 565 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 568 16 Applications of Constrained Optimization. . . . . . . . . . . . . . . . . . . . 571 16.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571 16.2 Design of Digital Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 572 16.2.1 Design of Linear-Phase FIR Filters Using QP . . . . . 572 16.2.2 Minimax Design of FIR Digital Filters Using SDP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 574 16.2.3 Minimax Design of IIR Digital Filters Using SDP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 578 16.2.4 Minimax Design of FIR and IIR Digital Filters Using SOCP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 586 16.2.5 Minimax Design of IIR Digital Filters Satisfying Multiple Specifications . . . . . . . . . . . . . . . . . . . . . . 587 16.3 Model Predictive Control of Dynamic Systems . . . . . . . . . . . . 591 16.3.1 Polytopic Model for Uncertain Dynamic Systems . . . 591 16.3.2 Introduction to Robust MPC . . . . . . . . . . . . . . . . . . 592 16.3.3 Robust Unconstrained MPC by Using SDP . . . . . . . 594 16.3.4 Robust Constrained MPC by Using SDP . . . . . . . . . 597 16.4 Optimal Force Distribution for Robotic Systems with Closed Kinematic Loops . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 602 16.4.1 Force Distribution Problem in Multifinger Dextrous Hands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 602 16.4.2 Solution of Optimal Force Distribution Problem by Using LP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 608 16.4.3 Solution of Optimal Force Distribution Problem by Using SDP . . . . . . . . . . . . . . . . . . . . . . . . . . . . 611 16.5 Multiuser Detection in Wireless Communication Channels . . . 614 16.5.1 Channel Model and ML Multiuser Detector . . . . . . . 615 16.5.2 Near-Optimal Multiuser Detector Using SDP Relaxation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 617 16.5.3 A Constrained Minimum-BER Multiuser Detector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 625 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 631 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 633 Appendix A: Basics of Linear Algebra . . . . . . . . . . . . . . . . . . . . . . . . . . . 635 Appendix B: Basics of Digital Filters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 673 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 691 xx Contents
- 24. About the Authors AndreasAntoniou received the B.Sc. and Ph.D. degrees in Electrical Engineering from the University of London, UK, in 1963 and 1966, respectively. He is a Life Member of the Association of Professional Engineers and Geoscientists of British Columbia, Canada, a Fellow of the Institution of Engineering and Technology, and a Life Fellow of the Institute of Electrical and Electronic Engineers. He served as the founding Chair of the Department of Electrical and Computer Engineering at the University of Victoria, BC, Canada and is now Professor Emeritus. He is the author of Digital Filters: Analysis, Design, and Signal Processing Applications published by McGraw-Hill in 2018 (previous editions of the book were published by McGraw-Hill under slightly different titles in 1979, 1993, and 2005). He served as Associate Editor/Editor of IEEE Transactions on Circuits and Systems from June 1983 to May 1987, as a Distinguished Lecturer of the IEEE Signal Processing Society in 2003, as General Chair of the 2004 International Symposium on Circuits and Systems, and as a Distinguished Lecturer of the IEEE Circuits and Systems Society during 2006–2007. He received the Ambrose Fleming Premium for 1964 from the IEE (best paper award), the CAS Golden Jubilee Medal from the IEEE Circuits and Systems Society in recognition of outstanding achievements in the area of circuits and systems, the BC Science Council Chairman’s Award for Career Achievement both in 2000, the Doctor Honoris Causa degree by the Metsovio National Technical University, Athens, Greece, in 2002, the IEEE Circuits and Systems Society Technical Achievement Award for 2005, the IEEE Canada Out- standing Engineering Educator Silver Medal for 2008, the IEEE Circuits and Systems Society Education Award for 2009, the Craigdarroch Gold Medal for Career Achievement for 2011, and the Legacy Award for 2011 both from the University of Victoria. Wu-Sheng Lu received the B.Sc. degree in Mathematics from Fudan University, Shanghai, China, in 1964, the M.S. degree in Electrical Engineering, and the Ph.D. degree in Control Science both from the University of Minnesota, Min- neapolis, in 1983 and 1984, respectively. He is a Member of the Association of Professional Engineers and Geoscientists of British Columbia, Canada, a Fellow of the Engineering Institute of Canada, and a Fellow of the Institute of Electrical and Electronics Engineers. He has been teaching and carrying out research in the areas of digital signal processing and application of optimization methods at the xxi
- 25. University of Victoria,BC, Canada, since 1987. He is the co-author with A. Antoniou of Two-Dimensional Digital Filters published by Marcel Dekker in 1992. He served as an Associate Editor of the Canadian Journal of Electrical and Computer Engineering in 1989, and Editor of the same journal from 1990 to 1992. He served as an Associate Editor for the IEEE Transactions on Circuits and Sys- tems, Part II, from 1993 to 1995 and for Part I of the same journal from 1999 to 2001 and 2004 to 2005. He received two best paper awards from the IEEE Asia Pacific Conference on Circuits and Systems in 2006 and 2014, the Outstanding Teacher Award of the Engineering Institute of Canada, Vancouver Island Branch, for 1988 and 1990, and the University of Victoria Alumni Association Award for Excellence in Teaching for 1991. xxii About the Authors
- 26. Abbreviations ADMM Alternating directionmethods of multipliers AMA Alternating minimization algorithms AWGN Additive white Gaussian noise BER Bit-error rate BFGS Broyden-Fletcher-Goldfarb-Shanno CCP Convex-concave procedure CDMA Code-division multiple access CMBER Constrained minimum BER CP Convex programming DFP Davidon-Fletcher-Powell DH Denavit-Hartenberg DNB Dual Newton barrier DS-CDMA Direct-sequence CDMA FDMA Frequency-division multiple access FIR Finite-duration impulse response FISTA Fast iterative shrinkage-thresholding algorithm FR Fletcher-Reeves GCO General constrained optimization GN Gauss-Newton HOG Histogram of oriented gradient HWDR Handwritten digits recognition IIR Infinite-duration impulse response IP Integer programming KKT Karush-Kuhn-Tucker LMI Linear matrix inequality LP Linear programming LSQI Least-squares minimization with quadratic inequality LU Lower-upper MAI Multiple access interference ML Maximum-likelihood MLE Maximum-likelihood estimation MNIST Modified National Institute for Standards and Technology MPC Model predictive control NAG Nesterov’s accelerated gradient xxiii
- 27. PAS Primal affinescaling PCCP Penalty convex-concave procedure PCM Predictor-corrector method PNB Primal Newton barrier QP Quadratic programming SCP Sequential convex programming SD Steepest-descent SDP Semidefinite programming SDPR-D SDP relaxation dual SDPR-P SDP relaxation primal SNR Signal-to-noise ratio SOCP Second-order cone programming SQP Sequential quadratic programming SVD Singular-value decomposition TDMA Time-division multiple access xxiv Abbreviations
- 28. 1 The Optimization Problem 1.1Introduction Throughout the ages, man has continuously been involved with the process of op- timization. In its earliest form, optimization consisted of unscientific rituals and prejudices like pouring libations and sacrificing animals to the gods, consulting the oracles, observing the positions of the stars, and watching the flight of birds. When the circumstances were appropriate, the timing was thought to be auspicious (or optimum) for planting the crops or embarking on a war. As the ages advanced and the age of reason prevailed, unscientific rituals were replaced by rules of thumb and later, with the development of mathematics, mathe- matical calculations began to be applied. Interest in the process of optimization has taken a giant leap with the advent of the digital computer in the early fifties. In recent years, optimization techniques advanced rapidly and considerable progress has been achieved. At the same time, digital com- puters became faster, more versatile, and more efficient. As a consequence, it is now possible to solve complex optimization problems which were thought intractable only a few years ago. The process of optimization is the process of obtaining the ‘best’, if it is possible to measure and change what is ‘good’ or ‘bad’. In practice, one wishes the ‘most’ or ‘maximum’ (e.g., salary) or the ‘least’ or ‘minimum’ (e.g., expenses). Therefore, the word ‘optimum’ is taken to mean ‘maximum’ or ‘minimum’ depending on the cir- cumstances; ‘optimum’ is a technical term which implies quantitative measurement and is a stronger word than ‘best’ which is more appropriate for everyday use. Like- wise, the word ‘optimize’, which means to achieve an optimum, is a stronger word than ‘improve’. Optimization theory is the branch of mathematics encompassing the quantitative study of optima and methods for finding them. Optimization practice, on the other hand, is the collection of techniques, methods, procedures, and algorithms that can be used to find the optima. Optimization problems occur in most disciplines like engineering, physics, math- ematics, economics, administration, commerce, social sciences, and even politics. © Springer Science+Business Media, LLC, part of Springer Nature 2021 A. Antoniou and W.-S. Lu, Practical Optimization, Texts in Computer Science, https://doi.org/10.1007/978-1-0716-0843-2_1 1
- 29. 2 1 TheOptimization Problem Optimization problems abound in the various fields of engineering like electrical, mechanical, civil, chemical, and building engineering. Typical areas of application are modeling, characterization, and design of devices, circuits, and systems; design of tools, instruments, and equipment; design of structures and buildings; process control; approximation theory, curve fitting, solution of systems of equations; fore- casting, production scheduling, quality control; maintenance and repair; inventory control, accounting, budgeting, etc. Some recent innovations rely almost entirely on optimization theory, for example, neural networks and adaptive systems. Most real- life problems have several solutions and occasionally an infinite number of solutions may be possible. Assuming that the problem at hand admits more than one solution, optimization can be achieved by finding the best solution of the problem in terms of some performance criterion. If the problem admits only one solution, that is, only a unique set of parameter values is acceptable, then optimization cannot be applied. Several general approaches to optimization are available, as follows: 1. Analytical methods 2. Graphical methods 3. Experimental methods 4. Numerical methods Analytical methods are based on the classical techniques of differential calculus. In these methods the maximum or minimum of a performance criterion is deter- mined by finding the values of parameters x1, x2, . . . , xn that cause the derivatives of f (x1, x2, . . . , xn) with respect to x1, x2, . . . , xn to assume zero values. The problem to be solved must obviously be described in mathematical terms before the rules of calculus can be applied. The method need not entail the use of a digital com- puter. However, it cannot be applied to highly nonlinear problems or to problems where the number of independent parameters exceeds two or three. A graphical method can be used to plot the function to be maximized or minimized if the number of variables does not exceed two. If the function depends on only one variable, say, x1, a plot of f (x1) versus x1 will immediately reveal the maxima and/or minima of the function. Similarly, if the function depends on only two variables, say, x1 and x2, a set of contours can be constructed. A contour is a set of points in the (x1, x2) plane for which f (x1, x2) is constant, and so a contour plot, like a topographical map of a specific region, will reveal readily the peaks and valleys of the function. For example, the contour plot of f (x1, x2) depicted in Fig. 1.1 shows that the function has a minimum at point A. Unfortunately, the graphical method is of limited usefulness since in most practical applications the function to be optimized depends on several variables, usually in excess of four. The optimum performance of a system can sometimes be achieved by direct experimentation. In this method, the system is set up and the process variables are adjusted one by one and the performance criterion is measured in each case. This method may lead to optimum or near optimum operating conditions. However, it can lead to unreliable results since in certain systems, two or more variables interact with
- 30. 1.1 Introduction 3 Fig.1.1Contour plot of f (x1, x2) A 10 f (x , x ) = 0 1 2 f (x , x ) = 50 1 2 1 x 2 x 20 30 40 50 each other, and must be adjusted simultaneously to yield the optimum performance criterion. The most important general approach to optimization is based on numerical meth- ods. In this approach, iterative numerical procedures are used to generate a series of progressively improved solutions to the optimization problem, starting with an initial estimate for the solution. The process is terminated when some convergence criterion is satisfied. For example, when changes in the independent variables or the performance criterion from iteration to iteration become insignificant. Numerical methods can be used to solve highly complex optimization problems of the type that cannot be solved analytically. Furthermore, they can be readily programmed on the digital computer. Consequently, they have all but replaced most other approaches to optimization. The discipline encompassing the theory and practice of numerical optimization methods has come to be known as mathematical programming [1–5]. During the past 40 years, several branches of mathematical programming have evolved, as follows: 1. Linear programming 2. Integer programming 3. Quadratic programming 4. Nonlinear programming 5. Dynamic programming Each one of these branches of mathematical programming is concerned with a spe- cific class of optimization problems. The differences among them will be examined in Sect.1.6.
- 31. 4 1 TheOptimization Problem 1.2 The Basic Optimization Problem Before optimization is attempted, the problem at hand must be properly formulated. A performance criterion F must be derived in terms of n parameters x1, x2, . . . , xn as F = f (x1, x2, . . . , xn) F is a scalar quantity which can assume numerous forms. It can be the cost of a product in a manufacturing environment or the difference between the desired per- formance and the actual performance in a system. Variables x1, x2, . . . , xn are the parameters that influence the product cost in the first case or the actual performance in the second case. They can be independent variables, like time, or control parameters that can be adjusted. The most basic optimization problem is to adjust variables x1, x2, . . . , xn in such a way as to minimize quantity F. This problem can be stated mathematically as minimize F = f (x1, x2, . . . , xn) (1.1) Quantity F is usually referred to as the objective or cost function. The objective function may depend on a large number of variables, sometimes as many as 100 or more. To simplify the notation, matrix notation is usually employed. If x is a column vector with elements x1, x2, . . . , xn, the transpose of x, namely, xT , can be expressed as the row vector xT = [x1 x2 · · · xn] In this notation, the basic optimization problem of Eq. (1.1) can be expressed as minimize F = f (x) for x ∈ En where En represents the n-dimensional Euclidean space. On many occasions, the optimization problem consists of finding the maximum of the objective function. Since max[ f (x)] = −min[− f (x)] the maximum of F can be readily obtained by finding the minimum of the negative of F and then changing the sign of the minimum. Consequently, in this and subsequent chapters we focus our attention on minimization without loss of generality. In many applications, a number of distinct functions of x need to be optimized simultaneously. For example, if the system of nonlinear simultaneous equations fi (x) = 0 for i = 1, 2, . . . , m needs to be solved, a vector x is sought which will reduce all fi (x) to zero simulta- neously. In such a problem, the functions to be optimized can be used to construct a vector F(x) = [ f1(x) f2(x) · · · fm(x)]T The problem can be solved by finding a point x = x∗ such that F(x∗) = 0. Very frequently, a point x∗ that reduces all the fi (x) to zero simultaneously may not
- 32. 1.2 The BasicOptimization Problem 5 exist but an approximate solution, i.e., F(x∗) ≈ 0, may be available which could be entirely satisfactory in practice. A similar problem arises in scientific or engineering applications when the func- tion of x that needs to be optimized is also a function of a continuous independent parameter (e.g., time, position, speed, frequency) that can assume an infinite set of values in a specified range. The optimization might entail adjusting variables x1, x2, . . . , xn so as to optimize the function of interest over a given range of the independent parameter. In such an application, the function of interest can be sampled with respect to the independent parameter, and a vector of the form F(x) = [ f (x, t1) f (x, t2) · · · f (x, tm)]T can be constructed, where t is the independent parameter. Now if we let fi (x) ≡ f (x, ti ) we can write F(x) = [ f1(x) f2(x) · · · fm(x)]T A solution of such a problem can be obtained by optimizing functions fi (x) for i = 1, 2, . . . , m simultaneously. Such a solution would, of course, be approximate because any variations in f (x, t) between sample points are ignored. Nevertheless, reasonable solutions can be obtained in practice by using a sufficiently large number of sample points. This approach is illustrated by the following example. Example 1.1 The step response y(x, t) of an nth-order control system is required to satisfy the specification y0(x, t) = ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ t for 0 ≤ t < 2 2 for 2 ≤ t < 3 −t + 5 for 3 ≤ t < 4 1 for 4 ≤ t as closely as possible. Construct a vector F(x) that can be used to obtain a function f (x, t) such that y(x, t) ≈ y0(x, t) for 0 ≤ t ≤ 5 Solution The difference between the actual and specified step responses, which constitutes the approximation error, can be expressed as f (x, t) = y(x, t) − y0(x, t) and if f (x, t) is sampled at t = 0, 1, . . . , 5, we obtain F(x) = [ f1(x) f2(x) · · · f6(x)]T
- 33. 6 1 TheOptimization Problem 0 1 2 3 4 5 1 2 3 y (x, t) y (x, t) 0 f (x, t) t Fig.1.2 Graphical construction for Example1.1 where f1(x) = f (x, 0) = y(x, 0) f2(x) = f (x, 1) = y(x, 1) − 1 f3(x) = f (x, 2) = y(x, 2) − 2 f4(x) = f (x, 3) = y(x, 3) − 2 f5(x) = f (x, 4) = y(x, 4) − 1 f6(x) = f (x, 5) = y(x, 5) − 1 The problem is illustrated in Fig. 1.2. It can be solved by finding a point x = x∗ such that F(x∗) ≈ 0. Evidently, the quality of the approximation obtained for the step response of the system will depend on the density of the sampling points and the higher the density of points, the better the approximation. Problems of the type just described can be solved by defining a suitable objective function in terms of the element functions of F(x). The objective function must be a scalar quantity and its optimization must lead to the simultaneous optimization of the element functions of F(x) in some sense. Consequently, a norm of some type must be used. An objective function can be defined in terms of the L p norm as F ≡ L p = m i=1 | fi (x)|p 1/p where p is an integer.1 1See Sect.A.8 for more details on vector and matrix norms. AppendixA also deals with other aspects of linear algebra that are important to optimization.
- 34. 1.2 The BasicOptimization Problem 7 Several special cases of the L p norm are of particular interest. If p = 1 F ≡ L1 = m i=1 | fi (x)| and, therefore, in a minimization problem like that in Example1.1, the sum of the magnitudes of the individual element functions is minimized. This is called an L1 problem. If p = 2, the Euclidean norm F ≡ L2 = m i=1 | fi (x)|2 1/2 is minimized, and if the square root is omitted, the sum of the squares is minimized. Such a problem is commonly referred to as a least-squares problem. In the case where p = ∞, if we assume that there is a unique maximum of | fi (x)| designated F̂ such that F̂ = max 1≤i≤m | fi (x)| then we can write F ≡ L∞ = lim p→∞ m i=1 | fi (x)|p 1/p = F̂ lim p→∞ m i=1 | fi (x)| F̂ p 1/p Since all the terms in the summation except one are less than unity, they tend to zero when raised to a large positive power. Therefore, we obtain F = F̂ = max 1≤i≤m | fi (x)| Evidently, if the L∞ norm is used in Example1.1, the maximum approximation error is minimized and the problem is said to be a minimax problem. Often the individual element functions of F(x) are modified by using constants w1, w2, . . . , wm as weights. For example, the least-squares objective function can be expressed as F = m i=1 [wi fi (x)]2 so as to emphasize important or critical element functions and de-emphasize unim- portant or uncritical ones. If F is minimized, the residual errors in wi fi (x) at the end of the minimization would tend to be of the same order of magnitude, i.e., error in |wi fi (x)| ≈ ε and so error in | fi (x)| ≈ ε |wi | Consequently, if a large positive weight wi is used with fi (x), a small residual error is achieved in | fi (x)|.
- 35. 8 1 TheOptimization Problem 1.3 General Structure of Optimization Algorithms Most of the available optimization algorithms entail a series of steps which are executed sequentially. A typical pattern is as follows: Algorithm 1.1 General optimization algorithm Step 1 (a) Set k = 0 and initialize x0. (b) Compute F0 = f (x0). Step 2 (a) Set k = k + 1. (b) Compute the changes in xk given by column vector Δxk where ΔxT k = [Δx1 Δx2 · · · Δxn] by using an appropriate procedure. (c) Set xk = xk−1 + Δxk (d) Compute Fk = f (xk) and ΔFk = Fk−1 − Fk. Step 3 Check if convergence has been achieved by using an appropriate criterion, e.g., by checking ΔFk and/or Δxk. If this is the case, continue to Step 4; otherwise, go to Step 2. Step 4 (a) Output x∗ = xk and F∗ = f (x∗). (b) Stop. In Step 1, vector x0 is initialized by estimating the solution using knowledge about the problem at hand. Often the solution cannot be estimated and an arbitrary solution may be assumed, say, x0 = 0. Steps 2 and 3 are then executed repeatedly until convergence is achieved. Each execution of Steps 2 and 3 constitutes one iteration, that is, k is the number of iterations. When convergence is achieved, Step 4 is executed. In this step, column vector x∗ = [x∗ 1 x∗ 2 · · · x∗ n ]T = xk and the corresponding value of F, namely, F∗ = f (x∗ ) are output. The column vector x∗ is said to be the optimum, minimum, solution point, or simply the minimizer, and F∗ is said to be the optimum or minimum value of the objective function. The pair x∗ and F∗ constitute the solution of the optimization problem. Convergence can be checked in several ways, depending on the optimization problem and the optimization technique used. For example, one might decide to stop the algorithm when the reduction in Fk between any two iterations has become insignificant, that is, |ΔFk| = |Fk−1 − Fk| εF (1.2)
- 36. 1.3 General Structureof Optimization Algorithms 9 where εF is an optimization tolerance for the objective function. Alternatively, one might decide to stop the algorithm when the changes in all variables have become insignificant, that is, |Δxi | εx for i = 1, 2, . . . , n (1.3) where εx is an optimization tolerance for variables x1, x2, . . . , xn. A third possi- bility might be to check if both criteria given by Eqs. (1.2) and (1.3) are satisfied simultaneously. There are numerous algorithms for the minimization of an objective function. However, we are primarily interested in algorithms that entail the minimum amount of effort. Therefore, we shall focus our attention on algorithms that are simple to apply, are reliable when applied to a diverse range of optimization problems, and entail a small amount of computation. A reliable algorithm is often referred to as a ‘robust’ algorithm in the terminology of mathematical programming. 1.4 Constraints In many optimization problems, the variables are interrelated by physical laws like the conservation of mass or energy, Kirchhoff’s voltage and current laws, and other system equalities that must be satisfied. In effect, in these problems certain equality constraints of the form ai (x) = 0 for x ∈ En where i = 1, 2, . . . , p must be satisfied before the problem can be considered solved. In other optimization problems a collection of inequality constraints might be imposed on the variables or parameters to ensure physical realizability, reliability, compatibility, or even to simplify the modeling of the problem. For example, the power dissipation might become excessive if a particular current in a circuit exceeds a given upper limit or the circuit might become unreliable if another current is reduced below a lower limit, the mass of an element in a specific chemical reaction must be positive, and so on. In these problems, a collection of inequality constraints of the form cj (x) ≥ 0 for x ∈ En where j = 1, 2, . . . , q must be satisfied before the optimization problem can be considered solved. An optimization problem may entail a set of equality constraints and possibly a set of inequality constraints. If this is the case, the problem is said to be a constrained optimization problem. The most general constrained optimization problem can be expressed mathematically as minimize f (x) for x ∈ En (1.4a) subject to: ai (x) = 0 for i = 1, 2, . . . , p (1.4b) cj (x) ≥ 0 for j = 1, 2, . . . , q (1.4c)
- 37. 10 1 TheOptimization Problem Fig.1.3 The double inverted pendulum θ 1 θ 2 Μ u(t) A problem that does not entail any equality or inequality constraints is said to be an unconstrained optimization problem. Constrained optimization is usually much more difficult than unconstrained opti- mization, as might be expected. Consequently, the general strategy that has evolved in recent years towards the solution of constrained optimization problems is to re- formulate constrained problems as unconstrained optimization problems. This can be done by redefining the objective function such that the constraints are simultane- ously satisfied when the objective function is minimized. Some real-life constrained optimization problems are given as Examples1.2 to 1.4 below. Example 1.2 Consider a control system that comprises a double inverted pendulum as depicted in Fig. 1.3. The objective of the system is to maintain the pendulum in the upright position using the minimum amount of energy. This is achieved by applying an appropriate control force to the car to damp out any displacements θ1(t) and θ2(t). Formulate the problem as an optimization problem. Solution The dynamic equations of the system are nonlinear and the standard practice is to apply a linearization technique to these equations to obtain a small-signal linear model of the system as [6] ẋ(t) = Ax(t) + fu(t) (1.5) where x(t) = ⎡ ⎢ ⎢ ⎣ θ1(t) θ̇1(t) θ2(t) θ̇2(t) ⎤ ⎥ ⎥ ⎦ , A = ⎡ ⎢ ⎢ ⎣ 0 1 0 0 α 0 −β 0 0 0 0 1 −α 0 α 0 ⎤ ⎥ ⎥ ⎦ , f = ⎡ ⎢ ⎢ ⎣ 0 −1 0 0 ⎤ ⎥ ⎥ ⎦ with α 0, β 0, and α = β. In the above equations, ẋ(t), θ̇1(t), and θ̇2(t) represent the first derivatives of x(t), θ1(t), and θ2(t), respectively, with respect to time, θ̈1(t) and θ̈2(t) would be the second derivatives of θ1(t) and θ2(t), and parameters α and β depend on system parameters such as the length and weight of each pendulum, the mass of the car, etc. Suppose that at instant t = 0 small nonzero
- 38. 1.4 Constraints 11 displacementsθ1(t) and θ2(t) occur, which would call for immediate control action in order to steer the system back to the equilibrium state x(t) = 0 at time t = T0. In order to develop a digital controller, the system model in Eq. (1.5) is discretized to become x(k + 1) = x(k) + gu(k) (1.6) where = I + ΔtA, g = Δtf, Δt is the sampling interval, and I is the identity matrix. Let x(0) = 0 be given and assume that T0 is a multiple of Δt, i.e., T0 = KΔt where K is an integer. We seek to find a sequence of control actions u(k) for k = 0, 1, . . . , K − 1 such that the zero equilibrium state is achieved at t = T0, i.e., x(T0) = 0. Let us assume that the energy consumed by these control actions, namely, J = K−1 k=0 u2 (k) needs to be minimized. This optimal control problem can be formulated analytically as minimize J = K−1 k=0 u2 (k) (1.7a) subject to: x(K) = 0 (1.7b) From Eq. (1.6), we know that the state of the system at t = KΔt is determined by the initial value of the state and system model in Eq. (1.6) as x(K) = K x(0) + K−1 k=0 K−k−1 gu(k) ≡ −h + K−1 k=0 gku(k) where h = −K x(0) and gk = K−k−1g. Hence the constraint in Eq. (1.7b) is equivalent to K−1 k=0 gku(k) = h (1.8) If we define u = [u(0) u(1) · · · u(K − 1)]T and G = [g0 g1 · · · gK−1], then the constraint in Eq. (1.8) can be expressed as Gu = h, and the optimal control problem at hand can be formulated as the problem of finding a u that solves the minimization problem
- 39. 12 1 TheOptimization Problem minimize uT u (1.9a) subject to: a(u) = 0 (1.9b) where a(u) = Gu − h. In practice, the control actions cannot be made arbitrarily large in magnitude. Consequently, additional constraints are often imposed on |u(i)|, for instance, |u(i)| ≤ m for i = 0, 1, . . . , K − 1 These constraints are equivalent to m + u(i) ≥ 0 m − u(i) ≥ 0 Hence if we define c(u) = ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ m + u(0) m − u(0) . . . m + u(K − 1) m − u(K − 1) ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ then the magnitude constraints can be expressed as c(u) ≥ 0 (1.9c) Obviously, the problem in Eqs. (1.9a)–(1.9c) fits nicely into the standard form of optimization problems given by Eqs. (1.4a)–(1.4c). Example 1.3 High performance in modern optical instruments depends on the qual- ity of components like lenses, prisms, and mirrors. These components have reflecting or partially reflecting surfaces, and their performance is limited by the reflectivities of the materials of which they are made. The surface reflectivity can, however, be altered by the deposition of a thin transparent film. In fact, this technique facilitates the control of losses due to reflection in lenses and makes possible the construction of mirrors with unique properties [7,8]. As is depicted in Fig. 1.4, a typical N-layer thin-film system consists of N layers of thin films of certain transparent media deposited on a glass substrate. The thickness and refractive index of the ith layer are denoted as xi and ni , respectively. The refractive index of the medium above the first layer is denoted as n0. If φ0 is the angle of incident light, then the transmitted ray in the (i − 1)th layer is refracted at an angle φi which is given by Snell’s law, namely, ni sin φi = n0 sin φ0 Given angle φ0 and the wavelength of light, λ, the energy of the light reflected from the film surface and the energy of the light transmitted through the film surface are usually measured by the reflectance R and transmittance T , which satisfy the relation R + T = 1
- 40. 1.4 Constraints 13 Fig.1.4An N-layer thin-film system n0 n3 n2 nN nN+1 layer 1 layer 2 layer 3 x2 x1 n1 Substrate φ1 φ0 φ2 layer N xn φN For an N-layer system, R is given by (see [9] for details) R(x1, . . . , xN , λ) = η0 − y η0 + y 2 y = c b b c = N k=1 cos δk ( j sin δk)/ηk jηk sin δk cos δk 1 ηN+1 where j = √ −1 and δk = 2πnk xk cos φk λ ηk = ⎧ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩ nk/ cos φk for light polarized with the electric vector lying in the plane of incidence nk cos φk for light polarized with the electric vector perpendicular to the plane of incidence The design of a multilayer thin-film system can now be accomplished as follows: Given a range of wavelengths λl ≤ λ ≤ λu and an angle of incidence φ0, find x1, x2, . . . , xN such that the reflectance R(x, λ) best approximates a desired re- flectance Rd(λ) for λ ∈ [λl, λu]. Formulate the design problem as an optimization problem.
- 41. 14 1 TheOptimization Problem Solution In practice, the desired reflectance is specified at grid points λ1, λ2, . . . , λK in the interval [λl, λu]; hence the design may be carried out by selecting xi such that the objective function J = K i=1 wi [R(x, λi ) − Rd(λi )]2 (1.10) is minimized, where x = [x1 x2 · · · xN ]T and wi 0 is a weight to reflect the importance of term [R(x, λi ) − Rd(λi )]2 in Eq. (1.10). If we let η = [1 ηN+1]T , e+ = [η0 1]T , e− = [η0 −1]T , and M(x, λ) = N k=1 cos δk ( j sin δk)/ηk jηk sin δk cos δk then R(x, λ) can be expressed as R(x, λ) = bη0 − c bη0 + c 2 = eT −M(x, λ)η eT +M(x, λ)η 2 Finally, we note that the thickness of each layer cannot be made arbitrarily thin or arbitrarily large and, therefore, constraints must be imposed on the elements of x as dil ≤ xi ≤ diu for i = 1, 2, . . . , N The design problem can now be formulated as the constrained minimization problem minimize J = K i=1 wi eT −M(x, λi )η eT +M(x, λi )η 2 − Rd(λi )2 subject to: xi − dil ≥ 0 for i = 1, 2, . . . , N diu − xi ≥ 0 for i = 1, 2, . . . , N Example 1.4 Quantities q1, q2, . . . , qm of a certain product are produced by m manufacturing divisions of a company, which are at distinct locations. The product is to be shipped to n destinations that require quantities b1, b2, . . . , bn. Assume that the cost of shipping a unit from manufacturing division i to destination j is ci j with i = 1, 2, . . . , m and j = 1, 2, . . . , n. Find the quantity xi j to be shipped from division i to destination j so as to minimize the total cost of transportation, i.e., minimize C = m i=1 n j=1 ci j xi j This is known as the transportation problem. Formulate the problem as an optimiza- tion problem.
- 42. 1.4 Constraints 15 SolutionNote that there are several constraints on variables xi j . First, each division can provide only a fixed quantity of the product, hence n j=1 xi j = qi for i = 1, 2, . . . , m Second, the quantity to be shipped to a specific destination has to meet the need of that destination and so m i=1 xi j = bj for j = 1, 2, . . . , n In addition, the variables xi j are nonnegative and thus, we have xi j ≥ 0 for i = 1, 2, . . . , m and j = 1, 2, . . . , n If we let c = [c11 · · · c1n c21 · · · c2n · · · cm1 · · · cmn]T x = [x11 · · · x1n x21 · · · x2n · · · xm1 · · · xmn]T A = ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ 1 1 · · · 1 0 0 · · · 0 · · · · · · · · · · · · 0 0 · · · 0 1 1 · · · 1 · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · 0 0 · · · 0 0 0 · · · 0 · · · 1 1 · · · 1 1 0 · · · 0 1 0 · · · 0 · · · 1 0 · · · 0 0 1 · · · 0 0 1 · · · 0 · · · 0 1 · · · 0 · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · 0 0 · · · 1 0 0 · · · 1 · · · 0 0 · · · 1 ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ b = [q1 · · · qm b1 · · · bn]T then the minimization problem can be stated as minimize C = cT x (1.11a) subject to: Ax = b (1.11b) x ≥ 0 (1.11c) where cT x is the inner product of c and x. The above problem, like those in Exam- ples1.2 and 1.3, fits into the standard optimization problem in Eqs. (1.4a)–(1.4c). Since both the objective function in Eq. (1.11a) and the constraints in Eqs. (1.11b) and (1.11c) are linear, the problem is known as a linear programming (LP) problem (see Sect.1.6.1). Example 1.5 An investment portfolio is a collection of a variety of securities owned by an individual whereby each security is associated with a unique possible return and a corresponding risk. Design an optimal investment portfolio that would minimize the risk involved subject to an acceptable return for a portfolio that comprises n securities.
- 43. 16 1 TheOptimization Problem Solution Some background theory on portfolio selection can be found in [10]. As- sume that the amount of resources to be invested is normalized to unity (e.g., 1 million dollars) and let xi represent the return of security i at some specified time in the future, say, in one month’s time. The return xi can be assumed to be a random variable and hence three quantities pertaining to security i can be evaluated, namely, the expected return μi = E[xi ] the variance of the return σ2 i = E[(xi − μi )2 ] and the correlation between the returns of the ith and jth securities ρi, j = E[(xi − μi )(x j − μj )] σi σj for i, j = 1, 2, . . . , n With these quantities known, constructing a portfolio amounts to allocating a fraction wi of the available resources to security i, for i = 1, 2, . . . , n. This leads to the constraints 0 ≤ wi ≤ 1 for i = 1, 2, . . . , n and n i=1 wi = 1 Given a set of investment allocations {wi , i = 1, 2, . . . , n}, the expected return of the portfolio can be deduced as E n i=1 wi xi = n i=1 wi μi The variance for the portfolio, which measures the risk of the investment, can be evaluated as E n i=1 wi xi − E( n i=1 wi xi ) 2 = E ⎧ ⎨ ⎩ n i=1 wi (xi − μi ) ⎡ ⎣ n j=1 wj (x j − μj ) ⎤ ⎦ ⎫ ⎬ ⎭ = E ⎡ ⎣ n i=1 n j=1 (xi − μi )(x j − μj )wi wj ⎤ ⎦ = n i=1 n j=1 E[(xi − μi )(x j − μj )]wi wj = n i=1 n j=1 (σi σj ρi j )wi wj = n i=1 n j=1 qi j wi wj where qi j = σi σj ρi j
- 44. 1.4 Constraints 17 Theportfolio can be optimized in several ways. One possibility would be to minimize the investment risk subject to an acceptable expected return μ∗, namely, minimize wi , 1 ≤ i ≤ n f (w) = n i=1 n j=1 qi j wi wj (1.12a) subject to: n i=1 μi wi ≥ μ∗ (1.12b) wi ≥ 0 for 1 ≤ i ≤ n (1.12c) n i=1 wi = 1 (1.12d) Anotherpossibilitywouldbetomaximizetheexpectedreturnsubjecttoanacceptable risk σ2 ∗ , namely maximize wi , 1 ≤ i ≤ n F(w) = n i=1 μi wi subject to: n i=1 n j=1 qi j wi wj ≤ σ2 ∗ wi ≥ 0 for 1 ≤ i ≤ n n i=1 wi = 1 1.5 The Feasible Region Any point x that satisfies both the equality as well as the inequality constraints is said to be a feasible point of the optimization problem. The set of all points that satisfy the constraints constitutes the feasible domain region of f (x). Evidently, the constraints define a subset of En. Therefore, the feasible region can be defined as a set2 R = {x : ai (x) = 0 for i = 1, 2, . . . , p and cj (x) ≥ 0 for j = 1, 2, . . . , q} where R ⊂ En. The optimum point x∗ must be located in the feasible region, and so the general constrained optimization problem can be stated as minimize f (x) for x ∈ R Any point x not in R is said to be a nonfeasible point. If the constraints in an optimization problem are all inequalities, the constraints divide the points in the En space into three types of points, as follows: 2The above notation for a set will be used consistently throughout the book.
- 45. Exploring the Varietyof Random Documents with Different Content
- 46. Fondie. A Novel.By E. C. Booth, author of “The Cliff End,” “The Doctor’s Lass.” Crown 8vo, 6s. This story of Fondie Bassiemoor, his life, and also that of the inhabitants of Wivvle, a Yorkshire village, is a big novel in every way. Mr. Booth’s unhurried method requires a large canvas, and although the story is long for a modern novel the feeling of the reader at the finish is that on no account would he have it shorter. Mr. Booth pictures the everyday life of a rural village in Yorkshire with all its types and characters clearly and lovingly drawn; the comedy and tragedy of life painted with the sure hand of an artist and master craftsman. The natural tone and accent of speech is reproduced, but there is nothing irritating in its transcription as the author renders the Yorkshire dialect in such manner and so naturally that no unusual effort is required to read it. The note of comedy is preserved through the greater part of the book, but the sadness of life is not ignored. To each their place. The author’s previous books have been unreservedly praised, but it is thought by competent judges that “Fondie” is a particular advance on any of his earlier work. For a comparison one must go to the early work of Thomas Hardy. Perhaps “Far from the Madding Crowd” is the closest. Mr. Booth’s “Fondie” will stand the comparison very well. MARY AGNES HAMILTON Dead Yesterday. By Mary Agnes Hamilton, author of “Less than the Dust,” “Yes.” Crown 8vo, 6s. This novel has been described by critics who have read it in manuscript as both clever and brilliant. It is notably modern in its feeling and outlook, its detail and allusions revealing its author’s interest in the artistic and social ideas which were current in 1914. The action of the story begins before the war, but is carried past August 1914, and finishes towards the end of 1915. It gives a very effective picture of an educated and bohemian coterie whose sophisticated attitude towards life is sharply challenged by the realization of the need to fight for national existence. MILDRED GARNER Harmony. A Novel. By Mildred Garner. Crown 8vo. 6s.
- 47. The scent ofold-fashioned flowers, the drowsy hum of bees, and the quiet spell of the countryside is realized in every page of “Harmony.” Peacewold is a harbour of refuge where gather those in need of the sympathy which the Little Blue Lady unfailingly has for her friends when they are distressed in spirit or body. To her comes Star worn out with months of settlement work in Bethnal Green, and Harmony whose sight is restored after years of blindness. Robin Grey, the austere Richard Wentworth and his son Bede, all come and she gives to each from the fulness of her spirit and faith. Willow, whose story the book is, also has reason to love the Little Blue Lady who has been as a mother to her. The book is distinguished for its shining faith and belief in the inherent goodness of human nature when subject to right influences. The searchings of heart when love comes and temporarily wrecks the harmony of Peacewold are shown to be for the good of those concerned and helpful to them in their development. “Harmony” is essentially a novel of sentiment and should certainly find many readers. It is earnest and sincere, and promises well for the author’s future as a successful novelist. RICHARD HARDING DAVIS Somewhere in France. Stories. By Richard Harding Davis, author of “With the Allies,” etc., etc. Illustrated. Crown 8vo, 3s. 6d. net. A new volume by the popular war correspondent. The stories are varied in theme, and are not solely devoted to war. The title of the book is obtained from the first story which is of spying and spies during the German advance on Paris. LESLIE MONTGOMERY Mr. Wildridge of the Bank. An Irish Novel. By Leslie Montgomery. Crown 8vo, 6s. Mr. Leslie Montgomery will be welcomed as an acquisition to the ranks of humorous novelists. Like George Birmingham he writes of the North of Ireland and shows the everyday life of a small town. The competition of the local managers of the two banks to secure the account of the heir to a fortune is very amusing and always strictly probable. How Mr. Wildridge gets the capital
- 48. subscribed for thewoollen factory: how the confiding Rector and his daughter are saved from dishonour, and how Orangemen and Sinn Feiners, Protestants and Catholics are cunningly induced to work for the prosperity of the town in order to ‘dish’ each other are all related in an easy and convincing way. The story is told in light comedy vein, at times becoming madcap farce, and yet it cannot be said that the bounds of possibility are ever surpassed. There is not an unpleasant or disagreeable character in the book, and the humour is at the expense of everyone in the town. Anthony Wildridge is always cultivated, adroit and audacious, and deserves all his success. At the close he discovers that he is younger and more susceptible than he thought he was. DUCKWORTH CO., Covent Garden, London, W.C.
- 49. Transcriber’s Notes The originalspelling was mostly preserved. A few obvious typographical errors were silently corrected. Further careful corrections, some after consulting other editions, are listed here (before/after): ... to an end just as the picture of the French count ... ... to an end just as the picture of the French court ... ... back to the table so that nobody “poked.” She ... ... backs to the table so that nobody “poked.” She ... ... and canons and things make a frightful noise.” ... ... and cannons and things make a frightful noise.” ... ... That life and such happiness in store for him is ... ... That life had such happiness in store for him is ... ... lounge seemed to have deserted her; and almost ... ... lounge seemed to have deserted her; and almost at ...
- 50. *** END OFTHE PROJECT GUTENBERG EBOOK BACKWATER: PILGRIMAGE, VOLUME 2 *** Updated editions will replace the previous one—the old editions will be renamed. Creating the works from print editions not protected by U.S. copyright law means that no one owns a United States copyright in these works, so the Foundation (and you!) can copy and distribute it in the United States without permission and without paying copyright royalties. Special rules, set forth in the General Terms of Use part of this license, apply to copying and distributing Project Gutenberg™ electronic works to protect the PROJECT GUTENBERG™ concept and trademark. Project Gutenberg is a registered trademark, and may not be used if you charge for an eBook, except by following the terms of the trademark license, including paying royalties for use of the Project Gutenberg trademark. If you do not charge anything for copies of this eBook, complying with the trademark license is very easy. You may use this eBook for nearly any purpose such as creation of derivative works, reports, performances and research. Project Gutenberg eBooks may be modified and printed and given away—you may do practically ANYTHING in the United States with eBooks not protected by U.S. copyright law. Redistribution is subject to the trademark license, especially commercial redistribution. START: FULL LICENSE
- 51. THE FULL PROJECTGUTENBERG LICENSE
- 52. PLEASE READ THISBEFORE YOU DISTRIBUTE OR USE THIS WORK To protect the Project Gutenberg™ mission of promoting the free distribution of electronic works, by using or distributing this work (or any other work associated in any way with the phrase “Project Gutenberg”), you agree to comply with all the terms of the Full Project Gutenberg™ License available with this file or online at www.gutenberg.org/license. Section 1. General Terms of Use and Redistributing Project Gutenberg™ electronic works 1.A. By reading or using any part of this Project Gutenberg™ electronic work, you indicate that you have read, understand, agree to and accept all the terms of this license and intellectual property (trademark/copyright) agreement. If you do not agree to abide by all the terms of this agreement, you must cease using and return or destroy all copies of Project Gutenberg™ electronic works in your possession. If you paid a fee for obtaining a copy of or access to a Project Gutenberg™ electronic work and you do not agree to be bound by the terms of this agreement, you may obtain a refund from the person or entity to whom you paid the fee as set forth in paragraph 1.E.8. 1.B. “Project Gutenberg” is a registered trademark. It may only be used on or associated in any way with an electronic work by people who agree to be bound by the terms of this agreement. There are a few things that you can do with most Project Gutenberg™ electronic works even without complying with the full terms of this agreement. See paragraph 1.C below. There are a lot of things you can do with Project Gutenberg™ electronic works if you follow the terms of this agreement and help preserve free future access to Project Gutenberg™ electronic works. See paragraph 1.E below.
- 53. 1.C. The ProjectGutenberg Literary Archive Foundation (“the Foundation” or PGLAF), owns a compilation copyright in the collection of Project Gutenberg™ electronic works. Nearly all the individual works in the collection are in the public domain in the United States. If an individual work is unprotected by copyright law in the United States and you are located in the United States, we do not claim a right to prevent you from copying, distributing, performing, displaying or creating derivative works based on the work as long as all references to Project Gutenberg are removed. Of course, we hope that you will support the Project Gutenberg™ mission of promoting free access to electronic works by freely sharing Project Gutenberg™ works in compliance with the terms of this agreement for keeping the Project Gutenberg™ name associated with the work. You can easily comply with the terms of this agreement by keeping this work in the same format with its attached full Project Gutenberg™ License when you share it without charge with others. 1.D. The copyright laws of the place where you are located also govern what you can do with this work. Copyright laws in most countries are in a constant state of change. If you are outside the United States, check the laws of your country in addition to the terms of this agreement before downloading, copying, displaying, performing, distributing or creating derivative works based on this work or any other Project Gutenberg™ work. The Foundation makes no representations concerning the copyright status of any work in any country other than the United States. 1.E. Unless you have removed all references to Project Gutenberg: 1.E.1. The following sentence, with active links to, or other immediate access to, the full Project Gutenberg™ License must appear prominently whenever any copy of a Project Gutenberg™ work (any work on which the phrase “Project
- 54. Gutenberg” appears, orwith which the phrase “Project Gutenberg” is associated) is accessed, displayed, performed, viewed, copied or distributed: This eBook is for the use of anyone anywhere in the United States and most other parts of the world at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this eBook or online at www.gutenberg.org. If you are not located in the United States, you will have to check the laws of the country where you are located before using this eBook. 1.E.2. If an individual Project Gutenberg™ electronic work is derived from texts not protected by U.S. copyright law (does not contain a notice indicating that it is posted with permission of the copyright holder), the work can be copied and distributed to anyone in the United States without paying any fees or charges. If you are redistributing or providing access to a work with the phrase “Project Gutenberg” associated with or appearing on the work, you must comply either with the requirements of paragraphs 1.E.1 through 1.E.7 or obtain permission for the use of the work and the Project Gutenberg™ trademark as set forth in paragraphs 1.E.8 or 1.E.9. 1.E.3. If an individual Project Gutenberg™ electronic work is posted with the permission of the copyright holder, your use and distribution must comply with both paragraphs 1.E.1 through 1.E.7 and any additional terms imposed by the copyright holder. Additional terms will be linked to the Project Gutenberg™ License for all works posted with the permission of the copyright holder found at the beginning of this work. 1.E.4. Do not unlink or detach or remove the full Project Gutenberg™ License terms from this work, or any files
- 55. containing a partof this work or any other work associated with Project Gutenberg™. 1.E.5. Do not copy, display, perform, distribute or redistribute this electronic work, or any part of this electronic work, without prominently displaying the sentence set forth in paragraph 1.E.1 with active links or immediate access to the full terms of the Project Gutenberg™ License. 1.E.6. You may convert to and distribute this work in any binary, compressed, marked up, nonproprietary or proprietary form, including any word processing or hypertext form. However, if you provide access to or distribute copies of a Project Gutenberg™ work in a format other than “Plain Vanilla ASCII” or other format used in the official version posted on the official Project Gutenberg™ website (www.gutenberg.org), you must, at no additional cost, fee or expense to the user, provide a copy, a means of exporting a copy, or a means of obtaining a copy upon request, of the work in its original “Plain Vanilla ASCII” or other form. Any alternate format must include the full Project Gutenberg™ License as specified in paragraph 1.E.1. 1.E.7. Do not charge a fee for access to, viewing, displaying, performing, copying or distributing any Project Gutenberg™ works unless you comply with paragraph 1.E.8 or 1.E.9. 1.E.8. You may charge a reasonable fee for copies of or providing access to or distributing Project Gutenberg™ electronic works provided that: • You pay a royalty fee of 20% of the gross profits you derive from the use of Project Gutenberg™ works calculated using the method you already use to calculate your applicable taxes. The fee is owed to the owner of the Project Gutenberg™ trademark, but he has agreed to donate royalties under this paragraph to the Project Gutenberg Literary Archive Foundation. Royalty
- 56. payments must bepaid within 60 days following each date on which you prepare (or are legally required to prepare) your periodic tax returns. Royalty payments should be clearly marked as such and sent to the Project Gutenberg Literary Archive Foundation at the address specified in Section 4, “Information about donations to the Project Gutenberg Literary Archive Foundation.” • You provide a full refund of any money paid by a user who notifies you in writing (or by e-mail) within 30 days of receipt that s/he does not agree to the terms of the full Project Gutenberg™ License. You must require such a user to return or destroy all copies of the works possessed in a physical medium and discontinue all use of and all access to other copies of Project Gutenberg™ works. • You provide, in accordance with paragraph 1.F.3, a full refund of any money paid for a work or a replacement copy, if a defect in the electronic work is discovered and reported to you within 90 days of receipt of the work. • You comply with all other terms of this agreement for free distribution of Project Gutenberg™ works. 1.E.9. If you wish to charge a fee or distribute a Project Gutenberg™ electronic work or group of works on different terms than are set forth in this agreement, you must obtain permission in writing from the Project Gutenberg Literary Archive Foundation, the manager of the Project Gutenberg™ trademark. Contact the Foundation as set forth in Section 3 below. 1.F. 1.F.1. Project Gutenberg volunteers and employees expend considerable effort to identify, do copyright research on, transcribe and proofread works not protected by U.S. copyright
- 57. law in creatingthe Project Gutenberg™ collection. Despite these efforts, Project Gutenberg™ electronic works, and the medium on which they may be stored, may contain “Defects,” such as, but not limited to, incomplete, inaccurate or corrupt data, transcription errors, a copyright or other intellectual property infringement, a defective or damaged disk or other medium, a computer virus, or computer codes that damage or cannot be read by your equipment. 1.F.2. LIMITED WARRANTY, DISCLAIMER OF DAMAGES - Except for the “Right of Replacement or Refund” described in paragraph 1.F.3, the Project Gutenberg Literary Archive Foundation, the owner of the Project Gutenberg™ trademark, and any other party distributing a Project Gutenberg™ electronic work under this agreement, disclaim all liability to you for damages, costs and expenses, including legal fees. YOU AGREE THAT YOU HAVE NO REMEDIES FOR NEGLIGENCE, STRICT LIABILITY, BREACH OF WARRANTY OR BREACH OF CONTRACT EXCEPT THOSE PROVIDED IN PARAGRAPH 1.F.3. YOU AGREE THAT THE FOUNDATION, THE TRADEMARK OWNER, AND ANY DISTRIBUTOR UNDER THIS AGREEMENT WILL NOT BE LIABLE TO YOU FOR ACTUAL, DIRECT, INDIRECT, CONSEQUENTIAL, PUNITIVE OR INCIDENTAL DAMAGES EVEN IF YOU GIVE NOTICE OF THE POSSIBILITY OF SUCH DAMAGE. 1.F.3. LIMITED RIGHT OF REPLACEMENT OR REFUND - If you discover a defect in this electronic work within 90 days of receiving it, you can receive a refund of the money (if any) you paid for it by sending a written explanation to the person you received the work from. If you received the work on a physical medium, you must return the medium with your written explanation. The person or entity that provided you with the defective work may elect to provide a replacement copy in lieu of a refund. If you received the work electronically, the person or entity providing it to you may choose to give you a second opportunity to receive the work electronically in lieu of a refund.
- 58. If the secondcopy is also defective, you may demand a refund in writing without further opportunities to fix the problem. 1.F.4. Except for the limited right of replacement or refund set forth in paragraph 1.F.3, this work is provided to you ‘AS-IS’, WITH NO OTHER WARRANTIES OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO WARRANTIES OF MERCHANTABILITY OR FITNESS FOR ANY PURPOSE. 1.F.5. Some states do not allow disclaimers of certain implied warranties or the exclusion or limitation of certain types of damages. If any disclaimer or limitation set forth in this agreement violates the law of the state applicable to this agreement, the agreement shall be interpreted to make the maximum disclaimer or limitation permitted by the applicable state law. The invalidity or unenforceability of any provision of this agreement shall not void the remaining provisions. 1.F.6. INDEMNITY - You agree to indemnify and hold the Foundation, the trademark owner, any agent or employee of the Foundation, anyone providing copies of Project Gutenberg™ electronic works in accordance with this agreement, and any volunteers associated with the production, promotion and distribution of Project Gutenberg™ electronic works, harmless from all liability, costs and expenses, including legal fees, that arise directly or indirectly from any of the following which you do or cause to occur: (a) distribution of this or any Project Gutenberg™ work, (b) alteration, modification, or additions or deletions to any Project Gutenberg™ work, and (c) any Defect you cause. Section 2. Information about the Mission of Project Gutenberg™
- 59. Project Gutenberg™ issynonymous with the free distribution of electronic works in formats readable by the widest variety of computers including obsolete, old, middle-aged and new computers. It exists because of the efforts of hundreds of volunteers and donations from people in all walks of life. Volunteers and financial support to provide volunteers with the assistance they need are critical to reaching Project Gutenberg™’s goals and ensuring that the Project Gutenberg™ collection will remain freely available for generations to come. In 2001, the Project Gutenberg Literary Archive Foundation was created to provide a secure and permanent future for Project Gutenberg™ and future generations. To learn more about the Project Gutenberg Literary Archive Foundation and how your efforts and donations can help, see Sections 3 and 4 and the Foundation information page at www.gutenberg.org. Section 3. Information about the Project Gutenberg Literary Archive Foundation The Project Gutenberg Literary Archive Foundation is a non- profit 501(c)(3) educational corporation organized under the laws of the state of Mississippi and granted tax exempt status by the Internal Revenue Service. The Foundation’s EIN or federal tax identification number is 64-6221541. Contributions to the Project Gutenberg Literary Archive Foundation are tax deductible to the full extent permitted by U.S. federal laws and your state’s laws. The Foundation’s business office is located at 809 North 1500 West, Salt Lake City, UT 84116, (801) 596-1887. Email contact links and up to date contact information can be found at the Foundation’s website and official page at www.gutenberg.org/contact
- 60. Section 4. Informationabout Donations to the Project Gutenberg Literary Archive Foundation Project Gutenberg™ depends upon and cannot survive without widespread public support and donations to carry out its mission of increasing the number of public domain and licensed works that can be freely distributed in machine-readable form accessible by the widest array of equipment including outdated equipment. Many small donations ($1 to $5,000) are particularly important to maintaining tax exempt status with the IRS. The Foundation is committed to complying with the laws regulating charities and charitable donations in all 50 states of the United States. Compliance requirements are not uniform and it takes a considerable effort, much paperwork and many fees to meet and keep up with these requirements. We do not solicit donations in locations where we have not received written confirmation of compliance. To SEND DONATIONS or determine the status of compliance for any particular state visit www.gutenberg.org/donate. While we cannot and do not solicit contributions from states where we have not met the solicitation requirements, we know of no prohibition against accepting unsolicited donations from donors in such states who approach us with offers to donate. International donations are gratefully accepted, but we cannot make any statements concerning tax treatment of donations received from outside the United States. U.S. laws alone swamp our small staff. Please check the Project Gutenberg web pages for current donation methods and addresses. Donations are accepted in a number of other ways including checks, online payments and
- 61. credit card donations.To donate, please visit: www.gutenberg.org/donate. Section 5. General Information About Project Gutenberg™ electronic works Professor Michael S. Hart was the originator of the Project Gutenberg™ concept of a library of electronic works that could be freely shared with anyone. For forty years, he produced and distributed Project Gutenberg™ eBooks with only a loose network of volunteer support. Project Gutenberg™ eBooks are often created from several printed editions, all of which are confirmed as not protected by copyright in the U.S. unless a copyright notice is included. Thus, we do not necessarily keep eBooks in compliance with any particular paper edition. Most people start at our website which has the main PG search facility: www.gutenberg.org. This website includes information about Project Gutenberg™, including how to make donations to the Project Gutenberg Literary Archive Foundation, how to help produce our new eBooks, and how to subscribe to our email newsletter to hear about new eBooks.