Call Girls Service Nashik Vaishnavi 7001305949 Independent Escort Service Nashik
ย
AOT3 Multivariable Optimization Algorithms.pdf
1. Advanced Optimization Theory
MED573
Multi Variable Optimization Algorithms
Dr. Aditi Sengupta
Department of Mechanical Engineering
IIT (ISM) Dhanbad
Email: aditi@iitism.ac.in
1
2. Introduction
2
โข Methods used to optimize functions having multiple design variables.
โข Some single variable optimization algorithms are used to perform
unidirectional search along desired direction.
โข Broadly classified into two categories:
(i) Direct search methods
(ii) Gradient-based methods
3. โข Optimality criteria differ from single-variable optimization.
โข In multi-variable optimization, the gradient of a function is not a scalar quantity โ it
is a vector quantity.
โข Let us assume that the objective function is a function of N variables represented by
x1, x2, โฆ, xN.
โข The gradient at any point x(t) is represented by โ๐๐ ๐ฅ๐ฅ ๐ก๐ก
which is N-dimensional
vector given by
โ๐๐ ๐ฅ๐ฅ ๐ก๐ก =
๐๐๐๐
๐๐๐ฅ๐ฅ1
,
๐๐๐๐
๐๐๐ฅ๐ฅ2
, โฆ ,
๐๐๐๐
๐๐๐ฅ๐ฅ๐๐
๐๐
at x(t)
3
Optimality Criteria
4. โข The second-order derivatives form a matrix, โ2๐๐(๐ฅ๐ฅ(๐ก๐ก)) known as Hessian matrix given by
โ2๐๐ ๐ฅ๐ฅ ๐ก๐ก =
๐๐2๐๐
๐๐๐ฅ๐ฅ1
2
๐๐2๐๐
๐๐๐ฅ๐ฅ1๐๐๐ฅ๐ฅ2
โฏ
๐๐2๐๐
๐๐๐ฅ๐ฅ1๐๐๐ฅ๐ฅ๐๐
๐๐2๐๐
๐๐๐ฅ๐ฅ1๐๐๐ฅ๐ฅ2
๐๐2๐๐
๐๐๐ฅ๐ฅ2
2 โฏ
๐๐2๐๐
๐๐๐ฅ๐ฅ2๐๐๐ฅ๐ฅ๐๐
โฎ โฎ โฑ โฎ
๐๐2๐๐
๐๐๐ฅ๐ฅ๐๐๐๐๐ฅ๐ฅ1
๐๐2๐๐
๐๐๐ฅ๐ฅ๐๐๐๐๐ฅ๐ฅ2
โฏ
๐๐2๐๐
๐๐๐ฅ๐ฅ๐๐
2
โข Now that we have the derivatives, we are ready to define the optimality criteria.
โข A point ฬ
๐ฅ๐ฅ is a stationary point if โ๐๐ ฬ
๐ฅ๐ฅ = 0.
โข The point is a minimum, a maximum or an inflection point if โ2
๐๐ ฬ
๐ฅ๐ฅ is positive definite,
negative definite or otherwise.
4
Optimality Criteria
5. โข The matrix โ2๐๐(๐ฅ๐ฅ(๐ก๐ก)) is defined to be positive definite if for any point y in the
search space the quantity ๐ฆ๐ฆ๐๐โ2๐๐ ๐ฅ๐ฅ ๐ก๐ก ๐ฆ๐ฆ โฅ 0.
โข The matrix โ2๐๐(๐ฅ๐ฅ(๐ก๐ก)) is defined to be negative definite if for any point y in the
search space the quantity ๐ฆ๐ฆ๐๐
โ2
๐๐ ๐ฅ๐ฅ ๐ก๐ก
๐ฆ๐ฆ โค 0.
โข If at some point y+ in the search space ๐ฆ๐ฆ+๐๐
โ2
๐๐ ๐ฅ๐ฅ ๐ก๐ก
๐ฆ๐ฆ+
โฅ 0 and for some other
point y-, ๐ฆ๐ฆโ๐๐โ2๐๐ ๐ฅ๐ฅ ๐ก๐ก ๐ฆ๐ฆโ โค 0 then the matrix is neither positive definite nor
negative definite.
โข Other test for positive definiteness โ all eigenvalues are positive or if all principal
determinants are positive.
5
Optimality Criteria
6. โข Successive unidirectional search techniques are used to find minimum along a search
direction.
โข Unidirectional search is a one-dimensional search performed by comparing function
values only along a specified direction.
โข Points that lie on a line (in N-dimensional space) passing through x(t) and oriented
along s(t) are considered in the search, expressed as
๐ฅ๐ฅ ๐ผ๐ผ = ๐ฅ๐ฅ(๐ก๐ก)
+ ๐ผ๐ผ๐ ๐ (๐ก๐ก)
Eq. (1)
where ฮฑ is a scalar quantity.
6
Unidirectional Search
7. โข We can rewrite the multivariable objective function in terms of a single
variable ฮฑ by substituting x by x(ฮฑ).
โข Then, using a single variable search method, minimum is found.
โข Once optimum value ฮฑ* is obtained, its corresponding point can also be
found.
7
Unidirectional Search
8. Minimize f(x1, x2) = (x1 โ 10)2 + (x2 โ 10)2
The minimum lies at point (10, 10)T from the contour plot of
the function. Here, function value = 0.
Let the point of interest be x(t) = (2, 1)T and we are interested
in finding minimum and corresponding function value in
search direction s(t) = (2, 5)T.
From right-angled triangle shown in dotted line, the optimal
point obtained is x* = (6.207, 11.517)T.
Let us see if we can obtain the solution by performing
unidirectional search along s(t).
8
Unidirectional Search: Example
10. We will use a bracketing algorithm to enclose optimum
point and then use a single-variable optimization method.
First, let us use bounding phase method. Assume initial
guess of x(0) = 0 and increment ฮ = 0.5.
The bounds for ฮฑ are obtained as (0.5, 3.5) with 6 function
evaluations. The bracketing points are then evaluated as
(3, 3.5)T and (9, 18.5)T.
Next, we use golden section search method to find
optimum point. Let us use a = 0.5 and b = 3.5. We obtain
ฮฑ* = 2.103 as minimum.
Substituting ฮฑ* = 2.1035, x(t) = (2, 1)T and s(t) = (2, 5)T in
Eq. (1), we obtain x* = (6.207, 11.517)T
10
Unidirectional Search: Example
11. โข In single-variable optimization, there are only two search directions a point
can be modified โ either in positive x-direction or negative x-direction.
โข In multi-objective optimization, each variable can be modified either in
positive or negative directions leading to 2N ways of modification.
โข One-variable-at-a-time algorithms cannot usually solve functions having
nonlinear interactions between design variables.
โข Thus, we need to completely eliminate concept of search direction, and
instead manipulate a set of points to create a better set of points (eg.
Simplex search method).
11
Direct Search Methods
12. 12
Simplex Search Method
โข For N variables, (N+1) points are to be used in the
initial simplex.
โข To avoid a zero-volume hypercube for a N-variable
function, (N+1) points in the simplex should not lie
along the same line.
โข At each iteration, the worst point in the simplex (xh)
is found first.
โข Then, a new simplex is formed from the old simplex
by some fixed rules that โsteerโ the search away
from the worst point in the simplex.
Four situations may arise depending on function
values of the simplex.
1. First, the centroid (xc) of all but the worst point
is determined.
2. The worst point in the simplex is reflected about
xc and a new point xr is found.
3. If function value at xr is better than best point of
initial simplex, reflection is considered to have
taken simplex to a โgood regionโ in search space.
13. 13
Simplex Search Method
4. An expansion along the line joining xc to xr is
performed, which is controlled by factor ฮณ.
5. If function value at xr is worse than worst point
of initial simplex, reflection is considered to have
taken simplex to a โbad regionโ in search space.
6. Thus, a contraction along the line joining xc to xr is made, controlled by factor ฮฒ.
7. Finally, if function value at xr is better than the worst point and worse than the next-to-worst
point in the simplex, contraction is made with ฮฒ > 0.
27. First, we need to calculate the first and second-order derivatives for gradient-based methods. This is done using central
difference techniques as:
๏ฟฝ
๐๐๐๐(๐ฅ๐ฅ)
๐๐๐ฅ๐ฅ๐๐ ๐ฅ๐ฅ(๐ก๐ก)
= (f ๐ฅ๐ฅ๐๐
๐ก๐ก
+ โ๐ฅ๐ฅ๐๐
๐ก๐ก
โ f(๐ฅ๐ฅ๐๐
๐ก๐ก
โ โ๐ฅ๐ฅ๐๐
(๐ก๐ก)
))/(2โ๐ฅ๐ฅ๐๐
(๐ก๐ก)
)
๏ฟฝ
๐๐2
๐๐(๐ฅ๐ฅ)
๐๐๐ฅ๐ฅ๐๐
2
๐ฅ๐ฅ(๐ก๐ก)
= (f ๐ฅ๐ฅ๐๐
๐ก๐ก
+ โ๐ฅ๐ฅ๐๐
๐ก๐ก
โ 2f(๐ฅ๐ฅ๐๐
๐ก๐ก
) + f(๐ฅ๐ฅ๐๐
๐ก๐ก
โ โ๐ฅ๐ฅ๐๐
(๐ก๐ก)
))/(โ๐ฅ๐ฅ๐๐
(๐ก๐ก)
)2
๏ฟฝ
๐๐2
๐๐(๐ฅ๐ฅ)
๐๐๐ฅ๐ฅ๐๐ ๐๐๐ฅ๐ฅ๐๐
๐ฅ๐ฅ(๐ก๐ก)
=
f ๐ฅ๐ฅ๐๐
๐ก๐ก
+ โ๐ฅ๐ฅ๐๐
๐ก๐ก
, ๐ฅ๐ฅ๐๐
๐ก๐ก
+ โ๐ฅ๐ฅ๐๐
๐ก๐ก
โ f ๐ฅ๐ฅ๐๐
๐ก๐ก
+ โ๐ฅ๐ฅ๐๐
๐ก๐ก
, ๐ฅ๐ฅ๐๐
๐ก๐ก
โ โ๐ฅ๐ฅ๐๐
๐ก๐ก
+ f ๐ฅ๐ฅ๐๐
๐ก๐ก
โ โ๐ฅ๐ฅ๐๐
๐ก๐ก
, ๐ฅ๐ฅ๐๐
๐ก๐ก
โ โ๐ฅ๐ฅ๐๐
๐ก๐ก
โ f ๐ฅ๐ฅ๐๐
๐ก๐ก
โ โ๐ฅ๐ฅ๐๐
๐ก๐ก
, ๐ฅ๐ฅ๐๐
๐ก๐ก
+ โ๐ฅ๐ฅ๐๐
๐ก๐ก
4โ๐ฅ๐ฅ๐๐
๐ก๐ก
โ๐ฅ๐ฅ๐๐
๐ก๐ก
For the complete first derivative vector of a N-variable function, 2N function evaluations are needed.
For the complete second derivative vector of a N-variable function, 3N function evaluations are needed.
For the complete mixed derivative vector of a N-variable function, 4N function evaluations are needed.
Thus, (2N2 +1) computations are needed for the Hessian matrix.
27
Gradient-Based Methods
28. โข By definition, the first derivative represents direction
of the maximum increase of the function value.
โข To get the minimum, we should be searching along
opposite to first derivative function.
28
Gradient-Based Methods
โข Any search direction d(t) would have smaller function value than that at the current
point x(t).
โข Thus, a search direction d(t) that satisfies the following relation is descent direction.
29. 29
Descent Direction
A search direction d(t) is a descent direction at point x(t) if the condition โ๐๐ ๐ฅ๐ฅ ๐ก๐ก . ๐๐(๐ก๐ก) โค 0 is
satisfied in the vicinity of the point x(t).
30. 30
Cauchyโs Steepest Descent Method
โข The search direction used in Cauchyโs method is the negative of the gradient at any
particular point x(k):
๐ ๐ (๐๐)
= โโ๐๐(๐ฅ๐ฅ ๐๐
)
โข Since this direction gives maximum descent in function values, it is known as steepest
descent method.
โข At every iteration, derivative is calculated at current point and unidirectional search is
performed in direction which is negative to the derivative direction to find minimum
along that direction.
โข The minimum point becomes current point and search is continued from this point.
โข Algorithm continues till gradient converges to sufficiently small quantity.