Chapter 2-1.pdf

R. Zakhama
Structural Optimisation course
Chapter 2: Mathematical
Programming
Part 1: Unconstrained Optimisation

R. Zakhama 2
Chap2: Mathematical Programming
I. Introduction
I.1 General problem formulation
min ( )
min ( ) min ( )
s.t. ( ) 0
s.t. ( ) 0 s.t. ( ) 0
( ) 0
X
X Z
f X
f X Z
g X
X Z
h X
X Z
X
ϕ
ψ θ
χ χ
χ

 

 
  
≤
 ≤  =
  
=
   ′
∈ ∈
 
  
∈





≤
≤
−
⇔
=
0
)
(
0
)
(
0
)
(
X
h
X
h
X
h
and
with
0
0
)
(
0
)
(

n
R
Y
X
Z
Y
Y
X
X
+
×
=
′






=



≥
=
+
⇔
≤
χ
χ
ψ
ψ

R. Zakhama 3
I.2 Classification of optimisation problems
• Linear programming
Algorithms: Simplex (Dantzig 1947), Interior point (Karmarkar 1984)
• Nonlinear programming
where f,g,h: Rn→ R ∈ C1 or C2
0
s.t.
min T







≥
≤
X
b
AX
X
C
X
0
)
(
0
)
(
s.t.
)
(
min







⊆
∈
=
≤
n
X
R
X
X
h
X
g
X
f
χ

R. Zakhama 4
Definition
Unconstrained optimisation problem
II. Unconstrained problems
Let χ ⊆ Rn , f: χ→ R and
Be(X*)={X∈Rn : DX-X*D≤ ε} where ε ≥ 0
Question: Under what condition on f and χ does the function f attain its
minimum in the set χ?
)
(
)
( *
χ
∈
∀
≤ x
X
f
X
f
X* ∈ χ is said to be global minimum of f over χ, if
)
(
min X
f
X χ
∈
Definition
)
(
)
( *
e
B
X
X
f
X
f ∩
∈
∀
≤ χ
X* ∈ χ is said to be a local minimum of f over χ, if

R. Zakhama 5
f
x
global
local
saddle point
f
x
global
FigII.1. : Global and local minima

R. Zakhama 6
f
x
local minimum x*
x*
• Every global minimum is also a local minimum
• It may not possible to identify a global minimum by finding all local minima
• f does not have a global minimum
FigII.1. : No Global minimum

R. Zakhama 7
II.1 Necessary and Sufficient Conditions for Optimality
What are necessary and sufficient conditions for a local minimum?
• The conditions that must be satisfied at the optimum point are called
necessary
If any point does not satisfy the necessary condition, it cannot be optimum
However, not every point that satisfy the necessary conditions are optimal
• Points satisfying the necessary conditions are called candidate optimum
points.
• If a candidate optimum point satisfies the sufficient condition, then it is
indeed optimum.
• If sufficiency conditions cannot be used or they are not satisfied, we may not
be able to draw any conclusions about the optimality of the candidate point.

R. Zakhama 8
• First order necessary condition for a function of Single Variable:
• Let x* be the minimum point, and investigate its neighbourhood (i.e.,
points x at a small distance d from x*).
• Based on Taylor series expansion (first order)
• Since d is small and can arbitrarily take any sign
NECESSARY CONDITION
0
)
(
)
(
)
( *
≥
∆
=
− x
f
x
f
x
f
0
)
(
'
)
( *
≥
=
∆ d
x
f
x
f
0
)
(
' *
=
x
f

R. Zakhama 9
• Second order sufficient condition for a function of Single
Variable:
• Based on Taylor series expansion (second order)
• If the necessary condition is satisfied, then
•Since d2 is always positive regardless of the sign of d
SUFFICIENT CONDITION
0
)
(
'
'
2
1
)
(
'
)
( 2
*
*
≥
+
=
∆ d
x
f
d
x
f
x
f
0
)
(
'
'
2
1
)
( 2
*
≥
=
∆ d
x
f
x
f
0
)
(
'
' *

x
f

R. Zakhama 10
• Optimality Conditions for Functions of Several Variables:
• The necessary condition
• The sufficient condition

R. Zakhama 11
Definiteness H Nature x*
Positive d. Minimum
Positive semi-d. Valley
Indefinite Saddlepoint
Negative semi-d. Ridge
Negative d. Maximum
)
( Hy
yT
i
λ
0

0
≥
0
≠
0
≤
0

• Stationary point nature summary :

R. Zakhama 12
II.2 Example 1
f(x)=(x-2)2 , f’(2)=0 f(x)=-x2 , f’(0)=0
f’’(2)=2 0 f’’(0)=-2 0
f
x
0
-1 1 2 3 4 5 6
4
0
8
12
16
f
x
-1 0 1
-1
0
FigII.2. : Example 1

R. Zakhama 13
( )
7
-
3x
min
2
R
x∈
II.2 Example 2
• Consider the problem:
• We first find the stationary points (which satisfy f’(x)=0):
• f’’(7/3) = 18 0  x* is a strict local minimum.
• Stationary points are found by solving a nonlinear equation:
• Finding the real roots of g(x) may not be always easy
- Consider the problem to minimise f(x)=x2+ex
- g(x)=2x+ex
- Need an algorithm to find x which satisfies g(x)=0.
3
7
0
)
7
3
(
6
0
(x)
' *
=

=
−

= x
x
f
0
(x)
'
)
( =
≡ f
x
g

R. Zakhama 14
II.3 One-Dimensional Minimisation
 Introduction
• Most n-dimensional search algorithms use one dimensional minimisation to
determine the minimum along a specified direction,
• where xq is the current point dq is the direction vector at that point, and α
is the variable which determines how far one needs to move along dq to
reach minimum.
• The function to be minimised, f(x), can now be expressed in terms of the
variable α.
Minimise f(x) = f(xq + αdq) = f(α)
at minimum α = α∗
• The different categories are:
• Zeroth order methods: use f
• First order methods: use f and the gradient of f
• Second order methods: use f, the gradient of f and the hessian of f
q
q
d
x
x α
+
=

R. Zakhama 15
 Unimodal functions
• Let φ : R→ R
• Consider the problem:
• Let x* be the minimum point of φ(x) and x* ∈ [a,b]
)
(
min x
R
x
φ
∈
Definition
The function φ is said to be unimodal on [a,b], if for a ≤ x1 ≤ x2 ≤ b
φ(x1) φ(x2)  x2 x*,
φ(x2) φ(x1)  x1 x* .

R. Zakhama 16
 Dichotomous Search (zeroth order)
• f is unimodal in the initial interval of uncertainty [a,b].
• Place λ and µ symmetrically, each at distance δ from the mid-point of [a,b].
f
x
λ µ b
a
FigII.3. : Dichotomous search

R. Zakhama 17
• Input: initial interval of uncertainty [a,b].
1. Initialisation: k = 0, ak = a, bk = b, δ(0), ε (final length of uncertainty interval).
2.
3. If f(λk) ≥ f(µk),
let ak+1 = λk , bk+1 = bk go to step 5
4. Otherwise If f(λk) f(µk),
let ak+1 = ak , bk+1 = µk
5. k=k+1
6. if (bk-ak)≤ε , exit
7. else go to step 2
• Output:
δ
µ
δ +
+
=
−
+
=
2
,
2
k
k
k
k
k
k
b
a
b
a
λ
2
* k
k b
a
x
+
=

R. Zakhama 18
Consider:
x* = -2.5652, f(x*) = -56.2626
7
19
6
3
5
4
1
min 2
3
4
−
+
−
− x
x
x
x
x
k ak bk bk-ak
0 -4 0 4
1 -4 -1.98 2.02
2 -3.0001 -1.98 1.0201
3 -3.0001 2.4849 0.5152
10 -2.5669 -2.5626 0.0043
20 -2.5652 -2.5652 4.65e-6
23 -2.5652 -2.5652 5.99e-7
…
…
…
…
…
…
…
…
…
…
…
…

R. Zakhama 19
 Golden Section Search (zeroth order)
• Input: initial interval of uncertainty [a,b].
1. Initialisation: k = 0, ak = a, bk = b, ε(0), τ = 1- 1/s = (3-√5 )/2 and s = Golden
Section Ratio
2. λk =ak + τ (bk - ak), µk =bk - τ (bk - ak)
3. If f(λk) ≥ f(µk),
let ak+1 = λk , bk+1 = bk , λk+1 = µk
µk+1 =bk+1 - τ (bk+1 – ak+1) , go to step 5
4. Otherwise If f(λk) ≤ f(µk),
let ak+1 = ak , bk+1 = µ k , µk+1 = λk
λk+1 =ak+1 + τ (bk+1 – ak+1)
5. k=k+1
6. if (bk-ak)≤ε , exit
7. else go to step 3
• Output:
2
* k
k b
a
x
+
=

R. Zakhama 20
 Bisection Search (first order)
• f ∈ C1.
• f is unimodal in the initial interval of uncertainty [a,b].
Idea: Compute f’(c) where c is the midpoint [a,b]
• if f’(c) = 0 then c is a minimum point.
• if f’(c) 0  [a,c] is the new interval of uncertainty.
• if f’(c) 0  [c,b] is the new interval of uncertainty.
c b
a c
a
FigII.4. : Bisection search

R. Zakhama 21
 Newton Method (second order)
• An iterative technique to find a root of a function.
• Problem: Find an approximate root of the function,
• An iteration of Newton’s method on
2
)
( 2
−
= x
x
f
2
)
( 2
−
= x
x
f
f
x
-2
-3 -1 0 1 2 3 4
0
5
10
xk
xk+1
x*
FigII.5. : Newton method

R. Zakhama 22
• Consider the problem to minimise f(x), x ∈ R.
• f ∈ C2.
• The problem of minimising the function f(x) is equivalent to obtaining the
root of the nonlinear equation,
• Based on Taylor series expansion,
• qk+1(x) vanishes,
0
)
(
'
)
( =
≡ x
f
x
q
)
)(
(
)
(
)
(
)
( 1
1
1 k
k
k
k
k
k x
x
x
f
x
f
x
f
x
q −
′
′
+
′
=
′
≡ +
+
+
)
(
)
(
1
k
k
k
k
x
f
x
f
x
x
′
′
′
−
=
+

R. Zakhama 23
• Consider the problem to minimise f(x), f ∈ C2.
• Need to find the roots of
1. Initialisation: Choose initial point x0 , ε 0 , and set k=0
2.
3. k=k+1
4. if |q(xk)| ε , exit
5. else go to step 2.
0
)
(
'
)
( =
≡ x
f
x
q
)
(
)
(
1
k
k
k
k
x
f
x
f
x
x
′
′
′
−
=
+

R. Zakhama 24
• Best convergence of all methods:
• Unless it diverges.
xk
f’
xk+1
xk+2
f’
xk
xk+1 xk+2
FigII.6. : Newton method convergence

R. Zakhama 25
II.4 Multi-Dimensional Minimisation
 Introduction
• Consider the problem to minimise f(x),
• Problem:
)
(
min x
f
n
R
x∈
2
1
2
2
2
1 5
5
min x
x
x
x
n
R
x
−
−
+
∈
x1
x2
0.1
0.2
0.3 0.4
FigII.7. : Contours plot
x1
x2
f

R. Zakhama 26
 Introduction
• Problem:
•
• Necessity condition: g(x) = 0 at x* = (2.5, 2.5)T
• Sufficient condition:
• x* is a strict local minimum
2
1
2
2
2
1 5
5
min x
x
x
x
n
R
x
−
−
+
∈



−
−
=
∇
=
5
2
5
2
)
(
)
(
2
1
x
x
x
f
x
g
definite
positive
is
2
0
0
2
)
( *








=
x
H

R. Zakhama 27
 Introduction
Definition
Let . If there exists a direction and δ 0 such that
for all α∈(0,δ), then d is said to be a descent
direction of f at
n
R
x ∈ n
R
d ∈
)
(
)
( x
f
d
x
f
+α
x
Result
Let f∈C1 and . Let . If then,
d is a descent direction of f at
n
R
x ∈ )
(
)
( x
f
x
g ∇
=
x
0
)
(
d
x
g T

R. Zakhama 28
 Introduction
FigII.8. : Descent direction
)}
(
)
(
:
{ 0
x
f
x
f
x
S =
=
)}
(
)
(
:
{ 0
x
f
x
f
x
S
=
)
( 0
x
g
0
x
}
0
)
(
)
(
:
{
at
of
ion
approximat
order
First
0
0
0
=
− x
x
x
g
x
x
S
T
0
d
0
d

R. Zakhama 29
 Introduction
(1) Initialise: x0 , k = 0
(2) while stopping condition is not satisfied at xk
(a) Find xk+1 such that f(xk+1) f(xk)
(b) k = k+1
endwhile
Output: x* = xk , a local minimum of f(x).
• How to find xk+1 in step 2(a) of the algorithm?
• Which stopping condition can be used?
• Does the algorithm converge? If yes how fast does it converge?
• Does the convergence and its speed depend on x0?

R. Zakhama 30
 Stopping conditions
• Stopping conditions for a minimisation problem:
Practical stopping conditions
•
•
•
te
semidefini
positive
is
)
(
and
0
)
( k
x
H
x
g =
ε
≤
)
(x
g
( )
)
(
1
)
( k
x
f
x
g +
≤ ε
ε
≤
− +
)
(
)
(
)
( 1
k
k
k
x
f
x
f
x
f

R. Zakhama 31
 Speed of convergence
We say the convergence is:
• Linear, if there exists a real τ ∈ ]0,1[, such that for all k≥1 we have:
• Superlinear, if
• Quadratic, if there exists a constant C 0, such that for all k≥1 we have:
*
*
1
x
x
x
x k
k
−
≤
−
+
τ
0
/ *
*
1
→
−
−
+
x
x
x
x k
k
2
*
*
1
x
x
C
x
x k
k
−
≤
−
+

R. Zakhama 32
 Steepest descent method (first order)
• Basic principle is to minimise the N-dimensional function by a series of 1D
line-minimisations:
• The steepest descent method chooses dk to be parallel to the gradient:
• Step-size αk is chosen to minimise f(xk + αkdk).
d
α
x
x k
k
k
k
+
=
+1
)
( k
k
x
f
d −∇
=
)
(
min
arg
0
k
k
k
d
x
f α
α
α
+
=

R. Zakhama 33
(1) Initialise: x0 , ε 0 and k = 0
(2) while
(a) dk = -g(xk)
(b) Find αk along dk such that
(c)
(d) k = k+1
endwhile
Output: x* = xk , a stationary point of f(x).
ε

)
( k
x
g
d
α
x
x k
k
k
k
+
=
+1
)
(
min
arg
0
k
k
k
d
x
f α
α
α
+
=

R. Zakhama 34
• The gradient is everywhere perpendicular to the contour lines.
• After each line minimisation the new gradient is always orthogonal to the
previous step direction (true of any line minimisation).
• Consequently, the iterates tend to zig-zag down the valley in a very inefficient
manner.
FigII.9. : Steepest descent method

R. Zakhama 35
 Conjugate gradient method (first order)
• This method is used for the quadratic function in the form:
where H(n,n) is a symmetric matrix positive definite, b ∈ Rn and c ∈ R
• This method converges in n iterations
• Each dk is chosen to be conjugate to all previous search directions with
respect to the Hessian H:
iables
design var
,
2
1
)
( n
c
x
b
Hx
x
x
f T
T
+
+
=
Definition
Let be a symmetric matrix. The vectors {d0,d1,…,dn-1} are
said to be H- conjugate if they are linearly independent and
n
n
R
H ×
∈
j
i
Hd
d j
T
i
≠
∀
= 0
j
i
Hd
d j
T
i
≠
∀
= 0

R. Zakhama 36
 Conjugate gradient method (first order)
(1) Initialise: x0 , ε 0, d0=-g0 and k = 0
(2) while
(a)
(b)
(c)
(d)
(e)
(f) k = k+1
endwhile
Output: x* = xk , a global minimum of f(x).
ε

k
g
k
T
k
k
T
k
k
Hd
d
d
g
=
α
k
k
k
k
d
x
x α
+
=
+1
b
Hx
g k
k
+
= +
+ 1
1
k
T
k
k
T
k
k
Hd
d
Hd
g 1
+
=
β
k
k
k
k
d
g
d β
+
−
= +
+ 1
1

R. Zakhama 37
 Fletcher-Reeves’ Conjugate gradient method (first order)
• For non-quadratic function, we know that any function can be approximated
locally by a quadratic function using a second order Taylor expansion.
• The Fletcher-Reeves method is an adaptation of the conjugate gradient
method for any non-quadratic function. The differences are:
- Find αk using a 1D search algorithm.
- Calculate βk such that:
k
T
k
k
T
k
k
g
g
g
g 1
1 +
+
=
β

R. Zakhama 38
 Fletcher-Reeves’ Conjugate gradient method (first order)
(1) Initialise: x0 , ε 0, d0=-g0 and k = 0
(2) while
(a)
(b)
(c) Compute
(d)
(e)
(f) k = k+1
endwhile
ε

k
g
)
(
min
arg
0
k
k
k
d
x
f α
α
α
+
=

k
k
k
k
d
x
x α
+
=
+1
1
+
k
g
k
T
k
k
T
k
k
g
g
g
g 1
+
=
β
k
k
k
k
d
g
d β
+
−
= +
+ 1
1

R. Zakhama 39
 Newton method (second order)
• Concept:
– Construct local quadratic approximation
– Minimise approximation
– Repeat
• Local approximation: 2nd order Taylor series of f at xk:
• First order necessary condition:
• is on the form,
• Classical Newton Method:
- Newton Direction:
- Step Length:
• Is dk a descent direction?
)
(
)
(
2
1
)
(
)
(
)
(
)
( k
k
T
k
k
T
k
k
q x
x
H
x
x
x
x
g
x
f
x
f
x
f −
−
+
−
+
=
≈
)
invertible
is
(assuming
)
(
0
)
( 1
1 k
k
k
k
k
q H
g
H
x
x
x
f −
+
−
=

=
∇
)
( 1
1 k
k
k
k
g
H
x
x −
+
−
= 1 k
k
k
k
d
x
x α
+
=
+
)
( 1 k
k
k
g
H
d −
−
=
1
=
k
α

R. Zakhama 40
• If f(x) is quadratic, then the solution is found in one step.
• The method has quadratic convergence (as in the 1D case).
• Risk of divergence, if the function f(x) non convex.
• To avoid divergence → Line search:
- Newton Direction:
- Update:
• If H=I then this reduces to steepest descent.
1 k
k
k
k
d
x
x α
+
=
+
FigII.10. : Line Search.
)
( 1 k
k
k
g
H
d −
−
=
1
+
k
x
k
x
k
ks
α d
d

R. Zakhama 41
(1) Initialise: x0 , ε 0 and k = 0
(2) while
(a)
(b)
(c)
(f) k = k+1
endwhile
ε

k
g
k
k
k
k
d
x
x α
+
=
+1
k
k
k
g
H
d 1
)
( −
−
=
)
(
min
arg
0
k
k
k
d
x
f α
α
α
+
=

R. Zakhama 42
 Comparison
• Problem: minimisation of Rosenbrock’s function
•
• Necessity condition: g(x) = 0 at x* = (1, 1)T
• Sufficient condition:
• x* is a strict local minimum
2
1
2
2
1
2 )
1
(
)
(
100
min
2
x
x
x
R
x
−
+
−
∈





−
−
−
−
−
=
∇
=
)
(
200
)
1
(
2
)
(
400
)
(
)
( 2
1
2
1
2
1
2
1
x
x
x
x
x
x
x
f
x
g
definite
positive
is
200
400
400
802
)
( *








−
−
=
x
H

R. Zakhama 43
Minimum at [1, 1]T
 Comparison
FigII.11. : Rosenbrock’s function.

R. Zakhama 44
 Comparison
FRCG Newton
SD
FigII.12. : Comparison

R. Zakhama 45
III. Problems
III.1 Problem 1
Consider the constrained minimisation problem
Minimise F = x2
Subject to
g1 = 1-x ≤ 0
g2 =x-2 ≤ 0
This represents the minimisation of x2 in the interval 1 ≤ x ≤ 2.
1 – Plot the function F, g1 and g2 on the interval 0 ≤ x ≤ 3.
2 – We can convert this to an equivalent unconstrained problem by using penalty
parameters. To do this we minimise the following unconstrained function:
Plot on the interval 0 ≤ x ≤ 3 for values of R = 5.0 and R = 0.5.
3 – For a value of R = 5.0 in part (2), and beginning with xl = 1.001 and xu = 1.999,
perform several iterations of the golden section method. Reduce the interval of
uncertainty to less than 0.1 in order to minimise the unconstrained function .








+
−
=
2
1
1
1
~
g
g
R
F
F
F
~
F
~

R. Zakhama 46
III. Problems
III.2 Problem 2
Given the function
1- Calculate {∇F} and [H] at the points {X1} = {0,0}, {X2} = {1,1} and {X3}T = {2,2}.
Check if [H] is positive definite at each of these points. Are the optimality conditions for
relative minimum satisfied at any of the points?
2- Apply three iterations of the method of steepest descent to this problem using {X1} as
initial point. .
5
2
2 1
2
1
2
2
2
1
2
4
1 +
−
+
+
−
= x
x
x
x
x
x
F

R. Zakhama 47
Sources
Presentation based on material from:
• Haftka Gürdal (1994): Elements of Structural Optimization
• van Keulen: TUD Optimization course
• Etman (2006): EM course Engineering Optimization

Chapter 2-1.pdf

Recommended

Recommended

More Related Content

Similar to Chapter 2-1.pdf

Similar to Chapter 2-1.pdf (20)

Recently uploaded

Recently uploaded (20)

Chapter 2-1.pdf