Global optimization

Optimization methods
Beatriz Penalver Bernabe
07/28/2015

General optimization problem
x = vector of variables
f = objective function
g = constraints (either equalities or inequalities)
X = set of bounds/integrality constraint
𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝑜𝑟 𝑚𝑎𝑥𝑖𝑚𝑖𝑧𝑒 𝑓(𝑥)
𝑠. 𝑡. 𝑔 𝑥 ≤ 0
x ∈ 𝑋

Optimization problems
LINEAR NON-LINEAR
CONTINUOSDISCRETE
LP
IP
NLP
MILP GO
Convex Non-convex
MINLP

Linear Programming (LP)
𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝐶 𝑇 𝑋
𝑠. 𝑡. 𝐴𝑋 − 𝐵 ≤ 0
𝑥𝑖 ≥ 0
Examples:
- Flux balance analysis (FBA) of
metabolic networks
- Flow through networks
Methods:
- SIMPLEX
Solvers:
- CPLEX (freely available from IBM for
academia in R, MATLAB, Python, GAMS)

Flux Balance Analysis
A + B  2C v1
C + B  D v2
Biochemical reactions
r1 r2 rn
A -1 0 … 0
B -1 -1 … 0
C 2 -1 … 0
D 0 1 … -1
……………..
M 0 0 … 1
*
v1
v2
…
…
…
vn
0
0
…
…
…
0
=
Mathematical representation
Z= c1 v1 + c2 v2 + c3 v3 … s.t. additional constrains (e.g., available
metabolites in the media)
Cellular objective
e.g., pyruvate production
e.g., biomass production
e.g., estradiol production
Genome-wide
metabolism
Steady-state assumption

SIMPLEX ALGORITHM
See this link for a simple step procedure of Dantzig’s algorithm
http://www.ams.jhu.edu/~castello/625.414/Handouts/SimplexProcedure.pdf

Integer Programming (IP)
𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝐶 𝑇 𝑌
𝑠. 𝑡. 𝐴𝑌 − 𝐵 ≤ 0
𝑦𝑖 𝑎𝑟𝑒 𝑖𝑛𝑡𝑒𝑔𝑒𝑟𝑠
Examples:
- Capital budgeting (e.g., identification of the
best investments sets subject to their
availability, b)
- Scheduling (e.g., class, faculty and students to
minimize the no-first choice)
Methods:
- Integer cuts (not very much used for IP by
itself)
- Branch and bound (BB)
Solvers:
- CPLEX (freely available from IBM for academia
in R, MATLAB, Python, GAMS)

Branch and Bound (BB)
𝑚𝑎𝑥𝑖𝑚𝑖𝑧𝑒 5𝑥1 + 8𝑥2
𝑠. 𝑡. 𝑥1 +𝑥2 ≤ 6
5𝑥1 + 9𝑥2 ≤ 45
𝑥1, 𝑥2 𝑎𝑟𝑒 𝑖𝑛𝑡𝑒𝑔𝑒𝑟𝑠
1. Relaxation of IP = LP, upper limit IP
2. Branch over one of the variables and
solve the relax LP
3. Determine whether to branch or bound
From http://web.mit.edu/15.053/www/AMP-Chapter-09.pdf

𝑠. 𝑡. 𝑥1 +𝑥2 ≤ 6
5𝑥1 + 9𝑥2 ≤ 45
1. Relaxation of IP = LP, upper limit IP
2. Branch over one of the variables and
solve the relax LP
3. Determine whether to branch or
bound
- Bound if infeasible or less than
previous optimization results

𝑠. 𝑡. 𝑥1 +𝑥2 ≤ 6
5𝑥1 + 9𝑥2 ≤ 45
1. Relaxation of IP = LP,
upper limit IP
2. Branch over one of the
variables and solve the
relax LP
3. Determine whether to
branch or bound

BB algorithm

Using cuts
𝐺𝑖𝑣𝑒𝑛 𝑡ℎ𝑖𝑠 𝑖𝑛𝑒𝑞𝑢𝑎𝑙𝑖𝑡𝑦 2𝑥1 + 3𝑥2 ≤ 4 𝑎𝑛𝑑 𝑥1, 𝑥2 𝑎𝑟𝑒 𝑖𝑛𝑡𝑒𝑔𝑒𝑟𝑠,
𝑎 𝑐𝑢𝑡𝑡𝑖𝑛𝑔 𝑝𝑙𝑎𝑛𝑒 𝑐𝑎𝑛 𝑏𝑒 𝑥1 + 𝑥2 ≤ 1
Gomory cut generation is in the supplementary slides

Mixed-Integer Linear Programming
(MILP)
𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝐷 𝑇 𝑋 + 𝐸 𝑇 𝑌
𝑠. 𝑡. 𝐴𝑋 + 𝐵𝑌 + 𝐶 ≤ 0
𝑦 𝑖
𝑎𝑟𝑒 𝑖𝑛𝑡𝑒𝑔𝑒𝑟𝑠
Examples:
- Facility location ( # of warehouses, transport cost)
- Scheduling batch processes
- Salesman
Methods:
- Branch and bound (BB)
- Branch and cut
Solvers:
- CPLEX (freely available from IBM for academia in R, MATLAB, Python, GAMS)
𝑥𝑖 ≥ 0

Non-Linear Programming (NLP)
𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝑓(𝑥)
𝑠. 𝑡. 𝑔 𝑥 ≤ 0
Examples:
- Minimization of reboiler duty in a distillation
column
- Minimum reactor volume for complex kinetic
processes
Methods:
- The strongly depend on the initial guess and
there are iterative
- KKT conditions to add constraints (see next)
- Variable elimination
Solvers:
- CONOPT, KNITRO, MINOS in GAMS
𝑥𝑖 ≥ 0
ℎ 𝑥 = 0

Unconstrained NLP
Newton’s method is
designed for these
problems (see
Sebastian’s slides)

Constrained NLP
- Karus-Kuhn-Tucker (KKT) Conditions
- KKT points x* can be a
minimum, maximum or
saddle point.
- KKT are used when
the NLP problem has
constraints

Optimization problems
LINEAR NON-LINEAR
CONTINUOSDISCRETE
LP
IP
NLP
MILP GO
ConvexNon-convex
MINLP

Convex functions and sets
Function f(x) is a convex function if for every pair of points in its domain X, the line
segment joining these two points is above the graph of the function, i.e., if for any
x1, x2 ∈ X and any λ ∈ 0,1 we have
𝑓 (𝜆𝑥1 1 − 𝜆 𝑥2 ≤ 𝜆𝑓 𝑥1 + 1 − 𝜆 𝑓(𝑥2)
Convex Non convex

Convex sets
Set C is a convex set if the line segment between any two points in C lies in C, i.e.,
if for any x1, x2 ∈ C and any 𝜃 ∈ 0,1 we have
𝜃𝑥1 + (1 − 𝜃)𝑥2 ∈ 𝐶
.
x1 .
x2
.
x1
.
x2
Convex Non convex

Mixed Integer Convex Non-Linear
Programming (MINLP)
𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝑓(𝑥, 𝑦)
𝑠. 𝑡. 𝑔 𝑥, 𝑦 ≤ 0
Examples:
- Feeding tray in a distillation column
- Modularity maximization
Methods:
- Branch and bound
- Decomposition methods
Solvers:
𝑥𝑖 ≥ 0
𝑦𝑗 𝑎𝑟𝑒 𝑖𝑛𝑡𝑒𝑔𝑒𝑟𝑠ℎ 𝑥, 𝑦 = 0
- SBB, Bonmin , DICOPT, AOA (AIMMS), α-ECP, MINOPT
Convex set

▪ Ravindran and Gupta (1985)
▪ Similar to the MILP branch-and-bound –
tree enumeration
▪ Suitable for problems with reasonable
number of integer variables and the NLP
relaxation is tight
Decomposition
• Outer-Approximation (OA) Duran &
Grossmann (1986)
• Generalized Benders Decomposition (GBD),
Geoffrion (1972)
• Extended Cutting Plane (ECP), Westerlund
and Pettersson (1995)
Solution algorithms for Convex MINLP

Branch and Bound for Convex
MINLP
• Similar to the branch-and-bound for solving MILP
• Difference is that in each node we solve a relaxed NLP problem (relaxing the
integer variables in MINLP); nonlinear program (NLP) relaxation – Lower bound
• Less stringent than other methods
• Successive solution of NLP subproblems
• Advantage: Tight formulation may require one NLP
• Disadvantage: Potentially many NLP subproblems
• Convergence global optimum: Unique solution of NLP (sufficient condition)
𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝑓 𝑥, 𝑦
𝑠. 𝑡. 𝑔 𝑥, 𝑦 ≤ 0
ℎ 𝑥, 𝑦 = 0
𝑥, 𝑦 ≤ 0
Relax NLP
y are continuous variables

General approach for
decomposition methods for MINLP
y are integers

Integer cut
The choice of all 0-1 combinations for 𝑦𝑗, 𝑗 ∈ 𝐽 is feasible, except the ones
for which 𝑦𝑗 = 0, 𝑗 ∈ 𝑁, 𝑦𝑗 = 1, 𝑗 ∈ 𝐵, where N and B are specified
partitions of J

NLP Upper Limit
NLP-U: nonlinear program at fixed yk Upper bound
Solution of NLP-U is a valid upper bound of MINLP solution

Master MILP problem
Develop a equivalent linear representation of the MINLP and apply
relaxation
Assume functions are convex and differentiable

Outer approximation
𝑓 𝑥 ≥ 𝑓 𝑥 𝑘
+ 𝛻𝑓 𝑥 𝑘 𝑇
(x − 𝑥 𝑘
)
𝑔 𝑥 ≥ 𝑔 𝑥 𝑘
+ 𝛻𝑔 𝑥 𝑘 𝑇
(x − 𝑥 𝑘
)
As f(x) and g(x) are convex, their corresponding outer approximations
for a feasible solution 𝑥 𝑘 ∈ 𝑋 can be modeled by
f(x)
Underestimate objective
function z
g(x)
Overestimated of
the feasible region

Outer approximation
Based on the solution of the NLP-U (xk, yk), we add the outer approximation cuts for
f(x) and g(x)

Generalized Benders
decomposition
Consider the Outer-Approximation (xk, yk)
Obtain a linear combination of (1) using the Karush-Kuhn-Tucker multipler μk
Similarly, cuts for infeasible subproblems can be derived

Generalized Benders
decomposition

Extending cutting plane
1. Guess (x0, y0)
2. Find xk, yk from the Master MILP from the Outer
Approximation
3. If 𝑔 𝑥 𝑘 + 𝐵𝑦 𝑘 ≤ 𝜀 => STOP
4. Otherwise k=k+1, linearize function at xk, yk and go
to step 2
No NLP

Convex vs Non-convex MINLP
Convex MINLP
▪ Branch and Bound
▪ Outer Approximation
▪ Generalized Benders Decomposition
▪ Extended Cutting Plane
▪ Major (local) MINLP solvers
Nonconvex MINLP - Global Optimization
▪ Convexification and Reformulation - nonconvex problems are much
more difficult to solve than convex ones
▪ Convex Envelopes
▪ Spatial Branch and Bound
▪ (Global) optimization solvers

Mixed Integer Non-Linear
Programming (MINLP)
𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝑓(𝑥, 𝑦)
𝑠. 𝑡. 𝑔(𝑥, 𝑦) ≤ 0
Examples:
- Modularity maximization
- Minimization of plant cost (fixed and variable) – chemical plant design
Methods:
- Convexification and Reformulation
- Convex Envelopes
- Spatial Branch and Bound
Solvers:
𝑥𝑖 ≥ 0
𝑦𝑗 𝑎𝑟𝑒 𝑖𝑛𝑡𝑒𝑔𝑒𝑟𝑠
- α BB, BARON (Branch and Reduce), OA for nonconvex MINLP, Branch
and Contract

Non-convexities
Effects of nonconvexities
▪ NLP supbroblems may have local
optima
▪ MILP master may cut-off global
optimum
Handling of nonconvexitieis
▪ Rigorous approach (global optimization)
- Replace nonconvex terms
- Using underestimators/convex envelopes
- Solve convex MINLP within spatial branch &
bound
Objective
Multiple minima
0 1
y
x
Global optimum
Cut off!

Convexification and Reformulation
Some nonconvex optimization problems can be reformulated as convex problem
(and solve with the convex optimization algorithms) – Convexification
A common case: Posynominals (ci are non-negative)
𝑓 𝑥 =
𝑖
𝑐𝑖
𝑗
𝑥𝑗
𝛼 𝑖𝑗
𝑥𝑗=𝑒 𝑢 𝑗
Replace the posynomials by an exponential function

Convex Envelopes
Most deterministic global optimization algorithms are based on spatial branch and
bound method
Branch-and-Bound for
MILP
Spatial Branch-and-Bound
for Nonconvex MINLP
Relaxation
Replacing binary/integer
variables as continuous
variables with bounds,
solve LPs
Replacing nonconvex terms
with their convex envelopes,
solve convex (MI)NLP
Branching
Branch binary/integer
variables only
Branch on both integer
variables and continuous
variables (due to the nonconvex
terms)
Complexity
NP-hard, but the worst case
can be bounded by 2n
NP-hard, much more difficult

What are convex envelopes?
Convex
envelop

McCormick envelopes for bilinear
terms
Convex envelopes of bilinear terms
McCormick under and over estimators
Over-estimators
Under-estimators

Other envelopes
If g(x) is twice derivable then the convex relation for g(x):
𝑐𝑜𝑛 𝑔 𝑥 = 𝑔 𝑥 +
𝑖
(𝑥𝑖
𝐿
− 𝑥𝑖)(𝑥𝑖 − 𝑥𝑖
𝑈
)

Step 1: Generate subproblems by branching on continuous
variables (subregions)
Step 2: Compute lower bound from convex MINLP (relaxation)
Step 3: Compute upper bound from local optimal solution to
nonconvex MINLP
Step 4: Continue until tolerance of bounds within tolerance
Spatial Branch and Bound Method
x
y Convex Relaxation
x
y
x
y
x
y
50

Graphic Example: Spatial-Branch-
and-Bound
Objective
Multiple minima
Convex envelop
LB – from MILP
LB
LB
UB = Upper bound
From NLP
LB < UB
LB > UB LB < UB
Global optimum search Branch and bound tree

Examples in Amaral’s lab of
optimization problems
• Modularity maximization
• Sequences of time series
• Non-evolutionary genetic code

Modularity maximization - MIQP

Modularity maximization - MIQP
Objective function
Constraints
Each node, n, belongs to only one module
Each link, l, should belong to the same
module, m, that the two nodes its belong,
n, e
Number of links in each module m
Total degree in each module m
There are more constraints in the paper…

Sequences in time series - MINLP
min
𝑖,𝑗
𝑁,𝑀
(𝑦𝑖,𝑗 − 𝑦𝑗 )2 + 𝑀
𝑠. 𝑡. 𝑦𝑗 = 𝐴𝑗 t 𝑖 -𝐵𝑗
𝑡𝑗−1 ≤ 𝑡𝑗 𝑗 𝑖𝑛 1, 2, … . , 𝑀
𝑀 ∈ [1,2,3, … , 𝑁]
𝐴𝑗 =
𝑐𝑜𝑣(𝑦𝑗, 𝑡𝑗)
𝑉𝑎𝑟 (𝑡𝑗)
𝐵𝑗 = 𝑦𝑗- 𝐴𝑗 𝑡𝑗
𝑡0 < 𝑡𝑗 ≤ 𝑡 𝑁

Non-evolutionary genetic code - IP
min
𝑖,𝑗
𝑁
𝐶𝑖,𝑗 𝜃𝑖,𝑗
𝑠. 𝑡
𝑙
𝑠𝑖,𝑙 = 1 𝑜𝑛𝑒 𝑎𝑚𝑖𝑛𝑜 𝑓𝑜𝑟 𝑒𝑎𝑐ℎ 𝑠𝑒𝑞
𝑖
𝑠𝑖,𝑙 ≤ 42 𝑚𝑢𝑙𝑡𝑖𝑝𝑙𝑒 𝑠𝑒𝑞𝑠 𝑓𝑜𝑟 𝑒𝑎𝑐ℎ 𝑎𝑚𝑖𝑛𝑜
𝑠𝑖,𝑙 ∈ [0,1]

Resources
Take the class ChE345: Process Optimization
GAMS, software for optimization
IBM academic initiative to obtain CPLEX
Some online resources:
1. Fenqi Yu webpage
https://optimization.mccormick.northwestern.edu/index.php/Main_Page
2. MIT optimization
http://web.mit.edu/15.053/www/AMP-Chapter-02.pdf
3. Argonne MINLP optimization
https://wiki.mcs.anl.gov/leyffer/images/b/b2/MINLP-Survey.pdf

Gomory’s cuts
Solution to the LP problem
Re-arranged into a integer and non-integer part

Gomory’s cuts
Normally Gomory’s cuts are not used by themselves as they are not very efficient
due to the large number of cuts to be added

Global optimization

More Related Content

What's hot

Similar to Global optimization

Recently uploaded

Global optimization