3. Convex
optimization
solvers
are
not
covered
• Convex
optimization
problem
involved
today
– LP:
perfectly
solved
since
1948;
• Simplex
algorithm(1947);
dual(1947);
dual
simplex
(1954);
Episode
algorithm
(1975,
polynomial
time);
– QP:
solving
difficulty
is
close
with
LP
(for
convex
QP)
• Interior
point
method
,
active
set
method,
conjugate
gradient
method
– SOCP:
‘has
very
efficient
method
to
solve
SOCP’
• Tools/Library
– CVX:
convex
optimization
library
on
Matlab
– CVXOPT:
free
library
on
Python
– Joptimizer:
on
Java
4. Optimization
problem
• Standard
form
• Domain
and
implicit
constraints
• Feasible
– Variable
x
is
feasible:
and
x
satisfice
all
constraints
– Problem
is
feasible:
at
least
exists
one
feasible
variable
x ∈D
5. Optimal
value
and
optimal
solution
• Optimal
value
• Optimal
solution
• Local
optimal
point
6. 4
Results
for
one
optimal
problem
• Not
feasible
• Unbounded
below
• Has
optimal
value,
but
no
optimal
point
• Has
optimal
value,
and
also
optimal
point
f1(x) = ex
7. Convex
optimization
• Stanford
form
• Feasible
set
of
a
convex
optimization
problem
is
convex
set
– Domain
of
convex
function
is
convex
set
– Sublevel
set
of
convex
function
is
convex
set
– Intersection
of
convex
sets
are
convex
set
8. Abstract
convex
optimization
problem
• Convex
optimization:
Minimize
a
convex
function
in
convex
set
• Convert
to
a
equivalent
convex
problem
may
help
solve
9. Local
optima
is
global
optima
– Consider
– We
have:
– We
can
easily
choose
a
small
,
which
contradicts
our
assumption
that
x
is
local
optimal
• E.g.
θ
10. Optimality
criterion
for
differentiable
f0
• X is optimal if and only if it’s feasible and
• It’s easy to prove x is optimal if
above
conditionholds
• Prove
if
x is optimal solution
– X
is
interior
of
feasible
set
(constraints
do
not
work
)
– X
is
in
boundary
of
feasible
set
(constraints
probably
works)
∇f0 (x) = 0
∇f0 (x) ≠ 0
∇f0 (x)T
(y − x) ≥ 0 for all feasible y
we have: f0 (y) ≥ f0 (x)+ ∇f0 (x)T
(y − x)
∇f0 (x)T
(y − x) ≥ 0, ∀y ∈X
11. Optimality
criterion
for
differentiable
f0
• If
x
is
optimal
and
in
the
boundary
(
),
then
∇f0 (x)T
(y − x) ≥ 0, ∀y ∈X
∇f0 (x) ≠ 0
Assume for some y, ∇f0 (x)T
(y − x) < 0
z(t) = ty + (1-t)x, t ∈[0,1]
d
dt
f0 (z(t))|t=0 = ∇f0 (x)T
(y − x) < 0
f0 (z(t)) < f0 (x)
12. Equivalent
convex
problem:
examples
• Two
problems
are
equivalent
if
the
solution
of
one
is
readily
obtained
from
the
the
other
one
– Some
transformations
preserve
convexity,
which
is
nice
• Example:
eliminating
equality
constraints
13. Equivalent
convex
problem:
examples
• Introducing
equality
constraints
• Introducing
slack
variables
for
linear
inequalities
15. Linear
Programming
(LP)
• Definition
– Convex
problem
with
affine
objective
and
constraint
functions
• Geometric
interpretation
– Feasible
set
is
a
polyhedron
– Level
set
is
hyper-‐plane
– Level
set
is
orthogonal
with
c
– Optimal
solution
usually
appear
in
vertex
G ∈Rm×n
,A ∈Rp×n
17. LP
example
(2)
• Chebyshev
center
of
a
polyhedron
B = {xc + u | || u ||2 ≤ r}
ai
T
x ≤ bi for all x ∈B if and only if
sup{ai
T
(xc + u)| || u ||2 ≤ r}=sup{ai
T
xc + || ai ||2|| u ||2 )| || u ||2 ≤ r}=ai
T
xc + r || ai ||2 ≤ bi
18. Linear
fractional
program
• Standard
form
• Transform
to
LP
y =
x
eT
x + f
, z=
1
eT
x + f
if y, z are solution of the new problem, then x = y / z is the solution of original problem
and also have same objective function
if x is solution of the original problem, then y and z are the solution of new problem
and have same objective function cT
y + dz = f0 (x)
19. Quadratic
program
(QP)
• Standard
form
– Minimize
a
convex quadratic
function
in
a
polyhedron
• Geometric
interpretation
– Feasible
set
is
a
polyhedron
– Level
set
is
ellipsoid
– Optimal
solution
usually
appear
in
boundary
P ∈S+
n
20. QP
examples
• Least
squares
with
bound
constraints
• Distance
between
two
polyhedron
minimize || Ax -b ||2
2
= xT
AT
Ax - 2bT
Ax + bT
b
subject to li ≤ xi ≤ ui , i = 1,...,n
AT
A ∈S+
n
Minimize || x1 - x2 ||2
2
, subject to A1x1 ≤ b1, A2x2 ≤ b2
22. Second-‐order
cone
program
i ||⋅||2 is Euclidean norm || y ||2 = y1
2
+...+ yn
2
i constraints are nonlinear, nondifferentiable, convex
Second order cone: {(z,t)| zT
z ≤ t2
,t ≥ 0}
23. SOCP
Example:
Parameter
uncertainty
in
LP
•
• Deterministic
model
– contains
must
hold
for
all
ai ∈εi = {ai + Piu | || u ||≤1}, Pi ∈Rn×n
ai ∈εi
sup{ai
T
x | ai ∈εi } ≤ bi
sup{ai
T
x | ai ∈εi } = ai
T
x + sup{uT
Pi
T
x | ||u ||≤1} = ai
T
x+ || Pi
T
x ||2
(when u =
Pi
T
x
|| Pi
T
x ||2
)
So ai
T
x+ || Pi
T
x ||2 ≤ b, which is SOCP
24. SOCP
Example:
Parameter
uncertainty
in
L
• Stochastic
model
ai is random variable; constraints must hold with probability η
bi − ai
T
x
|| ⋅i
1/2
∑ x ||
≥ Φ−1
(η)
25. Geometric
Programming
• Monomial
function
• Posynomial
function:
sum
of
monomial
function
• Operations
– Monomial
function:
multiple/divide
number;
multiple/divide
another
monomial
function
– Posynomial
function:
multiple/divide
number;
add/multiple/divide
monomial
function;
add
another
posynomial
function
28. Generalized
inequality
constraint
• Convex
problem
with
generalized
inequality
constraints
• Same
properties
are
standard
convex
problem
– Convex
feasible
set;
local
optimal
is
global;
optimality
criterion
for
differentiable
f0
when Ki = R+
n
, it's a normal convex problem
29. Cone
form
problem
• Special
case
with
affine
objective
and
constraints
• SOCP
is
a
special
case
of
cone
form
problem
minimize cT
x
subject to -(Ai x + bi ,ci
T
x + di ) =Ki
0, i = 1,...,m
Fx = g,
in which, Ki = {(y,t) ∈Rni +1
| || y ||2 ≤ t} is a second order cone
30. Review:
Cone
• Cone
• Convex
cone
• Positive
semi-‐definite
cone
• Proper
cone
• Dual
cone
if for any x ∈C and θ ≥ 0, we have θx ∈C, then we say set C is cone
if for any x1,x2 ∈C and θ1,θ2 ≥ 0, always have θ1x1 +θ2x2 ∈C
set of all positive semi - definite matrix : S+
n
= {X ∈Sn
| X = 0}
a cone which is convex, closed, solid, and pointed
dual cone of a cone K : K*
= {y | yT
x ≥ 0 for all x ∈K}
31. Review:
Generalized
inequality
• Define
partial
order
using
proper
cone
K
• Popular
proper
cones
and
corresponding
generalized
inequalities
• Dual
property
x =K y <=> y - x ∈K
x K y <=> y - x ∈int K
K = R+ , then =K is ≤ in R
K = R+
n
, then x =K y equals xi ≤ yi ,i = 1,...,n
K = S+
n
, then X =K Y equals Y - X is PSDM
x =K y iff for any λ =K* 0 and λ ≠ 0, we have λT
x < λT
y
