Main

LION11 1 / 40
The use of grossone in optimization: a survey and some
recent results
R. De Leone
School of Science and Technology
Universit`a di Camerino
June 2017

Outline of the talk
Outline of the talk
Single and Multi
Objective Linear
Programming
Nonlinear Optimization
Some recent results
LION11 2 / 40
Single and Multi Objective Linear Programming
Some recent results

Single and Multi Objective
Linear Programming
Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
Lexicographic rule and
RHS perturbation
RHS perturbation and
①
Lexicographic
multi-objective Linear
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
Some recent resultsLION11 3 / 40

Linear Programming and the Simplex Method
Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
min
x
cT x
subject to Ax = b
x ≥ 0
The simplex method proposed by George Dantzig in 1947
■ starts at a corner point (a Basic Feasible Solution, BFS)
■ veriﬁes if the current point is optimal
■ if not, moves along an edge to a new corner point
until the optimal corner point is identiﬁed or it discovers that the problem
has no solution.

Preliminary results and notations
Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
Let
X = {x ∈ IRn
: Ax = b, x ≥ 0}
where A ∈ IRm×n, b ∈ IRm, m ≤ n.
A point ¯x ∈ X is a Basic Feasible Solution (BFS) iff the columns of A
corresponding to positive components of ¯x are linearly independent.

Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
Let
X = {x ∈ IRn
: Ax = b, x ≥ 0}
Let ¯x be a BFS and deﬁne ¯I = I(¯x) := {j : ¯xj > 0} then
rank(A.¯I) = |¯I|. Note: |¯I| ≤ m

Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
Let
X = {x ∈ IRn
: Ax = b, x ≥ 0}
Let ¯x be a BFS and deﬁne ¯I = I(¯x) := {j : ¯xj > 0} then
rank(A.¯I) = |¯I|. Note: |¯I| ≤ m
Vertex Point, Extreme Points and Basic Feasible Solution Point coin-
cide
BFS ≡ Vertex ≡ Extreme Point

BFS and associated basis
Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
Assume now
rank(A) = m ≤ n
A base B is a subset of m linearly independent columns of A.
B ⊆ {1, . . . , n} , det(A.B) = 0
N = {1, . . . , n} − B
Let ¯x be a BFS. .

Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
Assume now
rank(A) = m ≤ n
B ⊆ {1, . . . , n} , det(A.B) = 0
N = {1, . . . , n} − B
Let ¯x be a BFS. .
If |{j : ¯xj > 0}| = m the BFS is said to be non–degenerate and
there is only a single base B := {j : ¯xj > 0} associated to ¯x
Non-degenerate BFS

Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
Assume now
rank(A) = m ≤ n
B ⊆ {1, . . . , n} , det(A.B) = 0
N = {1, . . . , n} − B
Let ¯x be a BFS. .
If |{j : ¯xj > 0}| < m the BFS is said to be degenerate and
there are more than one base B1, B2, . . . , Bl associated to ¯x with
{j : ¯xj > 0} ⊆ Bi
Degenerate BFS

Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
Assume now
rank(A) = m ≤ n
B ⊆ {1, . . . , n} , det(A.B) = 0
N = {1, . . . , n} − B
Let ¯x be a BFS. Let B a base associated to ¯x.
Then
¯xN = 0, ¯xB = A−1
.B b ≥ 0

Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
Assume now
rank(A) = m ≤ n
B ⊆ {1, . . . , n} , det(A.B) = 0
N = {1, . . . , n} − B
Let ¯x be a BFS. Let B a base associated to ¯x.

Convergence of the Simplex Method
Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
Convergence of the simplex method is ensured if all basis visited by the
method are nondegenerate

Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
In presence of degenerate BFS, the Simplex method may not terminate
(cycling)

Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
In presence of degenerate BFS, the Simplex method may not terminate
(cycling)
⇓
Hence, speciﬁc anti-cycling procedures must be implemented (Bland’s
rule, lexicographic order)

Lexicographic Rule
Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
At each iteration of the simplex method we choose the leaving variable
using the lexicographic rule

Lexicographic Rule
Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
Let B0 be the initial base and N0 = {1, . . . , n} − B0.
We can always assume, after columns reordering, that A has the form
A = A.Bo
... A.No

Lexicographic Rule
Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
Let
¯ρ = min
i: ¯Aijr >0
(A.−1
B b)i
¯Aijr
if such minimum value is reached in only one index this is the leaving
variable.
OTHERWISE

Lexicographic Rule
Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
Among the indices i for which
min
i: ¯Aijr >0
(A.−1
B b)i
¯Aijr
= ¯ρ
we choose the index for which
min
i: ¯Aijr >0
(A.−1
B A.Bo)i1
¯Aijr
If the minimum is reached by only one index this is the leaving variable.
OTHERWISE

Lexicographic Rule
Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
Among the indices reaching the minimum value, choose the index for which
min
i: ¯Aijr >0
(A.−1
B A.Bo)i2
¯Aijr
Proceed in the same way.
This procedure will terminate providing a single index since the rows of the
matrix (A.−1
B A.Bo) are linearly independent.

Lexicographic rule and RHS perturbation
Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
The procedure outlined in the previous slides is equivalent to perturb each
component of the RHS vector b by a very small quantity.
If this perturbation is small enough, the new linear programming problem is
nondegerate and the simplex method produces exactly the same pivot
sequence as the lexicographic pivot rule
However, is very difﬁcult to determine how small this perturbation must be.
More often a symbolic perturbation is used (with higher computational
costs)

Lexicographic rule and RHS perturbation and ①
Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
Replace bi with bi with
bi +
j∈Bo
Aij①−j
.

Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
bi +
j∈Bo
Aij①−j
.
Let
e =





①−1
①−2
...
①−m





and
b = A.−1
B (b + A.Boe) = A.−1
B b + A.−1
B A.Boe.

Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
bi +
j∈Bo
Aij①−j
.
Therefore b = (A.−1
B b)i +
m
k=1
(A.−1
B A.Bo)ik
①−k
and
min
i: ¯Aijr >0
(A.−1
B b)i +
m
k=1
(A.−1
B A.Bo)ik
①−k
¯Aijr
=
min
i: ¯Aijr >0
(A.−1
B b)i
¯Aijr
+
(A.−1
B A.Bo)i1
¯Aijr
①−1
+ . . . +
(A.−1
B A.Bo)im
¯Aijr
①−m

Lexicographic multi-objective Linear Programming
Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
lexmax
x
c1T
x, c2T
x, . . . , crT x
subject to Ax = b
x ≥ 0
The set
S := {Ax = b, x ≥ 0}
is bounded and non-empty.

Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
lexmax
x
c1T
x, c2T
x, . . . , crT x
subject to Ax = b
x ≥ 0
Preemptive Scheme
Starts considering the ﬁrst objective function alone:
max
x
c1x
subject to Ax = b
x ≥ 0
Let x∗1 be an optimal solution and β1 = c1T
x∗1.

Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
lexmax
x
c1T
x, c2T
x, . . . , crT x
subject to Ax = b
x ≥ 0
Preemptive Scheme
Then solve
max
x
c2T
x
subject to Ax = b
c1T
x = c1T
x∗1
x ≥ 0

Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
lexmax
x
c1T
x, c2T
x, . . . , crT x
subject to Ax = b
x ≥ 0
Preemptive Scheme
Repeat above schema until either the last problem is solved or an unique
solution has been determined.

Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
lexmax
x
c1T
x, c2T
x, . . . , crT x
subject to Ax = b
x ≥ 0
Non–Preemptive Scheme

Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
lexmax
x
c1T
x, c2T
x, . . . , crT x
subject to Ax = b
x ≥ 0
Non–Preemptive Scheme
There always exists a ﬁnite scalar MIR such that the solution of the above
problem can be obtained by solving the one single-objective LP problem
max
x
˜cT x
subject to Ax = b
x ≥ 0
where ˜c =
r
i=1
M−i+1
ci
.

Non–Preemptive grossone-based scheme
Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
Solve the LP
max
x
˜cT x
subject to Ax = b
x ≥ 0
where
˜c =
r
i=1
①−i+1
ci
Note that
˜cT
x = c1T
x ①0
+ c2T
x ①−1
+ . . . crT
x ①r−1

Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
Solve the LP
max
x
˜cT x
subject to Ax = b
x ≥ 0
where
˜c =
r
i=1
①−i+1
ci
Note that
˜cT
x = c1T
x ①0
+ c2T
x ①−1
+ . . . crT
x ①r−1
The main advantage of this scheme is that it does not require the
speciﬁcation of a real scalar value M

Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
Solve the LP
max
x
˜cT x
subject to Ax = b
x ≥ 0
where
˜c =
r
i=1
①−i+1
ci
Note that
˜cT
x = c1T
x ①0
+ c2T
x ①−1
+ . . . crT
x ①r−1
M. Cococcioni, M. Pappalardo, Y.D. Sergeyev

Theoretical results
Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
max
x
˜cT x
subject to Ax = b
x ≥ 0

Theoretical results
Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
max
x
˜cT x
subject to Ax = b
x ≥ 0
If the LP above has solution, there is always a solution that a vertex.
All optimal solutions of the lexicographic problem are feasible for the above
problem and have the objective value.
Any optimal solutions of the lexicographic problem is optimal for the above
problem, and viceversa.

Theoretical results
Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
max
x
˜cT x
subject to Ax = b
x ≥ 0
The dual problem is
min
π
bT π
subject to AT π ≤ ˜c
If ¯x is feasible for the primal problem and ¯π feasible for the dual problem
˜cT
¯x ≤ bT
¯π

Theoretical results
Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
max
x
˜cT x
subject to Ax = b
x ≥ 0
The dual problem is
min
π
bT π
subject to AT π ≤ ˜c
If x∗ is feasible for the primal problem and π∗ feasible for the dual problem
and
˜cT
x∗
= bT
π∗
x∗ is primal optimal and π∗ dual optimal.

The gross-simplex Algorithm
Outline of the talk
Single and Multi
Objective Linear
Programming
Linear Programming
and the Simplex
Method
Preliminary results
and notations
BFS and associated
basis
Convergence of the
Simplex Method
Lexicographic Rule
RHS perturbation
①
Lexicographic
Programming
Non–Preemptive
grossone-based
scheme
Theoretical results
The gross-simplex
Algorithm
Main issues:
1) Solve
AT
.Bπ = ˜cB
Use LU decomposition of A.B. Note: no divisions by gross-number are
required.
2) Calculate reduced cost vector
¯˜cN = ˜cN − AT
.N π
Also in this case only multiplications and additions of gross-numbers are
required.
¯˜cN =
7.331 ①−1
+ 0.331 ①−2
4 0 − 3.331 ①−1
− 0.33 ①−2
¯˜cN =
3.67 ①−1
0.17 ①−2
4 ①0
+ 0.33 ①−1
− 0.17 ①−2

Outline of the talk
Single and Multi
Objective Linear
Programming
The case of Equality
Constraints
Penalty Functions
Exactness of a Penalty
Function
Introducing ①
Convergence Results
Example 1
Example 2
Inequality Constraints
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 15 / 40

The case of Equality Constraints
Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 16 / 40
min
x
f(x)
subject to h(x) = 0
where f : IRn → IR and h : IRn → IRk
L(x, π) := f(x) +
k
j=1
πjhj(x) = f(x) + πT
h(x)

Penalty Functions
Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 17 / 40
A penalty function P : IRn → IR satisﬁes the following condition
P(x)
= 0 if x belongs to the feasible region
> 0 otherwise

Penalty Functions
Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 17 / 40
A penalty function P : IRn → IR satisﬁes the following condition
P(x)
= 0 if x belongs to the feasible region
> 0 otherwise
P(x) =
k
j=1
|hj(x)|
P(x) =
k
j=1
h2
j (x)

Exactness of a Penalty Function
Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modified LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 18 / 40
The optimal solution of the constrained problem
min
x
f(x)
subject to h(x) = 0
can be obtained by solving the following unconstrained minimization
problem
min f(x) +
1
σ
P(x)
for sufficiently small but fixed σ > 0.

Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 18 / 40
min
x
f(x)
subject to h(x) = 0
problem
min f(x) +
1
σ
P(x)
P(x) =
k
j=1
|hj(x)|

Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 18 / 40
min
x
f(x)
subject to h(x) = 0
problem
min f(x) +
1
σ
P(x)
P(x) =
k
j=1
|hj(x)|
Non–smooth function!

Introducing ①
Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 19 / 40
Let
P(x) =
k
j=1
h2
j (x)
Solve
min f(x) + ①P(x) =: φ (x, ①)

Convergence Results
Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 20 / 40
min
x
f(x)
subject to h(x) = 0
(1)

Convergence Results
Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 20 / 40
min
x
f(x)
subject to h(x) = 0
(1)
min
x
f(x) +
1
2
① h(x) 2
(2)

Convergence Results
Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 20 / 40
min
x
f(x)
subject to h(x) = 0
(1)
min
x
f(x) +
1
2
① h(x) 2
(2)
Let
x∗
= x∗0
+ ①−1
x∗1
+ ①−2
x∗2
+ . . .
be a stationary point for (2) and assume that the LICQ condition holds at
x∗0
then

Convergence Results
Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 20 / 40
min
x
f(x)
subject to h(x) = 0
(1)
min
x
f(x) +
1
2
① h(x) 2
(2)
Let
x∗
= x∗0
+ ①−1
x∗1
+ ①−2
x∗2
+ . . .
be a stationary point for (2) and assume that the LICQ condition holds at
x∗0
then
the pair x∗0, π∗ = h(1)(x∗) is a KKT point of (1).

Example 1
Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 21 / 40
min
x
1
2x2
1 + 1
6 x2
2
subject to x1 + x2 = 1
The pair (x∗, π∗) with x∗ =


1
4
3
4

, π∗ = −1
4 is a KKT point.

Example 1
Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 21 / 40
min
x
1
2x2
1 + 1
6 x2
2
subject to x1 + x2 = 1
The pair (x∗, π∗) with x∗ =


1
4
3
4

, π∗ = −1
4 is a KKT point.
f(x) + ①P(x) =
1
2
x2
1 +
1
6
x2
2 +
1
2
①(1 − x1 − x2)2

Example 1
Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 21 / 40
f(x) + ①P(x) =
1
2
x2
1 +
1
6
x2
2 +
1
2
①(1 − x1 − x2)2
First Order Optimality Condition
x1 + ①(x1 + x2 − 1) = 0
1
3 x2 + ①(x1 + x2 − 1) = 0

Example 1
Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 21 / 40
f(x) + ①P(x) =
1
2
x2
1 +
1
6
x2
2 +
1
2
①(1 − x1 − x2)2
x∗
1 =
1①
1 + 4①
, x∗
2 =
3①
1 + 4①

Example 1
Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 21 / 40
f(x) + ①P(x) =
1
2
x2
1 +
1
6
x2
2 +
1
2
①(1 − x1 − x2)2
x∗
1 =
1①
1 + 4①
, x∗
2 =
3①
1 + 4①
x∗
1 =
1
4
− ①−1
(
1
16
−
1
64
①−1
. . .)
x∗
2 =
3
4
− ①−1
(
3
16
−
3
64
①−1
. . .)

Example 1
Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 21 / 40
f(x) + ①P(x) =
1
2
x2
1 +
1
6
x2
2 +
1
2
①(1 − x1 − x2)2
x∗
1 =
1①
1 + 4①
, x∗
2 =
3①
1 + 4①
x∗
1 + x∗
2 − 1 =
1
4
−
1
16
①−1
+
1
64
①−2
. . .
+
3
4
−
3
16
①−1
+
3
64
①−2
. . . − 1
= −
4
16
①−1
−
3
16
①−1
+
4
64
①−2
. . .
and h(1)(x∗) = −1
4 = π∗

Example 2
Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 22 / 40
min x1 + x2
subject to x2
1 + x2
2 − 2 = 0
L(x, π) = x1 + x2 + π x2
1 + x2
2 − 2
The optimal solution is x∗ =
−1
−1
and the pair x∗, π∗ = 1
2 satisﬁes
the KKT conditions.

Example 2
Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 22 / 40
φ (x, ①) = x1 + x2 +
①
2
x2
1 + x2
2 − 2
2
First–Order Optimality Conditions



x1 + 2①x1 x2
1 + x2
2 − 2
2
= 0
x2 + 2①x2 x2
1 + x2
2 − 2
2
= 0
The solution is given by



x1 = −1 − ①−1 1
8 + ①−2
C
x2 = −1 − ①−1 1
8 + ①−2
C

Example 2
Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 22 / 40
Moreover
x2
1 + x2
2 − 2 = 1 +
1
64
①−2
+ ①−4
C2 1
4
①−1
− 2①−2
−
1
4
①−3
C +
1 +
1
64
①−2
+ ①−4
C2 1
4
①−1
− 2①−2
−
1
4
①−3
C
=
1
2
①−1
+
1
32
− 4C ①−2
+ −
1
2
C ①−3
+ −2C2

Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 23 / 40
min
x
f(x)
subject to g(x) ≤ 0
h(x) = 0
where f : IRn → IR, g : IRn → IRm h : IRn → IRk.
L(x, π, µ) := f(x) +
m
i=1
µigi(x) +
k
j=1
πjhj(x)
= f(x) + µT
g(x) + πT
h(x)

Modified LICQ condition
Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modified LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 24 / 40
Let x0 ∈ IRn. The Modified LICQ (MLICQ) condition is said to hold at x0 if
the vectors
∇gi(x0
), i : gi(x0
) ≥ 0, ∇hj(x0
), j = 1, . . . , k
are linearly independent.

Convergence Results
Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 25 / 40
min
x
f(x)
h(x) = 0
min
x
f(x) +
①
2
max{0, gi(x)} 2
+
①
2
h(x) 2
x∗
= x∗0
+ ①−1
x∗1
+ ①−2
x∗2
+ . . .
⇓ (MLICQ)

Convergence Results
Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 25 / 40
min
x
f(x)
h(x) = 0
min
x
f(x) +
①
2
max{0, gi(x)} 2
+
①
2
h(x) 2
x∗
= x∗0
+ ①−1
x∗1
+ ①−2
x∗2
+ . . .
⇓ (MLICQ)
x∗0
, µ∗
= g(1)
(x∗
), π∗
= h(1)
(x∗
)

The importance of CQs
Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 26 / 40
min x1 + x2
subject to x2
1 + x2
2 − 2
2
= 0
L(x, π) = x1 + x2 + π x2
1 + x2
2 − 2
2
The optimal solution is x∗ =
−1
−1

Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 26 / 40
φ (x, ①) = x1 + x2 +
①
2
x2
1 + x2
2 − 2
4
First–Order Optimality Conditions



1 + 4①x1 x2
1 + x2
2 − 2
3
= 0
1 + 4①x2 x2
1 + x2
2 − 2
3
= 0

Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 26 / 40
Let the solution of the above system be
x∗
1 = x∗
2 = A + B①−1
+ C①−2
with A, B, and C ∈ IR. Now
4①x∗
1 = 4A① + 4B + 4C①−1
and
1 + 4①x∗
1 (x∗
1)2
+ (x∗
2)2
− 2
3
= 1 + 4A① + 4B + 4C①−1
2A2 − 2 + 2AB①−1
+ D①−2
3
.

Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 26 / 40
1 + 4①x∗
1 (x∗
1)2
+ (x∗
2)2
− 2
3
= 1 + 4A① + 4B + 4C①−1
2A2
− 2 + 2AB①−1
+ D①−2
3
.

Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 26 / 40
1 + 4①x∗
1 (x∗
1)2
+ (x∗
2)2
− 2
3
= 1 + 4A① + 4B + 4C①−1
2A2
− 2 + 2AB①−1
+ D①−2
3
.
If 2A2 − 2 = 0 there is still a term of the order ① unless A = 0.

Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 26 / 40
1 + 4①x∗
1 (x∗
1)2
+ (x∗
2)2
− 2
3
= 1 + 4A① + 4B + 4C①−1
2A2
− 2 + 2AB①−1
+ D①−2
3
.
If 2A2 − 2 = 0 a term ①−1
can be factored out
1 + 4A① + 4B + 4C①−1
①−3
+2AB + D①−1
3
and the ﬁnite term cannot be equal to 0.

Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modified LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 26 / 40
1 + 4①x∗
1 (x∗
1)2
+ (x∗
2)2
− 2
3
= 1 + 4A① + 4B + 4C①−1
2A2
− 2 + 2AB①−1
+ D①−2
3
.
If 2A2 − 2 = 0 a term ①−1
can be factored out
1 + 4A① + 4B + 4C①−1
①−3
+2AB + D①−1
3
and the finite term cannot be equal to 0.
When Constraint Qualification conditions do not hold, the solution
of ∇F(x) = 0 does not provide a KKT pair for the constrained
problem.

Conjugate Gradient Method
Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 27 / 40
Data: Set k = 0, y0 = 0, r0 = b − Ay0.
If r0 = 0, then STOP. Else, set p0 = r0.
Step k: Compute αk = rT
k pk/pT
k Apk,
yk+1 = yk + αkpk,
rk+1 = rk − αkApk.
If rk+1 = 0, then STOP.
Else, set βk =
−rT
k+1Apk
pT
k Apk
=
rk+1
2
|rk
2
, and
pk+1 = rk+1 + βkpk,
k = k + 1.
Go to Step k.

pT
k Apk
Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modified LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 28 / 40
When the matrix A is positive definite then
λm(A) pk
2
≤ pT
k Apk
and pT
k Apk is bounded from below.
If the matrix A is not positive definite, then such a bound does not hold,
being potentially pT
k Apk = 0,.

pT
k Apk
Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modified LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 28 / 40
When the matrix A is positive definite then
λm(A) pk
2
≤ pT
k Apk
and pT
k Apk is bounded from below.
If the matrix A is not positive definite, then such a bound does not hold,
being potentially pT
k Apk = 0,.
R. De Leone, G. Fasano, Y.D. Sergeyev
Use
pT
k Apk = s①
where s = O(①−1
) if the Step k is a non-degenerate CG step, and
s = O(①−2
) if the Step k is a degenerate CG step.

Variable Metric Method for convex nonsmooth optimization
Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 29 / 40
xk+1
= xk
− αk Bk
−1
ξk
a
where ξk
a current aggregate subgradient, and the positive deﬁnite
variable-metric n × n matrix, approximation of the Hessian matrix.
Then
Bk+1
= Bk
+ ∆k
and
Bk+1
δk
≈ γk
with γk = gk+1 − gk (subgradients) and δk = xk+1 − xk and diagonal.
The focus on the updating technique of matrix Bk

Matrix Updating scheme
Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 30 / 40
min
b
Bδk − γk
subject to Bii ≥ ǫ
Bij = 0, i = j

Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 30 / 40
min
b
Bδk − γk
Bij = 0, i = j
Bk+1
ii = max ǫ,
γk
i
δk
1

Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 30 / 40
min
b
Bδk − γk
Bij = 0, i = j
M. Gaudioso, G. Giallombardo, M. Mukhametzhanov
¯γk
i =
γk
i if |γk
i | > ǫ
①−1
otherwise
¯δk
i =
δk
i if |δk
i | > ǫ
①−1
otherwise

Outline of the talk
Single and Multi
Objective Linear
Programming
Constraints
Penalty Functions
Function
Introducing ①
Convergence Results
Example 1
Example 2
Modiﬁed LICQ
condition
Convergence Results
The importance of
CQs
Conjugate Gradient
Method
pT
k Apk
Variable Metric
Method for convex
nonsmooth
optimization
Matrix Updating
scheme
LION11 30 / 40
min
b
Bδk − γk
Bij = 0, i = j
bk
i =



①−1
if 0 <
¯γk
i
¯δk
i
≤ ǫ
¯γk
i
¯δk
i
otherwise
Bk+1
ii = max ①−1
, bk
i

Some recent results
Outline of the talk
Single and Multi
Objective Linear
Programming
Some recent results
Quadratic Problems
∇F (x) = 0
A Generic Algorithm
Convergence
Gradient Method
Example A
Example B
Conclusions (?)
LION11 31 / 40

Quadratic Problems
Outline of the talk
Single and Multi
Objective Linear
Programming
Some recent results
Quadratic Problems
∇F (x) = 0
A Generic Algorithm
Convergence
Gradient Method
Example A
Example B
Conclusions (?)
LION11 32 / 40
min
x
1
2xT Mx
subject to Ax = b
x ≥ 0

Quadratic Problems
Outline of the talk
Single and Multi
Objective Linear
Programming
Some recent results
Quadratic Problems
∇F (x) = 0
A Generic Algorithm
Convergence
Gradient Method
Example A
Example B
Conclusions (?)
LION11 32 / 40
min
x
1
2xT Mx
subject to Ax = b
x ≥ 0
KKT conditions
Mx + q − AT
u − v = 0
Ax − b = 0
x ≥ 0, v ≥ 0, xT
v = 0

Quadratic Problems
Outline of the talk
Single and Multi
Objective Linear
Programming
Some recent results
Quadratic Problems
∇F (x) = 0
A Generic Algorithm
Convergence
Gradient Method
Example A
Example B
Conclusions (?)
LION11 32 / 40
min
x
1
2xT Mx
subject to Ax = b
x ≥ 0
min
1
2
xT
Mx +
①
2
Ax − b 2
2 +
①
2
max{0, −x} 2
2 =: F(x)

Quadratic Problems
Outline of the talk
Single and Multi
Objective Linear
Programming
Some recent results
Quadratic Problems
∇F (x) = 0
A Generic Algorithm
Convergence
Gradient Method
Example A
Example B
Conclusions (?)
LION11 32 / 40
min
x
1
2xT Mx
subject to Ax = b
x ≥ 0
min
1
2
xT
Mx +
①
2
Ax − b 2
2 +
①
2
max{0, −x} 2
2 =: F(x)
∇F(x) = Mx + q + ①AT
(Ax − b) − ① max{0, −x}

Quadratic Problems
Outline of the talk
Single and Multi
Objective Linear
Programming
Some recent results
Quadratic Problems
∇F (x) = 0
A Generic Algorithm
Convergence
Gradient Method
Example A
Example B
Conclusions (?)
LION11 32 / 40
min
x
1
2xT Mx
subject to Ax = b
x ≥ 0
min
1
2
xT
Mx +
①
2
Ax − b 2
2 +
①
2
max{0, −x} 2
2 =: F(x)
∇F(x) = Mx + q + ①AT
(Ax − b) − ① max{0, −x}
x = x(0)
+ ①−1
x(1)
+ ①−2
x(2)
+ . . .
b = b(0)
+ ①−1
b(1)
+ ①−2
b(2)
+ . . .
A ∈ IRm×n
rank(A) = m

∇F(x) = 0
Outline of the talk
Single and Multi
Objective Linear
Programming
Some recent results
Quadratic Problems
∇F (x) = 0
A Generic Algorithm
Convergence
Gradient Method
Example A
Example B
Conclusions (?)
LION11 33 / 40
0 = Mx + q + ①AT A x(0) + ①−1
x(1) + ①−2
x(2) + . . .
−b(0) − ①−1
b(1) − ①−2
b(2) + . . .
+① max 0, −x(0) − ①−1
x(1) − ①−2
x(2) + . . .

∇F(x) = 0
Outline of the talk
Single and Multi
Objective Linear
Programming
Some recent results
Quadratic Problems
∇F (x) = 0
A Generic Algorithm
Convergence
Gradient Method
Example A
Example B
Conclusions (?)
LION11 33 / 40
0 = Mx + q + ①AT A x(0) + ①−1
x(1) + ①−2
x(2) + . . .
−b(0) − ①−1
b(1) − ①−2
b(2) + . . .
+① max 0, −x(0) − ①−1
x(1) − ①−2
x(2) + . . .
Looking at the ① terms
Ax(0)
− b(0)
= 0
max 0, −x(0)
= 0 and hence x(0)
≥ 0

∇F(x) = 0
Outline of the talk
Single and Multi
Objective Linear
Programming
Some recent results
Quadratic Problems
∇F (x) = 0
A Generic Algorithm
Convergence
Gradient Method
Example A
Example B
Conclusions (?)
LION11 33 / 40
0 = Mx + q + ①AT A x(0) + ①−1
x(1) + ①−2
x(2) + . . .
−b(0) − ①−1
b(1) − ①−2
b(2) + . . .
+① max 0, −x(0) − ①−1
x(1) − ①−2
x(2) + . . .
Looking at the ①0
terms
Mx(0)
+ q + AT
Ax(1)
− b(1)
− v = 0
where
vj = max 0, −x
(1)
j
only for the indices j for which x
(0)
j = 0, otherwise vj = 0

∇F(x) = 0
Outline of the talk
Single and Multi
Objective Linear
Programming
Some recent results
Quadratic Problems
∇F (x) = 0
A Generic Algorithm
Convergence
Gradient Method
Example A
Example B
Conclusions (?)
LION11 33 / 40
0 = Mx + q + ①AT A x(0) + ①−1
x(1) + ①−2
x(2) + . . .
−b(0) − ①−1
b(1) − ①−2
b(2) + . . .
+① max 0, −x(0) − ①−1
x(1) − ①−2
x(2) + . . .
Set
u = Ax(1)
− b(1)
vj =
0 if x
(0)
j = 0
max 0, −x
(1)
j otherwise
Then
Mx(0)
+ q + AT
u − v = 0
v ≥ 0, vT
x0
= 0

A Generic Algorithm
Outline of the talk
Single and Multi
Objective Linear
Programming
Some recent results
Quadratic Problems
∇F (x) = 0
A Generic Algorithm
Convergence
Gradient Method
Example A
Example B
Conclusions (?)
LION11 34 / 40
min
x
f(x)
f(x) = ①f(1)
(x) + f(0)
(x) + ①−1
f(−1)
(x) + . . .
∇f(x) = ①∇f(1)
(x) + ∇f(0)
(x) + ①−1
∇f(−1)
(x) + . . .

A Generic Algorithm
Outline of the talk
Single and Multi
Objective Linear
Programming
Some recent results
Quadratic Problems
∇F (x) = 0
A Generic Algorithm
Convergence
Gradient Method
Example A
Example B
Conclusions (?)
LION11 34 / 40
min
x
f(x)
At iteration k

A Generic Algorithm
Outline of the talk
Single and Multi
Objective Linear
Programming
Some recent results
Quadratic Problems
∇F (x) = 0
A Generic Algorithm
Convergence
Gradient Method
Example A
Example B
Conclusions (?)
LION11 34 / 40
min
x
f(x)
At iteration k
If
∇f(1)
(xk
) = 0 and ∇f(0)
(xk
) = 0
STOP

A Generic Algorithm
Outline of the talk
Single and Multi
Objective Linear
Programming
Some recent results
Quadratic Problems
∇F (x) = 0
A Generic Algorithm
Convergence
Gradient Method
Example A
Example B
Conclusions (?)
LION11 34 / 40
min
x
f(x)
At iteration k
otherwise ﬁnd xk+1 such that

A Generic Algorithm
Outline of the talk
Single and Multi
Objective Linear
Programming
Some recent results
Quadratic Problems
∇F (x) = 0
A Generic Algorithm
Convergence
Gradient Method
Example A
Example B
Conclusions (?)
LION11 34 / 40
min
x
f(x)
At iteration k
If ∇f(1)(xk) = 0
f(1)
(xk+1
) ≤ f(1)
(xk
) + σ ∇f(1)
(xk
)
f(0)
(xk+1
) ≤ max
0≤j≤lk
f(0)
(xk−j
) + σ ∇f(0)
(xk
)

A Generic Algorithm
Outline of the talk
Single and Multi
Objective Linear
Programming
Some recent results
Quadratic Problems
∇F (x) = 0
A Generic Algorithm
Convergence
Gradient Method
Example A
Example B
Conclusions (?)
LION11 34 / 40
min
x
f(x)
At iteration k
If ∇f(1)(xk) = 0
f(0)
(xk+1
) ≤ f(0)
(xk
) + σ ∇f(0)
(xk
)
f(1)
(xk+1
) ≤ max
0≤j≤mk
f(1)
(xk−j
)

A Generic Algorithm
Outline of the talk
Single and Multi
Objective Linear
Programming
Some recent results
Quadratic Problems
∇F (x) = 0
A Generic Algorithm
Convergence
Gradient Method
Example A
Example B
Conclusions (?)
LION11 34 / 40
min
x
f(x)
m0 = 0, mk+1 ≤ max {mk + 1, M}
l0 = 0, kk+1 ≤ max {lk + 1, L}
σ(.) is a forcing function.
Non–monotone optimization technique, Zhang-Hager, Grippo-
Lampariello-Lucidi, Dai

Convergence
Outline of the talk
Single and Multi
Objective Linear
Programming
Some recent results
Quadratic Problems
∇F (x) = 0
A Generic Algorithm
Convergence
Gradient Method
Example A
Example B
Conclusions (?)
LION11 35 / 40
Case 1: ∃¯k such that ∇f(1)(xk) = 0, k ≥ ¯k
Then
f(1)
(xk+1
) ≤ max
0≤j≤mk
f(1)
(xk−j
), k ≥ ¯k
and hence
max
0≤i≤M
f(1)
(x
¯k+Ml+i
) ≤ max
0≤i≤M
f(1)
(x
¯k+M(l−1)+i
)
and
f(0)
(xk+1
) ≤ f(0)
(xk
) + σ ∇f(0)
(xk
) , k ≥ ¯k
Assuming that the level sets for f(1)(x0) and f(0)(x0) are compact sets,
then the sequence has at least one accumulation point x∗ and any
accumulation point satisﬁes ∇f(1)(x∗) = 0 and ∇f(0)(x∗) = 0

Convergence
Outline of the talk
Single and Multi
Objective Linear
Programming
Some recent results
Quadratic Problems
∇F (x) = 0
A Generic Algorithm
Convergence
Gradient Method
Example A
Example B
Conclusions (?)
LION11 35 / 40
Case 2: ∃ a subsequence jk such that ∇f(1)(xjk ) = 0
Then
f(1)
(xjk+1
) ≤ f(1)
(xjk
) + +σ ∇f(1)
(xjk
)
Again
max
0≤i≤M
f(1)
(xjk+Mt+i
) ≤ max
0≤i≤M
f(1)
(xjk+M(t−1)+i
)+σ ∇f(1)
(xjk
)
and hence ∇f(1)(xjk ) → 0.

Convergence
Outline of the talk
Single and Multi
Objective Linear
Programming
Some recent results
Quadratic Problems
∇F (x) = 0
A Generic Algorithm
Convergence
Gradient Method
Example A
Example B
Conclusions (?)
LION11 35 / 40
Case 2: ∃ a subsequence jk such that ∇f(1)(xjk ) = 0
Then
f(1)
(xjk+1
) ≤ f(1)
(xjk
) + +σ ∇f(1)
(xjk
)
Again
max
0≤i≤M
f(1)
(xjk+Mt+i
) ≤ max
0≤i≤M
f(1)
(xjk+M(t−1)+i
)+σ ∇f(1)
(xjk
)
and hence ∇f(1)(xjk ) → 0. Moreover,
max
0≤i≤L
f(0)
(xjk+Lt+i
) ≤ max
0≤i≤L
f(0)
(xjk+L(t−1)+i
)+σ ∇f(0)
(xjk
)
and hence ∇f(0)(xjk ) → 0.

Gradient Method
Outline of the talk
Single and Multi
Objective Linear
Programming
Some recent results
Quadratic Problems
∇F (x) = 0
A Generic Algorithm
Convergence
Gradient Method
Example A
Example B
Conclusions (?)
LION11 36 / 40
At iterations k calculate ∇f(xk).
If ∇f(1)(xk) = 0
xk+1
= min
α≥0,β≥0
f xk
− α∇f(1)
(xk
) − β∇f(0)
(xk
)
If ∇f(1)(xk) = 0
xk+1
= min
α≥0
f(0)
xk
− α∇f(0)
(xk
)

Example A
Outline of the talk
Single and Multi
Objective Linear
Programming
Some recent results
Quadratic Problems
∇F (x) = 0
A Generic Algorithm
Convergence
Gradient Method
Example A
Example B
Conclusions (?)
LION11 37 / 40
min
x
1
2x2
1 + 1
6 x2
2
subject to x1 + x2 − 1 = 0
f(x) =
1
2
x2
1 +
1
6
x2
2 +
1
2
①(1 − x1 − x2)2
x0
=
4
1
→ x1
=
0.31
0.69
→ x2
=
−0.1
0.39
→ x3
=
0.26
0.74
→
x4
=
−0.12
0.38
→ x5
=
0.25
0.75

Example B
Outline of the talk
Single and Multi
Objective Linear
Programming
Some recent results
Quadratic Problems
∇F (x) = 0
A Generic Algorithm
Convergence
Gradient Method
Example A
Example B
Conclusions (?)
LION11 38 / 40
min
x
x1 + x2
subject to x1
1 + x2
2 − 2 = 0
f(x) =
1
2
x2
1 +
1
6
x2
2 +
1
2
①(1 − x1 − x2)2
x0
=
0.25
0.75
→ x1
=
−1.22
−0.72
→ x2
=
−7.39
−6.89
→ x3
=
1.04
0.95
x4
=
−7.10
−7.19
→ x5
=
−1
−1

Conclusions (?)
Outline of the talk
Single and Multi
Objective Linear
Programming
Some recent results
Quadratic Problems
∇F (x) = 0
A Generic Algorithm
Convergence
Gradient Method
Example A
Example B
Conclusions (?)
LION11 39 / 40
■ The use of ① is extremely beneﬁcial in various aspects in Linear and
■ Difﬁcult problems in NLP can be approached in a simpler way using ①
■ A new convergence theory for standard algorithms (gradient, Newton’s,
Quasi-Newton) needs to be developed in theis new framework

Outline of the talk
Single and Multi
Objective Linear
Programming
Some recent results
Quadratic Problems
∇F (x) = 0
A Generic Algorithm
Convergence
Gradient Method
Example A
Example B
Conclusions (?)
LION11 40 / 40
Thanks for your attention

Main

Recommended

Recommended

More Related Content

Recently uploaded

Recently uploaded (20)

Featured

Featured (20)

Main