This document discusses convex programming problems and optimization problems. It defines convex sets and convex functions, and explains how to determine if a function is convex using its epigraph or differentiability properties. It then discusses various types of convex optimization problems including quadratic programming problems. It provides optimality conditions for constrained and unconstrained optimization using KKT conditions and discusses Wolfe's method for solving quadratic programming problems.

1. Convex Programming Problems
COP
Linear CPP
Non-linear
With Constrained
CPP
QPP
With
Unconstrained

2. Convex Set & Convex Function
Let S⊆ Rn. The set S is called a convex set if for 0≤ 𝜆 ≤ 1, 𝑥1, 𝑥2 ∈ 𝑆 ⇒ 𝜆𝑥1 + 1 − 𝜆 𝑥2 ∈ 𝑆.
Convex Set
Convex Function
Let S⊆ Rn
be a convex set and f: S → 𝑅. 𝑇ℎ𝑒𝑛 𝑓 𝑖𝑠 𝑐𝑎𝑙𝑙𝑒𝑑 𝑎 𝑐𝑜𝑛𝑣𝑒𝑥 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛 𝑖𝑓 𝑓𝑜𝑟 𝑎𝑙𝑙 𝑥1, 𝑥2 ∈ S and for all
0≤ 𝜆 ≤ 1, we have
𝑓 𝜆x1 + 1 − 𝜆 x2 ≤ 𝜆𝑓 𝑥1 + 1 − 𝜆 𝑓 𝑥2 .
0 𝑥1 𝑥2
𝑥
𝐵
𝐴
𝐶
𝑃
𝐵
𝐴𝐵 ≤ 𝐴𝐶

3. Convex Function by Epigraph
Epigraph
Let S⊆ 𝑅𝑛 be a convex set and 𝑓: 𝑆 → 𝑅. Then the set 𝐸𝑓 ⊆ 𝑅𝑛+1 given by Ef = { x, α : xϵ𝑆, 𝛼𝜖𝑅, 𝑓 𝑥 ≤ 𝛼} is
called epigraph of f.
Result:1 Let S⊆ 𝑅𝑛
be a convex set and 𝑓: 𝑆 → 𝑅.
Then 𝑓 is a convex function on S if and only if its epigraph Ef is a convex
set.
𝐸𝑓
x
f(x)

4. Convex Function by Differentiability
Differentiable Convex Functions
Let S⊆ Rn
be an open convex set and 𝑓: 𝑆 → 𝑅 be 𝑑𝑖𝑓𝑓𝑒𝑟𝑒𝑛𝑡𝑖𝑎𝑏𝑙𝑒. let 𝑓 be a convex function on S.
Then, for all x1, x2 ∈ 𝑆, 𝑤𝑒 ℎ𝑎𝑣𝑒 𝑓 x1 − 𝑓 x2 ≥ x1 − x2
T
∇𝑓(𝑥2)
Result:2 Let S⊆ Rn be an open convex set and 𝑓: 𝑆 → R be differentiable. Then f is a convex function
on S if and only if for all x1, x2 ∈ 𝑆, we have 𝒙𝟏 − 𝒙𝟐
𝑻[𝜵𝒇 𝒙𝟏 − 𝜵𝒇 𝒙𝟐 ] ≥ 𝟎.
Result:3 If f: R→ 𝑅 , then 𝐫𝐞𝐬𝐮𝐥𝐭 𝟐 means x1 − x2
T ∇𝑓′ 𝑥1 − ∇𝑓′ 𝑥2 ≥ 0, 𝑖. 𝑒. 𝑓𝑜𝑟 𝑥1 ≥ 𝑥2, 𝑓′ 𝑥1 ≥
𝑓′ 𝑥2 . 𝑇ℎ𝑢𝑠 𝑓′ is an increasing function which is the well known definition of convexity of real valued
function of real variable.
Result:4 Let S⊆ Rn
be an open convex set and 𝑓: 𝑆 → R be 𝑡𝑤𝑖𝑐𝑒 𝑑𝑖𝑓𝑓𝑒𝑟𝑒𝑛𝑡𝑖𝑎𝑏𝑙𝑒. Then f is a convex function on
S if and only if the Hessian matrix 𝐻𝑓 𝑥 is positive semi definite for all x ∈S.

5. Convex Optimization Problems
min f(x)
x∈ 𝑆
is convex optimization problem if
S⊆ 𝑅𝑛 convex set and 𝑓 is a convex on S
max f(x)
x∈ 𝑆
is convex optimization problem if
S⊆ 𝑅𝑛 convex set and 𝑓 is a concave on S
Result:5 If x∗
be a local min point of the convex optimization problem of above problem.
Then, x∗
is also its global min. point.
Result:6 The set of all optimal solutions of the convex optimization problem(above) is a convex set.
Result:7 Let S⊆ Rn
be a convex set and 𝑓: 𝑆 → R be a strictly convex function.
Then there is a unique minimizing point of 𝑓 over S.

6. Convex Programming Problems
An Optimization problems can be of the form,
min or max f(x)
Subject to 𝒈𝒊 𝒙 ≤ 𝒐𝒓 ≥ 𝟎 𝒊 = 𝟏, 𝟐, 𝟑, … , 𝒎
These problems are called mathematical programming problems.
If a Problem is convex programming problem, then it is one of the form
min f(x)
Subject to 𝑔𝑖 𝑥 ≤ 0
f, 𝑔𝑖, are convex
(i=1,2,3,…,m)
min f(x)
Subject to 𝑔𝑖 𝑥 ≥ 0
f is convex, 𝑔𝑖 are
concave
(i=1,2,3,…,m)
max f(x)
Subject to 𝑔𝑖 𝑥 ≤ 0
f is concave 𝑔𝑖, are
convex
(i=1,2,3,…,m)
max f(x)
Subject to 𝑔𝑖 𝑥 ≥ 0
f, 𝑔𝑖, are concave
(i=1,2,3,…,m)
⇓ ⇓ ⇓ ⇓

7. Optimality Condition
Result:8 If 𝑥∗ ∈ 𝑅 is a local min or local max point of 𝑓 over R then 𝑓′ 𝑥∗ = 0.
Result:9 (1) The point If 𝑥∗ ∈ 𝑅 such that 𝑓′ 𝑥∗ = 0, 𝑖𝑠 𝑎𝑛 𝑢𝑛𝑐𝑜𝑛𝑠𝑡𝑟𝑎𝑖𝑛𝑒𝑑
𝑙𝑜𝑐𝑎𝑙 min 𝑝𝑜𝑖𝑛𝑡 𝑜𝑓 𝑓 𝑜𝑣𝑒𝑟 𝑅 𝑖𝑓 𝑓′′ 𝑥∗ > 0.
(2) The point If 𝑥∗
∈ 𝑅 such that 𝑓′
𝑥∗
= 0, 𝑖𝑠 𝑎𝑛 𝑢𝑛𝑐𝑜𝑛𝑠𝑡𝑟𝑎𝑖𝑛𝑒𝑑
𝑙𝑜𝑐𝑎𝑙 max 𝑝𝑜𝑖𝑛𝑡 𝑜𝑓 𝑓 𝑜𝑣𝑒𝑟 𝑅 𝑖𝑓 𝑓′′
𝑥∗
< 0.
Result:10 (1) 𝐿𝑒𝑡 (𝑥∗, 𝑦∗) ∈ 𝑅2 𝑏𝑒 𝑎 𝑙𝑜𝑐𝑎𝑙 min 𝑜𝑟 𝑙𝑜𝑐𝑎𝑙 max 𝑜𝑓 𝑓 𝑜𝑣𝑒𝑟 𝑅2, 𝑡ℎ𝑒𝑛
𝜕𝑓
𝜕𝒙 𝒙∗,𝒚∗
= 𝟎,
𝜕𝑓
𝜕𝒚 𝒙∗,𝒚∗
= 𝟎,
(But that is only necessary result)
(2) 𝐿𝑒𝑡 (𝑥∗, 𝑦∗) ∈ 𝑅2 is local min if Hessian matrix 𝐻𝑓 𝑥∗, 𝑦∗ 𝑖𝑠 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑑𝑒𝑓𝑖𝑛𝑖𝑡𝑒.
𝑇ℎ𝑒𝑛, 𝑥∗
, 𝑦∗
𝑖𝑠 𝑎 𝑠𝑡𝑟𝑖𝑐𝑡 𝑙𝑜𝑐𝑎𝑙 min 𝑝𝑜𝑖𝑛𝑡 𝑜𝑣𝑒𝑟 𝑅2
.
(3) 𝐿𝑒𝑡 (𝑥∗, 𝑦∗) ∈ 𝑅2 is local max if Hessian matrix 𝐻𝑓 𝑥∗, 𝑦∗ 𝑖𝑠 𝑛𝑒𝑔𝑎𝑡𝑖𝑣𝑒 𝑑𝑒𝑓𝑖𝑛𝑖𝑡𝑒.
𝑇ℎ𝑒𝑛 𝑥∗
, 𝑦∗
𝑖𝑠 𝑎 𝑠𝑡𝑟𝑖𝑐𝑡 𝑙𝑜𝑐𝑎𝑙 m𝑎𝑥 𝑝𝑜𝑖𝑛𝑡 𝑜𝑣𝑒𝑟 𝑅2
.

8. Constrained Optimization
We Consider the general optimization problem
min 𝒇 𝒙
Subject to
𝒈𝒊 𝒙 ≤ 𝒃𝒊, (𝒊 = 𝟏, 𝟐, … , 𝒎) …(1)
How to Solve this types of problems using KKT
 First check objective function should be min. if
not then convert it.
 Add slack variables and write Lagrange's
function.
 Then Lagrange's function of above problem is
𝐿 𝑥, 𝑠, 𝜆 = 𝑓 𝑥 + 𝑖=1
𝑚
𝜆𝑖(𝑔𝑖 𝑥 + 𝑠𝑖
2
− 𝑏𝑖)
Where 𝜆 is Lagrange's multiplier, s is slack
variables.
Now optimality condition for optimal point (𝑥∗
, 𝑠∗
, 𝜆∗
) are
𝜵𝒇 𝒙∗ +
𝒊=𝟏
𝒎
𝝀𝒊
∗
𝜵𝒈𝒊(𝒙∗) = 𝟎
𝝀𝒊
∗
𝒈𝒊 𝒙∗
− 𝒃𝒊 = 𝟎 (i=1,2,3, … ,m)
𝒈𝒊 𝒙 ≤ 𝒃𝒊 (i=1,2,3, … ,m)
𝝀𝒊
∗
≥ 𝟎 (i=1,2,3, …, m)
These are the KKT condition for the problem 1.
𝑀𝑖𝑛 − 𝑥2
𝑠. 𝑡.
𝑥1
2
+ 𝑥2
2
≤ 4
−𝑥1
2
+ 𝑥2 ≤ 0
0,0 satisfy KKT condition but
not point of min.

9. Quadratic Programming Problem
Consider the general QPP
𝒎𝒊𝒏 𝒙𝑻
𝑸𝒙 + 𝒄𝑻
𝒙
𝒔. 𝒕. 𝑨𝒙 ≤ 𝒃
𝒙 ≥ 𝟎 …(2)
Where 𝑄 = 𝑞𝑖𝑗 𝑛∗𝑛
𝑆𝑦𝑚𝑚𝑡𝑒𝑟𝑖𝑐 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒
𝑠𝑒𝑚𝑖 − 𝑑𝑒𝑓𝑖𝑛𝑖𝑡𝑒 𝑚𝑎𝑡𝑟𝑖𝑥, 𝑐, 𝑥 ∈ 𝑅𝑛
, 𝑏 ∈ 𝑅𝑚
&
𝐴 = 𝑎𝑖𝑗 𝑚∗𝑛
KKT condition for QPP (2) is
𝟐𝑸𝒙 + 𝒄 + 𝑨𝑻𝝀 − 𝑰𝝁 = 𝟎
𝑨𝒙 − 𝒃 + 𝒔 = 𝟎
𝒙𝒋, 𝝀𝒊, 𝝁𝒋, 𝒔𝒊 ≥ 𝟎 ∀ 𝒊&𝒋
𝝀𝒊𝒔𝒊 = 𝟎 ∀ 𝒊 = 𝟏, 𝟐, 𝟑, … , 𝒎
𝝁𝒋𝒙𝒋 = 𝟎 ∀ 𝒋 = 𝟏, 𝟐, 𝟑, … 𝒏
𝑀𝑖𝑛 𝑥1 − 4 2
+ 𝑥2 − 4 2
𝑠. 𝑡.
𝑥1 + 𝑥2 ≤ 4
𝑥1 − 𝑥2 ≤ 1
𝑥1 ≥ 0, 𝑥2 ≥ 0
(2,2) is point of min.

10. Wolfe’s Method for QPP
KKT condition for QPP (3) with 𝒎𝒊𝒏 − 𝒇(𝒙) is
−𝟐𝑸𝒙 + 𝑨𝑻𝝀 − 𝑰𝝁 = 𝒄
𝑨𝒙 + 𝒔 = 𝒃
𝒙𝒋, 𝝀𝒊, 𝝁𝒋, 𝒔𝒊 ≥ 𝟎 ∀ 𝒊&𝒋
𝝀𝒊𝒔𝒊 = 𝟎 ∀ 𝒊 = 𝟏, 𝟐, 𝟑, … , 𝒎
𝝁𝒋𝒙𝒋 = 𝟎 ∀ 𝒋 = 𝟏, 𝟐, 𝟑, … , 𝒏
Consider the general QPP
𝒎𝒂𝒙 𝒇 𝒙 = 𝒙𝑻
𝑸𝒙 + 𝒄𝑻
𝒙
𝒔. 𝒕. 𝑨𝒙 ≤ 𝒃
𝒙 ≥ 𝟎 …(3)
Where 𝑄 = 𝑞𝑖𝑗 𝑛∗𝑛
𝑆𝑦𝑚𝑚𝑡𝑒𝑟𝑖𝑐 𝑛𝑒𝑔𝑎𝑡𝑖𝑣𝑒 𝑠𝑒𝑚𝑖 −
𝑑𝑒𝑓𝑖𝑛𝑖𝑡𝑒 𝑚𝑎𝑡𝑟𝑖𝑥, 𝑐, 𝑥 ∈ 𝑅𝑛, 𝑏 ∈ 𝑅𝑚 & 𝐴 = 𝑎𝑖𝑗 𝑚∗𝑛
𝑀𝑎𝑥 𝑧 = 𝑥1 + 𝑥2 − 𝑥1
2
+ 2 𝑥1𝑥2 − 2 𝑥2
2
𝑠. 𝑡.
2𝑥1 + 𝑥2 ≤ 1
𝑥1, 𝑥2 ≥ 0

11. Thank You