Basic terminology description in convex optimization. It is related to applied mathematics.

1. Basic Terminology Description in Convex Optimization
Domain: Applied Mathematics
Dr. Varun Kumar
Domain: Applied Mathematics Dr. Varun Kumar (IIIT Surat)Dr. Varun Kumar 1 / 14

2. Outlines
1 Introduction to Null Space of a Matrix
2 Introduction to Convex Set and Affine Set
3 Introduction to Half Space and Hyper Plane
4 Introduction to Norm and Norm Ball
5 Practical Application of Ellipse and Ellipsoid
6 References
Domain: Applied Mathematics Dr. Varun Kumar (IIIT Surat)Dr. Varun Kumar 2 / 14

3. Null space of a matrix
Let A ∈ Rm×n is a matrix of size m × n. Let a vector space ¯x exist in such
a way that
N(A) = Ax = 0m×1
where N(A) is a null space of matrix.
Note :
⇒ Basically, null space is a vector space.
⇒ Let x1, x2 ∈ N(A). It means that Ax1 = 0 and Ax2 = 0.
⇒ Linear combination of vector space, i.e αx1 + βx2 ∈ N(A)
⇒ αx1 + βx2 ∈ N(A) ⇒ A(αx1 + βx2) ⇒ αAx1 + βAx2 = 0
Domain: Applied Mathematics Dr. Varun Kumar (IIIT Surat)Dr. Varun Kumar 3 / 14

4. Introduction to convex set
Let ¯x1 and ¯x2 are the two points in n − dimensional space then the line
segment can be expressed as
¯z = θ ¯x1 + (1 − θ) ¯x2 ∀ 0 ≤ θ ≤ 1
Convex set:
For any ¯x1, ¯x2 ∈ S, then the entire line segment between ¯x1 and ¯x2 ∈ S
Domain: Applied Mathematics Dr. Varun Kumar (IIIT Surat)Dr. Varun Kumar 4 / 14

5. Example
Convex combination: Let x1, x2, ..., xk are k points in n − dimensional
signal space then
θ1x1 + θ2x2 + .... + θkxk ∀ 0 ≤ θ1, θ2, ..., θk ≤ 1
and θ1 + θ2 + ... + θk = 1
Convex hull
The region which complete the non-convex set into convex set is called as
convex hull.
Domain: Applied Mathematics Dr. Varun Kumar (IIIT Surat)Dr. Varun Kumar 5 / 14

6. Continued–
Affine set :
Let ¯x1 and ¯x2 are two points such that, ¯x1, ¯x2 ∈ S then the entire line
joining these two points also belongs to S . Here, the set S is called as
affine set.
Mathematical Description:
θx1 + (1 − θ)x2 ∀ θ
Domain: Applied Mathematics Dr. Varun Kumar (IIIT Surat)Dr. Varun Kumar 6 / 14

7. Continued–
Graphical representation of convex and affine set
Note:
⇒ Convex set is the special case of affine set.
⇒ It may be non-convex.
Example : Consider a line in 2D is 4x1 + 5x2 = 20
Domain: Applied Mathematics Dr. Varun Kumar (IIIT Surat)Dr. Varun Kumar 7 / 14

8. Example
Half spaces :
Here, a line divides the plane into two region, these two regions are called
as half spaces, i.e 4x1 + 5x2 < 20 and 4x1 + 5x2 > 20.
Note :
Half spaces are always convex, they may or may not be affine.
⇒ Let k number of points lies in the n-dimensional space, such that
a1x1 + a2x2 + ... + akxk = b ⇒ aT
x = b → Hyper plane
Domain: Applied Mathematics Dr. Varun Kumar (IIIT Surat)Dr. Varun Kumar 8 / 14

9. Introduction to norm
where a = [a1, a2, ..., ak]T and x = [x1, x2, ..., xk]T
⇒ Hyper plane divides the n- dimensional plane into two region, i.e
aTx > b and aTx < b.
⇒ Half spaces and hyper plane both are convex.
Introduction to norm
Let x1, x2,....xk be the elements of vector x. Mathematically, pth order
norm can be expressed as
lp =
k
i=1
xp
i
1
p
Example:
l1 = |x1| + |x2| + ... + |xk| → First order norm
l2 = |x1|2 + |x2|2 + ... + |xk|2 Second order/Euclidean norm
Domain: Applied Mathematics Dr. Varun Kumar (IIIT Surat)Dr. Varun Kumar 9 / 14

10. Graphical representation
l∞ = max{|x1|, ..., |xk|}
As per the norms definition: l1 > l2 > .... > l∞
Let (x1, x2) is a co-ordinate point in Cartesian co-ordinate system, where
x1 > x2 is shown in above figure. Hence,
l1 = x1 + x2 → Sum of two sides of a triangle.
l2 = x2
1 + x2
2 Euclidean distance of point (x1, x2) from origin.
l∞ = max{x1, x2} = x1 Larger distance either in x or y axis from origin.
Domain: Applied Mathematics Dr. Varun Kumar (IIIT Surat)Dr. Varun Kumar 10 / 14

11. Norm ball or Euclidean ball
Case 1: In case of two dimensional plane, let ¯x = [x1, x2] then its l2 norm
represents the interior of the circle. Mathematically,
¯x < rc ⇒ rc = Radius of circle
Case 2: In case of n-dimensional plane, let ¯x = [x1, x2, ....xk] then its l2
norm represents the interior of the sphere. Mathematically,
¯x < rs ⇒ rs = Radius of sphere
Domain: Applied Mathematics Dr. Varun Kumar (IIIT Surat)Dr. Varun Kumar 11 / 14

12. Practical application of ellipse and ellipsoid
Mathematical description
Case 1: In case of 2D plane, where any point is interior to the ellipse
x2
1
a2
+
x2
2
b2
≤ 1 ⇒ [x1, x2]
1
a 0
0 1
b
1
a 0
0 1
b
x1
x2
≤ 1 (1)
Let A =
a 0
0 b
and ¯x = [x1, x2]T
Domain: Applied Mathematics Dr. Varun Kumar (IIIT Surat)Dr. Varun Kumar 12 / 14

13. Continued–
From (1), it is clear that
¯xT
(A−1
)T
A−
1x ≤ 1 ⇒ (A−1
x)T
(A−1
x) ≤ 1 (2)
From (2), uTu = u 2, considering u = A−1x
Case 2: In case of n-dimensional signal, i.e ¯x = [x1, x2, ...., xk]T , when
points are interior to an ellipsoid,
x2
1
a2
1
+
x2
2
a2
2
+ ..... +
x2
k
a2
k
≤ 1 (3)
In this case, A = diag(a1, a2, ....ak) and
¯xT
(A−1
)T
A−
1x ≤ 1 ⇒ (A−1
x)T
(A−1
x) ≤ 1 (4)
Domain: Applied Mathematics Dr. Varun Kumar (IIIT Surat)Dr. Varun Kumar 13 / 14

14. References
J. Navarro, “A very simple proof of the multivariate chebyshev’s inequality,”
Communications in Statistics-Theory and Methods, vol. 45, no. 12, pp. 3458–3463,
2016.
M. I. Jordan and T. M. Mitchell, “Machine learning: Trends, perspectives, and
prospects,” Science, vol. 349, no. 6245, pp. 255–260, 2015.
Domain: Applied Mathematics Dr. Varun Kumar (IIIT Surat)Dr. Varun Kumar 14 / 14