adil raja, profile picture
Uploaded byadil raja
PDF, PPTX4,663 views

Convex Optimization

This document outlines an introduction to convex optimization. It begins with an introduction stating that convex optimization problems can be solved efficiently to find the global optimum. It then provides an outline covering convex sets, convex functions, convex optimization problems, and references. The body of the document defines convex sets as sets where a line segment between any two points lies entirely within the set. It also provides examples of convex sets including norm balls and intersections of convex sets. It defines convex functions as functions where the graph lies below any line segment between two points, and provides conditions for checking convexity using derivatives. Finally, it discusses convex optimization problems and solving them efficiently.

Embed presentation

Download as PDF, PPTX
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References CONVEX OPTIMIZATION Muhammad Adil Raja Roaming Researchers, Inc. August 31, 2014
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References OUTLINE INTRODUCTION CONVEX SETS CONVEX FUNCTIONS CONVEX OPTIMIZATION PROBLEMS REFERENCES
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References OUTLINE INTRODUCTION CONVEX SETS CONVEX FUNCTIONS CONVEX OPTIMIZATION PROBLEMS REFERENCES
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References OUTLINE INTRODUCTION CONVEX SETS CONVEX FUNCTIONS CONVEX OPTIMIZATION PROBLEMS REFERENCES
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References OUTLINE INTRODUCTION CONVEX SETS CONVEX FUNCTIONS CONVEX OPTIMIZATION PROBLEMS REFERENCES
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References OUTLINE INTRODUCTION CONVEX SETS CONVEX FUNCTIONS CONVEX OPTIMIZATION PROBLEMS REFERENCES
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References INTRODUCTION I I Many situations arise in machine learning where we would like to optimize the value of some function. I That is, given a function f : Rn ! R, we want to find x 2 Rn that minimizes (or maximizes) f (x). I In the general case, finding the global optimum of a function can be a very difficult task. I However, for a special class of optimization problems, known as convex optimization problems, we can efficiently find the global solution in many cases. I “efficiently” has both practical and theoretical connotations: 1. It means that we can solve many real-world problems in a reasonable amount of time.
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References INTRODUCTION II 2. And it means that theoretically we can solve problems in time that depends only polynomially on the problem size. I The goal of these section notes and the accompanying lecture is to give a very brief overview of the field of convex optimization.
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References CONVEX SETS I DEFINITION A set C is convex if, for any x, y 2 C and 2 with 0 1, x + (1 )y 2 C. I Intuitively, this means that if we take any two elements in C, and draw a line segment between these two elements, then every point on that line segment also belongs to C. I Figure 1 shows an example of one convex and one non-convex set. I The point x + (1 )y is called a convex combination of the points x and y.
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References CONVEX SETS II (a) (b) Figure 1: Examples of a convex set (a) and a non-convex set (b). FIGURE: Examples of a convex set (a) and a non-convex set. Examples All of Rn. It should be fairly obvious that given any x, y ! Rn, !x + (1 − !)y ! Rn. The non-negative orthant, Rn Examples I All of Rn. It should be fairly obvious that given any +. The non-negative orthant consists of all vectors in Rn whose elements are all non-negative: Rn + = {x : xi # 0 $i = 1, . . . , n}. To show x; y 2 n; x + (1 )y 2 n. I The non-negative orthant, n that this is a convex set, simply note that given any x, y ! Rn + and 0 % ! % 1, +. The non-negative (!x + (1 − !)y)i = !xi + (1 − !)yi # 0 $i. orthant consists of all vectors in n whose elements are all non-negative: Norm balls. Let · be some norm on Rn (e.g., ! the Euclidean norm, x2 = n x2i ). Then the set {x : x % 1} is a convex set. To see this, suppose x, y ! Rn,
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References CONVEX SETS III 1. n + = x : xi 08i = 1; :::; n. 2. To show that this is a convex set, simply note that given any x; y8Rn+ and 0 ? 1, (x + (1 )y)i = xi + (1 )yi 08i. I Norm balls. Let jj:jj be some norm on n (e.g., the Euclidean norm, x2 = qPni =1 x2 i ). Then the set x : jjxjj 1 is a convex set. I To see this,suppose x; y 2 Rn, with jjxjj 1; jjyjj 1,and 0 1. Then jjx + (1 )yjj jjxjj + jj(1 )yjj = jjxjj + (1 )jjyjj 1 where we used the triangle inequality and the positive homogeneity of norms.
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References CONVEX SETS IV I Affine subspaces and polyhedra. Given a matrix A 2 mn and a vector b 2 m, an affine subspace is the set x 2 n : Ax = b (note that this could possibly be empty if b is not in the range of A). Similarly, a polyhedron is the (again, possibly empty) set x 2 Rn : Ax b, where “” here denotes componentwise inequality (i.e., all the entries of Ax are less than or equal to their corresponding element in b). To prove this, first consider x; y 2 n such that Ax = Ay = b. Then for 0 1, A(x+(1)y) = Ax+(1)Ay = b+(1?)b = b. Similarly,for x; y 2 n that satisfy Ax b andAy b and 0 1, A(x+(1)y) = Ax+(1)Ay b+(1)b = b.
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References CONVEX SETS V I Intersections of convex sets. Suppose CT1,C2,...,Ck are convex sets. Then their intersection ki Ci kix x Cii 1k is also a convex set. =1 = : 2 8= ; :::; TTo see this, consider x; y 2 1 Ci and 0 =? 1. ki Then, x + (1 )y 2 Ci8i = 1; :::; k by the Tdefinition of a convex set. Therefore x + (1 )y 2 =1 Ci . Note, however, that the union of convex sets in general will not be convex. I Positive semidefinite matrices. The set of all symmetric positive semidefinite matrices, often times called the positive semidefinite cone and denoted Sn+ , is a convex set (in general, Sn nn denotes the set of symmetric n ? n matrices). Recall that a matrix A 2 nn is symmetric positive semidefinite if and only if A = AT and for all x 2 Rn, xT Ax 0. Now consider two symmetric positive semidefinite
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References CONVEX SETS VI matrices A;B 2 Sn+ and0 1. Then for any x 2 n, xT (A + (1 )B)x = xT Ax + (1 )xT Bx 0. The same logic can be used to show that the sets of all positive definite, negative definite, and negative semidefinite matrices are each also convex.
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References CONVEX FUNCTIONS I DEFINITION A function f : n ! is convex if its domain (denoted D(f)) is a convex set, and if, for all x; y 2 D(f ) and 2 R; 0 ? 1, f (x + (1 )y) f (x) + (1 )f (y). I Intuitively, the way to think about this definition is that if we pick any two points on the graph of a convex function and draw a straight line between them, then the portion of the function between these two points will lie below this straight line. This situation is pictured in Figure 2. I We say a function is strictly convex if Definition 2 holds with strict inequality for x?6= y and 0 1. We say that f is concave if – f is convex, and likewise that f is strictly concave if – f is strictly convex.
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References CONVEX FUNCTIONS II convex FIGURE: function. Graph of a convex By function. the definition By the definition of of convex functions, the graph must lie above the function. convex functions, the line connecting two points on the graph must lie above the function. Condition for Convexity
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References FIRST ORDER CONDITION FOR CONVEXITY I I Suppose a function f : n ! is differentiable. I The f is convex if and only if D(f ) is a convex set and for all x; y 2 D(f ), f (y) f (x) + 5x f (x)T (y x): I The function 5x f (x)T (y x): is called the first-order approximation to the function f at the point x. I Intuitively, this can be thought of as approximating f with its tangent line at the point x. I The first order condition for convexity says that f is convex if and only if the tangent line is a global underestimator of the function f . I In other words, if we take our function and draw a tangent line at any point, then every point on this line will lie below the corresponding point on f . I A function is differentiable if the gradient 5f (x) exists at all points x in the domain of f .
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References FIRST ORDER CONDITION FOR CONVEXITY II I The gradient is defined as 5x f (x) 2 n; (5x f (x))i = @f (x) @(x)i . I Similar to the definition of convexity, f will be strictly convex if this holds with strict inequality, concave if the inequality is reversed, and strictly concave if the reverse inequality is strict.
corresponding point on f. the definition of convexity, f will be strictly convex if this holds Convex with Optimization concave if the inequality is reversed, and strictly concave if the Muhammad reverse Adil inequality Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References FIRST ORDER CONDITION FOR CONVEXITY III Figure 3: Illustration of first-order condition for convexity. the gradient is defined as xf(x) # Rn, (xf(x))i = !f(x) . For a review on gradients !xi previous section notes on linear algebra. FIGURE: Illustration od the first-order condition for convexity.
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References SECOND ORDER CONDITION FOR CONVEXITY I 2x I Suppose a function f : n ! is twice differentiable. I Then f is convex if and only if D(f ) is a convex set and its Hessian is positive semidefinite: i.e., for any x 2 D(f ), 5f (x) 0. I Here, the notation “” when used in conjunction with matrices refers to positive semidefiniteness, rather than componentwise inequality. I In one dimension, this is equivalent to the condition that the second derivative f 00(x) always be positive (i.e., the function always has positive curvature).
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References SECOND ORDER CONDITION FOR CONVEXITY II 2x I Again analogous to both the definition and first order conditions for convexity, f is strictly convex if its Hessian is positive definite, concave if the Hessian is negative semidefinite, and strictly concave if the Hessian is negative definite. I A function is twice differentiable if the Hessian 5f (x) is defined for all points x in the domain of f . 2x I The Hessian is defined as 5f (x) 2 nn; 2x (5f (x))ij = @2f (x) @xi@xj . I For a symmetric matrix X 2 Sn;X 0 denotes that X is negative semidefinite. I As with inequalities, “” and “”.
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References JENSEN’S INEQUALITY I I Suppose we start with the inequality in the basic definition of a convex function f (x + (1 )y) f (x) + (1 )f (y)for0 1. I Using induction, this can be fairly easily extended to convex combinations of more than one point, f ( Pki =1 ixi ) Pki =1 i f (xi ) for Pki =1 i = 1; i 08i. I In fact, this can also be extended to infinite sums or integrals. I In the latter case, the inequality can be written as f ( R p(x)xdx) R R p(x)f (x)dx for p(x)dx = 1; p(x) 08x. I Because p(x) integrates to 1, it is common to consider it as a probability density, in which case the previous equation can be written in terms of expectations, f (E[x]) E[f (x)]. I This last inequality is known as JensenÕs inequality.
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References SUBLEVEL SETS I I Convex functions give rise to a particularly important type of convex set called an -sublevel set. I Given a convex function f : n ! and a real number 2 , the -sublevel set is defined as x 2 D(f ) : f (x) . I In other words, the -sublevel set is the set of all points x such that f (x) . I To show that this is a convex set, consider any x; y 2 D(f ) such that f (x) and f (y) . I Then f (x + (1 )y) f (x) + (1 )f (y) + (1 ) = .
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References EXAMPLES OF CONVEX FUNCTIONS I I We begin with a few simple examples of convex functions of one variable, then move on to multivariate functions. I Exponential. Let f : ! ; f (x) = eax for any a 2 . To show f is convex, we can simply take the second derivative f 00(x) = a2eax , which is positive for all x. I Negative logarithm. Let f : ! ; f (x) =?logx with domain D(f ) = R++. I Here, R++ denotes the set of strictly positive real numbers, x : x 0. I Then f 00(x) = 1=x2 0 for all x. I Affine functions. Let f : n ! ; f (x) = bT x + c for some b 2 n, c 2 . I In this case the Hessian, 52xf (x) = 0 for all x.
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References EXAMPLES OF CONVEX FUNCTIONS II 12 I Because the zero matrix is both positive semidefinite and negative semidefinite, f is both convex and concave. I In fact, affine functions of this form are the only functions that are both convex and concave. I Quadratic functions. Let f : n ! ; f (x) = xT Ax + bT x + c for a symmetric matrix A 2 Sn, b 2 n and c 2 . I The Hessian for this function is given by 52xf (x) = A. I Therefore, the convexity or non-convexity of f is determined entirely by whether or not A is positive semidefinite: if A is positive semidefinite then the function is convex (and analogously for strictly convex, concave, strictly concave).
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References EXAMPLES OF CONVEX FUNCTIONS III 22 I If A is indefinite then f is neither convex nor concave. I Note that the squared Euclidean norm f (x) = jjxjj= xT x is a special case of quadratic ni functions where A = I; b = 0; c = 0, so it is therefore a strictly convex function. I Norms. Let f ;n ! be some norm on n. Then by the triangle inequality and positive homogeniety of norms, for x; y 2 n; 0 1, f (x + (1 )y) f (x) + f (1 )y) = f (x) + (1 )f (y). I This is an example of a convex function where it is not possible to prove convexity based on the second or first order conditions. I The reason is that because norms are not generally differentiable Peverywhere (e.g., the 1-norm, jjxjj1 = =1 jxi j, is non-differentiable at all points where any xi is equal to zero).
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References EXAMPLES OF CONVEX FUNCTIONS IV I Nonnegative weighted sums of convex functions. Let f1; f2; :::; fk be convex functions and w1;w2; :::;wk be nonnegative real numbers. Then f (x) = Pki =1 wi fi (x) is a convex function, since f (x + (1?)y) = Pki =1 wi fi (x + (1 )y) (1) Pki =1 wi (fi (x) + (1 )fi (y)) (2) = Pki =1 wi fi (x) + (1 ) Pki =1 wi fi f (y)(3) = f (x) + (1 )f (x) (4)
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References CONVEX OPTIMIZATION PROBLEMS I I Armed with the definitions of convex functions and sets, we are now equipped to consider convex optimization problems. I Formally, a convex optimization problem in an optimization problem of the form minimize f (x) subject to x 2 C. I where f is a convex function, C is a convex set, and x is the optimization variable. I However, since this can be a little bit vague, we often write it often written as minimize f (x) subject to gi (x) 0, i = 1; :::;m hi (x) = 0; i = 1; :::; p I where f is a convex function, gi are convex functions, and hi are affine functions, and x is the optimization variable.
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References CONVEX OPTIMIZATION PROBLEMS II I It is important to note the direction of these inequalities: 1. A convex function gi must be less than zero. 2. This is because the 0-sublevel set of gi is a convex set, so the feasible region, which is the intersection of many convex sets, is also convex (recall that affine subspaces are convex sets as well). 3. If we were to require that gi 0 for some convex gi , the feasible region would no longer be a convex set, and the algorithms we apply for solving these problems would not longer be guaranteed to find the global optimum. 4. Also notice that only affine functions are allowed to be equality constraints. 5. Intuitively, you can think of this as being due to the fact that an equality constraint is equivalent to the two inequalities hi 0 and hi 0.
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References CONVEX OPTIMIZATION PROBLEMS III 6. However, these will both be valid constraints if and only if hi is both convex and concave, i.e., hi must be affine. I The optimal value of an optimization problem is denoted p (or sometimes f ) and is equal to the minimum possible value of the objective function in the feasible region. p = minf (x) : gi (x) 0; i = 1; :::;m; hi (x) = 0; i = 1; :::; p. I We allow p to take on the values +1 and 1 when the problem is either infeasible (the feasible region is empty) or unbounded below (there exists feasible points such that f (x) ! 1), respectively. I We say that x is an optimal point if f (x?) = p. I Note that there can be more than one optimal point, even when the optimal value is finite.
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References GLOBAL OPTIMALITY IN CONVEX PROBLEMS I I Defining the concepts of local optima and global optima are in order. I Intuitively, a feasible point is called locally optimal if there are no “nearby” feasible points that have a lower objective value. I Similarly, a feasible point is called globally optimal if there are no feasible points at all that have a lower objective value. I To formalize this a little bit more, following definitions are important.
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References GLOBAL OPTIMALITY IN CONVEX PROBLEMS II DEFINITION A point x is locally optimal if it is feasible (i.e., it satisfies the constraints of the optimization problem) and if there exists some R 0 such that all feasible points z with jjx zjj2 R, satisfy f (x) f (z). DEFINITION A point x is globally optimal if it is feasible and for all feasible points z, f (x) f (z). I We now come to the crucial element of convex optimization problems, from which they derive most of their utility.
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References GLOBAL OPTIMALITY IN CONVEX PROBLEMS III I The key idea is that for a convex optimization problem all locally optimal points are globally optimal. I LetÕs give a quick proof of this property by contradiction. I Suppose that x is a locally optimal point which is not globally optimal, i.e., there exists a feasible point y such that f (x) f (y). I By the definition of local optimality, there exist no feasible points z such that jjx?zjj2 Randf (z) f (x).
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References GLOBAL OPTIMALITY IN CONVEX PROBLEMS IV I But now suppose we choose the point z = y + (1 )x with = R2 jjx?yjj2 Then jjx zjj2 = jjx ( R 2jjxyjj2 y + (1 R 2jjxyjj2)x)jj2 (5) = jj R 2jjxyjj2 (x y)jj2 (6) = R=2 R: (7) I In addition, by the convexity of f we have f (z) = f (y + (1?)x) f (y) + (1?)f (x) f (x). I Furthermore, since the feasible set is a convex set, and since x and y are both feasible z = y + (1?) will be feasible as well. I Therefore, z is a feasible point, with jx?zj2 R and f (z) f (x).
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References GLOBAL OPTIMALITY IN CONVEX PROBLEMS V I This contradicts our assumption, showing that x cannot be locally optimal.
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References SPECIAL CASES OF CONVEX PROBLEMS I I For a variety of reasons, it is often times convenient to consider special cases of the general convex programming formulation. I For these special cases we can often devise extremely efficient algorithms that can solve very large problems, and because of this you will probably see these special cases referred to any time people use convex optimization techniques. I Linear Programming. We say that a convex optimization problem is a linear program (LP) if both the objective function f and inequality constraints gi are affine functions. I In other words, these problems have the form minimize cT x + d subject toGx h.
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References SPECIAL CASES OF CONVEX PROBLEMS II I Ax = b where x 2 n is the optimization variable, c 2 n, d 2 , G 2 mn, h 2 m, A 2 p?n, b 2 Rp are defined by the problem, and “” denotes elementwise inequality. I Quadratic Programming. We say that a convex optimization problem is a quadratic program (QP) if the inequality constraints gi are still all affine, but if the objective function f is a convex quadratic function. I In other words, these problems have the form, minimize 1 2xT Px + cT x + d subject to Gx h + Ax = nb. where again x 2 n is the optimization variable, c 2 n, d 2 , G 2 mn, h 2 Rm, A 2 pn, b 2 Rp are defined by the problem, but we also have P 2 S, a symmetric positive semidefinite matrix.
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References SPECIAL CASES OF CONVEX PROBLEMS III I Quadratically Constrained Quadratic Programming. We say that a convex optimization problem is a quadratically constrained quadratic program (QCQP) if both the objective f and the inequality constraints gi are convex quadratic functions, minimize 1 2xT Px + cT x + d subject to 1 2xTQix + r T i x + si 0; i = 1; :::;m Ax = b where, as before, x 2 n is the optimization variable, c 2 n, d 2 , A 2 pn, b 2 p, P 2 Sn+ , but we also have Qi 2 Sn+ ; ri 2 n; si 2 ,for i = 1,...,m. I Semidefinite Programming has become more and more prevalent in many different areas of machine learning research, so you might encounter these at some point, and it is good to have an idea of what they are.
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References SPECIAL CASES OF CONVEX PROBLEMS IV I We say that a convex optimization problem is a semidefinite program (SDP) if it is of the form minimizetr (CX) subject to tr (AiX) = bi , i = 1; :::; p X 0. I where the symmetric matrix X 2 Sn is the optimization variable, the symmetric matrices C, A1; :::; Ap 2 Sn are defined by the problem, and the constraint X 0 means that we are constraining X to be positive semidefinite. I This looks a bit different than the problems we have seen previously, since the optimization variable is now a matrix instead of a vector.
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References SPECIAL CASES OF CONVEX PROBLEMS V I It should be fairly obvious from the definitions that quadratic programs are more general than linear programs (since a linear program is just a special case of a quadratic program where P = 0), and likewise that quadratically constrained quadratic programs are more general than quadratic programs. I However, what is not obvious at all is that semidefinite programs are in fact more general than all the previous types. I That is, any quadratically constrained quadratic program (and hence any quadratic program or linear program) can be expressed as a semidefinte program. I
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References EXAMPLES OF CONVEX OPTIMIZATION I I Support Vector Machines. I Constrained Least Squares. I Maximum likelihood for Logistic Regression.
Convex Optimization Muhammad Adil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References REFERENCES I I The images and inspiration for preparing these slides have been taken from Andrew Ngs course on Machine Learning. I The course is available on Stanford engineering for all and Coursera.

More Related Content

PDF
An overview of gradient descent optimization algorithms.pdf
byvudinhphuong96
 
PPTX
Optimization tutorial
byNorthwestern University
 
PDF
Linear models for classification
bySung Yub Kim
 
PPTX
PRML Chapter 5
bySunwoo Kim
 
PDF
Lecture 4 neural networks
byParveenMalik18
 
PDF
Time Series Classification with Deep Learning | Marco Del Pra
byData Science Milan
 
PDF
Gradient descent method
bySanghyuk Chun
 
PPTX
Machine Learning Unit 2 Semester 3 MSc IT Part 2 Mumbai University
byMadhav Mishra
 
An overview of gradient descent optimization algorithms.pdf
byvudinhphuong96
 
Optimization tutorial
byNorthwestern University
 
Linear models for classification
bySung Yub Kim
 
PRML Chapter 5
bySunwoo Kim
 
Lecture 4 neural networks
byParveenMalik18
 
Time Series Classification with Deep Learning | Marco Del Pra
byData Science Milan
 
Gradient descent method
bySanghyuk Chun
 
Machine Learning Unit 2 Semester 3 MSc IT Part 2 Mumbai University
byMadhav Mishra
 

What's hot

PPTX
Fuzzy logic - Approximate reasoning
byDr. C.V. Suresh Babu
 
PDF
Introduction to optimization Problems
byElectronics & Communication Staff SCU Suez Canal University
 
PDF
Convex optimization methods
byDong Guo
 
PPTX
Direct Methods to Solve Linear Equations Systems
byLizeth Paola Barrero
 
PPTX
Lagrange's method
byKarnav Rana
 
PPTX
Convex optmization in communications
byDeepshika Reddy
 
PPTX
Linear differential equation
byPratik Sudra
 
PPTX
Complex analysis
bysujathavvv
 
PDF
Higher Order Differential Equation
byShrey Patel
 
PPT
LinearAlgebra.ppt
byvijaykumar838577
 
PPTX
Euler and improved euler method
bySohaib Butt
 
PDF
Introduction to the theory of optimization
byDelta Pi Systems
 
PDF
Integration in the complex plane
byAmit Amola
 
PDF
03 convexfunctions
bySufyan Sahoo
 
PPTX
Fuzzy Set
byEhsan Hamzei
 
PPTX
Optimization problems and algorithms
byAboul Ella Hassanien
 
PPTX
Vector space
byMehedi Hasan Raju
 
PDF
Lesson 17: The Method of Lagrange Multipliers
byMatthew Leingang
 
PPT
Eigen values and eigen vectors engineering
byshubham211
 
PDF
Mean Value Theorems
byEduron e-Learning Private Limited
 
Fuzzy logic - Approximate reasoning
byDr. C.V. Suresh Babu
 
Introduction to optimization Problems
byElectronics & Communication Staff SCU Suez Canal University
 
Convex optimization methods
byDong Guo
 
Direct Methods to Solve Linear Equations Systems
byLizeth Paola Barrero
 
Lagrange's method
byKarnav Rana
 
Convex optmization in communications
byDeepshika Reddy
 
Linear differential equation
byPratik Sudra
 
Complex analysis
bysujathavvv
 
Higher Order Differential Equation
byShrey Patel
 
LinearAlgebra.ppt
byvijaykumar838577
 
Euler and improved euler method
bySohaib Butt
 
Introduction to the theory of optimization
byDelta Pi Systems
 
Integration in the complex plane
byAmit Amola
 
03 convexfunctions
bySufyan Sahoo
 
Fuzzy Set
byEhsan Hamzei
 
Optimization problems and algorithms
byAboul Ella Hassanien
 
Vector space
byMehedi Hasan Raju
 
Lesson 17: The Method of Lagrange Multipliers
byMatthew Leingang
 
Eigen values and eigen vectors engineering
byshubham211
 
Mean Value Theorems
byEduron e-Learning Private Limited
 

Similar to Convex Optimization

PDF
Cs229 cvxopt
bycerezaso
 
PDF
Optimization introduction
byhelalmohammad2
 
PPTX
Math Notation using in optimation problem
bytrauvang2871
 
PDF
[Convex Optimization_slides Stephen Boyd] S.Boyd L.Vandenberghe .pdf
byBouaziziHechmi1
 
PDF
bv_cvxslides (1).pdf
bySantiagoGarridoBulln
 
PDF
complete-manual-of-multivariable-optimization.pdf
bySantiagoGarridoBulln
 
PPTX
Basic presentation of convex optimization 2018.pptx
byubaidqureshi27
 
PDF
lecture01_lecture01_lecture0001_ceva.pdf
byAnaNeacsu5
 
PDF
boyd 3.1
byKoki Isokawa
 
PDF
Numerical analysis convexity, concavity
bySHAMJITH KM
 
PPTX
CPP.pptx
byZHDCADHOCTEACHERS
 
PDF
02 basics i-handout
bysheetslibrary
 
PDF
PROCESS OPTIMIZATION (CHEN 421) LECTURE 1.pdf
byssuser8fa781
 
PPT
Optimization Introduction power point presentation
bymuskaanbhayana9
 
PPT
Optimization Introduction and the basics .ppt
byubaidqureshi27
 
PPT
Optimization Introduction power point presentation
bymuskaanbhayana9
 
PDF
Convex Analysis Pms28 Ralph Tyrell Rockafellar
byrippebunceaw
 
PPT
OI.ppt
byraj20072
 
PDF
35000120030_Aritra Kundu_Operations Research.pdf
byJuliusCaesar36
 
PDF
Optimization Techniques.pdf
byanandsimple
 
Cs229 cvxopt
bycerezaso
 
Optimization introduction
byhelalmohammad2
 
Math Notation using in optimation problem
bytrauvang2871
 
[Convex Optimization_slides Stephen Boyd] S.Boyd L.Vandenberghe .pdf
byBouaziziHechmi1
 
bv_cvxslides (1).pdf
bySantiagoGarridoBulln
 
complete-manual-of-multivariable-optimization.pdf
bySantiagoGarridoBulln
 
Basic presentation of convex optimization 2018.pptx
byubaidqureshi27
 
lecture01_lecture01_lecture0001_ceva.pdf
byAnaNeacsu5
 
boyd 3.1
byKoki Isokawa
 
Numerical analysis convexity, concavity
bySHAMJITH KM
 
CPP.pptx
byZHDCADHOCTEACHERS
 
02 basics i-handout
bysheetslibrary
 
PROCESS OPTIMIZATION (CHEN 421) LECTURE 1.pdf
byssuser8fa781
 
Optimization Introduction power point presentation
bymuskaanbhayana9
 
Optimization Introduction and the basics .ppt
byubaidqureshi27
 
Optimization Introduction power point presentation
bymuskaanbhayana9
 
Convex Analysis Pms28 Ralph Tyrell Rockafellar
byrippebunceaw
 
OI.ppt
byraj20072
 
35000120030_Aritra Kundu_Operations Research.pdf
byJuliusCaesar36
 
Optimization Techniques.pdf
byanandsimple
 

More from adil raja

PDF
ANNs.pdf
byadil raja
 
PDF
A Software Requirements Specification
byadil raja
 
PDF
NUAV - A Testbed for Development of Autonomous Unmanned Aerial Vehicles
byadil raja
 
PDF
DevOps Demystified
byadil raja
 
PDF
On Research (And Development)
byadil raja
 
PDF
Simulators as Drivers of Cutting Edge Research
byadil raja
 
PDF
The Knock Knock Protocol
byadil raja
 
PDF
File Transfer Through Sockets
byadil raja
 
PDF
Remote Command Execution
byadil raja
 
PDF
Thesis
byadil raja
 
PDF
CMM Level 3 Assessment of Xavor Pakistan
byadil raja
 
PDF
Data Warehousing
byadil raja
 
PDF
Implementation of a Non-Intrusive Speech Quality Assessment Tool on a Mid-Net...
byadil raja
 
PDF
Implementation of a Non-Intrusive Speech Quality Assessment Tool on a Mid-Net...
byadil raja
 
PDF
Real-Time Non-Intrusive Speech Quality Estimation for VoIP
byadil raja
 
PDF
VoIP
byadil raja
 
PDF
ULMAN GUI Specifications
byadil raja
 
PDF
Modeling the Effect of Packet Loss on Speech Quality: Genetic Programming Bas...
byadil raja
 
PDF
ULMAN-GUI
byadil raja
 
PDF
Modeling the Effect of Packet Loss on Speech Quality: Genetic Programming Bas...
byadil raja
 
ANNs.pdf
byadil raja
 
A Software Requirements Specification
byadil raja
 
NUAV - A Testbed for Development of Autonomous Unmanned Aerial Vehicles
byadil raja
 
DevOps Demystified
byadil raja
 
On Research (And Development)
byadil raja
 
Simulators as Drivers of Cutting Edge Research
byadil raja
 
The Knock Knock Protocol
byadil raja
 
File Transfer Through Sockets
byadil raja
 
Remote Command Execution
byadil raja
 
Thesis
byadil raja
 
CMM Level 3 Assessment of Xavor Pakistan
byadil raja
 
Data Warehousing
byadil raja
 
Implementation of a Non-Intrusive Speech Quality Assessment Tool on a Mid-Net...
byadil raja
 
Implementation of a Non-Intrusive Speech Quality Assessment Tool on a Mid-Net...
byadil raja
 
Real-Time Non-Intrusive Speech Quality Estimation for VoIP
byadil raja
 
VoIP
byadil raja
 
ULMAN GUI Specifications
byadil raja
 
Modeling the Effect of Packet Loss on Speech Quality: Genetic Programming Bas...
byadil raja
 
ULMAN-GUI
byadil raja
 
Modeling the Effect of Packet Loss on Speech Quality: Genetic Programming Bas...
byadil raja
 

Recently uploaded

PPTX
Top 3 winning teams announcement - TechSprint
bybajpaitusharoffon678
 
PDF
comprehensive analysis of cross-chain bridges in the DeFi - Garima Singh
byGarima Singh
 
PPTX
WOWVerse Workshop: Agentic AI & LLM Workshop
byram27belitkar
 
PPTX
Fake News Detection System | Plagiarism detection
bysurajkanojiya13
 
PDF
Fluid Mechanics, Heat Transfer, and Mass Transfer_ Chemical Engineering Pract...
bynikptl09
 
PDF
Day 01- Basic Knowledge for LPG system design.pdf
bykmas1612
 
PDF
EDIH TRAINING AI FOR COMPANIES: MODULE 4
byHristian Daskalov
 
PDF
God series - Physics video office crossed
byjangidankit8781
 
PDF
ERTMS-conference-WS8_Presentations_v0.pdf
byEduardo147194
 
PDF
JVM in the Age of AI: 2026 Edition - Deep Dive
byArtur Skowroński
 
PDF
Chip Designer's Code - Linux Terminal Part 6 - Text Viewers
byAmr Adel
 
PPTX
MicroAccelerometers & microfluidics.pptx
byThamizhvani TR
 
PPTX
Unit-2: Network Models--OSI Reference Model, TCP/IP Reference Model
bySuvarnaSharma5
 
PPTX
Speech to Text with Tone Correction.pptx
bysurajkanojiya13
 
PDF
Chris Elwell Woburn - An Experienced IT Executive
byChris Elwell Woburn
 
PDF
Chip Designer's Code - Linux Terminal Part 7.1 - Text Processing
byAmr Adel
 
PPTX
Electronics Device - EC25C01 - Semiconductor: Types, Conductivity,
byMathavan N
 
PPTX
The Half-Life of Preventive Maintenance: Why PM Compliance Fails & How to Res...
byMaintWiz Technologies Private Limited
 
PPTX
Explore how AI-powered maintenance checklists transform asset reliability and...
byMaintWiz Technologies Private Limited
 
PPT
Introduction to Concord Decorator by Stolle.ppt
bymorachatgpt123
 
Top 3 winning teams announcement - TechSprint
bybajpaitusharoffon678
 
comprehensive analysis of cross-chain bridges in the DeFi - Garima Singh
byGarima Singh
 
WOWVerse Workshop: Agentic AI & LLM Workshop
byram27belitkar
 
Fake News Detection System | Plagiarism detection
bysurajkanojiya13
 
Fluid Mechanics, Heat Transfer, and Mass Transfer_ Chemical Engineering Pract...
bynikptl09
 
Day 01- Basic Knowledge for LPG system design.pdf
bykmas1612
 
EDIH TRAINING AI FOR COMPANIES: MODULE 4
byHristian Daskalov
 
God series - Physics video office crossed
byjangidankit8781
 
ERTMS-conference-WS8_Presentations_v0.pdf
byEduardo147194
 
JVM in the Age of AI: 2026 Edition - Deep Dive
byArtur Skowroński
 
Chip Designer's Code - Linux Terminal Part 6 - Text Viewers
byAmr Adel
 
MicroAccelerometers & microfluidics.pptx
byThamizhvani TR
 
Unit-2: Network Models--OSI Reference Model, TCP/IP Reference Model
bySuvarnaSharma5
 
Speech to Text with Tone Correction.pptx
bysurajkanojiya13
 
Chris Elwell Woburn - An Experienced IT Executive
byChris Elwell Woburn
 
Chip Designer's Code - Linux Terminal Part 7.1 - Text Processing
byAmr Adel
 
Electronics Device - EC25C01 - Semiconductor: Types, Conductivity,
byMathavan N
 
The Half-Life of Preventive Maintenance: Why PM Compliance Fails & How to Res...
byMaintWiz Technologies Private Limited
 
Explore how AI-powered maintenance checklists transform asset reliability and...
byMaintWiz Technologies Private Limited
 
Introduction to Concord Decorator by Stolle.ppt
bymorachatgpt123
 

Convex Optimization

  • 1.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References CONVEX OPTIMIZATION Muhammad Adil Raja Roaming Researchers, Inc. August 31, 2014
  • 2.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References OUTLINE INTRODUCTION CONVEX SETS CONVEX FUNCTIONS CONVEX OPTIMIZATION PROBLEMS REFERENCES
  • 3.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References OUTLINE INTRODUCTION CONVEX SETS CONVEX FUNCTIONS CONVEX OPTIMIZATION PROBLEMS REFERENCES
  • 4.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References OUTLINE INTRODUCTION CONVEX SETS CONVEX FUNCTIONS CONVEX OPTIMIZATION PROBLEMS REFERENCES
  • 5.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References OUTLINE INTRODUCTION CONVEX SETS CONVEX FUNCTIONS CONVEX OPTIMIZATION PROBLEMS REFERENCES
  • 6.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References OUTLINE INTRODUCTION CONVEX SETS CONVEX FUNCTIONS CONVEX OPTIMIZATION PROBLEMS REFERENCES
  • 7.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References INTRODUCTION I I Many situations arise in machine learning where we would like to optimize the value of some function. I That is, given a function f : Rn ! R, we want to find x 2 Rn that minimizes (or maximizes) f (x). I In the general case, finding the global optimum of a function can be a very difficult task. I However, for a special class of optimization problems, known as convex optimization problems, we can efficiently find the global solution in many cases. I “efficiently” has both practical and theoretical connotations: 1. It means that we can solve many real-world problems in a reasonable amount of time.
  • 8.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References INTRODUCTION II 2. And it means that theoretically we can solve problems in time that depends only polynomially on the problem size. I The goal of these section notes and the accompanying lecture is to give a very brief overview of the field of convex optimization.
  • 9.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References CONVEX SETS I DEFINITION A set C is convex if, for any x, y 2 C and 2 with 0 1, x + (1 )y 2 C. I Intuitively, this means that if we take any two elements in C, and draw a line segment between these two elements, then every point on that line segment also belongs to C. I Figure 1 shows an example of one convex and one non-convex set. I The point x + (1 )y is called a convex combination of the points x and y.
  • 10.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References CONVEX SETS II (a) (b) Figure 1: Examples of a convex set (a) and a non-convex set (b). FIGURE: Examples of a convex set (a) and a non-convex set. Examples All of Rn. It should be fairly obvious that given any x, y ! Rn, !x + (1 − !)y ! Rn. The non-negative orthant, Rn Examples I All of Rn. It should be fairly obvious that given any +. The non-negative orthant consists of all vectors in Rn whose elements are all non-negative: Rn + = {x : xi # 0 $i = 1, . . . , n}. To show x; y 2 n; x + (1 )y 2 n. I The non-negative orthant, n that this is a convex set, simply note that given any x, y ! Rn + and 0 % ! % 1, +. The non-negative (!x + (1 − !)y)i = !xi + (1 − !)yi # 0 $i. orthant consists of all vectors in n whose elements are all non-negative: Norm balls. Let · be some norm on Rn (e.g., ! the Euclidean norm, x2 = n x2i ). Then the set {x : x % 1} is a convex set. To see this, suppose x, y ! Rn,
  • 11.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References CONVEX SETS III 1. n + = x : xi 08i = 1; :::; n. 2. To show that this is a convex set, simply note that given any x; y8Rn+ and 0 ? 1, (x + (1 )y)i = xi + (1 )yi 08i. I Norm balls. Let jj:jj be some norm on n (e.g., the Euclidean norm, x2 = qPni =1 x2 i ). Then the set x : jjxjj 1 is a convex set. I To see this,suppose x; y 2 Rn, with jjxjj 1; jjyjj 1,and 0 1. Then jjx + (1 )yjj jjxjj + jj(1 )yjj = jjxjj + (1 )jjyjj 1 where we used the triangle inequality and the positive homogeneity of norms.
  • 12.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References CONVEX SETS IV I Affine subspaces and polyhedra. Given a matrix A 2 mn and a vector b 2 m, an affine subspace is the set x 2 n : Ax = b (note that this could possibly be empty if b is not in the range of A). Similarly, a polyhedron is the (again, possibly empty) set x 2 Rn : Ax b, where “” here denotes componentwise inequality (i.e., all the entries of Ax are less than or equal to their corresponding element in b). To prove this, first consider x; y 2 n such that Ax = Ay = b. Then for 0 1, A(x+(1)y) = Ax+(1)Ay = b+(1?)b = b. Similarly,for x; y 2 n that satisfy Ax b andAy b and 0 1, A(x+(1)y) = Ax+(1)Ay b+(1)b = b.
  • 13.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References CONVEX SETS V I Intersections of convex sets. Suppose CT1,C2,...,Ck are convex sets. Then their intersection ki Ci kix x Cii 1k is also a convex set. =1 = : 2 8= ; :::; TTo see this, consider x; y 2 1 Ci and 0 =? 1. ki Then, x + (1 )y 2 Ci8i = 1; :::; k by the Tdefinition of a convex set. Therefore x + (1 )y 2 =1 Ci . Note, however, that the union of convex sets in general will not be convex. I Positive semidefinite matrices. The set of all symmetric positive semidefinite matrices, often times called the positive semidefinite cone and denoted Sn+ , is a convex set (in general, Sn nn denotes the set of symmetric n ? n matrices). Recall that a matrix A 2 nn is symmetric positive semidefinite if and only if A = AT and for all x 2 Rn, xT Ax 0. Now consider two symmetric positive semidefinite
  • 14.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References CONVEX SETS VI matrices A;B 2 Sn+ and0 1. Then for any x 2 n, xT (A + (1 )B)x = xT Ax + (1 )xT Bx 0. The same logic can be used to show that the sets of all positive definite, negative definite, and negative semidefinite matrices are each also convex.
  • 15.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References CONVEX FUNCTIONS I DEFINITION A function f : n ! is convex if its domain (denoted D(f)) is a convex set, and if, for all x; y 2 D(f ) and 2 R; 0 ? 1, f (x + (1 )y) f (x) + (1 )f (y). I Intuitively, the way to think about this definition is that if we pick any two points on the graph of a convex function and draw a straight line between them, then the portion of the function between these two points will lie below this straight line. This situation is pictured in Figure 2. I We say a function is strictly convex if Definition 2 holds with strict inequality for x?6= y and 0 1. We say that f is concave if – f is convex, and likewise that f is strictly concave if – f is strictly convex.
  • 16.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References CONVEX FUNCTIONS II convex FIGURE: function. Graph of a convex By function. the definition By the definition of of convex functions, the graph must lie above the function. convex functions, the line connecting two points on the graph must lie above the function. Condition for Convexity
  • 17.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References FIRST ORDER CONDITION FOR CONVEXITY I I Suppose a function f : n ! is differentiable. I The f is convex if and only if D(f ) is a convex set and for all x; y 2 D(f ), f (y) f (x) + 5x f (x)T (y x): I The function 5x f (x)T (y x): is called the first-order approximation to the function f at the point x. I Intuitively, this can be thought of as approximating f with its tangent line at the point x. I The first order condition for convexity says that f is convex if and only if the tangent line is a global underestimator of the function f . I In other words, if we take our function and draw a tangent line at any point, then every point on this line will lie below the corresponding point on f . I A function is differentiable if the gradient 5f (x) exists at all points x in the domain of f .
  • 18.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References FIRST ORDER CONDITION FOR CONVEXITY II I The gradient is defined as 5x f (x) 2 n; (5x f (x))i = @f (x) @(x)i . I Similar to the definition of convexity, f will be strictly convex if this holds with strict inequality, concave if the inequality is reversed, and strictly concave if the reverse inequality is strict.
  • 19.
    corresponding point onf. the definition of convexity, f will be strictly convex if this holds Convex with Optimization concave if the inequality is reversed, and strictly concave if the Muhammad reverse Adil inequality Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References FIRST ORDER CONDITION FOR CONVEXITY III Figure 3: Illustration of first-order condition for convexity. the gradient is defined as xf(x) # Rn, (xf(x))i = !f(x) . For a review on gradients !xi previous section notes on linear algebra. FIGURE: Illustration od the first-order condition for convexity.
  • 20.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References SECOND ORDER CONDITION FOR CONVEXITY I 2x I Suppose a function f : n ! is twice differentiable. I Then f is convex if and only if D(f ) is a convex set and its Hessian is positive semidefinite: i.e., for any x 2 D(f ), 5f (x) 0. I Here, the notation “” when used in conjunction with matrices refers to positive semidefiniteness, rather than componentwise inequality. I In one dimension, this is equivalent to the condition that the second derivative f 00(x) always be positive (i.e., the function always has positive curvature).
  • 21.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References SECOND ORDER CONDITION FOR CONVEXITY II 2x I Again analogous to both the definition and first order conditions for convexity, f is strictly convex if its Hessian is positive definite, concave if the Hessian is negative semidefinite, and strictly concave if the Hessian is negative definite. I A function is twice differentiable if the Hessian 5f (x) is defined for all points x in the domain of f . 2x I The Hessian is defined as 5f (x) 2 nn; 2x (5f (x))ij = @2f (x) @xi@xj . I For a symmetric matrix X 2 Sn;X 0 denotes that X is negative semidefinite. I As with inequalities, “” and “”.
  • 22.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References JENSEN’S INEQUALITY I I Suppose we start with the inequality in the basic definition of a convex function f (x + (1 )y) f (x) + (1 )f (y)for0 1. I Using induction, this can be fairly easily extended to convex combinations of more than one point, f ( Pki =1 ixi ) Pki =1 i f (xi ) for Pki =1 i = 1; i 08i. I In fact, this can also be extended to infinite sums or integrals. I In the latter case, the inequality can be written as f ( R p(x)xdx) R R p(x)f (x)dx for p(x)dx = 1; p(x) 08x. I Because p(x) integrates to 1, it is common to consider it as a probability density, in which case the previous equation can be written in terms of expectations, f (E[x]) E[f (x)]. I This last inequality is known as JensenÕs inequality.
  • 23.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References SUBLEVEL SETS I I Convex functions give rise to a particularly important type of convex set called an -sublevel set. I Given a convex function f : n ! and a real number 2 , the -sublevel set is defined as x 2 D(f ) : f (x) . I In other words, the -sublevel set is the set of all points x such that f (x) . I To show that this is a convex set, consider any x; y 2 D(f ) such that f (x) and f (y) . I Then f (x + (1 )y) f (x) + (1 )f (y) + (1 ) = .
  • 24.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References EXAMPLES OF CONVEX FUNCTIONS I I We begin with a few simple examples of convex functions of one variable, then move on to multivariate functions. I Exponential. Let f : ! ; f (x) = eax for any a 2 . To show f is convex, we can simply take the second derivative f 00(x) = a2eax , which is positive for all x. I Negative logarithm. Let f : ! ; f (x) =?logx with domain D(f ) = R++. I Here, R++ denotes the set of strictly positive real numbers, x : x 0. I Then f 00(x) = 1=x2 0 for all x. I Affine functions. Let f : n ! ; f (x) = bT x + c for some b 2 n, c 2 . I In this case the Hessian, 52xf (x) = 0 for all x.
  • 25.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References EXAMPLES OF CONVEX FUNCTIONS II 12 I Because the zero matrix is both positive semidefinite and negative semidefinite, f is both convex and concave. I In fact, affine functions of this form are the only functions that are both convex and concave. I Quadratic functions. Let f : n ! ; f (x) = xT Ax + bT x + c for a symmetric matrix A 2 Sn, b 2 n and c 2 . I The Hessian for this function is given by 52xf (x) = A. I Therefore, the convexity or non-convexity of f is determined entirely by whether or not A is positive semidefinite: if A is positive semidefinite then the function is convex (and analogously for strictly convex, concave, strictly concave).
  • 26.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References EXAMPLES OF CONVEX FUNCTIONS III 22 I If A is indefinite then f is neither convex nor concave. I Note that the squared Euclidean norm f (x) = jjxjj= xT x is a special case of quadratic ni functions where A = I; b = 0; c = 0, so it is therefore a strictly convex function. I Norms. Let f ;n ! be some norm on n. Then by the triangle inequality and positive homogeniety of norms, for x; y 2 n; 0 1, f (x + (1 )y) f (x) + f (1 )y) = f (x) + (1 )f (y). I This is an example of a convex function where it is not possible to prove convexity based on the second or first order conditions. I The reason is that because norms are not generally differentiable Peverywhere (e.g., the 1-norm, jjxjj1 = =1 jxi j, is non-differentiable at all points where any xi is equal to zero).
  • 27.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References EXAMPLES OF CONVEX FUNCTIONS IV I Nonnegative weighted sums of convex functions. Let f1; f2; :::; fk be convex functions and w1;w2; :::;wk be nonnegative real numbers. Then f (x) = Pki =1 wi fi (x) is a convex function, since f (x + (1?)y) = Pki =1 wi fi (x + (1 )y) (1) Pki =1 wi (fi (x) + (1 )fi (y)) (2) = Pki =1 wi fi (x) + (1 ) Pki =1 wi fi f (y)(3) = f (x) + (1 )f (x) (4)
  • 28.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References CONVEX OPTIMIZATION PROBLEMS I I Armed with the definitions of convex functions and sets, we are now equipped to consider convex optimization problems. I Formally, a convex optimization problem in an optimization problem of the form minimize f (x) subject to x 2 C. I where f is a convex function, C is a convex set, and x is the optimization variable. I However, since this can be a little bit vague, we often write it often written as minimize f (x) subject to gi (x) 0, i = 1; :::;m hi (x) = 0; i = 1; :::; p I where f is a convex function, gi are convex functions, and hi are affine functions, and x is the optimization variable.
  • 29.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References CONVEX OPTIMIZATION PROBLEMS II I It is important to note the direction of these inequalities: 1. A convex function gi must be less than zero. 2. This is because the 0-sublevel set of gi is a convex set, so the feasible region, which is the intersection of many convex sets, is also convex (recall that affine subspaces are convex sets as well). 3. If we were to require that gi 0 for some convex gi , the feasible region would no longer be a convex set, and the algorithms we apply for solving these problems would not longer be guaranteed to find the global optimum. 4. Also notice that only affine functions are allowed to be equality constraints. 5. Intuitively, you can think of this as being due to the fact that an equality constraint is equivalent to the two inequalities hi 0 and hi 0.
  • 30.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References CONVEX OPTIMIZATION PROBLEMS III 6. However, these will both be valid constraints if and only if hi is both convex and concave, i.e., hi must be affine. I The optimal value of an optimization problem is denoted p (or sometimes f ) and is equal to the minimum possible value of the objective function in the feasible region. p = minf (x) : gi (x) 0; i = 1; :::;m; hi (x) = 0; i = 1; :::; p. I We allow p to take on the values +1 and 1 when the problem is either infeasible (the feasible region is empty) or unbounded below (there exists feasible points such that f (x) ! 1), respectively. I We say that x is an optimal point if f (x?) = p. I Note that there can be more than one optimal point, even when the optimal value is finite.
  • 31.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References GLOBAL OPTIMALITY IN CONVEX PROBLEMS I I Defining the concepts of local optima and global optima are in order. I Intuitively, a feasible point is called locally optimal if there are no “nearby” feasible points that have a lower objective value. I Similarly, a feasible point is called globally optimal if there are no feasible points at all that have a lower objective value. I To formalize this a little bit more, following definitions are important.
  • 32.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References GLOBAL OPTIMALITY IN CONVEX PROBLEMS II DEFINITION A point x is locally optimal if it is feasible (i.e., it satisfies the constraints of the optimization problem) and if there exists some R 0 such that all feasible points z with jjx zjj2 R, satisfy f (x) f (z). DEFINITION A point x is globally optimal if it is feasible and for all feasible points z, f (x) f (z). I We now come to the crucial element of convex optimization problems, from which they derive most of their utility.
  • 33.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References GLOBAL OPTIMALITY IN CONVEX PROBLEMS III I The key idea is that for a convex optimization problem all locally optimal points are globally optimal. I LetÕs give a quick proof of this property by contradiction. I Suppose that x is a locally optimal point which is not globally optimal, i.e., there exists a feasible point y such that f (x) f (y). I By the definition of local optimality, there exist no feasible points z such that jjx?zjj2 Randf (z) f (x).
  • 34.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References GLOBAL OPTIMALITY IN CONVEX PROBLEMS IV I But now suppose we choose the point z = y + (1 )x with = R2 jjx?yjj2 Then jjx zjj2 = jjx ( R 2jjxyjj2 y + (1 R 2jjxyjj2)x)jj2 (5) = jj R 2jjxyjj2 (x y)jj2 (6) = R=2 R: (7) I In addition, by the convexity of f we have f (z) = f (y + (1?)x) f (y) + (1?)f (x) f (x). I Furthermore, since the feasible set is a convex set, and since x and y are both feasible z = y + (1?) will be feasible as well. I Therefore, z is a feasible point, with jx?zj2 R and f (z) f (x).
  • 35.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References GLOBAL OPTIMALITY IN CONVEX PROBLEMS V I This contradicts our assumption, showing that x cannot be locally optimal.
  • 36.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References SPECIAL CASES OF CONVEX PROBLEMS I I For a variety of reasons, it is often times convenient to consider special cases of the general convex programming formulation. I For these special cases we can often devise extremely efficient algorithms that can solve very large problems, and because of this you will probably see these special cases referred to any time people use convex optimization techniques. I Linear Programming. We say that a convex optimization problem is a linear program (LP) if both the objective function f and inequality constraints gi are affine functions. I In other words, these problems have the form minimize cT x + d subject toGx h.
  • 37.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References SPECIAL CASES OF CONVEX PROBLEMS II I Ax = b where x 2 n is the optimization variable, c 2 n, d 2 , G 2 mn, h 2 m, A 2 p?n, b 2 Rp are defined by the problem, and “” denotes elementwise inequality. I Quadratic Programming. We say that a convex optimization problem is a quadratic program (QP) if the inequality constraints gi are still all affine, but if the objective function f is a convex quadratic function. I In other words, these problems have the form, minimize 1 2xT Px + cT x + d subject to Gx h + Ax = nb. where again x 2 n is the optimization variable, c 2 n, d 2 , G 2 mn, h 2 Rm, A 2 pn, b 2 Rp are defined by the problem, but we also have P 2 S, a symmetric positive semidefinite matrix.
  • 38.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References SPECIAL CASES OF CONVEX PROBLEMS III I Quadratically Constrained Quadratic Programming. We say that a convex optimization problem is a quadratically constrained quadratic program (QCQP) if both the objective f and the inequality constraints gi are convex quadratic functions, minimize 1 2xT Px + cT x + d subject to 1 2xTQix + r T i x + si 0; i = 1; :::;m Ax = b where, as before, x 2 n is the optimization variable, c 2 n, d 2 , A 2 pn, b 2 p, P 2 Sn+ , but we also have Qi 2 Sn+ ; ri 2 n; si 2 ,for i = 1,...,m. I Semidefinite Programming has become more and more prevalent in many different areas of machine learning research, so you might encounter these at some point, and it is good to have an idea of what they are.
  • 39.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References SPECIAL CASES OF CONVEX PROBLEMS IV I We say that a convex optimization problem is a semidefinite program (SDP) if it is of the form minimizetr (CX) subject to tr (AiX) = bi , i = 1; :::; p X 0. I where the symmetric matrix X 2 Sn is the optimization variable, the symmetric matrices C, A1; :::; Ap 2 Sn are defined by the problem, and the constraint X 0 means that we are constraining X to be positive semidefinite. I This looks a bit different than the problems we have seen previously, since the optimization variable is now a matrix instead of a vector.
  • 40.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References SPECIAL CASES OF CONVEX PROBLEMS V I It should be fairly obvious from the definitions that quadratic programs are more general than linear programs (since a linear program is just a special case of a quadratic program where P = 0), and likewise that quadratically constrained quadratic programs are more general than quadratic programs. I However, what is not obvious at all is that semidefinite programs are in fact more general than all the previous types. I That is, any quadratically constrained quadratic program (and hence any quadratic program or linear program) can be expressed as a semidefinte program. I
  • 41.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References EXAMPLES OF CONVEX OPTIMIZATION I I Support Vector Machines. I Constrained Least Squares. I Maximum likelihood for Logistic Regression.
  • 42.
    Convex Optimization MuhammadAdil Raja Introduction Convex Sets Convex Functions Convex Optimization Problems References REFERENCES I I The images and inspiration for preparing these slides have been taken from Andrew Ngs course on Machine Learning. I The course is available on Stanford engineering for all and Coursera.