Download to read offline
![ANPARA THERMAL POWER STATION[1] sangam.pdf](https://cdn.slidesharecdn.com/ss_thumbnails/anparathermalpowerstation1sangam-251121115219-9261cde4-thumbnail.jpg?width=640&height=640&fit=bounds)
Basic presentation of convex optimization 2018.pptx
- 1. Artificial Intelligence: Representationand Problem Solving Optimization (1): Optimization and Convex Optimization 15-381 / 681 Instructors: Fei Fang (This Lecture) and Dave Touretzky feifang@cmu.edu Wean Hall 4126 04/03/2025 1
- 2. 04/03/2025 Fei Fang 2 Logistics Completethe CMU 'course hours worked' survey
- 3. 04/03/2025 Fei Fang 3 Recap Whatwe have learned so far… Search & Satisfiability in discrete space 8-queen, Graph coloring, 3-SAT,… Finite options This section: Generalize to discrete / continuous space
- 4. 04/03/2025 Fei Fang 4 Outline Optimization Convex Optimization
- 5. 04/03/2025 Fei Fang 5 Optimization Problem:Definition Optimization Problem: Determine value of optimization variable within feasible region/set to optimize optimization objective Optimization variable Feasible region/set Optimization objective Optimal solution: Optimal objective value s.t.
- 6. 04/03/2025 Fei Fang 6 Optimization Problem:Definition Optimization variable Discrete variables: Combinatorial optimization Continuous variables: Continuous optimization Mixed: Some variables are discrete, and some are continuous s.t.
- 7. 04/03/2025 Fei Fang 7 Optimization Problem:Definition Unconstrained optimization: Constrained optimization: Find a feasible point can already be difficult s.t.
- 8. 04/03/2025 Fei Fang 8 Optimization Problem:Definition Optimization objective : Feasibility problem Simple functions Linear function Convex function (this lecture) Complicated functions Even can be implicitly represented through an algorithm which takes as input, and outputs a value s.t.
- 9. 04/03/2025 Fei Fang 9 Optimization Problem:Definition Minimization can be converted to maximization (and vice versa) s.t. s.t. Same optimal solution Optimal objective value
- 10. 04/03/2025 Fei Fang 10 Optimization Problem:Example Example:Traveling Salesman Problem (TSP) Problem: cities, distance from city to city is , find a tour (a closed path that visits every city exactly once) with minimal total distance Variable : ordered list of cities being visited is the index of the city being visited Feasible set Objective function s.t.
- 11. 04/03/2025 Fei Fang 11 Optimization Problem:Example Example: 8-Queens Problem Variable : location of the queen in each column is the row index of the queen in column Feasible set Objective function (dummy) Variable : index of row and column of the queen Feasible set ; ; Objective function (dummy) s.t.
- 12. 04/03/2025 Fei Fang 12 Optimization Problem:Example Example: Linear Regression Problem: Find such that , Variable Feasible region Objective function ?
- 13. 04/03/2025 Fei Fang 13 Optimization Problem:How to Solve No general way to solve Many algorithms developed for special classes of optimization problems (i.e., when and satisfy certain constraints Convex optimization problem (CO) Linear Program (LP) (Mixed) Integer Linear Program (MILP) Quadratic program (QP), (Mixed) Integer Quadratic program (MIQP), Semidefinite program (SDP), Second-order cone program (SOCP), … Existing solvers and code packages for these problems Cplex (LP, MILP, QP), Gurobi (LP, MILP, MIQP), GLPK (LP, MILP), Cvxopt (CO), DSDP5 (SDP), MOSEK (QP, SOCP),Yalmip (SDP), …
- 14. 04/03/2025 Fei Fang 14 Optimization Problem:WhyUseful Why formulate problems as optimization problems? For many class of optimization problems, algorithms or algorithmic frameworks have been developed Decouple “representation” and “problem solving” Lazy mode Formulate a problem as an optimization problem Identify which class the formulation belongs to Call the corresponding solver Done!
- 15. 04/03/2025 Fei Fang 15 Convex Optimization:Definition Convex Optimization Problem: An optimization problem whose optimization objective is a convex function and feasible region is a convex set A special class of optimization problem s.t.
- 16. 04/03/2025 Fei Fang 16 Convex Optimization:Definition Convex combination A point between two points Given , a convex combination of them is any point of the form where When , is called a strict convex combination of 𝑥 𝑦 𝑧
- 17. 04/03/2025 Fei Fang 17 Convex Optimization:Definition Convex set Any convex combination of two points in the set is also in the set A set is convex if , ,
- 18. 04/03/2025 Fei Fang 18 Quiz 1:Convex Set Which of the following sets are convex? A: B: C: D: , where and are convex sets E:
- 19. 04/03/2025 Fei Fang 19 Convex Optimization:Definition Convex function Value in the middle point is lower than average value Let be a convex set.A function is convex in if , If , we simply say is convex
- 20. 04/03/2025 Fei Fang 20 Convex Optimization:Definition How to determine if a functions is convex? Prove by definition Use properties Sum of convex functions is convex If convex, then is convex Convexity is preserved under a linear transformation If , convex, then is convex If is a twice differentiable function of one variable, is convex on an interval iff (if and only if) its second derivative in
- 21. 04/03/2025 Fei Fang 21 Convex Optimization:Definition (not required) If is a twice continuously differentiable function of variables, is convex on iff its Hessian matrix of second partial derivatives is positive semidefinite on the interior of is positive semidefinite in if , is positive semidefinite in iff all eigenvalues of are non-negative Alternatively, prove
- 22. 04/03/2025 Fei Fang 22 Convex Optimization:Definition Concave function A function is concave if is convex Let be a convex set.A function is concave in if , The following is a convex optimization problem if is a concave function and is a convex set s.t.
- 23. 04/03/2025 Fei Fang 23 Convex Optimization:Definition Affine function An affine function is a function of the form where If a function is both convex and concave in , then is an affine function Proof (not required): Let . Clearly is convex and concave. By definition, satisfies (i) ; (ii) . Let be the canonical basis of , then . Therefore where . So
- 24. 04/03/2025 Fei Fang 24 Quiz 2:Convex Function Which of the following functions are convex? A: B: C: D: E: Counter Example for D:, , . , http://m.wolframalpha.com/input/?i=z%3D-xy
- 25. 04/03/2025 Fei Fang 25 Convex Optimization:Local Optima=Global Optima Given an optimization problem, a point is globally optimal if and Given an optimization problem, a point is locally optimal if and such that and Theorem 1: For a convex optimization problem, all locally optimal points are globally optimal s.t.
- 26. 04/03/2025 Fei Fang 26 Convex Optimization:Local Optima=Global Optima Proof ofTheorem 1: Prove by contradiction. (1) due to convexity of (2) (3) So is not local optima. Contradiction. Assume is a local optima with optimality radius , and . Let where .
- 27. 04/03/2025 Fei Fang 27 Convex Optimization:How to Solve Recall discrete setting Local search Iteratively improving an assignment Continuous and differentiable setting Iteratively improving value of Based on gradient
- 28. 04/03/2025 Fei Fang 28 Convex Optimization:How to Solve For , gradient is the vector of partial derivatives A multi-variable generalization of the derivative Point in the direction of steepest increase in
- 29. 04/03/2025 Fei Fang 29 Convex Optimization:How to Solve Gradient descent: iteratively update the value of A simple algorithm for unconstrained optimization Variants How to choose , e.g., How to update , e.g., How to define “convergence”, e.g., Algorithm: Gradient Descent Input: function , initial point , step size Initialize Repeat Until convergence
- 30. 04/03/2025 Fei Fang 30 Convex Optimization:How to Solve Theorem 2: For differentiable and small enough , for any with , can be convex or non-convex (Informal) For convex and differentiable and small enough , gradient descent converges to global optimum Proof (not required): (Recall Taylor expansion ) Consider a small , use Taylor expansion Last inequality holds for since
- 31. 04/03/2025 Fei Fang 31 Convex Optimization:How to Solve Projected Gradient Descent: iteratively update the value of while ensuring projects a point to the constraint set Variants How to choose , e.g., Algorithm: Projected Gradient Descent Input: function , initial point , step size Initialize Repeat Until convergence
- 32. 04/03/2025 Fei Fang 32 Convex Optimization:How to Solve Unconstrained and differentiable Gradient descent Set derivative to be 0 Closed form solution (Not covered) Newton’s Method (if twice differentiable) Constrained and differentiable Projected gradient descent (Not covered) Interior Point Method (Not covered) Non-differentiable -Subgradient Method Cutting Plane Method
- 33. 04/03/2025 Fei Fang 33 Convex Optimization:Apply Model a problem as a convex optimization problem Define variable, feasible set, objective function Prove it is convex (convex function + convex set) Solve the convex optimization problem Build up the model Call a solver Examples: fmincon (MATLAB), cvxpy (Python), cvxopt (Python), cvx (MATLAB) Map the solution back to the original problem
- 34. 04/03/2025 Fei Fang 34 Summary Optimization Problems ConvexPrograms Gradient Descent
- 35. 04/03/2025 Fei Fang 35 Convex Optimization:AdditionalResources Text book Convex Optimization, Chapters 1-4 Stephen Boyd and LievenVandenberghe Cambridge University Press https://web.stanford.edu/~boyd/cvxbook/ Online course Stanford University, Convex Optimization I (EE 364A), taught by Stephen Boyd http://ee364a.stanford.edu/courseinfo.html https://youtu.be/McLq1hEq3UY
- 36. 04/03/2025 Fei Fang 36 Acknowledgment Someslides are borrowed from previous slides made by J. Zico Kolter
Editor's Notes
- #3 Properties Finite options (brute force when the problem scale is small) (Often) NP-Complete Solution approach Brute force when the problem scale is small
- #5 X: vector with n real-valued entries Many important problems are about choosing a best configuration or a set of parameters to achieve some goal. f: F-> R^1 Optimal solution: Best value of optimization variables Optimal value: Best value of optimization objective
- #6 Can be discrete or continuous valued variables or mixed Example: shopping with given budget Shortest path, Traveling salesman problem
- #7 Example: shopping with given budget Shortest path, Traveling salesman problem
- #8 can be as simple as a linear function f(x) may even lack a closed form, but can only
- #9 Can be discrete or continuous valued variables or mixed Example: shopping with given budget Shortest path, Traveling salesman problem
- #10 Combinatorial optimization problem is the indicator function
- #12 Continuous Optimization Problem
- #14 Great news for practical use: represent a problem as an optimization problem, call a solver to get a solution or follow the existing solution framework to design the algorithm to get a solution Research efforts focus on: (i) represent a problem in the right form, convert problem formulations; (ii) design algorithms and algorithmic frameworks for a class of optimization problems
- #15 Intuitively, convex means the shape of it curve outwards. A convex set in a 2-D space is a shape whose surface has no indentations. A convex function is one whose shape bends up. Draw examples
- #18 A, B, C, D
- #19 (Chords lie above the function) Draw a plot
- #23 Let . Clearly is convex and concave. By definition, satisfies (i) ; (ii) . Let be the canonical basis of , then . Therefore where . So
- #24 A, B, C Counter Example for D:, , . ,
- #25 Key property: all local optima are global optima
- #26 Assume is a local optima with optimality radius , and . Let where . (1) due to convexity of (2) (3) So is not local optima. Contradiction.
- #29 Draw a figure for 1-D case and 2-D case
- #31 Draw a figure for 1-D case and 2-D case