Upcoming SlideShare
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Standard text messaging rates apply

Karmarkar's Algorithm For Linear Programming Problem

10,719

Published on

Published in: Technology
11 Likes
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

Views
Total Views
10,719
On Slideshare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
29
0
Likes
11
Embeds 0
No embeds

No notes for slide

Transcript

• 1. Karmarkar’s Algorithm AK Dhamija Introduction Karmarkar’s Algorithm Complexity LP Problem An Interior Point Method of Linear Programming Problem Klee-Minty Example Comparison Original Algorithm AK Dhamija Steps Iterations Transformation DIPR, DRDO Aﬃne Variant Three Concepts Example Concepts 1 & 2 November 20, 2009 & 3: Centering Iterations Rescaling Final Solution Another Example Further Issues References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 1/44
• 2. Overview Karmarkar’s Algorithm 1 Introduction Complexity AK Dhamija LP Problem Klee-Minty Example Introduction Comparison Complexity LP Problem 2 Original Algorithm Klee-Minty Steps Example Iterations Comparison Transformation Original Algorithm 3 Aﬃne Variant Steps Three Concepts Iterations Example Transformation Concepts 1 & 2 Aﬃne Variant & 3: Centering Iterations Three Concepts Rescaling Example Concepts 1 & 2 Final Solution & 3: Centering Another Example Iterations Rescaling 4 Further Issues Final Solution Another Example 5 References Further Issues References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 2/44
• 3. Algorithms Karmarkar’s Algorithm AK Dhamija Introduction Algorithms Complexity Complexity If n is the size of the problem (n may be number of variables in LP Problem) LP Problem Klee-Minty • Time Complexity - Time taken by algorithm Example Comparison • Space Complexity - Memory needed by algorithm √ Original • Increasing order of complexity : lg n, n, n, n lg n, n2 , n3 , 2n , n! Algorithm • O(n2 ) = c1 n2 , O(2n ) = c2 2n Steps Iterations • Compare O(n2 ) = 100n2 , O(2n ) = 2n Transformation • 22500 vs 32768 for n = 15 Aﬃne Variant • 1000000 vs 1267650600228229401496703205376 for n = 100 Three Concepts Example • O(n2 ) = c1 n2 is polynomial time (aka eﬃcient) algorithm and O(2n ) = c2 2n is exponential time (aka ineﬃcient) algorithm Concepts 1 & 2 & 3: Centering Iterations Rescaling Final Solution Another Example Further Issues References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 3/44
• 4. LP Problem Karmarkar’s Algorithm AK Dhamija Problem Instance Introduction • Set of n real variables e.g. x,y etc Complexity LP Problem • Set of restrictions in the form of linear inequalities e.g. x + y ≤ 5, y ≥ 0.2 etc Klee-Minty • Goal e.g. Maximize 4x + y Example Comparison • Solution: (x, y) = (5, 0) Original Algorithm Feasible Region Steps Iterations Transformation • Feasible region is convex Aﬃne Variant • Each constraint generates at most one edge in Three Concepts the feasible region Example Concepts 1 & 2 • As you move the goal line, G, to increase it’s & 3: Centering value, the point that will maximize it is one of Iterations the ’corners’. Rescaling Final Solution Another Example Further Issues References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 4/44
• 5. LP Algorithms Karmarkar’s Naive Algorithm Algorithm • Find all intersection points, check these points to satisfy all constraints i.e feasibility, and ﬁnd out AK Dhamija the point of optimum goal value • O(mn ) for n variables and m constraints - exponential in n Introduction Complexity LP Problem Simplex (George B. Dantzig, 1947) Klee-Minty Example • Start from a feasible corner, and Move to the one that gives the highest value to the goal function Comparison • STOPPING condition : none of the neighboring points gives a higher value than the current one Original Algorithm • Average time is linear; worst case time is exponential (covers all feasible corners) Steps • Remarkably successful in practice Iterations • Integer Programming is NP-complete Transformation Aﬃne Variant Ellipsoidal Method (Leonid Khachiyan, 1979) Three Concepts Example Concepts 1 & 2 • Runtime: polynomial & 3: Centering • Theoretical value; not useful in practice Iterations Rescaling Final Solution Interior point method (Narendra Karmarkar, 1984) Another Example • Runtime: polynomial Further Issues • Practical value, Recently these have become competitive in practice with simplex. References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 5/44
• 6. An example of Exponential Time Karmarkar’s Algorithm Klee-Minty Example AK Dhamija Introduction Complexity LP Problem Klee-Minty Example Comparison Original Algorithm Steps Iterations Transformation Aﬃne Variant Developments Three Concepts Example • The LP has n variables, n constraints and 2n extreme Concepts 1 & 2 & 3: Centering points Iterations Rescaling • The elementary simplex method, starting at x = 0, goes Final Solution Another Example through each of the extreme points before reaching the Further Issues optimum solution at (0, 0, ..., 5n ) References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 6/44
• 7. Klee-Minty Example Karmarkar’s Algorithm Here is the pivot sequence for n = 3, which goes through all 8 AK Dhamija extreme points, starting at the origin. Let s be the slack Introduction variables. Complexity LP Problem Klee-Minty Example Comparison Original Algorithm Steps Iterations Transformation Aﬃne Variant Three Concepts Example Concepts 1 & 2 & 3: Centering Iterations Rescaling Final Solution Another Example Exponential Time Algorithm in Worst case Further Issues References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 7/44
• 8. General Introduction Karmarkar’s Algorithm AK Dhamija Interior-point methods for linear programming Introduction Components Complexity LP Problem • A breakthrough development in linear programming during Klee-Minty Example the 1980s Comparison Original • Started in 1984 by a young mathematician at AT&T Bell Algorithm Steps Labs, N. Karmarkar Iterations Transformation • Polynomial-time algorithm which can solve huge linear Aﬃne Variant Three Concepts programming problems beyond the reach of the simplex Example Concepts 1 & 2 method. & 3: Centering Iterations • Karmarkar’s method stimulated development in both Rescaling Final Solution interior-point and simplex methods. Another Example Further Issues References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 8/44
• 9. Diﬀerence between Interior point methods and the simplex method Karmarkar’s Algorithm AK Dhamija The nature of trial solutions and Complexity Introduction Complexity LP Problem Simplex Method Klee-Minty Example • CPF (Corner Point Feasible) solutions Comparison • Worst Case:No of iterations can increase exponentially in the number of variables n : O(2n ) Original • Practically remarkably successful except some huge problems (eg Airlines Scheduling etc) Algorithm Steps Iterations Transformation Aﬃne Variant Interior Point Methods Three Concepts Example • Interior points (points inside the boundary of the feasible region) Concepts 1 & 2 & 3: Centering • Polynomial time Iterations • Advantageous in solving huge problems, but cumbersome for not-so-large problems Rescaling Final Solution Another Example Further Issues References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 9/44
• 10. Karmarkar’s Original Algorithm Karmarkar’s Algorithm Karmarkar assumes that the LP is given in Canonical form of AK Dhamija the problem Introduction • Min Z = CX Complexity LP Problem • such that AX = 0, 1X = 1, X ≥ 0 Klee-Minty Example Comparison Original Assumptions Algorithm Steps 1 1 1 • X0 = ( n , n , ...., n ) is a feasible solution Iterations Transformation • Minimum(Z) = 0 Aﬃne Variant Three Concepts Example Concepts 1 & 2 & 3: Centering To apply the algorithm to LP problem in standard form, a Iterations Rescaling transformation is needed Final Solution Another • Min Z = CX Example Further Issues • such that AX ≤ b, X ≥ 0 References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 10/44
• 11. Karmarkar’s Original Algorithm Karmarkar’s Algorithm Examples : Standard Vs Canonical Form AK Dhamija Standard Form Introduction Min Z = y1 + y2 (Z = CY) Complexity LP Problem such that Klee-Minty Example Comparison • y1 + 2y2 ≤ 2 (AY ≤ b) Original Algorithm • y1 , y2 ≥ 0 Steps Iterations Transformation Canonical Form Aﬃne Variant Three Concepts Min Z = 5x1 + 5x2 (Z = CX) Example Concepts 1 & 2 such that & 3: Centering Iterations Rescaling • 3x1 + 8x2 + 3x3 - 2x4 = 0 (AX = 0) Final Solution Another Example • x1 + x2 + x3 + x4 = 1 (1X = 1) Further Issues • xj ≥ 0, j = 1, 2, 3, 4 References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 11/44
• 12. Karmarkar’s Original Algorithm Karmarkar’s Algorithm AK Dhamija The Principle Idea Introduction Complexity Create a sequence of points x(0) , x(1) , x(2) , ..., x(k) having decreasing values of the objective function. LP Problem Klee-Minty In the kth step, the point x(k) is brought into the center of the simplexa by projective transformation. Example Comparison a n-dimensional unit simplex S is the set of points(x1 , x2 , ...xn )T satisfying Original Algorithm x1 + x2 + ... + xn = 1 and xj ≥ 0, j = 1, 2, ...n Steps Iterations Transformation Three key Concepts Aﬃne Variant • Projection of a vector onto the set of X satisfying AX = 0 Three Concepts Example • Karmarkar’s Centering Transformation Concepts 1 & 2 • Karmarkar’s Potential Function & 3: Centering Iterations Rescaling Final Solution Another Example Further Issues References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 12/44
• 13. Three Concepts Karmarkar’s Projection Algorithm • we want to move from a feasible point X0 to another feasible point X1 , that for some ﬁxed vector AK Dhamija v, will have a larger value of vX • if we choose to move in direction d = (d1 , d2 , ...dn ) that solves the optimization problem Introduction Max vd such that Complexity Ad = 0, d1 + d2 + ...dn = 0 (so that Ad remains feasible) LP Problem and ||d|| = 1 Klee-Minty then we will be moving in a feasible direction that maximizes the increase in vX per unit length Example moved. Comparison • The direction d that solves this optimization problem is given by the projection of v onto X Original satisfying AX = 0 and x1 + x2 + ... + xn = 0 and is given by [I − BT (BBT )−1 B]v, Algorithm A Steps where B = 1 Iterations Transformation Aﬃne Variant Three Concepts • Geometrically, if we can write v = p + w, where p Example satisﬁes Ap = 0 and w is perpendicular to all Concepts 1 & 2 vectors X satisfying AX = 0, then p is the & 3: Centering projection of v onto the set of X satisfying AX = 0. Iterations Rescaling Final Solution • For Example, v = (−2, −1, 7) is projected onto a Another set of 3-d vectors satisfying x3 = 0(ie x1 − x2 Example plane), then v = (−2, −1, 0) + (0, 0, 7). So here, p = (−2, −1, 0) is the vector in the set of X Further Issues satisfying AX = 0 that is closest to v. References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 13/44
• 14. Three Concepts Karmarkar’s Algorithm Karmarkar’s Centering Transformation AK Dhamija • If xk is a point in S, then f ([x1 , x2 , ...xn ]|xk ) transforms a point [x1 , x2 , ...xn ]T in S into a point [y1 , y2 , ...yn ] in S, where Introduction xj Complexity xk j LP Problem yj = r=n xr r=1 xk Klee-Minty r Example Comparison • Consider the LP Min z = x1 + 3x2 - 3x3 such that Original x2 - x3 = 0 Algorithm x1 + x2 + x3 = 1 Steps xi ≥ 0 Iterations This LP has a feasible solution [ 1 , 1 , 1 ]T and the optimal value of z is 0. 3 3 3 Transformation The feasible point [ 1 , 3 , 8 ]T yields the following transformation 4 8 3 Aﬃne Variant 1 3 3 f ([x1 , x2 , x3 ]|[ 4 , 8 , 8 ]) = Three Concepts 8x2 8x3 Example 4x1 3 3 [x1 , x2 , x3 ]|[ 8x2 8x3 , 8x2 8x3 , 8x2 8x3 ] Concepts 1 & 2 4x1 + + 4x1 + + 4x1 + + 3 3 3 3 3 3 & 3: Centering 1 , 1 , 1 ]|[ 1 , 3 , 3 ]) = [ 12 , 8 , 8 ] Iterations For example, f ([ 3 3 3 4 8 8 28 28 28 Rescaling Final Solution • We now refer to the variables x1 , x2 , ..., xn as being the original space and the variables y1 , y2 , ..., yn as being the transformed space and the unit simplex involving variables Another Example y1 , y2 , ..., yn will be called transformed unit simplex. Further Issues References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 14/44
• 15. Three Concepts Karmarkar’s Algorithm AK Dhamija Properties of Centering Transformation Introduction • f (xk |xk ) = [ n , n , ..., n ]T 1 1 1 Complexity f (.|xk ) maps xk into the center of the transformed unit simplex. LP Problem Klee-Minty • f (x|xk ) ∈ S and For x = x , f (x|xk ) = f (x |xk ) Example Any point in S is transformed into a point in the transformed unit simplex, and no two points in S Comparison can yield the same point (i.e. f is one-one mapping) Original • For any point [y1 , y2 , ...yn ]T in S, there is a unique point [x1 , x2 , ...xn ]T in S satisfying Algorithm f ([x1 , x2 , ...xn ]T |xk ) = [y1 , y2 , ...yn ]T Steps Iterations The point [x1 , x2 , ...xn ]T is given by Transformation xk yj j r=n xk y Aﬃne Variant r=1 r r Three Concepts we can write f −1 ([y1 , y2 , ...yn ]T |xk ) = [x1 , x2 , ...xn ]T Example The above two equations imply that f is a one-one onto mapping from s to S. Concepts 1 & 2 & 3: Centering • A point x) in S will satisfy Ax = 0 if A[diag(xk )]f (x|xk ) = 0 Iterations Feasible points in the original problem correspond to points y in the transformed unit simplex that Rescaling satisfy A[diag(xk )]y = 0 Final Solution Another Example Further Issues References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 15/44
• 16. Karmarkar’s Original Algorithm Karmarkar’s Step k = 0 Algorithm • Start with the solution point Y0 = X0 = ( n , n , ...., n )T 1 1 1 AK Dhamija • Compute step length parameters r = √ 1 n(n−1) and α = (n−1) 3n Introduction Complexity LP Problem Step k Klee-Minty Example Deﬁne Comparison • Dk = diag{Xk } = diag{xk1 , xk2 , ..., xkn } Original Algorithm • P= ADk 1 Steps Iterations Compute Transformation • cp = [I − PT (PPT )−1 P](CDk )T Aﬃne Variant Three Concepts • Ynew = Y0 - αr ||cp || p c Example Concepts 1 & 2 • Xk+1 = 1D Y D Y k new k new & 3: Centering Iterations • Z = CXk+1 Rescaling Final Solution • k=k+1 Another Example • Repeat iteration k until Z becomes less than prescribed tolerance Dk −1 X Further Issues Centering : X is brought to center by Y = −1 1D X k References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 16/44
• 17. Karmarkar’s Original Algorithm Karmarkar’s Algorithm AK Dhamija Introduction Complexity Certain Remarks LP Problem Klee-Minty Example • We move from the center of the transformed unit simplex in a direction opposite to the projection of Comparison CDk )T onto the transformation of the feasible region( the set of y satisfying Adiag{Xk }y = 0). As per our explanation of projection, this ensures that we maintain feasibility(in the transformed Original space) and move in a direction that maximizes the rate of decrease of CDk )T Algorithm Steps • By moving a distance αr from the center of the transformed unit simplex, we ensure that yk+1 Iterations will remain in the interior of the transformed simplex Transformation • When we use the inverse of Karmarkar’s centering transformation to transform yk+1 back into Aﬃne Variant xk+1 , the deﬁnition of projection imply that xk+1 will be feasible for the original LP Three Concepts Example Concepts 1 & 2 & 3: Centering Iterations Rescaling Final Solution Another Example Further Issues References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 17/44
• 18. Karmarkar’s Original Algorithm Karmarkar’s Algorithm AK Dhamija Potential Function : Why do we project CDk )T rather than C)T onto the transformed space? Introduction Complexity • Because we are projecting CDk T rather than CT , we can not be sure that each iteration will LP Problem decrease Z. In fact it is possible for CXk+1 > CXk to occur. Klee-Minty Example • Karmarkar’s Potential Function f (X) is deﬁned as f (X) = Comparison j=n T sumj=1 ln( CX x ), X = [x1 , x2 , ..., xn ]T j Original Algorithm • Karmarkar showed that if we project CDk )T (not C)T ) onto the feasible region in the Steps transformed space, then for some δ > 0, it will be true that for k = 0, 1, 2, ... f (Xk ) Iterations - f (Xk+1 ) ≥ δ. Transformation • So, each iteration of karmarkar’s algorithm decreases the potential function by an amount bounded Aﬃne Variant away from 0. Three Concepts Example • Karmarkar showed that if the potential function evaluated at xk is small enough, the Z = CXk Concepts 1 & 2 will be near 0. Because f (xk ) is decreased by at least δ per iteration, it follows that by choosing k & 3: Centering suﬃciently large, we can ensure that the Z-value for Xk is less than Iterations Rescaling Final Solution Another Example Further Issues References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 18/44
• 19. Karmarkar’s Original Algorithm - Two Iterations Karmarkar’s Algorithm AK Dhamija Problem Min Z = 2x2 - x3 Introduction such that Complexity LP Problem • x1 - 2x2 + x3 = 0 Klee-Minty Example • x1 + x2 + x3 = 1 Comparison • xj ≥ 0, j = 1, 2, 3 Original Solution : z = 0 at (0, 0.33334, 0.66667) Algorithm Steps Iterations Transformation Preliminary Step Aﬃne Variant • k=0 Three Concepts Example • X0 = ( n , n , ...., n )T = ( 1 , 1 , 1 )T 1 1 1 3 3 3 Concepts 1 & 2 & 3: Centering • r= √ 1 n(n−1) = √ 1 3(3−1) = √1 6 Iterations Rescaling • α= (n−1) 3n (3−1) 2 = (3)(3) = 9 Final Solution Another Example Further Issues References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 19/44
• 20. Karmarkar’s Original Algorithm - Two Iterations Karmarkar’s Iteration 0 Algorithm • Y0 = X0 = ( 3 , 1 , 3 )T 1 3 1 AK Dhamija • D0 = diag{X0 } = diag{ 1 , 1 , 3 } 3 3 1 Introduction • AD0 = (1, −2, 1)diag{X0 } = diag{ 1 , 1 , 3 } = ( 1 , −2 , 1 ) 3 3 1 3 3 3 Complexity 1 −2 1 LP Problem Klee-Minty • P= AD0 1 = 3 1 3 1 3 1 Example Comparison • CD0 = (0, 2, −1)diag{ 3 , 3 , 1 } = (0, 3 , −1 ) 1 1 3 2 3 Original Algorithm • PPT = 0.667 0 0 3 ; (PPT )−1 = 1.5 0 0 0.333 Steps   Iterations 0.5 0 0.5 Transformation • PT (PPT )−1 P =  0 1 0  0.5 0 0.5 Aﬃne Variant Three Concepts • cp = [I − PT (PPT )−1 P](CD0 )T = (0.167, 0, −0.167)T Example Concepts 1 & 2 • ||cp || = (0.167)2 + 0 + (−0.167)2 = 0.2362 & 3: Centering ( 2 )( √ ) 1 Iterations • c Ynew = Y0 - αr ||cp || = ( 1 , 3 , 1 )T - p 3 1 3 9 0.2362 6 (0.167, 0, −0.167)T = Rescaling Final Solution (0.2692, 0.3333, 0.3974)T Another Example • X1 = Ynew = (0.2692, 0.3333, 0.3974)T Further Issues • Z = CX1 = (0, 2, −1)(0.2692, 0.3333, 0.3974)T = 0.2692 References • k=0+1=1 AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 20/44
• 21. Karmarkar’s Original Algorithm - Two Iterations Karmarkar’s Iteration 1 Algorithm • D1 = diag{X1 } = diag{0.2692, 0.3333, 0.3974} AK Dhamija • AD1 = (1, −2, 1)diag{X1 } = diag{0.2692, 0.3333, 0.3974} = (0.2692, −0.6666, 0.3974) Introduction −0.6666 Complexity • P= AD1 1 = 0.2692 1 1 0.3974 1 LP Problem Klee-Minty • CD1 = (0, 2, −1)diag{0.2692, 0.3333, 0.3974} = (0, 0.6666, −0.3974) Example Comparison • PPT = 0.675 0 0 3 ; (PPT )−1 = 1.482 0 0 0.333 Original   Algorithm 0.441 0.067 0.492 Steps • PT (PPT )−1 P =  0.067 0.992 −0.059  Iterations 0.492 −0.059 0.567 Transformation • cp = [I − PT (PPT )−1 P](CD1 )T = (0.151, −0.018, −0.132)T Aﬃne Variant Three Concepts • ||cp || = (0.151)2 + (−0.018)2 + (−0.132)2 = 0.2014 Example ( 2 )( √ ) 1 Concepts 1 & 2 • c Ynew = Y0 - αr ||cp || = ( 1 , 3 , 1 )T - p 3 1 3 9 0.2014 6 (0.151, −0.018, −0.132)T = & 3: Centering Iterations (0.2653, 0.3414, 0.3928)T Rescaling Final Solution • D1 Ynew = diag{0.2692, 0.3333, 0.3974}(0.2653, 0.3414, 0.3928)T = Another (0.0714, 0.1138, 0.1561)T ; 1D1 Ynew = 0.3413 Example Further Issues • D Y 1 new = (0.2092, 0.3334, 0.4574)T X2 = 1D Y 1 new References • Z = CX1 = (0, 2, −1)(0.2092, 0.3334, 0.4574)T = 0.2094; k = 1 + 1 = 2 AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 21/44
• 22. Transforming a LP Problem into Karmarkar’s Special Form Karmarkar’s Steps for Conversion of the LP problem : Min Z = CX such that AX ≥ b, X ≥ 0 Algorithm AK Dhamija 1 Write the dual of given primal problem Min Z = bW such that AT W ≤ CT , W ≥ 0 Introduction Complexity 2 1 Introduce slack & surplus variables to primal and dual problems LP Problem 2 Combine these two problems. Klee-Minty Example 3 1 Introduce a bounding constraint xi + wi ≤ K), where K should be suﬃciently large Comparison to include all feasible solutions of original problem. Original 2 Introduce a slack variable in the bounding constraint and obtain xi + wi + s = K Algorithm Steps 4 1 Introduce a dummy variable d (subject to condition d = 1)to homogenize the constraints. Iterations 2 Replace the equations Transformation xi + wi + s = K and d = 1 with the following equations Aﬃne Variant xi + wi + s − Kd = 0 and xi + wi + s + d = K + 1 Three Concepts Example 5 Introduce the following transformation so as to obtain one on the RHS of the last equation Concepts 1 & 2 xj = (K + 1)yj , j = 1, 2, ...m + n & 3: Centering wj = (K + 1)ym+n+j , j = 1, 2, ...m + n Iterations s = (K + 1)y2m+2n+1 Rescaling d = (K + 1)y2m+2n+2 Final Solution Another Example 6 Introduce an artiﬁcial variable y2m+2n+3 (to be minimized) in all the equations such that the sum of the coeﬃcients in each homogenous equation is zero and coeﬃcient of the artiﬁcial variable in the Further Issues last equation is one. References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 22/44
• 23. Example of transforming a LP Problem into Karmarkar’s Special Form Karmarkar’s Steps for Conversion of the LP problem : Max Z = 2x1 + x2 such that x1 + x2 ≤ 5, x1 − x2 ≤ 3, Algorithm x1 , x2 are nonnegative AK Dhamija 1 Write the dual of given primal problem Min Z = 5w1 + 3w2 Introduction such that Complexity w1 + w2 ≥ 2 LP Problem w1 − w2 ≥ 1 Klee-Minty w1 , w2 ≥ 0 Example Comparison 2 Introduction of slack & surplus variables and combination of primal and dual problems x1 + x2 + x3 = 5 Original x1 − x2 + x4 = 3 Algorithm w1 + w2 − w3 = 2 Steps w1 − w2 − w4 = 1 Iterations 2x1 + x2 = 5w1 + 3w2 Transformation Aﬃne Variant 3 Addition of boundary constraint with slack variable s 4 4 i=1 xi + i=1 wi + s = K Three Concepts Example 4 Homogenized equivalent system with dummy variable d Concepts 1 & 2 x1 + x2 + x3 − 5d = 0 & 3: Centering x1 − x2 + x4 − 3d = 0 Iterations w1 + w2 − w3 − 2d = 0 Rescaling Final Solution w1 − w2 − w4 − d = 0 Another 2x1 + x2 − 5w1 − 3w2 = 0 4 4 Example i=1 xi + i=1 wi + s − Kd = 0 4 4 Further Issues i=1 xi + i=1 wi + s + d = (K + 1) with all variables non-negative References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 23/44
• 24. Example of transforming a LP Problem into Karmarkar’s Special Form Karmarkar’s Steps for Conversion of the LP problem : Max Z = 2x1 + x2 such that x1 + x2 ≤ 5, x1 − x2 ≤ 3, Algorithm x1 , x2 are nonnegative AK Dhamija 5 Introduction of transformations xj = (K + 1)yj , j = 1, 2, ...4 Introduction wj = (K + 1)y4+j , j = 1, 2, ...4 Complexity s = (K + 1)y9 LP Problem d = (K + 1)y10 Klee-Minty The system of equations thus obtained is as follows Example y1 + y2 + y3 − 5y10 = 0 Comparison y1 − y2 + y4 − 3y10 = 0 y5 + y6 − y7 − 2y10 = 0 Original y5 − y6 − y8 − y10 = 0 Algorithm 2y1 + y2 − 5y5 − 3y6 = 0 Steps 9 Iterations i=1 yi − Ky10 = 0 10 Transformation i=1 yi = 1 with all variables non-negative Aﬃne Variant Three Concepts 6 Introduce an artiﬁcial variable y11 Example Minimize y11 Concepts 1 & 2 subject to & 3: Centering y1 + y2 + y3 − 5y10 + 2y11 = 0 Iterations y1 − y2 + y4 − 3y10 + 2y11 = 0 Rescaling y5 + y6 − y7 − 2y10 + y11 = 0 Final Solution y5 − y6 − y8 − y10 + 2y11 = 0 Another 2y1 + y2 − 5y5 − 3y6 + 5y11 = 0 Example 9 i=1 yi − Ky10 + (K − 9)y11 = 0 Further Issues 11 i=1 yi = 1 with all variables non-negative References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 24/44
• 25. The Aﬃne Variant of Algorithm Karmarkar’s Algorithm AK Dhamija Introduction Three Concepts Complexity LP Problem • Concept 1: Shoot through the interior of the feasible Klee-Minty Example Comparison region towards an optimal solution Original • Concept 2: Move in the direction that improves the Algorithm Steps objective function value at the fastest feasible rate. Iterations Transformation • Concept 3: Transform the feasible region to place the Aﬃne Variant Three Concepts current trail solution near its center, thereby enabling the Example Concepts 1 & 2 fastest feasible rate for Concept 2 & 3: Centering Iterations Rescaling Final Solution Another Example Further Issues References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 25/44
• 26. An Example Karmarkar’s The Problem Algorithm Max Z = x1 + 2x2 AK Dhamija such that • x1 + x2 ≤ 8 Introduction Complexity • xj ≥ 0, j = 1, 2 LP Problem The optimal Solution is (x1 , x2 ) = (0, 8) with Z = 16 Klee-Minty Example Comparison Original Algorithm Steps Iterations Transformation Aﬃne Variant Three Concepts Example Concepts 1 & 2 & 3: Centering Iterations Rescaling Final Solution Another Example Further Issues References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 26/44
• 27. An Example Karmarkar’s Concept 1: Shoot through the interior of the feasible region toward an optimal solution Algorithm • The algorithm begins with an initial solution that lies in the interior of the feasible region AK Dhamija • Arbitrarily choose (x1 , x2 ) = (2, 2) to be the initial solution Introduction Complexity Concept 2: Move in a direction that improves the objective function value at the fastest feasible rate LP Problem Klee-Minty • The direction is perpendiculars to (and toward) the objective function line. Example Comparison • (3, 4) = (2, 2) + (1, 2), where the vector (1, 2) is the gradient( aka Coeﬃcients) of the Objective Function Original Algorithm Steps Iterations Transformation Aﬃne Variant Three Concepts Example Concepts 1 & 2 & 3: Centering Iterations Rescaling Final Solution Another Example Further Issues References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 27/44
• 28. An Example Karmarkar’s The algorithm actually operates on the augmented form Algorithm Max Z = CX such that AK Dhamija • AX = b • X≥0 Introduction Complexity LP Problem The Transformation Klee-Minty Example ⇒ Max Z = x1 + 2x2 = Max Z = x1 + 2x2 such that Comparison • x1 + x2 ≤ 8 = x1 + x2 + x3 = 8, where x3 is the slack ⇒ Original Algorithm • xj ≥ 0, j = 1, 2 = xj ≥ 0, j = 1, 2, 3 ⇒ Steps The optimal Solution is (x1 , x2 ) = (0, 8) with Z = 16 Iterations Transformation Aﬃne Variant The Matrices Three Concepts     x1 0 Example C= 1 2 0 X =  x2  A = 1 1 1 b= 8 0= 0  Concepts 1 & 2 x3 0 & 3: Centering Iterations Rescaling Final Solution The Solution Another Example • ⇒ ⇒ Initial Solution (2, 2) = (2, 2, 4), Optimum (0, 8) = (0, 8, 0) Further Issues • ⇒ Gradient of Objective Function (1, 2) = (1, 2, 0) References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 28/44
• 29. An Interior-point Algorithm Gradient of Objective function : C = 1 2 0 Karmarkar’s Algorithm Using Projected Gradient to implement Concept 1 & 2 AK Dhamija • ⇒ Adding the gradient to the initial leads to (3, 4, 4) = (2, 2, 4) + (1, 2, 0) = Infeasible • To remain feasible, the algorithm projects the point (3, 4, 4) down onto the feasible tetrahedron Introduction Complexity • The next trial solution moves in the direction of projected gradient i.e. the gradient projected onto the feasible region LP Problem Klee-Minty Example Comparison Original Algorithm Steps Iterations Transformation Aﬃne Variant Three Concepts Example Concepts 1 & 2 & 3: Centering Iterations Rescaling Final Solution Another Example Further Issues References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 29/44
• 30. An Interior-point Algorithm Karmarkar’s Using Projected Gradient to implement Concept 1 & 2 Algorithm   2 −1 −3 1 3 3 AK Dhamija • Projection Matrix P = I − AT (AAT )−1 A =  − 1  3 2 3 1  −3  −13 −13 2 3 Introduction   0 Complexity LP Problem • T Projected Gradient cP = PC =  1  Klee-Minty −1 Example       Comparison 2 2 0 • Move (2, 2, 4) towards cP to X =  2  + 4αcP =  2  + 4α  1  Original 4 4 −1 Algorithm Steps • α determines how far we move. large α ⇒ too close to the boundary and α ⇒ more iterations we Iterations have chosen α = 0.5, so the new trial solution move to: X = (2, 3, 3) Transformation Aﬃne Variant Three Concepts Example Concepts 1 & 2 & 3: Centering Iterations Rescaling Final Solution Another Example Further Issues References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 30/44
• 31. An Interior-point Algorithm Karmarkar’s Algorithm Concept 3: Transform the feasible region to place the current trail AK Dhamija solution near its center, thereby enabling a large improvement when Introduction concept 2 is implemented Complexity LP Problem Klee-Minty Centering scheme for implementing Concept 3 Example Comparison • Why : The centering scheme keeps turning the direction of the Original Algorithm projected gradient to point more nearly toward an optimal Steps solution as the algorithm converges toward this solution Iterations Transformation • How : Simply changing the scale for each of the variable so that Aﬃne Variant Three Concepts the trail solution becomes equidistant from the constraint Example Concepts 1 & 2 boundaries in the new coordinate system & 3: Centering Iterations ˜ • x is brought to the center X = (1, 1, 1) in the new coordinate Rescaling Final Solution system by X˜ = D−1 X, where D = diag{X} Another Example Further Issues References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 31/44
• 32. An Interior-point Algorithm Karmarkar’s Centering scheme for implementing Concept 3 Algorithm     1 0 0  1 0 0    x1 2 1 AK Dhamija  2 2 X = D−1 X =  0 ˜ 1 0   x2  =  0 1 0 2 = 1     2 2  0 0 1 x3 0 0 1 4 1 4 4 Introduction Complexity LP Problem The Problem in the new coordinate system becomes Klee-Minty Example Max Z = 2˜1 + 4˜2 x x Comparison such that Original Algorithm • 2˜1 + 2˜2 + 4˜3 = 8 x x x Steps • xj ≥ 0, j = 1, 2, 3 ˜ Iterations Transformation Aﬃne Variant Three Concepts Example Concepts 1 & 2 & 3: Centering Iterations Rescaling Final Solution Another Example Further Issues References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 32/44
• 33. Complete Illustration of the Algorithm Karmarkar’s Algorithm Iteration 1 AK Dhamija • Initial Solution (x1 , x2 , x3 ) = (2, 2, 4)     1 0 0 1 0 0  x1 Introduction •  2 1  ˜ −1 2 1 D = diag{x1 , x2 , x3 } =  0 0 X=D X= 0 0 x2    2 2  Complexity 0 0 1 0 0 1 x3 LP Problem   4 4 1 0 0    Klee-Minty 2 1 Example  2 1 = 0 0  2 = 1   Comparison 2 0 0 1 4 1 4 Original   Algorithm 2 0 0 Steps • ˜ A = AD = 1 1 1  0 2 0 = 2 2 4 Iterations 0 0 4 Transformation  2 0 0  1  Aﬃne Variant • ˜ CT = DCT =  0 2 0  2 = 2 4 0 Three Concepts 0 0 4 0 Example   5 −1 −1 Concepts 1 & 2 6 6 3 & 3: Centering • Projection Matrix P = I − AT (AAT )−1 A =  − 1 ˜ ˜ ˜ ˜  6 5 6 −1 3   Iterations −13 −13 1 3 Rescaling   Final Solution 1 Another • Projected Gradient cP = PCT˜ = 3  Example −2 Further Issues References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 33/44
• 34. Complete Illustration of the Algorithm Karmarkar’s Iteration 1 Algorithm • Identify v (how far to move) as the absolute value of the negative component of cP having the AK Dhamija largest value, so that v = | − 2| = 2   1 Introduction • In this coordinate system the algorithm moves from current solution to x =  1  + α cP = ˜ v Complexity 1   LP Problem     5 1 1 4 Klee-Minty  1  + 0.5  3  =  7  Example 2  4  1 −2 1 Comparison 2 Original Algorithm Steps Iterations Transformation Aﬃne Variant Three Concepts Example Concepts 1 & 2 & 3: Centering Iterations Rescaling Final Solution Another Example Further Issues References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 34/44
• 35. Complete Illustration of the Algorithm Karmarkar’s Algorithm Iteration 1     AK Dhamija x1 2 0 0 • ˜ In the original coordinate system the solution is X =  x2  DX =  0 2 0  x3 0 0 4 Introduction   5  5  4  2 Complexity  7  4 = 7  2 LP Problem 1 Klee-Minty 2 2 Example Comparison Original Algorithm Steps Iterations Transformation Aﬃne Variant Three Concepts Example Concepts 1 & 2 & 3: Centering Iterations Rescaling Final Solution Another Example Further Issues • This completes the iteration 1. The new solution will be used to start the next iteration References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 35/44
• 36. Complete Illustration of the Algorithm Karmarkar’s Algorithm Iteration 2 AK Dhamija • Current trial Solution (x1 , x2 , x3 ) = ( 5 , 7 , 2) 2 2  5    2 0 0 1 Introduction • D = diag{x1 , x2 , x3 } =  0 7 2 0 X=D ˜ −1 X= 1  Complexity 0 0 2 1 LP Problem     13 − 187 −9 2 − 11 Klee-Minty 18 12  Example • P =  − 18  7 41 90 − 14  cP =  133  45   60 Comparison 2 −9 − 14 45 37 45 − 41 15 Original Algorithm • The current solution becomes (0.83, 1.4, 0.5) which corresponds (2.08, 4.92, 1.0) in the original coordinate system Steps Iterations Transformation Aﬃne Variant Three Concepts Example Concepts 1 & 2 & 3: Centering Iterations Rescaling Final Solution Another Example Further Issues References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 36/44
• 37. Complete Illustration of the Algorithm Karmarkar’s Algorithm Iteration 3 AK Dhamija • Current trial Solution (x1 , x2 , x3 ) = (2.08, 4.92, 1.0)   2.08 0 0 Introduction • D = diag{x1 , x2 , x3 } =  0 4.92 0  Complexity 0 0 1.0 LP Problem   −1.63 Klee-Minty Example • cP =  1.05  Comparison −1.79 Original • The current solution becomes (0.54, 1.3, 0.5) which corresponds (1.13, 6.37, 0.5) in the original Algorithm coordinate system Steps Iterations Transformation Aﬃne Variant Three Concepts Example Concepts 1 & 2 & 3: Centering Iterations Rescaling Final Solution Another Example Further Issues References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 37/44
• 38. Complete Illustration of the Algorithm Karmarkar’s Eﬀect of rescaling of each iteration Algorithm Sliding the optimal solution toward (1, 1, 1) while the other BF solutions tend to slide away AK Dhamija Introduction Complexity LP Problem Klee-Minty Example Comparison Original Algorithm Steps Iterations Transformation (1) (2) Aﬃne Variant Three Concepts Example Concepts 1 & 2 & 3: Centering Iterations Rescaling Final Solution Another Example Further Issues (3) (4) References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 38/44
• 39. Complete Illustration of the Algorithm Karmarkar’s Algorithm More iterations AK Dhamija Starting from the current trial solution x, following the steps, x is moving toward the optimum (0, 8). When the trial solution is virtually unchanged from the proceeding one, then the algorithm has virtually Introduction converged to an optimal solution. We stop. Complexity LP Problem Klee-Minty Example Comparison Original Algorithm Steps Iterations Transformation Aﬃne Variant Three Concepts Example Concepts 1 & 2 & 3: Centering Iterations Rescaling Final Solution Another Example Further Issues References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 39/44
• 40. Another Example Karmarkar’s Algorithm The Problem AK Dhamija Max Z = 5x1 + 4x2 such that Introduction Complexity • 6x1 + 4x2 ≤ 24 LP Problem • x1 + 2x2 ≤ 6 Klee-Minty Example • −x1 + x2 ≤ 1 Comparison • x2 ≤ 2 Original • xj ≥ 0, j = 1, 2 Algorithm Steps Iterations Augmented Form Transformation Aﬃne Variant Max Z = 5x1 + 4x2 such that Three Concepts Example Concepts 1 & 2 • 6x1 + 4x2 + x3 = 24 & 3: Centering • x1 + 2x2 + x4 = 6 Iterations Rescaling • −x1 + x2 + x5 = 1 Final Solution • x2 + x6 = 2 Another Example • xj ≥ 0, j = 1, 2, 3, 4, 5, 6 Further Issues References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 40/44
• 41. Another Example Karmarkar’s The Matrices Algorithm     6 4 1 0 0 0 24 AK Dhamija  1 2 0 1 0 0   6  A=  −1  1 C= b=  5 4 0 0 0 0 1 0 0 1 0  Introduction 0 1 0 0 0 1 2 Complexity LP Problem Klee-Minty Example Comparison Original Algorithm Steps Iterations Transformation Aﬃne Variant Three Concepts Example Concepts 1 & 2 & 3: Centering Iterations Rescaling Final Solution Another Example Further Issues References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 41/44
• 42. Another Example Karmarkar’s Algorithm Starting from an initial solution X = (1, 1, 14, 3, 1, 1) AK Dhamija Introduction The Steps Complexity LP Problem D = diag{X} Klee-Minty Example ˜ A = AD Comparison Original ˜ c = DC Algorithm Steps ˜ ˜˜ ˜ P = I − AT (AAT )−1 A Iterations Transformation cP = P˜ c Aﬃne Variant   Three Concepts Example 1 ˜ = 1 + X α v cP Concepts 1 & 2 & 3: Centering Iterations Rescaling 1 Final Solution Another X = DX˜ Example Further Issues References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 42/44
• 43. Further Issues Karmarkar’s Algorithm AK Dhamija Interior-point methods is designed for dealing with big problems. Although the claim that it’s much faster than the simplex method is controversy, many tests on huge LP problems show its outperformance Introduction Complexity LP Problem Future Research Klee-Minty Example • Infeasible interior points method - remove the assumption that there always exits a nonempty interior Comparison • Methods applying to LP problems in standard form Original • Methods dealing with ﬁnding initial solution, and estimating the optimal solution Algorithm Steps • Methods working with primal-dual problems Iterations • Studies about moving step-long/short steps Transformation • Studies about eﬃcient implementation and complexity of various methods Aﬃne Variant Three Concepts Example Karmarkar’s paper not only started the development of interior point methods, but also encouraged rapid Concepts 1 & 2 & 3: Centering improvement of simplex methods Iterations Rescaling Final Solution Another Example Further Issues References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 43/44
• 44. References Karmarkar’s References Algorithm • N. Karmarkar, A New Polynomial - Time Algorithm for Linear Programming, Combinatorica 4 (4), AK Dhamija 1984, p. 373-395 • Yanrong Hu & Ce Yu, Paper review of ENGG*6140 Optimization Techniques – Interior-Point Introduction Methods for LP, 2003 Complexity • I. Lustig, R. Marsten, D. Shanno, Interior-point Methods for Linear Programming: Computational LP Problem State of the Art, ORSA Journal on Computing, 6 (1), 1994, p. 1-14 Klee-Minty Example • Hillier,Lieberman,Introduction to Operations Research (7th edition) 320-334 Comparison • Taha, Operations Research: An Introduction (6th edition) 336-345 Original • D. Gay, A Variant of Karmarkar’s Linear Programming Algorithm for Problems in Standard form, Algorithm Mathematical Programming 37 (1987) 81-90 Steps Iterations • E.R. Barnes, A Variation on Karmarkar’s Algorithm for Solving Linear Programming problems, Transformation Mathematical Programming 36, 1986, p. 174-182 Aﬃne Variant • V. Klee and G.J. Minty, How Good is the Simplex Algorithm? In O. Shisha, editor, Inequalities, III, pages 159-175. Academic Press, New York, NY, 1972. Three Concepts Example • Richard Bronson and Govindasamy Naadimuthu, Schaum’s Outlines, Operations Research, Second Concepts 1 & 2 Edition, Mcgraw Hill. & 3: Centering Iterations • Wayne L Winston, Operations Research, Applications and Algorithms, Fourth Edition, Thomson Rescaling Brooks/Cole, 2007 Final Solution Another Example Presentation available at Further Issues www.geocities.com/a_k_dhamija. References AK Dhamija, DIPR, DRDO Karmarkar’s Algorithm, An Interior Point Method of Linear Programming Problem 44/44