Computational complexityMeasure the degree of difficulty of an algorithm
Background What is an Algorithm? A problem can be solved by many algorithms   E.g., sorting data An algorithm is a method ...
Why consider Efficiency? There are often many algorithms to solve a problem How do we choose between them? (Usually) confl...
(Un programma che funziona) in fretta               vs.Un programma che (funziona in fretta)                Tecniche di pr...
Why consider Efficiency?                from the LinkedList javadoc page: There are often many algorithms to solve a probl...
How to Measure Efficiency? Critical resources   programmer’s effort   time, space (disk, RAM) Analysis   empirical (run pr...
Empirical Approach Implement each candidate   That could be lots of work – also error-prone Run it   Which inputs?   Worst...
Analytical Approach How to solve “which algorithm” problems without machines nor test data?                        Tecnich...
Analytical Approach An algorithm is a mapping For most algorithms, running time depends on “size” of the input Running tim...
Bubble sort              Tecniche di programmazione   A.A. 2012/2013
Analysis The bubble sort takes (n2-n)/2 “steps” Different implementations/assembly languages  Program A on an Intel Pentiu...
Analysis As a result:   Program A on Intel Pentium IV: T(n) ≈ 29n2   Program B on Motorola: T(n) ≈ 42n2   Program C on Int...
Analysis As processors change, the constants will always change   The exponent on n will not   We should not care about th...
Intuitive motivations Asymptotic notation captures behavior of functions for large values of x. Dominant term of 3x3 + 5x2...
[0, 2]         y = 3x 3+5x 2 –9                        y=x3                                                 y=x2          ...
[0, 5]         Tecniche di programmazione   A.A. 2012/2013
[0, 10]          Tecniche di programmazione   A.A. 2012/2013
[0, 100]                                    y = 5x 3                          y = 3x 3+5x 2 –9           Tecniche di progr...
Complexity Analysis O( · )   big o (big oh) Ω( · ) big omega Θ( · )   big theta                    Tecniche di programmazi...
O( · ) Upper Bounding Running Time Why?   Little-oh   “Order of”   D’Oh                                                   ...
Upper Bounding Running Time f(n) is O(g(n)) if f grows “at most as fast as” g21                       Tecniche di programm...
Big-O (formal) Let f and g be two functions such that            f (n ) : N → R + and g (n ) : N → R + if there exists pos...
Big-O (formal alt) Let f and g be two functions such that            f (n ) : N → R + and g (n ) : N → R + if there exists...
Example (log n)2 = O(n)24                 Tecniche di programmazione   A.A. 2012/2013
Notational Issues Big-O notation is a way of comparing functions Notation quite unconventional   e.g., 3x3 + 5x2 – 9 = O(x...
Common Misunderstanding 3x3 + 5x2 – 9 = O(x3) However, also true are:   3x3 + 5x2 – 9 = O(x4)   x3 = O(3x3 + 5x2 – 9)   si...
Lower Bounding Running Time f(n) is Ω(g(n)) if f grows “at least as fast as” g cg(n) is an approximation to f(n) bounding ...
Big-Omega (formal) Let f and g be two functions such that            f (n ) : N → R + and g (n ) : N → R + if there exists...
Tightly Bounding Running Time f(n) is Θ(g(n)) if f is essentially the same as g, to within a constant multiple29          ...
Big-Theta (formal) Let f and g be two functions such that             f (n ) : N → R + and g (n ) : N → R + if there exist...
Big-Θ, Big-O, and Big-Ω31               Tecniche di programmazione   A.A. 2012/2013
Big-Ω and Big-Θ Big-Ω: reverse of big-O. I.e.                     f (x ) = Ω(g (x ))                              iff     ...
Big-Ω and Big-Θ Big-Θ: domination in both directions. I.e.                     f (x ) = Θ(g (x ))                         ...
Problem Order the following from smallest to largest asymptotically. Group together all functions which are big-Θ of each ...
Solution  1x  13 + 1 x ln x lg 2 x  x + sin x, x + x ,13 + x  x ln x          2  x(ln x)     e   x  ( x + sin x)( x 20 − 1...
Practical approach36               Tecniche di programmazione   A.A. 2012/2013
Practical approach37               Tecniche di programmazione   A.A. 2012/2013
38   Tecniche di programmazione   A.A. 2012/2013
Would it be possible?     Algorithm     Foo                         Bar     Complexity   O(n2)                        O(2n...
Determination of Time Complexity Because of the approximations available through Big-Oh , the actual T(n) of an algorithm ...
Rule #1 Simple program statements are assumed to take a constant amount of time which is                         O(1)41   ...
Rule #2 Differences in execution time of simple statements is ignored42                        Tecniche di programmazione ...
Rule #3 In conditional statements the worst case is always used43                        Tecniche di programmazione   A.A....
Rule #4 – the “sum” rule The running time of a sequence of steps has the order of the running time of the largest E.g.,   ...
Rule #5 – the “product” rule If two processes are constructed such that second process is repeated a number of times for e...
Nested Loopsfor(int t=0; t<n; ++t) {         for(int u=0; u<n; ++u) {     O(n)             ++zap;                  O(1)   ...
Nested Loopsfor(int t=0; t<n; ++t) {         for(int u=0; u<n; ++u) {             ++zap;                      O(n*1)      ...
Nested Loopsfor(int t=0; t<n; ++t) {                 O(n)         for(int u=0; u<n; ++u) {             ++zap;             ...
Nested Loopsfor(int t=0; t<n; ++t) {         for(int u=0; u<n; ++u) {             ++zap;                      O(n2)       ...
Nested Loops     Note: Running time grows with nesting rather than the     length of the codefor(int t=0; t<n; ++t) {     ...
More Nested Loopsfor(int t=0; t<n; ++t) {         for(int u=t; u<n; ++u) {               n−t             ++zap;         }}...
Sequential statementsfor(int z=0; z<n; ++z)         zap[z] = 0;                                O(n)for(int t=0; t<n; ++t) ...
Conditionalsfor(int t=0; t<n; ++t) {         if(t%2) {             for(int u=t; u<n; ++u) {                 ++zap;        ...
Conditionalsfor(int t=0; t<n; ++t) {         if(t%2) {             for(int u=t; u<n; ++u) {                 ++zap;        ...
55   Tecniche di programmazione   A.A. 2012/2013
Tips Focus only on the dominant (high cost) operations and avoid a line-by-line exact analysis Break algorithm down into “...
Caveats Real time vs. complexity57                          Tecniche di programmazione   A.A. 2012/2013
Caveats Real time vs. complexity CPU time vs. RAM vs. disk58                       Tecniche di programmazione   A.A. 2012/...
Caveats Real time vs. complexity CPU time vs. RAM vs. disk Worse, Average or Best Case?59                      Tecniche di...
Worse, Average or Best Case?60
Worse, Average or Best Case? Depends on input problem instance type61
Basic Asymptotic Efficiency Classes
Basic Asymptotic Efficiency Classes
Basic Asymptotic Efficiency Classes
Basic Asymptotic Efficiency Classes
Basic Asymptotic Efficiency Classes
Basic Asymptotic Efficiency Classes
Basic Asymptotic Efficiency Classes
Basic Asymptotic Efficiency Classes
Evaluate the complexity Linear search?70                Tecniche di programmazione   A.A. 2012/2013
Evaluate the complexity Linear search     O(n) Dichotomic search71                   Tecniche di programmazione   A.A. 201...
Evaluate the complexity Linear search     O(n) Dichotomic search     O(log(n))72                   Tecniche di programmazi...
Basic Asymptotic Efficiency Classes
ArrayList vs. LinkedList                      ArrayList                        LinkedListadd(element)remove(object)get(ind...
ArrayList vs. LinkedList                       ArrayList                        LinkedListadd(element)             O(1)   ...
In theory, there is no differencebetween theory and practice.76               Tecniche di programmazione   A.A. 2012/2013
Licenza d’uso Queste diapositive sono distribuite con licenza Creative Commons “Attribuzione - Non commerciale - Condividi...
Upcoming SlideShare
Loading in …5
×

Computational complexity

786 views
701 views

Published on

Computational complexity: Measure the degree of difficulty of an algorithm

Teaching material for the course of "Tecniche di Programmazione" at Politecnico di Torino in year 2012/2013. More information: http://bit.ly/tecn-progr

Published in: Education, Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
786
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
25
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Computational complexity

  1. 1. Computational complexityMeasure the degree of difficulty of an algorithm
  2. 2. Background What is an Algorithm? A problem can be solved by many algorithms E.g., sorting data An algorithm is a method or a process followed to solve a problem I.e., a recipe An algorithm takes the input to a problem (function) and transforms it to the output I.e., a mapping Tecniche di programmazione A.A. 2012/2013
  3. 3. Why consider Efficiency? There are often many algorithms to solve a problem How do we choose between them? (Usually) conflicting goals: To design an algorithm that is easy to understand, code, debug software engineering To design an algorithm that makes efficient use of the computer’s resources data structures and algorithm analysis Tecniche di programmazione A.A. 2012/2013
  4. 4. (Un programma che funziona) in fretta vs.Un programma che (funziona in fretta) Tecniche di programmazione A.A. 2012/2013
  5. 5. Why consider Efficiency? from the LinkedList javadoc page: There are often many algorithms to solve a problem “All of the operations perform as could How do we choose between them? (Usually) conflicting goals: be expected for a doubly-linked list.” To design an algorithm that is easy to understand, code, debug software engineering To design an algorithm that makes efficient use of the computer’s resources data structures and algorithm analysis Tecniche di programmazione A.A. 2012/2013
  6. 6. How to Measure Efficiency? Critical resources programmer’s effort time, space (disk, RAM) Analysis empirical (run programs) analytical (asymptotic algorithm analysis) Worst case vs. Average case Tecniche di programmazione A.A. 2012/2013
  7. 7. Empirical Approach Implement each candidate That could be lots of work – also error-prone Run it Which inputs? Worst case, average case, or best case? Time it What machines Which OS? Tecniche di programmazione A.A. 2012/2013
  8. 8. Analytical Approach How to solve “which algorithm” problems without machines nor test data? Tecniche di programmazione A.A. 2012/2013
  9. 9. Analytical Approach An algorithm is a mapping For most algorithms, running time depends on “size” of the input Running time is expressed as T(n) some function T input size n Tecniche di programmazione A.A. 2012/2013
  10. 10. Bubble sort Tecniche di programmazione A.A. 2012/2013
  11. 11. Analysis The bubble sort takes (n2-n)/2 “steps” Different implementations/assembly languages Program A on an Intel Pentium IV: T(n) = 58*(n2-n)/2 Program B on a Motorola: T(n) = 84*(n2-2n)/2 Program C on an Intel Pentium V: T(n) = 44*(n2-n)/2 Note that each has an n2 term as n increases, the other terms will drop out Tecniche di programmazione A.A. 2012/2013
  12. 12. Analysis As a result: Program A on Intel Pentium IV: T(n) ≈ 29n2 Program B on Motorola: T(n) ≈ 42n2 Program C on Intel Pentium V: T(n) ≈ 22n2 Tecniche di programmazione A.A. 2012/2013
  13. 13. Analysis As processors change, the constants will always change The exponent on n will not We should not care about the constants As a result: Program A: T(n) ≈ n2 Program B: T(n) ≈ n2 Program C: T(n) ≈ n2 Bubble sort: T(n) ≈ n2 Tecniche di programmazione A.A. 2012/2013
  14. 14. Intuitive motivations Asymptotic notation captures behavior of functions for large values of x. Dominant term of 3x3 + 5x2 – 9 is 3x3 As x becomes larger and larger, other terms become insignificant and only 3x3 remains in the picture Tecniche di programmazione A.A. 2012/2013
  15. 15. [0, 2] y = 3x 3+5x 2 –9 y=x3 y=x2 y=x Tecniche di programmazione A.A. 2012/2013
  16. 16. [0, 5] Tecniche di programmazione A.A. 2012/2013
  17. 17. [0, 10] Tecniche di programmazione A.A. 2012/2013
  18. 18. [0, 100] y = 5x 3 y = 3x 3+5x 2 –9 Tecniche di programmazione A.A. 2012/2013
  19. 19. Complexity Analysis O( · ) big o (big oh) Ω( · ) big omega Θ( · ) big theta Tecniche di programmazione A.A. 2012/2013
  20. 20. O( · ) Upper Bounding Running Time Why? Little-oh “Order of” D’Oh Oh! Tecniche di programmazione A.A. 2012/2013
  21. 21. Upper Bounding Running Time f(n) is O(g(n)) if f grows “at most as fast as” g21 Tecniche di programmazione A.A. 2012/2013
  22. 22. Big-O (formal) Let f and g be two functions such that f (n ) : N → R + and g (n ) : N → R + if there exists positive constants c and n0 such that f (n ) ≤ cg (n ), for all n > n0 then we can write f (n ) = O( g (n ))22 Tecniche di programmazione A.A. 2012/2013
  23. 23. Big-O (formal alt) Let f and g be two functions such that f (n ) : N → R + and g (n ) : N → R + if there exists positive constants c and n0 such that f (n ) 0 ≤ lim =c<∞ n →∞ g (n ) then we can write f (n ) = O( g (n ))23 Tecniche di programmazione A.A. 2012/2013
  24. 24. Example (log n)2 = O(n)24 Tecniche di programmazione A.A. 2012/2013
  25. 25. Notational Issues Big-O notation is a way of comparing functions Notation quite unconventional e.g., 3x3 + 5x2 – 9 = O(x3) Doesn’t mean “3x3 + 5x2 – 9 equals the function O(x3)” “3x3 + 5x2 – 9 is big oh of x3” But “3x3+5x2 –9 is dominated by x3” Tecniche di programmazione A.A. 2012/2013
  26. 26. Common Misunderstanding 3x3 + 5x2 – 9 = O(x3) However, also true are: 3x3 + 5x2 – 9 = O(x4) x3 = O(3x3 + 5x2 – 9) sin(x) = O(x4) Note: Usage of big-O typically involves mentioning only the most dominant term “The running time is O(x2.5)” Tecniche di programmazione A.A. 2012/2013
  27. 27. Lower Bounding Running Time f(n) is Ω(g(n)) if f grows “at least as fast as” g cg(n) is an approximation to f(n) bounding from below27 Tecniche di programmazione A.A. 2012/2013
  28. 28. Big-Omega (formal) Let f and g be two functions such that f (n ) : N → R + and g (n ) : N → R + if there exists positive constants c and n0 such that f (n ) ≥ cg (n ), for all n > n0 then we can write f (n ) = Ω( g (n ))28 Tecniche di programmazione A.A. 2012/2013
  29. 29. Tightly Bounding Running Time f(n) is Θ(g(n)) if f is essentially the same as g, to within a constant multiple29 Tecniche di programmazione A.A. 2012/2013
  30. 30. Big-Theta (formal) Let f and g be two functions such that f (n ) : N → R + and g (n ) : N → R + if there exists positive constants c1, c2 and n0 such that c1 g (n ) ≤ f (n ) ≤ c2 g (n ), for all n > n0 then we can write f (n ) = Θ( g (n ))30 Tecniche di programmazione A.A. 2012/2013
  31. 31. Big-Θ, Big-O, and Big-Ω31 Tecniche di programmazione A.A. 2012/2013
  32. 32. Big-Ω and Big-Θ Big-Ω: reverse of big-O. I.e. f (x ) = Ω(g (x )) iff g (x ) = O (f (x )) so f(x) asymptotically dominates g(x) Tecniche di programmazione A.A. 2012/2013
  33. 33. Big-Ω and Big-Θ Big-Θ: domination in both directions. I.e. f (x ) = Θ(g (x )) iff f (x ) = O (g (x )) && f (x ) = Ω(g (x )) Tecniche di programmazione A.A. 2012/2013
  34. 34. Problem Order the following from smallest to largest asymptotically. Group together all functions which are big-Θ of each other: 1 1x + sin x, ln x, x + x , ,13 + ,13 + x, e , x , x x e x x x ( x + sin x)( x − 102), x ln x, x(ln x) , lg 2 x 20 2 Tecniche di programmazione A.A. 2012/2013
  35. 35. Solution 1x 13 + 1 x ln x lg 2 x x + sin x, x + x ,13 + x x ln x 2 x(ln x) e x ( x + sin x)( x 20 − 102) x e xx Tecniche di programmazione A.A. 2012/2013
  36. 36. Practical approach36 Tecniche di programmazione A.A. 2012/2013
  37. 37. Practical approach37 Tecniche di programmazione A.A. 2012/2013
  38. 38. 38 Tecniche di programmazione A.A. 2012/2013
  39. 39. Would it be possible? Algorithm Foo Bar Complexity O(n2) O(2n) n = 100 10s 4s n = 1000 12s 4.5s39 Tecniche di programmazione A.A. 2012/2013
  40. 40. Determination of Time Complexity Because of the approximations available through Big-Oh , the actual T(n) of an algorithm is not calculated T(n) may be determined empirically Big-Oh is usually determined by application of some simple 5 rules40 Tecniche di programmazione A.A. 2012/2013
  41. 41. Rule #1 Simple program statements are assumed to take a constant amount of time which is O(1)41 Tecniche di programmazione A.A. 2012/2013
  42. 42. Rule #2 Differences in execution time of simple statements is ignored42 Tecniche di programmazione A.A. 2012/2013
  43. 43. Rule #3 In conditional statements the worst case is always used43 Tecniche di programmazione A.A. 2012/2013
  44. 44. Rule #4 – the “sum” rule The running time of a sequence of steps has the order of the running time of the largest E.g., f(n) = O(n2) g(n) = O(n3) f(n) + g(n) = O(n3)44 Tecniche di programmazione A.A. 2012/2013
  45. 45. Rule #5 – the “product” rule If two processes are constructed such that second process is repeated a number of times for each n in the first process, then O is equal to the product of the orders of magnitude for both products E.g., For example, a two-dimensional array has one for loop inside another and each internal loop is executed n times for each value of the external loop. The function is O(n2)45 Tecniche di programmazione A.A. 2012/2013
  46. 46. Nested Loopsfor(int t=0; t<n; ++t) { for(int u=0; u<n; ++u) { O(n) ++zap; O(1) }} 46 Tecniche di programmazione A.A. 2012/2013
  47. 47. Nested Loopsfor(int t=0; t<n; ++t) { for(int u=0; u<n; ++u) { ++zap; O(n*1) }} 47 Tecniche di programmazione A.A. 2012/2013
  48. 48. Nested Loopsfor(int t=0; t<n; ++t) { O(n) for(int u=0; u<n; ++u) { ++zap; O(n) }} 48 Tecniche di programmazione A.A. 2012/2013
  49. 49. Nested Loopsfor(int t=0; t<n; ++t) { for(int u=0; u<n; ++u) { ++zap; O(n2) }} 49 Tecniche di programmazione A.A. 2012/2013
  50. 50. Nested Loops Note: Running time grows with nesting rather than the length of the codefor(int t=0; t<n; ++t) { for(int u=0; u<n; ++u) { ++zap; O(n2) }} 50 Tecniche di programmazione A.A. 2012/2013
  51. 51. More Nested Loopsfor(int t=0; t<n; ++t) { for(int u=t; u<n; ++u) { n−t ++zap; }} n(n − 1) n 2 − n ∑i =0 (n − i ) = n −1 2 = 2 =O n 2 ( ) 51 Tecniche di programmazione A.A. 2012/2013
  52. 52. Sequential statementsfor(int z=0; z<n; ++z) zap[z] = 0; O(n)for(int t=0; t<n; ++t) { for(int u=t; u<n; ++u) { ++zap; O(n2) }} Running time: max(O(n), O(n2)) = O(n2) 52 Tecniche di programmazione A.A. 2012/2013
  53. 53. Conditionalsfor(int t=0; t<n; ++t) { if(t%2) { for(int u=t; u<n; ++u) { ++zap; O(n) } } else { zap = 0; O(1) }} 53 Tecniche di programmazione A.A. 2012/2013
  54. 54. Conditionalsfor(int t=0; t<n; ++t) { if(t%2) { for(int u=t; u<n; ++u) { ++zap; } O(n2) } else { zap = 0; }} 54 Tecniche di programmazione A.A. 2012/2013
  55. 55. 55 Tecniche di programmazione A.A. 2012/2013
  56. 56. Tips Focus only on the dominant (high cost) operations and avoid a line-by-line exact analysis Break algorithm down into “known” pieces Identify relationships between pieces Sequential is additive Nested (loop / recursion) is multiplicative Drop constants Keep only dominant factor for each variable56 Tecniche di programmazione A.A. 2012/2013
  57. 57. Caveats Real time vs. complexity57 Tecniche di programmazione A.A. 2012/2013
  58. 58. Caveats Real time vs. complexity CPU time vs. RAM vs. disk58 Tecniche di programmazione A.A. 2012/2013
  59. 59. Caveats Real time vs. complexity CPU time vs. RAM vs. disk Worse, Average or Best Case?59 Tecniche di programmazione A.A. 2012/2013
  60. 60. Worse, Average or Best Case?60
  61. 61. Worse, Average or Best Case? Depends on input problem instance type61
  62. 62. Basic Asymptotic Efficiency Classes
  63. 63. Basic Asymptotic Efficiency Classes
  64. 64. Basic Asymptotic Efficiency Classes
  65. 65. Basic Asymptotic Efficiency Classes
  66. 66. Basic Asymptotic Efficiency Classes
  67. 67. Basic Asymptotic Efficiency Classes
  68. 68. Basic Asymptotic Efficiency Classes
  69. 69. Basic Asymptotic Efficiency Classes
  70. 70. Evaluate the complexity Linear search?70 Tecniche di programmazione A.A. 2012/2013
  71. 71. Evaluate the complexity Linear search O(n) Dichotomic search71 Tecniche di programmazione A.A. 2012/2013
  72. 72. Evaluate the complexity Linear search O(n) Dichotomic search O(log(n))72 Tecniche di programmazione A.A. 2012/2013
  73. 73. Basic Asymptotic Efficiency Classes
  74. 74. ArrayList vs. LinkedList ArrayList LinkedListadd(element)remove(object)get(index)set(index, element)add(index, element)remove(index)contains(object)indexOf(object) 74 Tecniche di programmazione A.A. 2012/2013
  75. 75. ArrayList vs. LinkedList ArrayList LinkedListadd(element) O(1) O(1)remove(object) O(n) + O(n) O(n) + O(1)get(index) O(1) O(n)set(index, element) O(1) O(n) + O(1)add(index, element) O(1) + O(n) O(n) + O(1)remove(index) O(n) O(n) + O(1)contains(object) O(n) O(n)indexOf(object) O(n) O(n) 75 Tecniche di programmazione A.A. 2012/2013
  76. 76. In theory, there is no differencebetween theory and practice.76 Tecniche di programmazione A.A. 2012/2013
  77. 77. Licenza d’uso Queste diapositive sono distribuite con licenza Creative Commons “Attribuzione - Non commerciale - Condividi allo stesso modo (CC BY-NC-SA)” Sei libero: di riprodurre, distribuire, comunicare al pubblico, esporre in pubblico, rappresentare, eseguire e recitare questopera di modificare questopera Alle seguenti condizioni: Attribuzione — Devi attribuire la paternità dellopera agli autori originali e in modo tale da non suggerire che essi avallino te o il modo in cui tu usi lopera. Non commerciale — Non puoi usare questopera per fini commerciali. Condividi allo stesso modo — Se alteri o trasformi questopera, o se la usi per crearne unaltra, puoi distribuire lopera risultante solo con una licenza identica o equivalente a questa. http://creativecommons.org/licenses/by-nc-sa/3.0/77 Tecniche di programmazione A.A. 2012/2013

×