Upcoming SlideShare
×

# Computational complexity

786 views
701 views

Published on

Computational complexity: Measure the degree of difficulty of an algorithm

Teaching material for the course of "Tecniche di Programmazione" at Politecnico di Torino in year 2012/2013. More information: http://bit.ly/tecn-progr

Published in: Education, Technology
2 Likes
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

Views
Total views
786
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
25
0
Likes
2
Embeds 0
No embeds

No notes for slide

### Computational complexity

1. 1. Computational complexityMeasure the degree of difficulty of an algorithm
2. 2. Background What is an Algorithm? A problem can be solved by many algorithms E.g., sorting data An algorithm is a method or a process followed to solve a problem I.e., a recipe An algorithm takes the input to a problem (function) and transforms it to the output I.e., a mapping Tecniche di programmazione A.A. 2012/2013
3. 3. Why consider Efficiency? There are often many algorithms to solve a problem How do we choose between them? (Usually) conflicting goals: To design an algorithm that is easy to understand, code, debug software engineering To design an algorithm that makes efficient use of the computer’s resources data structures and algorithm analysis Tecniche di programmazione A.A. 2012/2013
4. 4. (Un programma che funziona) in fretta vs.Un programma che (funziona in fretta) Tecniche di programmazione A.A. 2012/2013
5. 5. Why consider Efficiency? from the LinkedList javadoc page: There are often many algorithms to solve a problem “All of the operations perform as could How do we choose between them? (Usually) conflicting goals: be expected for a doubly-linked list.” To design an algorithm that is easy to understand, code, debug software engineering To design an algorithm that makes efficient use of the computer’s resources data structures and algorithm analysis Tecniche di programmazione A.A. 2012/2013
6. 6. How to Measure Efficiency? Critical resources programmer’s effort time, space (disk, RAM) Analysis empirical (run programs) analytical (asymptotic algorithm analysis) Worst case vs. Average case Tecniche di programmazione A.A. 2012/2013
7. 7. Empirical Approach Implement each candidate That could be lots of work – also error-prone Run it Which inputs? Worst case, average case, or best case? Time it What machines Which OS? Tecniche di programmazione A.A. 2012/2013
8. 8. Analytical Approach How to solve “which algorithm” problems without machines nor test data? Tecniche di programmazione A.A. 2012/2013
9. 9. Analytical Approach An algorithm is a mapping For most algorithms, running time depends on “size” of the input Running time is expressed as T(n) some function T input size n Tecniche di programmazione A.A. 2012/2013
10. 10. Bubble sort Tecniche di programmazione A.A. 2012/2013
11. 11. Analysis The bubble sort takes (n2-n)/2 “steps” Different implementations/assembly languages Program A on an Intel Pentium IV: T(n) = 58*(n2-n)/2 Program B on a Motorola: T(n) = 84*(n2-2n)/2 Program C on an Intel Pentium V: T(n) = 44*(n2-n)/2 Note that each has an n2 term as n increases, the other terms will drop out Tecniche di programmazione A.A. 2012/2013
12. 12. Analysis As a result: Program A on Intel Pentium IV: T(n) ≈ 29n2 Program B on Motorola: T(n) ≈ 42n2 Program C on Intel Pentium V: T(n) ≈ 22n2 Tecniche di programmazione A.A. 2012/2013
13. 13. Analysis As processors change, the constants will always change The exponent on n will not We should not care about the constants As a result: Program A: T(n) ≈ n2 Program B: T(n) ≈ n2 Program C: T(n) ≈ n2 Bubble sort: T(n) ≈ n2 Tecniche di programmazione A.A. 2012/2013
14. 14. Intuitive motivations Asymptotic notation captures behavior of functions for large values of x. Dominant term of 3x3 + 5x2 – 9 is 3x3 As x becomes larger and larger, other terms become insignificant and only 3x3 remains in the picture Tecniche di programmazione A.A. 2012/2013
15. 15. [0, 2] y = 3x 3+5x 2 –9 y=x3 y=x2 y=x Tecniche di programmazione A.A. 2012/2013
16. 16. [0, 5] Tecniche di programmazione A.A. 2012/2013
17. 17. [0, 10] Tecniche di programmazione A.A. 2012/2013
18. 18. [0, 100] y = 5x 3 y = 3x 3+5x 2 –9 Tecniche di programmazione A.A. 2012/2013
19. 19. Complexity Analysis O( · ) big o (big oh) Ω( · ) big omega Θ( · ) big theta Tecniche di programmazione A.A. 2012/2013
20. 20. O( · ) Upper Bounding Running Time Why? Little-oh “Order of” D’Oh Oh! Tecniche di programmazione A.A. 2012/2013
21. 21. Upper Bounding Running Time f(n) is O(g(n)) if f grows “at most as fast as” g21 Tecniche di programmazione A.A. 2012/2013
22. 22. Big-O (formal) Let f and g be two functions such that f (n ) : N → R + and g (n ) : N → R + if there exists positive constants c and n0 such that f (n ) ≤ cg (n ), for all n > n0 then we can write f (n ) = O( g (n ))22 Tecniche di programmazione A.A. 2012/2013
23. 23. Big-O (formal alt) Let f and g be two functions such that f (n ) : N → R + and g (n ) : N → R + if there exists positive constants c and n0 such that f (n ) 0 ≤ lim =c<∞ n →∞ g (n ) then we can write f (n ) = O( g (n ))23 Tecniche di programmazione A.A. 2012/2013
24. 24. Example (log n)2 = O(n)24 Tecniche di programmazione A.A. 2012/2013
25. 25. Notational Issues Big-O notation is a way of comparing functions Notation quite unconventional e.g., 3x3 + 5x2 – 9 = O(x3) Doesn’t mean “3x3 + 5x2 – 9 equals the function O(x3)” “3x3 + 5x2 – 9 is big oh of x3” But “3x3+5x2 –9 is dominated by x3” Tecniche di programmazione A.A. 2012/2013
26. 26. Common Misunderstanding 3x3 + 5x2 – 9 = O(x3) However, also true are: 3x3 + 5x2 – 9 = O(x4) x3 = O(3x3 + 5x2 – 9) sin(x) = O(x4) Note: Usage of big-O typically involves mentioning only the most dominant term “The running time is O(x2.5)” Tecniche di programmazione A.A. 2012/2013
27. 27. Lower Bounding Running Time f(n) is Ω(g(n)) if f grows “at least as fast as” g cg(n) is an approximation to f(n) bounding from below27 Tecniche di programmazione A.A. 2012/2013
28. 28. Big-Omega (formal) Let f and g be two functions such that f (n ) : N → R + and g (n ) : N → R + if there exists positive constants c and n0 such that f (n ) ≥ cg (n ), for all n > n0 then we can write f (n ) = Ω( g (n ))28 Tecniche di programmazione A.A. 2012/2013
29. 29. Tightly Bounding Running Time f(n) is Θ(g(n)) if f is essentially the same as g, to within a constant multiple29 Tecniche di programmazione A.A. 2012/2013
30. 30. Big-Theta (formal) Let f and g be two functions such that f (n ) : N → R + and g (n ) : N → R + if there exists positive constants c1, c2 and n0 such that c1 g (n ) ≤ f (n ) ≤ c2 g (n ), for all n > n0 then we can write f (n ) = Θ( g (n ))30 Tecniche di programmazione A.A. 2012/2013
31. 31. Big-Θ, Big-O, and Big-Ω31 Tecniche di programmazione A.A. 2012/2013
32. 32. Big-Ω and Big-Θ Big-Ω: reverse of big-O. I.e. f (x ) = Ω(g (x )) iff g (x ) = O (f (x )) so f(x) asymptotically dominates g(x) Tecniche di programmazione A.A. 2012/2013
33. 33. Big-Ω and Big-Θ Big-Θ: domination in both directions. I.e. f (x ) = Θ(g (x )) iff f (x ) = O (g (x )) && f (x ) = Ω(g (x )) Tecniche di programmazione A.A. 2012/2013
34. 34. Problem Order the following from smallest to largest asymptotically. Group together all functions which are big-Θ of each other: 1 1x + sin x, ln x, x + x , ,13 + ,13 + x, e , x , x x e x x x ( x + sin x)( x − 102), x ln x, x(ln x) , lg 2 x 20 2 Tecniche di programmazione A.A. 2012/2013
35. 35. Solution 1x 13 + 1 x ln x lg 2 x x + sin x, x + x ,13 + x x ln x 2 x(ln x) e x ( x + sin x)( x 20 − 102) x e xx Tecniche di programmazione A.A. 2012/2013
36. 36. Practical approach36 Tecniche di programmazione A.A. 2012/2013
37. 37. Practical approach37 Tecniche di programmazione A.A. 2012/2013
38. 38. 38 Tecniche di programmazione A.A. 2012/2013
39. 39. Would it be possible? Algorithm Foo Bar Complexity O(n2) O(2n) n = 100 10s 4s n = 1000 12s 4.5s39 Tecniche di programmazione A.A. 2012/2013
40. 40. Determination of Time Complexity Because of the approximations available through Big-Oh , the actual T(n) of an algorithm is not calculated T(n) may be determined empirically Big-Oh is usually determined by application of some simple 5 rules40 Tecniche di programmazione A.A. 2012/2013
41. 41. Rule #1 Simple program statements are assumed to take a constant amount of time which is O(1)41 Tecniche di programmazione A.A. 2012/2013
42. 42. Rule #2 Differences in execution time of simple statements is ignored42 Tecniche di programmazione A.A. 2012/2013
43. 43. Rule #3 In conditional statements the worst case is always used43 Tecniche di programmazione A.A. 2012/2013
44. 44. Rule #4 – the “sum” rule The running time of a sequence of steps has the order of the running time of the largest E.g., f(n) = O(n2) g(n) = O(n3) f(n) + g(n) = O(n3)44 Tecniche di programmazione A.A. 2012/2013
45. 45. Rule #5 – the “product” rule If two processes are constructed such that second process is repeated a number of times for each n in the first process, then O is equal to the product of the orders of magnitude for both products E.g., For example, a two-dimensional array has one for loop inside another and each internal loop is executed n times for each value of the external loop. The function is O(n2)45 Tecniche di programmazione A.A. 2012/2013
46. 46. Nested Loopsfor(int t=0; t<n; ++t) { for(int u=0; u<n; ++u) { O(n) ++zap; O(1) }} 46 Tecniche di programmazione A.A. 2012/2013
47. 47. Nested Loopsfor(int t=0; t<n; ++t) { for(int u=0; u<n; ++u) { ++zap; O(n*1) }} 47 Tecniche di programmazione A.A. 2012/2013
48. 48. Nested Loopsfor(int t=0; t<n; ++t) { O(n) for(int u=0; u<n; ++u) { ++zap; O(n) }} 48 Tecniche di programmazione A.A. 2012/2013
49. 49. Nested Loopsfor(int t=0; t<n; ++t) { for(int u=0; u<n; ++u) { ++zap; O(n2) }} 49 Tecniche di programmazione A.A. 2012/2013
50. 50. Nested Loops Note: Running time grows with nesting rather than the length of the codefor(int t=0; t<n; ++t) { for(int u=0; u<n; ++u) { ++zap; O(n2) }} 50 Tecniche di programmazione A.A. 2012/2013
51. 51. More Nested Loopsfor(int t=0; t<n; ++t) { for(int u=t; u<n; ++u) { n−t ++zap; }} n(n − 1) n 2 − n ∑i =0 (n − i ) = n −1 2 = 2 =O n 2 ( ) 51 Tecniche di programmazione A.A. 2012/2013
52. 52. Sequential statementsfor(int z=0; z<n; ++z) zap[z] = 0; O(n)for(int t=0; t<n; ++t) { for(int u=t; u<n; ++u) { ++zap; O(n2) }} Running time: max(O(n), O(n2)) = O(n2) 52 Tecniche di programmazione A.A. 2012/2013
53. 53. Conditionalsfor(int t=0; t<n; ++t) { if(t%2) { for(int u=t; u<n; ++u) { ++zap; O(n) } } else { zap = 0; O(1) }} 53 Tecniche di programmazione A.A. 2012/2013
54. 54. Conditionalsfor(int t=0; t<n; ++t) { if(t%2) { for(int u=t; u<n; ++u) { ++zap; } O(n2) } else { zap = 0; }} 54 Tecniche di programmazione A.A. 2012/2013
55. 55. 55 Tecniche di programmazione A.A. 2012/2013
56. 56. Tips Focus only on the dominant (high cost) operations and avoid a line-by-line exact analysis Break algorithm down into “known” pieces Identify relationships between pieces Sequential is additive Nested (loop / recursion) is multiplicative Drop constants Keep only dominant factor for each variable56 Tecniche di programmazione A.A. 2012/2013
57. 57. Caveats Real time vs. complexity57 Tecniche di programmazione A.A. 2012/2013
58. 58. Caveats Real time vs. complexity CPU time vs. RAM vs. disk58 Tecniche di programmazione A.A. 2012/2013
59. 59. Caveats Real time vs. complexity CPU time vs. RAM vs. disk Worse, Average or Best Case?59 Tecniche di programmazione A.A. 2012/2013
60. 60. Worse, Average or Best Case?60
61. 61. Worse, Average or Best Case? Depends on input problem instance type61
62. 62. Basic Asymptotic Efficiency Classes
63. 63. Basic Asymptotic Efficiency Classes
64. 64. Basic Asymptotic Efficiency Classes
65. 65. Basic Asymptotic Efficiency Classes
66. 66. Basic Asymptotic Efficiency Classes
67. 67. Basic Asymptotic Efficiency Classes
68. 68. Basic Asymptotic Efficiency Classes
69. 69. Basic Asymptotic Efficiency Classes
70. 70. Evaluate the complexity Linear search?70 Tecniche di programmazione A.A. 2012/2013
71. 71. Evaluate the complexity Linear search O(n) Dichotomic search71 Tecniche di programmazione A.A. 2012/2013
72. 72. Evaluate the complexity Linear search O(n) Dichotomic search O(log(n))72 Tecniche di programmazione A.A. 2012/2013
73. 73. Basic Asymptotic Efficiency Classes