Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.

Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.

Like this presentation? Why not share!

- Algorithm Design and Complexity - C... by Traian Rebedea 11640 views
- Algorithm Design and Complexity - C... by Traian Rebedea 5077 views
- Algorithm Design and Complexity - C... by Traian Rebedea 12517 views
- Algorithm Design and Complexity - C... by Traian Rebedea 6372 views
- Algorithm and flowchart by IamPe Khamkhum 410 views
- Basic Problems and Solving Algorithms by Nopadon Juneam 751 views

28,938 views

28,822 views

28,822 views

Published on

No Downloads

Total views

28,938

On SlideShare

0

From Embeds

0

Number of Embeds

2

Shares

0

Downloads

0

Comments

0

Likes

8

No embeds

No notes for slide

- 1. Algorithm Design and Complexity Course 1 & 2
- 2. Introduction Algorithms and problems We need to be able to provide solutions to problems Any domain has problems that require an algorithmic solution Find the best solution from a wide range of choices Learn methods to develop solutions Problem => Idea => Solution => Algorithm => Pseudocode => Code => Compiled Program We need to design algorithms
- 3. Introduction (2) Any non-trivial problem accepts a wide range of solutions Need to compare these solution in order to find the best one => Complexity Need to show that the devised solution solves the problem => Correctness Not all the problems, have an algorithmic solution!
- 4. Introduction (3) Some problems are similar (or slightly variations) Accept similar solutions Need to learn to discover two problems that are similar Some methods used for designing algorithms provide solutions for different problems Need to understand these methods in order to know to use them for as many problems as possible The problems that can be solved using one method have some common properties!
- 5. Course Info Lectures: Traian Rebedea Ph.D. @ University Politehnica of Bucharest, CS Dept. traian.rebedea@cs.pub.ro / trebedea@gmail.com Lecturer @ Computer Science Department Interests: NLP, IR/IE, ML, TEL/CSCL, AI in general Published over 25 papers at important conferences Published 4 book chapters (2 in important international books) Worked at 3 companies, founded 1 company http://www.informatik.uni-trier.de/~ley/db/indices/atree/r/Rebedea:Traian.html http://ro.linkedin.com/in/trebedea Course Website: http://adcfils.wordpress.com/
- 6. Course Overview Introduction Complexity of Algorithms Correctness of Algorithms (short) Algorithm Design Techniques Graph Algorithms & Applications Problems and Decidability Searches, topological sort, articulation points, bridges, strongly connected components, minimum spanning trees, shortest paths, network flow Classes of Problems
- 7. Grading Exam: 40p Lab & Assignments: 60p Assignments: 45p (3 x 15p) Lab activity: 15p The lab assistant decides how to grade the lab activity Rules: Minimum 30p for lab & assignments Minimum 15p for the exam Minimum 50p for total You are not allowed to copy solutions from colleagues or WWW (Measure of Software Similarity - MOSS)
- 8. Textbooks & More Info Cormen T.H, Leiserson C.E, Rivest R.L and Stein C, Introduction to Algorithms, Second Edition. MIT Press, 2001 Baase S and A. Van Gelder. Computer Algorithms. Introduction to Design & Analysis, Addison-Wesley, 2000 References for each chapter Introduction to Algorithms @ MIT Coursera: Algorithms: Design and Analysis (part 1 and part 2) Click here for link also has video lectures https://www.coursera.org/course/algo, https://www.coursera.org/course/algo2 Websites for programming exercises: TopCoder, Infoarena, Talent Buddy, HackerRank, Project Euler
- 9. Problems and Algorithms Problems 1 – n Algorithms Problem: Sorting 1+ algorithms to solve each problem An algorithm usually solves only 1 problem Given an array with n numbers A[n], arrange the elements in the array such that any two consecutive elements are sorted (A[i] <= A[i + 1] for i = 1..n-1) Arrays A[1..n] Algorithms: Quick Sort, Merge Sort, Heap Sort, Bubble Sort, Insertion Sort, Selection Sort, Radix Sort, Bucket Sort, …
- 10. Problems & Computability There are a lot of problems We would like to find solutions for all of them Not all the problems can be solved! The problems that can be solved are called computable or decidable problems The problems that cannot be solved are: Very difficult Not clear enough (need for subjective reasoning)
- 11. Problems & Computability We would all like to know: Which stock bonds shall rise tomorrow? Which football team would win a game? Who shall I marry? Is there a God? Are there any aliens in the universe? Who is the most beautiful girl in the world? Need for subjective thinking: what does “the most beautiful” mean?
- 12. Example This example should be understood as a metaphor Problem: Is there any alien life in the universe? The physics and astronomy parts of it may be wrong Assumption: the universe is infinite and there are an infinite number of celestial bodies Solution: Explore all the planets, suns and other celestial bodies Use any exploring method, it can be as good as you want Explore the celestial body with a perfect scanner If you find life on it => ANSWER:YES! Alien life exists Else: continue to the next celestial body
- 13. Example (2) The previous solution has a flaw It never answers NO! If the answer to our problem would be NO, then we must wait an infinite amount of time We cannot stop the solution at any moment in time and conclude for certain that the answer is NO, because maybe we still have a celestial body with alien life on it that has not been explored yet! This kind of problem is called undecidable!
- 14. Undecidable Problems Problems that cannot be solved with an algorithmic solution We can devise an algorithm, but that algorithm shall never finish in some situations… therefore we cannot know the answer to our problem! Quick info: A decision problem = a problem for which the answer is {yes, no} Is n a prime number ? An optimum problem = a problem for each we need to find the optimum solution out of a set of possible solutions Which is the shortest path between two vertices in a graph ? Optimum = minimum or maximum
- 15. Undecidable Problems (2) Any decision problem can be: Decidable – can always solve the problem with an algorithm that always finishes! Semi-decidable – can devise a pseudo-algorithm for solving the problem that only finishes if the answer is YES, but it never finishes otherwise! Therefore, we can never know whether the answer is YES or NO, but if the algorithm stops than the answer is for sure YES Undecidable – can not know if the answer is YES… The previous example is a semi-decidable problem However, most of the times all the problems that are not decidable (semi- or un-decidable) are called undecidable
- 16. Undecidable Problems (3) Why the example is a metaphor? Because, any problem that is not decidable should have an infinite space of the problem! All the problems that have a finite number of states that form the space of exploration for that problem are decidable E.g. there are a finite number of arranging the numbers in an array E.g. there are a finite number of ways to arrange a set of queens on a chess board
- 17. Undecidable Problems (4) Very difficult problems have an infinite space that should be explored in order to find the solution! Quick info: There are an infinitely uncountable number of problems in this world There are only an infinitely countable number of programs in this world Alonso-Church thesis states that all the problems that can be computed (are decidable) are the ones that can have a program associated to them, that is used for solving them Only a infinitely countable set of problems are decidable The rest are not decidable
- 18. Classic Problems that are NOT Decidable Halting Problem: Given a program P and an input x, does P(x) halts/finishes? YES if P(x) finishes in a finite amount of time NO if P(x) never finishes (may be because it loops forever) Barber Problem/Paradox Post’s Correspondence Problem
- 19. Barber Paradox http://en.wikipedia.org/wiki/Barber_paradox Suppose there is a town with just one male barber; and that every man in the town keeps himself clean-shaven: some by shaving themselves, some by attending the barber. It seems reasonable to imagine that the barber obeys the following rule: He shaves all and only those men in town who do not shave themselves. Does the barber shave himself?
- 20. Barber Paradox (2) 1. 2. The situation presented is in fact impossible: If the barber does not shave himself, he must abide by the rule and shave himself. If he does shave himself, according to the rule he will not shave himself
- 21. Post’s correspondence problem http://en.wikipedia.org/wiki/Post_correspondence_problem The input of the problem consists of two finite lists: of words over some alphabet A having at least two symbols. A solution to this problem is a sequence of indices with K >=1 and for all k, such that The decision problem then is to decide whether such a solution exists or not.
- 22. Complexity of Algorithms Need to find the best algorithm for solving a problem Is algorithm A better than algorithm B ? A measure for the performance of an algorithm Simple practical solution: Implement the algorithms Measure their running times on a given machine But we want to measure the performance of an algorithm: Independent of the machine and language it is implemented in Without wasting time for implementing it
- 23. Complexity of Algorithms (2) We need a theoretical framework for measuring the performance of an algorithm Performance Time = How quick does the algorithm compute the results ? Space = How much memory does it need ? Focus on time performance Moore’s Law: processing power evolves less quickly than storage capacity Space constraints are rarely an issue: related to RAM size
- 24. Running Time Measure of the time complexity of an algorithm It is a theoretical measure that is dependent of the input data and the processing performed by an algorithm We define the running time as a function that only depends on the size of the input data The size of the input data is measured by positive integers For arrays: A[n] For graphs: G(V, E), |V| = n, |E| = m For multiplying 3 matrices: lines1, columns1, columns2
- 25. Running Time (2) We shall only discuss in this chapter running times that are dependent of a single parameter: T(n) The discussion can be easily extended to more parameters T(n) is the running time for an algorithm that has an input data of size n T(n) is a function T(n): N → R+
- 26. Example – Insertion Sort http://en.wikipedia.org/wiki/Insertion_sort Problem: Sorting an array A[n] Solution: Insertion sort Every repetition of the main loop of insertion sort removes an element from the input data, inserting it into the correct position in the already-sorted list, until no input elements remain. The already-sorted list is the sub-array on the left side Usually, the removed element is the next one
- 27. Insertion Sort - Pseudocode InsertionSort( A[1..n] ) 1. FOR (j = 2 .. n) 2. x = A[j] 3. i=j–1 4. 5. 6. 7. 8. // element to be inserted // position on the right side of // the sorted sub-array WHILE (i > 0 AND x < A[i]) // while not in position A[i + 1] = A[i] // move to right i-// continue A[i + 1] = x RETURN A
- 28. Example From Erik D. Demaine and Charles E. Leiserson – Introduction to Algorithms@ MIT
- 29. Analysis of Complexity What is the complexity of Insertion Sort ? General solution for the running time: Each simple instruction takes a constant amount of time This is clearly a simplification as the execution of an instruction depends on the operands Simple instructions: assignments, logical, mathematical between numbers, print/scan of a number, return, … Complex instructions: calls to other functions T(n) = sum over the running time of each instruction The running time of an instruction = Running time to execute it once * Number of times it is executed
- 30. Analysis of Complexity – General Instruction nbr Running time – execute once Number of time it is executed 1 C1 (a constant) n 2 C2 n–1 3 C3 n–1 4 C4 T1 5 C5 T2 6 C6 T2 7 C7 n–1 8 C8 1 T(n) = C1*n + (C2+C3+C7)*(n-1) + C4*T1 + (C5+C6)*T2 T1 = (j=2..n) tj T2 = (j=2..n) (tj – 1) Difficult to say more about the general form of the running time
- 31. Analysis of Complexity – General (2) General form of the running time cannot be expanded because it depends on the structure of the input data T1, T2 = ?? However, there are interesting special cases that can be easily computed: Worst case Best case Average case ?
- 32. Worst Case Complexity Happens when the array is sorted descending In this case, all the elements x = A[j] are lower than all the previous elements Therefore, they must be moved to the beginning of the array Thus: tj = j (from j – 1 to 0) T1 = (j=2..n) j = n*(n+1) / 2 – 1 T2 = (j=2..n) (j - 1) = n*(n-1) / 2 Tworst(n) = a*n2 + b*n + c quadratic time
- 33. Best Case Complexity Happens when the array is sorted ascending In this case, all the elements x = A[j] are higher than all the previous elements Therefore, they are not moved Thus: tj = 1 T1 = (j=2..n) 1 = n-1 T2 = (j=2..n) (1 - 1) = 0 Tbest(n) = b1*n + c1 linear time
- 34. Average Case Complexity It is interesting to compute it precisely It is very difficult to compute it precisely! Should take into consideration the distribution of the input data and sum up over all possible instances of the input data averaged by their distribution See example of formula in blackboard Not feasible Simpler solution: on average, an element x = A[j] in inserted in the middle of the already-sorted list Recompute T1 and T2 for this case => still a quadratic solution
- 35. Conclusions General formula for the complexity of an algorithm is usually incomplete due to the influence of the structure of the input data Average case is interesting, but difficult to compute Solution: only compute worst case! Makes sense for a lot of applications: ABS braking algorithm should be good on worst case, the same for computing reports for your boss, … On most occasions, average case has the same order of growth as the worst case complexity
- 36. hope you’re not sleeping yet
- 37. Asymptotic Notations Current simplifications for computing complexity of algorithms: Constant amount of time for simple instructions Interested in worst case most of the time Only interested in the asymptotic behavior of T(n) Asymptotic: T(n) | n → INF These notations are not used only for running times, but for any function of the form f(n) : N → R+ |sin(n)|, 1/n, 2sin(n)+1 are functions that cannot be running times
- 38. Big-O Notation http://en.wikipedia.org/wiki/Big_O_notation Upper asymptotic bound
- 39. Big- Notation Omega Lower asymptotic bound
- 40. Notation Theta Order of growth
- 41. Remarks It is important to compute the order of growth for algorithms The asymptotic notations define sets of functions See picture on blackboard Sometimes, the Big-O notation is used as a substituent of the notation notation – equivalence relation for functions of the form f(n) : N → R+ Big-O, Big- notations – partial order relations
- 42. Equivalence Relation Three important properties: 3. Reflexivity Symmetry Transitivity It partitions the functions into equivalence classes: 1. 2. Each class has a representative function Obtained by removing all lower degree terms and removing any constants from the highest degree term (1), (log log n), (log n), (n log n), (n2), (n3), … , (2n), (n!), (nn)
- 43. Partial Order Relations 1. 2. 3. Three important properties: Reflexivity Anti-symmetry Transitivity f(n) O(g(n)) … f(n) “<=“ g(n) f(n) (g(n)) … f(n) “>=“ g(n) f(n) (g(n)) … f(n) “~=“ g(n) Partial order because some functions cannot be compared: e.g. n and nsin(n)+1
- 44. Small-o and Small- Notations f(n) O(g(n)) can be: Tight: 2*n2 + n O(n2) Loose: 5*n O(n2) Therefore, small-o is always loose: Similarly: f(n) o(g(n)) … f(n) “<“ g(n) f(n) (g(n)) … f(n) “>“ g(n)
- 45. Asymptotic Notations Used in Equations For any function on the left side that is part of the set defined by that asymptotic notation there is a function on the right side that is part of the set defined by that asymptotic notation
- 46. Exercises – Set 1 What is the complexity of the following algorithms ? Matrix_add_1 (A[n][n],B[n][n]) { for (i = 1,n) { for (j = 1,n) { C[i][j] = 0 } } for (i = 1,n) { for (j = 1,n) { C[i][j] = A[i][j] + B[i][j] } } return C }
- 47. Matrix_add_2 (A[n][n],B[n][n]) { for (i = 1,n) { for (j = 1,n) { B[i][j] = A[i][j] + B[i][j] } } return B }
- 48. Maximum Subsequence Sum Problem Given (possibly negative) integers a1, a2, ..., an, find the maximum value of (k=i..j)Σak. The maximum subsequence sum is defined to be 0 if all the integers are negative
- 49. Let A[1] … A[N] be an array of integers that contains a sequence of length N. Let sum and maxSum be integers initialized to 0. For integer i = 1 to N do Let sum = 0 For integer j = i to N do Let sum = sum + A[ j ] If( sum > maxSum ) then Let maxSum = sum Return maxSum
- 50. There also exist solutions in: (n) (n log n) http://www.wou.edu/~broegb/Cs345/MaxSubsequenceSum.pdf (n3)
- 51. Exercises – Set 2
- 52. Keep in Mind Are there things more important than the performance of an algorithm ? May be: Correctness Modularity Maintainability Robustness User-friendliness Programmer time Extensibility Reliability
- 53. References CLRS – Chapters 1-3 MIT OCW – Introduction to Algorithms – video lectures 1-2

No public clipboards found for this slide

×
### Save the most important slides with Clipping

Clipping is a handy way to collect and organize the most important slides from a presentation. You can keep your great finds in clipboards organized around topics.

Be the first to comment