Anlysis and design of algorithms part 1

  • 512 views
Uploaded on

Analyzing Algorithms and problems. Classifying functions by their asymptotic growth rate. Recursive procedures. Recurrence equations - Substitution Method, Changing variables, Recursion Tree, Master …

Analyzing Algorithms and problems. Classifying functions by their asymptotic growth rate. Recursive procedures. Recurrence equations - Substitution Method, Changing variables, Recursion Tree, Master Theorem. Design Techniques- Divide and Conquer, Dynamic Programming, Greedy, Backtracking

More in: Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
512
On Slideshare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
34
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Analysis and Design of Algorithms Deepak John Department Of Computer Applications , SJCET-Pala
  • 2. What is AlgorithmWhat is Algorithm  Algorithm  is any well-defined computational procedure that takes some is any well defined computational procedure that takes some value, or set of values, as input and produces some value, or set of values, as output.  is thus a sequence of computational steps that transform the input into the output.  is a tool for solving a well - specified computational is a tool for solving a well specified computational problem.
  • 3. What is a programWhat is a program  A program is the expression of an algorithm in a programming languageg g  a set of instructions which the computer will follow to solve a problem
  • 4. Importance of Analyze AlgorithmImportance of Analyze Algorithm  Need to recognize limitations of various algorithms for solving a problemfor solving a problem  Need to understand relationship between problem size and running timeand running time  Need to learn how to analyze an algorithm's running time without coding itg  Need to learn techniques for writing more efficient code  Need to recognize bottlenecks in code as well as which parts of code are easiest to optimize
  • 5. Floor and CeilingFloor and Ceiling
  • 6. 3 useful formulas
  • 7. Analyzing Algorithm and Problems  An algorithm is a method or process to solve a problem satisfying the following properties:the following properties:  Correctness  Amount Of Work Done A d W C A l i Average and Worst Case Analysis  Amount of space used  Simplicity and Clarity  Optimality  Implementation and programming  Lower bounds and the complexity of problems Lower bounds and the complexity of problems
  • 8.  Correctness  Preconditions (characteristics of i/p ,it is expected to work)and post conditions(result it is to produce for each i/p)conditions(result it is to produce for each i/p)  Solution method  Implementation  Amount Of Work Done Amount Of Work Done  Highly depend on the programming language used and the programmers style  Also referred as complexity of algorithmp y g  Average and Worst Case Analysis Let Dn be the set of inputs of size n and I be an element of Dn. Let t(I) be the number of basic operations performed by the algorithm on input l i ( ) i h i b f b i Worst case complexity W(n) is the maximum number of basic operations performed by the algorythm on any size of input n. =max {t(I) | I Dn  Average case complexity A(n) Average case complexity A(n) Let Pr(I) be the probability that input I occurs. Then A(n)= P r ( ) ( )I t IA(n)= P r ( ) ( ) I D n I t I  
  • 9.  Amount of space used - storage space (instructions ,constants etc andg p ( , extra space for input)  Simplicity ,clarity and Optimality -an algorithm is optimal if there is no algorithm in the class that performs fewer basis operations  Lower bounds and the complexity of problems Lower bounds and the complexity of problems
  • 10. Algorithm Complexity  Worst Case Complexity:  Provides an upper bound on running time Provides an upper bound on running time  the function defined by the maximum number of steps taken on any instance of size n  Best Case Complexity:  the function defined by the minimum number of steps taken on any instance of size ninstance of size n  Average Case Complexity:  Provides the expected running time Provides the expected running time  the function defined by the average number of steps taken on any instance of size n
  • 11. Best Worst and Average Case ComplexityBest, Worst, and Average Case Complexity
  • 12. Sequential Search, Unordered I t E K h E i ith t i i d d 0 1) dInput: E, n, K, where E is an array with n entries indexed 0, …, n-1), and K is the item sought. For simplicity, we assume that K and the entries of E are integers, as is n. Output: Returns ans, the location of K in E (-1 if K is not found.)  Algorithm: Step (Specification) int seqSearch(int[] E, int n, int K) 1. int ans, index; 2 1 // A f il2. ans = -1; // Assume failure. 3. for (index = 0; index < n; index++) 4 if (K == E[index])4. if (K E[index]) 5. ans = index; // Success! 6. break; // Done!; 7. return ans;
  • 13. Analysis of the Algorithm • Basic Operation: Comparison of x with an array entryp p y y • Worst-Case Analysis: Let W(n) be a function. W(n) is the maximum number of basic operations performed by the algorithm on any input i F lsize n. For our example, clearly W(n) = n. The worst cases occur when K appears only in the last position in theThe worst cases occur when K appears only in the last position in the array and when K is not in the array at all. •
  • 14. Average-case Analysis:g y A(n) for the success case, Ii represent the event that K appears in the i th position in the array. Let t(I) be the number of comparisons done for input Ifor input I. n-1 Asucc(n)=∑ Pr(Ii | succ)t(Ii)= 1 0 (1 / ) ( 1 ) n i n i     Asucc(n) ∑ Pr(Ii | succ)t(Ii) i=0 A (n) for the fail case. (1 / ) ( ( 1 ) / 2 ) ( 1 ) / 2 n n n n   ( ) A(fail)=n Combine both cases ,Let q be the probability that K is in the array A(n)=Pr(succ) Asucc(n)+Pr(fail)Afail(n) =q((n+1)/2)+(1- q)n (1 ( /2)) ( /2)=n(1-(q/2))+(q/2)
  • 15. Classifying functions by their asymptotic growth rateClassifying functions by their asymptotic growth rate  The running time of an algorithm as input size is called the t ti i tiasymptotic running time  The notations (, O, , o, w ) describe different rate-of-growth relations between the defining function and the defined set ofg functions.  O(g(n)), Big-Oh of g of n, the Asymptotic Upper Bound;  (g(n)), Theta of g of n, the Asymptotic Tight Bound; and  (g(n)), Omega of g of n, the Asymptotic Lower Bound.
  • 16. Big-O •We use O notation to give an upper bound on a function to within a constantWe use O otat o to g ve a uppe bou d o a u ct o to w t a co sta t factor. •Used for bound the worst case running time of the algorithm on every input •Let f(n) and g(n) be two functions. We write          0 such thatandconstantspositiveexistthere: ncngOnf  Let f(n) and g(n) be two functions. We write f(n) = O(g(n)) or f = O(g)     0allfor0 nnncgnf 
  • 17. Big Omega – Notation  – A lower bound  Used to bound the best case running time of an algorithm      h hdi iihf          0 0 allfor0 such thatandconstantspositiveexistthere: nnncgnf ncngnf   f(n) ( ) f(n) cg(n) n0
  • 18. -notation   provides a tight bound            021 021 allfor0 such thatand,,constantspositiveexistthere: nnngcnfngc nccngnf                 ngnfngOnfngnf  AND c2g(n) f(n) c1g(n) n0n0
  • 19. Relations Between , , ORelations Between , , O Theorem : For any two functions g(n) and f(n), f(n) = (g(n)) iff f(n) = O(g(n)) and f(n) = (g(n)).f( ) (g( )) f( ) (g( )) f( ) (g( )) I.e., (g(n)) = O(g(n))  (g(n)) In practice, asymptotically tight bounds are obtained from asymptotic upper and lower boundsasymptotic upper and lower bounds.
  • 20. (g(n)), functions that grow at least as fast as g(n) >= (g(n)), functions that grow at the same rate as g(n) g(n) = g(n) <= O(g(n)), functions that grow no faster than g(n)
  • 21. o-notationo notation For a given function g(n), the set little-o: f(n)=o(g(n)) there exist positive constants c and n0 where c > 0, n0 > 0 such that for all n  n0, we have 0  f(n) < cg(n). f(n) becomes insignificant relative to g(n) as n approaches infinity:f( ) g g( ) pp y lim [f(n) / g(n)] = 0 n g(n) is an upper bound for f(n) that is not asymptotically tightg(n) is an upper bound for f(n) that is not asymptotically tight.
  • 22. -notation f(n)=(g(n)) there exist positive constants c and n0 where c > 0, n0 > 0  notation For a given function g(n), the set little-omega: such that for all n  n0, we have 0  cg(n) < f(n)}. f(n) becomes arbitrarily large relative to g(n) as n approaches infinity: lim [f(n) / g(n)] = . n g(n) is a lower bound for f(n) that is not asymptotically tight.g( ) f( ) y p y g
  • 23. Theoretical analysis of time efficiencyTheoretical analysis of time efficiency Time efficiency is analyzed by determining the number of repetitions of the basic operation as a function of input sizerepetitions of the basic operation as a function of input size  Basic operation: the operation that contributes the most towards the running time of the algorithm input size T(n) ≈ copC(n) running time execution time for basic operation or cost Number of times basic operation is executed or cost
  • 24. Recursive Procedure  A procedure that is defined in terms of itself  In a computer language a function that calls itself In a computer language a function that calls itself  A recursive algorithm is a problem solution that has been expressed in terms of two or more easier to solve sub blproblems
  • 25. Content of a Recursive MethodContent of a Recursive Method  Base case(s).  V l f th i t i bl f hi h f Values of the input variables for which we perform no recursive calls are called base cases (there should be at least one base case).  Every possible chain of recursive calls must eventually reach a base case. R i ll Recursive calls.  Calls to the current method.  Each recursive call should be defined so that it makes progress Each recursive call should be defined so that it makes progress towards a base case.
  • 26. Recurrence equationRecurrence equation Merge Sort • T(n) = (1) if n=1• T(n) = (1) if n=1 T(n/2)+ T(n/2)+ (n) if n>1 • Ignore details, T(n) = 2T(n/2)+ (n).Ignore details, T(n) 2T(n/2) (n). Recurrence Equation Solving technique  Substitution method  Master method  Recursion Tree method
  • 27. The Substitution Method  Used to establish either upper or lower bound on a recurrence  T o steps: Two steps: 1. Guess the form of the solution. 2. Use mathematical induction to find the constants and show that the solution works. Example T(n) = 2T(n/2) + nT(n) = 2T(n/2) + n Guess (#1) T(n) = O(n) Need T(n) <= cn for some constant c>0Need T(n) < cn for some constant c>0 Assume T(n/2) <= cn/2Inductive hypothesis Thus T(n) <= 2cn/2 + n = (c+1) n( ) ( ) Our guess was wrong!!
  • 28. T(n) = 2T(n/2) + n Guess (#2) T(n) = O(n2)( ) ( ) ( ) Need T(n) <= cn2 for some constant c>0 Assume T(n/2) <= cn2/4 Inductive hypothesis Thus T(n) <= 2cn2/4 + n = cn2/2+ n Works for all n as long as c>=2 !!But there is a lot of “slack”
  • 29. Solve T(n)=2T(n/2)+n  Guess the solution: T(n)=O(n lg n),  i.e., T(n) cn lg n for some c.  Prove the solution by induction:  Suppose this bound holds for n/2 i e Suppose this bound holds for n/2, i.e.,  T(n/2) cn/2 lg (n/2).  T(n)  2((cn/2 lg (n/2))+n( ) (( g ( ))   cn lg (n/2)+n  = cn lg n - cn lg 2 +n  = cn lg n cn +n = cn lg n - cn +n   cn lg n (as long as c1)  Works for all n as long as c>=1 !!This is the correct guess.
  • 30. 1. Making a good guess A good guess is vital when applying this method. If the initial guess i h d b dj d l ( i )is wrong, the guess needs to be adjusted later.(Experience) 2. Subleties  G ess is correct b t ind ction proof not ork Guess is correct, but induction proof not work.  Problem is that inductive assumption not strong enough.  Solution: revise the guess by subtracting a lower-order term.g y g  Example: T(n)=T(n/2)+T(n/2)+1.  Guess T(n)=O(n), i.e., T(n)  cn for some c. H T( )   /2  /2 1 1 hi h d i l However, T(n) c n/2+c n/2+1 =cn+1, which does not imply T(n)  cn for any c.  Attempting T(n)=O(n2) will work, but overkill.  New guess T(n)  cn – b will work as long as b  1.
  • 31. 3. Avoiding Pitfall • It is easy to guess T(n)=O(n) (i e T(n)  cn) for T(n)=2T(n/2)+n• It is easy to guess T(n)=O(n) (i.e., T(n)  cn) for T(n)=2T(n/2)+n. • And wrongly prove: – T(n)  2(c n/2)+n •  cn+n • =O(n).  wrongly !!!! • Problem is that it does not pro e the t f of T( ) • Problem is that it does not prove the exact form of T(n)  cn.
  • 32. 4. Changing Variables • Suppose T(n)=2T(n)+lg n• Suppose T(n)=2T(n)+lg n. • Rename m=lg n. so T(2m)=2T(2m/2)+m. • Domain transformation: – S(m)=T(2m), so S(m)=2S(m/2)+m. – Which is similar to T(n)=2T(n/2)+n. So the solution is S( ) O( lg )– So the solution is S(m)=O(m lg m). – Changing back to T(n) from S(m), the solution is T(n) =T(2m)=S(m)=O(m lg m)=O(lg n lg lg n).
  • 33. The Recursion-tree MethodThe Recursion tree Method  Idea:  Each node represents the cost of a single subproblem.  Sum up the costs with each level to get level cost. S ll h l l l Sum up all the level costs to get total cost.
  • 34. Example of recursion treeExample of recursion tree Solve T(n) = T(n/4) + T(n/2) + n2:
  • 35. Example of recursion treeExample of recursion tree Solve T(n) = T(n/4) + T(n/2) + n2: T(n)
  • 36. Example of recursion treeExample of recursion tree Solve T(n) = T(n/4) + T(n/2) + n2: n2 T(n/4) T(n/2)
  • 37. Example of recursion treeExample of recursion tree Solve T(n) = T(n/4) + T(n/2) + n2: n2 (n/4)2 (n/2)2 T( /16) T( /8) T( /8) T( /4)T(n/16) T(n/8) T(n/8) T(n/4)
  • 38. Example of recursion treeExample of recursion tree Solve T(n) = T(n/4) + T(n/2) + n2: n2 ( /16)2 ( /8)2 ( /8)2 ( /4)2 (n/4)2 (n/2)2 (n/16)2 (n/8)2 (n/8)2 (n/4)2 (1)
  • 39. Example of recursion treeExample of recursion tree Solve T(n) = T(n/4) + T(n/2) + n2: 2nn2 ( /16)2 ( /8)2 ( /8)2 ( /4)2 (n/4)2 (n/2)2 (n/16)2 (n/8)2 (n/8)2 (n/4)2 (1)
  • 40. Example of recursion treeExample of recursion tree Solve T(n) = T(n/4) + T(n/2) + n2: 5 2nn2 ( /16)2 ( /8)2 ( /8)2 ( /4)2 (n/4)2 (n/2)2 2 16 5 n (n/16)2 (n/8)2 (n/8)2 (n/4)2 (1)
  • 41. Example of recursion treeExample of recursion tree Solve T(n) = T(n/4) + T(n/2) + n2: 5 2nn2 ( /16)2 ( /8)2 ( /8)2 ( /4)2 (n/4)2 2 16 5 n 225 (n/2)2 (n/16)2 (n/8)2 (n/8)2 (n/4)2 2 256 25 n … (1)
  • 42. Example of recursion treeExample of recursion tree Solve T(n) = T(n/4) + T(n/2) + n2: 5 2nn2 ( /16)2 ( /8)2 ( /8)2 ( /4)2 (n/4)2 2 16 5 n 225 (n/2)2 (n/16)2 (n/8)2 (n/8)2 (n/4)2 2 256 25 n … (1)     1 3 16 52 16 5 16 52 nTotal =     161616 = (n2) geometric series
  • 43. Recursion Tree for T(n)=3T(n/4)+(n2)Recursion Tree for T(n) 3T(n/4) (n ) T(n) cn2 T(n/4) T(n/4) T(n/4) cn2 ( /4)2 c(n/4)2 c(n/4)2T(n/4) T(n/4) T(n/4) c(n/4)2 c(n/4) c(n/4) T(n/16) T(n/16) T(n/16)T(n/16)T(n/16)T(n/16) T(n/16) T(n/16) T(n/16) (a) (b) (c) cn2 2 2 cn2 (3/16)cn2 c(n/4)2 c(n/4)2 c(n/4)2 c(n/16)2 c(n/16)2 c(n/16)2c(n/16)2c(n/16)2c(n/16)2 c(n/16)2 c(n/16)2 c(n/16)2 (3/16)cn2 (3/16)2cn2log 4 n c(n/16) c(n/16) c(n/16)c(n/16)c(n/16)c(n/16) c(n/16) c(n/16) c(n/16) T(1)T(1) T(1)T(1) T(1)T(1) (nlog 4 3 )T(1)T(1) T(1)T(1) T(1)T(1) ( ) 3log4n = nlog 4 3 Total O(n2) (d)
  • 44.  Tree has log4 n+1 levels (0,1,... log4 n ),ie subproblem size for a node at deph ‘i’ is (n/4)i .when the subproblem hits n=1,when (n/4)i =1p ( ) p , ( ) or i=log4 n .  Each level has 3 times more nodes than the level above,so the b f d t d th ‘i’ i 3inumber of nodes at depth ‘i’ is 3i .  Each node i has a cost of c. (n/4i )2 .  Total cost over all nodes at depth i is 3i (n/4i )2 =(3/16)i cn2. Total cost over all nodes at depth i is, 3 .(n/4 ) =(3/16) cn  Last level depth log4 n has 3. log4 n nodes ,each with cost T(1). total cost= (nlog 4 3 )( )
  • 45. Master Method/Theorem  The master method applies to recurrences of the form  T(n) = a T(n/b) + f (n) , 1 ( h b f b bl ) a  1 (the number of subproblems).  b>1, (n/b is the size of each subproblem).  f(n) is a given function and is asymptotically positive.( ) g y p y p for T(n) = aT(n/b)+f(n), n/b may be n/b or n/b. If f( ) O( log a ) f 0 th T( ) ( log a )1. If f(n)=O(nlogb a-) for some >0, then T(n)= (nlogb a ). 2. If f(n)= (nlogb a ), then T(n)= (nlogb a lg n). 3. If f(n)=(nlogb a+) for some >0, and if af(n/b) cf(n) for some( ) ( ) , ( ) ( ) c<1 and all sufficiently large n, then T(n)= (f(n)). In each of the three cases ,we are comparing the function f(n) with the function nlogb a. the function n gb
  • 46. Application of Master TheoremApplication of Master Theorem  T(n) = 9T(n/3)+n;  a=9 b=3 f(n) =n a=9,b=3, f(n) =n  nlogb a = nlog3 9 =  (n2)  f(n)=O(nlog3 9-) for =1f( ) ( )  By case 1, T(n) = (n2).  T(n) = T(2n/3)+1  a=1,b=3/2, f(n) =1  nlogb a = nlog3/2 1 =  (n0) =  (1)  By case 2, T(n)= (lg n).
  • 47. Application of Master TheoremApplication of Master Theorem  T(n) = 3T(n/4)+nlg n;  a=3 b=4 f(n) =nlg n a 3,b 4, f(n) nlg n  nlogb a = nlog4 3 =  (n0.793)  f(n)= (nlog4 3+) for 0.2  Moreover, for large n, c=3/4.  af(n/b) =3(n/4)lg (n/4)  (3/4)nlg n = cf(n)  By case 3 T(n) = (f(n))= (nlg n) By case 3, T(n) = (f(n))= (nlg n).
  • 48. Algorithm DesignAlgorithm Design The strategies which may be used in the design of algorithms,The strategies which may be used in the design of algorithms, including:  Divide-and-conquer algorithms  Dynamic programming  Greedy algorithms  Backtracking algorithms Backtracking algorithms
  • 49. Divide and Conquer The most well known algorithm design strategy:g g gy 1. Divide instance of problem into two or more smaller instances 2. Solve smaller instances recursively 3 Obtain solution to original (larger) instance by combining these3. Obtain solution to original (larger) instance by combining these solutions
  • 50. Time complexity of the general algorithmTime complexity of the general algorithm  Time complexity: T(n)=    2T(n/2)+S(n)+M(n) b , n  1 , n < 1 where S(n) : time for splitting M(n) : time for merging bb : a constant  e.g. Binary search  e g quick sort e.g. quick sort  e.g. merge sort
  • 51. Merge Sort—divide-and-conquerMerge Sort divide and conquer  Divide: divide the n-element sequence into two subproblems of n/2 elements each.p  Conquer: sort the two subsequences recursively using merge sort. If the length of a sequence is 1, do nothing i i i l d i dsince it is already in order.  Combine: merge the two sorted subsequences to produce the sorted answerproduce the sorted answer.
  • 52. Dynamic Programming  One disadvantage of using Divide-and-Conquer is that the process of recursively solving separate sub-instances canprocess of recursively solving separate sub instances can result in the same computations being performed repeatedly since identical sub-instances may arise.  The idea behind dynamic programming is to avoid this pathology Th h d ll li h hi b i i i The method usually accomplishes this by maintaining a table of sub-instance results.
  • 53.  Dynamic Programming is an algorithm design technique for optimization problems. In such problem there can be many solutions. E h l ti h l d i h t fi d l ti ith thEach solution has a value, and we wish to find a solution with the optimal value.  Like divide and conquer, DP solves problems by combining solutions to subproblems.  DP reduces computation by  Solving subproblems in a bottom-up fashion Solving subproblems in a bottom up fashion.  Storing solution to a subproblem the first time it is solved.  Looking up the solution when subproblem is encountered again.  E l Example: Fibonacci numbers computed by iteration.
  • 54. Basic Outline of Dynamic ProgrammingBasic Outline of Dynamic Programming To solve a problem, we need a collection of sub-problems that satisfy a few properties:that satisfy a few properties: 1. There are a polynomial number of sub-problems. 2. The solution to the problem can be computed easily fromp p y the solutions to the sub-problems. 3. There is a natural ordering of the sub-problems from “smallest" to “largest". 4. There is an easy-to-compute recurrence that allows us to t th l ti t b bl f th l ticompute the solution to a sub-problem from the solutions to some smaller sub-problems.
  • 55. Elements of Dynamic Programming (DP) DP is used to solve problems with the following characteristics:  Simple subproblems  We should be able to break the original problem to smaller subproblems that have the same structuresubproblems that have the same structure  Optimal substructure of the problems  The optimal solution to the problem contains within optimal solutions to its subproblems.  Overlapping sub-problems  there exist some places where we solve the same subproblem more there exist some places where we solve the same subproblem more than once.
  • 56. Steps in Dynamic Programming 1. Characterize structure of an optimal solution. 2 Define value of optimal solution recursively2. Define value of optimal solution recursively. 3. Compute the value of an optimal solution 4 Construct an optimal solution from computed values4. Construct an optimal solution from computed values.
  • 57. Knapsack Problem by DPKnapsack Problem by DP  Given some items, pack the knapsack to get the maximum total value. Each item has some weight and some value. Total weight thatg g we can carry is no more than some fixed number W.  So we must consider weights of items as well as their value. Item # Weight Value 1 1 8 2 3 62 3 6 3 5 5
  • 58.  Given a knapsack with maximum capacity W, and a set S consisting of n items  Each item i has some weight wi and benefit value bi (all wi , bi and W are integer values)  P bl H t k th k k t hi i t t l Problem: How to pack the knapsack to achieve maximum total value of packed items?
  • 59. Optimal Binary Search Trees  Problem  Given sequence K = k1 < k2 <··· < kn of n sorted keys, with a Given sequence K k1 k2 kn of n sorted keys, with a search probability pi for each key ki.  Want to build a binary search tree (BST) with minimum expected hsearch cost.  Actual cost = number of items examined.  For key k cost = depth (k )+1 where depth (k ) = depth of k in For key ki, cost = depthT(ki)+1, where depthT(ki) = depth of ki in BST T . • Expected Search Cost   n pk TE )(depth1 ]incostsearch[   i iiT pk 1 )(depth1
  • 60. Example  Consider 5 keys with search probabilities: p1 = 0.25, p2 = 0.2, p3 = 0.05, p4 = 0.2, p5 = 0.3.p1 p2 p3 p4 p5 k2 i depthT(ki) depthT(ki)·pi 1 1 0.25 k1 k4 2 0 0 3 2 0.1 4 1 0.2 k3 k5 5 2 0.6 1.15 Therefore, E[search cost] = 2.15.
  • 61. ExampleExample  p1 = 0.25, p2 = 0.2, p3 = 0.05, p4 = 0.2, p5 = 0.3. i depthT(ki) depthT(ki)·pi 1 1 0.25 k2 2 0 0 3 3 0.15 4 2 0.4 5 1 0 3 k1 k5 5 1 0.3 1.10 k4 Therefore, E[search cost] = 2.10. k3 This tree turns out to be optimal for this set of keys3 This tree turns out to be optimal for this set of keys.
  • 62. Step 1:Optimal SubstructureStep 1:Optimal Substructure  Any subtree of a BST contains keys in a contiguous range k k for some 1 ≤ i ≤ j ≤ nrange ki, ..., kj for some 1 ≤ i ≤ j ≤ n. T T  If T is an optimal BST andp T contains subtree T with keys ki, ... ,kj , then T must be an optimal BST for keys ki, ..., kj.
  • 63. Optimal SubstructureOptimal Substructure  One of the keys in ki, …,kj, say kr, where i ≤ r ≤ j, must be the root of an optimal subtree for these keysmust be the root of an optimal subtree for these keys.  Left subtree of kr contains ki,...,kr1.  Ri ht bt f k t i k +1 k kr  Right subtree of kr contains kr+1, ...,kj. T fi d ti l BST To find an optimal BST:  Examine all candidate roots kr , for i ≤ r ≤ j  Determine all optimal BSTs containing k k and containing ki kr-1 kr+1 kj  Determine all optimal BSTs containing ki,...,kr1 and containing kr+1,...,kj
  • 64. Step 2:Recursive Solutionp  Find optimal BST for ki,...,kj, where i ≥ 1, j ≤ n, j ≥ i1. When j = i1, the tree is empty.  Define e[i, j ] = expected search cost of optimal BST for ki,...,kj.  If j = i1, then e[i, j ] = 0.  If j ≥ i,  Select a root kr, for some i ≤ r ≤ j .  Recursively make an optimal BSTs  f k k th l ft bt d for ki,..,kr1 as the left subtree, and  for kr+1,..,kj as the right subtree.
  • 65. Recursive Solution  When the OPT subtree becomes a subtree of a node:  Depth of every node in OPT subtree goes up by 1.  E t d h t i b (i j) i th f ll th Expected search cost increases by w(i,j) ,is the sum of all the probabilities in the subtree  If kr is the root of an optimal BST for ki,..,kj :r p i j e[i, j ] = e[i, r1] + e[r+1, j] + w(i, j).  But, we don’t know kr. Hence,
  • 66. Step 3:Computing the expected search costStep 3:Computing the expected search cost For each subproblem (i,j), store:For each subproblem (i,j), store:  expected search cost in a table use only entries e[i, j ], where j ≥ i1.  root[i, j ] = root of subtree with keys ki,..,kj, for 1 ≤ i ≤ j ≤ n.  w[1..n+1, 0..n] = sum of probabilities[ ] p  w[i, i1] = 0 for 1 ≤ i ≤ n.  w[i, j ] = w[i, j-1] + pj for 1 ≤ i ≤ j ≤ n.
  • 67. Greedy algorithm  Like dynamic programming, used to solve optimization problems.  Greedy algorithms do not always yield optimal solutions but for many Greedy algorithms do not always yield optimal solutions, but for many problems they do.  A greedy algorithm always makes the choice that looks best at the moment.  Problems exhibit optimal substructure and the greedy-choice property. A d l i h k i h A h h A greedy algorithm works in phases. At each phase:  You take the best you can get right now, without regard for future consequencesco seque ces  You hope that by choosing a local optimum at each step, you will end up at a global optimum
  • 68. Greedy algorithms don'tGreedy algorithms don t  Do not consider all possible paths D id f h i Do not consider future choices  Do not reconsider previous choices  Do not always find an optimal solution greedy strategy usually progresses in a top-down fashion, making one greedy choice after another, reducing each given problem instance to a smaller one.
  • 69. Types of Solutions Produced by Greedy AlgorithmsTypes of Solutions Produced by Greedy Algorithms  Optimal Solutions – The best possible answer that any algorithm could find to the problemalgorithm could find to the problem  Good Solutions – A solution that is near optimal and could be good-enough for some problems  Bad Solutions – A solution that is not acceptable Bad Solutions A solution that is not acceptable  Worst Possible Solution – The solution that is farthest from the goal
  • 70. Elements of Greedy AlgorithmsElements of Greedy Algorithms 1. Determine the optimal substructure of the problem. 2 Greedy Choice Property2. Greedy Choice Property
  • 71. Traveling salesman  A salesman must visit every city (starting from city A), and wants to cover the least possible distance H i i i ( d d) if He can revisit a city (and reuse a road) if necessary  He does this by using a greedy algorithm: He goes to the next nearest city from wherever he is  From A he goes to B  From B he goes to D Thi i i l i A B C2 4  This is not going to result in a shortest path!  The best result he can get now will 3 3 4 4 g be ABDBCE, at a cost of 16  An actual least-cost path from A is ADBCE at a cost of 14 E D ADBCE, at a cost of 14 E
  • 72. Greedy vs dynamic programmingGreedy vs. dynamic programming Dynamic programming: • Make a choice at each step.p • Choice depends on knowing optimal solutions to subproblems. Solve subproblems first. • Solve bottom-up. Greedy: M k h i t h t• Make a choice at each step. • Make the choice before solving the subproblems. • Solve top-downSolve top down.
  • 73. BacktrackingBacktracking  Backtracking is a systematic way to go through all the possible configurations of a search space.  is a methodical way of trying out various sequences of decisions, until you find one that “works”.  Recursion can be used for elegant and easy implementation of Recursion can be used for elegant and easy implementation of backtracking.  Backtracking ensures correctness by enumerating all possibilities.  We can represent the solution space for the problem using a state space tree  The root of the tree represents 0 choices,p ,  Nodes at depth 1 represent first choice  Nodes at depth 2 represent the second choice, etc.
  • 74. Backtracking AlgorithmBacktracking Algorithm  Backtracking is a modified depth-first search of a tree.  Approach Approach 1. Tests whether solution has been found 2. If found solution, return it 3. Else for each choice that can be made a) Make that choice b) Recurb) Recur c) If recursion returns a solution, return it 4. If no choices remain, return failure  Some times called “search tree”
  • 75. Coloring a mapColoring a map  You wish to color a map with not more than four colors  red, yellow, green, blue  Adjacent countries must be in different colors  You don’t have enough information to choose colors  Each choice leads to another set of choices Each choice leads to another set of choices  One or more sequences of choices may (or may not) lead to a solution  Many coloring problems can be solved with backtracking
  • 76. Example ProblemsExample Problems
  • 77. Solve :T(n)=2T(√n)+1 Solution By Changing Variables <==2c lg m-2c S(m)<==c.lg my g g n=2m ie m=lg n Then T(2m )=2T(2m/2 )+1 =O(lg m) T(n)=O(lg m) Ie =O(lg lg n) Assume S(m)=T(2m ) Then S(m)=2S(m/2)+1 Ie O(lg lg n) By guessing S(m)=O lg m Ie <==c.lg m S(m/2)<==c lg m/2S(m/2)<==c.lg m/2 S(m)<==2.c.lg m/2 <==2.c.lg m-lg 22.c.lg m lg 2