Introduction to Algorithms
                           6.046J/18.401J
                                      LECTURE 15
    ...
Dynamic programming
     Design technique, like divide-and-conquer.
Example: Longest Common Subsequence (LCS)
• Given two ...
Dynamic programming
     Design technique, like divide-and-conquer.
Example: Longest Common Subsequence (LCS)
• Given two ...
Dynamic programming
     Design technique, like divide-and-conquer.
Example: Longest Common Subsequence (LCS)
• Given two ...
Dynamic programming
     Design technique, like divide-and-conquer.
Example: Longest Common Subsequence (LCS)
• Given two ...
Brute-force LCS algorithm
  Check every subsequence of x[1 . . m] to see
  if it is also a subsequence of y[1 . . n].




...
Brute-force LCS algorithm
  Check every subsequence of x[1 . . m] to see
  if it is also a subsequence of y[1 . . n].
  An...
Towards a better algorithm
Simplification:
1. Look at the length of a longest-common
   subsequence.
2. Extend the algorit...
Towards a better algorithm
Simplification:
1. Look at the length of a longest-common
   subsequence.
2. Extend the algorit...
Towards a better algorithm
Simplification:
1. Look at the length of a longest-common
   subsequence.
2. Extend the algorit...
Recursive formulation
Theorem.
                    c[i–1, j–1] + 1           if x[i] = y[j],
 c[i, j] =          max{c[i–1...
Recursive formulation
Theorem.
           c[i–1, j–1] + 1           if x[i] = y[j],
 c[i, j] = max{c[i–1, j], c[i, j–1]} o...
Recursive formulation
Theorem.
           c[i–1, j–1] + 1           if x[i] = y[j],
 c[i, j] = max{c[i–1, j], c[i, j–1]} o...
Proof (continued)
Claim: z[1 . . k–1] = LCS(x[1 . . i–1], y[1 . . j–1]).
  Suppose w is a longer CS of x[1 . . i–1] and
  ...
Proof (continued)
Claim: z[1 . . k–1] = LCS(x[1 . . i–1], y[1 . . j–1]).
  Suppose w is a longer CS of x[1 . . i–1] and
  ...
Dynamic-programming
         hallmark #1

                   Optimal substructure
              An optimal solution to a p...
Dynamic-programming
         hallmark #1

                   Optimal substructure
              An optimal solution to a p...
Recursive algorithm for LCS
  LCS(x, y, i, j)
    if x[i] = y[ j]
        then c[i, j] ← LCS(x, y, i–1, j–1) + 1
        e...
Recursive algorithm for LCS
  LCS(x, y, i, j)
    if x[i] = y[ j]
        then c[i, j] ← LCS(x, y, i–1, j–1) + 1
        e...
Recursion tree
m = 3, n = 4:                            3,4
                                          3,4

            2,4...
Recursion tree
m = 3, n = 4:                            3,4
                                          3,4

            2,4...
Recursion tree
m = 3, n = 4:                            3,4
                                          3,4

            2,4...
Dynamic-programming
         hallmark #2

                Overlapping subproblems
              A recursive solution conta...
Dynamic-programming
         hallmark #2

                Overlapping subproblems
              A recursive solution conta...
Memoization algorithm
Memoization: After computing a solution to a
subproblem, store it in a table. Subsequent calls
check...
Memoization algorithm
Memoization: After computing a solution to a
subproblem, store it in a table. Subsequent calls
check...
Memoization algorithm
Memoization: After computing a solution to a
subproblem, store it in a table. Subsequent calls
check...
Dynamic-programming
           algorithm
IDEA:                              A B C B D                                     ...
Dynamic-programming
           algorithm
IDEA:                              A B C B D                                     ...
Dynamic-programming
           algorithm
IDEA:                              A B C B D                                     ...
Dynamic-programming
           algorithm
IDEA:                              A B C B D                                     ...
Upcoming SlideShare
Loading in...5
×

Mychurch File Upload

317

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
317
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
20
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Mychurch File Upload

  1. 1. Introduction to Algorithms 6.046J/18.401J LECTURE 15 Dynamic Programming • Longest common subsequence • Optimal substructure • Overlapping subproblems Prof. Charles E. Leiserson November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.1
  2. 2. Dynamic programming Design technique, like divide-and-conquer. Example: Longest Common Subsequence (LCS) • Given two sequences x[1 . . m] and y[1 . . n], find a longest subsequence common to them both. November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.2
  3. 3. Dynamic programming Design technique, like divide-and-conquer. Example: Longest Common Subsequence (LCS) • Given two sequences x[1 . . m] and y[1 . . n], find a longest subsequence common to them both. “a” not “the” November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.3
  4. 4. Dynamic programming Design technique, like divide-and-conquer. Example: Longest Common Subsequence (LCS) • Given two sequences x[1 . . m] and y[1 . . n], find a longest subsequence common to them both. “a” not “the” x: A B C B D A B y: B D C A B A November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.4
  5. 5. Dynamic programming Design technique, like divide-and-conquer. Example: Longest Common Subsequence (LCS) • Given two sequences x[1 . . m] and y[1 . . n], find a longest subsequence common to them both. “a” not “the” x: A B C B D A B BCBA = LCS(x, y) y: B D C A B A functional notation, but not a function November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.5
  6. 6. Brute-force LCS algorithm Check every subsequence of x[1 . . m] to see if it is also a subsequence of y[1 . . n]. November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.6
  7. 7. Brute-force LCS algorithm Check every subsequence of x[1 . . m] to see if it is also a subsequence of y[1 . . n]. Analysis • Checking = O(n) time per subsequence. • 2m subsequences of x (each bit-vector of length m determines a distinct subsequence of x). Worst-case running time = O(n2m) = exponential time. November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.7
  8. 8. Towards a better algorithm Simplification: 1. Look at the length of a longest-common subsequence. 2. Extend the algorithm to find the LCS itself. November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.8
  9. 9. Towards a better algorithm Simplification: 1. Look at the length of a longest-common subsequence. 2. Extend the algorithm to find the LCS itself. Notation: Denote the length of a sequence s by | s |. November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.9
  10. 10. Towards a better algorithm Simplification: 1. Look at the length of a longest-common subsequence. 2. Extend the algorithm to find the LCS itself. Notation: Denote the length of a sequence s by | s |. Strategy: Consider prefixes of x and y. • Define c[i, j] = | LCS(x[1 . . i], y[1 . . j]) |. • Then, c[m, n] = | LCS(x, y) |. November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.10
  11. 11. Recursive formulation Theorem. c[i–1, j–1] + 1 if x[i] = y[j], c[i, j] = max{c[i–1, j], c[i, j–1]} otherwise. November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.11
  12. 12. Recursive formulation Theorem. c[i–1, j–1] + 1 if x[i] = y[j], c[i, j] = max{c[i–1, j], c[i, j–1]} otherwise. Proof. Case x[i] = y[ j]: 1 2 i m x: L 1 2 = j n y: L November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.12
  13. 13. Recursive formulation Theorem. c[i–1, j–1] + 1 if x[i] = y[j], c[i, j] = max{c[i–1, j], c[i, j–1]} otherwise. Proof. Case x[i] = y[ j]: 1 2 i m x: L 1 2 = j n y: L Let z[1 . . k] = LCS(x[1 . . i], y[1 . . j]), where c[i, j] = k. Then, z[k] = x[i], or else z could be extended. Thus, z[1 . . k–1] is CS of x[1 . . i–1] and y[1 . . j–1]. November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.13
  14. 14. Proof (continued) Claim: z[1 . . k–1] = LCS(x[1 . . i–1], y[1 . . j–1]). Suppose w is a longer CS of x[1 . . i–1] and y[1 . . j–1], that is, | w | > k–1. Then, cut and paste: w || z[k] (w concatenated with z[k]) is a common subsequence of x[1 . . i] and y[1 . . j] with | w || z[k] | > k. Contradiction, proving the claim. November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.14
  15. 15. Proof (continued) Claim: z[1 . . k–1] = LCS(x[1 . . i–1], y[1 . . j–1]). Suppose w is a longer CS of x[1 . . i–1] and y[1 . . j–1], that is, | w | > k–1. Then, cut and paste: w || z[k] (w concatenated with z[k]) is a common subsequence of x[1 . . i] and y[1 . . j] with | w || z[k] | > k. Contradiction, proving the claim. Thus, c[i–1, j–1] = k–1, which implies that c[i, j] = c[i–1, j–1] + 1. Other cases are similar. November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.15
  16. 16. Dynamic-programming hallmark #1 Optimal substructure An optimal solution to a problem (instance) contains optimal solutions to subproblems. November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.16
  17. 17. Dynamic-programming hallmark #1 Optimal substructure An optimal solution to a problem (instance) contains optimal solutions to subproblems. If z = LCS(x, y), then any prefix of z is an LCS of a prefix of x and a prefix of y. November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.17
  18. 18. Recursive algorithm for LCS LCS(x, y, i, j) if x[i] = y[ j] then c[i, j] ← LCS(x, y, i–1, j–1) + 1 else c[i, j] ← max{ LCS(x, y, i–1, j), LCS(x, y, i, j–1)} November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.18
  19. 19. Recursive algorithm for LCS LCS(x, y, i, j) if x[i] = y[ j] then c[i, j] ← LCS(x, y, i–1, j–1) + 1 else c[i, j] ← max{ LCS(x, y, i–1, j), LCS(x, y, i, j–1)} Worst-case: x[i] ≠ y[ j], in which case the algorithm evaluates two subproblems, each with only one parameter decremented. November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.19
  20. 20. Recursion tree m = 3, n = 4: 3,4 3,4 2,4 2,4 3,3 3,3 1,4 1,4 2,3 2,3 2,3 2,3 3,2 3,2 1,3 1,3 2,2 2,2 1,3 1,3 2,2 2,2 November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.20
  21. 21. Recursion tree m = 3, n = 4: 3,4 3,4 2,4 2,4 3,3 3,3 1,4 1,4 2,3 2,3 2,3 2,3 3,2 3,2 m+n 1,3 1,3 2,2 2,2 1,3 1,3 2,2 2,2 Height = m + n ⇒ work potentially exponential. November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.21
  22. 22. Recursion tree m = 3, n = 4: 3,4 3,4 2,4 2,4 same 3,3 3,3 subproblem 1,4 1,4 2,3 2,3 2,3 2,3 3,2 3,2 m+n 1,3 1,3 2,2 2,2 1,3 1,3 2,2 2,2 Height = m + n ⇒ work potentially exponential., but we’re solving subproblems already solved! November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.22
  23. 23. Dynamic-programming hallmark #2 Overlapping subproblems A recursive solution contains a “small” number of distinct subproblems repeated many times. November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.23
  24. 24. Dynamic-programming hallmark #2 Overlapping subproblems A recursive solution contains a “small” number of distinct subproblems repeated many times. The number of distinct LCS subproblems for two strings of lengths m and n is only mn. November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.24
  25. 25. Memoization algorithm Memoization: After computing a solution to a subproblem, store it in a table. Subsequent calls check the table to avoid redoing work. November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.25
  26. 26. Memoization algorithm Memoization: After computing a solution to a subproblem, store it in a table. Subsequent calls check the table to avoid redoing work. LCS(x, y, i, j) if c[i, j] = NIL then if x[i] = y[j] then c[i, j] ← LCS(x, y, i–1, j–1) + 1 same else c[i, j] ← max{ LCS(x, y, i–1, j), as before LCS(x, y, i, j–1)} November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.26
  27. 27. Memoization algorithm Memoization: After computing a solution to a subproblem, store it in a table. Subsequent calls check the table to avoid redoing work. LCS(x, y, i, j) if c[i, j] = NIL then if x[i] = y[j] then c[i, j] ← LCS(x, y, i–1, j–1) + 1 same else c[i, j] ← max{ LCS(x, y, i–1, j), as before LCS(x, y, i, j–1)} Time = Θ(mn) = constant work per table entry. Space = Θ(mn). November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.27
  28. 28. Dynamic-programming algorithm IDEA: A B C B D A B Compute the 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 table bottom-up. B 0 0 1 1 1 1 0 0 1 1 1 1 1 1 1 1 D 0 0 1 1 1 2 0 0 1 1 1 2 2 2 2 2 C 0 0 1 0 0 1 2 2 2 2 2 2 2 2 2 2 A 0 1 1 0 1 1 2 2 2 3 3 2 2 2 3 3 B 0 1 2 0 1 2 2 3 3 3 4 2 3 3 3 4 A 0 1 2 0 1 2 2 3 3 4 4 2 3 3 4 4 November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.28
  29. 29. Dynamic-programming algorithm IDEA: A B C B D A B Compute the 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 table bottom-up. B 0 0 1 1 1 1 0 0 1 1 1 1 1 1 1 1 Time = Θ(mn). D 0 0 1 1 1 2 2 2 0 0 1 1 1 2 2 2 C 0 0 1 0 0 1 2 2 2 2 2 2 2 2 2 2 A 0 1 1 0 1 1 2 2 2 3 3 2 2 2 3 3 B 0 1 2 0 1 2 2 3 3 3 4 2 3 3 3 4 A 0 1 2 0 1 2 2 3 3 4 4 2 3 3 4 4 November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.29
  30. 30. Dynamic-programming algorithm IDEA: A B C B D A B Compute the 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 table bottom-up. B 0 0 1 1 1 1 0 0 1 1 1 1 1 1 1 1 Time = Θ(mn). D 0 0 1 1 1 2 2 2 0 0 1 1 1 2 2 2 Reconstruct C 0 0 1 0 0 1 2 2 2 2 2 2 2 2 2 2 LCS by tracing backwards. A 0 1 1 0 1 1 2 2 2 3 3 2 2 2 3 3 B 0 1 2 0 1 2 2 3 3 3 4 2 3 3 3 4 A 0 1 2 0 1 2 2 3 3 4 4 2 3 3 4 4 November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.30
  31. 31. Dynamic-programming algorithm IDEA: A B C B D A B Compute the 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 table bottom-up. B 0 0 1 1 1 1 0 0 1 1 1 1 1 1 1 1 Time = Θ(mn). D 0 0 1 1 1 2 2 2 0 0 1 1 1 2 2 2 Reconstruct C 0 0 1 0 0 1 2 2 2 2 2 2 2 2 2 2 LCS by tracing backwards. A 0 1 1 0 1 1 2 2 2 3 3 2 2 2 3 3 Space = Θ(mn). B 0 1 2 0 1 2 2 3 3 3 4 2 3 3 3 4 Exercise: A 0 1 2 0 1 2 2 3 3 4 4 2 3 3 4 4 O(min{m, n}). November 7, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L15.31
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×