Successfully reported this slideshow.

# Lecture10.pdf

Upcoming SlideShare
Suffix Tree and Suffix Array
×

# Lecture10.pdf

Pattern matching

Pattern matching

## More Related Content

### Lecture10.pdf

1. 1. Combinatorial Pattern Matching CS 466 Saurabh Sinha
2. 2. Genomic Repeats • Example of repeats: – ATGGTCTAGGTCCTAGTGGTC • Motivation to find them: – Genomic rearrangements are often associated with repeats – Trace evolutionary secrets – Many tumors are characterized by an explosion of repeats
3. 3. Genomic Repeats • The problem is often more difficult: – ATGGTCTAGGACCTAGTGTTC • Motivation to find them: – Genomic rearrangements are often associated with repeats – Trace evolutionary secrets – Many tumors are characterized by an explosion of repeats
4. 4. l -mer Repeats • Long repeats are difficult to find • Short repeats are easy to find (e.g., hashing) • Simple approach to finding long repeats: – Find exact repeats of short l-mers (l is usually 10 to 13) – Use l -mer repeats to potentially extend into longer, maximal repeats
5. 5. l -mer Repeats (cont’d) • There are typically many locations where an l -mer is repeated: GCTTACAGATTCAGTCTTACAGATGGT • The 4-mer TTAC starts at locations 3 and 17
6. 6. Extending l -mer Repeats GCTTACAGATTCAGTCTTACAGATGGT • Extend these 4-mer matches: GCTTACAGATTCAGTCTTACAGATGGT • Maximal repeat: CTTACAGAT
7. 7. Maximal Repeats • To find maximal repeats in this way, we need ALL start locations of all l -mers in the genome • Hashing lets us find repeats quickly in this manner
8. 8. Hashing: Maximal Repeats • To find repeats in a genome: – For all l -mers in the genome, note the start position and the sequence – Generate a hash table index for each unique l -mer sequence – In each index of the hash table, store all genome start locations of the l -mer which generated that index – Extend l -mer repeats to maximal repeats
9. 9. Pattern Matching • What if, instead of finding repeats in a genome, we want to find all sequences in a database that contain a given pattern? • This leads us to a different problem, the Pattern Matching Problem
10. 10. Pattern Matching Problem • Goal: Find all occurrences of a pattern in a text • Input: Pattern p = p1…pn and text t = t1…tm • Output: All positions 1< i < (m – n + 1) such that the n-letter substring of t starting at i matches p • Motivation: Searching database for a known pattern
11. 11. Exact Pattern Matching: Running Time • Naïve runtime: O(nm) • On average, it’s more like O(m) – Why? • Can solve problem in O(m) time ? – Yes, we’ll see how (later)
12. 12. Generalization of problem: Multiple Pattern Matching Problem • Goal: Given a set of patterns and a text, find all occurrences of any of patterns in text • Input: k patterns p1,…,pk, and text t = t1…tm • Output: Positions 1 < i < m where substring of t starting at i matches pj for 1 < j < k • Motivation: Searching database for known multiple patterns • Solution: k “pattern matching problems”: O(kmn) • Solution: Using “Keyword trees” => O(kn+m) where n is maximum length of pi
13. 13. Keyword Trees: Example • Keyword tree: – Apple
14. 14. Keyword Trees: Example (cont’d) • Keyword tree: – Apple – Apropos
15. 15. Keyword Trees: Example (cont’d) • Keyword tree: – Apple – Apropos – Banana
16. 16. Keyword Trees: Example (cont’d) • Keyword tree: – Apple – Apropos – Banana – Bandana
17. 17. Keyword Trees: Example (cont’d) • Keyword tree: – Apple – Apropos – Banana – Bandana – Orange
18. 18. Keyword Trees: Properties – Stores a set of keywords in a rooted labeled tree – Each edge labeled with a letter from an alphabet – Any two edges coming out of the same vertex have distinct labels – Every keyword stored can be spelled on a path from root to some leaf
19. 19. Multiple Pattern Matching: Keyword Tree Approach • Build keyword tree in O(kn) time; kn is total length of all patterns • Start “threading” at each position in text; at most n steps tell us if there is a match here to any pi • O(kn + nm) • Aho-Corasick algorithm: O(kn + m)
20. 20. Aho-Corasick algorithm
21. 21. “Fail” edges in keyword tree Dashed edge out of internal node if matching edge not found
22. 22. “Fail” edges in keyword tree • If currently at node q representing word L(q), find the longest proper suffix of L(q) that is a prefix of some pattern, and go to the node representing that prefix • Example: node q = 5 L(q) = she; longest proper suffix that is a prefix of some pattern: “he”. Dashed edge to node q’=2
23. 23. Automaton • Transition among the different nodes by following edges depending on next character seen (“c”) • If outgoing edge with label “c”, follow it • If no such edge, and are at root, stay • If no such edge, and at non-root, follow dashes edge (“fail” transition); DO NOT CONSUME THE CHARACTER (“c”) Example: search text “ushers” with the automaton
24. 24. Aho-Corasick algorithm • O(kn) to build the automaton • O(m) to search a text of length m • Key insight: – For every character “consumed”, we move at most one level deeper (away from root) in the tree. Therefore total number of such “away from root” moves is <= m – Each fail transition moves us at least one level closer to root. Therefore total number of such “towards root” moves is <= m (you cant climb up more than you climbed down)
25. 25. Suffix tree • Build a tree from the text • Used if the text is expected to be the same during several pattern queries • Tree building is O(m) where m is the size of the text. This is preprocessing. • Given any pattern of length n, we can answer if it occurs in text in O(n) time • Suffix tree = “modified” keyword tree of all suffixes of text
26. 26. Suffix Tree=Collapsed Keyword Trees • Similar to keyword trees, except edges that form paths are collapsed – Each edge is labeled with a substring of a text – All internal edges have at least two outgoing edges – Leaves labeled by the index of the pattern.
27. 27. Suffix Tree of a Text • Suffix trees of a text is constructed for all its suffixes ATCATG TCATG CATG ATG TG G quadratic Keyword Tree Suffix Tree Time is linear in the total size of all suffixes, i.e., it is quadratic in the length of the text
28. 28. Suffix Trees: Advantages • Suffix trees build faster than keyword trees ATCATG TCATG CATG ATG TG G quadratic Keyword Tree Suffix Tree linear (Weiner suffix tree algorithm) Time to find k patterns of length n: O(m + kn)
29. 29. Suffix Trees: Example
30. 30. Multiple Pattern Matching: Summary • Keyword and suffix trees are used to find patterns in a text • Keyword trees: – Build keyword tree of patterns, and thread text through it • Suffix trees: – Build suffix tree of text, and thread patterns through it
31. 31. Approximate vs. Exact Pattern Matching • So far all we’ve seen exact pattern matching algorithms • Usually, because of mutations, it makes much more biological sense to find approximate pattern matches • Biologists often use fast heuristic approaches (rather than local alignment) to find approximate matches
32. 32. Heuristic Similarity Searches • Genomes are huge: Smith-Waterman quadratic alignment algorithms are too slow • Alignment of two sequences usually has short identical or highly similar fragments • Many heuristic methods (i.e., FASTA) are based on the same idea of filtration – Find short exact matches, and use them as seeds for potential match extension
33. 33. Query Matching Problem • Goal: Find all substrings of the query that approximately match the text • Input: Query q = q1…qw, text t = t1…tm, n (length of matching substrings), k (maximum number of mismatches) • Output: All pairs of positions (i, j) such that the n-letter substring of q starting at i approximately matches the n-letter substring of t starting at j, with at most k mismatches
34. 34. Query Matching: Main Idea • Approximately matching strings share some perfectly matching substrings. • Instead of searching for approximately matching strings (difficult) search for perfectly matching substrings (easy).
35. 35. Filtration in Query Matching • We want all n-matches between a query and a text with up to k mismatches • “Filter” out positions we know do not match between text and query • Potential match detection: find all matches of l -tuples in query and text for some small l • Potential match verification: Verify each potential match by extending it to the left and right, until (k + 1) mismatches are found
36. 36. Filtration: Match Detection • If x1…xn and y1…yn match with at most k mismatches, they must share an l -tuple that is perfectly matched, with l = n/(k + 1) • Break string of length n into k+1 parts, each each of length n/(k + 1) – k mismatches can affect at most k of these k+1 parts – At least one of these k+1 parts is perfectly matched
37. 37. Filtration: Match Verification • For each l -match we find, try to extend the match further to see if it is substantial query Extend perfect match of length l until we find an approximate match of length n with k mismatches text