Your SlideShare is downloading. ×
0
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Succinct Data Structure for Analyzing Document Collection
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Succinct Data Structure for Analyzing Document Collection

15,146

Published on

Invited talk at the workshop ALSIP2011. The topic includes, Rank/Select data structure, and its implementation, wavelet trees and its applications, suffix arrays, suffix trees, Burrows Wheeler's …

Invited talk at the workshop ALSIP2011. The topic includes, Rank/Select data structure, and its implementation, wavelet trees and its applications, suffix arrays, suffix trees, Burrows Wheeler's Transform, FM-index, Top-k document query.

Published in: Technology, Education
0 Comments
12 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
15,146
On Slideshare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
132
Comments
0
Likes
12
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  1. 2011/12/1 ALSIP2011@Takamatsu Succinct Data Structure for Analyzing Document Collection Preferred Infrastructure Inc. Daisuke Okanohara hillbig@preferred.jp
  2. Agenda Succinct Data Structures (20 min.) Succinct data Full-text  Rank/Select, Wavelet Tree Structures Index Full-text Index (30 min.)  Suffix Array/Tree, BWT Compressed Full- text Index Compressed Full-Text Index (20 min.)  FM-index Analyzing Document Analyzing Document Collection (20 min.) Collection  Document Statistics (for mining)  Top-k query
  3. Background Very large document collections appear everywhere  Twitter : 200 billion tweets per year. (6K tweets per sec.)  Modern DNA Sequencing  E.g. Illumina HiSeq 2000 produces 40G bases per day Analyzing these large document collections require new data structures and algorithms
  4. Notation T[0, n] : string T[0]T[1]…T[n] T[i, j] : subarray from T[i] to T[j] inclusively T[i, j) : subarray from T[i] to T[j-1] lg m : log2 m (# of bits to represent m cases) Σ : the alphabet set of string σ := |Σ| : the number of alphabets n : length of input string m : length of query string
  5. PreliminariesEmpirical Entropy (1/2) Empirical entropy indicates the compressibility of the string Zero-order empirical entropy of T H0(T) := ∑c n(c)/n log (n/p(c))  n(c) : the number of occurrences of c in T  n : the length of T  The lower-bound for the average code length of compressor not using context informationT = abccbaabcab, n(a)=4, n(b)=4, n(c)=3, H0(T) = 1.57 bit
  6. Empirical Entropy (2/2) k-th order empirical entropy of T Hk(T) = ∑w∈Σk-1 nw/n H0(Tw) :  Tw : the characters followed by a substring w  nw : the number of occurrences of a string w in T.  The lower-bound for the average code length of compressor using the following substring of length k-1 as a context information T = abccbaabcab Ta=bac Tb=acaa Tc=bcb, H2(T) = 0.33 bit Hierarchy of entropy: log σ ≧ H0(T) ≧ H1(T) ≧ … Hk(T)  Typical English Text, lg σ=8 H0=4.5 , H5=1.7
  7. Succinct Data Structure
  8. Rank/Select DictionaryFor a bit vector B[0, n), B[i] ∈{0, 1}, consider following operations Lookupc(B, i) : B[i] Rankc(B, i) : The number of c’s in B[0…i) Selectc(B, i) : The (i+1)-th occurrence of c in BExample:i 0123456789B 0111011001 Rank1(B,6)=4 Select0(B,2)=2The number of 1’s in B[0, 6) The (2+1)-th occurrence of 0 in B
  9. Rank/Select Dictionary Usage (1/2)Subset For U = {0, 1, 2, …, n-1}, a subset of U, V⊆U can be represented by a bit vector B[0, n-1], s.t. B[i] = 1 if i∈V, and B[i]=0 otherwise. Select(B, i) is the (i+1)-th smallest element in V, Rank(B, i) = |{e|e∈V, e<i}| V = {0, 4, 7, 10, 15} i 0123456789012345 B = 1000100100100001
  10. Rank/Select Dictionary Usage (2/2)Partial Sums For a non-negative integer array P[0, m), we concatenate each unary codes of P[i] to build a bit array B[0, n)  An unary code of a non-negative integer p≧0 is 0p1  E.g. unary codes of 0, 3, 10 are 1, 0001, 00000000001  From the definition, n=m+∑p[i] and the number of ones is m P[i] = Select1(B, i) - Select1(B, i-1). Cum[k] = ∑i=0…kP[i]. = Select1(B, k) - k (# of 0’s before (k+1)-th 1) Find r s.t. Cum[r] ≦ x < Cum[r+1] by Rank1(B, Select0(B, x))P = {3, 5, 2, 0, 4, 7}B = 000100000100110000100000001Cum[4] = Select(B, 4) – 4 = 14
  11. Data structures for Rank/Select Bit vector B[0, n) with m ones can be represented in lg (n, m) bits  lg (n, m) ≒ m lg2e + m lg (n/m) when m << n The implementations of Rank/Select Dictionary are divided into the case when the bit vector is dense, sparse, and very sparse. Case 1. Dense 01101011010100110101100101 Case 2. Sparse (RRR) 00001001000010001000100010 Case 3. Very Sparse (SDArray) 00000000000010000000001000
  12. Rank/Select DictionaryCase 1. Dense Divide B into large blocks of length l=(log n)2 Divide each large block into small blocks of length s=(log n)/2 For each large block, store L[i]=Rank1(B, il) For each small block, store S[i]=Rank1(B, is) - Rank1(B, is / l) Build a table P(X) that stores the number of ones in X of length s Rank1(B, i) = L[i/l] + S[i/s] + P(B[i/s*s, i))  three table lookup. O(1) time L[1] Total size of L, S, P is o(n) +S[4] Select can also be supported in O(1) time using n+o(n) bits +P[X] In practice, P(X) is often replaced with popcount(X)
  13. Rank/Select DictionaryCase 2. Sparse (RRR) Case 1. is redundant when m << n A small block X is represented by the class and the offset.  The class C(X) is the number of ones in X  e.g . X=101, C(X) = 2.  The offset O(X) is the lexicographical rank among C(X)  e.g. X0=011, X1=101, X2 =110, O(X0)=0, O(X1)=1, O(X2)=2 Store a pointer of every (log n)2/2 blocks and an offset for each elementB = 111110100101010000100111010C = 3 2 1 2 1 0 1 3 1O = 0 2 2 1 1 0 2 2 1
  14. Rank/Select ImplementationCase 2. Sparse (RRR) Ida: when m << n, many classes (# of ones in small block) are small, and the offsets are also small Total size is nH0(B) + o(n) bits. Rank is supported in O(1) time  We can also support select in O(1) time using the same working space. Note: RRR is also efficient when ones and zeros are clustered.  The following bit is not sparse, but can be efficiently stored by RRR encoding. 000000100000100011111111110111110111111111100000000010000001 many 0s many 1s many 0s
  15. Rank/Select ImplementationCase 3. Very Sparse (SD array) Case 2. is not efficient for very sparse case.  o(n) term cannot be negligible when nH0 is very small Let S[i] := select1(B, i) and t = lg (n/m). L[i] := the lowest t bits of S[i] P[i] := S[i] / 2t Since P is weakly increasing integers, use prefix sums to represent H  Concatenate unary codes of P[i]-P[i-1]. B=0001011000000001 S[0, 3] = [3, 5, 6, 15], t = lg(16 / 4) = 2. L[0, 3] = [3, 1, 2, 3] P[i] = [0, 1, 1, 3], H = 1011001
  16. Rank/Select ImplementationCase 3. Very Sparse (SDArray) L is stored explicitly, and H is stored with Rank/Select dictionary. Size H : 2m bits (H[m] ≦ m) L : m log2 (n/m) bits. select(B, i) = (select1(H, i)-i)2t + L[i]  O(1) time rank(B, i) = linear search from 1 + rank1(H, select0(H, i))  expected O(1) time.
  17. Rank/Select for large alphabets Consider Rank/Select on a string with many alphabets I will explain only wavelet trees here since it is very very useful ! I will not discuss other topics of rank / select for large alphabets  Multi-array wavelet trees [Ferragina+ ACM Transaction 07]  Permutation based approach [Golynski SODA 06]  Alphabet partitioning [Barbey+ ISSAC 10]
  18. Wavelet Tree Wavelet Tree = alphabet tree + bit vectors in each node T = ahcbedghcfaehcgda b c d e f g h
  19. Wavelet Tree Wavelet Tree = alphabet tree + bit vectors in each node ahcbedghcfaehcgd Filter characters according to its value 010010110 For each character c := T[i] (i = 0…n), we move c to the left child and store 0 acbdc hegh if c is the descendant of the left child. We move c to the right child and store 1 otherwise.a b c d e f g h
  20. Wavelet Tree Wavelet Tree = alphabet tree + bit vectors in each node Recursively continue this process. Store a alphabet tree and bit vectors of each node ahcbedghcfaehcgd 0100101101011010 0 1acbdcacd heghfehg01011011 10110011aba cdccd efe hghhg010 01001 010 10110a b c d e f g h
  21. Wavelet Tree (contd.) Total bit length is n lg σ bits  The height is l2σ, and there are n bits at each depth. 0100101101011010 01011011 10110011 Height is lg σ010 01001 010 10110 a b c d e f g n
  22. Wavelet Tree (Lookup) Lookup(T, pos) pos := 9 e.g. Lookup(T, 9) 0100101101011010 b := B[pos] = 0 pos := Rankb(B, pos) = 4 0 1 b := B[pos] = 0 01011011 10110011 pos := Rankb(B, pos) = 2010 01001 010 10110 b := B[pos] = 0 a b c d e f g n Return “c”
  23. Wavelet Tree (Rank) Rankc (T, pos) ahcbedghcfaehcgd e.g. Rankc(T, 9) 0100101101011010 b := B[9] = 0 pos := Rank0(B, 0) = 4 0 1 b := B[4] = 4 01011011 10110011 pos := Rank1(B, 4) = 2 b := B[2] = 0010 01001 010 10110 pos := Rank0(B, 2) = 1 a b c d e f g n Return 1
  24. Wavelet Tree (Select) Selectc (T, i) Example Selectc(T, 1) 0100101101011010 i := Select0(B,4) = 8 0 1 01011011 10110011 i := Select1(B,2) = 4 i := Select0(B,1) = 2010 01001 010 10110 a b c d e f g n i=1
  25. Wavelet Tree (Quantile List) Wavelet Tree is so called double-sorted array Quantile(T, p, sp, ep) : Return the (p+1)-th largest value in T[sp, ep) E.g. Quantile(T, 5, 4, 12) 0100101101011010 # of zeros in B[4,12] is 3 3 < (5+1) and go to right 0 1 01011011 # of zeros in B[1,6] is 3 10110011 3 > 2 and go to left # of zeros in B[0, 2] is 2010 01001 010 10110 2 = 2 and go to right a b c d e f g n Return “f” 43782614 ahcbedghcfaehcgd
  26. Wavelet Tree (top-k Frequent List ) FreqList (T, k, sp, ep) : Return k most frequent values in T[sp, ep) E.g. FreqList(T, 5, 4, 12) Greedy Search on the tree. 0100101101011010 Larger ranges will be examined first. 0 1 01011011 10110011 Q := priority queue {# of elems, node} Q.push {[ep-sp, root]} while q = Q.pop()010 01001 010 10110 If q is leaf, report q and return Q.push([0’s in range, q.left]) Q.push([1’s in range, q.right]) a b c d e f g n end while
  27. Wavelet Tree (RangeFreq, RangeList) RangeList (T, sp, ep, min, max) : Return {T[i]} s.t. min≦ [i]<max, sp≦i<ep RangeFreq(T, sp, ep, min, max) : Return |RangeList(T, sp, ep, min, max)| E.g. RangeList(T, 4, 12, c, e) 0123457890123456 Enumerate nodes that go to between 0100101101011010 min and max 0 1 For RangeList, # of enumerated 01011011 10110011 nodes is O(log σ + occ) For RangeFreq, # of enumerated n odes is O(log σ)010 01001 010 10110 a b c d e f g n
  28. Wavelet Tree Summary Input T[0…n), 0≦T[i]<σ Many operations can be supported in O(log σ) time using wavelet tree Description TimeLookup T[i] O(log σ)Rankc(T, p) the number of c in T[0…p) O(log σ)Selectc(T, i) the pos of (i+1)-th c in T O(log σ)Quantile(T, s, e, r) (r+1)-th largest val in T[s, e) O(log σ)FreqList(T, k, s, e) k most frequent values in T[s, e) Expected O(klog σ) Worst O(σ)RangeFreq(T, s, e, x, y) items x ≦ T[i] < y in T[s, e) O(log σ)RangeList(T, s, e, x, y) List T[i] s.t. x ≦ T[i] < y in T[s, e) O(log σ + occ)NextValue(T, s, e, x) smallest T[r] > x, s.t. s ≦ r ≦ e O(log σ)
  29. Wavelet tree improvements Huffman Tree shaped Wavelet Tree (HWT)  If the alphabet tree corresponds to the Huffman tree for T, the size of the wavelet tree becomes nH0 + o(nH0) bits.  Expected query time is also O(H0) when the query distribution is same as in T Wavelet Tree for large alphabets  For large alphabet set (e.g. σ ≒, n) the overhead of alphabet tree, and bit vectors could large.  Concatenate bit vectors of same depth. In traversal of tree, we iteratively compute the range of the original bit array.
  30. 2D Search using Wavelet Tree  Given a 2D data set D={(xi, yi)}, find all points in [sx, sy] x [ex, ey]  By using a wavelet tree, wen can solve this in O(log ymax + occ) time Remove columns with no data. If there are more than one data in the same column, we copy the column so that each column has exactly one data. 0 1 2 3 4 5 6 0 1 2 2 4 4 6 6 60 * 0 * Store the x information using a bit array1 * * 1 * * Xb = 1010110011001112 23 * * * 3 * * * Since each column has4 * 4 * one value, we can convert it to the string5 * 5 * T =3136130466 * 6 *
  31. 2D Search using Wavelet Tree (contd.) [sx, sy] x [ex, ey] region is found by  s’ := Select0(X, sx)+1 , e’ := Select0(X, ex)+1  s’’ := Rank1(X, s’) e’’ := Rank1(X, e’)  Perform RangeFreq(s’’, e’’, sy, ey) 0 1 2 3 4 5 6 0 * 1 * * X = 101011001100111 2 3 * * * T =313613046 4 * 5 * 6 *
  32. Full-text Index
  33. Full-text Index Given a text T[0, n) and a query Q[0, m), support following operations. occ(T, Q) : Report the number of occurrence of Q in T locate(T, Q) :Report the positions of occurrence of Q in T Examplei 0123456789AT abracadabra occ(T, “abra”)=2locate(T, “abra”)={0, 7}
  34. Suffix Array Let Si = T[i, n] be the i-th suffix of T Sort {Si}i=0…n by its lexicographical order Store sorted index in SA[0…n] in n lg n bits (SSA[i] < SSA[i+1]) S0 abracadabra$ S11 $ 11 S1 bracadabra$ S10 a$ 10 S2 racadabra$ S7 abra$ 7 S3 acadabra$ S0 abracadabra$ 0 S4 cadabra$ S3 acadabra$ 3 S5 adabra$ S5 adabra$ 5 S6 dabra$ S8 bra$ 8 S7 abra$ S1 bracadabra$ 1 S8 bra$ S4 cadabra$ 4 S9 ra$ S6 dabra$ 6 S10 a$ S9 ra$ 9 S11 $ S2 racadabra$ 2
  35. Search using SA Given a query Q[0…m), find the range [sp, ep) s.t. sp := max{i | Q ≦ SSA[i]} Q=“ab”, sp=2, ep=4 ep := min{i | Q > SSA[i]} S11 $ S10 a$ Since {SSA[i]} are sorted, sp and ep S7 abra$ can be found by binary search in O(m log n) time S0 abracadabra$  Each string comparison require O(m) time. S3 acadabra$ The occurrence positions are SA[sp, ep) S5 adabra$ S8 bra$  if ep=sp then Q is not found on T S1 bracadabra$ S4 cadabra$ Occ(T, Q) : O(m log n) time. S6 dabra$ Locate(T, Q) : O(m log n + Occ) time S9 ra$ S2 racadabra$
  36. Suffix Tree 11 10 $ A trie for all suffixes of T $ bra 7  Remove all nodes with only one child. $ c c 0  Store suffix indexes (SA) at leaves  Store depth information at internal nodes, and a 3 d edge information is recovered from depth and SA . 5 Fact: # of internal nodes is at most N-1 bra $ c 8  Proof: Consider inserting a suffix one by one. At each insertion, at most one internal node will c 1 be build. d 4 ra 6 Many string operations can be supported by using $ 9 a suffix tree. c 2
  37. Burrows Wheeler Transform (BWT) i SA BWT Suffix Convert T into BWT as follows 0 11 a $ BWT[i] := T[SA[i] – 1] 1 10 r a$  if SA[i] = 0, BWT[i] = T[n] 2 7 d abra$ abracdabra$->ard$rcaaaabb 3 0 $ abracadabra$ 4 3 r acadabra$ 5 5 c adabra$ 6 8 a bra$ 7 1 a bracadabra$ 8 4 a cadabra$ 9 6 a dabra$ 10 9 b ra$ 11 2 b racadabra$
  38. Characteristics of BWT Reversible transformation  We can recover T from only BWT Easily compressed  Since similar suffixes tend to have preceding characters, same characters tend to gather in BWT. bzip2 Implicit suffix information  BWT has same information as suffix arrays, and can be used for full-text index (so called FM-index).
  39. Example : Before BWTWhen Farmer Oak smiled, the corners of his mouth spread till they werewithin an unimportant distance of his ears, his eyes were reduced to chinks,and diver gingwrinkles appeared round them, extending upon hiscountenance like the rays in a rudimentary sketch of the rising sun. HisChristian name was Gabriel, and on working days he was a young man ofsound judgment, easy motions, proper dress, and general good character. OnSundays he was a man of misty views, rather given to postponing, andhampered by his best clothes and umbrella : upon the whole, one who felthimself to occupy morally that vast middle space of Laodiceanneutrality whichlay between the Communion people of the parish and the drunken section, --that is, he went to church, but yawned privately by the time the con+gegationreached the Nicene …
  40. Example : After BWTooooooioororooorooooooooorooorromrrooomooroooooooormoorooororioooroormmmmmuuiiiiiIiuuuuuuuiiiUiiiiiioooooooooooorooooiiiioooioiiiiiiiiiiioiiiiiieuiiiiiiiiiiiiiiiiiouuuuouuUUuuuuuuooouuiooriiiriirriiiiriiiiiiaiiiiioooooooooooooiiiouioiiiioiiuiiuiiiiiiiiiiiiiiiiiiiiiiioiiiiioiuiiiiiiiiiiiiioiiiiiiiiiiiiioiiiiiiuiiiioiiiiiiiiiiiioiiiiiiiiiioiiiioiiiiiiioiiiaiiiiiiiiiiiiiiiiioiiiiiioiiiiiiiiiiiiiiiuiiiiiiiiiiiiiiiiiioiiiiiiiioiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiuuuiioiiiiiuiiiiiiiiiiiiiiiiiiiiiiiioiiiiuioiuiiiiiiioiiiiiiiuiiiiiiiiiiiiiiiiiiiiiiiiiiiiiioaoiiiiioioiiiiiiiioooiiiiiooioiiioiiiiiouiiiiiiiiiiiiooiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiioiiiiiiiiiiiiiiiiiiioiooiiiiiiiiiiioiiiiiuiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiioiiiiiiiiiiiiioiiiuiiiiiiiiiioiiiiiiiiiiiiuoiiioiiioiiiiiiiiiiiiiiiiiiiiiiuiiiiuuiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiuiuiiiiiuuiiiiiiiiiiiiiiiiiiiiiiiiuiiiiiiiiiiiiiiiiiiiiiiiiiiiioiiiiiiioiiiiiiiiiiiiiiiiiiiiioiiiiiiiiioiiiiuiiiioiiiioiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiioiioiiiiiiuiiiiiiiiiiiiiiiooiiiiiiiiiiiiiiiiiiiioooiiiiiiiioiiiiouiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiioiiioiiiiiiioiiiiiioiiiiuiuoiiiiiiiiioiiiiiiioiiiiiiiiiiiiiiiiiiiiiiiiiiiioioi….
  41. LF-mapping i BWT Suffix (First char is F) Let F[i] = T[SA[i]] 0 a1 $  First characters of suffixes 1 r a1$ Fact : the order of same chars 2 d a2bra$ in BWT and F is same 3 $ a3bracadabra$ 4 r a4cadabra$ 5 c a5dabra$ 6 a2 bra$ 7 a3 bracadabra$ 8 a4 cadabra$ 9 a5 dabra$ 10 b ra$ 11 b racadabra$
  42. LF-mapping (1/3) i BWT Suffix (First char is F) Let F[i] = T[SA[i]] 0 a1  First characters of suffixes 1 r a1 Fact : the order of same chars 2 d a2 in BWT and F is same 3 $ a3bracadabra$ Proof: the order of chars in BWT 4 r a4 is determined by the following 5 c a5 Same suffixes. 6 a2 the order of chars in F is also 7 a3 bracadabra$ determined by the following 8 a4 suffixes. These are same ! 9 a5 10 11
  43. LF-mapping (2/3) i BWT Suffix (First char is F) Let F[i] = T[SA[i]] 0 a $  First characters of sorted suffixes 1 r a$ 2 d abra$ Let C[c] be the number of 3 $ abracadabra$ characters smaller than c in T  C[c] = beginning position of F[i] = c 4 r acadabra$ For B[i]=c, its corresponding char 5 c adabra$ in F[i] is found by Rankc(T, i) + C[c] 6 a bra$ 7 a bracadabra$ 8 a cadabra$ 9 a dabra$ 10 b ra$ 11 b racadabra$
  44. LF-mapping (3/3) Let ISA[0, n) s.t. ISA[r]=p if SA[p] = r  SA[i] converts the rank into the position, and ISA[i] vice versa. For c := BWT[r], we define two functions LF(r) := Rankc(T, r) + C[c] = ISA[SA[r] -1] PSI(r) := Selectc(T, r– C[c]) = ISA[SA[r] + 1]  LF and PSI are inverse functions each other. Using LF and PSI, we can move a text in its position order.  PSI = forward by one char , LF = backward by one char e.g. T = abracadabra SA[LF[r]] SA[r] SA[PSI[r]]
  45. Decoding of BWT using LF-mappingr := beg // the rank of T[0, n), i.e. ISA[0]while Output BWT[r] r := PSI(r)until r != begThat’s all !In practice, PSI values are pre-computed using O(n) time
  46. Compressed Full-text Index
  47. Compressed Full-Text Index Full-text can be compressed  Moreover, we can recover original text from compressed index (self-index) I explain FM-index only here. Other important compressed full-text indices  Compressed Suffix Arrays  LZ-index  LZ77-based
  48. FM-index P’=ra appear in P P=a appeared in [1, 5) i BWT Suffix Using LF-mapping for searching 0 a $ pattern. 1 r a$ 2 d abra$ Assume that a range [sp, ep) 3 $ abracadabra$ correspond to a pattern P. 4 r acadabra$ Then if a pattern P’ = cP occurs, 5 c adabra$ then [sp, ep) should be preceded by c 6 a bra$ 7 a bracadabra$ 8 a cadabra$ 9 a dabra$ 10 b ra$ 11 b racadabra$
  49. FM-index P’=ra appear in P P=a appeared in [1, 5) i BWT Suffix Using LF-mapping for searching 0 a $ pattern. 1 r1 a$ 2 d abra$ Assume that a range [sp, ep) 3 $ abracadabra$ correspond to a pattern P. 4 r2 acadabra$ Then if a pattern P’ = cP occurs, 5 c adabra$ then [sp, ep) is preceded by c 6 a bra$ 7 a bracadabra$ The corresponding suffixes can be 8 a cadabra$ found by Rankc(BWT, sp) + C[c] 9 a dabra$ and Rankc(BWT, ep) + C[c] 10 b r1a$ 11 b r2acadabra$
  50. FM-indexInput query : Q[0, m)Output a range [sp, ep) corresponding to Qsp := 0, ep := nfor i = m-1 to 0 c := Q[c] sp := C[c] + Rankc(BWT, sp) ep := C[c] + Rankc(BWT, ep)end forReturn (sp, ep)Occ(T, Q) = ep - spLocate(T, Q) = SA[sp, ep)
  51. Storing Sampled SA’s i SA R Store sampled SAs instead of original SA. 0 0 Sample SA[p] s.t. SA[p] = tk for (t=0…) 1 0 We also store a bit array R to indicate the index of 2 0 sampled SAs. 3 0 1 4 0 Recall that LF(r) = ISA[SA[r]-1] for c = BWT[c]. 5 0 If SA[p] is not sampled, we use LF to move on SA 6 8 1  SA[r] = SA[LF(r)]-1 = SA[LF2(r)]-2 = … = SA[LFk(r)]-k 7 0 We move on SA until the sampled point is found 8 4 1 using at most k LF operations. 9 0 10 0 Set k = log1+εn. The size of A is (n/k) log n = o(n) 11 0 The lookup of SA[i] requires O(log1+εn logσ) time.
  52. Grammar Compression on SA [R. Gonzalez+ CPM07] SA can also be compressed using grammar based compressions Let DSA[i] := SA[i] – SA[i-1] (for i > 0) If BWT[i] = BWT[i+1] = … = BWT[i+run] then DSA[i+1, i+run] = DSA[k+1, k+run] for k := LF[i]  DSA contains many repetition if BWT has many runs. DSA can be compressed using grammar compression  [R. Gonzelez+ CPM07] used Re-pair but other grammar compressions can be used as long as they support fast decompression.
  53. Compression Boosting in FM-index The size of FM-index using wavelet trees is nH0 + o(nH0) bits We can improve the space to nHK by compression boosting Compression Boosting [P. Ferragina+ ACM T 07]  Divide BWT into blocks using contexts of length k  Build a separate wavelet tree for each block Implicit Compression Boosting [Makinen+ SPIRE 07]  Wavelet Tree + compressed bit vectors using RRR in Wavelet Tree The size is close to optimal: the overhead is at most σk+1 log n, which is o(n) when k ≦ ((1-ε)logσn)-1 b aaaa.. b aabb.. Divide BWT’s a aacc.. using following b aacd.. suffix of length d abcd.. d abcc..
  54. Compression Boosting using fixed length block[Karkkaine+ SPIRE 11] Dividing BWT into blocks of fixed length can also achieve nHK  Block size is σ(log n)2  To analyze the working space, we calculate the overhead incurred of conversion from context-sensitive partition into fixed-length partition Note. This encoding is very similar to the encoding used in bzip2
  55. SSA: Huffman WTAFFMI : Context Sensitive Partition of BWTFixed Block : Fixed Block+ RRR : Represent bit vectorsin WT using RRR encoding
  56. Summary of state-of-the-art Give an input String T, compute BWT for T (=B). Divide B into blocks of fixed length Each block is stored in a Huffman-shaped wavelet tree. A bit-vector in a wavelet tree is stored using RRR representation. For English, 2-3 bits per character, 1 pattern 0.06 msec.
  57. Analyzing Document Collection
  58. Document Query DL(D, P) : Document Listing  List the distinct documents where P appears DLF(D, P) :Document Listing with Frequency  The result of DL(D, P) + frequency of P in each document Top(k, D, P) : Top-k Retrieval  List the k documents where P appears with largest scores S(D, P)  e.g. S(D, P) : term frequency, document’s score (e.g. Page rank) In the previous section we consider hit positions, and here we consider hit documents  We need to convert hit positions into hit documents.  This is not easy.
  59. Index for document collection For a document collection D={d1, d2, …, dD} Build T = d1#d2#d3#...#dD and build an full-text index for T  We also use a bit array to indicate the beginning positions of docs.  Given a hit position in T, we can convert it to the hit doc in O(1) time. Baseline of DL and DLF  Perform Locate(T, Q) and convert each hit positions into a hit doc each independently.  However, this requires O(occ(T, Q)) time, and occ(T, Q) could be much larger than # of hit documents
  60. DocArray and PrevArray (1/2)[Muthukrishnan SODA 02] DocArray : D[0, n] s.t. D[i] is the doc ID of SA[i].  DL(D, Q) is now to list all distinct values in the hit range D[sp, ep) PrevArray : P[0, n] s.t. P[i]=max{j < I, D[j]=D[i]}  P[i] can be simulated as selectD[i] (D, rankD[i](D, i-1)) i 0 1 2 3 4 5 6 D 66 77 66 88 77 77 88 P -1 -1 0 -1 1 4 3
  61. DocArray and PrevArray (2/2) Alg. DocList reports all distinct doc IDs in [sp, ep) in O(|DL|) time.  Call DocList(sp, sp, ep)DocList(gsp, sp, ep) i 0 1 2 3 4 5 6 IF sp=ep D 66 77 66 88 77 88 88 RETURN; (min, pos) := RMQ(P, sp, ep) P -1 -1 0 -1 1 3 5 IF min ≧ gsp RMQ(P, 2, 7) = -1 (=P[3]) RETURN; Report D[3] Report D[pos] DocList(gsp, sp, pos) DocList(gsp, pos+1, ep) RMQ(P, 2, 3) = 0 (=P[2]) RMQ(P, 5, 7) = 3 (=P[5]) Report D[2] Return
  62. Wavelet Tree + Grammar Compression[Navarro+ SEA 11] Build a wavelet tree for D. Using Freqlist(k, sp, ep), we can solve DL and DLF in O(log σ) time. However storing D requires n lg |D| bits. This is very large. As in compression of suffix arrays using grammar compression, D can also be compressed by grammar compression Apply a grammar compression to bit arrays in wavelet trees, and support rank/select operations Currently, this is the fastest/smallest approach in practice.
  63. Suffix Tree based approach (1/4)[W.K. Hon+ FOCS 09] Let ST be the generalized suffix tree for a collection of documents A leaf l is marked with document d if l corresponds to d An internal node l is marked with d if at least two children of v contain leaves marked with d  An internal node can be marked with many values d. 1 c 2 a b c 3 7 a 10 c 4 5 6 8 9 11 12 13 14 a c b a a c c b c
  64. Suffix Tree based approach (2/4)  In every node v marked with d, we store a pointer ptr(v, d) to its lowest ancestor u such that u is also marked with d  ptr(v, d) also has a weight corresponds to relevance score.  If such ancestor does not exist, ptr(v, d) points to a super root node. 1 cptr(7, a) = 2 2 a b c ptr(13, a) = 2 3 7 a 10 c 4 5 6 8 9 11 12 13 14 a c b a a c c b c
  65. Suffix Tree Based Approach (3/4) Lemma 1. The total number of pointers ptr(*,*) in T is at most 2n Lemma 2. Assume that document d contains a pattern P and v is the node corresponding to P. Then there exist a unique pointer ptr(u, d), such that u is in the subtree of v and ptr(u, d) points to an ancestor of v 1 c Example: when v = 10. ptr(10, c)=2, 2 a b c and ptr(13, b)=2. These ptrs correspond to c the occurrence of P in b and c. 10 11 12 13 c c b
  66. Suffix Tree Based Approach (4/4) To enumerate document corresponding to the node x, find all ptrs that begins in the subtree of x, and points to the ancestor of x.  These ptrs uniquely corresponds to each hit documents (Lemma) x
  67. Geometric Interpretation pdpd(v, c) := -1 c a bdepth of ptr(v, c) 0 c c 1 a c b a c bplace all ptrs in 2 a a c ca point (x, pd(v,c)) 1 2 2 2 4 5 6 7 8 9 10 11 12 13 14 depth 1 c 0 2 a b c 1 3 7 a 10 c 2 4 5 6 8 9 11 12 13 14 3 a c b a a c c b c
  68. Document Query = three-sided 2D query When v := locus(Q) d := depth(v), and r be the maximal index assigned to v or its descendants. DL can be solve by RangeList(v, r, 0, d-1) pd -1 v=2 c a b v=10 0 c c 1 a c b a c b 2 a a c c 1 2 2 2 4 5 6 7 8 9 10 11 12 13 14
  69. Top-K query Each ptr(v, d) can store any score function depended on Q, D. For each node v, f ptr(x, d) points to vTop-K Theorem [Navarro+ SODA 12] k most highly weighted points in the range [a, b] x [0, h] can be reported in decreasing order of their weights in O(h+k) time In practice, we can use a priority queue on each node.
  70. Bibliography[Navarro+ SEA 12] Practical Compressed Document Retrieval, G. Navarro, S.J. Puglisi, and D. Valenzuela, SEA 2012[Navarro+ SODA 12] Top-k Document Retrieval in Optimal Time and LinearSpace, SODA 2012[T. Gagie+ TCS+ 11] New Algorithms on Wavelet Trees and Applications toInformation Retrieval, TCS 2011[G. Navarro+ 11] Practical Top-K Document Retrieval in Reduced Space[Karkkaine+ SPIRE 11] Fixed Block Compression Boosting in FM-indexes, J.Karkkainen, S. J. Puglisi, SPIRE 2011[W.K. Hon+ FOCS 09] Space-Efficient Framework for Top-k String RetrievalProblems, W. K. Hon, R. Shah, J. S. Vitter, FOCS 2009
  71. [Navarro+ SEA 11] “Practical Compressed Document Retrieval”, GonzaloNavarro , Simon J. Puglisi, and Daniel Valenzuela,, SEA 2011[Navarro+ SODA 12] Top-k Document Retrieval in Optimal Time and LinearSpace , Gonzalo Navarro, Yakov Nekrich, SODA 2012[Gagie+ JTCS 11] New Algorithms on Wavelet Trees and Applications toInformation Retrieval, Travis Gagie , Gonzalo Navarro , Simon J. Puglisi, J. ofTCS 2011

×