1. A suffix array is an array containing all suffixes of a string sorted in lexicographic order. It allows for efficient string searches, including prefix searches, substring searches, and near neighbor searches. 2. The suffix array can be used to find the starting positions of all occurrences of a pattern or substring within the text in time linear to the pattern length. It supports finding the longest common prefix between two strings in constant time. 3. Suffix arrays are useful for text indexing and string searching, including applications like full text search, bioinformatics, data compression, string mining, and more. They allow locating patterns in strings and calculating similarities between strings.

1. Suffix Array
Solr 2011/12/19
1

2. • (@nobu_k)
• Preferred Infrastructure (PFI FI)
•
•
• Sedue(2 )
2

3. Suffix Array
• Suffix Array(SA):
• ( ) 1
• Sedue
• SA
•
• +Sedue
• ” - ”
3

4. •
•
• ( )
• n-gram(q-gram)
•
4

5. Suffix Array
•
•
• n-gram
•
•
5

6. Suffix( )
0: mississippi
1: ississippi
2: ssissippi
3: sissippi
mississippi
4: issippi
5: ssippi
6: sippi
7: ippi
8: ppi
9: pi
10: i
6

7. Suffix Array
0: mississippi 10: i
1: ississippi 7: ippi
2: ssissippi 4: issippi
3: sissippi 1: ississippi
4: issippi 0: mississippi
5: ssippi 9: pi
6: sippi 8: ppi
7: ippi 6: sippi
8: ppi 3: sissippi
9: pi 5: ssippi
10: i 2: ssissippi
7

8. 10: i
7: ippi • mississippi ’si’
4: issippi
1: ississippi • ’si’
0: mississippi
9: pi •
8: ppi
6:
3:
sippi
sissippi
•
5: ssippi
2: ssissippi • 3 6
8

9. 10: i SA[i]:
7: ippi
4: issippi 10 7 4 1 0 9 8 6 3 5 2
1: ississippi T[i]:
0: mississippi
9: pi m i s s i s s i p p i
8: ppi
6: sippi
3: sissippi 6
5: ssippi T[SA[6]]
2: ssissippi → T[8]
→ “ppi”
9

10. (1/3)
T[i]:
1 2 3 ... n
SA
SA[i]
10

11. (2/3)
RedBull !!
1. RedBull *2
RedBull SA[i]
2.
RedBull
1 2 3 ... n
11

12. (3/3)
3.
RedBull
1 2 3 ... n
4.
( 1, 3), ( 2, 4), ( 3, 2),...,( n, 2)
12

13. • SA
• +
• /n-gram
• SA
•
13

14. SA
• (n-gram )
•
• n-gram
•
•
• “THIS IS IT”
• proximity
14

15. SA
•
•
•
•
• HDD
• ( )
•
•
15

16. •
• ( )
• SAIS
•
• HDD
• (dc3, dc7)
• Sedue Haskell C++
• @tanakh++
16

17. • ( )
•
•
• 1 100GB/day
• Sedue
• SA n-gram
• n-gram
• SA n-gram
•
17

18. HDD
• HDD
• OK
•
•
• SSD
• SSD
• Sedue 20 (80MB)
• SA[i]
18

19. VS
1. SA
•
2.
• SSD+ 500
3.
• O(N) CPU
4.
•
• malloc
19

20. • Sedue 1 56
• : 40
• : 16 (UTF-16)
• 2 3
•
• =
•
• SSD
•
20

21. SA
•
• 4(+1)
• 2-gram
•
• % OK
•
• ” ”
•
21

22. •
•
•
•
•
22

23. : groonga
• Sedue groonga
•
•
• Sedue groonga!!
23

24. :
•
•
• (http://jubat.us/)
• http://github.com/jubatus
• @JubatusOfficial
• with NTT PF
24

25. : Fluentd
• Ruby
• Treasure Data, Inc.
• @frsyuki, @kzk_mover
• Solr
• gem install fluentd
• Visit http://fluentd.org/doc/ now!!
25

26. •
26