Поисковые сервера и библиотеки как Solr/Lucene/Elastic выполняют поиск по ключевым словам. Иногда появляется "странное" требование поиска по подстрокам, также известное в SQL как WHERE field LIKE %infix%. В докладе речь пойдёт про структуры данных индекса Lucene; о том почему этот поиск алгоритмически сложный; и как выполнить его с помощью имеющихся структур данных с минимальной кастомизацией Solr. Доклад не рассматривает SQL, Solr SQL, и суффиксные массивы; не требует от слушателей специфических знаний про поисковые машины, так как посвящён структурам данных обратного индекса и конечным трансляторам (FST).

1. SEARCH LIKE %SQL%
SUBSTRING SEARCH IN LUCENE / SOLR / ELASTIC
MAY 27, 2017

2. 2
• work in Search 6 years
• Apache Lucene committer for 1.5 years
• speak at LuceneRevolution, BerlinBuzzwords
ABOUT ME

3. 3
• Simple and multi-term query
• Prefix* query
• *suffix query
• *substring* query
• AnalyzingInfixSuggester
AGENDA

4. 4

5. 5

6. 6
Any comprehensive text search service
• Patent
• Legal
• Chemistry
• Bioinformatics
• SQL legacy
CUSTOMER PROFILE

7. 7
Business Problem/Opportunity
• Ill searches for *infix*
CHALLENGE

8. 8
…at all?
Or what’s fast at comparison to it?
WHY IT’S A PROBLEM?

9. 9
text:foo OR text:bar
text:foo AND text:bar
THESE SEARCHES ARE (CONSIDERED AS) FAST

10. 10
text:foo text:bar
+text:foo +text:bar
THESE SEARCHES ARE (CONSIDERED AS) FAST

11. 11
text:foo text:bar
+text:foo +text:bar
O(n)
THESE SEARCHES ARE (CONSIDERED AS) FAST

12. 12
•text:[sci TO scj]
•text:sci*
WHY THESE ARE SILL FAST?

13. 13
TERM EXPANSION
• discipline
• luscious
• science
• scilla
• scissors
text:[sci TO scj]
text:sci*
text:(science scilla scissors)
O(t)+O(n)

14. 14
TERM EXPANSION
• discipline
• luscious
• science
• scilla
• scissors
text:[sci TO scj]
text:sci*
text:(science scilla scissors)
O(t)+O(n)

15. 15

16. 16
WHAT’S THEN?
• asci
• disci
• discipline
• lemnisci
• luscious
• menisci
text:*sci

17. 17
WHAT’S THEN?
text:*sci
• asci
• disci
• discipline
• lemnisci
• luscious
• menisci

18. 18

19. 19
text:*sci
ReversedWildcardFilterFactory
0enilpicsid
0icsa
0icsid
0icsinem
0icsinmel
0suoicsul
asci
disci
discipline
lemnisci
luscious
menisci
text:0ics*
WHAT’S THEN? – REVERSE!

20. 20
ReversedWildcardFilterFactory

21. 21

22. 22

23. 23
WHAT’S THEN? – REVERSE!
text:*sci
ReversedWildcardFilterFactory
0enilpicsid/0
0icsa/10
0icsid/20
0icsinem/30
0icsinmel/40
0suoicsul/50
asci/60
disci/70
discipline/80
lemnisci/90
luscious/100
menisci/110

24. 24
ReversedWildcardFilterFactory

25. 25
Well.. Postings
asci/0 8, 9, 10, 14, 18, 23, 24, 26, 31, 35
disci/10 8, 11, 14, 18, 18, 18, 21, 23, 25, 27
discipline/20 4, 5, 6, 6, 9, 13, 13, 14, 18, 22
lemnisci/30 3, 4, 7, 9, 9, 9, 12, 13, 17, 20
luscious/40 3, 3, 5, 9, 9, 12, 14, 19, 23, 28
menisci/50 0, 2, 5, 6, 11, 13, 17, 22, 27

26. 26
Well.. Postings .. ah yeah..
0enilpicsid
0icsa
0icsid
0icsinem
0icsinmel
0suoicsul
asci/0 8, 9, 10, 14, 18, 23, 24, 26, 31, 35
disci/10 8, 11, 14, 18, 18, 18, 21, 23, 25, 27
discipline/20 4, 5, 6, 6, 9, 13, 13, 14, 18, 22
lemnisci/30 3, 4, 7, 9, 9, 9, 12, 13, 17, 20
luscious/40 3, 3, 5, 9, 9, 12, 14, 19, 23, 28
menisci/50 0, 2, 5, 6, 11, 13, 17, 22, 27

27. 27
Well.. Postings .. ah yeah.. (and positions!)
0enilpicsid/0 4, 5, 6, 6, 9, 13, 13, 14, 18, 22
0icsa/10 8, 9, 10, 14, 18, 23, 24, 26, 31, 35
0icsid/20 8, 11, 14, 18, 18, 18, 21, 23, 25, 27
0icsinem/30 0, 2, 5, 6, 11, 13, 17, 22, 27
0icsinmel/40 3, 4, 7, 9, 9, 9, 12, 13, 17, 20
0suoicsul/50 3, 3, 5, 9, 9, 12, 14, 19, 23, 28
asci/60 8, 9, 10, 14, 18, 23, 24, 26, 31, 35
disci/70 8, 11, 14, 18, 18, 18, 21, 23, 25, 27
discipline/80 4, 5, 6, 6, 9, 13, 13, 14, 18, 22
lemnisci/90 3, 4, 7, 9, 9, 9, 12, 13, 17, 20
luscious/100 3, 3, 5, 9, 9, 12, 14, 19, 23, 28
menisci/110 0, 2, 5, 6, 11, 13, 17, 22, 27

28. 28
benchmark khludnevm$ ant run-task -Dtask.alg=conf/index-5m.alg -
Dtask.mem=1000m
…
[java] --> Round 0--
>1: solr.server:org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrC
lient-->org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient
[java]
[java] ------------> starting task: StopSolrServer
[java]
[java] ------------> Report sum by Prefix (AddDocs) and Round (1
about 1 out of 13)
[java]
Operation round recsPerRun elapsedSec avgUsedMem avgTotalMem
[java] AddDocs 0 5000001 1,
100.41 102,215,792 257,425,408
[java]
[java] Reopen Times:
[java] 1166
[java] ####################
[java] ### D O N E !!! ###
[java] ####################
BUILD SUCCESSFUL
Total time: 19 minutes 28 seconds
$ ant run-task -Dtask.alg=conf/index-5m-reverse.alg -Dtask.mem=1000m
[java] ------------> starting task: Rounds
…....
[java] ------------> Report sum by Prefix (AddDocs) and Round (1 about 1 out
of 13)
[java] Operation round
recsPerRun rec/s elapsedSec avgUsedMem avgTotalMem
[java] AddDocs 0
5000001 3,556.96 1,405.69 75,075,400 257,425,408
[java]
[java] Reopen Times:
[java] 3114
[java] ####################
[java] ### D O N E !!! ###
[java] ####################
BUILD SUCCESSFUL
Total time: 26 minutes 50 seconds
$ du -hs ../example/schemaless/solr/gettingstarted/data/*
28G ../example/schemaless/solr/gettingstarted/data/index-reverse
13G ../example/schemaless/solr/gettingstarted/data/index-simple

29. 29

30. 30

31. 31
text:*sci*
AND THEN

32. 32
luscious
EdgeNGramFilter + ReversedWildcardFilter
EdgeNGram Reverse Sort
luscious suoicsul csul
lusciou uoicsul icsul
luscio oicsul l
lusci icsul oicsul
lusc csul sul
lus sul suoicsul
lu ul ul
l l uoicsul
*sci* -> ics*

33. 33
luscious
EdgeNGramFilter + ReversedWildcardFilter
EdgeNGram Reverse Sort
luscious suoicsul csul
lusciou uoicsul icsul
luscio oicsul l
lusci icsul oicsul
lusc csul sul
lus sul suoicsul
lu ul ul
l l uoicsul
*sci* -> ics*

34. 34
a
csa
csid
csinem
csinmel
csul
d
el
em
enilpicsid
icsa
icsid
icsinem
icsinmel
icsul
id
ilpicsid
inem
inmel
l
lpicsid
m
mel
nem
nilpicsid
nmel
oicsul
picsid
sa
sid
sinem
sinmel
sul
suoicsul
ul
uoicsul

35. 35
https://discuss.codechef.com/questions/21385/a-tutorial-on-suffix-arrays
https://issues.apache.org/jira/browse/SOLR-9974
http://labs.carrotsearch.com/jsuffixarrays.html
SUFFIX ARRAY

36. 36
AUTO-COMPLETE

37. 37
AnalyzingInfixSuggester
LUCENE-3922: Add Japanese Kanji number normalization to Kuromoji
SOLR-4945: Japanese Autocomplete and Highlighter broken
4945
autocomplete
broken
highlighter
japanese
solr

38. 38
AnalysingInfixSuggester TO RESCUE!

39. 39
http://localhost:8901/solr/gettingstarted/suggest?
suggest.dictionary = body_txt_en & suggest.q = sci
<response>
<lst name="responseHeader"><int
name="QTime">4</int>
</lst>
<lst name="suggest">
<lst name="body_txt_en">
<lst name="sci">
<int name="numFound">1000</int>
<arr name="suggestions">
<str>scienc</str>
<str>scientif</str>
<str>scientist</str>
<str>disciplin</str>
<str>sci</str>
<str>conscious</str>
<str>category:sci</str>
<str>fascin</str>
<str>discipl</str>
<str>consciou</str>
<str>unconsci</str>
<str>conscienc</str>
<str>oscil</str>
<str>neurosci</str>
<str>interdisciplinari</str>
<str>disciplinari</str>
<str>scissor</str>
<str>ascii</str>
<str>scientolog</str>
<str>scimitar</str>
<str>conscienti</str>
<str>pseudosci</str>
<str>rescind</str>
<str>priscilla</str>
<str>subconsci</str>
<str>brescia</str>
<str>scion</str>
<str>category:scientif</str>
<str>infobox_scientist</str>
<str>www.newscientist.com</str>
<str>resuscit</str>
<str>plebiscit</str>
<str>user:scimitar</str>
<str>multidisciplinari</str>
<str>fascia</str>
<str>scifi</str>
<str>geoscienc</str>
<str>www.scifi.com</str>
<str>www.sciencemag.org</str>
<str>omnisci</str>
<str>scipio</str>
<str>neuroscientist</str>
<str>scientologist</str>
….

40. 40

41. 41

42. 42

43. 43

44. 44

45. 45
AnalysingInfixSuggester FOR SUBSTRING SEARCH
•existing scalable algorithm
•minor customization
•no postings explosion
•potentially supports NRT
•embeddable as fileformat

46. 46

47. 47
REFERENCES
What is in a Lucene index? Adrien Grand
https://www.youtube.com/watch?v=T5RmMNDR5XI
Automata Invasion. Robert Muir, Michael Mccandless
https://www.youtube.com/watch?v=pd2jvy2IbJE
В поисках Tommy Hilfiger, Михаил Хлуднев
https://www.youtube.com/watch?v=Azf4oUL-Dqc
A new Lucene suggester based on infix matches
http://blog.mikemccandless.com/2013/06/a-new-lucene-suggester-based-on-infix.html

48. 48
ANYWAY
text:*a*

49. 49
CONTACTS
Mikhail_Khludnev@EPAM.COM
mkhl@apache.org
https://plus.google.com/+MikhailKhludnev