Search LIKE %SQL% - Mikhail Khludnev, EPAM
•
1 like•655 views
Presented at Lucene/Solr Revolution 2017
Recommended
Recommended
More Related Content
Similar to Search LIKE %SQL% - Mikhail Khludnev, EPAM
Similar to Search LIKE %SQL% - Mikhail Khludnev, EPAM (20)
More from Lucidworks
More from Lucidworks (20)
Recently uploaded
Recently uploaded (20)
Search LIKE %SQL% - Mikhail Khludnev, EPAM
- 1. SEARCH LIKE %SQL% INFIX SEARCH IN LUCENE / SOLR / ELASTIC SEP 15, 2017
- 2. 2 Talk Title Speaker Name Company SEARCH LIKE %SQL% Mikhail Khludnev EPAM
- 3. 3 • work in Search for 6 years • Apache Lucene/Solr committer for 2 years • speak at LuceneRevolution, BerlinBuzzwords • chief search engineer in EPAM ABOUT ME
- 4. 4 WE ARE
- 5. 5 ESTABLISHED & EXPANDING GLOBAL VERTICALS Award-winning Wealth Management Platform Deep Expertise in Current and Emerging FinTech Working with 5 of the 10 Largest Investment Banks Leading Digital Transformation for Global Retailers Working with largest online travel association (OTA) & largest global hospitality company Recognized M&E Leader by Independent Research Analysts Working with 4 out of the 4 Top Broadcast Networks and 14 out of the top 30 TV Networks to transform consumer-driven media R&D Domain Experts with 700+ Complex Solutions & Services Supporting the Entire Drug Discovery Workflow Working with 9 of the 10 Top Pharma Companies 24-Year History of Leading Product Development Working with 30+ of the top 100 ISVs FINANCIAL SERVICES TRAVEL & CONSUMER SOFTWARE & HI-TECHLIFE SCIENCES AND HEALTHCARE MEDIA & ENTERTAINMENT EMERGING Deep Expertise Offers Innovative Solutions Working with industries ranging from Energy and Utilities to Telecom and Automotive
- 6. 6 • Term and boolean query • Prefix* query • *suffix query • *infix* query • Approaching Suggester • Derivative Terms AGENDA
- 7. 7 • Endeca • MarkLogic • FAST, Google Search Appliance • Sphinx • Apache Lucene • Apache Solr • Elastic SEARCH ENGINES
- 8. 8
- 9. 9
- 10. 10 CUSTOMER PROFILE Any comprehensive text search service • Patent • Legal • Chemistry • Bioinformatics • SQL legacy
- 11. 11 WHERE LIKE %infix% *infix*
- 12. 12 Business Problem/Opportunity • Ill searches for *infix* CHALLENGE Bank of England
- 13. 13 Business Problem/Opportunity • Ill searches for *infix* CHALLENGE
- 14. 14 …at all? Or what’s fast at comparison to it? WHY IT’S A PROBLEM?
- 15. 15 text:foo OR text:bar text:foo AND text:bar THESE SEARCHES ARE (CONSIDERED AS) FAST
- 16. 16 text:foo OR text:bar text:foo AND text:bar O(r) << O(Dall) r – results Dall – all docs THESE SEARCHES ARE (CONSIDERED AS) FAST
- 17. 17 • text:[sci TO scj] • text:sci* WHY THESE ARE STILL FAST?
- 18. 18 TERM EXPANSION • discipline • luscious • science • scilla • scissors text:[sci TO scj] text:sci* text:(science OR scilla OR scissors) O(t)+O(r) t – query terms r - results
- 19. 19 TERM EXPANSION • discipline • luscious • science • scilla • scissors text:[sci TO scj] text:sci* text:(science OR scilla OR scissors) O(t)+O(r)
- 21. 21 ms
- 22. 22 WHAT’S THEN? • asci • disci • discipline • lemnisci • luscious • menisci text:*sci
- 23. 23 WHAT’S THEN? text:*sci O(Tall)+O(r) • asci • disci • discipline • lemnisci • luscious • menisci Tall – all terms r - results
- 25. 25 ms
- 26. 26 0 1000 2000 3000 4000 5000 6000 prefix* *suffix RESPONSE TIME, ms
- 27. 27 text:*sci ReversedWildcardFilterFactory 0enilpicsid 0icsa 0icsid 0icsinem 0icsinmel 0suoicsul asci disci discipline lemnisci luscious menisci text:0ics* WHAT’S THEN? – REVERSE!
- 29. 29
- 30. 30
- 31. 31 WHAT’S THEN? – REVERSE! text:*sci ReversedWildcardFilterFactory 0enilpicsid/0 0icsa/10 0icsid/20 0icsinem/30 0icsinmel/40 0suoicsul/50 asci/60 disci/70 discipline/80 lemnisci/90 luscious/100 menisci/110
- 33. 33 Well.. Postings asci/0 8, 9, 10, 14, 18, 23, 24, 26, 31, 35 disci/10 8, 11, 14, 18, 18, 18, 21, 23, 25, 27 discipline/20 4, 5, 6, 6, 9, 13, 13, 14, 18, 22 lemnisci/30 3, 4, 7, 9, 9, 9, 12, 13, 17, 20 luscious/40 3, 3, 5, 9, 9, 12, 14, 19, 23, 28 menisci/50 0, 2, 5, 6, 11, 13, 17, 22, 27
- 34. 34 Well.. Postings .. ah yeah.. 0enilpicsid 0icsa 0icsid 0icsinem 0icsinmel 0suoicsul asci/0 8, 9, 10, 14, 18, 23, 24, 26, 31, 35 disci/10 8, 11, 14, 18, 18, 18, 21, 23, 25, 27 discipline/20 4, 5, 6, 6, 9, 13, 13, 14, 18, 22 lemnisci/30 3, 4, 7, 9, 9, 9, 12, 13, 17, 20 luscious/40 3, 3, 5, 9, 9, 12, 14, 19, 23, 28 menisci/50 0, 2, 5, 6, 11, 13, 17, 22, 27
- 35. 35 Well.. Postings .. ah yeah.. (and positions!) 0enilpicsid/0 4, 5, 6, 6, 9, 13, 13, 14, 18, 22 0icsa/10 8, 9, 10, 14, 18, 23, 24, 26, 31, 35 0icsid/20 8, 11, 14, 18, 18, 18, 21, 23, 25, 27 0icsinem/30 0, 2, 5, 6, 11, 13, 17, 22, 27 0icsinmel/40 3, 4, 7, 9, 9, 9, 12, 13, 17, 20 0suoicsul/50 3, 3, 5, 9, 9, 12, 14, 19, 23, 28 asci/60 8, 9, 10, 14, 18, 23, 24, 26, 31, 35 disci/70 8, 11, 14, 18, 18, 18, 21, 23, 25, 27 discipline/80 4, 5, 6, 6, 9, 13, 13, 14, 18, 22 lemnisci/90 3, 4, 7, 9, 9, 9, 12, 13, 17, 20 luscious/100 3, 3, 5, 9, 9, 12, 14, 19, 23, 28 menisci/110 0, 2, 5, 6, 11, 13, 17, 22, 27
- 36. 36 benchmark khludnevm$ ant run-‐task -‐Dtask.alg=conf/index-‐5m.alg -‐ Dtask.mem=1000m … [java] -‐-‐> Round 0-‐-‐>1: solr.server:org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient -‐-‐>org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient [java] [java] -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐> starting task: StopSolrServer [java] [java] -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐> Report sum by Prefix (AddDocs) and Round (1 about 1 out of 13) [java] Operation round recsPerRun elapsedSec avgUsedMem avgTotalMem [java] AddDocs 0 5000001 1, 100.41 102,215,792 257,425,408 [java] [java] Reopen Times: [java] 1166 [java] #################### [java] ### D O N E !!! ### [java] #################### BUILD SUCCESSFUL Total time: 19 minutes 28 seconds $ ant run-‐task -‐Dtask.alg=conf/index-‐5m-‐reverse.alg -‐Dtask.mem=1000m [java] -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐> starting task: Rounds ….... [java] -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐> Report sum by Prefix (AddDocs) and Round (1 about 1 out of 13) [java] Operation round recsPerRun rec/s elapsedSec avgUsedMem avgTotalMem [java] AddDocs 0 5000001 3,556.96 1,405.69 75,075,400 257,425,408 [java] [java] Reopen Times: [java] 3114 [java] #################### [java] ### D O N E !!! ### [java] #################### BUILD SUCCESSFUL Total time: 26 minutes 50 seconds $ du -‐hs ../example/schemaless/solr/gettingstarted/data/* 28G ../example/schemaless/solr/gettingstarted/data/index-‐reverse 13G ../example/schemaless/solr/gettingstarted/data/index-‐simple
- 37. 37 0 5 10 15 20 25 30 Baseline (5M en wiki) Reversed Main Index INDEX SIZE, GB 13G 28G
- 38. 38
- 39. 39
- 41. 41 discipline EdgeNGramFilter + ReversedWildcardFilter EdgeNGram Sort discipline cipline iscipline discipline scipline e cipline ine ipline ipline pline iscipline line line ine ne ne pline e scipline*sci* -> sci*
- 43. 43 0 5 10 15 20 25 30 35 Baseline (5M en wiki) Reversed EdgeNGramm Main Index INDEX SIZE, GB 13G 28G ~60G
- 45. 45 SUGGESTER
- 46. 46 AnalyzingInfixSuggester LUCENE-3922: Add Japanese Kanji number normalization to Kuromoji SOLR-4945: Japanese Autocomplete and Highlighter broken 4945 autocomplete broken highlighter japanese solr
- 48. 48
- 49. 49 AnalysingInfixSuggester FOR infix SEARCH • feed AnalysingInfixSuggester with main index’s terms • enable EdgeNGramFilter for AnalysingInfixSuggester discipline iscipline scipline cipline ipline pline line ine ne e
- 50. 50 • 14 M terms -> 79 M EdgeNGramms • 10 min • 3.3 G (25%) BUILDING SUGGESTER INDEX discipline iscipline scipline cipline ipline pline line ine ne e
- 51. 51 ms
- 52. 52 0 5 10 15 20 25 30 35 Baseline (5M en wiki) Reversed EdgeNGramm Suggester Main Index Suggester Index INDEX SIZE, GB 13G 28G 13G+3.3G ~60G
- 53. 53 http://localhost:8901/solr/gettingstarted/suggest? suggest.dictionary = body_txt_en & suggest.q = sci <response> <lst name="responseHeader"><int name="QTime">4</int> </lst> <lst name="suggest"> <lst name="body_txt_en"> <lst name="sci"> <int name="numFound">1000</int> <arr name="suggestions"> <str>scienc</str> <str>scientif</str> <str>scientist</str> <str>disciplin</str> <str>sci</str> <str>conscious</str> <str>category:sci</str> <str>fascin</str> <str>discipl</str> <str>consciou</str> <str>unconsci</str> <str>conscienc</str> <str>oscil</str> <str>neurosci</str> <str>interdisciplinari</str> <str>disciplinari</str> <str>scissor</str> <str>ascii</str> <str>scientolog</str> <str>scimitar</str> <str>conscienti</str> <str>pseudosci</str> <str>rescind</str> <str>priscilla</str> <str>subconsci</str> <str>brescia</str> <str>scion</str> <str>category:scientif</str> <str>infobox_scientist</str> <str>www.newscientist.com</str> <str>resuscit</str> <str>plebiscit</str> <str>user:scimitar</str> <str>multidisciplinari</str> <str>fascia</str> <str>scifi</str> <str>geoscienc</str> <str>www.scifi.com</str> <str>www.sciencemag.org</str> <str>omnisci</str> <str>scipio</str> <str>neuroscientist</str> <str>scientologist</str> …. <int name="QTime">4</int>
- 54. 54 AnalysingInfixSuggester FOR infix SEARCH • feed AnalysingInfixSuggester with main index’s terms • enable EdgeNGramFilter for AnalysingInfixSuggester • override wildcard expansion by calling AnalysingInfixSuggester
- 55. 55 *infix* SEARCH 3834 ms *sci* 142 ms *sci*
- 56. 56
- 57. 57
- 58. 58
- 59. 59 0 1000 2000 3000 4000 5000 6000 prefix* *suffix *substr* *suggester* RESPONSE TIME, ms
- 60. 60 AnalysingInfixSuggester FOR infix SEARCH • existing scalable algorithm • minor customization • no postings explosion • potentially supports NRT
- 61. 61 discipline discipline Suggester asci ci cious cipline .. sci scious scipline us uscious discipline asci disci discipline lemnisci luscious menisci School discipline .. is a required set of actions by a teacher towards a student … Main Index
- 62. 62 discipline discipline Derivative Terms asci ci cious cipline .. sci scious scipline us uscious discipline asci disci discipline lemnisci luscious menisci School discipline .. is a required set of actions by a teacher towards a student …
- 63. 63 • A slight index format change • many terms refer to the same postings list • API is : • indexWriter.deriveTerms(“name”, “name_edge”, new EdgeNgrammTokenFilter()); • search: name_edge:sci* • Hijacking and Injecting codecs LUCENE-7863 • Promising for deep taxonomies. Derivative terms
- 64. 64 *INFIX* SEARCH WITH DERIVATIVE TERMS 127 ms *sci*
- 65. 65 ms
- 66. 66 0 1000 2000 3000 4000 5000 6000 prefix* *suffix *substr* *suggester* *derived* RESPONSE TIME, ms
- 67. 67 0 5 10 15 20 25 30 35 Baseline (5M en wiki) Reversed EdgeNGramm Suggester Derived Terms Main Index Suggester Index INDEX SIZE, GB 13G 28G 13G+3.3G 17G ~60G
- 68. 68 REFERENCES What is in a Lucene index? Adrien Grand https://www.youtube.com/watch?v=T5RmMNDR5XI Automata Invasion. Robert Muir, Michael Mccandless https://www.youtube.com/watch?v=pd2jvy2IbJE • Lucene Search Essentials: Scorers, Collectors and Custom Queries, Mikhail Khludnev https://www.youtube.com/watch?v=X9YovpYj6uo A new Lucene suggester based on infix matches http://blog.mikemccandless.com/2013/06/a-new-lucene-suggester-based-on- infix.html
- 69. 69 REFERENCES What is in a Lucene index? Adrien Grand https://www.youtube.com/watch?v=T5RmMNDR5XI Automata Invasion. Robert Muir, Michael Mccandless https://www.youtube.com/watch?v=pd2jvy2IbJE В поисках Tommy Hilfiger, Михаил Хлуднев https://www.youtube.com/watch?v=Azf4oUL-Dqc A new Lucene suggester based on infix matches http://blog.mikemccandless.com/2013/06/a-new-lucene-suggester-based-on-infix.html