Make Plone Search Act Like Google Using Solr

3,525 views

Published on

Solr is a powerful open source search engine server which has become a popular choice for extending the search capabilities of Plone sites. The default configuration works well, but how do you answer the client's request to "Make my search just like Google's"?

In this talk we will take a look at the various options that are available for configuring Solr's schema and configuration. We will discuss how to set up stop words, spell checking, n-grams and alternate query handlers. We will see what effect these settings will have on the search results and find out how to debug problems when they arise.

Published in: Technology, News & Politics

Make Plone Search Act Like Google Using Solr

  1. 1. Make Plone Search ActLike Google Using Solr Clayton Parker | Senior Web Developer PLONE CONFERENCE 2011
  2. 2. Who Am I
  3. 3. What will we learn? PLONE CONFERENCE 2011
  4. 4. What will we learn? PLONE CONFERENCE 2011• Intro to Solr
  5. 5. What will we learn? PLONE CONFERENCE 2011• Intro to Solr• Brief overview of Plone integration points
  6. 6. What will we learn? PLONE CONFERENCE 2011• Intro to Solr• Brief overview of Plone integration points• Solr configuration
  7. 7. What will we learn? PLONE CONFERENCE 2011• Intro to Solr• Brief overview of Plone integration points• Solr configuration• Solr schema setup
  8. 8. What will we learn? PLONE CONFERENCE 2011• Intro to Solr• Brief overview of Plone integration points• Solr configuration• Solr schema setup• Debugging tips and tricks
  9. 9. PLONE CONFERENCE 2011What is Solr ?
  10. 10. Version Madness PLONE CONFERENCE 2011 1.x (up to 1.4) 1.5 (number abandoned) 3.x (merge of Lucene and Solr)
  11. 11. Books PLONE CONFERENCE 2011
  12. 12. PLONE CONFERENCE 2011Integration
  13. 13. PLONE CONFERENCE 2011alm.solrindex
  14. 14. PLONE CONFERENCE 2011collective.solr
  15. 15. Solr Configuration
  16. 16. Query Handlers PLONE CONFERENCE 2011• Standard• Disjunction Max (DisMax)• Extended DisMax (experimental)
  17. 17. DisMax PLONE CONFERENCE 2011• Multiple index searches• Boosting• Friendlier to end users
  18. 18. DisMax PLONE CONFERENCE 2011 Index Name qf=SearchableText^1.0 substring^0.2 Weight
  19. 19. MinShouldMatch PLONE CONFERENCE 2011 mm=100% All terms required mm=50% Half of the terms required mm=-2 All but two terms required
  20. 20. MinShouldMatch PLONE CONFERENCE 2011 mm=2<-25% 9<-3 2 or less 3-9 terms all more than 9 terms are but 25% terms all but required required three are required
  21. 21. Spelling Component PLONE CONFERENCE 2011<searchComponent name="spellcheck" class="solr.SpellCheckComponent"> <lst name="spellchecker"> <str name="name">default</str> <str name="classname">solr.IndexBasedSpellChecker</str> <str name="buildOnCommit">true</str> <str name="spellcheckIndexDir">path/to/spellcheck</str> <!-- The field that will contain the dynamic spelling data --> <str name="field">spell</str> <str name="accuracy">0.5</str> </lst> <!-- Control indexing and query of spelling data --> <str name="queryAnalyzerFieldType">spell-text</str></searchComponent>
  22. 22. Spelling Schema PLONE CONFERENCE 2011<fieldType name="spell-text" class="solr.TextField"> <analyzer> <tokenizer class="solr.ICUTokenizerFactory"/> <filter class="solr.ICUFoldingFilterFactory"/> <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/> </analyzer></fieldType>
  23. 23. Solr Schema
  24. 24. Index vs Query PLONE CONFERENCE 2011 http://www.cominvent.com/2011/04/04/solr-architecture-diagram/
  25. 25. PLONE CONFERENCE 2011
  26. 26. PLONE CONFERENCE 2011Character Filters
  27. 27. PLONE CONFERENCE 2011Character Filters Tokenizer
  28. 28. PLONE CONFERENCE 2011Character Filters Tokenizer Filters
  29. 29. PLONE CONFERENCE 2011Character Filters Tokenizer Filters
  30. 30. Complete Field PLONE CONFERENCE 2011 <fieldType name="text" class="solr.TextField" positionIncrementGap="100"> <analyzer type="index"> <charFilter class="solr.MappingCharFilterFactory" mapping="mapping.txt"/> <tokenizer class="solr.ICUTokenizerFactory"/> <filter class="solr.ICUFoldingFilterFactory"/> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/> <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/> <filter class="solr.EnglishPorterFilterFactory" protected="protwords.txt"/> <filter class="solr.WordDelimiterFilterFactory" splitOnCaseChange="1" splitOnNumerics="1" stemEnglishPossessive="1" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" preserveOriginal="1"/> </analyzer> <analyzer type="query"> <charFilter class="solr.MappingCharFilterFactory" mapping="mapping.txt"/> <tokenizer class="solr.ICUTokenizerFactory"/> <filter class="solr.ICUFoldingFilterFactory"/> <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/> <filter class="solr.EnglishPorterFilterFactory" protected="protwords.txt"/> <filter class="solr.WordDelimiterFilterFactory" splitOnCaseChange="1" splitOnNumerics="1" stemEnglishPossessive="1" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" preserveOriginal="1"/> <filter class="solr.PositionFilterFactory"/> </analyzer> </fieldType>
  31. 31. Copy Field PLONE CONFERENCE 2011 <copyField source="SearchableText" dest="spell"/> <copyField source="SearchableText" dest="substring"/>
  32. 32. PLONE CONFERENCE 2011Character Filters• Process text before tokenizing• Remove irrelevant characters
  33. 33. Pattern Replace PLONE CONFERENCE 2011 <charFilter class="solr.PatternReplaceCharFilterFactory" pattern="[^a-zA-z0-9_-]" replacement="" replace="all"/>That WAS a narrow escape! said Alice, a good deal frightened That WAS a narrow escape said Alice a good deal frightened
  34. 34. Mapping PLONE CONFERENCE 2011 <charFilter class="solr.MappingCharFilterFactory" mapping="mapping.txt"/> # œ => oe "u0153" => "oe" # ß => ss "u00DF" => "ss"
  35. 35. HTML Strip PLONE CONFERENCE 2011 <charFilter class="solr.HTMLStripCharFilterFactory"/>
  36. 36. PLONE CONFERENCE 2011Tokenizers• Split raw text into tokens / terms• Typically the first step
  37. 37. Whitespace Tokenizer PLONE CONFERENCE 2011 <tokenizer class="solr.WhitespaceTokenizerFactory"/> That WAS a narrow escape! said Alice That WAS a narrow escape! said Alice
  38. 38. ICU Tokenizer PLONE CONFERENCE 2011 <tokenizer class="solr.ICUTokenizerFactory"/> That WAS a narrow escape! said Alice That WAS a narrow escape said Alice
  39. 39. Pattern Tokenizer PLONE CONFERENCE 2011<tokenizer class="solr.PatternTokenizerFactory" pattern=";s*" /> one; two; three one two three
  40. 40. Path Hierarchy PLONE CONFERENCE 2011 <tokenizer class="solr.PathHierarchyTokenizerFactory"/> /usr/local/etc/nginx /usr /usr/local /usr/local/etc /usr/local/etc/nginx
  41. 41. PLONE CONFERENCE 2011Token Filters• Process after tokenizing• Normalization of terms
  42. 42. Lower Case PLONE CONFERENCE 2011 <filter class="solr.LowerCaseFilterFactory"/> Foo bAr BAZ foo bar baz
  43. 43. ASCII Folding PLONE CONFERENCE 2011 <filter class="solr.ASCIIFoldingFilterFactory"/> idée bête grüßen idee bete grussen
  44. 44. ICU Folding PLONE CONFERENCE 2011 <filter class="solr.ICUFoldingFilterFactory"/> Idée BÊTE GrüßeN idee bete grussen
  45. 45. Pattern Replace PLONE CONFERENCE 2011<filter class="solr.PatternReplaceFilterFactory" pattern="[^a-zA-z0-9_-]" replacement="" replace="all"/> That That WAS WAS a a narrow narrow escape! escape said said Alice Alice
  46. 46. Word Delimiter PLONE CONFERENCE 2011<filter class="solr.WordDelimiterFilterFactory" splitOnCaseChange="1" splitOnNumerics="1" stemEnglishPossessive="1" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" preserveOriginal="1"/> StudlyCaps1234-5678 Caps StudlyCaps1234-5678 Studly 1234 5678
  47. 47. Edge N Gram PLONE CONFERENCE 2011 <filter class="solr.EdgeNGramFilterFactory" minGramSize="4" maxGramSize="100" side="front"/> Conqueror Conqueror Conquero Conquer Conque Conqu Conq
  48. 48. Stop Words PLONE CONFERENCE 2011 <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/> That narrow WAS escape a said narrow Alice escape good said deal Alice frightened a good deal frightened
  49. 49. Synonyms PLONE CONFERENCE 2011<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/> # synonyms.txt foozball foosball foosball # add multiple terms baby-foot foozball, foosball, baby-foot tele television # merge into one t.v. television tv, t.v., tele => television tv television
  50. 50. Language Stemming PLONE CONFERENCE 2011 <filter class="solr.EnglishPorterFilterFactory" protected="protwords.txt"/> dry dri drying dri dried dri
  51. 51. Language Stemming PLONE CONFERENCE 2011<filter class="solr.ElisionFilterFactory" articles="stopwordarticles.txt"/> quil il ne ne comprend comprend pas pas langlais anglais<filter class="solr.EnglishPorterFilterFactory" language="French"/> considere consider consideres consider considerent consider
  52. 52. Solr Debugging
  53. 53. Schema Browser PLONE CONFERENCE 2011
  54. 54. Analysis PLONE CONFERENCE 2011
  55. 55. Analysis PLONE CONFERENCE 2011
  56. 56. Analysis PLONE CONFERENCE 2011
  57. 57. Analysis PLONE CONFERENCE 2011
  58. 58. Analysis PLONE CONFERENCE 2011
  59. 59. Search Interface PLONE CONFERENCE 2011
  60. 60. Crafting a URL PLONE CONFERENCE 2011 http://localhost:8983/solr/select? qf=SearchableText^1.0&rows=10&fl=*,score&debugQuery=on& explainOther=True&indent=true&defType=dismax&q=test q=test qf=SearchableText^1.0 defType=dismax debugQuery=on explainOther=on indent=on
  61. 61. Verbose XML* PLONE CONFERENCE 2011 <lst name="responseHeader"> <int name="status">0</int> <int name="QTime">2</int> <lst name="params"> <str name="explainOther">True</str> <str name="fl">*,score</str> <str name="debugQuery">on</str> <str name="indent">true</str> <str name="q">test</str> <str name="qf">SearchableText^1.0</str> <str name="rows">10</str> <str name="defType">dismax</str> </lst> </lst>* like there is any other kind
  62. 62. Verbose XML* PLONE CONFERENCE 2011 <result name="response" numFound="2" start="0" maxScore="0.70710677"> <doc> <float name="score">0.70710677</float> <int name="docid">-643919099</int> </doc> <doc> <float name="score">0.3788861</float> <int name="docid">-643919097</int> </doc> </result>* like there is any other kind
  63. 63. Verbose XML*<lst name="debug"> PLONE CONFERENCE 2011 <str name="rawquerystring">test</str> <str name="querystring">test</str> <str name="parsedquery">+DisjunctionMaxQuery((SearchableText:test)) ()</str> <str name="parsedquery_toString">+(SearchableText:test) ()</str> <lst name="explain"> <str name="-643919099">0.70710677 = (MATCH) sum of: 0.70710677 = (MATCH) fieldWeight(SearchableText:test in 4), product of: 1.4142135 = tf(termFreq(SearchableText:test)=2) 1.0 = idf(docFreq=5, maxDocs=6) 0.5 = fieldNorm(field=SearchableText, doc=4) </str> <str name="-643919097">0.3788861 = (MATCH) sum of: 0.3788861 = (MATCH) fieldWeight(SearchableText:test in 0), product of: 1.7320508 = tf(termFreq(SearchableText:test)=3) 1.0 = idf(docFreq=5, maxDocs=6) 0.21875 = fieldNorm(field=SearchableText, doc=0) </str></lst>* like there is any other kind
  64. 64. Links PLONE CONFERENCE 2011• Solr (http://lucene.apache.org/solr)• Solr Wiki (http://wiki.apache.org/solr)• Books (http://www.packtpub.com/books/all?keys=solr)• SolrIndex (http://pypi.python.org/pypi/alm.solrindex/)• collective.solr (http://pypi.python.org/pypi/collective.solr)
  65. 65. Flickr Credits PLONE CONFERENCE 2011• http://www.flickr.com/photos/naturegeak/5642083189/ (who)• http://www.flickr.com/photos/eklektikos/2541408630/ (schema)• http://www.flickr.com/photos/sidelong/13954593/ (char filter)• http://www.flickr.com/photos/benimoto/2214240119/ (tokenizers)• http://www.flickr.com/photos/chaunceydavis/3264077445/ (filters)• http://www.flickr.com/photos/comedynose/3271760209/ (configuration)• http://www.flickr.com/photos/nicksart/4821509371/ (debugging) Thanks to
  66. 66. Questions? Check out .co m/d emos s ixfeetup

×