Solr is a powerful open source search engine server which has become a popular choice for extending the search capabilities of Plone sites. The default configuration works well, but how do you answer the client's request to "Make my search just like Google's"? In this talk we will take a look at the various options that are available for configuring Solr's schema and configuration. We will discuss how to set up stop words, spell checking, n-grams and alternate query handlers. We will see what effect these settings will have on the search results and find out how to debug problems when they arise.

1. Make Plone Search Act
Like Google Using Solr
Clayton Parker | Senior Web Developer
PLONE CONFERENCE 2011

2. Who Am I

3. What will we learn? PLONE CONFERENCE 2011

4. What will we learn? PLONE CONFERENCE 2011
• Intro to Solr

5. What will we learn? PLONE CONFERENCE 2011
• Intro to Solr
• Brief overview of Plone integration points

6. What will we learn? PLONE CONFERENCE 2011
• Intro to Solr
• Brief overview of Plone integration points
• Solr configuration

7. What will we learn? PLONE CONFERENCE 2011
• Intro to Solr
• Brief overview of Plone integration points
• Solr configuration
• Solr schema setup

8. What will we learn? PLONE CONFERENCE 2011
• Intro to Solr
• Brief overview of Plone integration points
• Solr configuration
• Solr schema setup
• Debugging tips and tricks

9. PLONE CONFERENCE 2011
What is Solr ?

10. Version Madness PLONE CONFERENCE 2011
1.x
(up to 1.4)
1.5
(number abandoned)
3.x
(merge of Lucene and Solr)

11. Books PLONE CONFERENCE 2011

12. PLONE CONFERENCE 2011
Integration

13. PLONE CONFERENCE 2011
alm.solrindex

14. PLONE CONFERENCE 2011
collective.solr

15. Solr Configuration

16. Query Handlers PLONE CONFERENCE 2011
• Standard
• Disjunction Max (DisMax)
• Extended DisMax (experimental)

17. DisMax PLONE CONFERENCE 2011
• Multiple index searches
• Boosting
• Friendlier to end users

18. DisMax PLONE CONFERENCE 2011
Index Name
qf=SearchableText^1.0 substring^0.2
Weight

19. MinShouldMatch PLONE CONFERENCE 2011
mm=100%
All terms required
mm=50%
Half of the terms required
mm=-2
All but two terms required

20. MinShouldMatch PLONE CONFERENCE 2011
mm=2<-25% 9<-3
2 or less 3-9 terms all more than 9
terms are but 25% terms all but
required required three are
required

21. Spelling Component PLONE CONFERENCE 2011
<searchComponent name="spellcheck" class="solr.SpellCheckComponent">
<lst name="spellchecker">
<str name="name">default</str>
<str name="classname">solr.IndexBasedSpellChecker</str>
<str name="buildOnCommit">true</str>
<str name="spellcheckIndexDir">path/to/spellcheck</str>
<!-- The field that will contain the dynamic spelling data -->
<str name="field">spell</str>
<str name="accuracy">0.5</str>
</lst>
<!-- Control indexing and query of spelling data -->
<str name="queryAnalyzerFieldType">spell-text</str>
</searchComponent>

22. Spelling Schema PLONE CONFERENCE 2011
<fieldType name="spell-text" class="solr.TextField">
<analyzer>
<tokenizer class="solr.ICUTokenizerFactory"/>
<filter class="solr.ICUFoldingFilterFactory"/>
<filter class="solr.StopFilterFactory"
ignoreCase="true" words="stopwords.txt"/>
</analyzer>
</fieldType>

23. Solr Schema

24. Index vs Query PLONE CONFERENCE 2011
http://www.cominvent.com/2011/04/04/solr-architecture-diagram/

25. PLONE CONFERENCE 2011

26. PLONE CONFERENCE 2011
Character Filters

27. PLONE CONFERENCE 2011
Character Filters
Tokenizer

28. PLONE CONFERENCE 2011
Character Filters
Tokenizer
Filters

29. PLONE CONFERENCE 2011
Character Filters
Tokenizer
Filters

30. Complete Field PLONE CONFERENCE 2011
<fieldType name="text" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<charFilter class="solr.MappingCharFilterFactory" mapping="mapping.txt"/>
<tokenizer class="solr.ICUTokenizerFactory"/>
<filter class="solr.ICUFoldingFilterFactory"/>
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
ignoreCase="true" expand="true"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/>
<filter class="solr.EnglishPorterFilterFactory" protected="protwords.txt"/>
<filter class="solr.WordDelimiterFilterFactory" splitOnCaseChange="1"
splitOnNumerics="1" stemEnglishPossessive="1" generateWordParts="1"
generateNumberParts="1" catenateWords="0" catenateNumbers="0"
catenateAll="0" preserveOriginal="1"/>
</analyzer>
<analyzer type="query">
<charFilter class="solr.MappingCharFilterFactory" mapping="mapping.txt"/>
<tokenizer class="solr.ICUTokenizerFactory"/>
<filter class="solr.ICUFoldingFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/>
<filter class="solr.EnglishPorterFilterFactory" protected="protwords.txt"/>
<filter class="solr.WordDelimiterFilterFactory" splitOnCaseChange="1"
splitOnNumerics="1" stemEnglishPossessive="1" generateWordParts="1"
generateNumberParts="1" catenateWords="0" catenateNumbers="0"
catenateAll="0" preserveOriginal="1"/>
<filter class="solr.PositionFilterFactory"/>
</analyzer>
</fieldType>

31. Copy Field PLONE CONFERENCE 2011
<copyField source="SearchableText" dest="spell"/>
<copyField source="SearchableText" dest="substring"/>

32. PLONE CONFERENCE 2011
Character Filters
• Process text before tokenizing
• Remove irrelevant characters

33. Pattern Replace PLONE CONFERENCE 2011
<charFilter class="solr.PatternReplaceCharFilterFactory"
pattern="[^a-zA-z0-9_-]" replacement="" replace="all"/>
'That WAS a narrow escape!' said Alice, a good deal frightened
That WAS a narrow escape said Alice a good deal frightened

34. Mapping PLONE CONFERENCE 2011
<charFilter class="solr.MappingCharFilterFactory"
mapping="mapping.txt"/>
# œ => oe
"u0153" => "oe"
# ß => ss
"u00DF" => "ss"

35. HTML Strip PLONE CONFERENCE 2011
<charFilter class="solr.HTMLStripCharFilterFactory"/>

36. PLONE CONFERENCE 2011
Tokenizers
• Split raw text into tokens / terms
• Typically the first step

37. Whitespace Tokenizer PLONE CONFERENCE 2011
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
'That WAS a narrow escape!' said Alice
'That
WAS
a
narrow
escape!'
said
Alice

38. ICU Tokenizer PLONE CONFERENCE 2011
<tokenizer class="solr.ICUTokenizerFactory"/>
'That WAS a narrow escape!' said Alice
That
WAS
a
narrow
escape
said
Alice

39. Pattern Tokenizer PLONE CONFERENCE 2011
<tokenizer class="solr.PatternTokenizerFactory" pattern=";s*" />
one; two; three
one
two
three

40. Path Hierarchy PLONE CONFERENCE 2011
<tokenizer class="solr.PathHierarchyTokenizerFactory"/>
/usr/local/etc/nginx
/usr
/usr/local
/usr/local/etc
/usr/local/etc/nginx

41. PLONE CONFERENCE 2011
Token Filters
• Process after tokenizing
• Normalization of terms

42. Lower Case PLONE CONFERENCE 2011
<filter class="solr.LowerCaseFilterFactory"/>
Foo
bAr
BAZ
foo
bar
baz

43. ASCII Folding PLONE CONFERENCE 2011
<filter class="solr.ASCIIFoldingFilterFactory"/>
idée
bête
grüßen
idee
bete
grussen

44. ICU Folding PLONE CONFERENCE 2011
<filter class="solr.ICUFoldingFilterFactory"/>
Idée
BÊTE
GrüßeN
idee
bete
grussen

45. Pattern Replace PLONE CONFERENCE 2011
<filter class="solr.PatternReplaceFilterFactory"
pattern="[^a-zA-z0-9_-]" replacement="" replace="all"/>
'That That
WAS WAS
a a
narrow narrow
escape!' escape
said said
Alice Alice

46. Word Delimiter PLONE CONFERENCE 2011
<filter class="solr.WordDelimiterFilterFactory"
splitOnCaseChange="1" splitOnNumerics="1" stemEnglishPossessive="1"
generateWordParts="1" generateNumberParts="1"
catenateWords="0" catenateNumbers="0" catenateAll="0"
preserveOriginal="1"/>
StudlyCaps1234-5678
Caps
StudlyCaps1234-5678 Studly
1234
5678

47. Edge N Gram PLONE CONFERENCE 2011
<filter class="solr.EdgeNGramFilterFactory"
minGramSize="4" maxGramSize="100" side="front"/>
Conqueror
Conqueror
Conquero
Conquer
Conque
Conqu
Conq

48. Stop Words PLONE CONFERENCE 2011
<filter class="solr.StopFilterFactory"
ignoreCase="true" words="stopwords.txt"/>
That narrow
WAS escape
a said
narrow Alice
escape good
said deal
Alice frightened
a
good
deal
frightened

49. Synonyms PLONE CONFERENCE 2011
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
ignoreCase="true" expand="true"/>
# synonyms.txt foozball
foosball foosball
# add multiple terms baby-foot
foozball, foosball, baby-foot
tele television
# merge into one t.v. television
tv, t.v., tele => television tv television

50. Language Stemming PLONE CONFERENCE 2011
<filter class="solr.EnglishPorterFilterFactory"
protected="protwords.txt"/>
dry dri
drying dri
dried dri

51. Language Stemming PLONE CONFERENCE 2011
<filter class="solr.ElisionFilterFactory" articles="stopwordarticles.txt"/>
qu'il il
ne ne
comprend comprend
pas pas
l'anglais anglais
<filter class="solr.EnglishPorterFilterFactory" language="French"/>
considere consider
consideres consider
considerent consider

52. Solr Debugging

53. Schema Browser PLONE CONFERENCE 2011

54. Analysis PLONE CONFERENCE 2011

55. Analysis PLONE CONFERENCE 2011

56. Analysis PLONE CONFERENCE 2011

57. Analysis PLONE CONFERENCE 2011

58. Analysis PLONE CONFERENCE 2011

59. Search Interface PLONE CONFERENCE 2011

60. Crafting a URL PLONE CONFERENCE 2011
http://localhost:8983/solr/select?
qf=SearchableText^1.0&rows=10&fl=*,score&debugQuery=on&
explainOther=True&indent=true&defType=dismax&q=test
q=test
qf=SearchableText^1.0
defType=dismax
debugQuery=on
explainOther=on
indent=on

61. Verbose XML* PLONE CONFERENCE 2011
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">2</int>
<lst name="params">
<str name="explainOther">True</str>
<str name="fl">*,score</str>
<str name="debugQuery">on</str>
<str name="indent">true</str>
<str name="q">test</str>
<str name="qf">SearchableText^1.0</str>
<str name="rows">10</str>
<str name="defType">dismax</str>
</lst>
</lst>
* like there is any other kind

62. Verbose XML* PLONE CONFERENCE 2011
<result name="response" numFound="2" start="0" maxScore="0.70710677">
<doc>
<float name="score">0.70710677</float>
<int name="docid">-643919099</int>
</doc>
<doc>
<float name="score">0.3788861</float>
<int name="docid">-643919097</int>
</doc>
</result>
* like there is any other kind

63. Verbose XML*
<lst name="debug">
PLONE CONFERENCE 2011
<str name="rawquerystring">test</str>
<str name="querystring">test</str>
<str name="parsedquery">+DisjunctionMaxQuery((SearchableText:test)) ()</str>
<str name="parsedquery_toString">+(SearchableText:test) ()</str>
<lst name="explain">
<str name="-643919099">
0.70710677 = (MATCH) sum of:
0.70710677 = (MATCH) fieldWeight(SearchableText:test in 4), product of:
1.4142135 = tf(termFreq(SearchableText:test)=2)
1.0 = idf(docFreq=5, maxDocs=6)
0.5 = fieldNorm(field=SearchableText, doc=4)
</str>
<str name="-643919097">
0.3788861 = (MATCH) sum of:
0.3788861 = (MATCH) fieldWeight(SearchableText:test in 0), product of:
1.7320508 = tf(termFreq(SearchableText:test)=3)
1.0 = idf(docFreq=5, maxDocs=6)
0.21875 = fieldNorm(field=SearchableText, doc=0)
</str>
</lst>
* like there is any other kind

64. Links PLONE CONFERENCE 2011
• Solr (http://lucene.apache.org/solr)
• Solr Wiki (http://wiki.apache.org/solr)
• Books (http://www.packtpub.com/books/all?keys=solr)
• SolrIndex (http://pypi.python.org/pypi/alm.solrindex/)
• collective.solr (http://pypi.python.org/pypi/collective.solr)

65. Flickr Credits PLONE CONFERENCE 2011
• http://www.flickr.com/photos/naturegeak/5642083189/ (who)
• http://www.flickr.com/photos/eklektikos/2541408630/ (schema)
• http://www.flickr.com/photos/sidelong/13954593/ (char filter)
• http://www.flickr.com/photos/benimoto/2214240119/ (tokenizers)
• http://www.flickr.com/photos/chaunceydavis/3264077445/ (filters)
• http://www.flickr.com/photos/comedynose/3271760209/ (configuration)
• http://www.flickr.com/photos/nicksart/4821509371/ (debugging)
Thanks to

66. Questions?
Check out
.co m/d emos
s ixfeetup