An Introduction to Solr

Searching with Solr
Tom Hill
[email_address]
eBig Java SIG, June 18th, 2008

Tonight's Talk
Tonight's Talk should run about 1 1/2 hours
About Solr
Background & overview
Installing & Bringing Up Solr
Rest Interface & Java Client
Configuring Solr

Why Implement Search?
Does your site need search?
Do you need to implement it, or is Google enough?
Just text or Structured Data?
Do you need to control ranking?

What is Solr?
Web application for text search
A wrapper around Apache Lucene
Lucene is a library (.jar file)
Solr is a web app (.war file)
Written at CNet, now at Apache

What is Lucene?
Text search library in Java
Fast, feature rich.
Written by Doug Cutting
Active Apache development community
Versions also in C++, C#, Ruby, Python, Delphi, Lisp, etc...

Why Solr?
Reliable
Fast
Supported
Open Source
Tunable Scoring

Solr Versions
Current Version is 1.2
A year old
1.3 is coming "sometime"
Large number of features in HEAD
Use the latest from subversion for new projects

Alternatives to Solr
Just Use Google
Use Lucene
Use Your Database
Commercial Libraries
Write your own

What Solr is Not
A replacement for a relational database
An embedded database*
Fully cross platform :-(
Replication depends on unix FS
Admin scripts are bash(minor)

Solr Sites
CNet (Reviews & Products)
Internet Archive (Collections)
Netflix (Movies)
Zvents (Events)
StripSearch.ws (Comics)
And many more

Features Here's a quick look at some of the features of Solr, as implemented on Zvents.com

Faceted Navigation
Groups the results by category
Can do multiple facets at once
Returns matching counts

Additional Constraints

Synonyms, etc.

Solr Overview

Simple Webapp Web Servers[1..n] Database Master Database Slaves[0..n] Solr Master Solr Slaves[0..n]

Scaling Solr
Master/Slave architecture
Writes to master/reads to slaves
Replication: Periodic transfers, not continuous
Rsync

Updates
Updates flush caches, bad for performance
Master therefor much slower than slaves
So send all queries to slaves
Depends on your update rates

Solr's Data Model
Solr maintains a collection of documents
A document is a collection of fields & values
A field can occur multiple times in a document
Documents are immutable.
They can be deleted, and a new version added, however.

Querying
Http request
http://localhost:8080/comix/select/?q=java

Solr Query Syntax
Lucene Query Syntax + a bit
paris
city:paris
title:"The Right Way" AND text:go
id:[* TO *]

Solr Query Syntax II
-inStock:false
te?t
theat*
te*t
test~

Using Solr
Getting data into Solr
Getting data out of Solr

Getting Data Into Solr
POST it.
<add> <doc> <field name="employeeId">05991</field> <field name="office">Bridgewater</field> <field name="skills">Perl</field> <field name="skills">Java</field> </doc> [<doc> ... </doc>[<doc> ... </doc>]] </add>

POST it.
<add> < doc > <field name="employeeId">05991</field> <field name="office">Bridgewater</field> <field name="skills">Perl</field> <field name="skills">Java</field> </ doc > [<doc> ... </doc>[<doc> ... </doc>]] </add>

POST it.
<add> <doc> <field name=" employeeId "> 05991 </field> <field name="office">Bridgewater</field> <field name="skills">Perl</field> <field name="skills">Java</field> </doc> [<doc> ... </doc>[<doc> ... </doc>]] </add>

Committing
Nothing shows up in the index until you commit
You can just POST <commit/> to http:// host : port /solr/update

Getting Data Out http://localhost:8080/comix/select/?q=data&indent=on
<response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">0</int>
<lst name="params">
<str name="indent">on</str>
<str name="q">data</str>
</lst>
</lst>
<result name="response" numFound="2" start="0">
<doc>
<str name="id">strip.3136</str>
<str name="release_date">1992-05-07</str>
<date name="timestamp">2008-02-28T10:06:01.682Z</date>
<str name="type">strip</str>
</doc>
</result>
</response>

Getting Data Out http://localhost:8080/comix/select/?q=data&indent=on { "responseHeader":{ "status":0, "QTime":1, "params":{ "wt":"json", "rows":["1", "1"], "start":"0", "indent":"on", "q":"data", "version":"2.2"}}, "response":{"numFound":2,"start":0,"docs":[ { "feature_id":"3", "release_date":"1992-05-07", "id":"strip.3136", "timestamp":"2008-02-28T10:06:01.682Z"}] }} JSON format

Debug Query Option
Add &debugQuery=on to request params
Returns parsed form of query
<str name="rawquerystring">c.i.a</str><str name="querystring">c.i.a</str><str name="parsedquery">PhraseQuery(text:"c i a")</str><str name="parsedquery_toString">text:"c i a"</str>

Debug Query Option II
Add &debugQuery=on to request params
Returns scoring information
<str name="id=strip.2781,internal_docid=29854"> 2.6219895 = (MATCH) fieldWeight(text:calvin in 29854), product of: 1.0 = tf(termFreq(text:calvin)=1) 2.6219895 = idf(docFreq=6222) 1.0 = fieldNorm(field=text, doc=29854) </str> <str name="id=strip.4078,internal_docid=31151"> 2.6219895 = (MATCH) fieldWeight(text:calvin in 31151), product of: 1.0 = tf(termFreq(text:calvin)=1) 2.6219895 = idf(docFreq=6222) 1.0 = fieldNorm(field=text, doc=31151) </str>

Deleting Data
POST
<delete><id>35</id></delete> <delete><query>city:paris</query></delete>

Command Line Control
curl http://localhost:8983/solr/update -H "Content-type: text/xml" --data-binary '<commit/>'
<?xml version="1.0" encoding="UTF-8"?><response><lst name="responseHeader">
</lst></response>
</lst></response>
</lst></response>

Solr in 3 minutes!
Download Solr from Apache
Untar
"ant example"
Start the example app
Load data into Solr
Query

Solr in Ten Minutes <Context docBase="/var/solr/apache-solr-1.2.0.war" debug="0" crossContext="true" > <Environment name="solr/home" type="java.lang.String" value="/var/solr" override="true" /></Context>
Copy Solr's example/solr dir to /var/solr
Edit schema.xml and solrconfig.xml
Load data into Solr
In $CATALINA_HOME/conf/Catalina/localhost/foo.xml

Directory Layout
${solr.home}/conf
schema.xml
solrconfig.xml
${solr.home}/data
${solr.home}/logs
${solr.home}/bin

Java Solr Client
Called SolrJ
Not in Solr 1.2.
I grabbed from the HEAD from svn
Works with Solr 1.2
Add/Delete/Query/Commit/Optimize

Adding Docs w/SolrJ Given Map<String, String> fields; CommonsHttpSolrServer server = new CommonsHttpSolrServer( url ); SolrInputDocument doc= new SolrInputDocument(); for (Map.Entry<String, String> e : fields.entrySet()){ doc.addField(e.getKey(), e.getValue()); } UpdateResponse res = server .add( doc);

Deleting Docs w/SolrJ CommonsHttpSolrServer server = new CommonsHttpSolrServer( url ); UpdateResponse res; res = server .deleteById("100"); res = server .deleteByQuery("city:paris");

Simple Query CommonsHttpSolrServer server = new CommonsHttpSolrServer( url ); SolrQuery query = new SolrQuery(); query.setQuery("dance"); QueryResponse rsp = server .query(query);

More Interesting Query CommonsHttpSolrServer server = new CommonsHttpSolrServer( url ); SolrQuery query = new SolrQuery(); query.setQuery("dance"); query.setFacet( true ); query.addFacetField("city"); query.setFacetMinCount(1); query.addSortField( "price", SolrQuery.ORDER.asc ); QueryResponse rsp = server .query(query);

Query Responses
QueryResponse qr = server .query(query);
SolrDocumentList docs = qr.getResults();
List<FacetField> lf = qr.getFacetFields();
for (FacetField ff: lf) {
String fieldName = ff.getName();
List<FacetField.Count> lc = ff.getValues();
for (FacetField.Count c: lc) {
String countName = c.getName();
long count = c.getCount();
}
}

Other Commands
Commit
server.commit()
Optimize
server.optimize()
Not too complicated!

Request Handlers
Request handler define how the query is processed.
Two main types
StandardRequestHandler
DisMaxRequestHandler
You can implement your own
Changing in Solr 1.3

"Standard" Request Handler
Accepts Solr Query Syntax
I tend to use it for my queries, not user queries.

DisMaxRequestHandler
Recommended for user queries
Allows simple users keywords to be applied to multiple fields, with weighting.
Boost Functions
Boost Queries

Boost Functions
Allow you to influence scoring at run time
Computationally Expensive!
Really useful for tuning scoring
linear(x,2,4) returns 2*x+4
x is a field

The Solr Schema
schema.xml
Defines types used in this webapp
Defines the fields and their types
Defines "copyFields"
READ THE EXAMPLE SCHEMA.XML

Types
Types define processing for a field
How the words are split (Whitespace? Punctuation? CIA != C.I.A.)
Stemming
Case Folding, etc
Predefined date, int, float, etc
c

Analysis: Index and Query Time
Types have two modes
Index Time
Query Time

Simple Text Field <fieldType name="text" class="solr.TextField" positionIncrementGap= "100"> <analyzer type="index"> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/></analyzer><analyzer type="query"><tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/> <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/></analyzer></fieldType> <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/></analyzer></fieldType> <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/></analyzer></fieldType>

Analysis & Facets
Make sure to use an untokenized field for faceting.
"San Jose" != "San" "Jose"

Fields
Elements of a document
Both predefined & dynamic
Fields may occur multiple times
Maybe indexed and/or stored

Example Fields <field name="id" type="string" indexed="true" stored="true" required="true" /><field name="name" type="text" indexed="true" stored="true"/><field name="alphaNameSort" type="alphaOnlySort" indexed="true" stored="false"/>

Copy Fields
Two main uses
To analyze a field in two different ways
To concatenate fields

The Solr Config File
solrconfig.xml
Defines request handlers, defaults, caches,
Read the example solrconfig.xml

Configuring DisMax
Parameter defaults set in solrconfig.xml
Can be overridden in each request
Except for params labeled invariant

DisMax Config Example <requestHandler name="dismax" class="solr.DisMaxRequestHandler" > <lst name="defaults"> <str name="echoParams">explicit</str> <float name="tie">0.01</float> <str name="qf"> text^0.5 features^1.0 name^1.2 sku^1.5 id^10.0 manu^1.1 cat^1.4 </str> <str name="pf"> text^0.2 features^1.1 name^1.5 manu^1.4 manu_exact^1.9 </str> ... </requestHandler>

DisMax Config Example <requestHandler name="dismax" class="solr.DisMaxRequestHandler" > ... <str name="bf"> ord(poplarity)^0.5 recip(rord(price),1,1000,1000)^0.3 </str> <str name="fl"> id,name,price,score </str>... </requestHandler>

DisMax Config Example <requestHandler name="dismax" class="solr.DisMaxRequestHandler" > ... <str name="mm"> 2<-1 5<-2 6<90% </str> <int name="ps">100</int> <str name="q.alt">*:*</str> </lst> </requestHandler>

Wrap Up

Resources
Solr http://lucene.apache.org/solr/
wiki, mailing list, jira (bugs/features)
Lucene http://lucene.apache.org /

Lucene In Action

Building Search Applications with Lucene, lingpipe and Gate Manu Konchady Manu Konchady Manu Konchady

Other Presentations
Yonik Seely's Solr & Lucene
http://people.apache.org/~yonik/presentations/
Slideshare.net
Search for solr, or search for lucene

Thanks! Thanks for coming. Feel free to email me if you have questions about Solr Tom Hill [email_address]

Extra Slides Things I didn't have time for in the presentation. Some of them unfinished.

Search Engines are not the Same as Users
Search engines have different usage patterns than users

Response Writers
http://localhost:8983/solr/select/?q=text_t%3Atiger&version=2.2&start=0&rows=10&indent=on& wt=ruby
http://localhost:8983/solr/select/?q=text_t%3Atiger&version=2.2&start=0&rows=10&indent=on& wt=xml

Explain
Just why did the documents come up in that order?

Data Matters
Gigo
The better the data is, the better the search will be.

Watch Your Caches
Just like any other app, check your statistics
What's the hit rate for your caches?

Setting Up Replication
Run rsyncd on the master
Run snapshot on the master at intervals
Run snappuller on the slaves at (different) intervals.
Scripts don't print errors!
Check the logs
Use bash -xv

Autowarming
Runs after an update to the index
Updates flush caches
Runs some queries to populate caches again
Can be a problem, with frequent updates
Don't autowarm master, if updating lots

Tour Of Solr's Web UI

Programming Collective Intelligence A Really Fun Book

Geographic Searching
Local Lucene & Local Solr
http://locallucene.wiki.sourceforge.net
There's also geolucene, but it's not being actively developed, as far as I can tell.
http://www.gossamer-threads.com/lists/l ucene/java-dev/53378

http://localhost:8983/solr/admin/stats.jsp#update Are there commits pending?

http://localhost:8983/comix/admin/analysis.jsp?name=text&val=wi-fi Analysis Explanation

http://logmania.masakiplus.net	1093
http://hadoopbigdata.wordpress.com	79
http://www.slideshare.net	47
http://tejaspillai.wordpress.com	42
http://www.scoop.it	42
http://webcache.googleusercontent.com	7
http://feeds.feedburner.com	5

An Introduction to Solr

by tomhill

Accessibility

Categories

Upload Details

Usage Rights

7 Embeds 1,315

Statistics

An Introduction to Solr Presentation Transcript