A presentation given at the Bruin Tech-a-Thon 2013 mini conference as an introduction to Apache Lucene and Apache Solr.

1. Make Your Data
Searchable
With Solr in 25 Minutes
Kai Chan
BruinTech Tech-a-Thon, November 19, 2013

2. The Goal
data
data
find this

3. The Goal
• objectives
o find something in the (text) data
o get the results fast
o get the most relevant results first
o avoid getting the not-so-relevant results first
• (one) solution: Solr

4. What Solr is
• used by high-profile websites like Twitter
… and interesting projects like NewsScape
• open-source, full-text search platform
• uses Lucene for indexing and searching
• standalone process/program (typically)
• REST-like API over HTTP
• different output formats (XML, JSON, CSV)

5. How to Talk To Solr
• have front-end/browser make HTTP
requests
• language-specific clients
o .Net
o Java
o PHP
o Python
o Ruby
• integration with other applications
o Moodle
o Drupal
Plone

6. How Solr works
Solr
query
(i.e. search criteria)
result
(i.e. things being looked for)

7. How Solr works
Solr
query
(i.e. search criteria)
result
(i.e. things being looked for)
Solr
index
index

8. How Solr works
Solr
Solr
query
(i.e. search criteria)
result
(i.e. things being looked for)
Solr
data to be searched
index
index

9. How Solr works
Solr
Solr
query
(i.e. search criteria)
query
(i.e. search criteria)
result’
(i.e. things being looked for)
Solr
index
index
index’
additions
updates
deletions
result
(i.e. things being looked for)

10. How Data Are Organized
collection
document document document
field
field
field
field
field
field
field
field
field

11. collection
document document document
subject
date
from
subject
date
from
date
from
reply-to
text text
reply-to
text
How Data Are Organized

12. collection
document document document
subject
date
from
title
SKU
price
last name
phone
text description
first name
address
How Data Are Organized

13. Solr Field Definition
• field
o name
o type
o options
• field type
o text: "string", "text_general"
o numeric: "int", "long", "float", "double"
• options
o indexed: content can be searched
o stored: content can be returned at search-time
o multivalued: multiple values per field & document

14. Solr Dynamic Field
• define field by naming convention
• "amount_i": int, index, stored
• "tag_ss": string, indexed, stored, multivalued
name type indexed stored multiValued
*_i int true true false
*_l long true true false
*_f float true true false
*_d double true true false
*_s string true true false
*_ss string true true true
*_t text_general true true false
*_txt text_general true true true

15. Getting Data into Solr
• submit (post) files to Solr
o XML
o JSON
o CSV
• have Solr pull data from database or file
o RDBMS
o XML data locally (file) or remotely (HTTP)
o extract data (XPath)
o manipulate data (regex replace, strip HTML tags)

16. Searching Data in Solr
• send request to http://host:port/solr/search
• parameters
o q - main query
o fl - fields to return
o sort - sort criteria
o wt - response writer (e.g. xml, json)
o indent - set to true for pretty-printing

17. Query Syntax
• basic format: field name “:”
word/phrasetext:negotiation
text:"debt ceiling"

18. Query Syntax
• several clauses: separated by
spacetext:negotiation
subject:debt
• make the word/phrase required: “+”
prefix+text:negotiation
+subject:debt
• make the word/phrase prohibited: “-”
prefixtext:negotiation -
subject:debt

19. Additional Things Solr Can Do
• other types of queries
o range
o fuzzy
o wildcard
o regex
o proximity
o spatial
o join
• sorting
• faceted search
• … and more

20. Conclusion
• more about
Solr:http://lucene.apache.org/solr/
• Solr reference
guide:http://www.apache.org/dyn/closer.cgi/l
ucene/solr/ref-guide/
• my e-mail:kai@ssc.ucla.edu
• questions?