Solr

Solr Windows Installation Download and install Tomcat for Windows using the MSI installer. Install it with the tcnative.dll file. Say you installed it in c: omcat Check if Tomcat is installed correctly by going to http://localhost:8080/ Change the c: omcatconfserver.xml file to add the URIEncoding Connector element as shown below. <Connector port=&quot;8080&quot; maxHttpHeaderSize=&quot;8192&quot; URIEncoding=&quot;UTF-8&quot; maxThreads=&quot;150&quot; minSpareThreads=&quot;25&quot; maxSpareThreads=&quot;75&quot; enableLookups=&quot;false&quot; redirectPort=&quot;8443&quot; acceptCount=&quot;100&quot; connectionTimeout=&quot;20000&quot; disableUploadTimeout=&quot;true&quot; /> Download and unzip the Solr distribution zip file into (say) c: empsolrZip

Download and install Tomcat for Windows using the MSI installer. Install it with the tcnative.dll file. Say you installed it in c: omcat

Check if Tomcat is installed correctly by going to http://localhost:8080/

Change the c: omcatconfserver.xml file to add the URIEncoding Connector element as shown below.

<Connector port=&quot;8080&quot; maxHttpHeaderSize=&quot;8192&quot; URIEncoding=&quot;UTF-8&quot;

maxThreads=&quot;150&quot; minSpareThreads=&quot;25&quot; maxSpareThreads=&quot;75&quot;

enableLookups=&quot;false&quot; redirectPort=&quot;8443&quot; acceptCount=&quot;100&quot;

connectionTimeout=&quot;20000&quot; disableUploadTimeout=&quot;true&quot; />

Download and unzip the Solr distribution zip file into (say) c: empsolrZip

Solr Windows Installation Make a directory called solr where you intend the application server to function, say c:websolr Copy the contents of the examplesolr directory c: empsolrZipexamplesolr to c:websolr Stop the Tomcat service Copy the *solr*.war file from c: empsolrZipdist to the Tomcat webapps directory c: omcatwebapps Rename the *solr*.war file solr.war Use the system tray icon to configure Tomcat to start with the following Java option: -Dsolr.solr.home=c:websolr Alternative to the previous step goto C: omcatconfCatalinalocalhost and create a file named ”solr.xml” having this line of code (see below in the notes) . But to run the server this way you will not keep the solr.war in webapps folder of tomcat but rather in some other folder like in this case I have kept it in ${catalina.home}/newsolr/solr.war. Start the Tomcat service Go to the solr admin page to verify that the installation is working. It will be at http://localhost:8080/solr/admin

Make a directory called solr where you intend the application server to function, say c:websolr

Copy the contents of the examplesolr directory c: empsolrZipexamplesolr to c:websolr

Stop the Tomcat service

Copy the *solr*.war file from c: empsolrZipdist to the Tomcat webapps directory c: omcatwebapps

Rename the *solr*.war file solr.war

Use the system tray icon to configure Tomcat to start with the following Java option: -Dsolr.solr.home=c:websolr

Alternative to the previous step goto C: omcatconfCatalinalocalhost and create a file named ”solr.xml” having this line of code (see below in the notes) . But to run the server this way you will not keep the solr.war in webapps folder of tomcat but rather in some other folder like in this case I have kept it in ${catalina.home}/newsolr/solr.war.

Start the Tomcat service

Go to the solr admin page to verify that the installation is working. It will be at http://localhost:8080/solr/admin

In Solr and Lucene, an index is built of one or more Documents. A Document consists of one or more Fields. A Field consists of a name, content, and metadata telling Solr how to handle the content. For instance, Fields can contain strings, numbers, booleans, or dates, as well as any types you wish to add. A Field can be described using a number of options that tell Solr how to treat the content during indexing and searching.

In Solr and Lucene, an index is built of one or more Documents. A Document consists of one or more Fields. A Field consists of a name, content, and metadata telling Solr how to handle the content. For instance, Fields can contain strings, numbers, booleans, or dates, as well as any types you wish to add. A Field can be described using a number of options that tell Solr how to treat the content during indexing and searching.

Field attributes The contents of a stored Field are saved in the index. This is useful for retrieving and highlighting the contents for display but is not necessary for the actual search. For example, many applications store pointers to the location of contents rather than the actual contents of a file. stored Indexed Fields are searchable and sortable. You also can run Solr's analysis process on indexed Fields, which can alter the content to improve or change results. The following section provides more information about Solr's analysis process. indexed Description Attribute name

Example &quot;Solr Home&quot; Directory ============================= This directory is provided as an example of what a &quot;Solr Home&quot; directory should look like. It's not strictly necessary that you copy all of the files in this directory when setting up a new instance of Solr, but it is recommended. Basic Directory Structure ------------------------- The Solr Home directory typically contains the following subdirectories... conf/ This directory is mandatory and must contain your solrconfig.xml and schema.xml. Any other optional configuration files would also be kept here. data/ This directory is the default location where Solr will keep your index, and is used by the replication scripts for dealing with snapshots. You can override this location in the solrconfig.xml and scripts.conf files. Solr will create this directory if it does not already exist. lib/ This directory is optional. If it exists, Solr will load any Jars found in this directory and use them to resolve any &quot;plugins&quot; specified in your solrconfig.xml or schema.xml (ie: Analyzers, Request Handlers, etc...) bin/ This directory is optional. It is the default location used for keeping the replication scripts.

Basic Directory Structure

-------------------------

The Solr Home directory typically contains the following subdirectories...

conf/

This directory is mandatory and must contain your solrconfig.xml

and schema.xml. Any other optional configuration files would also

be kept here.

data/

This directory is the default location where Solr will keep your

index, and is used by the replication scripts for dealing with

snapshots. You can override this location in the solrconfig.xml

and scripts.conf files. Solr will create this directory if it

does not already exist.

lib/

This directory is optional. If it exists, Solr will load any Jars

found in this directory and use them to resolve any &quot;plugins&quot;

specified in your solrconfig.xml or schema.xml (ie: Analyzers,

Request Handlers, etc...)

bin/

This directory is optional. It is the default location used for

keeping the replication scripts.

What Is Solr SOLR is a REST layer for Lucene Began life at CNET to provide a robust search system Joined Apache Incubator in January 2006 Graduated to Lucene sub-project status in January 2007 A full text search server based on Lucene XML/HTTP Interfaces Loose Schema to define types and fields Web Administration Interface Extensive Caching Index Replication Extensible Open Architecture Written in Java5, deployable as a WAR

SOLR is a REST layer for Lucene

Began life at CNET to provide a robust search system

Joined Apache Incubator in January 2006

Graduated to Lucene sub-project status in January 2007

A full text search server based on Lucene

XML/HTTP Interfaces

Loose Schema to define types and fields

Web Administration Interface

Extensive Caching

Index Replication

Extensible Open Architecture

Written in Java5, deployable as a WAR

Why use SOLR? Easy to set up and get started Powerful full text searching Cross platform - Java and REST Under active development Fast Adds extra functionality on top of Lucene: replication CSV importing JSON results results highlighting synonym support

Easy to set up and get started

Powerful full text searching

Cross platform - Java and REST

Under active development

Fast

Adds extra functionality on top of Lucene:

replication

CSV importing

JSON results

results highlighting

synonym support

Adding Documents HTTP POST to /update <add><doc boost=“2”> <field name=“article”>05991</field> <field name=“title”>Apache Solr</field> <field name=“subject”>An intro...</field> <field name=“category”>search</field> <field name=“category”>lucene</field> <field name=“body”>Solr is a full...</field> </doc></add>

HTTP POST to /update

<add><doc boost=“2”>

<field name=“article”>05991</field>

<field name=“title”>Apache Solr</field>

<field name=“subject”>An intro...</field>

<field name=“category”>search</field>

<field name=“category”>lucene</field>

<field name=“body”>Solr is a full...</field>

</doc></add>

Deleting Documents Delete by Id <delete><id>05591</id></delete> Delete by Query (multiple documents) <delete> <query>manufacturer:microsoft</query> </delete>

Delete by Id

<delete><id>05591</id></delete>

Delete by Query (multiple documents)

<delete>

<query>manufacturer:microsoft</query>

</delete>

Commit <commit/> makes changes visible closes IndexWriter removes duplicates opens new IndexSearcher newSearcher/firstSearcher events cache warming “ register” the new IndexSearcher <optimize/> same as commit, merges all index segments.

<commit/> makes changes visible

closes IndexWriter

removes duplicates

opens new IndexSearcher

newSearcher/firstSearcher events

cache warming

“ register” the new IndexSearcher

<optimize/> same as commit, merges all

index segments.

Lucene syntax Required search term – use a “+” +ipod +belkin Field-specific searching – use fieldName name:ipod manu:belkin Wildcard searching – use * or ? ip?d belk* *deo (currently requires modifying solr source) Range searching timestamp:[2006-07-16T12:30:00Z to *] Time needs to be full ISO Proximity searching – use a “~” &quot;video ipod&quot;~3 – up to 3 words apart Fuzzy searches – use a “~” ipod~ will find ipod and ipods belkin~0.7 will find words close spellings

Required search term – use a “+”

+ipod

+belkin

Field-specific searching – use fieldName

name:ipod

manu:belkin

Wildcard searching – use * or ?

ip?d

belk*

*deo (currently requires modifying solr source)

Range searching

timestamp:[2006-07-16T12:30:00Z to *]

Time needs to be full ISO

Proximity searching – use a “~”

&quot;video ipod&quot;~3 – up to 3 words apart

Fuzzy searches – use a “~”

ipod~ will find ipod and ipods

belkin~0.7 will find words close spellings

Default Query Syntax Lucene Query Syntax [; sort specification] 1. mission impossible; releaseDate desc 2. +mission +impossible –actor:cruise 3. “mission impossible” –actor:cruise 4. title:spiderman^10 description:spiderman 5. description:“spiderman movie”~10 6. +HDTV +weight:[0 TO 100] 7. Wildcard queries: te?t, te*t, test*

Lucene Query Syntax [; sort specification]

1. mission impossible; releaseDate desc

2. +mission +impossible –actor:cruise

3. “mission impossible” –actor:cruise

4. title:spiderman^10 description:spiderman

5. description:“spiderman movie”~10

6. +HDTV +weight:[0 TO 100]

7. Wildcard queries: te?t, te*t, test*

Full control panel interface Start row/max rows – pagination Output type standard (xml), python, json, ruby, xslt Enable highlighting fields to highlight works on wildcard matches

Start row/max rows – pagination

Output type

standard (xml), python, json, ruby, xslt

Enable highlighting

fields to highlight

works on wildcard matches

Default Parameters Query Arguments for HTTP GET/POST to /select

Query Arguments for HTTP GET/POST to /select

Search Results http://localhost:8983/solr/select?q=video&start=0&rows=2&fl=name,price <response><responseHeader><status>0</status> <QTime>1</QTime></responseHeader> <result numFound=&quot;16173&quot; start=&quot;0&quot;> <doc> <str name=&quot;name&quot;>Apple 60 GB iPod with Video</str> <float name=&quot;price&quot;>399.0</float> </doc> <doc> <str name=&quot;name&quot;>ASUS Extreme N7800GTX/2DHTV</str> <float name=&quot;price&quot;>479.95</float> </doc> </result> </response>

http://localhost:8983/solr/select?q=video&start=0&rows=2&fl=name,price

<response><responseHeader><status>0</status>

<QTime>1</QTime></responseHeader>

<result numFound=&quot;16173&quot; start=&quot;0&quot;>

<doc>

<str name=&quot;name&quot;>Apple 60 GB iPod with Video</str>

<float name=&quot;price&quot;>399.0</float>

</doc>

<doc>

<str name=&quot;name&quot;>ASUS Extreme N7800GTX/2DHTV</str>

<float name=&quot;price&quot;>479.95</float>

</doc>

</result>

</response>

Caching IndexSearcher’s view of an index is fixed • Aggressive caching possible • Consistency for multi-query requests filterCache – unordered set of document ids matching a query resultCache – ordered subset of document ids matching a query documentCache – the stored fields of documents userCaches – application specific, custom query handlers

IndexSearcher’s view of an index is fixed

• Aggressive caching possible

• Consistency for multi-query requests

filterCache – unordered set of document ids

matching a query

resultCache – ordered subset of document ids

matching a query

documentCache – the stored fields of documents

userCaches – application specific, custom query

handlers

Warming for Speed Lucene IndexReader warming field norms, FieldCache, tii – the term index Static Cache warming Configurable static requests to warm new Searchers Smart Cache Warming (autowarming) Using MRU items in the current cache to prepopulate the new cache Warming in parallel with live requests

Lucene IndexReader warming

field norms, FieldCache, tii – the term index

Static Cache warming

Configurable static requests to warm new

Searchers

Smart Cache Warming (autowarming)

Using MRU items in the current cache to prepopulate

the new cache

Warming in parallel with live requests

Smart Cache Warming

Schema Lucene has no notion of a schema Sorting - string vs. numeric Ranges - val:42 included in val:[1 TO 5] ? Lucene QueryParser has date-range support, but must guess. Defines fields, their types, properties Defines unique key field, default search field, Similarity implementation

Lucene has no notion of a schema

Sorting - string vs. numeric

Ranges - val:42 included in val:[1 TO 5] ?

Lucene QueryParser has date-range support,

but must guess.

Defines fields, their types, properties

Defines unique key field, default search

field, Similarity implementation

Field Definitions Field Attributes: name, type, indexed, stored, multiValued, omitNorms <field name=&quot;id“ type=&quot;string&quot; indexed=&quot;true&quot; stored=&quot;true&quot;/> <field name=&quot;sku“ type=&quot;textTight” indexed=&quot;true&quot; stored=&quot;true&quot;/> <field name=&quot;name“ type=&quot;text“ indexed=&quot;true&quot; stored=&quot;true&quot;/> <field name=“reviews“ type=&quot;text“ indexed=&quot;true“ stored=“false&quot;/> <field name=&quot;category“ type=&quot;text_ws“ indexed=&quot;true&quot; stored=&quot;true“ multiValued=&quot;true&quot;/> Dynamic Fields, in the spirit of Lucene! <dynamicField name=&quot;*_i&quot; type=&quot;sint“ indexed=&quot;true&quot; stored=&quot;true&quot;/> <dynamicField name=&quot;*_s&quot; type=&quot;string“ indexed=&quot;true&quot; stored=&quot;true&quot;/> <dynamicField name=&quot;*_t&quot; type=&quot;text“ indexed=&quot;true&quot; stored=&quot;true&quot;/>

Field Attributes: name, type, indexed, stored, multiValued,

omitNorms

<field name=&quot;id“ type=&quot;string&quot; indexed=&quot;true&quot; stored=&quot;true&quot;/>

<field name=&quot;sku“ type=&quot;textTight” indexed=&quot;true&quot; stored=&quot;true&quot;/>

<field name=&quot;name“ type=&quot;text“ indexed=&quot;true&quot; stored=&quot;true&quot;/>

<field name=“reviews“ type=&quot;text“ indexed=&quot;true“ stored=“false&quot;/>

<field name=&quot;category“ type=&quot;text_ws“ indexed=&quot;true&quot; stored=&quot;true“

multiValued=&quot;true&quot;/>

Dynamic Fields, in the spirit of Lucene!

<dynamicField name=&quot;*_i&quot; type=&quot;sint“ indexed=&quot;true&quot; stored=&quot;true&quot;/>

<dynamicField name=&quot;*_s&quot; type=&quot;string“ indexed=&quot;true&quot; stored=&quot;true&quot;/>

<dynamicField name=&quot;*_t&quot; type=&quot;text“ indexed=&quot;true&quot; stored=&quot;true&quot;/>

Search Relevancy

Configuring Relevancy <fieldtype name=&quot;text&quot; class=&quot;solr.TextField&quot;> <analyzer> <tokenizer class=&quot;solr.WhitespaceTokenizerFactory&quot;/> <filter class=&quot;solr.LowerCaseFilterFactory&quot;/> <filter class=&quot;solr.SynonymFilterFactory&quot; synonyms=&quot;synonyms.txt“/> <filter class=&quot;solr.StopFilterFactory“ words=“stopwords.txt”/> <filter class=&quot;solr.EnglishPorterFilterFactory&quot; protected=&quot;protwords.txt&quot;/> </analyzer> </fieldtype>

<fieldtype name=&quot;text&quot; class=&quot;solr.TextField&quot;>

<analyzer>

<tokenizer class=&quot;solr.WhitespaceTokenizerFactory&quot;/>

<filter class=&quot;solr.LowerCaseFilterFactory&quot;/>

<filter class=&quot;solr.SynonymFilterFactory&quot;

synonyms=&quot;synonyms.txt“/>

<filter class=&quot;solr.StopFilterFactory“

words=“stopwords.txt”/>

<filter class=&quot;solr.EnglishPorterFilterFactory&quot;

protected=&quot;protwords.txt&quot;/>

</analyzer>

</fieldtype>

copyField Copies one field to another at index time Usecase: Analyze same field different ways copy into a field with a different analyzer boost exact-case, exact-punctuation matches language translations, thesaurus, soundex <field name=“title” type=“text”/> <field name=“title_exact” type=“text_exact” stored=“false”/> <copyField source=“title” dest=“title_exact”/> Usecase: Index multiple fields into single searchable field

Copies one field to another at index time

Usecase: Analyze same field different ways

copy into a field with a different analyzer

boost exact-case, exact-punctuation matches

language translations, thesaurus, soundex

<field name=“title” type=“text”/>

<field name=“title_exact” type=“text_exact” stored=“false”/>

<copyField source=“title” dest=“title_exact”/>

Usecase: Index multiple fields into single searchable

field

Web Admin Interface Show Config, Schema, Distribution info Query Interface Statistics Caches: lookups, hits, hitratio, inserts, evictions, size RequestHandlers: requests, errors UpdateHandler: adds, deletes, commits, optimizes IndexReader, open-time, index-version, numDocs, maxDocs, Analysis Debugger Shows tokens after each Analyzer stage Shows token matches for query vs index

Show Config, Schema, Distribution info

Query Interface

Statistics

Caches: lookups, hits, hitratio, inserts, evictions, size

RequestHandlers: requests, errors

UpdateHandler: adds, deletes, commits, optimizes

IndexReader, open-time, index-version, numDocs,

maxDocs,

Analysis Debugger

Shows tokens after each Analyzer stage

Shows token matches for query vs index

Selling Points Fast Powerful & Configurable High Relevancy Mature Product Same features as software costing $$$ Leverage Community Lucene committers, IR experts Free consulting: shared problems & solutions

Fast

Powerful & Configurable

High Relevancy

Mature Product

Same features as software costing $$$

Leverage Community

Lucene committers, IR experts

Free consulting: shared problems & solutions

Where are we going? OOTB Simple Faceted Browsing Automatic Database Indexing Federated Search HA with failover Alternate output formats (JSON, Ruby) Highlighter integration Spellchecker Alternate APIs (Google Data, OpenSearch)

OOTB Simple Faceted Browsing

Automatic Database Indexing

Federated Search

HA with failover

Alternate output formats (JSON, Ruby)

Highlighter integration

Spellchecker

Alternate APIs (Google Data, OpenSearch)