Apache Solr is an open source search engine based on Apache Lucene. It provides features like hit highlighting, faceted search, indexing and searching large amounts of data from databases or XML files. Solr is written in Java and can be run on Tomcat, JBoss or Jetty. It allows searching and filtering of data through queries and faceted navigation. Solr supports sharding and replication across multiple servers for high performance and scalability.

1. Apache Solr Ratification
cdevecchi@gmail.com

2. Solr – What is it?
• Apache Project
• Open source engine based in lucene
• APIs XML/HTTP e JSON

3. Features
• Lemmatization
• Hit Highlight
• Dictionaries
• Geosearch
• Faceted Search
• Caching
• Index Replication and Databases Integration

4. Characteristics
• Java -> Tomcat / Jboss / Jetty
• Schema
• Client solrj
• Jmx statistics

5. Query
• Highlighting
– Activated by query (hl=true)
• Text Analysis
– Use dictionary and thesaurus
– Relevancy searchs
– Spelling suggestions
– Search by similarity (“More like this”)
– Fuzzy (Damerau-Levenshtein distance)

6. Query
• Querying data
– Words
– Words by field
– Orderly (sort)
• Faceted Search
– Categories

7. Query
• Faceted Search, the queries could be a problem?
– Exemple
http://localhost:8983/solr/select?
q=video&rows=0&facet=true&facet.field=inStock
&facet.query=price:[*+TO+500]
&facet.query=price:[500+TO+*]
&facet.prefix=xx&facet.limit=5&facet.mincount=1

8. Data indexing
• Solr XML native
• CSV
• Database (DIH)
• Rich Documents
• Crawler

9. Index
• Index is being larger than you imagine?
• Could be adjusted in:
– Index size segments
– Merge index segments

10. Collections
• It is possible to create separated index by documents kind

11. Data Replication
• Master / Slave
- Index
- Config files

12. Sharding
• ZooKeeper
– http://hadoop.apache.org/zookeeper/

13. SolrCaching
• Put searched docs on cache
• Two implementations
– Solr.search.LRUCache (LRU= Least Recently Used in
memory)
– Solr.search.FastLRUCache (a partir da versão 1.4)
• How to use
– filterCache
– queryResultCache
– documentCache (sobe tudo em memória)

14. Cluster – Carrot2
• Search Results Clustering Engine
• Search in many nodes
• Live Demo
– http://search.carrot2.org/stable/search

15. Crawling
• Apache Nutch
– Search, parse and parallel indexing or distributed indexing
– Many formats
• Ex. plain text, html, xml, zip, .doc, javascript, rss, pdf, etc
– Cluster
– MapReduce
– Distributed Filesystem (via hadoop)

16. Backup / Snapshot
• Active by scripts (solr-tools)
• Index snapshots
• Diferencial backups
– $solr_data/yyyymmdd

17. Architecture (Master/Slave)

18. Architecture (Índice Distribuído)

19. Indexing Tests
• Indexing tests
• 7k xml sized, with 111 fields
• 1,2 milion docs on index
• VM -> 2GB RAM, processor 2.33 Ghz

20. Indexing Tests
90
44

21. Search Tests

22. QPS
61
0
37
5
38 38

23. References
• http://lucene.apache.org/solr/
• http://wiki.apache.org/solr/
• http://project.carrot2.org/
• http://download.carrot2.org/head/manual/index.html#chapter.introduction
• http://wiki.apache.org/solr/ZooKeeperIntegration