This document provides an introduction to Lucene and related projects. It discusses how Lucene allows for indexing and searching of text documents, with features like relevance sorting, wildcard searches, and range queries. The document also introduces related projects like Solr, which provides a search server and REST API on top of Lucene, and Tika, which allows extracting text and metadata from various file formats. Overall, the document gives a high-level overview of Lucene and its ecosystem for text search and information retrieval.

1. Lucene Intro

2. About Me
• Cristian Vat
• Java Developer / Geek / Enthusiast
• Contact
• @deathy
• ... or TM JUG mailing list

3. About YOU
• Heard about Lucene / Solr ?
• Used Lucene / Solr ?

4. Databases / Text Search

5. Databases
• Select/Search on (usually) exact values or
ranges
• Group/Summarize Results
• Sort results by value(s) of certain result
column(s)

6. Text Search
• Search for individual words/tokens
• Search long text documents
• More language-aware
• “Sorting” by Relevance of results by default

7. IR Quick Intro

8. IR = Information Retrieval

9. IR Quick Intro
• Doc 1: “I did enact Julius Caesar: I was killed i’
the Capitol; Brutus killed me.”
• Doc 2: “So let it be with Caesar. The noble
Brutus hath told you Caesar was ambitious:”

10. IR Quick Intro
• Index
• “I” -> Doc 1
• “Caesar” -> Doc 1, Doc 2
• “enact” -> Doc 1
• “noble” -> Doc 1

11. IR Quick Intro
• Search
• caesar
• c?es*
• caesar AND noble
• “Julius Caesar”
• Caesar NOT Brutus

12. Lucene Ecosystem
...and many more

13. Lucene
• IR Library
• Just API for Indexing/Searching
• No GUI
• No parsers for different file formats

14. Lucene
• Fast
• Thread-Safe/Multi-Threaded indexing and
searching
• No dependencies! (not even logging
framework)

15. Solr
• Search Server / Layer over Lucene
• Provides REST-like HTTP (JSON/XML) API
• Client libraries in Java, PHP, Python, Ruby,
Perl, .NET, ...

16. Solr
• More structured indexes
• Replication / Distribution, Master-Slave, etc.
• Faceted Search / Filtering
• Indexing of rich document types (via Tika)

17. Tika
• “Content Analysis Toolkit”
• Text and Metadata extraction from various
rich document types
• Used by Solr for indexing rich document
types

18. Lucene (in more detail)

19. Lucene Index Structure
• Index = One or more Documents
• Document = one or more Fields with values
• NO Schema/Structure restrictions

20. Adding documents

21. Lucene Search

22. Query Parser
• AND, OR, NOT ( +/- )
• “apache AND lucene NOT solr” ( “+apache
+lucene -solr” )
• Range Queries
• year:[1994 TO 2011]
• Wildcard/Fuzzy:
• “ap?che”, “apac*”, “appche”˜0.8

23. Sorting or Results
• Default sort by Relevance
• Possible to use custom sort fields

24. Relevance
• Score is calculated for each document based
on individual document/fields and the current
search query

25. For the nerds
http://lucene.apache.org/core/old_versioned_docs/versions/3_5_0/api/core/org/apache/lucene/search/Similarity.html

26. Analysis
• From long continuous text to individual
tokens/words used for indexing

27. Analysis
• Text -> Tokenizer -> (TokenFilter)* -> Tokens

28. Tokenizer
• Splits main text into words, by whitespace,
punctuation, other rules
• Text: “So, it has come to this!”
• Tokens: [ “So”, “it”, “has”, “come”, “to”, “this” ]

29. Token Filters
• Change existing tokens or add new ones
• Case-Folding
• Synonyms
• Stemming

30. Token Filters
• Text: “The Pandorica was constructed to
ensure the safety of the Alliance.”
• Tokens: [“The”, “Pandorica”, “was”,
“constructed”, “to”, “ensure”, “the”, “safety”,
“of”, “the”, “Alliance” ]
• Filtered: [ “pandorica”, “was”, “construct”,
“to”, “ensure”, “safe”, “of”, “alliance” ]

31. Q&A
Questions?

32. Thanks