Lucene and MySQL

Lucene with MySQL
Farhan “Frank” Mashraqi
DBA
Fotolog, Inc.
[email_address]
[email_address]

Introduction
Farhan Mashraqi
Senior MySQL DBA of Fotolog, Inc.
Known on Planet MySQL as “Frank Mash”

What is Lucene?
Started in 1997 “self serving project”
2001: Apache folks adopts Lucene
Open Source Information Retrieval (IR) Library
available from the Apache Software Foundation
Search and Index any textual data
Doesn’t care about language, source and format of data

Lucene?
Not a turnkey search engine
Standard
for building open-source based large-scale search applications
a high performance, scalable, cross-platform search toolkit
Today: translated into C++, C#, Perl, Python, Ruby
for embedded and customizable search
widely adopted by OEM software vendors and enterprise IT departments

Lucene DB Web Aggregate Data File System Index Documents Index Search Index Search Results User Query LUCENE Application

What types of queries it supports
Single and multi-term queries
Phrase queries
Wildcards
Result ranking
+apple –computer +pie
country:USA
country:USA AND state:CA

Cons
Need Java resources (programmers)
JSP experience plus
Implementation and Maintenance Cost
By default
No installer or wizard for setup (it’s a toolkit  )
No administration or command line tools (demo avail.)
No spider
Coding yourself is always an option
No complex script language support by default
3 rd party tools available

Cons 2
No built-in support for (Demos avail. for how to implement)
HTML format
PDF format
Microsoft Office Documents
Advanced XML queries
“ How tos” available.
No database gateway
Integrates with MySQL with little work
Web interface
JSP sample available
Missing enterprise support

Lucene Libraries
The Lucene libraries include core search components such as a document indexer, index searcher, query parser, and text analyzer.

Who is behind Lucene?
Doug Cutting (Author)
Previously at Excite
Apache Software Foundation

Who uses Lucene?
IBM
- IBM OmniFind Yahoo! Edition
CNET
http://reviews.cnet.com/
http://www.mail-archive.com/java-user@lucene.apache.org/msg02645.html
Wikipedia
Fedex
Akamai’s EdgeComputing platform
Technorati
FURL
Sun
Open Solaris Source Browser

When to use Lucene?
Search applications
Search functionality for existing applications
Search enabling database application

When not to use?
Not ideal for
Adding generic search to site
Enterprise systems needing support for proprietary formats
Extremely high volume systems
Through a better architecture this can be solved
Investigate carefully if
You need more than 100 QPS per system
Highly volatile data
Updates are actually Deletes and Additions
Additions visible to new sessions only

Why Lucene?
What problems does Lucene solve?
Full text with MySQL
Pros and Cons
Powerful features
Simple API
Scalable, cost-effective, efficient Indexing
Powerful Searching through multiple query types

Powerful features
Simple API
Sort by any field
Simultaneous updates and searching

Core Index Classes
IndexWriter
Directory
Analyzer
Document
Field

IndexWriter
IndexWriter
Creates new index
Adds document to new index
Gives you “write” access but no “read” access
Not the only class used to modify an index
Lucene API can be used as well

Directory
Directory
Represents location of the Lucene Index
Abstract class
Allows its subclasses to store the index as they see fit
FSDirectory
RAMDirectory
Interface Identical to FSDirectory

Analyzer
Analyzer
Text passed through analyzer before indexing
Specified in the IndexWriter constructor
Incharge of extracting tokens out of text to be indexed
Rest is eliminated
Several implementation available (stop words, lower case etc)

Document
Document
Collection of fields (virtual document)
Chunk of data
Fields of a document represent the document or meta-data associated with that document
-Original source of Document data (word PDF) irrelevant
Metadata indexed and stored separately as fields of a document
Text only: java.lang.String and java.io.Reader are the only things handled by core

Field 1
Field
Document in an index contains one or more fields (in a class called Field)
Each field represents data that is either queried against or retrieved from index during search.
Four different types:
Keyword
Isn’t analyzed
But indexed and stored in the index
Ideal for:
URLs
Paths
SSN
Names
Orginal value is reserved in entirety

Field types
Unindexed
Neither analyzed nor indexed
Value stored in index as is
Fields that need to be displayed with search results (URL etc)
But you won’t search based on these fields
Because original values are stored
Don’t store fields with very large values
Especially if index size will be an issue

Field types
Unstored
Opposite of UnIndexed
Field type is analyzed and indexed but isn’t stored in the index
Suitable for indexing a large amount of text that’s not going to be needed in original form
E.g.
HTML of a webpage etc

Field types
Text
Analyzed and indexed
Field of this type can be searched against
Be careful about the field size
If data indexed is String, it will be stored
If Data is from a Reader
It will not be stored

Note:
Field.Text(String, String) and Field.Text(String, Reader) are different.
(String, String) stores the field data
(String, Reader) does not
To index a String, but not store it, use
Field.UnStored(String, String)

Classes for Basic Search Operations
IndexSearcher
Opens an index in read-only mode
Offers a number of search methods
Some of which implemented in Searcher class
IndexSearcher is = new IndexSearcher(
FSDirectory.getDirectory("/tmp/index", false));
Query q = new TermQuery(new Term("contents", "lucene"));
Hits hits = is.search (q);

Term
Basic unit for searching
Consists of pair of string elements: name of field and value of field
Term objects are involved in indexing process
Term objects can be constructed and used with TermQUery
Query q = new TermQuery( new Term("contents", "lucene") );
Hits hits = is.search(q);

Query
A number of query subclasses
BooleanQuery
PhraseQuery
PrefixQuery
PhrasePrefixQuery
RangeQuery
FilteredQuery
SpanQuery

TermQuery
Most basic type of query supported by Lucene
Used for matching documents that contain fields with specific values
Hits
Simple container of pointers to ranked search results.
Hits instances don’t load from index all documents that match a query but only a small portion (performance)

Indexing
Multiple type indexing
Scalable
High Performance
“ over 20MB/minute on Pentium M 1.5GHz”
Incremental indexing and batch indexing have same cost
Index Size
index size roughly 20-30% the size of text indexed
Compare to MySQL’s FULL-TEXT index size
Cost-effective
1 MB heap (small RAM needed)

Powerful Searching & Sorting
Ranked Searching
Multiple Powerful Query Types
phrase queries, wildcard queries, proximity queries, range queries and more
Fielded Searching
fielded searching (e.g., title, author, contents)
Date Range Searching
date-range searching
Multiple Index Searching with Merged Results
Sort by any field

How to Integrate Your Application With Lucene
Install JDK (5 or 6)
Testing Lucene Demo

Prerequisites: JDK
Installing JDK
For downloading visit the JDK5 http://java.sun.com/javase/downloads/index_jdk5.jsp page
or JDK 6 download page http://java.sun.com/javase/downloads/index.jsp
Once downloaded:
Change Permissions
[root@srv31 jdk-install]# chmod 755 jdk-1_5_0_09-linux-i586.bin
Install
[root@srv31 jdk-install]# ./jdk-1_5_0_09-linux-i586.bin

Testing Lucene Demo
Step 2: Testing Lucene Demo
Set up your environment
vi /root/.bashrc
export PATH=/var/www/html/java/jdk1.5.0_09/bin:$PATH export CLASSPATH=.:/var/www/html/java/jdk1.5.0_09:/var/www/html/java/jdk1.5.0_09/lib:/var/www/html/java/jdk1.5.0_09/lib/lucene-2.1.0/lucene-core-2.1.0.jar:/var/www/html/java/jdk1.5.0_09/lib/lucene-2.1.0/lucene-demos-2.1.0.jar:/var/www/html/java/jdk1.5.0_09/lib/xmlrpc-3.0a1.jar
Now get and place in /var/www/html/java/jdk1.5.0_09/lib/lucene-2.1.0/
Lucene Java
http://www.apache.org/dyn/closer.cgi/lucene/java/
XMLRPC Library
[root@srv31 lib]# wget http://mirror.candidhosting.com/pub/apache/lucene/java/lucene-2.1.0.zip [root@srv31 lib]# unzip lucene-2.1.0.zip [root@srv31 lib]# cp -p lucene-2.1.0/lucene-core-2.1.0.jar ../lib/ [root@srv31 lib]# cp -p lucene-2.1.0/lucene-demos-2.1.0.jar ../lib/ [root@srv31 lib]# cp -p /var/www/html/java/jdk1.5.0_06/lib/xmlrpc-3.0a1.jar /var/www/html/java/jdk1.5.0_09/lib/xmlrpc-3.0a1.jar Now "dot" the above file: [root@srv31 lib]# . /root/.bashrc

Testing Lucene Demo 2
Believe it or not, we are now ready to test the Lucene Demo.
Indexing
I just let it loose on a randomly picked directory to give you an idea: [root@srv31 lib]# java org.apache.lucene.demo.IndexFiles /var/www/html/java/jdk1.5.0_09/ adding /var/www/html/java/jdk1.5.0_09/include/jni.h adding /var/www/html/java/jdk1.5.0_09/include/linux/jawt_md.h adding /var/www/html/java/jdk1.5.0_09/include/linux/jni_md.h adding /var/www/html/java/jdk1.5.0_09/include/jvmti.h adding /var/www/html/java/jdk1.5.0_09/include/jvmdi.h Optimizing... 157013 total milliseconds

Testing Lucene Demo 3
[root@srv31 lib]# java org.apache.lucene.demo.SearchFiles
Query: java
Searching for: java
1159 total matching documents
1. /var/www/html/java/jdk1.5.0_09/jre/lib/locale/zh.GBK/LC_MESSAGES/sunw_java_plugin.mo
2. /var/www/html/java/jdk1.5.0_09/jre/lib/locale/zh/LC_MESSAGES/sunw_java_plugin.mo
3. /var/www/html/java/jdk1.5.0_09/jre/lib/locale/ko/LC_MESSAGES/sunw_java_plugin.mo
4. /var/www/html/java/jdk1.5.0_09/jre/lib/locale/zh_HK.BIG5HK/LC_MESSAGES/sunw_java_plugin.mo
5. /var/www/html/java/jdk1.5.0_09/jre/lib/locale/zh_TW.BIG5/LC_MESSAGES/sunw_java_plugin.mo
6. /var/www/html/java/jdk1.5.0_09/jre/lib/locale/zh_TW/LC_MESSAGES/sunw_java_plugin.mo
7. /var/www/html/java/jdk1.5.0_09/demo/jfc/Stylepad/README.txt
8. /var/www/html/java/jdk1.5.0_09/demo/jfc/Notepad/README.txt
9. /var/www/html/java/jdk1.5.0_09/demo/plugin/jfc/Stylepad/README.txt
10. /var/www/html/java/jdk1.5.0_09/demo/plugin/jfc/Notepad/README.txt
more (y/n) ?

Loading data from MySQL
…
String url = "jdbc:mysql://127.0.0.1/odp";
Connection con = DriverManager.getConnection(url, “user", “pass");
Statement Stmt = con.createStatement();
ResultSet RS = Stmt.executeQuery
("SELECT * FROM " +
" articles" );

Loading data from MySQL 2
while (RS.next()) {
// System.out.print(""" + RS.getString(1) + """);
try {
final Document doc = new Document();
// create Document
doc.add(Field.Text("title", RS.getString("title")));
doc.add(Field.Text("type", "article"));
doc.add(Field.Text("author", RS.getString("author")));
doc.add(Field.Text("body", RS.getString("body")));
doc.add(Field.Text("extended", RS.getString("extended")));
…

Loading data from MySQL 3
…
doc.add(Field.Text("tags", RS.getString("tags")));
doc.add(Field.UnIndexed("permalink", RS.getString("permalink") ));
doc.add(Field.UnIndexed("id", RS.getString("id")));
doc.add(Field.UnIndexed("member_id", RS.getString("member_id")));
doc.add(Field.UnIndexed("portal_id", RS.getString("portal_id")));
//doc.add(Field.Text("id", RS.getString("id")));
writer.addDocument(doc);
}
catch (IOException e) { System.err.println("Unable to index student"); }
}
// close connection

Searching Data using XML RPC
public static void searchArticles( final String search, final int numberOfResults)
throws Exception
{
final Query query;
Analyzer analyzer = new StandardAnalyzer();
query = QueryParser.parse(search, "title", analyzer);
final ArrayList ids = new ArrayList();
try {
final IndexReader reader = IndexReader.open(INDEX_DIR);
final IndexSearcher searcher = new IndexSearcher(reader);
final Hits hits = searcher.search(query);
for (int i = 0; i != hits.length() && i != numberOfResults; ++i) {
final Document doc = hits.doc(i);
// id field needs to be added //ids.add(new Integer(doc.getField("id").stringValue()));
…

Searching Data using XML RPC 2
…
ids.add(new Integer(doc.getField("id").stringValue()));
System.out.println("Found + " + doc.getField("id").stringValue() );
System.out.println("--Title = " + doc.getField("title").stringValue() );
System.out.println("--Type = " + doc.getField("type").stringValue() );
System.out.println("--Body = " + doc.getField("body").stringValue() );
System.out.println("--Author = " + doc.getField("author").stringValue() );
System.out.println("--Extended = " + doc.getField("extended").stringValue() );
System.out.println("--Tags = " + doc.getField("tags").stringValue() );
System.out.println("--Permalink = " + doc.getField("permalink").stringValue() );
System.out.println("--Member Id = " + doc.getField("member_id").stringValue() );
System.out.println("--Portal Id = " + doc.getField("portal_id").stringValue() );

Searching Data using XML RPC 3
}
searcher.close();
reader.close();
}
catch (IOException e) {
System.out.println("Error while reading student data from index");
}
}

Future of Lucene
Advanced Linguistics Modules that integrate with Lucene
Support for complex script languages
Basis Technologies’ Rosette® Linguistics Platform
The same linguistic software that powers multilingual web search on Google, Live.com, Yahoo! and leading enterprise search engines
“ allows Lucene-based applications to index and search text in multiple languages concurrently, including complex script languages such as Arabic, Chinese, Farsi, Japanese and Korean. “
www.basistech.com/lucene

What are the ports of Lucene
Lucene4c - C
CLucene - C++
MUTIS - Delphi
Lucene.Net - a straight C#/.NET port of Lucene by the Apache Software Foundation , fully compatible with it.
Plucene - Perl
Kinosearch - Perl
Pylucene - Lucene interfaced with a Python front-end
Ferret and RubyLucene - Ruby
Zend Framework (Search) - PHP
Montezuma - Common Lisp

Where to get help about Lucene?
http://lucene.apache.org/java/docs/mailinglists.html
IRC

Books about Lucene
Lucene in Action
Erik Hatcher and Otis Gospodnetic

Questions?

http://www.techpresentations.org	2650
http://www.slideshare.net	142
http://www.techiegyan.com	26
http://www.fachak.com	21
http://translate.googleusercontent.com	7
http://qwerty7777.blogspot.com	3
http://www.linkedin.com	2
http://fachak.com	1
http://www.x608.com	1
http://74.125.95.132	1
http://thesis.aloneinkyoto.net	1
http://static.slideshare.net	1
http://74.125.77.132	1
http://74.125.93.104	1
http://216.239.59.104	1
http://www.aloneinkyoto.net	1

Lucene and MySQL

by frankmashraqi

Accessibility

Categories

Upload Details

Usage Rights

16 Embeds 2,860

Statistics

Lucene and MySQL Presentation Transcript