Lucene and MySQL

Loading...

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

0 comments

Post a comment

    Post a comment
    Embed Video
    Edit your comment Cancel

    21 Favorites & 1 Group

    Lucene and MySQL - Presentation Transcript

    1. Lucene with MySQL
      • Farhan “Frank” Mashraqi
      • DBA
      • Fotolog, Inc.
      • [email_address]
      • [email_address]
    2. Introduction
      • Farhan Mashraqi
      • Senior MySQL DBA of Fotolog, Inc.
      • Known on Planet MySQL as “Frank Mash”
    3. What is Lucene?
      • Started in 1997 “self serving project”
      • 2001: Apache folks adopts Lucene
      • Open Source Information Retrieval (IR) Library
        • available from the Apache Software Foundation
      • Search and Index any textual data
        • Doesn’t care about language, source and format of data
    4. Lucene?
      • Not a turnkey search engine
      • Standard
        • for building open-source based large-scale search applications
        • a high performance, scalable, cross-platform search toolkit
      • Today: translated into C++, C#, Perl, Python, Ruby
        • for embedded and customizable search
          • widely adopted by OEM software vendors and enterprise IT departments
    5. Lucene DB Web Aggregate Data File System Index Documents Index Search Index Search Results User Query LUCENE Application
    6. What types of queries it supports
      • Single and multi-term queries
      • Phrase queries
      • Wildcards
      • Result ranking
      • +apple –computer +pie
      • country:USA
      • country:USA AND state:CA
    7. Cons
      • Need Java resources (programmers)
        • JSP experience plus
      • Implementation and Maintenance Cost
      • By default
        • No installer or wizard for setup (it’s a toolkit  )
        • No administration or command line tools (demo avail.)
        • No spider
          • Coding yourself is always an option
        • No complex script language support by default
          • 3 rd party tools available
    8. Cons 2
        • No built-in support for (Demos avail. for how to implement)
          • HTML format
          • PDF format
          • Microsoft Office Documents
          • Advanced XML queries
            • “ How tos” available.
          • No database gateway
            • Integrates with MySQL with little work
          • Web interface
            • JSP sample available
          • Missing enterprise support
    9. Lucene Libraries
        • The Lucene libraries include core search components such as a document indexer, index searcher, query parser, and text analyzer.
    10. Who is behind Lucene?
      • Doug Cutting (Author)
        • Previously at Excite
      • Apache Software Foundation
    11. Who uses Lucene?
      • IBM
        • - IBM OmniFind Yahoo! Edition
      • CNET
        • http://reviews.cnet.com/
        • http://www.mail-archive.com/java-user@lucene.apache.org/msg02645.html
      • Wikipedia
      • Fedex
      • Akamai’s EdgeComputing platform
      • Technorati
      • FURL
      • Sun
        • Open Solaris Source Browser
    12. When to use Lucene?
      • Search applications
      • Search functionality for existing applications
      • Search enabling database application
    13. When not to use?
      • Not ideal for
        • Adding generic search to site
        • Enterprise systems needing support for proprietary formats
        • Extremely high volume systems
          • Through a better architecture this can be solved
        • Investigate carefully if
          • You need more than 100 QPS per system
        • Highly volatile data
          • Updates are actually Deletes and Additions
          • Additions visible to new sessions only
    14. Why Lucene?
      • What problems does Lucene solve?
        • Full text with MySQL
          • Pros and Cons
      • Powerful features
      • Simple API
      • Scalable, cost-effective, efficient Indexing
      • Powerful Searching through multiple query types
    15. Powerful features
      • Simple API
      • Sort by any field
      • Simultaneous updates and searching
    16. Core Index Classes
      • IndexWriter
      • Directory
      • Analyzer
      • Document
      • Field
    17. IndexWriter
      • IndexWriter
        • Creates new index
        • Adds document to new index
        • Gives you “write” access but no “read” access
        • Not the only class used to modify an index
          • Lucene API can be used as well
    18. Directory
      • Directory
        • Represents location of the Lucene Index
        • Abstract class
          • Allows its subclasses to store the index as they see fit
          • FSDirectory
          • RAMDirectory
            • Interface Identical to FSDirectory
    19. Analyzer
      • Analyzer
        • Text passed through analyzer before indexing
        • Specified in the IndexWriter constructor
          • Incharge of extracting tokens out of text to be indexed
          • Rest is eliminated
        • Several implementation available (stop words, lower case etc)
    20. Document
      • Document
        • Collection of fields (virtual document)
        • Chunk of data
        • Fields of a document represent the document or meta-data associated with that document
        • -Original source of Document data (word PDF) irrelevant
        • Metadata indexed and stored separately as fields of a document
        • Text only: java.lang.String and java.io.Reader are the only things handled by core
    21. Field 1
      • Field
        • Document in an index contains one or more fields (in a class called Field)
        • Each field represents data that is either queried against or retrieved from index during search.
        • Four different types:
          • Keyword
            • Isn’t analyzed
            • But indexed and stored in the index
            • Ideal for:
              • URLs
              • Paths
              • SSN
              • Names
            • Orginal value is reserved in entirety
    22. Field types
          • Unindexed
            • Neither analyzed nor indexed
            • Value stored in index as is
            • Fields that need to be displayed with search results (URL etc)
              • But you won’t search based on these fields
            • Because original values are stored
              • Don’t store fields with very large values
              • Especially if index size will be an issue
    23. Field types
          • Unstored
            • Opposite of UnIndexed
            • Field type is analyzed and indexed but isn’t stored in the index
            • Suitable for indexing a large amount of text that’s not going to be needed in original form
            • E.g.
              • HTML of a webpage etc
    24. Field types
          • Text
            • Analyzed and indexed
            • Field of this type can be searched against
            • Be careful about the field size
            • If data indexed is String, it will be stored
            • If Data is from a Reader
              • It will not be stored
    25. Note:
      • Field.Text(String, String) and Field.Text(String, Reader) are different.
        • (String, String) stores the field data
        • (String, Reader) does not
      • To index a String, but not store it, use
        • Field.UnStored(String, String)
    26. Classes for Basic Search Operations
      • IndexSearcher
        • Opens an index in read-only mode
        • Offers a number of search methods
          • Some of which implemented in Searcher class
          • IndexSearcher is = new IndexSearcher(
          • FSDirectory.getDirectory("/tmp/index", false));
          • Query q = new TermQuery(new Term("contents", "lucene"));
          • Hits hits = is.search (q);
    27. Classes for Basic Search Operations
      • Term
        • Basic unit for searching
        • Consists of pair of string elements: name of field and value of field
        • Term objects are involved in indexing process
        • Term objects can be constructed and used with TermQUery
          • Query q = new TermQuery( new Term("contents", "lucene") );
          • Hits hits = is.search(q);
    28. Classes for Basic Search Operations
      • Query
        • A number of query subclasses
          • BooleanQuery
          • PhraseQuery
          • PrefixQuery
          • PhrasePrefixQuery
          • RangeQuery
          • FilteredQuery
          • SpanQuery
    29. Classes for Basic Search Operations
      • TermQuery
        • Most basic type of query supported by Lucene
        • Used for matching documents that contain fields with specific values
      • Hits
        • Simple container of pointers to ranked search results.
        • Hits instances don’t load from index all documents that match a query but only a small portion (performance)
    30. Indexing
      • Multiple type indexing
        • Scalable
        • High Performance
          • “ over 20MB/minute on Pentium M 1.5GHz”
          • Incremental indexing and batch indexing have same cost
        • Index Size
          • index size roughly 20-30% the size of text indexed
          • Compare to MySQL’s FULL-TEXT index size
      • Cost-effective
        • 1 MB heap (small RAM needed)
    31. Powerful Searching & Sorting
      • Ranked Searching
      • Multiple Powerful Query Types
        • phrase queries, wildcard queries, proximity queries, range queries and more
      • Fielded Searching
        • fielded searching (e.g., title, author, contents)
      • Date Range Searching
        • date-range searching
      • Multiple Index Searching with Merged Results
      • Sort by any field
    32. How to Integrate Your Application With Lucene
      • Install JDK (5 or 6)
      • Testing Lucene Demo
    33. Prerequisites: JDK
      • Installing JDK
        • For downloading visit the JDK5 http://java.sun.com/javase/downloads/index_jdk5.jsp page
          • or JDK 6 download page http://java.sun.com/javase/downloads/index.jsp
        • Once downloaded:
          • Change Permissions
            • [root@srv31 jdk-install]# chmod 755 jdk-1_5_0_09-linux-i586.bin
          • Install
            • [root@srv31 jdk-install]# ./jdk-1_5_0_09-linux-i586.bin
    34. Testing Lucene Demo
      • Step 2: Testing Lucene Demo
        • Set up your environment
          • vi /root/.bashrc
            • export PATH=/var/www/html/java/jdk1.5.0_09/bin:$PATH export CLASSPATH=.:/var/www/html/java/jdk1.5.0_09:/var/www/html/java/jdk1.5.0_09/lib:/var/www/html/java/jdk1.5.0_09/lib/lucene-2.1.0/lucene-core-2.1.0.jar:/var/www/html/java/jdk1.5.0_09/lib/lucene-2.1.0/lucene-demos-2.1.0.jar:/var/www/html/java/jdk1.5.0_09/lib/xmlrpc-3.0a1.jar
          • Now get and place in /var/www/html/java/jdk1.5.0_09/lib/lucene-2.1.0/
            • Lucene Java
              • http://www.apache.org/dyn/closer.cgi/lucene/java/
            • XMLRPC Library
              • [root@srv31 lib]# wget http://mirror.candidhosting.com/pub/apache/lucene/java/lucene-2.1.0.zip [root@srv31 lib]# unzip lucene-2.1.0.zip [root@srv31 lib]# cp -p lucene-2.1.0/lucene-core-2.1.0.jar ../lib/ [root@srv31 lib]# cp -p lucene-2.1.0/lucene-demos-2.1.0.jar ../lib/ [root@srv31 lib]# cp -p /var/www/html/java/jdk1.5.0_06/lib/xmlrpc-3.0a1.jar /var/www/html/java/jdk1.5.0_09/lib/xmlrpc-3.0a1.jar Now "dot" the above file: [root@srv31 lib]# . /root/.bashrc
    35. Testing Lucene Demo 2
      • Believe it or not, we are now ready to test the Lucene Demo.
        • Indexing
          • I just let it loose on a randomly picked directory to give you an idea: [root@srv31 lib]# java org.apache.lucene.demo.IndexFiles /var/www/html/java/jdk1.5.0_09/ adding /var/www/html/java/jdk1.5.0_09/include/jni.h adding /var/www/html/java/jdk1.5.0_09/include/linux/jawt_md.h adding /var/www/html/java/jdk1.5.0_09/include/linux/jni_md.h adding /var/www/html/java/jdk1.5.0_09/include/jvmti.h adding /var/www/html/java/jdk1.5.0_09/include/jvmdi.h Optimizing... 157013 total milliseconds
    36. Testing Lucene Demo 3
      • [root@srv31 lib]# java org.apache.lucene.demo.SearchFiles
      • Query: java
      • Searching for: java
      • 1159 total matching documents
      • 1. /var/www/html/java/jdk1.5.0_09/jre/lib/locale/zh.GBK/LC_MESSAGES/sunw_java_plugin.mo
      • 2. /var/www/html/java/jdk1.5.0_09/jre/lib/locale/zh/LC_MESSAGES/sunw_java_plugin.mo
      • 3. /var/www/html/java/jdk1.5.0_09/jre/lib/locale/ko/LC_MESSAGES/sunw_java_plugin.mo
      • 4. /var/www/html/java/jdk1.5.0_09/jre/lib/locale/zh_HK.BIG5HK/LC_MESSAGES/sunw_java_plugin.mo
      • 5. /var/www/html/java/jdk1.5.0_09/jre/lib/locale/zh_TW.BIG5/LC_MESSAGES/sunw_java_plugin.mo
      • 6. /var/www/html/java/jdk1.5.0_09/jre/lib/locale/zh_TW/LC_MESSAGES/sunw_java_plugin.mo
      • 7. /var/www/html/java/jdk1.5.0_09/demo/jfc/Stylepad/README.txt
      • 8. /var/www/html/java/jdk1.5.0_09/demo/jfc/Notepad/README.txt
      • 9. /var/www/html/java/jdk1.5.0_09/demo/plugin/jfc/Stylepad/README.txt
      • 10. /var/www/html/java/jdk1.5.0_09/demo/plugin/jfc/Notepad/README.txt
      • more (y/n) ?
    37. Loading data from MySQL
      • String url = "jdbc:mysql://127.0.0.1/odp";
      • Connection con = DriverManager.getConnection(url, “user", “pass");
      • Statement Stmt = con.createStatement();
      • ResultSet RS = Stmt.executeQuery
      • ("SELECT * FROM " +
      • " articles" );
    38. Loading data from MySQL 2
      • while (RS.next()) {
      • // System.out.print(""" + RS.getString(1) + """);
      • try {
      • final Document doc = new Document();
      • // create Document
      • doc.add(Field.Text("title", RS.getString("title")));
      • doc.add(Field.Text("type", "article"));
      • doc.add(Field.Text("author", RS.getString("author")));
      • doc.add(Field.Text("body", RS.getString("body")));
      • doc.add(Field.Text("extended", RS.getString("extended")));
    39. Loading data from MySQL 3
      • doc.add(Field.Text("tags", RS.getString("tags")));
      • doc.add(Field.UnIndexed("permalink", RS.getString("permalink") ));
      • doc.add(Field.UnIndexed("id", RS.getString("id")));
      • doc.add(Field.UnIndexed("member_id", RS.getString("member_id")));
      • doc.add(Field.UnIndexed("portal_id", RS.getString("portal_id")));
      • //doc.add(Field.Text("id", RS.getString("id")));
      • writer.addDocument(doc);
      • }
      • catch (IOException e) { System.err.println("Unable to index student"); }
      • }
      • // close connection
    40. Searching Data using XML RPC
      • public static void searchArticles( final String search, final int numberOfResults)
      • throws Exception
      • {
      • final Query query;
      • Analyzer analyzer = new StandardAnalyzer();
      • query = QueryParser.parse(search, "title", analyzer);
      • final ArrayList ids = new ArrayList();
      • try {
      • final IndexReader reader = IndexReader.open(INDEX_DIR);
      • final IndexSearcher searcher = new IndexSearcher(reader);
      • final Hits hits = searcher.search(query);
      • for (int i = 0; i != hits.length() && i != numberOfResults; ++i) {
      • final Document doc = hits.doc(i);
      • // id field needs to be added //ids.add(new Integer(doc.getField("id").stringValue()));
    41. Searching Data using XML RPC 2
      • ids.add(new Integer(doc.getField("id").stringValue()));
      • System.out.println("Found + " + doc.getField("id").stringValue() );
      • System.out.println("--Title = " + doc.getField("title").stringValue() );
      • System.out.println("--Type = " + doc.getField("type").stringValue() );
      • System.out.println("--Body = " + doc.getField("body").stringValue() );
      • System.out.println("--Author = " + doc.getField("author").stringValue() );
      • System.out.println("--Extended = " + doc.getField("extended").stringValue() );
      • System.out.println("--Tags = " + doc.getField("tags").stringValue() );
      • System.out.println("--Permalink = " + doc.getField("permalink").stringValue() );
      • System.out.println("--Member Id = " + doc.getField("member_id").stringValue() );
      • System.out.println("--Portal Id = " + doc.getField("portal_id").stringValue() );
    42. Searching Data using XML RPC 3
      • }
      • searcher.close();
      • reader.close();
      • }
      • catch (IOException e) {
      • System.out.println("Error while reading student data from index");
      • }
      • }
    43. Future of Lucene
      • Advanced Linguistics Modules that integrate with Lucene
        • Support for complex script languages
          • Basis Technologies’ Rosette® Linguistics Platform
            • The same linguistic software that powers multilingual web search on Google, Live.com, Yahoo! and leading enterprise search engines
            • “ allows Lucene-based applications to index and search text in multiple languages concurrently, including complex script languages such as Arabic, Chinese, Farsi, Japanese and Korean. “
          • www.basistech.com/lucene
    44. What are the ports of Lucene
      • Lucene4c - C
      • CLucene - C++
      • MUTIS - Delphi
      • Lucene.Net - a straight C#/.NET port of Lucene by the Apache Software Foundation , fully compatible with it.
      • Plucene - Perl
      • Kinosearch - Perl
      • Pylucene - Lucene interfaced with a Python front-end
      • Ferret and RubyLucene - Ruby
      • Zend Framework (Search) - PHP
      • Montezuma - Common Lisp
    45. Where to get help about Lucene?
      • http://lucene.apache.org/java/docs/mailinglists.html
      • IRC
    46. Books about Lucene
      • Lucene in Action
        • Erik Hatcher and Otis Gospodnetic
    47. Questions?

    + frankmashraqifrankmashraqi, 3 years ago

    custom

    7470 views, 21 favs, 12 embeds more stats

    Presented by Fotolog. Lucene is a powerful, high-pe more

    More info about this document

    © All Rights Reserved

    Go to text version

    • Total Views 7470
      • 6232 on SlideShare
      • 1238 from embeds
    • Comments 0
    • Favorites 21
    • Downloads 0
    Most viewed embeds
    • 1201 views on http://www.techpresentations.org
    • 18 views on http://www.fachak.com
    • 8 views on http://www.techiegyan.com
    • 3 views on http://qwerty7777.blogspot.com
    • 1 views on http://74.125.95.132

    more

    All embeds
    • 1201 views on http://www.techpresentations.org
    • 18 views on http://www.fachak.com
    • 8 views on http://www.techiegyan.com
    • 3 views on http://qwerty7777.blogspot.com
    • 1 views on http://74.125.95.132
    • 1 views on http://74.125.77.132
    • 1 views on http://74.125.93.104
    • 1 views on http://216.239.59.104
    • 1 views on http://www.aloneinkyoto.net
    • 1 views on http://static.slideshare.net
    • 1 views on http://thesis.aloneinkyoto.net
    • 1 views on http://www.x608.com

    less

    Flagged as inappropriate Flag as inappropriate
    Flag as inappropriate

    Select your reason for flagging this presentation as inappropriate. If needed, use the feedback form to let us know more details.

    Cancel
    File a copyright complaint
    Having problems? Go to our helpdesk?

    Categories

    Groups / Events