What’s New in Apache Lucene 2.9

What’s New in Apache Lucene 2.9 A Lucid Imagination Technical White Paper

Abstract Apache Lucene is a high-performance, cross-platform, full-featured Information Retrieval library in open source, suitable for nearly every application that requires full-text search features. Since its introduction nearly 10 years ago, Apache Lucene has become a competitive player for developing extensible, high-performance full-text search solutions. The experience accumulated over time by the community of Lucene committers and contributors and the innovations they have engineered have delivered significant ongoing advances in Lucene’s capabilities. This white paper describes the new features and improvements in the latest version, Apache Lucene 2.9. It is intended mainly for programmers familiar with the broad base of Lucene’s capabilities, though those new to Lucene should also find it a useful exploration of the newest features. In the simplest terms, Lucene is now faster and more flexible than before. Historic weak points have been improved to open the way for innovative new features like near-real-time search, flexible indexing, and high-performance numerical range queries. Many new features have been added, new APIs introduced, and critical bugs have been fixed—all with the same goal: improving Lucene’s state-of-the-art search capabilities. What’s New in Lucene 2.9 A Lucid Imagination Technical White Paper • October 2009 Page ii

Table of Contents Introduction ............................................................................................................................................................ 1 Core Features and Improvements .................................................................................................................. 3 Numeric Capabilities and Numeric Range Queries .............................................................................. 3 New TokenStream API .................................................................................................................................... 7 Per-Segment Search ...................................................................................................................................... 11 Near Realtime Search (NRS) ...................................................................................................................... 12 MultiTermQuery-Related Improvements ............................................................................................. 13 Payloads ............................................................................................................................................................. 14 Additions to Lucene Contrib .......................................................................................................................... 16 New Contrib Analyzers ................................................................................................................................ 16 Lucene Spatial (formerly known as LocalLucene) ............................................................................ 16 Lucene Remote and Java RMI .................................................................................................................... 18 New Flexible QueryParser .......................................................................................................................... 18 Minor Changes and Improvements in Lucene 2.9 ............................................................................. 19 Strategies for Upgrading to Lucene 2.9 ..................................................................................................... 21 Upgrade to 2.9—Recommended Actions .............................................................................................. 21 Upgrade to 2.9—Optional Actions ........................................................................................................... 22 References ............................................................................................................................................................ 23 Next Steps ............................................................................................................................................................. 24 APPENDIX: Choosing Lucene or Solr .......................................................................................................... 25 What’s New in Lucene 2.9 A Lucid Imagination Technical White Paper • October 2009 Page iii

Introduction Apache Lucene is a high-performance, cross-platform, full-featured Information Retrieval library, in open source, suitable for nearly every application that requires full-text search features. Lucene currently ranks among the top 15 open source projects and is one of the top 5 Apache projects, with installations at over 4,000 companies. Downloads of Lucene, and its server implementation Solr, have grown nearly tenfold over the past three years; Solr is the fastest-growing Lucene subproject. Lucene and Solr offer an attractive alternative to proprietary licensed search and discovery software vendors.1 With the release of version 2.9 in September 2009, the Apache Lucene community delivered the latest upgrade of Lucene. This white paper aims to address key issues for you if you have an Apache Lucene-based application, and need to upgrade existing code to work well with this latest version, so that you may take advantage of the various improvements and prepare for the next major release. If you do not have a Lucene application, the paper should also give you a good overview of the innovations in this release. Unlike the previous 2.4.1 release (March 2009), Lucene 2.9 is more than just a bug-fix release. It introduces multiple performance improvements, new features, better runtime behavior, API changes, and bug-fixes at a variety of levels. The 2.9 release improves Lucene in several key aspects, which make it an even more compelling alternative to other solutions. Most notably: • Improvements for Near-Realtime Search capabilities make documents searchable almost instantaneously. • A new, straightforward API for handling Numeric Ranges both simplifies development and virtually wipes out performance overhead. • Analysis API has been replaced for more streamlined, flexible text handling. 1 See the Appendix for a discussion of when to choose Lucene or Solr. What’s New in Lucene 2.9 A Lucid Imagination Technical White Paper • October 2009 Page 1

And, behind the scenes, the groundwork has been laid for yet more indexing flexibility in future releases. Lucene Contrib also adds new utility packages, introduced with this release: • An extremely flexible query parser framework opens new possibilities for programmers to more easily create their own query parsing syntax. • Local-Lucene and its geo-search capabilities, now donated to Apache, provide this near-mandatory functionality for state-of-the-art search. • Various contributions have markedly improved support for languages like Arabic, Persian, and Chinese. Some important notes on compatibility: because previous minor releases also contained performance improvements and bug fixes, programmers have been accustomed to upgrading to a new Lucene version just by replacing the JAR file in their classpath. And, in those past cases, Lucene-based apps could be upgraded flawlessly without recompiling the software components accessing or extending Apache Lucene. However, this may not be so with Lucene 2.9. Lucene 2.9 introduces several back-compatibility-breaking changes that may well require changes in your code that uses the library. A drop-in library replacement is not guaranteed to be successful; at a minimum, it is not likely to be flawless. As a result, we recommend that if you are upgrading from a previous Lucene release, you should at least recompile any software components directly accessing or extending the library. In the latter case, recompilation alone will most likely not be sufficient. More details on these dependencies are discussed in the “Upgrading Lucene” section of the paper. We’ve also noted any significant compatibility issues with this label: [BACK-COMPATIBILITY]. Finally, it is important to note that Lucene 2.9 will be the last release supporting the Java 1.4 platform. While the majority of programmers are already running on either version 1.5 or 1.6 platforms (1.6 is our recommended JVM), Java 1.4 reached its end of service life in October 2008. This document is not intended to be a comprehensive overview of Lucene 2.9 in all its functions, but rather the new key features and capabilities. Always check the Lucid Imagination Certified distribution and the official Lucene Website (http://lucene.apache.org) for the most up-to-date release information. What’s New in Lucene 2.9 A Lucid Imagination Technical White Paper • October 2009 Page 2

Core Features and Improvements Numeric Capabilities and Numeric Range Queries One of Apache Lucene's basic properties is its representation of internal searchable values (terms) as UTF-8 encoded characters. Every value passed to Lucene must be converted into a string in order to be searchable. At the same time, Lucene is frequently applied to search numeric values and ranges, such as prices, dates, or other numeric field attributes. Historically, searching over numeric ranges has been a weak point of the library. However, the 2.9 release comes with a tremendous improvement for searching numeric values, especially for range queries. Prior to Lucene 2.9, numeric values were encoded with leading zeros, essentially as a full- precision value. Values stored with full precision ended up creating many unique terms in the index. Thus, if you needed to retrieve all documents in a certain range (e.g., from $1.50 to $1500.0) Lucene had to iterate through a lot of terms whenever many documents with unique values were indexed. Consequently, execution of queries with large ranges and lots of unique terms could be extremely slow as a result of this overhead. Many workaround techniques have evolved over the years to improve the performance of ranges, such as encoding date ranges in multiple fields with separate fields for year, month, and day. But at the end of the day, every programmer had to roll his or her own way of searching ranges efficiently. In Lucene 2.9, NumericUtils and its relatives (NumericRangeQuery / NumericRangeFilter) introduce native numeric encoding and search capabilities. Numeric Java primitives (long, int, float, and double) are transformed into prefix- encoded representations with increasing precision. Internally each prefix precision is generated by stripping off the least significant bits indicated by the precisionStep. Each value is subsequently converted to a sequence of 7-bit ASCII characters (due to the UTF-8 term encoding in the index, 8 or more bits would split into two or more bytes) resulting in a predictable number of prefix-terms that can be calculated ahead of time. The figure below illustrates such a Prefix Tree. What’s New in Lucene 2.9 A Lucid Imagination Technical White Paper • October 2009 Page 3

Example of a Prefix Tree, where the leaves of the tree hold the actual term values and all the descendants of a node have a common prefix associated with the node. Bold circles mark all relevant nodes to retrieve a range from 215 to 977. The generated terms are indexed just like any other string values passed to Lucene. Under the hood, Lucene associates distinct terms with all documents containing the term, so that all documents containing a numeric value with the same prefix are “grouped” together, meaning the number of terms that need to be searched is reduced tremendously. This stands in contrast to the relatively less efficient encoding scheme in previous releases, where each unique numeric value was indexed as a distinct term based on the number of terms in the index. What’s New in Lucene 2.9 A Lucid Imagination Technical White Paper • October 2009 Page 4

Directory directory = new RAMDirectory(); Analyzer analyzer = new WhitespaceAnalyzer(); IndexWriter writer = new IndexWriter(directory, analyzer, IndexWriter.MaxFieldLength.UNLIMITED); for (int i = 0; i < 20000; i++) { Document doc = new Document(); doc.add(new Field("id", String.valueOf(i), Field.Store.YES, Field.Index.NOT_ANALYZED_NO_NORMS)); String num = Integer.toString(i); String paddedValue = "00000".substring(0, 5 - num.length()) + num; doc.add(new Field("oldNumeric", paddedValue, Field.Store.YES, Field.Index.NOT_ANALYZED_NO_NORMS)); writer.addDocument(doc); } writer.close(); Indexing a zero-padded numeric value for use with an ordinary RangeQuery. You can also use the native encoding of numeric values beyond range searches. Numeric fields can be loaded in the internal FieldCache, where they are used for sorting. Zero- padding of numeric primitives (see code example above) is no longer needed as the trie- encoding guarantees the correct ordering without requiring execution overhead or extra coding. The code listing below instead uses the new NumericField to index a numeric Java primitive using 4-bit precision. Like the straightforward NumericField, querying numeric ranges also provides a type-safe API. NumericRangeQuery instances are created using one of the provided static constructors for the corresponding Java primitive. What’s New in Lucene 2.9 A Lucid Imagination Technical White Paper • October 2009 Page 5

Directory directory = new RAMDirectory(); Analyzer analyzer = new WhitespaceAnalyzer(); IndexWriter writer = new IndexWriter(directory, analyzer, IndexWriter.MaxFieldLength.UNLIMITED); for (int i = 0; i < 20000; i++) { Document doc = new Document(); doc.add(new Field("id", String.valueOf(i), Field.Store.YES, Field.Index.NOT_ANALYZED_NO_NORMS)); doc.add(new NumericField("newNumeric", 4, Field.Store.YES, true).setIntValue(i)); writer.addDocument(doc); } writer.close(); Indexing numeric values with the new NumericField type The example below shows a numeric range query using an int primitive with the same precision used in the indexing example. If different precision values are used at index or search time, numeric queries can yield unexpected behavior. What’s New in Lucene 2.9 A Lucid Imagination Technical White Paper • October 2009 Page 6

IndexSearcher searcher = new IndexSearcher(directory, true); Query query = NumericRangeQuery.newIntRange("newNumeric", 4, 10, 10000, true, false); TopDocs docs = searcher.search(query, null, 10); assertNotNull("Docs is null", docs); assertEquals(9990, docs.totalHits); for (int i = 0; i < docs.scoreDocs.length; i++) { ScoreDocs d= docs.scoreDocs[i]; assertTrue(sd.doc >= 10 && sd.doc < 10000); } Searching numeric values with the new NumericRangeQuery Improvements resulting from new Lucene numeric capabilities are equally significant in versatility and performance. Now, Lucene can cover almost every use-case related to numeric values. Moreover, range searches or sorting on float or double values up to fast date searches (dates converted to time stamps) will execute in less than 100 milliseconds in most cases. By comparison, the old approach using padded full-precision values could take up to 30 seconds or more depending on the underlying index. New TokenStream API Almost every programmer who has extended Lucene has worked with its analysis function. Text analysis is common to almost every use-case, and is among the best known Lucene APIs. Since its early days, Lucene has used a “Decorator Pattern” to provide a pluggable and flexible analysis API, allowing a combination of existing and customized analysis implementations. The central analysis class TokenStream enumerates a sequence of tokens from either a document's fields or from a query. Commonly, multiple TokenStream instances are chained, each applying a separate analysis step to text terms represented by a Token class that encodes all relevant information about a term. Prior to Lucene 2.9, TokenStream operated exclusively on Token instances transporting term information through the analysis chain. With this release, the token-based API has been marked as deprecated. It is completely replaced by an attribute-based API. What’s New in Lucene 2.9 A Lucid Imagination Technical White Paper • October 2009 Page 7

Here’s how it has changed. Rather than receiving a Token instance from one of the two TokenStream.next() methods, the new API follows a stateful approach instead. To advance in the stream, consumers call TokenStream.incrementToken(), which returns a Boolean result indicating if the end of the stream has been reached. Information gathered during the analysis process is encoded in attributes accessible via the new TokenStream base class AttributeSource. In contrast to the older Token class, the Attribute-based approach separates specific term characteristics from others not necessarily related. Each TokenStream adds the attributes it is specifically targeting at construction time (see code listing below) and keeps a reference to it throughout its lifetime. This provides type-safe access to all attributes relevant for a particular TokenStream instance. protected CharReplacementTokenStream(TokenStream input) { super(input); termAtt = (TermAttribute) addAttribute(TermAttribute.class); } Adding a TermAttribute at construction time Inside TokenStream.incrementToken(), a token stream only operates on attributes that have been declared in the constructor. For instance, if you have Lucene replacing a character like a German umlaut in a term, only the TermAttribute (declared at construction time in the code listing above) is used. (Other attributes like PositionIncrementAttribute or PayloadAttribute are ignored by this TokenStream as they might not be needed in this particular use-case.) What’s New in Lucene 2.9 A Lucid Imagination Technical White Paper • October 2009 Page 8

public boolean incrementToken() throws IOException { if (input.incrementToken()) { final char[] termBuffer = termAtt.termBuffer(); final int termLength = termAtt.termLength(); if (replaceChar(termBuffer, termLength)) { termAtt.setTermBuffer(output, 0, outputPos); } return true; } return false; } Replacing characters using the new attribute based API. What the above example does not demonstrate is the full power of the new token API. There, we replaced one or more characters in the token and discarded the original one. Yet, in many use-cases, the original token should be preserved in addition to the modified one. Using the old API required a fair bit of work and logic to handle such a common use-case. In contrast, the new attribute-based approach allows capture and restoration of the state of attributes, which makes such use-cases almost trivial. The example below shows a version of the previous example improved for Lucene 2.9, in which the original term attribute is restored once the stream is advanced. What’s New in Lucene 2.9 A Lucid Imagination Technical White Paper • October 2009 Page 9

public boolean incrementToken() throws IOException { if (state != null) { restoreState(state); state = null; return true; } if (input.incrementToken()) { final char[] termBuffer = termAtt.termBuffer(); final int termLength = termAtt.termLength(); if (replaceChar(termBuffer, termLength)) { state = captureState(); termAtt.setTermBuffer(output, 0, outputPos); } return true; } return false; } Replacing characters and additionally emitting the original term text using the new Attribute based API (position increments are omitted). The separation of attributes makes it possible to add arbitrary properties to the analysis chain without using a customized Token class. Attributes are then made type-safely accessible by all subsequent TokenStream instances, and can eventually be used by the consumer. This way, you get a generic way to add various kind of custom information, such as part-of-speech tags, payloads, or average document length to the token stream. Unfortunately, Lucene 2.9 doesn't yet provide functionality to persist custom Attribute implementation to the underlying index. This improvement, part of what is often referred to as "flexible indexing," is under active development and is proposed for one of the upcoming Lucene releases. Beyond the generalizability of this API, one of its most significant improvements is its effective reuse of Attribute instances across multiple iterations of analysis. Attribute What’s New in Lucene 2.9 A Lucid Imagination Technical White Paper • October 2009 Page 10

implementations are created during TokenStream instantiation and are reused each time the stream advances to a successive increment. Even if a stream is used for another analysis, the same Attribute instances may be used, provided the stream is reusable. This greatly reduces the rate of object creation, streamlining execution and minimizing any required garbage collection. While the new API provides full back-compatibility, it is strongly recommended to update any existing custom TokenStream implementations to exclusively use incrementToken instead of one of the overhead-heavy next() methods. If you are trying to update your custom TokenStream or one of its subclass (TokenFilter and Tokenizer) implementations, it is recommended that you use the abstract BaseTokenStreamTestCase class, which provides various utility functions for testing against the new and old API. The test case is freely available for download in the source distribution of Apache Lucene 2.9. Per-Segment Search Since the early days of Apache Lucene, documents have been stored at the lowest level in a segment—a small but entirely independent index. On the highest abstraction level, Lucene combines segments into one large index and executes searches across all visible segments. As more and more documents are added to an index, Lucene buffers your documents in RAM and flushes them to disk periodically. Depending on a variety of factors, Lucene either incrementally adds documents to an existing segment, or creates entirely new segments. To reduce the negative impact of an increasing number of segments on search performance, Lucene tries to combine/merge multiple segments into larger ones. For optimal search performance, Lucene can optimize an index that essentially merges all existing segments into a single segment. Prior to Lucene 2.9, search logic resided at the highest abstraction level, accessing a single IndexReader no matter how many segments the index was composed of. Similarly the FieldCache was associated with the top-level IndexReader, and then had to be invalidated each time an index was reopened. With Lucene 2.9, the search logic and the FieldCache have moved to a per-segment level. While this has introduced a little more internal complexity, the benefit of the tradeoff is a new per-segment index behavior that yields a rich variety of performance improvements for unoptimized indexes. What’s New in Lucene 2.9 A Lucid Imagination Technical White Paper • October 2009 Page 11

In most applications, existing segments rarely change internally, and this property had not been effectively utilized in previous versions of Lucene. IndexReader.reopen(), first added in Lucene 2.4, now has the ability to add new or changed segments to an already existing top-level IndexReader instead of reloading all existing segments. The FieldCache also takes advantage of rarely changing segments. Cache instances of unchanged or updated segments can remain in memory or need only be rebuilt instead of invalidating the FieldCache entirely. Depending on the number of changed index segments, this can heavily reduce I/O as well as garbage collection costs, compared to reopening the entire index. Previous versions of Lucene also suffered from long warming time for sorting and function queries. Those use-cases have been improved as the warm-up of reopened searchers is now much faster. It's worth mentioning that Per-Segment Search doesn't yield improvements in all situations. If an IndexReader is opened on an optimized index, all pre-existing segments are merged into a single one, which then loads in its entirety. In other situations, perhaps more common, where some changes have been committed to the index and a new IndexReader instance is obtained by calling IndexReader.reopen() on a previously opened reader, the new per-segment capabilities can dramatically speed up reopening. But in this case, opening a new IndexReader using one of the overloaded static IndexReader.open() methods will create an entirely new reader instance and therefore can't take advantage of any per-segment capabilities. Near Realtime Search (NRS) More and more, Lucene programmers are pursuing real-time or near real-time requirements with their search applications. Previous Lucene versions did a decent job with the incremental changes characteristic of this scenario, capturing those changes and making them available for searching. Lucene 2.9 adds significant new capabilities for addressing the requirements of high-change document environments. First of all, the IndexWriter – in general responsible for modifying the underlying index and flushing documents to disk – —now offers a way to obtain an IndexReader instance directly from the writer. The newly obtained reader then not only reflects the documents already flushed to disk, but also makes all uncommitted documents still residing in memory almost instantly searchable. What’s New in Lucene 2.9 A Lucid Imagination Technical White Paper • October 2009 Page 12

The reader instance returned by IndexWriter.getReader() supports reopening the reader as long as the writer releasing the reader has not been committed. Once it is committed, re-opening the reader will result in an AlreadyClosedExecption. It is important to understand why this feature is referred to as “near real-time” rather than “real-time.” When IndexWriter.getReader() is called for the very first time, Lucene needs to consume a reasonable amount of additional resources (i.e., RAM, CPU-cycles, and file descriptors) to make uncommitted documents searchable. Due to this additional work, uncommitted documents will not always be available instantaneously. Nonetheless, in most cases, the performance gained with this feature will be better than just reopening the index, or the traditional simpler approach of opening a brand new reader instance. To keep the latency as low as possible, the IndexWriter offers an optional “pre-warmup” functionality, by which newly merged segments can be prepared for real-time search. If you are new to this feature, you should be aware that the pre-warmup API is still marked experimental and might change in future releases. MultiTermQuery-Related Improvements In Lucene 2.4, many standard queries, such as FuzzyQuery, WildcardQuery, and PrefixQuery were re-factored and subclassed under MultiTermQuery. Lucene 2.9 adds some improvements under the hood, resulting into much better performance for those queries. [BACK- COMPATIBILITY] 2 In Lucene 2.9, multi-term queries now use a constant score internally, based on the assumption that most programmers don't care about the interim score of the queries resulting from the term expansion that takes place during query rewriting. 2 This could be a back-compatibility issue if one of those classes has been subclassed. What’s New in Lucene 2.9 A Lucid Imagination Technical White Paper • October 2009 Page 13

Although constant-scoring is now the default behavior, the older scoring mode is still available for multiterm queries in 2.9. Beyond that, you can choose one of the following scoring modes: Filtered constant score: rewrites the multiterm query into a ConstantScoreQuery in combination with a filter to match all relevant documents. BooleanQuery constant score: rewrites the multiterm query into a ConstantScoreQuery based on a BooleanQuery by translating each term into an optional Boolean clause. This mode still has a limitation of maxClauseCount and might raise an exception if the query has too many Boolean clauses. Conventional scoring (not recommended): rewrites the multiterm query into an ordinary BooleanQuery. Automatic constant score (default): tries to choose the best constant score mode (Filter or BooleanQuery) based on term and document counts from the query. If the number of terms and documents is small enough, BooleanQuery is chosen, otherwise the query rewrites to a filter-backed ConstantScoreQuery. You can change the scoring mode by passing an implementation of RewriteMethod to MultiTermQuery.setRewriteMethod() as shown in the code example below. PrefixQuery prefixQuery = new PrefixQuery(new Term("aField", "luc")); prefixQuery.setRewriteMethod( MultiTermQuery.CONSTANT_SCORE_FILTER_REWRITE); Explicitly setting a filtered constant-score RewriteMethod on a PrefixQuery Payloads The Payloads feature, though originally added in a previous version of Lucene, remains pretty new to most programmers. A payload is essentially a byte array that is associated with a particular term in the index. Payloads can be associated with a single term during text analysis and subsequently committed directly to the index. On the search side, these byte arrays are accessible to influence the scoring for a particular term, or even to filter entire documents. What’s New in Lucene 2.9 A Lucid Imagination Technical White Paper • October 2009 Page 14

For instance, if your Lucene application is analyzing the phrase “Gangs of New York”, payloads can encode information about the terms “New” and “York” together, so that they are treated as a paired term for the name of a city, or can specify that “Gangs” is a noun rather than a verb. Prior to 2.9, payloads were exposed via a query called BoostingTermQuery, which has now been renamed to PayloadTermQuery. By using this query type, you can query Lucene to find all occurrences where “New” is a part of a city name like “New York” or “New Orleans”. In comparison with previous versions, Lucene 2.9 also provides more control and flexibility for payload scoring. You can pass a custom PayloadFunction to the constructor of a payload-aware query. Each payload is fed back to the custom function, which calculates the score based on the cumulative outcomes of payload occurrences. This improvement becomes even more useful when payloads are used in combination with span queries. Spans represent a range of term positions in a document, whereas in turn, payloads can help scoring based on the distance between terms. For instance, using a PayloadNearQuery , documents can be scored differently if terms are in the same sentence or paragraph if that information is encoded in the payload. At a higher abstraction level, another payload aware TokenFilter has been added. DelimitedPayloadTokenFilter splits tokens separated by a predefined character delimiter, where the first part of the token is the token itself and the second part after the delimiter represents the payload. For example, it can parse an e-mail address, for example carol.smith@apache.org, by making “carol.smith” the token, and creating a payload to represent the domain name, “apache.org”. A customizable payload-encoder takes care of encoding the values while everything else magically happens inside the filter. Besides being a convenient way to add payloads to existing search functionality, this class also serves as a working example of how to use payloads during the analysis process3. 3 See http://www.lucidimagination.com/blog/2009/08/05/getting-started-with-payloads/ for more information. What’s New in Lucene 2.9 A Lucid Imagination Technical White Paper • October 2009 Page 15

Additions to Lucene Contrib So far, we’ve reviewed key new features and improvements introduced in the Apache Lucene core API. This section outlines the major additions and improvements to Lucene Contrib packages. Contrib packages are parts of Lucene which do not necessarily belong to the API core but are often helpful in building Lucene applications. New Contrib Analyzers The Analysis package in Lucene Contrib has always been a valuable source for almost every Lucene programmer. The latest release brings several noteworthy improvements especially in terms of language support. • Better support for Chinese: Chinese, like many Asian languages, does not use white spaces to delimit one word from another, nor is punctuation used at all. Smart-CN provides an analyzer with improved tokenization and capabilities in splitting individual characters. While Smart-CN is part of the analyzers contrib module, it is distributed in its own JAR file because of the large (6MB) file resources it depends on. • “Light10”-based Arabic analysis: a new Analyzer based on a high-performance stemming algorithm (Light10) applying lightweight prefix and suffix removal to Arabic text. • Persian Analyzer: applying character normalization and Persian stopword removal to Persian-only or mixed language text. • Reverse String filter, as in leading wildcards: to support a search feature like leading wildcards efficiently, one of the common tricks/approaches is to index terms in reverse order. A leading wildcard effectively becomes a trailing wildcard if searched against a field with reversed tokens. Lucene Spatial (formerly known as LocalLucene) Geospatial search has become a very common use-case, especially with the advent of mobile devices. Almost every new mobile platform supports a “nearby” search feature. End users seeking data on something near their current location (restaurants, movie theatres, What’s New in Lucene 2.9 A Lucid Imagination Technical White Paper • October 2009 Page 16

plumbers, etc.) expect both that results are limited to within a certain range, and that results can be ranked by distance from the end user’s location. In early 2009, an open source project formerly known as LocalLucene was donated to Apache Lucene and integrated as a contrib package. Lucene Spatial extends Lucene capabilities with support for geographical and location-based search. While Lucene Spatial doesn't have any distance scoring capabilities, it can effectively help to filter and sort based on geographical information like longitude and latitude values. Filtering is an especially common use-case, when combined with a full-text query. In searching for “French restaurant” within 5 miles from a specific location, the filter restricts down the search space to documents with location fields within 5 miles; the rest of the search operation is implemented in core Lucene. Lucene Spatial has a couple of different ways to encode geographic information: GeoHash: a hierarchical spatial data structure that subdivides space into buckets in a grid shape. GeoHash takes the even bits from the longitude value while the odd bits are taken from the latitude value. The result is an arbitrary precision base 32-encoded string that offers the property of gradually removing characters from the end of the string to reduce the size and precision of the code. Nearby places are likely to have similar prefixes due to this property. Cartesian Tiers: projects the world on a flat surface. Overlays to this projection are created as grids (Cartesian Tiers), with each tier having an increasing number (always by a power of two) of grid boxes on it dividing up the projection. Location data can be placed within one of the grid boxes with different precision depending on the number of grid boxes on the tier. Both of the above allow efficient storage of geo-information in a Lucene index. In contrast to plain latitude and longitude values indexed in separate fields, GeoHash and Cartesian Tiers encode in a single field. Note that despite its previous releases under a different name (LocalLucene), the Lucene Spatial API still isn't considered stable and might change in future releases. What’s New in Lucene 2.9 A Lucid Imagination Technical White Paper • October 2009 Page 17

Lucene Remote and Java RMI The historic dependency on Java RMI from the Lucene core has now been removed: Lucene Remote is now partitioned into an optional contrib package. While the package itself doesn't add any functionality to Lucene it introduces a critical back-compatibility issue likely to be relevant for many programmers. In prior versions, the core-interface Searchable extended java.rmi.Remote to enable searches on remote indexes. If you had taken advantage of this convenience, you will now have to add the new Lucene-remote JAR file to the classpath and change their code to use the new remote base interface RMIRemoteSearchable as shown below. final RMIRemoteSearchable remoteObject = ...; final String remoteObjectName = ...; Naming.rebind (remoteObjectName, remoteObject); Searchable searchable = (Searchable)Naming.lookup(remoteObjectName); Using RemoteSearchable with Lucene 2.9 New Flexible QueryParser Lucene’s built-in query parser has been a burden on developers trying to extend the default query syntax. While changing certain parts of it, such as query instantiation, could be readily achieved by subclassing the parser, changing the actual syntax required deep knowledge of the JavaCC parser-generator. The new contrib package QueryParser provides a complete query parser framework, which is fully compliant with the core parser but enables flexible customization by using a modular architecture. The basic idea of the new query parser is to separate the syntax from the semantics of a query, internally represented as a tree. Ultimately the parser splits up in three stages: 1. Parsing stage: transforms the query text (syntax) into a QueryNode tree. This stage is exploited by a single interface (SyntaxParser) mandatory for custom implementation of this stage. What’s New in Lucene 2.9 A Lucid Imagination Technical White Paper • October 2009 Page 18

2. Query-Node processing stage: once the QueryNode tree is created, a chain of processors start working on the tree. While walking down the tree, a processor can apply query optimizations, child reordering, or term tokenization even before the query is actually executed. 3. Building stage: the final stage builds the actual Lucene Query object by mapping QueryNode types to associated builders. Each builder subsequently applies the actual conversion into a Lucene query. The snippet below, taken from the new standard QueryParser implementation, shows how the stages are exposed at the API's top level. QueryNode queryTree = this.syntaxParser.parse(query, getField()); queryTree = this.processorPipeline.process(queryTree); return (Query) this.builder.build(queryTree); To provide a smooth transition from the existing core parser to the new API, this contrib package also contains an implementation fully compliant with the standard query syntax. This not only helps the switch to the new query parser but it also serves as an example of how to use and extend the API. That said, the standard implementation is based on the new query parser API and therefore it can't simply replace a core parser as is. If you have been replacing Lucene's current query parser, you can use QueryParserWrapper instead, which preserves the old query parser interface but calls the new parser framework. One final caveat: the QueryParserWrapper is marked as deprecated, as the new query parser will be moved to the core in the upcoming release and eventually replace the old API. Minor Changes and Improvements in Lucene 2.9 Beside the improvements and entirely new features, Lucene 2.9 contains several minor improvements worth mentioning. The following points are a partial outline of minor changes. What’s New in Lucene 2.9 A Lucid Imagination Technical White Paper • October 2009 Page 19

Term vector-based highlighter: a new term highlighter implementation based on term vectors (essentially a view of terms, offsets, and positions in a documents field). It supports features like N-Gram fields and phrase-unit highlighting with slops and yields good performance on large documents. The downside is that it requires a lot more disk space due to stored term vectors. [BACK- COMPATIBILITY] Collector replaces HitCollector: the low-level HitCollector was deprecated and replaced with a new Collector class. Collector offers a more efficient API to collect hits across sequential IndexReader instances. The most significant improvement here is that score calculation is now decoupled from collecting hits or skipped entirely if not needed—a nice new efficiency. Improved String “interning”: Lucene 2.9 internally uses a custom String intern cache instead of Java's default String.intern(). The lockless implementation yields minor internal performance improvements. New n-gram distance: a new n-gram–based distance measure was added to the contrib spellcheck package. [BACK- COMPATIBILITY] Weight is now an abstract class: the Weight interface was refactored to an abstract class including minor method signature changes. ExtendedFieldCache marked deprecated: All methods and parsers from the interface ExtendedFieldCache have been moved into FieldCache. ExtendedFieldCache is now deprecated and contains only a few declarations for binary backwards compatibility. [BACK- COMPATIBILITY] MergePolicy interface changed: MergePolicy now requires an IndexWriter instance to be passed upon instantiation. As a result, IndexWriter was removed as a method argument from all MergePolicy methods. For a complete list of improvements, bug-fixes, compatibility, and runtime behavior changes you should consult the CHANGES.txt file included in the Lucene distribution (http://lucene.apache.org/java/2_9_0/changes/Changes.html). What’s New in Lucene 2.9 A Lucid Imagination Technical White Paper • October 2009 Page 20

Strategies for Upgrading to Lucene 2.9 In the main, a Lucene-based application will benefit from the improvements in 2.9, even as its new features such as numeric capabilities and the new TokenStream API do require code modifications and may require reindexing in order to take full advantage. That said, compared to previous version changes, an upgrade to version 2.9 requires a more sophisticated upgrade procedure. True, there are many cases in which an upgrade won't require code changes, as changes limited to “expert” APIs won't affect applications only using high-level functionality. All the same, even if an application complies with Lucene 2.9, it is likely that some of the changes in runtime characteristics can introduce unexpected behaviors. In the sections below, we’ll offer some brief suggestions for making the transition. In the months ahead, the next major version of Apache Lucene will be underway. That said, one cannot assume that an upgrade to 2.9 is not worthwhile given the numbering of the next release (i.e., version 3.0). It is also slated to be a deprecation release, in which all deprecated-marked code in Lucene 2.9 will be removed. Some parts of the API might be modified in order to make use of Java Generics, but in general the upgrade from 2.9 to 3.0 should be as seamless as earlier upgrades have been. Once you have replaced the usage of any deprecated API(s) in your code, you should then be able to upgrade the next time simply by replacing the Lucene JAR file. Upgrade to 2.9—Recommended Actions At a minimum, if you plan an upgrade of your search application to Lucene 2.9, you should recompile your application against the new version before the application is rolled out in a production environment. The most critical issues will immediately raise a compile-time error once the JAR is in the classpath. For those of you using Lucene from a single location, for example, in the JRE's ext directory, you should make sure that 2.9 is the only Lucene version accessible. In cases where an application relies on extending Lucene in any particular way and the upgrade doesn't raise a compile-time error, it is recommended that you add a test-case for the extension based on the behavior executed against the older version of Lucene. What’s New in Lucene 2.9 A Lucid Imagination Technical White Paper • October 2009 Page 21

It is also extremely important that you backup and archive your index before opening it with Lucene 2.9, as it will make changes to the index that may not be readable by previous versions. Again, we strongly recommend a careful reading of the CHANGES.txt file included in every Lucene distribution, especially the sections on back-compatibility policy and on changes in runtime behavior. Careful study followed by proper planning and testing should prevent you from running into any surprises once the new Lucene 2.9-based application goes into production. Upgrade to 2.9—Optional Actions Lucene 2.9 includes many new features that are not required for use of the new release. Nevertheless, 2.9 has numerous parts of the API marked as deprecated, since they are to be removed in the next release. To prepare for the next release and further improvements in this direction, it is strongly recommended that you replace any deprecated API during the upgrade process. Applications using any kind of numeric searches can improve their performance heavily by replacing custom solution with Lucene's Numeric Capabilities described earlier in this white paper. Last but not least, the new TokenStream API will replace the older API entirely in the next release. Custom TokenStream, TokenFilter, and Tokenizer implementations should be updated to the attribute-based API. Here, the source distribution contains basic test cases that can help you safely upgrade. Finally, to reiterate, you would do best to write new added test cases against their current Lucene version, and upgrade the test and your code once you have gained enough confidence in the stability of the upgrade. What’s New in Lucene 2.9 A Lucid Imagination Technical White Paper • October 2009 Page 22

References http://lucene.apache.org/java/2_9_0/index.html http://lucene.apache.org/java/2_9_0/changes/Changes.html http://lucene.apache.org/java/2_9_0/changes/Contrib-Changes.html http://www.lucidimagination.com/Community/Hear-from-the-Experts/Podcasts-and- Videos/Interview-Uwe-Schindler http://wiki.apache.org/lucene-java/NearRealtimeSearch http://wiki.apache.org/lucene-java/Payloads http://www.lucidimagination.com/blog/2009/08/05/getting-started-with-payloads/ http://wiki.apache.org/lucene-java/ConceptsAndDefinitions http://wiki.apache.org/lucene-java/FlexibleIndexing http://wiki.apache.org/lucene-java/Java_1.5_Migration http://www.lucidimagination.com/How-We-Can-Help/webinar-Lucene-29 http://www.nsshutdown.com/projects/lucene/whitepaper/locallucene_v2.html http://www.lucidimagination.com/Community/Hear-from-the-Experts/Podcasts-and- Videos/Interview-Ryan-McKinley http://ocw.kfupm.edu.sa/user062/ICS48201/NLLight%20Stemming%20for%20Arabic% 20Information%20Retrieval.pdf https://javacc.dev.java.net/ What’s New in Lucene 2.9 A Lucid Imagination Technical White Paper • October 2009 Page 23

Next Steps For more information on how Lucid Imagination can help search application developers, employees, customers, and partners find the information they need, please visit http://www.lucidimagination.com to access blog posts, articles, and reviews of dozens of successful implementations. Certified Distributions from Lucid Imagination are complete, supported bundles of software which include additional bug fixes, performance enhancements, along with our free 30-day Get Started program. Coupled with one of our support subscriptions, a Certified Distribution can provide a complete environment to develop, deploy, and maintain commercial-grade search applications. Certified Distributions are available at www.lucidimagination.com/Downloads. Please e-mail specific questions to: Support and Service: support@lucidimagination.com Sales and Commercial: sales@lucidimagination.com Consulting: consulting@lucidimagination.com Or call: 1.650.353.4057 What’s New in Lucene 2.9 A Lucid Imagination Technical White Paper • October 2009 Page 24

APPENDIX: Choosing Lucene or Solr The great improvements in the capabilities of Lucene and Solr open source search technology have created rapidly growing interest in using them as alternatives for their search applications. As is often the case with open source technology, online community documentation provides rich details on features and variations, but does little to provide explicit direction on which technologies would be the best choice. So when is Lucene preferable to Solr and vice versa? There is in fact no single answer, as Lucene and Solr bring very similar underlying technology to bear on somewhat distinct problems. Solr is versatile and powerful, a full- featured, production-ready search application server requiring little formal software programming. Lucene presents a collection of directly callable Java libraries, with fine- grained control of machine functions and independence from higher-level protocols. In choosing which might be best for your search solution, the key questions to consider are application scope, deployment environment, and software development preferences. If you are new to developing search applications, you should start with Solr. Solr provides scalable search power out of the box, whereas Lucene requires solid information retrieval experience and some meaningful heavy lifting in Java to take advantage of its capabilities. In many instances, Solr doesn’t even require any real programming. Solr is essentially the “serverization” of Lucene, and many of its abstract functions are highly similar, if not the just the same. If you are building an app for the enterprise sector, for instance, you will find Solr almost a 100% match to your business requirements: it comes ready to run in a servlet container such as Tomcat or Jetty, and ready to scale in a production Java environment. Its RESTful interfaces and XML-based configuration files can greatly accelerate application development and maintenance. In fact, Lucene programmers have often reported that they find Solr to contain “the same features I was going to build myself as a framework for Lucene, but already very well implemented.” Once you start with Solr, and you find yourself using a lot of the features Solr provides out of the box, you will likely be better off using Solr’s well organized extension mechanisms instead of starting from scratch using Apache Lucene. What’s New in Lucene 2.9 A Lucid Imagination Technical White Paper • October 2009 Page 25

If, on the other hand, you do not wish to make any calls via HTTP, and wish to have all of your resources controlled exclusively by Java API calls that you write, Lucene may be a better choice. Lucene can work best when constructing and embedding a state-of-the-art search engine, by allowing programmers to assemble and compile inside a native Java application. Some programmers set aside the convenience of Solr in order to more directly control the large set of sophisticated features with low-level access, data, or state manipulation, and choose Lucene instead, for example, with byte-level manipulation of segments or intervention in data I/O. Investment at the low level enables development of extremely sophisticated, cutting edge text search and retrieval capabilities. As for features, the latest version of Solr generally encapsulates the latest version of Lucene. As the two are in many ways functional siblings, spending time gaining a solid understanding how Lucene works internally can help you understand Apache Solr and its extension of Lucene's workings. No matter which you choose, the power of open source search is yours to harness. More information on both Lucene and Solr can be found at www.lucidimagination.com. What’s New in Lucene 2.9 A Lucid Imagination Technical White Paper • October 2009 Page 26

What’s New in Apache Lucene 2.9

by LucidWorks

Accessibility

Categories

Upload Details

Usage Rights

Statistics

What’s New in Apache Lucene 2.9 Document Transcript