Numeric Range Queries in Lucene and Solr
Upcoming SlideShare
Loading in...5
×
 

Numeric Range Queries in Lucene and Solr

on

  • 873 views

Presentation covers core lucene/solr stuff which is used in numeric range queries. There are several examples, algorithm discovered by Uwe is briefly explained.

Presentation covers core lucene/solr stuff which is used in numeric range queries. There are several examples, algorithm discovered by Uwe is briefly explained.

Statistics

Views

Total Views
873
Views on SlideShare
872
Embed Views
1

Actions

Likes
0
Downloads
3
Comments
3

1 Embed 1

http://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • @hcedric you are welcome)

    @GregMiller13 good question. Just because I didn't run any profiling myself I can only guess, so it will be better to check what I am saying.

    For example, let's take BooleanQueryRewrite mode:
    1) You have less clauses for NumericRangeQuery which saves you from TooMuchClausesException (at least it was the case for versions prior to solr/lucene 4)
    2) You have less work for scorers to do, also to create scorers you need to access TermsEnum and DocsEnum
    @see TermQuery#scorer
    TermsEnum termsEnum = getTermsEnum(context);
    DocsEnum docs = termsEnum.docs(acceptDocs, null);
    So, in case of non NumericRangeQuery you have more TermQueries and have to open more TermsEnum and DocsEnum.
    I believe it's not so cheap operation.

    In case of FILTER_REWRITE there is no scorers leap-frog anymore, however you still need TermsEnum and DocsEnum.

    3) Another thing to think about is fancy VInt/delta encoding which is used for index. I wonder if it can also affect performance due to fact that we kinda group posting lists in case of numeric fields.
    For example, you may have
    term1 -> [1,3,5,7,9] delta for each is 2
    term2 -> [2,4,6,8,10] delta for each is 2
    numericterm -> [1,2,3,4,5,6,7,8,9,10] delta for each is 1

    And AFAIK if delta is less than 255 it is stored as byte, so in case of grouped terms we have smaller delta.

    4) I heard that even for FST term seek is not so cheap operation.

    Conclusion: I think that the main reason is 2, however it may be not the only reason, If you decide to find the real reason please share your findings with me.
    Are you sure you want to
    Your message goes here
    Processing…
  • This is probably a really naive question but can you describe the performance challenge with querying numeric ranges in a little more detail? Is the issue finding all unique numeric terms in the index, or is the challenge merging all the document lists for every unique term? With the FST index structure now in Lucene I would assume it's the latter since this data structure seems like it would be efficient for finding all terms with given prefixes out of the box and without using this trie trick. Can you confirm if this is the case?
    Are you sure you want to
    Your message goes here
    Processing…
  • Very helpful presentation for hacking my way into the lucene numeric queries. Thank you.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Numeric Range Queries in Lucene and Solr Numeric Range Queries in Lucene and Solr Presentation Transcript

  • Numeric Range Queries in Lucene and Solr kirilchukvadim@gmail.com
  • Agenda: ● What is RangeQuery ● Which field type to use for Numerics ● Range stuff under the hood (run!) ● NumericRangeQuery ● Useful links
  • Agenda: ● What is RangeQuery ● Which field type to use for Numerics ● Range stuff under the hood (run!) ● NumericRangeQuery ● Useful links
  • Range Queries: A range query is a type of query that matches all documents where some value is between an upper and lower boundary: Give me: ● Jeans with price from 200 to 300$ ● Car with length from 5 to 10m ● ...
  • Range Queries: In solr range query is as simple as: q = field:[100 TO 200] We will talk about Numeric Range Queries but you can use range queries for text too: q = field:[A TO Z]
  • Agenda: ● What is RangeQuery ● Which field type to use for Numerics ● Range stuff under the hood (run!) ● NumericRangeQuery ● Useful links (relax)
  • Which field type? Which field type to use for “range” fields (let’s stick with int) in schema? ● solr.IntField ● or maybe solr.SortableIntField ● or maybe solr.TrieIntField
  • Which field type? Let’s assume we have: ● 11 documents, id: 1,2,3,..11 ● each doc has single value “int” price field ● document id is the same as it’s price ● q = *:* "numFound": 11, "docs": [ { "id": 1, “price_field": 1 }, { "id": 2, “price_field": 2 }, ... { "id": 11, “price_field": 11 }]
  • Which field type - solr.IntField q = price_field:[1 TO 10]
  • Which field type - solr.IntField q = price_field:[1 TO 10] "numFound": 2, "start": 0, "docs": [ { "price_field": 1 }, { "price_field": 10 } ] }
  • Which field type - solr.IntField Store and index the text value verbatim and hence don't correctly support range queries, since the lexicographic ordering isn't equal to the numeric ordering [1,10],11,2,3,4,5,6,7,8,9 Interesting, but “sort by” works fine.. Clever comparator knows that values are ints!
  • Which field type - solr.SortableIntField ● q = price_field:[1 TO 10] ○ "numFound": 10 ● “Sortable”, in fact, refer to the notion of making the numbers have correctly sorted order. It’s not about “sort by” actually! ● Processed and compared as strings!!! tricky string encoding: NumberUtils.int2sortableStr(...) ● Deprecated and will be removed in 5.X ● What should i use then?
  • Which field type - solr.TrieIntField ● q = price_field:[1 TO 10] ○ "numFound": 10 ● Recommended as replacement for IntField and SortableIntField in javadoc ● Default for primitive fields in reference schema ● Said to be fast for range queries (actually depends on precision step) ● Tricky and, btw wtf is precision step?
  • Agenda: ● What is RangeQuery ● Which field type to use for Numerics ● Range stuff under the hood (run!) ● NumericRangeQuery ● Useful links
  • Under the hood - Index
  • Under the hood - Index NumericTokenStream is where half of magic happens! ● precision step = 1 ● value = 11 00000000 00000000 00000000 00001011 ● Let’s see how it will be indexed!
  • Under the hood - Index Field with precisionStep=1
  • Under the hood - Index shift=0 00001011 11 shift=1 00001010 10 = 5 << 1 shift=2 00001000 8 = 2 << 2 shift=3 00001000 8 = 1 << 3 shift=4 00000000 0 = 0 << 4 shift=5 00000000 0 = 0 << 5 continue…
  • Under the hood - Index How much for an integer? 11111111 11111111 11111111 11111111 Algorithm requires to index all 32/precisionStep terms So, for “11” we have 11, 10, 8, 8, 0, 0, 0, 0, 0….0
  • Under the hood - Index Okay! We indexed 32 tokens for the field. (TermDictionary! Postings!) Where is the trick? Stay tuned!
  • Under the hood - Query
  • Under the hood - Query Sub-classes of FieldType could override #getRangeQuery(...) to provide their own range query implementation. If not, then likely you will have: MultiTermQuery rangeQuery = TermRangeQuery. newStringRange(...) TrieField overrides it. And here comes...
  • Agenda: ● What is RangeQuery ● Which field type to use for Numerics ● Range stuff under the hood (run!) ● NumericRangeQuery ● Useful links
  • Numeric Range Query (Decimal) ● Decimal example, precisionStep = ten ● q = price:[423 TO 642]
  • Numeric Range Query (Binary) ● precisionStep = 1 ● q = price:[3 TO 12] 0 1 2 3 4 5 6 7 8 9 10 11 12 13
  • Numeric Range Query (Binary) ● precisionStep = 1 ● q = price:[3 TO 12] SHIFT = 1 0 0 1 1 2 3 2 3 4 5 6 4 7 8 5 9 10 6 11 12 13
  • ... Numeric Range Query (Binary) ● precisionStep = 1 ● q = price:[3 TO 12] 0 0 0 1 0 0 0 0 1 1 2 1 2 3 2 3 4 5 6 3 4 7 8 5 9 10 6 11 12 13
  • Numeric Range Query (Binary) ● precisionStep = 1 ● q = price:[3 TO 12] 0 1 0 0 0 0 1 1 2 1 2 3 2 3 4 5 6 3 4 7 8 5 9 10 6 11 12 13
  • Numeric Range Query (How?) So, the questions is: How to create query for the algorithm?
  • Numeric Range Query (How?) Let’s come back to TrieField#getRangeQuery(...) There are several options: ● field is multiValued, hasDocValues, not indexed ○ super#getRangeQuery ● field is hasDocValues, not indexed ○ new ConstantScoreQuery ( FieldCacheRangeFilter.newIntRange(...) ) ● otherwise ta-da ○ NumericRangeQuery.newIntRange(...)
  • Numeric Range Query (How?) NumericRangeQuery extends MultiTermQuery which is: An abstract Query that matches documents containing a subset of terms provided by a FilteredTermsEnum enumeration. This query cannot be used directly(abstract); you must subclass it and define getTermsEnum(Terms, AttributeSource) to provide a FilteredTermsEnum that iterates through the terms to be matched.
  • Numeric Range Query (How?) Let’s understand how #getTermsEnum works. Returns new NumericRangeTermsEnum(...) The main part is: NumericUtils.splitIntRange(...)
  • Numeric Range Query (How?) Algorithm uses binary masks very much: for (int shift=0; noRanges(); shift += precisionStep): diff = 1L << (shift + precisionStep); mask = ((1L << precisionStep) - 1L) << shift; diff=2 0 0 1 1 2 3 Diff is distance between upper level neighbors Mask is to check if currentLevel node has nodes lower or upper. (1,3 hasLower, 0,2 hasUpper)
  • Numeric Range Query (How?) hasLower = (minBound & mask) != 0L; hasUpper = (maxBound & mask) != mask; if (hasLower) addRange(builder, valSize, minBound, minBound | mask, shift); if (hasUpper) addRange(builder, valSize, maxBound & ~mask, maxBound, shift);
  • Numeric Range Query (How?) hasLower = (minBound & mask) != 0L; hasUpper = (maxBound & mask) != mask; nextMinBound = (hasLower ? (minBound + diff) : minBound) & ~mask; nextMaxBound = (hasUpper ? (maxBound - diff) : maxBound) & ~mask;
  • Numeric Range Query (How?) // If we are in the lowest precision or the next precision is not available. addRange(builder, valSize, minBound, maxBound, shift); // exit the split recursion loop (FOR)
  • Numeric Range Query (How?) ● ● ● ● ● shift = 0 diff = 0b00000010 = 2 mask = 0b00000001 = 1 hasLower = (3 & 1 != 0)? = true hasUpper = (12 & 1 != 1)? = true ○ addRange 3..(3 | 1) = 3..3 ○ addRange 12..(12 & ~1) = 12..12 ● nextMin = (3 + 2) & ~1 = 4 ● nextMax = (12 - 2) & ~1 = 10 0 1 2 3 4 5 6 7 8 9 10 11 12 13
  • Numeric Range Query (How?) ● ● ● ● ● ● ● ● min:4; max:10 shift = 1 diff = 0b00000100 = 4 mask = 0b00000010 = 2 hasLower = (4 & 2 != 0) ? = false hasUpper = (10 & 2 != 2) ? = false nextMin = min nextMax = max 0 0 1 1 2 3 2 3 4 5 6 4 7 8 5 9 10 6 11 12 13
  • Numeric Range Query (How?) ● ● ● ● ● ● ● ● min:4; max:10 shift = 2 diff = 0b00001000 = 8 mask = 0b00000100 = 4 hasLower = (4 & 4 != 0) ? = true hasUpper = (10 & 4 != 4) ? = true nextMin = (4 + 8) & ~4 = 8 => min > max END nextMax = (10 - 8) & ~4 = 0 => range 1..2 shift = 2 2 3 0 1 0 0 1 1 2 3 2 3 4 5 6 4 7 8 5 9 10 6 11 12 13
  • Numeric Range Query (How?) TestNumericUtils#testSplitIntRange assertIntRangeSplit(lower, upper, precisionStep, expectBounds, shifts) assertIntRangeSplit(3, 12, 1, true, Arrays.asList( -2147483645,-2147483645, // 3,3 -2147483636,-2147483636, // 12,12 536870913, 536870914), // 1, 2 for shift == 2 Arrays.asList(0, 0, 2) ); // Crappy unsigned int conversions are done in the asserts
  • Numeric Range Query (How?) So, NumericTermsEnum generates and remembers all ranges to match.
  • Numeric Range Query (How?) Basically TermsEnum is an Iterator to seek or step through terms in some order. In our case order is: 0 1 2 3 4 5 6 7 8 9 10 11 12 Then (shift = 1): 0 1 2 3 4 5 6 Then (shift = 2) 0 2 1 ... 3 13
  • Numeric Range Query (How?) Actually we have FilteredTermsEnum: 1. Only red terms are accepted by our enumerator 2. If term is not accepted we advance: FilteredTermsEnum#nextSeekTerm(currentTerm) TermsEnum#seekCeil(termToSeek) Seek term depends on currentTerm and generated ranges.
  • Numeric Range Query (How?) Ok, now we have TermsEnum for MiltiTermQuery and enum is able to seek through only those terms which match appropriate sub ranges. The question is how to convert TermsEnum to Query!?
  • Numeric Range Query (How?) The last trick is query#rewrite() method of MultiTermQuery (rewrite is always called on query before performing search): public final Query rewrite(IndexReader reader) { return rewriteMethod.rewrite(reader, this); } Oh, “rewriteMethod” how interesting… It defines how the query is rewritten.
  • Numeric Range Query (How?) There are plenty of different rewrite methods, but most interesting for us are: ● CONSTANT_SCORE_* ○ BOOLEAN_QUERY_REWRITE ○ FILTER_REWRITE ○ AUTO_REWRITE_DEFAULT
  • Numeric Range Query (How?) BOOLEAN_QUERY_REWRITE 1. Collect terms (TermCollector) by using #getTermsEnum(...) 2. For each term create TermQuery 3. return BooleanQuery with all TermQuery as leafs
  • Numeric Range Query (How?) FILTER_REWRITE 1. 2. 3. 4. 5. Get termsEnum by using #getTermsEnum(...) Create FixedBitSet Get DocsEnum for each term Iterate over docs and bitSet.set(docid); return ConstantScoreQuery over filter (bitSet)
  • Numeric Range Query (How?) AUTO_REWRITE_DEFAULT If the number of documents to be visited in the postings exceeds some percentage of the maxDoc() for the index then FILTER_REWRITE is used, otherwise BOOLEAN_REWRITE is used.
  • Agenda: ● .. ● I promised. Precision Step! ● ...
  • Precision step So, what is precision step and how it affects performance? ● Defines how much terms to index for each value ○ Lower step values mean more precisions and consequently more terms in index ○ indexedTermsPerValue = bitsPerVal / pStep ○ Lower precision terms are non unique, so term dictionary doesn’t grow much, however postings file does
  • Precision step So, what is precision step and how it affects performance? ● ... ○ Smaller precision step means less number of terms to match, which optimizes query speed ○ But more terms to seek in index ○ You can index with a lower precision step value and test search speed using a multiple of the original step value. ○ Ideal step is found by testing only
  • Precision step (Results) According to NumericRangeQuery javadoc: ● Opteron64 machine, Java 1.5, 8 bit precision step ● 500k docs index ● TermRangeQuery in BooleanRewriteMode took about 30-40 seconds ● TermRangeQuery in FilterRewriteMode took about 5 seconds ● NumericRangeQuery took < 100ms
  • Agenda: ● What is RangeQuery ● Which field type to use for Numerics ● Range stuff under the hood (run!) ● NumericRangeQuery ● Useful links
  • Useful links ● http://searchhub.org/2009/05/13/exploringlucene-and-solrs-trierange-capabilities/ ● http://www.panfmp.org/ ● http://epic.awi.de/17813/1/Sch2007br.pdf ● http://lucene.apache. org/core/4_3_1/core/org/apache/lucene/search/ NumericRangeQuery.html ● http://en.wikipedia.org/wiki/Range_tree ● me http://plus.google.com/+VadimKirilchuk