Solr sparse faceting
•Download as ODP, PDF•
0 likes•998 views
Multiple time and space enhancements for high cardinality Solr String faceting. Presented at Berlin Buzzwords 2015. Code available at http://tokee.github.io/lucene-solr/
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Solr sparse faceting
Similar to Solr sparse faceting (20)
Recently uploaded
Recently uploaded (20)
Solr sparse faceting
- 1. 1/47 Solr sparse faceting Everything counts in large amounts @TokeEskildsen State and University Library, Denmark https://tokee.github.io/lucene-solr/
- 3. 3/47 Nothing To Fear ● 500TB+ web resources from Danish Net Archive ● Estimated 50TB Solr index data when finished ● 3 machines of 16 CPU cores, 256GB RAM, 25 * 900GB SSD – Each machine holds: 25 Solrs ● Each Solr holds: 1 optimized shard with 900GB / 250M docs ● Shards build externally, one at a time ● (Optimizations also relevant for smaller setups)
- 4. 4/47 Pipeline counter = new int[ordinals] for docID: result.getDocIDs() for ordinal: getOrdinals(docID) counter[ordinal]++ for ordinal = 0 ; ordinal < counters.length ; ordinal++ priorityQueue.add(ordinal, counter[ordinal]) ord term counter 0 A 0 1 B 3 2 C 0 3 D 1006 4 E 1 5 F 1 6 G 0 7 H 0 8 I 3
- 5. 5/47
- 6. 6/47
- 7. 7/47 Recycle counter = pool.getCounter() for docID: result.getDocIDs() for ordinal: getOrdinals(docID) counter[ordinal]++ for ordinal = 0 ; ordinal < counters.length ; ordinal++ priorityQueue.add(ordinal, counter[ordinal]) pool.release(counter)
- 8. 8/47
- 9. 9/47
- 10. 10/47
- 11. 11/47 Counting counter = pool.getCounter() for docID: result.getDocIDs() for ordinal: getOrdinals(docID) counter[ordinal]++ for ordinal = 0 ; ordinal < counters.length ; ordinal++ priorityQueue.add(ordinal, counter[ordinal]) pool.release(counter) ord term counter 0 A 0 1 B 3 2 C 0 3 D 1006 4 E 1 5 F 1 6 G 0 7 H 0 8 I 3
- 12. 12/47 ord counter 0 0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 tracker N/A N/A N/A N/A N/A N/A N/A N/A N/A
- 13. 13/47 ord counter 0 0 1 0 2 0 3 1 4 0 5 0 6 0 7 0 8 0 tracker 3 N/A N/A N/A N/A N/A N/A N/A N/A
- 14. 14/47 ord counter 0 0 1 1 2 0 3 1 4 0 5 0 6 0 7 0 8 0 tracker 3 1 N/A N/A N/A N/A N/A N/A N/A
- 15. 15/47 ord counter 0 0 1 1 2 0 3 2 4 0 5 0 6 0 7 0 8 0 tracker 3 1 N/A N/A N/A N/A N/A N/A N/A
- 16. 16/47 ord counter 0 0 1 3 2 0 3 1006 4 1 5 1 6 0 7 0 8 3 tracker 3 1 8 4 5 N/A N/A N/A N/A
- 17. 17/47 counter = pool.getCounter() for docID: result.getDocIDs() for ordinal: getOrdinals(docID) if counter[ordinal]++ == 0 && tracked < maxTracked tracker[tracked++] = ordinal if tracked < maxTracked for i = 0 ; i < tracked ; i++ priorityQueue.add(tracker[i], counter[tracker[i]]) else for ordinal = 0 ; ordinal < counter.length ; ordinal++ priorityQueue.add(ordinal, counter[ordinal]) Sparse counting ord counter 0 0 1 3 2 0 3 1006 4 1 5 1 6 0 7 0 8 3 tracker 3 1 8 4 5 N/A N/A N/A N/A
- 18. 18/47
- 19. 19/47 Get the Balance Right Phase 1) All shards perform faceting The Merger calculates top-X terms Phase 2) The term counts are requested from the shards that did not return them in phase 1 for term: query.getTerms() result.add(term, searcher.numDocs( query(field:term), base.getDocIDs() ).hitCount)
- 21. 21/47 Alternative fine counting counter = pool.getCounter() for docID: result.getDocIDs() for ordinal: getOrdinals(docID) counter.increment(ordinal) for term: query.getTerms() result.add(term, counter.get(getOrdinal(term))) }Same as phase 1
- 22. 22/47 Stripped counter = pool.getCounter(key) for term: query.getTerms() result.add(term, counter.get(getOrdinal(term)))
- 24. 24/47 250,000,000 docs / 900GB, optimized Field References Max docs/term Terms domain 250,000,000 3,000,000 1,100,000 url 250,000,000 56,000 200,000,000 links 5,800,000,000 5,000,000 610,000,000
- 25. 25/47 Term distributions domain 1.1M url 200M links 600M
- 26. 26/47 Clean Title:/home/te/Dropbox/sb/net/pack_tra Creator:Dia v0.97.2 CreationDate:Wed May 20 22:26:18 2015
- 27. 27/47 World Full Of Nothing Title:/home/te/Dropbox/sb/net/pack_tra Creator:Dia v0.97.2 CreationDate:Wed May 20 14:55:25 2015 Title:/home/te/Dropbox/sb/net/pack_tra Creator:Dia v0.97.2 CreationDate:Wed May 20 14:56:08 2015 domain: 4 MB url: 780 MB links: 2350 MB int[ordinals] PackedInts(ordinals, maxBPV) domain: 3 MB (72%) url: 420 MB (53%) links: 1760 MB (75%)
- 28. 28/47 Title:/home/te/Dropbox/sb/net/pack_tra Creator:Dia v0.97.2 CreationDate:Wed May 20 22:38:49 2015 Title:/home/te/Dropbox/sb/net/pack_tra Creator:Dia v0.97.2 CreationDate:Wed May 20 22:26:18 2015 Platonic ideal Harsh reality Plane 4 Plane 3 Plane 2 Plane 1 Construction Time Again
- 29. 29/47 Title:/home/te/Dropbox/sb/net/pack_tra Creator:Dia v0.97.2 CreationDate:Wed May 20 22:38:49 2015Plane 4 Plane 3 Plane 2 Plane 1 L: 0 ≣ 000000
- 30. 30/47 Title:/home/te/Dropbox/sb/net/pack_tra Creator:Dia v0.97.2 CreationDate:Wed May 20 22:38:49 2015Plane 4 Plane 3 Plane 2 Plane 1 L: 0 ≣ 000000 L: 1 ≣ 000001
- 31. 31/47 Title:/home/te/Dropbox/sb/net/pack_tra Creator:Dia v0.97.2 CreationDate:Wed May 20 22:38:49 2015Plane 4 Plane 3 Plane 2 Plane 1 L: 0 ≣ 000000 L: 1 ≣ 000001 L: 2 ≣ 000010
- 32. 32/47 Title:/home/te/Dropbox/sb/net/pack_tra Creator:Dia v0.97.2 CreationDate:Wed May 20 22:38:49 2015Plane 4 Plane 3 Plane 2 Plane 1 L: 0 ≣ 000000 L: 1 ≣ 000001 L: 2 ≣ 000010 L: 3 ≣ 000011
- 33. 33/47 Title:/home/te/Dropbox/sb/net/pack_tra Creator:Dia v0.97.2 CreationDate:Wed May 20 22:38:49 2015Plane 4 Plane 3 Plane 2 Plane 1 L: 0 ≣ 000000 L: 1 ≣ 000001 L: 2 ≣ 000010 L: 3 ≣ 000011 L: 4 ≣ 000100 L: 5 ≣ 000101 L: 6 ≣ 000110 L: 7 ≣ 000111 ... L: 12 ≣ 001100
- 34. 34/47 Title:/home/te/Dropbox/sb/net/pack_tra Creator:Dia v0.97.2 CreationDate:Wed May 20 22:38:49 2015Plane 4 Plane 3 Plane 2 Plane 1 L: 0 ≣ 000000 L: 1 ≣ 000001 L: 2 ≣ 000010 L: 3 ≣ 000011 L: 4 ≣ 000100 L: 5 ≣ 000101 L: 6 ≣ 000110 L: 7 ≣ 000111 ... L: 12 ≣ 001100 if counter[ordinal]++ == 0 && tracked < maxTracked tracker[tracked++] = ordinal ?
- 35. 35/47 Title:/home/te/Dropbox/sb/net/pack_tra Creator:Dia v0.97.2 CreationDate:Wed May 20 22:39:03 2015 Now This is Fun Title:/home/te/Dropbox/sb/net/pack_tra Creator:Dia v0.97.2 CreationDate:Wed May 20 22:38:49 2015 Plane 1 Plane 2 Plane 3 Plane 4
- 36. 36/47 Title:/home/te/Dropbox/sb/net/pack_tra Creator:Dia v0.97.2 CreationDate:Wed May 20 22:39:03 2015 Plane 1 Plane 2 Plane 3 Plane 4 L: 0 ≣ 000000
- 37. 37/47 Title:/home/te/Dropbox/sb/net/pack_tra Creator:Dia v0.97.2 CreationDate:Wed May 20 22:39:03 2015 Plane 1 Plane 2 Plane 3 Plane 4 L: 0 ≣ 000000 L: 1 ≣ 000001
- 38. 38/47 Title:/home/te/Dropbox/sb/net/pack_tra Creator:Dia v0.97.2 CreationDate:Wed May 20 22:39:03 2015 Plane 1 Plane 2 Plane 3 Plane 4 L: 0 ≣ 000000 L: 1 ≣ 000001 L: 2 ≣ 000011
- 39. 39/47 Title:/home/te/Dropbox/sb/net/pack_tra Creator:Dia v0.97.2 CreationDate:Wed May 20 22:39:03 2015 Plane 1 Plane 2 Plane 3 Plane 4 L: 0 ≣ 000000 L: 1 ≣ 000001 L: 2 ≣ 000011 L: 3 ≣ 000101
- 40. 40/47 Title:/home/te/Dropbox/sb/net/pack_tra Creator:Dia v0.97.2 CreationDate:Wed May 20 22:39:03 2015 Plane 1 Plane 2 Plane 3 Plane 4 L: 0 ≣ 000000 L: 1 ≣ 000001 L: 2 ≣ 000011 L: 3 ≣ 000101 L: 4 ≣ 000111 L: 5 ≣ 001001 L: 6 ≣ 001011 L: 7 ≣ 001101 ... L: 12 ≣ 010111
- 41. 41/47 The Bottom Line Title:/home/te/Dropbox/sb/net/pack_tra Creator:Dia v0.97.2 CreationDate:Wed May 20 22:39:03 2015 Title:/home/te/Dropbox/sb/net/pack_tra Creator:Dia v0.97.2 CreationDate:Wed May 20 14:55:25 2015 Title:/home/te/Dropbox/sb/net/pack_tra Creator:Dia v0.97.2 CreationDate:Wed May 20 14:56:08 2015 domain: 4 MB url: 780 MB links: 2350 MB domain: 3 MB (72%) url: 420 MB (53%) links: 1760 MB (75%) domain: 1 MB (30%) url: 66 MB ( 8%) links: 311 MB (13%) int[ordinals] PackedInts(ordinals, maxBPV) N-plane-z
- 42. 42/47
- 43. 43/47 Kitchen sink 250,000,000 docs / 900GB, optimized Field References Max docs/term Terms domain 250,000,000 3,000,000 1,100,000 url 250,000,000 56,000 200,000,000 links 5,800,000,000 5,000,000 610,000,000 x 9 shards x 3 concurrent requests
- 44. 44/47 Shouldn't Have Done That
- 45. 45/47 Some Great Reward 8GB heap per 900GB shard
- 46. 46/47 Dream On ● Threaded counting ● Monotonically increasing tracker for nplane-z ● Regexp filtering ● Fine count skipping ● Counter capping
Editor's Notes
- Highly scientific. Very happy with Solr faceting for general work. Principles primarily, applicable to non-Solr settings.
- State & University + Royal Library. Harvest 4 times/year. Hardware: Save money. Static shards. Generation 1: 25 shards. Generation 2: Currently 9 shards. We&apos;ll use this. Apologies for small sample. 100GB free cache mem (0.5%-1.23% of index size) Danish Net Archive: http://netarkivet.dk/in-english/ Next: We need aggregations, so faceting. String faceting is focus here.
- Bird&apos;s eye view of vanilla Solr fc-faceting. 1) Allocate counter 2) Update counter 3) Extract top-X DocValues all the way. Works really well for a few million values. Next: Performance: 200M values.
- Not time series, but bucketed by result size. Most real-world results in the lower end. Non-warmed, random queries, limit=25. 3 concurrent searches. 200M values. Note the high variance for same-size result sets.
- Standard JRE 1.7 garbage collector – no tuning. Full GC means delay for the client. Standard GC means higher CPU load.
- Upon release, the counter is cleared by the pool. Faster than re-allocating. Optionally the clearing is done in the background by a dedicated thread.
- Without and with reuse of counters. Probably possible to reduce full GC on reuse.
- With and without reuse of counters. The last we see of vanilla Solr (the tricks are stacked)
- Quartiles plot. Box is 75%, median and 25%. Lower median with reuse. Smaller variance with reuse. Downside: Static overhead. Next: Why the 500ms lower bound?
- Vanilla Solr extraction of top-X terms scales linear to the number of unique terms. Previous chart: 200M terms = 200M counters. Next: Track the changes.
- No change as ord 3 has already been incremented.
- Problem: Random memory access. Slower beyond ~8% (so turn off automatically).
- Non-sparse & sparse. Downside: 8% overhead. Next: Few Solr&apos;s stand alone - SolrCloud.
- With distributed faceting, fine counting is necessary.
- Pit of Pain coined by Andy Jackson, UKWA. 1K-10K very high, due to the highest chance of fine-counting. &lt; 100: Every shard probably returned all know terms. &gt; 100K: Most of the terms in top-X probably returned by most shards. 10M+: True load starts to show (rare). Note: 1M cardinality field. We count just skip finecounting, but we don&apos;t want to cheat that much. Next: What to do about it?
- Simple strategy: As phase 1 was fast, just repeat it and lookup the specific terms.
- Since we already did the counting in phase 1, we&apos;ll just re-use the filled counter structure. Cleanup problem handled by background thread.
- Yes, re-use and sparse multipliers does stack. Downside: Not much. Maybe large result sets? Next: From performance to memory: Counter representation.
- A typical shard from Net Archive Search v2. References are from documents to terms. High-cardinality in different ways. We want to facet on url and links to visualize connectivity hotspots. Next: All counters are not equal.
- Common theme: Many with max count 1.
- We flip the visualization. Squares are counter-bits needed.
- int[ordinals] uses 4 bytes/counter. PackedInts uses as many bits per value as is needed to represent the highest value of any counter.This works well for fields with a low max count (such as url), but not so well for fields with a high max count (domain and links). Note: The int[ordinals] is not to scale. It really goes up to 32.
- Black squares are counter value bits. Blue squares signify if the counter it belongs to is so large that it continues on the next plane. All previous blue squares must be counted. This requires a fast rank function to work (3% overhead). Creation is slow! Full term-iteration. Amortized plane-accesses for increments in n-plane is 2 (1 + 50% chance of p2 + 25% chance of p3 + ...). Each plane-shift requires rank lookup and overflow-bit lookup: 2 memory reads Details at https://sbdevel.wordpress.com/2015/03/13/n-plane-packed-counters-for-faceting/
- Start with 0.
- Set first bit. No overflow -&gt; no problem.
- Plane 1 overflows for L. Count blue overflow bits: There are 5. At position 5 in Plane 2, update the bits.
- This works very well as most counters have a max of 1.
- This works very well as most counters have a max of 1.
- This works very well as most counters have a max of 1.
- This works very well as most counters have a max of 1.
- This works very well as most counters have a max of 1.
- This works very well as most counters have a max of 1.
- N-plane-z instantiations beyond the first re-uses the overflow bits. Lowest possible: 0.7MB (15%), 27.9MB (4%), 132.3MB (6%)
- Single shard. 640M values. Obvious speed degradation that follows space savings. Insignificant with small result sets due to sparse. TODO: Worst case (*:*) array: packed: nplanez Next: Scale it up.
- I can&apos;t see how this could possibly go wrong.
- Worst case: About 50 billion references to 9 billion values must be processed. (10 minutes) Two strategies presents themselves: Threading for exact counts and sampling. Threading is currently implemented.
- Screenshot taken immediately after kitchen sink test.
- Available on GitHub. Currently Solr 4.8, promise to go for 4.10.4 Real Soon Now. 5.x later.