Large scale net_archive_toke_eskildsen_iipc_workshop_2015
•Download as ODP, PDF•
3 likes•1,234 views
Scaling principles and case study for building a Solr index of material from a net archive (harvested web resources). Focus is multiple TB of index, multiple billions of documents, < 2 second response time for faceted search, few concurrent users. And doing it all without much money. The slides are annotated (turn on "Notes").
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (16)
Similar to Large scale net_archive_toke_eskildsen_iipc_workshop_2015
Similar to Large scale net_archive_toke_eskildsen_iipc_workshop_2015 (20)
Recently uploaded
Recently uploaded (20)
Large scale net_archive_toke_eskildsen_iipc_workshop_2015
- 1. Toke Eskildsen te@statsbiblioteket.dk IIPC Technical Training Workshop 2015 – Large Scale Net Archive Indexing - 1/26 Scaling Net Archive Indexing & Search IIPC Technical Training Workshop 2014 @TokeEskildsen Low-level search guy (boss says “System Architect”)
- 2. Toke Eskildsen te@statsbiblioteket.dk IIPC Technical Training Workshop 2015 – Large Scale Net Archive Indexing - 2/26 Scaling SolrCloud indexing ● CPU for analysis, bulk read & write for Solr ● Homogeneous shards (law of large numbers) ● Solr index update entry point might be bottleneck (so use more entry points) ● Routing overhead ● Splitting and moving shards ● Schema changes might require parallel rebuild
- 3. Toke Eskildsen te@statsbiblioteket.dk IIPC Technical Training Workshop 2015 – Large Scale Net Archive Indexing - 3/26 Static independent shards ● Easy scaling – Predictable resource requirements ● Selective shard rebuilding ● Trivial backup ● Lower overall requirements – Half the JVM heap requirements – Single segment→Higher performance – Less disk cache competition ● Temporal locality – Better disk cache utilization with few users – Hot spot problem with more users – Ranking suffers (in theory) ● No document-level updates! ~250M docs / 900GB shard
- 4. Toke Eskildsen te@statsbiblioteket.dk IIPC Technical Training Workshop 2015 – Large Scale Net Archive Indexing - 4/26 Static independent shards search Shard 01 Shard 02 Shard 03 Searcher 1 ZooKeeper
- 5. Toke Eskildsen te@statsbiblioteket.dk IIPC Technical Training Workshop 2015 – Large Scale Net Archive Indexing - 5/26 Building static shards ● Not standard Solr ● Sample setup (distribution optional) – 24 CPU cores (more would be nice) – 1 Solr indexer @ 40 GB RAM – 1 Archon tracking (W)ARC files – 1 Arctika controlling webarchive-discovery (Tika) – 40 webarchive-discovery (Tika) @ 1 GB RAM – Final shards: 250M docs, 900GB, fully optimized Archon + Arctika: https://github.com/netarchivesuite/netsearch
- 6. Toke Eskildsen te@statsbiblioteket.dk IIPC Technical Training Workshop 2015 – Large Scale Net Archive Indexing - 6/26 Static independent shards index Shard 4 Indexer 1 Shard 5 Indexer 2 Shard 1 Shard 2 Shard 3 Searcher 1 WAD = webarchive-discovery from UKWA: https://github.com/ukwa/webarchive-discovery WAD 1 Arctika 1 WAD 2... WAD n WAD 1 Arctika 1 WAD 2... WAD n ARC-path ARC-path ARC-path ARC-path Archon ARC 1 Storage ARC 2... ARC n
- 7. Toke Eskildsen te@statsbiblioteket.dk IIPC Technical Training Workshop 2015 – Large Scale Net Archive Indexing - 7/26 Measuring search performance ● Mimick real world scenarios – Unique queries ● preferably logged from production – Warmed caches – Concurrent searches (if relevant) – Measured time, not reported Qtime ● Capture setup data – Index size, shard count, document count, free cache memory, sar logs
- 8. Toke Eskildsen te@statsbiblioteket.dk IIPC Technical Training Workshop 2015 – Large Scale Net Archive Indexing - 8/26 Predicting scaling requirements ● All else is rarely equal – Disk cache / index size ratio – CPU cores / shard – Slowest shard dictates total response time ● 3 or more measurement points ● Use 2 or more shards ● Visualize measurements
- 9. Toke Eskildsen te@statsbiblioteket.dk IIPC Technical Training Workshop 2015 – Large Scale Net Archive Indexing - 9/26 SolrCloud distributed search ● Phase 1 – Tophits calculation (fast) – Simple faceting (medium to slow) ● Phase 2 – Document resolving (fast) – Facet fine count (medium to very slow) ● Coordination and merge overhead
- 10. Toke Eskildsen te@statsbiblioteket.dk IIPC Technical Training Workshop 2015 – Large Scale Net Archive Indexing - 10/26 Interval popularity (aka long tail)
- 11. Toke Eskildsen te@statsbiblioteket.dk IIPC Technical Training Workshop 2015 – Large Scale Net Archive Indexing - 11/26 ms over time
- 12. Toke Eskildsen te@statsbiblioteket.dk IIPC Technical Training Workshop 2015 – Large Scale Net Archive Indexing - 12/26 hits, ms
- 13. Toke Eskildsen te@statsbiblioteket.dk IIPC Technical Training Workshop 2015 – Large Scale Net Archive Indexing - 13/26 log(hits), ms
- 14. Toke Eskildsen te@statsbiblioteket.dk IIPC Technical Training Workshop 2015 – Large Scale Net Archive Indexing - 14/26 Bucketed percentiles (candlesticks)
- 15. Toke Eskildsen te@statsbiblioteket.dk IIPC Technical Training Workshop 2015 – Large Scale Net Archive Indexing - 15/26 Abstract search hardware ● IOPS – Needed for concurrent users and/or many shards ● Latency – 1 request = 1 thread / shard (lying a bit) – Lower latency → more IOPS ● Tapes < Spinning drives < SSDs < RAM – But the truth is in the mix
- 16. Toke Eskildsen te@statsbiblioteket.dk IIPC Technical Training Workshop 2015 – Large Scale Net Archive Indexing - 16/26 Case study: Net Archive Search at State and University Library, Denmark
- 17. Toke Eskildsen te@statsbiblioteket.dk IIPC Technical Training Workshop 2015 – Large Scale Net Archive Indexing - 17/26 Standard request ● Free-text matching in 6 fields ● Phrase matching i 1 field ● Grouping on URL (not used in the tests) ● Faceting – URL (~6b uniques, 7b references) – Host & domain (millions of uniques, 7b references) – 3 small ones (year, format, public suffix)
- 18. Toke Eskildsen te@statsbiblioteket.dk IIPC Technical Training Workshop 2015 – Large Scale Net Archive Indexing - 18/26 Solr version & schema ● Solr 4.8.1 + SOLR-5894 patch (optional) ● Piggy backing UKWA work ● DocValues on all large facet fields (essential)
- 19. Toke Eskildsen te@statsbiblioteket.dk IIPC Technical Training Workshop 2015 – Large Scale Net Archive Indexing - 19/26 Clever Solr config tweaks This space intentionally left blank
- 20. Toke Eskildsen te@statsbiblioteket.dk IIPC Technical Training Workshop 2015 – Large Scale Net Archive Indexing - 20/26 CPU
- 21. Toke Eskildsen te@statsbiblioteket.dk IIPC Technical Training Workshop 2015 – Large Scale Net Archive Indexing - 21/26 Disk cache RAM %index mean median 110 0.49 658 141 98 0.44 1004 170 54 0.24 2164 361 27 0.12 5620 913 7 0.03 8546 3012
- 22. Toke Eskildsen te@statsbiblioteket.dk IIPC Technical Training Workshop 2015 – Large Scale Net Archive Indexing - 22/26 Concurrent requests
- 23. Toke Eskildsen te@statsbiblioteket.dk IIPC Technical Training Workshop 2015 – Large Scale Net Archive Indexing - 23/26 Concurrent requests (less faceting)
- 24. Toke Eskildsen te@statsbiblioteket.dk IIPC Technical Training Workshop 2015 – Large Scale Net Archive Indexing - 24/26 Faceting impact mitigation Sparse faceting: http://tokee.github.io/lucene-solr/
- 25. Toke Eskildsen te@statsbiblioteket.dk IIPC Technical Training Workshop 2015 – Large Scale Net Archive Indexing - 25/26 Fewer, smaller facets
- 26. Toke Eskildsen te@statsbiblioteket.dk IIPC Technical Training Workshop 2015 – Large Scale Net Archive Indexing - 26/26 ● Measure thrice & visualise ● Common Solr rules of thumbs are not always applicable at Net Archive scale ● Static shards makes scaling easier ● SSDs works very well for us (22TB costs £7500) ● Full distributed faceting is doable but heavy Danish Net Archive: http://netarkivet.dk/in-english/ More Solr tech talk: http://sbdevel.wordpress.com
Editor's Notes
- The two main drawbacks of running a very large SolrCloud is The memory overhead of handling both search & indexing for each shard Having to re-build the whole index in one go, when a larger change to the schema is introduced
- From a search perspective, searching the whole collection of shards is done exactly as normally with SolrCloud: ZooKeeper keeps track of shard status and searches are distributed.
- Static shard indexing is fully independent of SolrCloud (ant ZooKeeper). Archon is a database, keeping track of the total amount of ARC-files and their status (non-indexed, being indexed, indexed, failed). There is only one Archon per logical collection (the whole Net Archive is one logical collection). Arctika is responsible for spawning WADs (webarchive-discovery from UKWA). It asks Archon for the path of an ARC in need of indexing and starts a new JVM, using WAD to analyze the single ARC and send the result into an indexer. It keeps doing this until the index has reached the wanted size; then it optimized the index and it is ready to be copied to the searcher. There is one Arctika per Solr Indexer. Each indexer is a plain Solr setup with a single shard.
- Single-shard Solr installations only uses 1 phase to compute the result. The speed of phase 2 in a SolrCloud search varies from much faster than phase 2 to very much slower, primarily dictated by how much fine-counting is needed for faceting. The merge-overhead is normally very small (few milliseconds).
- All testing done on the full index with 25 shards @ 900GB, with a total of 7 billion documents.
- Hyper Threading does have an effect. The number of cores beyone 8 has little influence on response speed for smaller result sets. Note: Scaling down requirements and scaling up concurrent requests is likely to require more CPU power.
- Graph for 22TB index on 25 SSDs. The amount of disk cache was controlled by a tiny program memeater (see https://github.com/tokee/memeater). Disk cache, or rather lack of disk cache, has profound impact on Solr performance. For SSDs as well as spinning drives. The difference being the amount needed overall and the worst-case penalty for cache-misses (not shown on the graph). At Statsbiblioteket we would like “normal” searches to return within 2 seconds. While we might get by with 100GB free memory for cache, 54GB would not be enough with current requirements.
- For full faceting, just 2 concurrent requests makes it too slow for result sets larger than 10M documents, which is not a good thing. Mean performance is 2-3 QPS.
- Turning off URL faceting helps a great deal. Performance is on average ~10 QPS. Throughput rises from 1-4 concurrent threads, after which there is no further gain.
- no_facet (blue) is plain search without faceting. skip_facets (purple) is faceting without fine-counting (phase 1 only). This gives imprecise results. We have not yet measured how much that influences the result. sparse_facets (green) is sparse faceting (http://tokee.github.io/lucene-solr/), an optimization of standard Solr faceting. solr_facet (orange) is standard Solr faceting. Notice how the measurement points is missing for more than 10.000 hits – this is due to the slow speed and limited test time.
- Faceting without the URL field makes it feasible to use standard Solr faceting and nearly free to use faceting without fine-counting or sparse faceting, as long as the result set is below 100M documents.
- Although SSDs themselves are very inexpensive, building a single machine with 25 drives requires quite a controller. It might be better to buy smaller machines, maybe with 4-6 drives @ 1TB, 64GB RAM and 4 CPU cores? Start with a single machine and measure!