Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Tuning ElasticSearch for multi-terabyte analytics


Published on

A talk by Andrew Clegg at the ElasticSearch London meetup in November 2013 on how Pearson does large-scale analytical queries on ElasticSearch.

Published in: Technology, Business

Tuning ElasticSearch for multi-terabyte analytics

  1. 1. ElasticSearch London Tuning ElasticSearch for multi-terabyte analytics or… “Counting stuff is hard” Andrew Clegg Data Analytics & Visualization Team Pearson @andrew_clegg
  2. 2. Introduction
  3. 3. Our data Over 11 billion “docs” in production cluster. Each doc is around 1-2KB of JSON. ~60 million docs/day == ~700 docs/sec. Higher than this during peak times. Much higher when backfilling historical data. Conversely: not many end users yet: 5-20 on a typical day.
  4. 4. Our architecture Palomino
  5. 5. Getting data in Hardware (Yes, actual hardware!) Cisco UCS servers, 24 cores, 96GB memory. 8 x 1TB disks. 7 for data, 1 for log files, temp files, etc. Reads/writes parallelized across segments. Currently 5 of these in production cluster. 10GB switch.
  6. 6. Getting data in Index configuration We don’t store any data in ElasticSearch. All we need is facet counts. Disable _source, _all, and individual field storage. Disable term vectors and norms. No analysis on text fields (just unbroken strings). No date autodetection.
  7. 7. Getting data in Weekly rolling indices mean shard level can increase as traffic does Time (new index each week) Shard count NB currently we have steady state so it’s set to 5 shards each week. 3 replicas per shard (including primary). Real-time implies: can’t disable replication during indexing!
  8. 8. Getting data in Client configuration Multiple writer threads on multiple machines: currently 6 x 3. Bulk API: currently up to 1000 docs per batch. Incoming docs queued until batch limit, or time or size limits, reached. (e.g. 1000 docs or 100000 bytes or 2 secs since last batch)
  9. 9. Getting data in Other things we could do -- but currently don’t Tune indexer thread pool size? Tune segment merge policy? Reduce flush interval? Even without these, our current record is over 20,000 docs indexed/sec. (And think the bottleneck was the client machines…)
  10. 10. Getting data out Typical queries Date histogram and terms facet are the most common by far. So we wrote our own versions with some optimizations :-) Field data cache size important for speed: currently 30% of 80GB heap. (In fact it actually uses much more than this, with ES 0.90.2. Upgrade planned!) We always use search_type=count.
  11. 11. Getting data out Facet workflow Client request Data nodes: ● Find matching records ● Perform groupings and counts (and any other calculations) ● Return to master Arbitrary master node: ● Parses query ● Distributes subqueries to data nodes (including itself) ● Combines results (reduce function) ● Returns to client
  12. 12. Getting data out Facet plugin optimizations Approximate data structures and sampling mode: Trade between speed/memory and accuracy. Uses Lucene’s BytesRef & BytesRefHash instead of String & HashSet. Micro-caching of local calculations, e.g. timestamp rounding. Explicit “render” phase after “reduce” phase: Defer as much as possible until then.
  13. 13. Getting data out General advice for plugin writers Minimize object creation/destruction and type conversions. Use arrays of primitives, or Trove collections, where possible. Reuse buffers. Release objects as soon as possible when no longer needed. Lucene has some neat tricks: bit fields, fast hashing algorithms. So does ElasticSearch: CacheRecycler lets you reuse collections.
  14. 14. Getting data out Hints for query performance tuning Tools like jmap, jstat, Visual VM and MAT are very helpful. Use ES “hot threads” API to see where it’s spending its time. Set up unit/integration tests with time and RAM instrumentation.
  15. 15. Getting data out Other things we could do -- but currently don’t Non-data nodes to parse queries, and handle reduce & render phases. Garbage collector tuning. (Note to self: see if Trove still crashes Java 7 JVM under G1 GC…) Use SSDs :-)
  16. 16. Thanks! Any questions?