Elasticsearch for Logs & Metrics - a deep dive

1. Elasticsearch for logs and metrics (a deep dive) Rafał Kuć and Radu Gheorghe Sematext Group, Inc.

2. About us Logsene SPM ES API metrics ... Products Services

3. Agenda Index layout Cluster layout Per-index tuning of settings and mappings Hardware+OS options Pipeline patterns

4. Daily indices are a good start ... indexing, most searches Indexing is faster in smaller indices Cheap deletes Search only needed indices “Static” indices can be cached

5. The Black Friday problem* * for logs. Metrics usually don’t suffer from this

6. Typical indexing performance graph for one shard* * throttled so search performance remains decent At this point it’s better to index in a new shard Typically 5-10GB, YMMV

7. INDEX Y U NO AS FAST

8. INDEX Y U NO AS FAST more merges more expensive (+uncached) searches Mostly because

9. Rotate by size* * use Field Stats for queries or rely on query cache: https://github.com/elastic/kibana/issues/6644

10. Aliases; Rollover Index API* * 5.0 feature

11. Slicing data by time For spiky ingestion, use size-based indices Make sure you rotate before the performance drop (test on one node to get that limit)

12. Multi tier architecture (aka hot/cold) Client Client Client Data Data Data ... Data Data Data Master Master Master We can optimize data nodes layer Ingest Ingest Ingest

13. Multi tier architecture (aka hot/cold) logs_2016.11.07 indexing es_hot_1 es_cold_1 es_cold_2

14. Multi tier architecture (aka hot/cold) logs_2016.11.07 logs_2016.11.08 indexing m ove es_hot_1 es_cold_1 es_cold_2 curl -XPUT localhost:9200/logs_2016.11.07/_settings -d '{ "index.routing.allocation.exclude.tag" : "hot", "index.routing.allocation.include.tag": "cold" }'

15. Multi tier architecture (aka hot/cold) logs_2016.11.08 logs_2016.11.07 indexing es_hot_1 es_cold_1 es_cold_2

16. Multi tier architecture (aka hot/cold) logs_2016.11.11 logs_2016.11.07 logs_2016.11.09 logs_2016.11.08 logs_2016.11.10 indexing, most searches long running searches good CPU, best possible IO heap, IO for backup/replication and stats es_hot_1 es_cold_1 es_cold_2 SSD or RAID0 for spinning

17. Hot - cold architecture summary Costs optimization - different hardware for different tier Performance - above + fewer shards, less overhead Isolation - long running searches don't affect indexing

18. Elasticsearch high availability & fault tolerance Dedicated masters is a must discovery.zen.minimum_master_nodes = N/2 + 1 Keep your indices balanced not balanced cluster can lead to instability Balanced primaries are also good helps with backups, moving to cold tier, etc total_shards_per_node is your friend

19. Elasticsearch high availability & fault tolerance When in AWS - spread between availability zones bin/elasticsearch -Enode.attr.zone=zoneA cluster.routing.allocation.awareness.attributes: zone We need headroom for spikes leave at least 20 - 30% for indexing & search spikes Large machines with many shards? look out for GC - many clusters died because of that consider running smaller ES instances but more

20. Which settings to tune Merges → most indexing time Refreshes → check refresh_interval Flushes → normally OK with ES defaults

21. Relaxing the merge policy Less merges ⇒ faster indexing/lower CPU while indexing Slower searches, but: - there’s more spare CPU - aggregations aren’t as affected, and they are typically the bottleneck especially for metrics More open files (keep an eye on them!) Increase index.merge.policy.segments_per_tier ⇒ more segments, less merges Increase max_merge_at_once, too, but not as much ⇒ reduced spikes Reduce max_merged_segment ⇒ no more huge merges, but more small ones

22. And even more settings Refresh interval (index.refresh_interval)* - 1s -> baseline indexing throughput - 5s -> +25% to baseline throughput - 30s -> +75% to baseline throughput Higher indices.memory.index_buffer_size higher throughput Lower indices.queries.cache.size for high velocity data to free up heap Omit norms (frequencies and positions, too?) Don't store fields if _source is used Don't store catch-all (i.e. _all) field - data copied from other fields * https://sematext.com/blog/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/

23. Let’s dive deeper into storage Not searches on a field, just aggregations ⇒ index=false Not sorting/aggregating on a field ⇒ doc_values=false Doc values can be used for retrieving (see docvalue_fields), so: ● Logs: use doc values for retrieving, exclude them from _source* ● Metrics: short fields normally ⇒ disable _source, rely on doc values Long retention for logs? For “old” indices: ● set index.codec=best_compression ● force merge to few segments * though you’ll lose highlighting, update API, reindex API...

24. Metrics: working around sparse data Ideally, you’d have one index per metric type (what you can fetch with one call) Combining them into one (sparse) index will impact performance (see LUCENE-7253) One doc per metric: you’ll pay with space Nested documents: you’ll pay with heap (bitset used for joins) and query latency

25. What about the OS? Say no to swap Disk scheduler: CFQ for HDD, deadline for SSD Mount options: noatime, nodiratime, data=writeback, nobarrier because strict ordering is for the weak

26. And hardware? Hot tier. Typical bottlenecks: CPU and IO throughput indexing is CPU-intensive flushes and merges write (and read) lots of data Cold tier: Memory (heap) and IO latency more data here ⇒ more indices&shards ⇒ more heap ⇒ searches hit more files many stats calls are per shard ⇒ potentially choke IO when cluster is idle Generally: network storage needs to be really good (esp. for cold tier) network needs to be low latency (pings, cluster state replication) network throughput is needed for replication/backup

27. AWS specifics c3 instances work, but there’s not enough local SSD ⇒ EBS gp2 SSD* c4 + EBS give similar performance, but cheaper i2s are good, but expensive d2s are better value, but can’t deal with many shards (spinning disk latency) m4 + gp2 EBS are a good balance gp2 → PIOPS is expensive, spinning is slow 3 IOPS/GB, but caps at 160MB/s or 10K IOPS (of up to 256kb) per drive performance isn’t guaranteed (for gp2) ⇒ one slow drive slows RAID0 Enhanced Networking (and EBS Optimized if applicable) are a must * And used local SSD as cache. With --cachemode writeback for async writing: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Logical_Vol ume_Manager_Administration/lvm_cache_volume_creation.html block size?

28. The pipeline read buffer deliver

29. The pipeline read buffer deliver Log shipper reason #1

30. The pipeline read buffer deliver Log shipper reason #1 Files? Sockets? Network? What if buffer fills up? Processing before/after buffer? How? Others besides Elasticsearch? How to buffer if $destination is down? Overview of 6 log shippers: sematext.com/blog/2016/09/13/logstash-alternatives/

31. Types of buffers buffer application.log Log file can act as a buffer Memory and/or disk of the log shipper or a dedicated tool for buffering

32. Where to do processing Logstash (or Filebeat or…) Buffer (Kafka/Redis) here Logstash Elasticsearch

33. Where to do processing Logstash Buffer (Kafka/Redis) here Logstash Elasticsearch something else

34. Where to do processing Logstash Buffer (Kafka/Redis) here Logstash Elasticsearch something else Outputs need to be in sync

35. Where to do processing Logstash Kafka Logstash Elasticsearch something else LogstashElasticsearch offset other offset here here, too

36. Where to do processing (syslog-ng, fluentd…) input here Elasticsearch something else here

37. Where to do processing (rsyslogd…) input here here here

38. Zoom into processing Ideally, log in JSON Otherwise, parse For performance and maintenance (i.e. no need to update parsing rules) Regex-based (e.g. grok) Easy to build rules Rules are flexible Slow & O(n) on # of rules Tricks: Move matching patterns to the top of the list Move broad patterns to the bottom Skip patterns including others that didn’t match Grammar-based (e.g. liblognorm, PatternDB) Faster. O(1) on # of rules. References: Logagent Logstash rsyslog syslog-ng sematext.com/blog/2015/05/18/tuning-elasticsearch-indexing-pipeline-for-logs/ www.fernuni-hagen.de/imperia/md/content/rechnerarchitektur/rainer_gerhards.pdf

39. Back to buffers: check what happens if when they fill up Local files: when are they rotated/archived/deleted? TCP: what happens when connection breaks/times out? UNIX sockets: what happens when socket blocks writes? UDP: network buffers should handle spiky load Check/increase net.core.rmem_max net.core.rmem_default Unlike UDP&TCP, both DGRAM and STREAM local sockets are reliable/blocking

40. Let’s talk protocols now UDP: cool for the app, but not reliable TCP: more reliable, but not completely Application-level ACKs may be needed: No failure/backpressure handling needed App gets ACK when OS buffer gets it ⇒ no retransmit if buffer is lost* * more at blog.gerhards.net/2008/05/why-you-cant-build-reliable-tcp.html sender receiver ACKs Protocol Example shippers HTTP Logstash, rsyslog, syslog-ng, Fluentd, Logagent RELP rsyslog, Logstash Beats Filebeat, Logstash Kafka Fluentd, Filebeat, rsyslog, syslog-ng, Logstash

41. Wrapping up: where to log? critical? UDP. Increase network buffers on destination, so it can handle spiky traffic Paying with RAM or IO? UNIX socket. Local shipper with memory buffers, that can drop data if needed Local files. Make sure rotation is in place or you’ll run out of disk! no IO RAM yes

42. Flow patterns (1 of 5) application.log application.log Logstash Logstash Elasticsearch Easy&flexible Overhead

43. Flow patterns (2 of 5) application.log application.log Filebeat Filebeat Elasticsearch (with Ingest) Light&simple Harder to scale processing sematext.com/blog/2016/04/25/elasticsearch-ingest-node-vs-logstash-performance/

44. Flow patterns (3 of 5) Elasticsearch files, sockets (syslog?), localhost TCP/UDP Logagent Fluentd rsyslog syslog-ng Light, scales No central control sematext.com/blog/2016/09/13/logstash-alternatives/

45. Flow patterns (4 of 5) ElasticsearchKafka Filebeat Logagent Fluentd rsyslog syslog-ng Good for multiple destinations More complex something else Logstash, custom consumer

46. Flow patterns (5 of 5)

47. Thank you! Rafał Kuć rafal.kuc@sematext.com @kucrafal Radu Gheorghe radu.gheorghe@sematext.com @radu0gheorghe Sematext info@sematext.com http://sematext.com @sematext Join Us! We are hiring! http://sematext.com/jobs

48. Pictures https://pixabay.com/get/e831b60920f71c22d2524518a33219c8b66ae3d11eb611429df9c77f/scuba-diving-147683_1280.png https://pixabay.com/static/uploads/photo/2012/04/18/12/17/firewood-36866_640.png http://i3.kym-cdn.com/entries/icons/original/000/004/006/y-u-no-guy.jpg http://memepress.wpgoods.com/wp-content/uploads/2013/06/neutral-feel-like-a-sir-clean-l1.png