1.
Elasticsearch for logs and metrics
(a deep dive)
Rafał Kuć and Radu Gheorghe
Sematext Group, Inc.

2.
About us
Logsene
SPM
ES API
metrics
...
Products Services

3.
Agenda
Index layout
Cluster layout
Per-index tuning of settings and mappings
Hardware+OS options
Pipeline patterns

4.
Daily indices are a good start
...
indexing, most searches
Indexing is faster in smaller indices
Cheap deletes
Search only needed indices
“Static” indices can be cached

5.
The Black Friday problem*
* for logs. Metrics usually don’t suffer from this

6.
Typical indexing performance graph for one shard*
* throttled so search performance remains decent
At this point it’s better to index in a new shard
Typically 5-10GB, YMMV

7.
INDEX
Y U NO AS FAST

8.
INDEX
Y U NO AS FAST
more merges
more expensive
(+uncached) searches
Mostly because

9.
Rotate by size*
* use Field Stats for queries or rely on query cache:
https://github.com/elastic/kibana/issues/6644

10.
Aliases; Rollover Index API*
* 5.0 feature

11.
Slicing data by time
For spiky ingestion, use size-based indices
Make sure you rotate before the performance drop
(test on one node to get that limit)

12.
Multi tier architecture (aka hot/cold)
Client
Client
Client
Data
Data
Data
...
Data
Data
Data
Master
Master
Master
We can optimize data nodes layer
Ingest
Ingest
Ingest

13.
Multi tier architecture (aka hot/cold)
logs_2016.11.07
indexing
es_hot_1 es_cold_1 es_cold_2

14.
Multi tier architecture (aka hot/cold)
logs_2016.11.07
logs_2016.11.08
indexing
m
ove
es_hot_1 es_cold_1 es_cold_2
curl -XPUT localhost:9200/logs_2016.11.07/_settings -d '{
"index.routing.allocation.exclude.tag" : "hot",
"index.routing.allocation.include.tag": "cold"
}'

15.
Multi tier architecture (aka hot/cold)
logs_2016.11.08 logs_2016.11.07
indexing
es_hot_1 es_cold_1 es_cold_2

16.
Multi tier architecture (aka hot/cold)
logs_2016.11.11
logs_2016.11.07
logs_2016.11.09
logs_2016.11.08
logs_2016.11.10
indexing, most searches long running searches
good CPU, best possible IO heap, IO for backup/replication and stats
es_hot_1 es_cold_1 es_cold_2
SSD or RAID0 for spinning

17.
Hot - cold architecture summary
Costs optimization - different hardware for different tier
Performance - above + fewer shards, less overhead
Isolation - long running searches don't affect indexing

18.
Elasticsearch high availability & fault tolerance
Dedicated masters is a must
discovery.zen.minimum_master_nodes = N/2 + 1
Keep your indices balanced
not balanced cluster can lead to instability
Balanced primaries are also good
helps with backups, moving to cold tier, etc
total_shards_per_node is your friend

19.
Elasticsearch high availability & fault tolerance
When in AWS - spread between availability zones
bin/elasticsearch -Enode.attr.zone=zoneA
cluster.routing.allocation.awareness.attributes: zone
We need headroom for spikes
leave at least 20 - 30% for indexing & search spikes
Large machines with many shards?
look out for GC - many clusters died because of that
consider running smaller ES instances but more

20.
Which settings to tune
Merges → most indexing time
Refreshes → check refresh_interval
Flushes → normally OK with ES defaults

21.
Relaxing the merge policy
Less merges ⇒ faster indexing/lower CPU while indexing
Slower searches, but:
- there’s more spare CPU
- aggregations aren’t as affected, and they are typically the bottleneck
especially for metrics
More open files (keep an eye on them!)
Increase index.merge.policy.segments_per_tier ⇒ more segments, less merges
Increase max_merge_at_once, too, but not as much ⇒ reduced spikes
Reduce max_merged_segment ⇒ no more huge merges, but more small ones

22.
And even more settings
Refresh interval (index.refresh_interval)*
- 1s -> baseline indexing throughput
- 5s -> +25% to baseline throughput
- 30s -> +75% to baseline throughput
Higher indices.memory.index_buffer_size higher throughput
Lower indices.queries.cache.size for high velocity data to free up heap
Omit norms (frequencies and positions, too?)
Don't store fields if _source is used
Don't store catch-all (i.e. _all) field - data copied from other fields
* https://sematext.com/blog/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/

23.
Let’s dive deeper into storage
Not searches on a field, just aggregations ⇒ index=false
Not sorting/aggregating on a field ⇒ doc_values=false
Doc values can be used for retrieving (see docvalue_fields), so:
● Logs: use doc values for retrieving, exclude them from _source*
● Metrics: short fields normally ⇒ disable _source, rely on doc values
Long retention for logs? For “old” indices:
● set index.codec=best_compression
● force merge to few segments
* though you’ll lose highlighting, update API, reindex API...

24.
Metrics: working around sparse data
Ideally, you’d have one index per metric type (what you can fetch with one call)
Combining them into one (sparse) index will impact performance (see LUCENE-7253)
One doc per metric: you’ll pay with space
Nested documents: you’ll pay with heap (bitset used for joins) and query latency

25.
What about the OS?
Say no to swap
Disk scheduler: CFQ for HDD, deadline for SSD
Mount options: noatime, nodiratime, data=writeback, nobarrier
because strict ordering is for the weak

26.
And hardware?
Hot tier. Typical bottlenecks: CPU and IO throughput
indexing is CPU-intensive
flushes and merges write (and read) lots of data
Cold tier: Memory (heap) and IO latency
more data here ⇒ more indices&shards ⇒ more heap
⇒ searches hit more files
many stats calls are per shard ⇒ potentially choke IO when cluster is idle
Generally:
network storage needs to be really good (esp. for cold tier)
network needs to be low latency (pings, cluster state replication)
network throughput is needed for replication/backup

27.
AWS specifics
c3 instances work, but there’s not enough local SSD ⇒ EBS gp2 SSD*
c4 + EBS give similar performance, but cheaper
i2s are good, but expensive
d2s are better value, but can’t deal with many shards (spinning disk latency)
m4 + gp2 EBS are a good balance
gp2 → PIOPS is expensive, spinning is slow
3 IOPS/GB, but caps at 160MB/s or 10K IOPS (of up to 256kb) per drive
performance isn’t guaranteed (for gp2) ⇒ one slow drive slows RAID0
Enhanced Networking (and EBS Optimized if applicable) are a must
* And used local SSD as cache. With --cachemode writeback for async writing:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Logical_Vol
ume_Manager_Administration/lvm_cache_volume_creation.html
block
size?

28.
The pipeline
read buffer deliver

29.
The pipeline
read buffer deliver
Log shipper
reason #1

30.
The pipeline
read buffer deliver
Log shipper
reason #1
Files? Sockets? Network?
What if buffer fills up?
Processing
before/after buffer?
How?
Others besides Elasticsearch?
How to buffer if $destination is down?
Overview of 6 log shippers: sematext.com/blog/2016/09/13/logstash-alternatives/

31.
Types of buffers
buffer
application.log Log file can act as a buffer
Memory and/or disk of the log shipper
or a dedicated tool for buffering

32.
Where to do processing
Logstash
(or Filebeat or…)
Buffer
(Kafka/Redis)
here
Logstash Elasticsearch

33.
Where to do processing
Logstash
Buffer
(Kafka/Redis)
here
Logstash Elasticsearch
something
else

34.
Where to do processing
Logstash
Buffer
(Kafka/Redis)
here
Logstash Elasticsearch
something
else
Outputs
need to be
in sync

35.
Where to do processing
Logstash Kafka Logstash Elasticsearch
something
else
LogstashElasticsearch
offset
other
offset
here
here,
too

36.
Where to do processing (syslog-ng, fluentd…)
input
here
Elasticsearch
something
else
here

37.
Where to do processing (rsyslogd…)
input
here
here
here

38.
Zoom into processing
Ideally, log in JSON
Otherwise, parse
For performance and maintenance
(i.e. no need to update parsing rules)
Regex-based (e.g. grok)
Easy to build rules
Rules are flexible
Slow & O(n) on # of rules
Tricks:
Move matching patterns to the top of the list
Move broad patterns to the bottom
Skip patterns including others that didn’t match
Grammar-based
(e.g. liblognorm, PatternDB)
Faster. O(1) on # of rules. References:
Logagent
Logstash
rsyslog syslog-ng
sematext.com/blog/2015/05/18/tuning-elasticsearch-indexing-pipeline-for-logs/
www.fernuni-hagen.de/imperia/md/content/rechnerarchitektur/rainer_gerhards.pdf

39.
Back to buffers: check what happens if when they fill up
Local files: when are they rotated/archived/deleted?
TCP: what happens when connection breaks/times out?
UNIX sockets: what happens when socket blocks writes?
UDP: network buffers should handle spiky load
Check/increase
net.core.rmem_max
net.core.rmem_default
Unlike UDP&TCP,
both DGRAM and STREAM
local sockets
are reliable/blocking

40.
Let’s talk protocols now
UDP: cool for the app, but not reliable
TCP: more reliable, but not completely
Application-level ACKs may be needed:
No failure/backpressure handling needed
App gets ACK when OS buffer gets it
⇒ no retransmit if buffer is lost*
* more at blog.gerhards.net/2008/05/why-you-cant-build-reliable-tcp.html
sender receiver
ACKs
Protocol Example shippers
HTTP Logstash, rsyslog, syslog-ng, Fluentd, Logagent
RELP rsyslog, Logstash
Beats Filebeat, Logstash
Kafka Fluentd, Filebeat, rsyslog, syslog-ng, Logstash

41.
Wrapping up: where to log?
critical?
UDP. Increase network
buffers on destination,
so it can handle spiky
traffic
Paying with
RAM or IO?
UNIX socket. Local
shipper with memory
buffers, that can
drop data if needed
Local files. Make sure
rotation is in place or
you’ll run out of disk!
no
IO RAM
yes

42.
Flow patterns (1 of 5)
application.log
application.log
Logstash
Logstash
Elasticsearch
Easy&flexible
Overhead

43.
Flow patterns (2 of 5)
application.log
application.log
Filebeat
Filebeat
Elasticsearch
(with Ingest)
Light&simple
Harder to scale processing
sematext.com/blog/2016/04/25/elasticsearch-ingest-node-vs-logstash-performance/

44.
Flow patterns (3 of 5)
Elasticsearch
files,
sockets (syslog?),
localhost TCP/UDP
Logagent
Fluentd
rsyslog
syslog-ng
Light, scales
No central control
sematext.com/blog/2016/09/13/logstash-alternatives/

45.
Flow patterns (4 of 5)
ElasticsearchKafka
Filebeat
Logagent
Fluentd
rsyslog
syslog-ng
Good for multiple destinations
More complex
something
else
Logstash,
custom consumer

46.
Flow patterns (5 of 5)

47.
Thank you!
Rafał Kuć
rafal.kuc@sematext.com
@kucrafal
Radu Gheorghe
radu.gheorghe@sematext.com
@radu0gheorghe
Sematext
info@sematext.com
http://sematext.com
@sematext
Join Us! We are hiring!
http://sematext.com/jobs

48.
Pictures
https://pixabay.com/get/e831b60920f71c22d2524518a33219c8b66ae3d11eb611429df9c77f/scuba-diving-147683_1280.png
https://pixabay.com/static/uploads/photo/2012/04/18/12/17/firewood-36866_640.png
http://i3.kym-cdn.com/entries/icons/original/000/004/006/y-u-no-guy.jpg
http://memepress.wpgoods.com/wp-content/uploads/2013/06/neutral-feel-like-a-sir-clean-l1.png