2. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
2
This presentation contains
• Lots of JSON objects
• Pictures of cows, elks and
musical instruments
• A bit of Go code
• Anomaly detection via
moving averages
5. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
5
the company behind
Elasticsearch
Logstash
Kibana
6. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
6
also known as
the
ELK
stack
Photo
credit:
https://www.flickr.com/photos/lsmith2010/8215026548
7. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
7
Distributed from day 1
Logstash 1.5 release party
8. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
8
Open source culture
Image
credit:
https://www.flickr.com/photos/tappnel/5798812875
• We live in GitHub
• We talk Pull Requests
• Conferences
• Community
• Blog posts
9. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
9
That’s not all
Found Beats
Watcher ShieldMarvel
ES
for
Hadoop
10. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
10
Started an open source project
Packetbeat
11. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
11
We wanted to do
better monitoring and troubleshooting for
distributed systems
12. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
12
Idea
look at the communication between
services
13. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
13
Capture network packets
• Passively listen to network
packets
• Works on a copy of the
traffic
• It doesn’t add latency
• It cannot break your
application
Image
credit:
https://www.flickr.com/photos/bigdrumthump/3223280727
14. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
14
Packet capturing
1.
Using
port
mirroring 2.
As
an
“agent”
15. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
15
Sniffing from a technical PoV
• libpcap (tcpdump), supports
all Unix like systems
• Winpcap, supports Windows
• For Go, gopacket provides
bindings and more
• High speed API for packet
capturing on Linux:
af_packet
Image
credit:
https://www.flickr.com/photos/57881779@N04/7930362242/
17. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
17
Matching requests and responses
• Pipelining complicates
matching the requests with
the responses.
18. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
18
Create a JSON object for each request-response pair
22. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
22
• Control information: IP addresses, TCP/
UDP ports
• Request type: HTTP method, SQL query,
RPC function
• Request data: form parameters, payload
• Response data: status code, error
message, payload
• Timing data: timestamp of the request,
latency
Packet data
23. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
23
Packetbeat: Overview
24. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
24
There’s more to apps than packets
Packetbeat
Listens
to
the
“beat”
of
the
network
packets.
Topbeat
Listens
to
the
“beat”
of
the
operating
system
metrics.
Image
credits:
https://www.flickr.com/photos/7147684@N03/921738874/
https://www.flickr.com/photos/bigdrumthump/3223280727
https://www.flickr.com/photos/jadeashleyphotography/6584949945/
https://www.flickr.com/photos/mitosettembremusica/2839965900/
Filebeat
Listens
to
the
“beat”
of
logs.
Metricsbeat
Listens
to
the
internal
“beat”
of
systems
via
APIs.
25. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
25
Topbeat
• Like the Unix top
command but
sending the data
periodically to
Elasticsearch
• Works also on
Windows
26. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
26
Topbeat system wide and per process stats
CPU
“steal”
time
Total
/
used
/
free
memory
CPU
stats
Per
process
stats
CPU
time
consumed
Process
pid,
name,
parent
pid,
etc.
Memory
used
27. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
27
Topbeat output objects
File
system
stats
Mount
point
Device
name
Total,
used,
free
disk
space
28. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
28
Filebeat
• Do one thing well:
• Send log files to Logstash & Elasticsearch
• Light on consumed resources
• Easy to deploy on multiple platforms
• A “Beat” based on the Logstash-Forwarder
source code
29. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
29
Beats have libbeat in common
• Go library
• Provides common things for all
Beats:
• logging, service handling,
configuration file handling,
CLI flags
• Outputs and filters
Dev
guide
for
creating
a
new
Beat:
https://www.elastic.co/guide/en/beats/libbeat/current/index.html
30. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
30
Deployment: directly to ES
• Option 1: Insert
directly into
Elasticsearch via
the bulk API
• Security can be
provided via
Shield and HTTPs
31. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
31
Deployment: Send to Logstash
• Option 2: Insert via
Logstash
• Uses the Lumberjack
protocol which offers
security
• Gives the opportunity of
enriching or modifying the
data
32. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
32
Getting insights from the data
• Elasticsearch aggregations
• Split the data into
buckets
• Apply a function over the
data
• Freely combine them by
nesting
• Work with multiple shards
33. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
33
Date histogram
•Splits data in buckets of time
34. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
34
Date histogram response
35. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
35
Percentiles aggregation
95th percentile value means that 95% of the
values are smaller than it
36. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
36
Percentile aggregation response
37. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
37
Percentile aggregation
•Approximate values
•T-digests algorithm by Ted Dunning
•Accurate for small sets of values
•More accurate for extreme percentiles
38. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
38
Date histogram nested with percentiles
39. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
39
Date histogram with nested percentiles
42. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
42
Latency histogram
43. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
43
Histogram by response time
• Splits data in buckets by response time
• [0-10ms), [10ms-20ms), …
44. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
44
Histogram by response time
45. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
45
Add a date histogram
46. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
46
Response times repartition
48. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
48
Slowest RPC methods
•Combines terms and percentiles aggregations
49. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
49
Terms aggregation
• Buckets are dynamically built: one per unique value
• By default: top 10 by document count
• Approximate because each shard can have a different
50. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
50
Order by 99th percentile
52. www.elastic.co
52
• New in Elasticsearch 2.0
(currently in beta)
• Work on the results of
other aggregations
Pipeline aggregations
53. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
53
Derivative aggregation
• Metric
constantly
growing
• Take first order
derivate to see
the speed of
growth
54. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
54
Simple moving average
• [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
• with a window size of 5:
• (1 + 2 + 3 + 4 + 5) / 5 = 3
• (2 + 3 + 4 + 5 + 6) / 5 = 4
• (3 + 4 + 5 + 6 + 7) / 5 = 5
• etc.
55. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
55
Exponentially weighted moving average
• Older values become exponentially less important
56. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
56
Moving average - dynamic thresholds
• yellow - measured values
• purple - moving average (ewma)
• green - threshold, mean + (3 * standard deviation)
57. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
57
Request
Extended
stats
agg
for
mean
and
std
deviation
Moving
averages
aggs
for
mean
and
std
Bucket
script
agg
Details:
https://www.elastic.co/blog/staying-‐in-‐control-‐with-‐moving-‐averages-‐part-‐1
58. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
58
Cyclic trends - anomalies
• EWMA lags behind too much
• The values constantly hit the threshold
59. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
59
Cyclic trends - anomalies
• Holt-Winters (triple exponential) model works better for
seasonal data
• Requires two periods to bootstrap the algorithm
60. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
60
Thanks
• https://discuss.elastic.co/c/beats
• Twitter: @tudor_g
• We have stickers!