1. Get real-time insights from your application with
Packetbeat & Elasticsearch

2. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
2
This presentation contains
• Lots of JSON objects
• Pictures of cows, elks and
musical instruments
• A bit of Go code
• Anomaly detection via
moving averages

3. www.elastic.co
3
Tudor Golubenco
twitter.com/tudor_g
github.com/tsg
I am

4. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
4
I work for

5. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
5
the company behind
Elasticsearch
Logstash
Kibana

6. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
6
also known as
the
ELK
stack
Photo
credit:
https://www.flickr.com/photos/lsmith2010/8215026548

7. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
7
Distributed from day 1
Logstash 1.5 release party

8. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
8
Open source culture
Image
credit:
https://www.flickr.com/photos/tappnel/5798812875
• We live in GitHub
• We talk Pull Requests
• Conferences
• Community
• Blog posts

9. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
9
That’s not all
Found Beats
Watcher ShieldMarvel
ES
for
Hadoop

10. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
10
Started an open source project
Packetbeat

11. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
11
We wanted to do
better monitoring and troubleshooting for
distributed systems

12. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
12
Idea
look at the communication between
services

13. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
13
Capture network packets
• Passively listen to network
packets
• Works on a copy of the
traffic
• It doesn’t add latency
• It cannot break your
application
Image
credit:
https://www.flickr.com/photos/bigdrumthump/3223280727

14. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
14
Packet capturing
1.
Using
port
mirroring 2.
As
an
“agent”

15. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
15
Sniffing from a technical PoV
• libpcap (tcpdump), supports
all Unix like systems
• Winpcap, supports Windows
• For Go, gopacket provides
bindings and more
• High speed API for packet
capturing on Linux:
af_packet
Image
credit:
https://www.flickr.com/photos/57881779@N04/7930362242/

16. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
16
Decoding

17. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
17
Matching requests and responses
• Pipelining complicates
matching the requests with
the responses.

18. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
18
Create a JSON object for each request-response pair

19. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
19
SQL example

20. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
20
Memcache example

21. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
21
DNS example

22. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
22
• Control information: IP addresses, TCP/
UDP ports
• Request type: HTTP method, SQL query,
RPC function
• Request data: form parameters, payload
• Response data: status code, error
message, payload
• Timing data: timestamp of the request,
latency
Packet data

23. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
23
Packetbeat: Overview

24. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
24
There’s more to apps than packets
Packetbeat
Listens
to
the
“beat”
of
the
network
packets.
Topbeat
Listens
to
the
“beat”
of
the
operating
system
metrics.
Image
credits:
https://www.flickr.com/photos/7147684@N03/921738874/
https://www.flickr.com/photos/bigdrumthump/3223280727
https://www.flickr.com/photos/jadeashleyphotography/6584949945/
https://www.flickr.com/photos/mitosettembremusica/2839965900/
Filebeat
Listens
to
the
“beat”
of
logs.
Metricsbeat
Listens
to
the
internal
“beat”
of
systems
via
APIs.

25. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
25
Topbeat
• Like the Unix top
command but
sending the data
periodically to
Elasticsearch
• Works also on
Windows

26. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
26
Topbeat system wide and per process stats
CPU
“steal”
time
Total
/
used
/
free
memory
CPU
stats
Per
process
stats
CPU
time
consumed
Process
pid,
name,
parent
pid,
etc.
Memory
used

27. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
27
Topbeat output objects
File
system
stats
Mount
point
Device
name
Total,
used,
free
disk
space

28. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
28
Filebeat
• Do one thing well:
• Send log files to Logstash & Elasticsearch
• Light on consumed resources
• Easy to deploy on multiple platforms
• A “Beat” based on the Logstash-Forwarder
source code

29. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
29
Beats have libbeat in common
• Go library
• Provides common things for all
Beats:
• logging, service handling,
configuration file handling,
CLI flags
• Outputs and filters
Dev
guide
for
creating
a
new
Beat:
https://www.elastic.co/guide/en/beats/libbeat/current/index.html

30. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
30
Deployment: directly to ES
• Option 1: Insert
directly into
Elasticsearch via
the bulk API
• Security can be
provided via
Shield and HTTPs

31. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
31
Deployment: Send to Logstash
• Option 2: Insert via
Logstash
• Uses the Lumberjack
protocol which offers
security
• Gives the opportunity of
enriching or modifying the
data

32. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
32
Getting insights from the data
• Elasticsearch aggregations
• Split the data into
buckets
• Apply a function over the
data
• Freely combine them by
nesting
• Work with multiple shards

33. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
33
Date histogram
•Splits data in buckets of time

34. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
34
Date histogram response

35. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
35
Percentiles aggregation
95th percentile value means that 95% of the
values are smaller than it

36. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
36
Percentile aggregation response

37. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
37
Percentile aggregation
•Approximate values
•T-digests algorithm by Ted Dunning
•Accurate for small sets of values
•More accurate for extreme percentiles

38. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
38
Date histogram nested with percentiles

39. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
39
Date histogram with nested percentiles

40. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
40
Result

41. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
41
Kibana config

42. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
42
Latency histogram

43. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
43
Histogram by response time
• Splits data in buckets by response time
• [0-10ms), [10ms-20ms), …

44. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
44
Histogram by response time

45. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
45
Add a date histogram

46. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
46
Response times repartition

47. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
47
Kibana config

48. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
48
Slowest RPC methods
•Combines terms and percentiles aggregations

49. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
49
Terms aggregation
• Buckets are dynamically built: one per unique value
• By default: top 10 by document count
• Approximate because each shard can have a different

50. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
50
Order by 99th percentile

51. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
51
Kibana config

52. www.elastic.co
52
• New in Elasticsearch 2.0
(currently in beta)
• Work on the results of
other aggregations
Pipeline aggregations

53. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
53
Derivative aggregation
• Metric
constantly
growing
• Take first order
derivate to see
the speed of
growth

54. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
54
Simple moving average
• [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
• with a window size of 5:
• (1 + 2 + 3 + 4 + 5) / 5 = 3
• (2 + 3 + 4 + 5 + 6) / 5 = 4
• (3 + 4 + 5 + 6 + 7) / 5 = 5
• etc.

55. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
55
Exponentially weighted moving average
• Older values become exponentially less important

56. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
56
Moving average - dynamic thresholds
• yellow - measured values
• purple - moving average (ewma)
• green - threshold, mean + (3 * standard deviation)

57. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
57
Request
Extended
stats
agg
for
mean
and
std
deviation
Moving
averages
aggs
for
mean
and
std
Bucket
script
agg
Details:
https://www.elastic.co/blog/staying-­‐in-­‐control-­‐with-­‐moving-­‐averages-­‐part-­‐1

58. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
58
Cyclic trends - anomalies
• EWMA lags behind too much
• The values constantly hit the threshold

59. www.elastic.coCopyright Elastic 2015 Copying, publishing and/or distributing
without written permission is strictly prohibited
59
Cyclic trends - anomalies
• Holt-Winters (triple exponential) model works better for
seasonal data
• Requires two periods to bootstrap the algorithm

60. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
60
Thanks
• https://discuss.elastic.co/c/beats
• Twitter: @tudor_g
• We have stickers!