Monica ist Mit-Schöpferin von Elastic Beats. Bevor sie Beats erfand, arbeitete sie als Core Developer für IPTEGO, einem Start-Up Unternehmen aus Berlin, das eine komplette Monitoring und Trouble-Shooting Solution für VoIP Netzwerke anbietet. Das Produkt wurde weltweit verkauft, und wird derzeit von großen Firmen der Telekommunikationsbranche verwendet.

1. Monitor your infrastructure
with the Elastic Beats
Monica Sarbu

2. 2
Monica Sarbu
Team lead, Beats team
Email: monica@elastic.co
Twitter: @monicasarbu

3. @monicasarbu
Monitor your servers
3
Apache logs

4. @monicasarbu
Monitor your servers
4
memory %
CPU %
Apache logs

5. @monicasarbu
Monitor your servers
5
Apache metrics
memory %
CPU %
Apache logs

6. @monicasarbu
Monitor your servers
6
Apache metrics
memory %
CPU %
HTTP transactions
Apache logs

7. @monicasarbu
Multiple data types, one storage
7
Apache metrics
memory %
CPU %
HTTP transactions
Apache logs

8. @monicasarbu
Scalable from day 1
8

9. Beats are lightweight shippers
that collect and ship all kinds of
operational data to Elasticsearch

10. @monicasarbu 10
Elastic
Stack
Kibana
Elasticsearch
Beats Logstash

11. @monicasarbu
The Beats
11
30+ other community Beats shipping

12. Filebeat
12

13. tail -f

14. tail -f
over the Network

15. tail -f
over the Network
with extra powers
http://www.clipartpanda.com/clipart_images/witches-clip-art-6144127

16. Multiline
JSON logs
Filtering

17. @monicasarbu
Send raw log lines
17
{
message: “55.3.244.1 GET /index.html 15824 0.043”,
…
}

18. @monicasarbu
Parse log lines by defining grok patterns
18
I
N
G
E
S
T
or

19. @monicasarbu
Grok patterns
19
%{IP:client} %{WORD:method} %{URIPATHPARAM:request}
%{NUMBER:bytes} %{NUMBER:duration}
{
message: “55.3.244.1 GET /index.html 15824 0.043”,
…
}

20. @monicasarbu
After parsing
20
{
message: “55.3.244.1 GET /index.html 15824 0.043”
client: “55.3.244.1”,
method: “GET”,
request: “/index.html”
bytes: 15824,
duration: 0.043
…
}

21. Handle back-pressure

22. @monicasarbu
Why back-pressure is key?
22

23. @monicasarbu
Synchronous sending
23
batch of messages
ack
stream of log linesreadreadacked
registry file

24. Filebeat adapts its speed
automatically to as much as the
next stage can process

25. @monicasarbu
When next stage is down …
• Filebeat patiently waits
• Log lines are not lost
• It doesn’t allocate memory
• It doesn’t buffer log lines on disk
25

26. At-Least-Once
delivery

27. #velo
@monicasarbu
No such think as “exactly once”
27

28. #velo
@monicasarbu
28
batch of messages
ack
batch of messages
same batch of messages
ack
duplicates!
28

29. Filebeat
Collect container logs
29

30. @monicasarbu
Docker logging drivers
30
https://docs.docker.com/engine/admin/logging/overview/

31. @monicasarbu
001 Gelf driver + Logstash
Pros:
• logs send directly to
Logstash
31
Cons:
• UDP based, no delivery
guarantees, no
congestion control

32. @monicasarbu
010 json-file driver + Filebeat
Pros:
• Simple to setup as it’s the
default driver
• Easy to add container
metadata (name, labels,
etc.)
• `docker logs` works
32
Cons:
• json-file driver can slow
down Docker container

33. @monicasarbu
011 Syslog driver + Syslog server + Filebeat
Pros:
• Good control over the
path where the files are
written, rotation strategies,
etc.
33
Cons:
• you need to manage the
syslog server
• metadata is serialized as
string, needs to be de-
serialized again
• multiline is difficult
because data from
containers can be mixed

34. @monicasarbu
100 Journald driver + Filebeat
Pros:
• journald is often already
available
• convenient support for
container metadata
(name, labels, etc.)
• `docker logs` works
34
Cons:
• Filebeat doesn’t yet
support journald
• You can use the
community Beat,
Journalbeat

35. @monicasarbu
101 Shared volume + Filebeat
Pros:
• If your app can rotate it’s
own logs, it’s very easy to
setup
• Scales well
35
Cons:
• Difficult to pass container
metadata (name, labels,
etc.)

36. @monicasarbu
Conclusion
“At-least-once” guarantees and handle
back-pressure:
• json-file driver + Filebeat
• Syslog driver + Filebeat
• Shared volume + Filebeat
• Journald driver + Filebeat (in the
future)
36
No guarantees:
• Gelf driver + Logstash
• Fluentd driver + Logstash

37. Metricbeat
37
new in 5.0

38. @monicasarbu
One Metricbeat module for each service
38
+
Add your own

39. @monicasarbu
Metricbeat system module
39
CPU Mem diskIO filesystem
processesload network cores

40. Metricbeat
Collect container metrics
40

41. @monicasarbu
Querying the Docker API
• CPU and memory
• Docker container information
• network (in/out bytes, dropped)
• diskIO (reads/writes)
• status of containers (# of stopped, running, etc)
41

42. @monicasarbu
Docker module in Metricbeat
• Get container metrics by querying the Docker API
• Has access to container names and labels
• Easy to setup
42
in progress

43. @monicasarbu
Reading cgroup data from /proc/
• Doesn’t require access to the Docker API
(can be a security issue)
• Works for any container runtime (Docker,
rkt, runC, LXD, etc.)
• Cannot get the container name and labels
only the container ID
43

44. @monicasarbu
System module + cgroup data
• if cgroup option is enabled (by default is
disabled)
• Automatically enhances process data with
cgroup information
44

45. @monicasarbu
Run as a container
45
App1 App2 App3
Host

46. 46
Elasticsearch as time series DB

47. #velo
@monicasarbu
Elasticsearch BKD trees
47
• Added for Geo-points
• faster to index
• faster to query
• more disk-efficient
• more memory efficient

48. @monicasarbu
0
10000
20000
30000
40000
50000
60000
70000
80000
float half float scaled float
(factor = 4000)
scaled float
(factor = 100)
On Disk Usage in kb
Points disk usage (kb) docs_values disk usage (kb)
Float values
48
• half floats
• scaled floats (using a
scaling factor) - great for
things like percentage
points

49. #velo
@monicasarbu
Why Elasticsearch for time series
• Horizontal scalability. Mature and
battle tested cluster support.
• Flexible aggregations (incl moving
averages & Holt Winters)
• One system for both logs and
metrics
• Timelion UI, Grafana
• Great ecosystem: e.g. alerting
tools
49

50. Packetbeat
50

51. @monicasarbu
How Packetbeat works
51
1 2 3 4
capture
network traffic
decodes
network traffic
correlates
request &
response into
transactions
send
transactions to
Elasticsearch

52. @monicasarbu
Supported traffic decoders
52
+
Add your own
http:// Thrift DNS
ICMP AMQP

53. @monicasarbu
Unknown traffic, use flows
•Look into data for which we don’t
understand the application layer protocol
•TLS
•Protocols we don’t yet support
•Get data about IP / TCP / UDP layers
•number of packets & bytes
•retransmissions
•inter-arrival time
53

54. @monicasarbu
Monitor traffic exchanged by containers
54
App1
Host
App2 App3
Packetbeat
traffic exchanged between your containers

55. 55
Demo: Metricbeat, Filebeat,
Packetbeat
Multiple data types, one view in Kibana

56. Thank you
• github.com/elastic/beats
• discuss.elastic.co
• @elastic #elasticbeats
• #beats on freenode
56