Take your operational visibility to the next level by bringing your logs, metrics, and now APM data under one roof. Learn how Elasticsearch efficiently combines these types of data in a single store and see how Kibana is used to search logs, analyze metrics, and leverage APM features for better performance monitoring and faster troubleshooting.

1. 1
Álvaro Lobato
Observability team lead
March 2019
Logging, Metrics, and APM: The Operations Trifecta

2. Logs
Metrics
APM

3. 3
Benefits of Logs + Metrics + APM in one stack

4. 4
Unified Dashboards
Same UI for KPI summaries and root cause analysis

5. 5
Unified Alerting
Trigger off any operational data to provide unified SLA monitoring

6. 6
Unified Machine Learning
Correlate multiple data sources for more intelligent anomaly detection

7. 7
Operational gains
Single technology for operational data saves on administrative costs

8. 8
Elastic Stack for logs

9. Metrics vs Logs
64.242.88.10 - - [07/Mar/2017:16:10:02 -0800] "GET /mailman/listinfo/hsdivision HTTP/1.1" 200 6291
64.242.88.10 - - [07/Mar/2017:16:11:58 -0800] "POST /twiki/bin/view/TWiki/WikiSyntax HTTP/1.1" 404 7352
64.242.88.10 - - [07/Mar/2017:16:20:55 -0800] "GET /twiki/bin/view/Main/DCCAndPostFix HTTP/1.1" 200 5253
For each event, print out what happened.
Logs are chronological records of events

10. Making logging more turnkey with ‘modules’
• Turnkey experience for specific data types
• Data to dashboard in just one step
• Automated parsing and enrichment
• Default dashboards, alerts, ML jobs

11. Logging modules
System
• Linux / MacOS
• Windows Events
Containers
• Docker
• Kubernetes
Databases
• MySQL
• PostgreSQL
Queues
• Kafka
• Redis
Web servers
• Apache
• Nginx
Audit data
• Filesystem
• System calls
WINLOGBEATFILEBEATAUDITBEAT
Infrastructure Applications

12. 12
Log File Import
Automatic Structure Discovery

13. 13
Ad-hoc log search and visualization
Kibana Discover, Visualize, Dashboard

14. 14
Elastic Stack for metrics

15. Metrics vs Logs
64.242.88.10 - - [07/Mar/2017:16:10:02 -0800] "GET /mailman/listinfo/hsdivision HTTP/1.1" 200 6291
64.242.88.10 - - [07/Mar/2017:16:11:58 -0800] "POST /twiki/bin/view/TWiki/WikiSyntax HTTP/1.1" 404 7352
64.242.88.10 - - [07/Mar/2017:16:20:55 -0800] "GET /twiki/bin/view/Main/DCCAndPostFix HTTP/1.1" 200 5253
For each event, print out what happened.
Logs are chronological records of events
07/Mar/2017 16:10:00 all 2.58 0.00 0.70 1.12 0.05 95.55 server1 containerX regionA
07/Mar/2017 16:20:00 all 2.56 0.00 0.69 1.05 0.04 95.66 server2 containerY regionB
07/Mar/2017 16:30:00 all 2.64 0.00 0.65 1.15 0.05 95.50 server2 containerZ regionC
Every x minutes, measure the CPU load and print it out, and annotate with meta-data.
Metrics are periodic measurements of numeric KPIs

16. Elasticsearch for search and numerical analytics
Inverted Index for full-text search Columnar store for structured data
BKD Trees for numerical operations Rollups save space

17. 17
Elastic Stack as a Metrics Solution

18. Metrics modules
System
• Linux
• MacOS
• Windows
• Perfmon
Infrastructure
Cloud
• AWS
• GCP
• Azure
• DigitalOcean
• Alibaba
Containers
• Docker
• Kubernetes
Virtualization
• vSphere
PACKETBEATMETRICBEAT
Network
• Netflow
• Packets
• TLS Envelope
Storage
• Ceph
HEARTBEAT

19. Applications
Datastores
• MySQL
• PostgreSQL
• MongoDB
• Couchbase
• Aerospike
• Graphite
Web servers
• Apache
• Nginx
Other
• HAProxy
• Zookeeper
Queues
• Kafka
• Redis
• RabbitMQ
Caches
• Memcached
Uptime
• Heartbeat
Custom apps
• JMX/Jolokia
• PHP-FPM
• Golang
Metrics modules PACKETBEATMETRICBEATHEARTBEAT

20. Visualizing time series data
Time Series Visual Builder

21. Visualizing time series data
Annotations

22. 22
Elastic Stack for APM

23. What is APM?
Example
08:32:10 Request "/api/checkout"
08.32:11 Response "/api/checkout 500 ERROR"

24. What is APM?
Example
08:32:10 Request "/api/products/top"
08.32:17 Response "/api/products/top 200 OK"
7 seconds - zZzzZZz

25. How does APM work?
Data
processor
apm-server
Data storage
elasticsearch
Browser
Agent
Web server
Agent
Web server
Agent
Web server
Agent
UI
kibana
Browser
Agent
Browser
Agent

26. • Focuses on search experience on top of APM data
• ‘Just another index’ in Elastic Stack
Elastic APM
APM adds end-user experience and application-level monitoring to the stack
Language support
● Python
● Node.js
● Ruby
● RUM
● Java
● Go
● .NET (in dev)

27. APM is another index in Elasticsearch
Need another visualization? Build a dashboard, no need to wait for your vendor

28. Single transaction
Distributed Tracing
Transaction
Span
Span
Span
HTTP request Response

29. Distributed tracing example
Distributed Tracing
Trace A
Transaction 1
Span
Span
Span
Transaction 2
Span
Transaction 3
Span
Span

30. Distributed Tracing
Trace and map across multiple services
• See the end-to-end view and
navigate to individual transactions
• Based on the notion of a end-to-
end Trace ID across services
• Investigating compatibility with
OpenTracing API and aligning
with W3C trace context spec

31. 31
DEMO

32. 32
12:00 in London room
APM Group Discussion

33. 33
Come to Speaker AMA!
Questions?