The document discusses how logs, metrics, and application performance monitoring (APM) data can be integrated using the Elastic Stack for operational benefits. It describes how logs, metrics, and APM data can be visualized on unified dashboards and monitored with unified alerting. Machine learning can also correlate data from the different sources. The Elastic Stack provides modules to ingest various types of log and metrics data. Elasticsearch has evolved to efficiently store and analyze both textual log data and numerical metrics data. The Kibana UI allows exploring and visualizing the different data types.

1. David Sztykman
February 2019
Logging, Metrics, and APM:
The Operations Trifecta

2. 2
Benefits of Logs + Metrics + APM in one stack

3. Logs
Metrics
APM

4. 4
Unified Dashboards
Same UI for KPI summaries and root-cause analysis

5. 5
Unified Alerting
Trigger off any operational data to provide unified SLA monitoring

6. 6
Unified Machine Learning
Correlate multiple data sources for more intelligent anomaly detection

7. 7
Operational Gains
Using a single technology for operational data saves on administrative costs

8. 8
Elastic Stack for logs

9. Metrics vs Logs
Logs are chronological records of events
64.242.88.10 - - [07/Jan/2019:16:10:02 -0800] "GET /mailman/listinfo/hsdivision HTTP/1.1" 200 6291
64.242.88.10 - - [07/Jan/2019:16:11:58 -0800] "POST /twiki/bin/view/TWiki/WikiSyntax HTTP/1.1" 404 7352
64.242.88.10 - - [07/Jan/2019:16:20:55 -0800] "GET /twiki/bin/view/Main/DCCAndPostFix HTTP/1.1" 200 5253
For each event, print out what happened.

10. • Turnkey experience for specific data types
• Data to dashboard in just one step
• Automated parsing and enrichment
• Default dashboards, alerts, ML jobs
Logging Metrics Security
Making logging more turnkey with modules

11. Logging Modules
System
• Linux / MacOS
• Windows Events
Containers
• Docker
• Kubernetes
Databases
• MySQL
• PostgreSQL
Queues
• Kafka
• Redis
Web servers
• Apache
• Nginx
Audit data
• Filesystem
• System calls
Infrastructure Applications
WINLOGBEATFILEBEATAUDITBEAT

12. Log File Import
Automatic Structure Discovery

13. Ad-hoc log search and visualization
Kibana Discover, Visualize, Dashboard

14. 14
Elastic Stack for metrics

15. 64.242.88.10 - - [07/Jan/2019:16:10:02 -0800] "GET /mailman/listinfo/hsdivision HTTP/1.1" 200 6291
64.242.88.10 - - [07/Jan/2019:16:11:58 -0800] "POST /twiki/bin/view/TWiki/WikiSyntax HTTP/1.1" 404 7352
64.242.88.10 - - [07/Jan/2019:16:20:55 -0800] "GET /twiki/bin/view/Main/DCCAndPostFix HTTP/1.1" 200 5253
For each event, print out what happened.
Metrics vs Logs
Logs are chronological records of events
Metrics are periodic measurement of numeric KPIs
07/Jan/2019 16:10:00 all 2.58 0.00 0.70 1.12 0.05 95.55 server1 containerX regionA
07/Jan/2019 16:20:00 all 2.56 0.00 0.69 1.05 0.04 95.66 server2 containerY regionB
07/Jan/2019 16:30:00 all 2.64 0.00 0.65 1.15 0.05 95.50 server2 containerZ regionC
Every x minutes, measure the CPU load, print it out, and annotate with meta-data.

16. 16
Evolution of Elasticsearch into a Metrics Store

17. Elasticsearch beginnings
Primarily used for application search
Search engine
Inverted index primary data structure, and is
great for search
2010

18. 2012 Columnar storage
Structured data storage, resulting in compact
storage and faster analytics
Elasticsearch evolves to support analytics
https://www.elastic.co/blog/elasticsearch-as-a-column-store
Columnar Store, Built on Lucene "doc values"
Search engine
Inverted index primary data structure, and is
great for search
2010

19. 2014 Aggregation Framework
Analytics features to slice and dice data along
various dimensions
Aggregation Framework
Out-of-this-world aggregations
https://www.elastic.co/blog/out-of-this-world-aggregations
Search engine
Inverted index primary data structure, and is
great for search
2010
2012 Columnar storage
Structured data storage, resulting in compact
storage and faster analytics

20. BKD trees and sparse fields
Data structures optimized for numbers. Faster
analytics, lower storage footprint
2016
2014 Aggregation Framework
Analytics features to slice and dice data along
various dimensions
Elasticsearch storage efficiencies
BKD Trees & Sparse Fields
https://www.elastic.co/blog/searching-numb3rs-in-5.0
1-Dimension
2-Dimensions
Sparse Data
Search engine
Inverted index primary data structure, and is
great for search
2010
2012 Columnar storage
Structured data storage, resulting in compact
storage and faster analytics

21. Rollups
Roll up or aggregate older data into bigger
time buckets and save on disk space
2018
Rollup support for long-term retention
Added in Elasticsearch 6.3
https://www.elastic.co/blog/data-rollups-in-elasticsearch-you-know-for-saving-space
Search engine
Inverted index primary data structure, and is
great for search
2010
BKD trees and sparse fields
Data structures optimized for numbers. Faster
analytics, lower storage footprint
2016
2014 Aggregation Framework
Analytics features to slice and dice data along
various dimensions
2012 Columnar storage
Structured data storage, resulting in compact
storage and faster analytics

22. Elasticsearch for search and numerical analytics
Inverted Index for full-text search Columnar store for structured data
BKD Trees for numerical operations Rollups save space

23. 23
Evolution of the rest of Elastic Stack into a
Metrics Solution

24. Metrics Modules
Infrastructure
System
• Linux
• MacOS
• Windows
• Perfmon
Cloud
• AWS
• GCP
• Azure
• DigitalOcean
• Alibaba
Containers
• Docker
• Kubernetes
Virtualization
• vSphere
Network
• Netflow
• Packets
• TLS Envelope
Storage
• Ceph
PACKETBEATMETRICBEATHEARTBEAT
Infrastructure

25. Metrics Modules
Infrastructure
PACKETBEATMETRICBEATHEARTBEAT
Uptime
• Heartbeat
Custom apps
• JMX/Jolokia
• PHP-FPM
• Golang
Datastores
• MySQL
• PostgreSQL
• MongoDB
• Couchbase
• Aerospike
• Graphite
Queues
• Kafka
• Redis
• RabbitMQ
Caches
• Memcached
Web servers
• Apache
• Nginx
Other
• HAProxy
• Zookeeper
Applications

26. Visualizing time series data
Time Series Visual Builder

27. Visualizing time series data
Annotations

28. 28
Elastic Stack for APM

29. Why APM?
Example: Slow response or load times
03:43:45 Request "GET cyclops.ESProductDetailView"
03:43:57 Response "cyclops.ESProductDetailView 200 OK"
12 seconds - zZzzZZz

30. Why APM?
Example: Errors & Exceptions
03:43:59 Request "POST /api/checkout"
03:43:59 Response "/api/checkout 500 ERROR"

31. How APM works
Agents, API, and APM Server
Data
processor
apm-server
Data storage
Elasticsearch
Browser
Agent
Web server
Agent
Web server
Agent
UI
Kibana
Browser
Agent
Browser
Agent
Web server
Agent

32. Elastic APM
APM adds end-user experience and application-level monitoring to the stack
● Python
● Node.js
● Ruby
● RUM (Real User Monitoring)
Language Support
● Java
● Go
● .NET (in dev)
• Focuses on search experience on top of APM data
• Just another index in Elastic Stack
• Active roadmap to expand programming languages

33. Dedicated APM UI
Great overview and drill-down with industry-standard visualizations

34. Ad-hoc search in a curated UI
Combine a custom
workflow with the
freedom of search

35. APM is just another index in Elasticsearch
Need another visualization? Build a dashboard, no need to wait for your vendor

36. • Correlate data from different sources
• Ability to re-use analysis content
• Ability to re-use Elastic-provided content
Correlation between logs, metrics, and APM
Benefits
• Version 0.1 published: github.com/elastic/ecs
• Working with internal groups to validate
• Community feedback welcome!
Status
Elastic Common Schema

37. 37
DEMO

38. Questions?