Visibility into your infrastructure is critical, and the Elastic (ELK) Stack, brings its logging strengths to your metrics use case. Discover how simplified data onboarding with hundreds of prebuilt integrations, automated insights with alerting and machine learning, and new visual tools built for exploring infrastructure metrics are streamlining the monitoring use case.

1. 1
Why you should use
Elastic for Infrastructure
Metrics
Karl Degenhardt
Senior Solutions Architect

2. 2
Evolving Architectures ~↑ Monitoring Complexity
Hardware & software trends
are evolving in tandem
Higher resource utilization
increases monitoring complexity
• Orchestration/Hypervisor
• Dynamic/ephemeral jobs
• You can no longer "point" to
where that job lives
Shift to cloud-native yields
maintainable code, with costs
• Traditional licensing models don't scale
as well as your applications
• Hurdles with autoscaling
Monitoring Complexity

3. 3
Applications
VMs/Containers
Other DBs,
Services &
Middleware
Orchestration Infrastructure
APM
Metrics
Logs
Uptime
Uptime
APM Metrics
APM Logs
APM
APM
Metrics
Logs
Uptime
Metrics
Logs
Uptime
APM

4. 4
• Support the full stack
• Easily ingest from new sources
• Monitor dynamic ecosystems
• Ability to interact with your data
– Aggregations and visualizations
– Different views based on who is looking
• Rich and flexible alerting
• Long term, reliable storage
• Bonus points for full Observability
Needs from a monitoring solution
Core features and functionally

5. 5
Ingesting Metrics to
Elastic

6. 6

7. 7
Instructions
right in Kibana
Growing list of integrations
● Download and install
Metricbeat
● Edit the configuration for
destination
● Enable and configure the
module
● Start the beats
● Explore!

8. 8
● Deploy Elastic Agent
● Choose the integration type
● Register and configure the data
source
● Specify the data you want to
collect
● Explore!
Elastic Fleet
Centralized ingest and configuration

9. 9
Use your existing shippers
Core features and functionality
Your App
Prometheus
Exporter
Your App
Prometheus
Exporter
Metricbeat +
Elasticsearch
Prometheus
Server
Metricbeat +
Elasticsearch Azure Monitor

10. 10
Autodiscover
Automatically monitor new containers
● Perfect for dynamic ecosystems
● Automatically picks up new
instances
● Works with K8s, Docker, AWS, etc.
● Hints based auto-discovery for K8s
● Full context backed by Elastic
Common Schema

11. 11
Elastic for time series

12. Storing Metrics in Elasticsearch
● Metrics stored as numeric fields
○ Depending on expected values:
float, double, integer...
● Dimensions/labels normally stored
as keyword
● Several metrics per document
○ more efficient
○ one doc per combination of
dimensions (time series)
{
"@timestamp": "2018-09-27T10:08:38",
"system": {
"cpu": {
"nice": 8,
"user": 2,
},
“load”: 1.2,
},
"host": "frontend01.bigorg.dev",
"zone": “europe-west”,
...
}
Data model

13. Storing Metrics in Elasticsearch
{
"@timestamp": "2018-09-27T10:08:38",
"system": {
"cpu": {
"nice": 8,
"user": 2,
},
"load": 1.2,
},
"host": "frontend01.bigorg.dev",
"zone": "europe-west",
...
}
Correlation

14. 14
Elastic Common Schema
Established, predictable fields

15. ● Several types for numbers
double, integer, float
depending on size needs…
● Distributed Histograms (7.6
● IPs
query by IP/subnet
● Geo
Map your metrics
● Dates
Rich typing and
filtering
Much more than single type
numbers and string labels

16. Powerful aggregations
• Common metric aggs (sum, avg, count, min, max…)
• With more choices on top!
– Mutate data / calculate metrics at query time with scripting
– Grouping is not limited to labels: Geo proximity, filters, ranges

17. Index lifecycle management
Reduce storage costs as data ages
1
2
3
1 2 3
Hot Nodes Cold Nodes
Warm
Nodes
1

18. Rollups
Reduce storage costs as data ages

19. Distributed by design
• Horizontally scalable
• Cross cluster search
• Cross cluster replication
Easy to scale

20. 20
Powerful data store
Beyond Time Series
● Inverted index + columnar store
● Optimized numeric field types (BKD
● Powerful aggregations framework
● Fast response even for
high-cardinality queries
● ILM & Data Rollups
● With all of the benefits of the
Elastic Stack

21. 21
Making metrics
actionable with Elastic

22. 22
Dashboards &
Visualizations
Out-of-the-box visibility
● Ship with most integrations
● Mix and match for your needs
● Leverage Kibana drilldowns for
custom navigation paths
● Of course, dedicated Metrics
and Logs apps

23. 23
Metrics App
Birds-eye view or drill down

24. 24
Integrated Alerting
Automatically detect and alert
● Many types of alerts
● Prefiltering based on context
● Multiple facets per alert
○ CPU and Memory
○ Network TX and RX
● Automatically split alerts on
chosen field (per
container/pod/host)
● Deviations in logging rates

25. 25
Machine Learning
Automatically detect and alert
● Automate anomaly detection at
scale and across disparate data
sources
● Find patterns in your logs
● Automatically call out anomalies
and outliers

26. 26
Full Observability
Unified data, UI and alerting

27. 27
Thank You!