The document discusses the complexities of unified observability in cloud-native environments, emphasizing the need for centralized logging, metrics, and application performance monitoring (APM). It highlights the evolution of monitoring architectures and the transition from traditional licensing models to more scalable options that integrate various data sources. Additionally, it underlines the importance of a unified data layer and machine learning for actionable insights in managing application performance and operational efficiency.

1. Tanya Bragin
Senior Director, Product Management
April 2020
Logs, Metrics, and APM for
Unified Observability

3. Higher resource utilization
increases monitoring complexity
• Orchestration/Hypervisor
• Dynamic/ephemeral jobs
• You can no longer "point" to where
that job lives
Shift to cloud-native yields
maintainable code, with costs
• Traditional licensing models don't
scale as well as your applications
• Hurdles with autoscaling
Monitoring Complexity
Hardware & software trends are evolving in tandem
Evolving Architectures ~↑ Monitoring Complexity

4. Applications VMs/Containers
Other DBs,
Services &
Middleware
Orchestration InfrastructureUptime
Metrics
Logs
Network
Network
APM Metrics
APM Logs
APM
APM
Metrics
Logs
Network
Metrics
Logs
Network
APM
Network
APM
Uptime

5. Development
& DevOps Teams
Log
Monitoring Team
Interface status
Flows (Netflow, sFlow,
IPFIX)
Real traffic (packet
analysis)
Network Tool
Infra
Monitoring Team
Web Logs
App Logs
Database Logs
Container Logs
Middleware Logs
Log Tool
Network
Monitoring Team
Real User Monitoring
Txn Perf Monitoring
Distributed Tracing
Uptime
Response Time
APM & Uptime Tools
Container Metrics
Host Metrics
Database Metics
Network Metrics
Storage Metrics
Metrics Tool
Status Quo: Siloed Collection of Tools

6. How many tools does your org
currently use for monitoring
your systems?

7. APM & Uptime NetworkMetricsLogs
Elastic Approach to Observability
Interface status
Flows (Netflow,
sFlow, IPFIX)
Real traffic (packet
analysis)
Web Logs
App Logs
Database Logs
Container Logs
Middleware Logs
Real User Monitoring
Txn Perf Monitoring
Distributed Tracing
Uptime
Response time
Container Metrics
Host Metrics
Database Metics
Network Metrics
Storage Metrics
Elastic Common Schema

8. Unified User Interface
Same UI for KPI summaries and root-cause analysis

9. Unified Data Layer with Common Schema
Open data keeps your data out of silos and delivers maximum business value
• Ship from anywhere — and correlate across your data these sources
• The data is yours — no API rate limiting, no data black boxes
• Cloud native scale — no constraints on dimensions and cardinality

10. Correlate all data sources with unified machine learning and anomaly detection
Unified Machine Learning and Alerting

11. APM & Uptime NetworkMetricsLogs
Elastic Approach to Observability
Interface status
Flows (Netflow,
sFlow, IPFIX)
Real traffic (packet
analysis)
Web Logs
App Logs
Database Logs
Container Logs
Middleware Logs
Real User Monitoring
Txn Perf Monitoring
Distributed Tracing
Uptime
Response time
Container Metrics
Host Metrics
Database Metics
Network Metrics
Storage Metrics
Elastic Common Schema

12. Elastic Stack for logs

13. Adopt an open approach to centralized logging
Turnkey data ingestion, intuitive search interface

14. Make logs actionable with machine learning
Improve analyst efficiency: 10,000 foot view to a single log line

15. Turn log events into intelligence
Real-time dashboards based on log data, at scale

16. Meet audit requirements with log lifecycle management
Index lifecycle management
Policy based data management that optimizes
your cluster behind the scenes
Hot. Warm. Cold. Frozen.Log archival and re-hydration
Robust snapshot management via API or
Snapshot Management UI
Cold storage with online search
Specialized indices for efficient long-term
retention of logs
You’re in control of how your data is tiered

17. Elastic Stack for metrics

18. Elastic Stack as a Metrics Store
BKD trees
Data structures optimized for numerical time
series analysis.
Columnar storage
Structured data storage, resulting in compact
storage and faster analytics
Rollups and Index Lifecycle Management
Aggregate older data into bigger time buckets
Aggregations framework
Analytics features to slice and dice data along
various dimensions
2012
2016
2014
2018
Prometheus support
Support for ingesting data from Prometheus
exporters and servers
2019
Improved support for histograms
Dedicated histogram data type in
Elasticsearch
2020

19. Turnkey data on-boarding
100s of data sources at your fingertips

20. Turn metrics into intelligence
Flexible time-series analytics and data visualization

21. Make logs more valuable with metrics
From KPIs to logs

22. Combine SLA monitoring with logs
Easy-to-consume interface, unified with the rest of observability data

23. Make your data actionable with alerting
In-context alert creation

24. Make your data actionable with alerting
In-context alert creation

25. Make your data actionable with alerting
In-context alert creation

26. Elastic Stack for network and APM

27. 27
Elastic for Network
Packetbeat joins Elastic
Added real-time packet analytics to the stack
2016
2017
Elastic adds support for SNMP
Network device and interface health
information
2019
2018
Elastic adds support for Netflow
Out of the box support for flow-level visibility
From interfaces to flows to packets
Add Network view in the SIEM app
Network activity view relevant for security and
observability alike

28. 28
Elastic APM
Elastic joins forces with Opbeat
A next-generation APM solution designed for
developers
2017
2018
Distributed tracing
Auto-instrumentation and support for
OpenTracing, W3C Trace Context header
2020
2019
Elastic APM GA & more agents
Agents for Python, Node.js, Ruby, Javascript;
Real User Monitoring, Java, …
Enterprise-ready free and open APM
● Java
● .NET
● Node.js
● Javascript
Language Support
● Python
● Ruby
● Go
● PHP (in dev)
• Turnkey agents
• Auto-instrumentation for common
frameworks
• Designed to be lightweight

29. 29
Elastic APM
Elastic joins forces with Opbeat
A next-generation APM solution designed for
developers
2017
2018
Distributed tracing
Auto-instrumentation and support for
OpenTracing, W3C Trace Context header
2020
2019
Elastic APM GA & more agents
Agents for Python, Node.js, Ruby, Javascript;
Real User Monitoring, Java, …
Service Map, annotations
Fully features user interface for navigating
APM data
Enterprise-ready free and open APM

30. Avoid lock-in with open source APM agents
Support for open standards - Jaeger, OpenTracing, OpenMetrics, W3C Trace context

31. Track transactions from browser to backend
End-to-end distributed tracing

32. Reduce MTTR by streamlining analyst workflow
Navigate traces, metrics, and logs in one UI for faster issue resolution

33. Understand your dependencies in real time
Dependency mapping

34. Get more value from your trace data
Flexible data retention for detailed traces per application class
• Stop throwing away valuable traces
before they were analyzed
• Apply machine learning to detailed trace
data to gain insights
• Set up data retention policies per
application class to contain costs

35. 35
Demo

36. What now?
Try it yourself!

37. Next up: Workplace Search