Jason Schmidt and Elijah Zupancic, Solutions Architects for NGINX, dive deep into the work behind adding OpenTelemetry to the MARA project. They describe how to add instrumentation to a legacy microservices application, integrate a microservices application with existing OpenTelemetry instrumentation, and deploy a collector architecture in a way that balances flexibility, ease of management, and future upgrade paths. The session is designed to help demystify OpenTelemetry and provide working examples that illustrate best practices within the MARA framework. Speakers: Elijah Zupancic Jason Schmidt

1. MARA
The Journey to OpenTelemetry
Jason Schmidt Solutions Architect
Elijah Zupancic Solutions Architect

2. ©2022 F5
2

3. ©2022 F5
3
A Brief Introduction to
OpenTelemetry

4. ©2022 F5
4
OpenTelemetry Benefits
Standards-based agents
Cloud integration
Automated code instrumentation
Support for developer frameworks
Any code, any time
No cardinality limits

5. ©2022 F5
5
OpenTelemetry Core Concepts
● Tracing
○ Built from spans
○ Emitted by services
○ What is happening for a given operation
● Metrics
○ A measurement about a service, captured at runtime
○ Consists of counters, measures, and observers
● Logging
○ A timestamped text record
○ Can be structured or unstructured
● Baggage
○ Key/Value format
○ Contextual data
○ Passed between spans

6. ©2022 F5
6
OpenTelemetry Core Components
● Specification
○ OTLP Protocol
○ API Definition
○ SDK Requirements
● Collector
○ Vendor agnostic
○ Receive, process, export
● Language SDK
● Automatic Instrumentation

7. ©2022 F5
7
Current Component Status
API SDK Protocol
Tracing Stable Stable Stable
Metrics Stable Mixed Stable
Logging Draft Draft Beta
Baggage Stable* Stable n/a

8. ©2022 F5
8
MARA: Telemetry (So Far)

9. ©2022 F5
9
Where We Started
● Application: Bank of Sirius
○ Forked from Google’s Bank of Anthos
○ Configured to use Google CloudTrace
● Overall Challenges
○ Add management/monitoring
● Telemetry Challenges
○ Move away from CloudTrace
○ Determine how to best instrument and collect traces
○ Collect metrics
○ Collect logs

10. ©2022 F5
10
Considerations
● Avoid Lock In
○ Ability to switch between observability technologies
● Ease of Use
○ Reduction in friction for implementation
○ Automated instrumentation when possible
● Visualization Tooling
○ Ability to use and correlate data to make decisions
● Resource Use
○ How much to dedicate to telemetry?

11. ©2022 F5
11
Our Observability Wishlist
● Logging
● Tracing
● Metrics
● Health checks
● Runtime state introspection
● Heap/core dumps
● Exception/error aggregation and notification

12. ©2022 F5
12
Our Observability Options

13. ©2022 F5
13
The Winner….
● OpenTelemetry solves the bulk of our issues
● However…
○ Some components are not fully stable
○ Not many production examples
○ Overall “newness”
● The Solution….
○ OpenTelemetry tracing / collector
○ Other technologies as a stop-gap
○ Also allows for benchmarking
○ OTEL vs other solutions

14. ©2022 F5
14
Where We are Today
Tracing
• OTEL Collector
• OTEL SDK in Bank of Sirius
Logging
• Elasticsearch
• Filebeat
• Kibana
Metrics
• Prometheus
• StatsD
• Postgres Prometheus Exporter
Visualization
• Jaeger
• Grafana

15. ©2022 F5
15
MARA: Telemetry (The Future)

16. ©2022 F5
16
MARA and OpenTelemetry: The Vision
Integration of
OTEL tracing
into MARA
1
Convert
Metrics to use
OTEL
2
Convert Log
Management
to use OTEL
3
Operationalize
OTEL
4

17. ©2022 F5
17
Other Areas of Interest
Deploy
OpenTelemetry
Demo Web Store
on MARA
Send OTEL data
to third party
services
Integrate OTEL in
other parts of the
MARA framework

18. ©2022 F5
18
Community Engagement
● Track and adopt OpenTelemetry enhancements
● Extend OpenTelemetry reach to…
○ Other MARA components
○ Other NGINX projects
● Contribute to OpenTelemetry projects
● Provide feedback on experience
○ Blogs
○ Talks
○ Videos

19. ©2022 F5
19
Challenges

20. ©2022 F5
20
Known Challenges
● Maturity of components
○ Some components still in beta
○ Early in life cycle of other components
● Instrumentation type
○ Manual versus Auto instrumentation
● Operational concerns
○ Resource concerns (cpu / mem)
○ Performance impact
○ Long term data storage concerns
● Standards / Best Practices