My contribution to the "Grafana & Friends" Meetup. This presentation goes into the context in the Observability landscape, the basics of OpenTelemetry with its signals and lookout what to expect next.

1. Grafana & Friends Munich
OpenTelemetry and Kafka: from front- to backend with OpenTelemetry

2. $ whoami - Sebastian
Sebastian Poxhofer
SRE @ N26 & Freelancer
Github: @secustor
Open Source Enthusiast & Contributor

3. Agenda
- Context
- Concepts
- Lookout
- Demo
- QA

4. Why should I care?

5. What’s out there? / The state of observability
- Microservices lead to a more complex debugging process
- OSS
- Observability pillars are separated
- Proprietary products
- Heavy vendor lock in, especially if SDKs are used
- Public clouds
- Mostly support only their own products
- Limited imports available
- Product changes require system redesigns
- Metadata is not standardized

6. Origins

7. One protocol to rule them all
The pillars of observability (signals):
- Traces ( GA )
- Metrics ( GA )
- Logs ( Protocol is stable )

8. The container for monitoring?

9. But why OpenTelemetry and not Y?
- Semantic conventions for metadata
- Open standard
- Supported by :
- 300+ companies
- 2,000+ contributors
- 100,000+ contributions in the last year
- CNCF ( incubating )
- FAANG ( MAANA? 🤔) + M

10. CONCEPTS

11. Distributed traces
- Trace
- TraceId
- Span
- SpanId
- ParentSpanId
- Attributes
- Start- and Endtime
- Events
- Timestamp
- Attributes
- Baggage
- Attributes

12. Context propagation
- HTTP/2
- GRPC
- Messaging
- Depends on instrumentation
- Kafka
- RabbitMQ
- ...

13. Metrics
- Metrics
- Name
- Attributes
- Value type ( integer, float, … )
- Unit of measurement
- Types
- Sum
- Gauge
- Histogram
- Exponential Histogram

14. Logs
- Severity
- TraceData ( TraceId & SpanId )
- Timestamp
- Attributes
- Fields of structured logs
- Exception data
- Body

15. Components
- Open Telemetry Protocol ( OTLP )
- SDKs
- Instrumentation libraries
- Auto instrumentation
- Manual instrumentation
- Collector
- Kubernetes Operator
- Specification ( standards for metadata )

16. Instrumentations
Supported Languages:
- Golang
- Java
- JavaScript/TypeScript
- .NET
- Swift
- C++
- Python
- ...
Distributions:
- Core
- Contrib
- Instrumentation

17. The collector pipeline

18. Collector components
Receiver:
- OTLP
- AWS ( xRay, Container
Insights )
- GCP ( Sub & Span)
- Jaeger/Zipkin
- Prometheus
- Filelog
- Kafka
- …
Processor:
- Batch
- MemoryLimiter
- GroupByAttribute
- GroupByTrace
- Filter
- SpanMetrics
- K8Attributes
- ...
Exporter:
- OTLP
- AWS ( xRay, CloudWatch, …)
- GCP
- AzureMonitor
- Jaeger/Zipkin
- Kafka
- Loki
- Splunk
- …

19. What’s next?

20. Status
- Auto instrumentation available
- Java
- JS
- Python
- Ruby
- Opentelemetry Operator in development
- Java, Python, NodeJS agent injection possible
- Horizontal autoscaling for Prometheus ingestion
- Prometheus CRD support

21. Lookout
- Agent Management Protocol
- eBPF tracing
- Profiling signal
- .NET auto instrumentation

22. Links
- Instrumentations: https://opentelemetry.io/registry/
- Collector
- Receivers:
https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver
- Exporters:
https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter
- Demo: https://github.com/open-telemetry/opentelemetry-demo

23. Demo

24. Q&A