"Can you determine how a given event came to be? Is it an aggregation, a combination of multiple events with different sources? What are its origins? As event driven architectures become more sophisticated, with features such as stateful stream processing, data joining, and multi-cluster flows, it becomes harder to trace the path of an event, its origins and touch points. At the same time, it also becomes more important. Using code examples and usage scenarios we will dive into the tracing capabilities of OpenTelemetry for Kafka clients, including those using the Consumer/Producer and Kafka Streams libraries, as well as the Connect and ksqlDB platforms. This will culminate in an end-to-end tracing pipeline demonstration. This talk will cover the following topics: - Distributed tracing concepts, including context propagation and the OpenTelemetry implementation stack - OpenTelemetry’s Kafka instrumentation, what is supported out of the box, code examples, edge cases, challenges and solutions - A demonstration of an end-to-end tracing implementation In this session, you will gain an understanding of the importance of end-to-end traceability, and several tools & examples for improving observability in your own distributed event driven applications."

1. A Practical Guide To End-to-End
Tracing In Event Driven Architectures
by Roman Kolesnev

2. Roman - UK developer at PIE Labs
PIE Labs, Confluent
• What is Distributed Tracing ?
• What is OpenTelemetry?
• Kafka Client instrumentation
• Kafka Streams instrumentation
• ksqlDB instrumentation
• Kafka Connect instrumentation
Who we are, what we’ll talk about…

3. Components of a DT system
• Instrumentation
• Collection
• Visualisation
https://blog.gurock.com/distributed-tracig/

4. What makes up a trace?
https://docs.logz.io/user-guide/distributed-tracing/what-is-tracing

5. Why Distributed Tracing?
Adding context to the message and process flow.
• Dependency graph
• Record of Event flow
• Log correlation
• Contextual metrics
• Answer questions like:
“This result looks weird. Show me all the intermediate
states, so I can debug where the weirdness started…”

6. https://drive.google.com/file/d/1qZ0XlYK_H0RWisRVJ8UugZWawVk2mOnM/view

7. Context and Context?
• Remote Trace Context

8. Context and Context?
• Local Trace Context

9. Section: What is OpenTelemetry?
• What is Distributed Tracing (DT)?
• What is OpenTelemetry?
• Kafka Client instrumentation
• Kafka Streams instrumentation
• ksqlDB instrumentation
• Kafka Connect instrumentation

10. Overview of OpenTelemetry
1. Standardised, vendor-agnostic
2. High-quality, ubiquitous, and portable
3. Collection of tools, APIs, and SDKs
4. Instrument, generate
5. Collect, and export
6. To an observability back-end (not OT - e.g. Jaeger)
7. To help you analyze your software’s performance and behavior

11. Support for Kafka in OpenTelemetry
Kafka Clients:
• Javaagent
• Tracing Wrappers
• Tracing Interceptors
Kafka Streams:
• Javaagent
• Supply Kafka Clients with Tracing
OpenTelmetry instrumentation

12. • What is Distributed Tracing (DT)?
• What is OpenTelemetry?
• Kafka Client instrumentation
• Kafka Streams instrumentation
• ksqlDB instrumentation
• Kafka Connect instrumentation
Section: Kafka Client instrumentation

13. Tracing Consumer Producer Applications
Application Flow
Trace

14. Tracing Consumer Producer Applications
Kafka Receive Telemetry Enabled

15. Tracing Consumer Producer Applications
Kafka Receive Telemetry Enabled

16. Tracing Consumer Producer Applications
Kafka Receive Telemetry Enabled

17. Tracing Consumer Producer Applications
Kafka Receive Telemetry Disabled

18. Tracing Consumer Producer Applications
Kafka Receive Telemetry Disabled

19. Tracing Consumer Producer Applications
Kafka Receive Telemetry Disabled

20. Comparison of instrumentation methods
Interceptor / Wrapper
Javaagent

21. • What is Distributed Tracing (DT)?
• What is OpenTelemetry?
• Kafka Client instrumentation
• Kafka Streams instrumentation
• ksqlDB instrumentation
• Kafka Connect instrumentation
Section: Kafka Streams instrumentation

22. Kafka Streams support
• Kafka Streams specific Process span implementation in Javaagent
• Stateless Kafka Streams processing pipelines
• Limitations:
• Non-javaagent implementation limitations
• Stateful operation support
• Caching in stateful operations

23. Kafka Streams support - Stateless
Javaagent
Interceptor / Wrapper

24. Kafka Streams support - Stateless
Javaagent
Interceptor / Wrapper

25. Kafka Streams - Stateful - No Caching

26. Kafka Streams - Stateful - No Caching - Aggregate Span

27. Kafka Streams - Stateful - No Caching - Aggregate Span

28. Kafka Streams - Stateful - No Caching - Aggregate Span

29. Kafka Streams - Stateful - No Caching - Aggregate Span

30. Kafka Streams - Stateful - No Caching - Wrapped Store

31. Kafka Streams - Stateful - No Caching - Wrapped Store

32. Kafka Streams - Stateful - No Caching - Wrapped Store

33. Kafka Streams - Stateful - Caching

34. Kafka Streams - Stateful - Caching - Aggregate

35. Kafka Streams - Stateful - Caching - Aggregate

36. Kafka Streams - Stateful - Caching - Aggregate

37. Kafka Streams support - summary
• Supported as is with Javaagent:
• Stateless
• Stateful - with limitations - single thread context, - no caching
• Wrapping State Store approach
• Inlining Span creation into Stateful operations

38. • What is Distributed Tracing (DT)?
• What is OpenTelemetry?
• Kafka Client instrumentation
• Kafka Streams instrumentation
• ksqlDB instrumentation
• Kafka Connect instrumentation
Section: kSQL instrumentation

39. ksqlDB support - summary
• ksqlDB runs Kafka Streams jobs under the hood
• Using Javaagent with ksqlDB - limitations
• Trace grouping implications

40. kSQL support - stateless

41. kSQL support - stateful

42. kSQL support - stateful

43. Javaagent extension - Principles
• Aspect oriented programming (AOP)
• ByteBuddy and code injection
• Service provider interface (SPI)

44. Javaagent extension - Injection of state store

45. Javaagent extension - Injection of state store

46. https://drive.google.com/file/d/1OXMxYNKSClM3RKs4_cRzCWZ1Aj21VSrN/view

47. Javaagent extension - SPI
• SPI extension points
• Span exporter wrapping for service name override
• Examples of other useful extensions (attribute propagation, span
suppression)

48. kSQL support - stateful - service name cleanup

49. kSQL support - stateful - service name cleanup

50. • What is Distributed Tracing (DT)?
• What is OpenTelemetry?
• Kafka Client instrumentation
• Kafka Streams instrumentation
• ksqlDB instrumentation
• Kafka Connect instrumentation
Section: Kafka Connect instrumentation

51. Kafka Connect support - Source Task

52. Kafka Connect support - Source Task

53. Kafka Connect support - Source Task

54. Kafka Connect support - Source Task

55. Kafka Connect support - Sink Task

56. Kafka Connect support - Sink Task

57. Kafka Connect support - Sink Task

58. Thank you!
confluent.io/community/ask-the-community
rkolesnev@confluent.io
https://github.com/rkolesnev/kafka-opentelemetry