The shift from monolithic applications to microservices led to many new challenges we haven’t had before. Especially analyzing problems and tracking down erroneous components of a distributed system has become much more difficult as slicing and decoupling applications advances. We now have to answer questions like: How do we find out which services were involved when processing a specific request and how long did it take to respond? How do we figure out which service is causing a request to fail and why? These issues are addressed by Distributed Tracing tools like Zipkin, Jaeger, OpenTracing and OpenCensus. But how can we leverage the data we are gathering using such tools to gain new insights into our business processes, instead of just focussing on the technical aspects?

1. DISTRIBUTED TRACING
TAKING
TO THE NEXT LEVEL

2. WHO ARE WE?
▸ Frank Pfleger
@frankpfleger
frank@trasier.com
▸ Lukasz Pielak
@lukaszpielak
lukasz@trasier.com

3. WHAT IS THIS TALK ABOUT
▸ The Fundamental Concepts
▸ The Current State
▸ The Next Level
TAKING DISTRIBUTED TRACING TO THE NEXT LEVEL

4. LOGGINGMETRICS TRACING
OBSERVABILITY

5. LOGGING

6. LOGGING

7. LOGGING

8. LOGGING - MDC
Source: https://www.baeldung.com/mdc-in-log4j-2-logback

9. TRACEABILITY OF SOFTWARE
MONOLITHIC
ARCHITECTURE

10. TRACEABILITY OF SOFTWARE
MICROSERVICES

11. TRACEABILITY OF SOFTWARE
MDC CONTEXT PROPAGATION
TK TK TK
TKTK
TK
TK
TK TK

12. DISTRIBUTED
TRACING

13. DISTRIBUTED TRACING
BRINGING EVENTS IN
CAUSAL ORDER
IN
DISTRIBUTED ENVIRONMENT

14. DISTRIBUTED TRACING
Span TracerTrace Context
(span id, trace id)
Trace

15. DISTRIBUTED TRACING
WHY DISTRIBUTED TRACING?
▸ Latency visualization
▸ Service dependency visualization
▸ Hidden architecture revealed
▸ Error analysis
▸ Infrastructure check
▸ Version check

16. ▸ Trace Context propagation
▸ in-process
▸ inter-process
▸ Clock skew / synchronization
▸ Performance and stability
▸ Data ingestion, storage, retention
▸ GDPA
DISTRIBUTED TRACING
PROBLEMS OF DISTRIBUTED TRACING
SERVICE B
SERVICE A
Latency measurement points
Host 1
Host 2

17. CORRELATE LOG ENTRIES
TRACE COMMUNICATION BETWEEN SERVICES
USE DISTRIBUTED TRACING
USE TRACE-ID IN THE MDC
DON’T SILENTLY SWALLOW EXCEPTIONS

20. THERE IS ALWAYS AN ALTERNATIVE ;)

21. THE CURRENT STATE OF
DISTRIBUTED TRACING

22. BOOKING PAYMENT
USER INTERFACE
Live Reporting
(Performance monitoring)
OFFER
DATASTORE
BigData Store
(High-Performance / Indexing)
WEBSHOP
Context propagation
(Service Instrumentation)
Common language
(Specification / Trace-Context)
COLLECTOR Decoupled Reporter
(Asynchronous collecting)

23. APM - PROVIDERS

24. TOOLING LANDSCAPE

25. GOAL: MAKE OBSERVABILITY EASY FOR MODERN APPLICATIONS
vendor-neutral API
wide vendor support
tracer implantation effort
https://opentracing.io/docs/overview/what-is-tracing/

26. GOAL: MAKE OBSERVABILITY EASY FOR MODERN APPLICATIONS
libraries used for collecting tracing and metrics data
ready to use implementation for database engines, kubernetes etc.
tightly coupled to the implementations
opinionated implementation for capturing observability signals
COMPLEMENTARY NOT CONTRADICTORY
vendor-neutral API
wide vendor support
tracer implantation effort

27. GOAL: MAKE OBSERVABILITY EASY FOR MODERN APPLICATIONS
libraries used for collecting tracing and metrics data
ready to use implementation for database engines, kubernetes etc.
tightly coupled to the implementations
opinionated implementation for capturing observability signals
COMPLEMENTARY NOT CONTRADICTORY
vendor-neutral API with default implementations
+ =
vendor-neutral API
wide vendor support
tracer implantation effort

28. GOAL: MAKE OBSERVABILITY EASY FOR MODERN APPLICATIONS
https://github.com/open-telemetry/opentelemetry-java
https://www.w3.org/2018/distributed-tracing/
https://github.com/w3c/trace-context

29. DEMO
https://github.com/trasiercom/springboot-example/branches

30. TAKING
DISTRIBUTED TRACING
TO THE NEXT LEVEL

31. DISTRIBUTED TRACING
credits to jaeger documentation

32. DISTRIBUTED TRACING

33. THE NEXT LEVEL
BUSINESS TRACING
EASY BUG TRIAGE
BUSINESS ANALYSIS
PREDICTIVE ANALYSIS
MOCK SERVICES
REPLAY CONVERSATIONS

34. BUSINESS TRACING
CONVERSATION AS A WHOLE

35. CUSTOMER CHARGED TOO MUCH?
INSPECT PAYLOAD OF TRANSACTIONS

36. ON-THE-FLY ERROR DETECTION
https://ui.trasier.com/#/account/170522/space/nova-test/search/id/267cd7b7-2d6a-474b-ade2-bf0453881f72?spanId=f53e4982-a7c0-4117-b98c-91260eefbdbb&view=editor&editortab=response

37. DEMO
https://github.com/trasiercom/springboot-example/branches

38. THE NEXT LEVEL
EASY BUG TRIAGE
https://ui.trasier.com/#/account/123456/space/sys-test/search/id/267cd7b7-2d6a-474b-ade2-bf0453881f72
Allocating bugs to the responsible team can be
tricky when there is only a functional description.

39. THE NEXT LEVEL
MOCK SERVICES
OFFER THIRD PARTY SERVICE
TEST-CLIENT
X-CID=ABC123
HEADER
Testing can be hard when you’re integrating with many services.
Especially unreliable integration systems can increase the development
costs unexpectedly.
TEST 1 | MOCK ID=ABC123
BUSINESS TRACING BACKEND

40. THE NEXT LEVEL
MOCK SERVICES
Testing can be hard when you’re integrating with many services.
Especially unreliable integration systems can increase the development
costs unexpectedly.
OFFER THIRD PARTY SERVICE
TEST-CLIENT
MODE=READ
HEADER
X-CID=ABC123
TEST 1 | MOCK ID=ABC123
BUSINESS TRACING BACKEND

41. THE NEXT LEVEL
MOCK SERVICES
Change on the integrator side (Bug, Error, API Change)
Probably bug in our system
Run tests against real backend and against mock at the same time
Test against mock50 (100%) SUCCESS, 0 (0%) ERROR
Test against real backend40 (80%) SUCCESS, 10 (20%) ERROR
Test against real backend40 (80%) SUCCESS, 10 (20%) ERROR
Test against mock40 (80%) SUCCESS, 10 (20%) ERROR

42. THE NEXT LEVEL
REPLAY CONVERSATIONS
OFFER THIRD PARTY SERVICE
REPLAY-CLIENT
BUSINESS TRACING BACKEND
MODE=REPLAY
HEADER
X-CID=ABC123

43. THE NEXT LEVEL
BUSINESS ANALYSIS
The data stored by the business tracing system is the perfect foundation for
extensive research and data analysis. Gathering the equivalent information from
your existing data stores is expensive and may not even be possible.

44. THE NEXT LEVEL
BUSINESS ANALYSIS
The data stored by the business tracing system is the perfect foundation for
extensive research and data analysis. Gathering the equivalent information from
your existing data stores is expensive and may not even be possible.

45. THE NEXT LEVEL
PREDICTIVE ANALYSIS
The chance is to finally take the step to predictive analysis on your actual
business data and processes.
Offer creation errors during the day per product id.
Offer creation errors during the day.

46. BUSINESS OBSERVABILITY
IS AS IMPORTANT AS
TECHNICAL TRACING

47. Q & MAYBE A;-)
▸ Lukasz Pielak
@lukaszpielak
lukasz@trasier.com
▸ Frank Pfleger
@frankpfleger
frank@trasier.com

48. ▸ Trasier
▸ https://trasier.com
▸ https://github.com/trasiercom/springboot-example/branches
▸ OpenTracing
▸ https://opentracing.io
▸ OpenCensus
▸ https://opencensus.io
▸ OpenTelemetry
▸ https://opentelemetry.io
▸ Zipkin
▸ https://zipkin.io
▸ Jaeger
▸ https://jaegertracing.io
THANK YOU
adesso Schweiz AG
Bahnhaldenstrasse 7
CH-8052 Zurich
T +41 58 520 98 00
adesso Schweiz AG
Morgenstrasse 129
CH-3018 Bern
T +41 58 520 97 00
adesso Schweiz AG
Viaduktstrasse 8
CH-4051 Basel
T +41 58 520 97 20
info@adesso.ch
www.adesso.ch