With great distribution comes also great responsibility. The usage of multiple (micro-)services increases the risk of losing the overview of traceability of errors and lagging performance. With the help of Sleuth and Zipkin, we have some helpful instrumentation tools which can provide a helping hand in tracing down slow performance issues. In this session, we will explore how we can setup Sleuth into your Spring Boot services and use it into your logging monitoring to add some extra overview in your services. Next, we will use Zipkin to find the culprit which is slowing down our services.

1. { Tracing performance with some help of
Sleuth, Zipkin & ELK }
Rafaëla Breed
Amsterdam | April 2-3, 2019

2. Rafaëla Breed
• So#ware engineer @ Luminis Amsterdam
• Java developer
• Spring(Boot)
• Photography, traveling

3. Hmmm… breakfast

4. New company, new application

5. Let’s just start developing our applica4on

6. Everything works fine, but…
…there comes the tester/business/client: !
“It does not work” "
“So, what does not work?” #
“Well, it is slow” $
“All right, do you know which step is being slow exactly?” #
“When I push the toast bu>on, it takes 5 seconds to get the results.
That is way too slow. We have to service our customers as fast as possible and
……” %

7. Microservice
Microservice
Microservice
MicroserviceMicroservice
What do we know?
Breakfast service
Get toasted bread
Machine service
Get toaster
informa/on
Energy service
Get power usage
Temperature
service
Get temperature
Preference service
Get preference of
toast
Request Request
Request
Request
Request
Response
Response
Response Response
Response

8. What do we have?

9. ELK: Elas)cSearch, Logstash, Kibana
• Applica(ons
• Collect
• Parse
• Transform
• Store
• Search
• Analyze • Visualize
• Explore
Maven Logback GELF
dependency

10. Kibana
Log messages of the applications
Fields used
in logs
Filters
Search

11. Let’s try & dive in our logging

12. Summary of demo
• Easy search & filter by ELK
• But… Not easy to track specific events… yet

13. What we would like to have
•Correlation between operations
•Each starting operation => unique id
•Each next operation => unique id

14. What Sleuth can offer
• Depicted from Dapper Infrastructure, Zipkin, Htrace
• Implementable via Maven/Gradle
• Brave is used as tracing library
• Every first remote call starts a trace (aka uber-span)
• Every following remote call starts a span (aka child span)
• Traces are passed to next service(s)

15. Adding Sleuth to your project
• To every SpringBoot app we add:
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<ar@factId>spring-cloud-sleuth</ar@factId>
<version>${spring.cloud.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<ar@factId>spring-cloud-starter-sleuth</ar@factId>
</dependency>
</dependencies>
• Or take a look at
https://cloud.spring.io/spring-
cloud-sleuth/
• Using custom log appender(s)?
Add the properties X-B3-TraceId
and X-B3-SpanId to your
configuration!

16. Trace
Summary
of demo
Span E
Parent Span A
Span D
Parent Span B
Span C
Parent Span B
Span B
Parent Span A
Span A
Request Request
Request
Request
Request
Response
Response
Response Response
Response
Breakfast service
Get toasted bread
Machine service
Get toaster
informa/on
Energy service
Get power usage
Temperature
service
Get temperature
Preference service
Get preference of
toast

17. Summary of demo

18. Zipkin
• Developed by Twitter in 2012
• Adapted and open-sourced in 2015 as OpenZipkin
• Distributed tracing system
• Trace down your system(s)

19. Adding Zipkin client to your project
• To every SpringBoot app we add:
<!-- Assuming you already added the spring cloud sleuth dependency in
your dependencyManagement -->
<dependency>
<groupId>org.springframework.cloud</groupId>
<arDfactId>spring-cloud-starter-zipkin</arDfactId>
</dependency>

20. Sampler for Zipkin
• Do you want to put each request into Zipkin?
• Up-front decision
• Probability 0…1 à chance 0 to 100%
• Default = ProbabilityBasedSampler 0.1
• ALWAYS_SAMPLE
• Custom declaraHve sampling
• Custom sampling on request
• Rate-limiHng sampler
• X-B3-Flags overrides (client request header)

21. Getting a Zipkin server
• Github project
• Docker images (production ready)
• Cloud solutions for AWS/Azure available
Zipkin Server
Collector Storage API UI

22. Summary of demo
• The Zipkin client sends enriched information of Sleuth to your Zipkin
Server
• The Zipkin Server provides a user interface in which you can overview
the details of the events logged by Sleuth/Zipkin Client

23. Summary of demo
• The user interface with clear graphics of events with 2mes

24. Overview what will happen (a bit under the hood)
Spring boot applica.on
Spring Cloud Sleuth
Spring Cloud Zipkin
Brave’s
Tracing
Zipkin’s
Reporter
Traces, span
Zipkin server
Sampler
Zipkin’s
Sender ->
[KAFKA, RABBIT, WEB]

27. Our Cucumber tests
• TDD, business scenario’s
• Logs tests & apps available

28. Our test log

29. Our test report

30. Our application logs

31. “Über-trace” aka conversa1on
Trace Microservice
Trace
Trace
Our future wishes
Request
Response
Microservice
Breakfast service
Get toasted bread
+
Request
Microservice
Response
Breakfast service
Get fried egg
+
Request
Response
Breakfast service
Get fried bacon
Trace
RequestResponse
Microservice
Breakfast service
Get coffee

32. Conversa)on ID specs
• Client may pass request header as X-B3-ConvID
• If not passed à application creates it
• Subsequent calls of services receive it & use it
• Loggable & passed to Zipkin à searchable
• Client response header has X-B3-ConvID

33. Adding a custom tracefilter
• Extend your class with GenericFilterBean
• Ensure that the @Order is before or a'er
TraceWebServletAutoConfigura>on.TRACING_FILTER_ORDER,
dependent on what you want to provide

34. Propaga'on fields
• Or bagage
• Addable fields passed through services with trace and every span
Trace
Microservice
Span
Microservice
Span
X-B3-CONVID
X-B3-CONVID
X-B3-CONVID

35. Tags
• Addable fields for specific span for details / lookup
• Will be passed to Zipkin
Microservice
Span
Zipkin server
X-B3-CONVID
X-B3-CONVID
X-B3-CONVID

36. Summary of demo
• Manipulate pass through of Sleuth with a custom filter
• Propaga+on fields will be passed to the whole trace
• A tag is used to tag a span & is also passed to Zipkin
• Make conversa<on id or trace id available in your reports!

37. But…
…what about applications without Spring Boot?
…what about applications without Java?
…are there alternatives to Zipkin?

38. Distributed tracing
opencencus.io & opentracing.io

39. Summary
• Easy review of application calls with ELK, Sleuth, Zipkin
• More clear separation of events by traces & spans with Sleuth
• More easy search for performance latencies by visualization in Zipkin
• Extra information by extra custom interceptors/filters
• Logging and tracing will go hand-in-hand
Golden circle of observability

40. Thank you for your ,me!
• Ques%ons?
• Please give feedback in the app
• Presenta%on & code at
h:ps://github.com/RDBreed/sleuthzipkindemo
Observability & serverless?
14:10-14:50 by Yan Cui @ room 2