This document discusses distributed tracing using Spring Cloud and Zipkin. It begins with introductions from the presenters. The agenda then outlines tracing concepts like tracers, tracing systems, and log correlation with ELK. It describes using Spring Cloud Sleuth and Zipkin on Pivotal Cloud Foundry. New features are highlighted and a demo is planned. Metrics integration with PCF is also discussed. The goal is to help with latency analysis for microservices.

1. Distributed Tracing : Latency analysis for
your microservices using Spring Cloud &
Zipkin
By Reshmi Krishna
@reshmi9k
1
By Marcin
Grzejszczak
@mgrzejszczak

2. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
About us
● Spring Cloud developer
● Working mostly on Sleuth / Contract / Pipelines
Twitter: @mgrzejszczak
Blog: http://toomuchcoding.com
2

3. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
About us
● Senior Platform Architect, Pivotal Cloud Foundry
● Working with enterprise customers to transform
they way they build software
● Previous Life - Software Engineer on Wall Street
Twitter : @reshmi9k
MeetUp : Cloud-Native-New-York
3

4. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Agenda
● Tracing, Tracers and Tracing Systems
● Log Correlation with ELK
● Demo: Spring Cloud Sleuth, Zipkin on PCF
● New & Noteworthy features
● PCF Metrics
4

5. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 5

6. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Everything is going to be okay!
6

7. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Until it’s not
7

8. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

9. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Let’s Debug

10. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
It doesn’t look like this
Service1
Service2
Service4
Service3
Web
Frontend

11. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
More like this

12. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

13. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Troubleshooting Latency Issues
▪ When was the event? How long did it take?
▪ How do I know it was slow?
▪ Why did it take so long?
▪ Which microservice was responsible?

14. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Distributed Tracing
▪ Distributed Tracing is a process of collecting end-to-end transaction graphs in near
real time
▪ A trace represents the entire journey of a request
▪ A span represents single operation call
▪ Distributed Tracing systems are often used for this purpose. Zipkin is an example
▪ As a request is flowing from one microservice to another, tracers add logic to create
unique trace Id, span Id

15. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Dapper Paper By Google
@reshmi9k
@reshmi9k
This paper described Dapper, which is Google’s production distributed
systems tracing infrastructure
Design Goals :
Low overhead
Application-level transparency
Scalability

16. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Visualization - Traces & Spans
service1
Trace Id : 1, Span Id : 1
service4
Trace Id : 1, Parent Id : 2, Span Id : 4
service2
Trace Id : 1, Parent Id : 1, Span Id : 2
service3
Trace Id : 1, Parent Id : 2, Span Id : 3

17. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Tracer: Spring Cloud Sleuth
● Adds trace and span ids to the Slf4J MDC
● Tracers have instrumentation or sampling policy
● Loosely based on HTrace, but Zipkin (Dapper) compatible
● If spring-cloud-sleuth-zipkin is available then the app will generate and collect
Zipkin-compatible traces via HTTP
18

18. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 19
Log Correlation with ELK

19. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Which server is broken?

20. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
SSH and grep for ERROR to find it?

21. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Log correlation with Sleuth
@reshmi9k
@reshmi9k
We take care of passing tracing information between threads / libraries /
contexts for:
● Hystrix
● Rest Template
● Feign
● WebClient
● WebFlux
● Project Reactor
● Messaging via Spring Integration
● Zuul
If you don’t do anything unexpected there’s nothing you need to do to make
Sleuth work. Check the docs for more info.

22. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Spring Cloud Sleuth logging format
@reshmi9k
@reshmi9k
We set a logging format for you...

23. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Now let’s aggregate the logs!
@reshmi9k
@reshmi9k
Instead of SSHing to the machines to grep logs lets aggregate them!
● With Cloud Foundry’s (CF) Loggregator the logs from different instances
are streamed into a single place
● You can harvest your logs with Logstash Forwarder / FileBeat
● You can use ELK stack to stream and visualize the logs

24. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
SERVICE 1
/start
REQUEST
RESPONSE
SERVICE 2
SERVICE 3
REQUEST
RESPONSE
REQUEST
RESPONSE
SERVICE 4
REQUEST
RESPONSE
“Hello from service3”
“Hello from service4”
“Hello from service2, response from
service3 [Hello from service3] and from
service4 [Hello from service4]”

25. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
SERVICE 1
/readtimeout
REQUEST
BOOM!
SERVICE 2
REQUEST
BOOM!
REQUEST
BOOM!

26. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Log correlation with Spring Cloud Sleuth
DEMO

27. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

28. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

29. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

30. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Introduction to Zipkin
31

31. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Zipkin
● Zipkin is a distributed tracing system
● Implementation based on Dapper paper
● Aggregate spans into trace trees
● Manages both collection and lookup of the data
● In 2015, OpenZipkin became the primary fork

32. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Zipkin Architecture
zipkin.jar

33. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Moving to cloud
34

34. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Integrate distributed tracing to your apps
Spring Cloud
Sleuth
Collector
Span
Store
Transport
Mq/Http/Log
Spring Cloud
Sleuth
Spring Cloud
Sleuth
Spring Cloud
Sleuth
Query
ServerZipkin UI
ZIPKIN
APP
APP
APP
APP

35. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Magic of Sleuth!!!
Build.gradle
1. compile "org.springframework.cloud:spring-cloud-starter-zipkin"
2. compile "org.springframework.amqp:spring-rabbit"
app.yml
1. spring.sleuth.sampler. percentage: 1.0
2. logging.level.org.springframework.cloud.sleuth: DEBUG
36

36. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Let’s Demo

37. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
New And Noteworthy
48

38. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
New and noteworthy - Dalston
● Creating spans via annotations
○ especially useful for interfaces (e.g. Spring Data)
● Baggage
○ Sleuth specific (via baggage_ header prefix)
● Span adjusters
○ list of custom adjusters of spans before reporting

39. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
New and noteworthy - Edgware
● RestTemplateBuilder support
○ the built RestTemplate is tracing aware
● Zipkin load balanced/discovery using Eureka
○ pass the service id in the URL
● Reactor Support
○ tracing context passed through Flux
● Zipkin2 Support
○ starter-zipkin points to Zipkin2
● Deprecation of Stream server
○ migrate to Zipkin2

40. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
New and noteworthy - Finchley
● WebFlux & WebClient support
○ tracing context passed through the Flux
○ WebClient is tracing aware
● Removal of Stream server
○ you should use Zipkin 2 native messaging support
○ migrate by having both the old Stream server and the new Zipkin 2
store spans to the same storage
○ start migrating apps from Sleuth Stream client to Zipkin 2
● Removal of OAuth2 support
○ will be added once it’s re-added in Security

41. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
DEMO
52

42. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
PCF Metrics
53

43. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
PCF Metrics
● Container Metrics: Three graphs measuring CPU, memory, and disk usage
percentages
● Network Metrics: Three graphs measuring requests, HTTP errors, and
response times
● Custom Metrics: User-customizable graphs for measuring app performance,
such as Spring Boot Actuator metrics
● App Events: A graph of update, start, stop, crash, SSH, and staging failure
events
● Logs: A list of app logs that you can search, filter, and download
54

44. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
PCF Metrics
55
Trace Explorer: A dependency graph that traces a request
as it flows through your apps and their endpoints, along with
the corresponding logs

45. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Integrating Distributed Tracing with PCF Metrics
Spring:
● Spring Boot v1.4.3 or later.
● Spring Cloud Sleuth v1.0.12 or later.
● Add the following to your app dependency file:
56
compile "org.springframework.cloud:spring-cloud-starter-sleuth"

46. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Let’s Demo

47. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Summary
● Distributed tracing allows you to quickly see latency issues in your system
● Log correlation allows you to match logs for a given trace
● Zipkin is a great tool to visualize the latency graph and system dependencies
● Spring Cloud Sleuth integrates with Zipkin and grants you log correlation
● PCF brings it all together
61

48. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 62

49. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
THANK YOU !!!
● https://github.com/marcingrzejszczak/vagrant-elk-box/tree/presentation -
code for this presentation (clone and run getReadyForConference.sh -
NOTE: you need Vagrant!)
● https://cloud.spring.io/spring-cloud-sleuth/
Sleuth’s documentation
● https://github.com/reshmik/distributed-tracing-for-pcf.git
Spring Cloud Sleuth & Zipkin on PCF
● https://github.com/openzipkin/zipkin-java
Repo with Spring Boot Zipkin server
● http://docsbrewing-zipkin-server.cfapps.io
Zipkin deployed to PCF for Brewery Sample app
● http://docs.pivotal.io/pcf-metrics/1-4/using.html
PCF Metrics
63

50. Unless otherwise indicated, these slides are © 2013-2017 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-
NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Disclaimer Statement
The following is intended to outline the general direction of Pivotal's offerings. It is
intended for information purposes only and may not be incorporated into any
contract. Any information regarding pre-release of Pivotal offerings, future
updates or other planned modifications is subject to ongoing evaluation by Pivotal
and is subject to change. This information is provided without warranty or any
kind, express or implied, and is not a commitment to deliver any material, code, or
functionality, and should not be relied upon in making purchasing decisions
regarding Pivotal's offerings. These purchasing decisions should only be based on
features currently available. The development, release, and timing of any
features or functionality described for Pivotal's offerings in this presentation
remain at the sole discretion of Pivotal. Pivotal has no obligation to update
forward looking information in this presentation.
64

51. Learn More. Stay Connected.
65
#springone@s1
p