Distributed tracing is a critical tool for building and operating modern software applications. By giving you an end-to-end view of your application, tracing can help you build a better understanding of your software faster. Starting with the basics, this session will show you how to instrument your application in a way that will give you maximum leverage, whether you are working to improve baseline performance or diagnose an ongoing incident. We’ll use examples derived from the more than 10 billion traces we’ve analyzed from organizations such as Twilio, Lyft, and Github and show you how to use tracing to see your software from every angle.

1. From 1 to 1010 Traces:
Getting Up to Speed With
Distributed Tracing
spoons@lightstep.com (aka Daniel Spoonhower)
CTO and Co-founder, LightStep
@save_spoons

2. What are 10 billion traces good for?

3. What does your software look like?

9. Network connections fail… and programming with failure is hard
- Doesn’t matter if you’re using microservices, functions, …
Production is different (for better or worse)
- Version skew
- Configuration differences
- Scale
Problems occur at service boundaries

11. Tracing is a diagnostic that reveals…
… how a set of services coordinate to handle individual user requests
… from mobile or browser to backends to databases (end-to-end)
… including metadata (logs, tags, other annotations)
Provides a request-centric view that explains performance problems
Distributed Tracing 101

15. Trace
- Represents an end-to-end request
- Consists of 1 to 1000s of spans
Span
- Represents some work done by a single service
- Timed event + metadata
Terminology
start
duration
finish
service
operation

16. Sound good?
????

17. https://opentracing.io/registry/
start
finish
operation

18. Aside: how does it work?
Sampling?
Storage?
Analysis?

19. So far:
- Defined the basics of distributed tracing
- Showed an example of how it can explain a performance problem
- Described what you need to do to start tracing
Next:
- How do you get more value from distributed tracing?
Checking in

20. But what about… $$$?
… Or just finding the logs you need
Who would like more logging?

21. Puts each log entry in context
Assigns relative value to each entry
Solution: attach logs to each trace

22. Key-value pairs associated with
spans
- enable search, aggregation
Tags
http.status_code=200
error=true
release=2019-01-16
host=10.1.221.36
customer_id=vf4hzgwp
request_id=3877433638511

23. Cardinality = number of different values for a tag
“I heard I should be careful with tags”
Most tracing tools don’t put limits on cardinality
Cardinality

24. Trace = 1..N spans
Span = {
Start, Finish
Service, Operation
Logs, Tags
}
Putting it all together

25. overloaded host
big customer

27. Tracing means following requests end-to-end
But finding the root cause often requires comparing requests
- What makes this request unique?
- What makes it similar to others?
End-to-end and in context

28. 100% coverage?
- Not necessarily: lots of value from front-end proxies, key services
Your tool reflects what you already know about your application
- ... and reveals something new
Success criteria

30. Connections offer important clues to identifying performance problems
- Where (unexpected) failures occur
- Where misaligned assumptions are surfaced
Distributed tracing follows those connections
Observing service-to-service connections in
production

31. Questions?
@save_spoons
@lightstephq
lightstep.com