In this slide, we go through the Google Dapper, OpenTracing, Jaeger to OpenTelemetry. By reading and studying the history of Dapper, we could lean the experience and design theory of a large-scale distributed tracing system and then know how it affects other solutions, like OpenTracing and Jaeger. We also discuss the difference between the OpenTracing and Jaeger and also demonstrate how Jaeger works and looks like. After, we talked about the future of OpenTracing, the new organization called OpenTelemetry, what's its goal and how to do that.

1. Opentracing 101
HungWei Chiu

2. HungWei Chiu
•MTS @ ONF
•SDNDS-TW/CNTUG
•Linux/Network/Container/
Kubernetes
•Kuberentes Courses @Hiskio

3. Agenda
Dapper
Opentracing
Jaeger
OpenTelemetry

4. Why Dapper
All most all open source solution are implemented
based on Dapper
ZipKin(Twitter)
Appdash(Sourcegraph)
Jaeger(Uber)

5. Dapper
Dapper, a Large-Scale Distributed Systems Tracing
Infrastructure
Published by Google (2010)
Used over two years (before 2010)
Link: https://research.google/pubs/pub36356/

6. Preface
Modern Internet services are implemented as distributed
system.
Complexe, large-scale
Maybe developed by different teams/languages.
Span many 1000+ machines across physical facilities.

7. Preface
Google built Dapper to provide Google's developers
with more information about the behavior of complex
distributed system.
Understanding system behavior requires observing
related activities across many different program and
machines.

8. Design Goals
Low overhead
Application-level transparency
Programmer should not to be aware of the tracing system.
Scalability
Handle the size of Google's services and clusters in the
next few years.

9. Distributed Tracing in Dapper
One service
Five Servers

10. Trace Tree and Span

11. Annotation in Span

12. Collection Pipeline

13. Overhead
Trace generation is the most critical segment.
In the Dapper libraries are creating/destroying spans and
annotations, logging to local disk.
Average of creation (2.2GHz x86 Server)
Span: 176 ns
Root Span: 204 ns

14. Overhead
CPU never user more 0.3%
of one core of production
machine.
Network:
Less than 0.01% of the
network traffic in Google's
production environment.

15. Web search cluster
Sampling is indeed necessary
Still an adequate amount of
trace data for high-volume
service when sampling rate as
low as 1/1024
Using lower sampling frequency
has the added benefit of
allowing data to persist longer.

16. Experience
Google AdWords
The teams used Dapper iteratively from the first system prototype through launch.
Performance
Optimize performance
Identify unnecessary serial request along the critical path
Correctness
Database (read-only/read-write)
Understanding
Testing

17. OpenTracing

18. OpenTracing
MicroService provides a powerful architecture but not without its
own challenges.
Debugging and observing transactions across services.
Transactions are no in-memory calls or stack traces anymore.
Distributed tracing wants to solve this problem.
Provides a solution to describe and analyze the cross-process
transactions.

19. OpenTracing
OpenTracing is composed of an API specification,
frameworks and libraries that have implemented the
specification, and documentation for the project.
Allows SEs to add instrumentations to their application code
using APIs
Supported languages
Golang, Java, JS, Python, C++, C#..etc

20. Distributed Tracing Model

21. OpenTracing Units

22. Opentracing Units
Trace
The description of a transaction as it moves through a distributed system.
Represent calls made between the downstream services
Span
A named, timed operation representing a piece of the workflow
Includes
Operation name
Start and End timestamp
Key/Value attribute
Parent's Span identifier
Span Context
Carries data across process boundaries
SpanID, TraceID
Key: Value items.

23. Opentracing Units
Tags
Key:Value pairs that enable user-defined annotation of spans in
order to query and filter.
Logs
Key:Value pairs that are useful for capturing timed log messages.
Baggage Items
Carried by Span Context

24. https://opentracing.io/docs/overview/tracers/

25. Jaeger

26. Jaeger
Distributed tracing system released as open sourced by Uber.
Cloud Native Computing Foundation graduated project.
OpenTracing compatible data model and instrumentation libraries.
Golang/Node/Java/Python/C++
Storage backends
Cassandra, Elasticsearch, memory

27. Jaeger vs. OpenTracing
https://medium.com/jaegertracing/jaeger-and-opentelemetry-1846f701d9f2

28. Deployment

29. Docker-Compose Kubernetes- Operator

30. Demo Project
Project: hotrod
Demo application that consists of several microservices
Run everything via docker-compose
https://github.com/jaegertracing/jaeger/tree/master/examples/hotrod

31. Demo Project
https://github.com/jaegertracing/jaeger/tree/master/examples/hotrod
Frontend
Customer Driver Route
RedisMysql

32. Demo

35. gRPC

36. Jaeger vs. OpenTracing
https://medium.com/jaegertracing/jaeger-and-opentelemetry-1846f701d9f2

37. OpenTelemetry

38. OpenTelemetry
(OpenCensus and OpenTracing) are OpenTelemetry
Telemetry (Tracing, Metrics, Logs)
OpenTelemetry provides the libraries, agent and other
components to you
Capture metrics, traces, metadata and logs and then sends
them to your backends like (Prometheus, Jaeger, Zipkin)

39. https://medium.com/jaegertracing/jaeger-and-opentelemetry-1846f701d9f2

40. API SDK Processing Exporter Collector
Telemetry
Backend
Extensibility
https://blog.thundra.io/what-is-opentelemetry-and-what-does-it-bring

43. Q&A