What is Distributed Tracing (DT), why it may be useful for you.
Design of DT, how OpenTracing and OpenCensus works for Elixir/Erlang projects (libraries, problems, my experience)
10. Design DT: Use Cases
❖ Log one request through all the services
11. Design DT: Use Cases
❖ Log one request through all the services
❖ Gather all operations information (result, time)
12. Design DT: Use Cases
❖ Log one request through all the services
❖ Gather all operations information (result, time)
❖ Build Dependency Graph
13. Design DT: Use Cases
❖ Log one request through all the services
❖ Gather all operations information (result, time)
❖ Build Dependency Graph
❖ Analytics (“Daper” paper)
14. Design DT: Use Cases
❖ Log one request through all the services
❖ Gather all operations information (result, time)
❖ Build Dependency Graph
❖ Analytics (“Daper” paper)
❖ Tags, Logs, Artifacts for each operation
15. Design DT: Use Cases
❖ Log one request through all the services
❖ Gather all operations information (result, time)
❖ Build Dependency Graph
❖ Analytics (“Daper” paper)
❖ Tags, Logs, Artifacts for each operation
❖ Lines of Business analytics
16. Design DT: Use Cases
❖ Log one request through all the services
❖ Gather all operations information (result, time)
❖ Build Dependency Graph
❖ Analytics (“Daper” paper)
❖ Tags, Logs, Artifacts for each operation
❖ Lines of Business analytics
❖ QoS, Traffic Control
20. Design DT: Idea
❖ User Request ID -> to pass to every subsystem:
❖ HTTP: headers
21. Design DT: Idea
❖ User Request ID -> to pass to every subsystem:
❖ HTTP: headers
❖ gRPC: additional field / auto wrapping
22. Design DT: Idea
❖ User Request ID -> to pass to every subsystem:
❖ HTTP: headers
❖ gRPC: additional field / auto wrapping
❖ Event Bus: additional field / auto wrapping
23. Design DT: Idea
❖ User Request ID -> to pass to every subsystem:
❖ HTTP: headers
❖ gRPC: additional field / auto wrapping
❖ Event Bus: additional field / auto wrapping
❖ Subsystem to have sub-request ID
24. Design DT: Idea
❖ User Request ID -> to pass to every subsystem:
❖ HTTP: headers
❖ gRPC: additional field / auto wrapping
❖ Event Bus: additional field / auto wrapping
❖ Subsystem to have sub-request ID
❖ Relation to the previous subsystem (parent/child, sequence, …)
29. Design DT: Problems
❖ Too many traces -> OOM or CPU is 100%
❖ Too few traces -> miss problems
❖ Decide “on the fly” is difficult
30. OpenTracing
❖ Cloud Native Computing Foundation (cncf.io) incubating project
❖ Uber, Apple, Pinterest, Couchbase
❖ API specification, libraries
31. OpenTracing: Concepts
❖ Trace
❖ Span: name, start time, end time
❖ Span: kv tags, kv logs, baggage items
❖ SpanContext
❖ Scopes + Threading + ActiveSpan
❖ Tracers: API + ready solutions
❖ Carriers: API to inject/extract SpanContext
32. OpenTracing: Flow
1. get SpanContext or start Trace => span.start(SpanContext)
2. span.store(tags/metrics/logs/baggage)
3.
4. span.finish()
33. OpenTracing: Flow
1. get SpanContext or start Trace => span.start(SpanContext)
2. span.store(tags/metrics/logs/baggage)
3. run another function with SpanContext
4. span.finish()
34. OpenTracing: Flow
1. get SpanContext or start Trace => span.start(SpanContext)
2. span.store(tags/metrics/logs/baggage)
3. send async message with SpanContext
4. span.finish()
35. OpenTracing: Flow
1. get SpanContext or start Trace => span.start(SpanContext)
2. span.store(tags/metrics/logs/baggage)
3. HTTP request with SpanContext in headers
4. span.finish()
45. defmodule Nested do
use ExRay, pre: :before_fun, post: :after_fun
…
@trace kind: :critical
def fred(a, b), do: blee(a, b)
…
defp before_fun(ctx) do
Span.open(ctx.target, @req_id)
|> :otter.tag(:kind, ctx.meta[:kind])
|> :otter.log(">>> #{ctx.target} with #{ctx.args |> inspect}")
end
end
OpenTracing: Ex_Ray (Elixir)
46. ❖ Less code needed
❖ Low quality code
❖ Memory leaks
❖ Exceptions are not re-raised in wrappers
❖ No default agreements
OpenTracing: Ex_Ray (Elixir): Summary
47. OpenCensus
❖ Started in Google
❖ Large community (Microsoft, Datadog, Prometheus, …)
❖ Automatic Context Propagation
❖ Reference implementation of the official W3C HTTP tracing header
48. OpenCensus: Concepts
❖ Trace, Span - similar to OpenTracing
❖ Link between spans: child/parent/unknown
❖ Sampling: Always/Never/Probabilistic (1 in 10000)/RateLimiting (10 per
sec)
❖ Automatic Context Propagation
❖ Stats/Metrics
52. OpenCensus Erlang
❖ Public GitHub repo for all Elixir/Erlang libs
❖ Libs for web-servers (Elli, Cowboy, Phoenix, …)
❖ Integrate with minimum effort
53. OpenCensus Erlang
❖ ETS table for Span data + GC for abandoned Spans
❖ Track SpanContext: process dict / variable
❖ Parse transform or manual context tracking
❖ Logger can receive SpanContext
❖ Metrics
57. OpenCensus Elixir
❖ Uses opencencus-erlang (e.g. prepare headers with SpanContext)
❖ Implements a macro:
with_child_span “span1” do
…
end
58. ❖ Uses “Phoenix Instrumenter”
❖ Creates Span for any Controller or View
❖ Integration (config.exs):
instrumenters: [OpencensusPhoenix.Instrumenter]
OpenCensus Elixir: Phoenix
59. ❖ Integrates into any pipeline with “Plug”
❖ Gets parent Span from headers
❖ Creates child Span with new attributes (call function to get them)
❖ Integration:
defmodule MyApp.TracePlug do
# some custom configuration
end
plug MyApp.TracePlug
OpenCensus Elixir: Plug
60. OpenCensus BEAM: Summary
❖ A lot of libraries ready to be used
❖ Seamless integration with other languages
❖ You need to understand the concept
62. Summary
❖ A lot of advantages: Introspection, Analytics, LoB, QoS
❖ Think about sending metrics with OpenCensus
❖ Easy to integrate even with Erlang/Elixir
63. Breaking News
❖ Update: May 21st
❖ OpenTracing + OpenCensus
=> OpenTelemetry
❖ Backward compatibility for both projects
❖ Nov 2019: readonly mode for
OpenTracing, OpenCensus