https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK Of the three observability data types supported by OpenTelemetry (metrics, logs, and traces) the latter is the one with most potential. Tracing gives users insights into how requests are processed by microservices in a modern, cloud-native architecture. Jaeger and Grafana can visualize a single trace, showing how an individual request traversed your entire system. This helps for distributed debugging and analysis, but using traces only this way is limiting. What if you stored tracing data in a SQL database? You could ask global questions about your system. You could find slow communication paths, where the error rate spiked since the last deployment, or where the request rate suddenly dropped. Thus, tracing can be used proactively to help you spot issues before your customers do. This talk will show you how to do all the above by ingesting OpenTelemetry traces into a PostgreSQL/TimescaleDB database, and building custom dashboards using SQL to make the most out of your tracing data. BIO John Pruitt is a software engineer at Timescale. His work focuses on database/SQL development for the Promscale open-source observability tool, and currently on adding support for OpenTelemetry tracing. Prior to joining Timescale, John grew the DBA team at Shipt. Most of the balance of his career was spent building custom time-series applications in the energy industry and leading data warehousing efforts at regional banks. KEY TAKE-AWAYS FROM THE TALK - What is distributed tracing - Why viewing individual traces is of limited value - How SQL can be used to analyze and visualize traces - What insights can be unlocked using SQL against traces

1. What More Can I Learn from My
OpenTelemetry Traces?
Analyzing traces with SQL
February 1, 2022
John Pruitt

2. What we will
cover today 👀
01
02
Context: Tracing..? 🤔
Our Example System
● Request Rates / Error Rates
● Request Durations
● Service Dependencies
● Upstream Spans
● Downstream Spans
04 Q&A!
03 Building Grafana Dashboards for Traces

3. tsdb.co/opentelemetry-demo

4. 01
Context
Tracing?

5. A trace is a tree of spans
SPAN1
SPAN2 SPAN3 SPAN4
SPAN5 SPAN6 SPAN7 SPAN8
SPAN9 SPAN10

6. A trace is also a time series
SPAN1
SPAN2 SPAN3 SPAN4
SPAN5 SPAN6 SPAN7 SPAN8
SPAN9 SPAN10
TIME

7. A trace in Jaeger 👇

8. By the end of this talk, you will be able to build this 🙌

9. 02
Our Example System
An Absurd Password Generator

10. LOAD
GENERATOR
DIGIT
UPPER
LOWER
SPECIAL
OTEL
COLLECTOR
PROMSCALE
COLLECTOR
PROMSCALE

11. 03
Building Grafana
Dashboards for Traces
Using simple (but powerful) SQL

13. Request Rates / Error Rates
A simple dashboard to get started

14. 1 Requests per second

15. SELECT
time_bucket('1 minute', start_time) as time,
count(*) / 60.0 as req_per_sec
FROM ps_trace.span s
WHERE s.start_time >= now() - interval '5 minutes'
AND parent_span_id is null -- just the root spans
GROUP BY 1
ORDER BY 1
1 Requests per second

16. 2 Requests per second

17. SELECT
time_bucket('1 second', start_time) as time,
count(*) as req_per_sec
FROM ps_trace.span s
WHERE s.start_time >= now() - interval '5 minutes'
AND parent_span_id is null -- just the root spans
GROUP BY 1
ORDER BY 1
2 Requests per second

18. Errors by service

19. Errors by service
SELECT
service_name,
count(*) as num_err
FROM ps_trace.span
WHERE $__timeFilter(start_time)
AND status_code = 'STATUS_CODE_ERROR'
GROUP BY 1

20. Errors by operation

21. Errors by operation
SELECT
x.service_name,
x.span_name,
x.num_err::numeric / x.num_total as err_rate
FROM
(
SELECT
service_name,
span_name,
count(*) filter (where status_code = 'STATUS_CODE_ERROR') as num_err,
count(*) as num_total
FROM ps_trace.span
WHERE $__timeFilter(start_time)
GROUP BY 1, 2
) x
ORDER BY err_rate desc

22. Error rates by operation over time

23. Error rates by operation over time
SELECT
x.time,
x.service_name,
x.span_name,
x.num_err::numeric / x.num_total as err_rate
FROM
(
SELECT
time_bucket('1 minute', start_time) as time,
service_name,
span_name,
count(*) filter (where status_code = 'STATUS_CODE_ERROR') as num_err,
count(*) as num_total
FROM ps_trace.span
WHERE $__timeFilter(start_time)
GROUP BY 1, 2, 3
) x
ORDER BY time

24. Request Durations
A somewhat more complex dashboard…

25. Request durations

26. Slowest requests

27. SELECT
trace_id,
start_time,
duration_ms,
FROM ps_trace.span
WHERE $__timeFilter(start_time)
AND parent_span_id is null
ORDER BY duration_ms DESC
LIMIT 10
Slowest requests

28. Histogram of request durations

29. Histogram of request durations
SELECT duration_ms
FROM ps_trace.span
WHERE $__timeFilter(start_time)
AND parent_span_id is null

30. Distribution of request durations over time

31. Distribution of request durations over time
SELECT
start_time as time,
duration_ms
FROM ps_trace.span
WHERE $__timeFilter(start_time)
AND parent_span_id is null
ORDER BY 1

32. Request duration percentiles over time

33. SELECT
r.time,
'p' || lpad((p.p * 100.0)::int::text, 2, '0') as percentile,
approx_percentile(p.p, percentile_agg(r.duration_ms)) as duration
FROM
(
SELECT
time_bucket('1 minute', start_time) as time,
duration_ms
FROM ps_trace.span
WHERE $__timeFilter(start_time)
AND parent_span_id is null
) r
CROSS JOIN
(
SELECT unnest(ARRAY[.01, .5, .75, .9, .95, .99]) as p
) p
GROUP BY r.time, p.p
ORDER BY r.time
Request duration percentiles over time

34. Service Dependencies
A real-time minimap!

35. Service dependencies

36. SELECT
value#>>'{}' as id,
value#>>'{}' as title
FROM _ps_trace.tag
WHERE key = 'service.name'
SELECT
p.service_name || '->' || k.service_name as id,
p.service_name as source,
k.service_name as target,
k.span_name as "mainStat",
count(*) as "secondaryStat"
FROM ps_trace.span p
INNER JOIN ps_trace.span k
ON (p.trace_id = k.trace_id
AND p.span_id = k.parent_span_id
AND p.service_name != k.service_name)
WHERE $__timeFilter(p.start_time)
GROUP BY 1, 2, 3, 4
Service dependencies

37. Service dependencies

38. SELECT
p.service_name as source,
k.service_name as target,
k.span_name,
count(*) as calls,
sum(k.duration_ms) as total_exec_ms,
avg(k.duration_ms) as avg_exec_ms
FROM ps_trace.span p
INNER JOIN ps_trace.span k
ON (p.trace_id = k.trace_id
AND p.span_id = k.parent_span_id
AND p.service_name != k.service_name)
WHERE $__timeFilter(p.start_time)
GROUP BY 1, 2, 3
ORDER BY total_exec_ms DESC
Service dependencies

39. Upstream Spans
Analyzing the traces’ tree structure

40. Upstream spans

41. WITH RECURSIVE x AS
(
SELECT
trace_id, span_id, parent_span_id,
service_name, span_name
FROM ps_trace.span
WHERE $__timeFilter(start_time)
AND service_name = '${service}'
AND span_name = '${operation}'
UNION ALL
SELECT
s.trace_id, s.span_id, s.parent_span_id,
s.service_name, s.span_name
FROM x
INNER JOIN ps_trace.span s
ON (x.trace_id = s.trace_id
AND x.parent_span_id = s.span_id)
)
SELECT
md5(service_name || '-' || span_name) as id,
span_name as title,
service_name as "subTitle",
count(*) as "mainStat"
FROM x
GROUP BY service_name, span_name
Upstream spans (nodes)

42. WITH RECURSIVE x AS
(
SELECT
trace_id, span_id, parent_span_id, service_name, span_name,
null::text as id,
null::text as target,
null::text as source
FROM ps_trace.span
WHERE $__timeFilter(start_time)
AND service_name = '${service}'
AND span_name = '${operation}'
UNION ALL
SELECT
s.trace_id, s.span_id, s.parent_span_id, s.service_name, s.span_name,
md5(s.service_name || '-' || s.span_name || '-' || x.service_name || '-' || x.span_name) as id,
md5(x.service_name || '-' || x.span_name) as target,
md5(s.service_name || '-' || s.span_name) as source
FROM x
INNER JOIN ps_trace.span s
ON (x.trace_id = s.trace_id
AND x.parent_span_id = s.span_id)
)
SELECT DISTINCT x.id, x.target, x.source
FROM x
WHERE id is not null
Upstream spans (edges)

43. Downstream Spans
Analyzing both the tree + the time series

44. Downstream spans

45. WITH RECURSIVE x AS
(
SELECT trace_id, span_id, parent_span_id, service_name, span_name
FROM ps_trace.span
WHERE $__timeFilter(start_time)
AND service_name = '${service}' AND span_name = '${operation}'
UNION ALL
SELECT s.trace_id, s.span_id, s.parent_span_id, s.service_name, s.span_name
FROM x
INNER JOIN ps_trace.span s
ON (x.trace_id = s.trace_id
AND x.span_id = s.parent_span_id)
)
SELECT
md5(service_name || '-' || span_name) as id,
span_name as title, service_name as "subTitle", count(*) as "mainStat"
FROM x
GROUP BY service_name, span_name
Downstream spans (nodes)

46. WITH RECURSIVE x AS
(
SELECT trace_id, span_id, parent_span_id, service_name, span_name,
null::text as id,
null::text as source,
null::text as target
FROM ps_trace.span
WHERE $__timeFilter(start_time)
AND service_name = '${service}'
AND span_name = '${operation}'
UNION ALL
SELECT s.trace_id, s.span_id, s.parent_span_id, s.service_name, s.span_name,
md5(s.service_name || '-' || s.span_name || '-' || x.service_name || '-' ||
x.span_name) as id,
md5(x.service_name || '-' || x.span_name) as source,
md5(s.service_name || '-' || s.span_name) as target
FROM x
INNER JOIN ps_trace.span s
ON (x.trace_id = s.trace_id
AND x.span_id = s.parent_span_id)
)
SELECT DISTINCT x.id, x.source, x.target
FROM x
WHERE id is not null
Downstream spans (edges)

47. Total execution time by operation

48. WITH RECURSIVE x AS
(
SELECT s.trace_id, s.span_id, s.parent_span_id, s.service_name, s.span_name,
s.duration_ms - coalesce(
(
SELECT sum(z.duration_ms)
FROM ps_trace.span z
WHERE s.trace_id = z.trace_id AND s.span_id = z.parent_span_id
), 0.0) as duration_ms
FROM ps_trace.span s
WHERE $__timeFilter(s.start_time) AND s.service_name = '${service}' AND s.span_name = '${operation}'
UNION ALL
SELECT s.trace_id, s.span_id, s.parent_span_id, s.service_name, s.span_name,
s.duration_ms - coalesce(
(
SELECT sum(z.duration_ms)
FROM ps_trace.span z
WHERE s.trace_id = z.trace_id AND s.span_id = z.parent_span_id
), 0.0) as duration_ms
FROM x
INNER JOIN ps_trace.span s ON (x.trace_id = s.trace_id AND x.span_id = s.parent_span_id)
)
SELECT service_name, span_name, sum(duration_ms) as total_exec_time
FROM x
GROUP BY 1, 2
ORDER BY 3 DESC
Total execution time by operation

49. Total execution time by operation over time

50. WITH RECURSIVE x AS
(
SELECT time_bucket('15 seconds', s.start_time) as time,
s.trace_id, s.span_id, s.parent_span_id, s.service_name, s.span_name,
s.duration_ms - coalesce(
(
SELECT sum(z.duration_ms)
FROM ps_trace.span z
WHERE s.trace_id = z.trace_id AND s.span_id = z.parent_span_id
), 0.0) as duration_ms
FROM ps_trace.span s
WHERE $__timeFilter(s.start_time) AND s.service_name = '${service}' AND s.span_name = '${operation}'
UNION ALL
SELECT time_bucket('15 seconds', s.start_time) as time,
s.trace_id, s.span_id, s.parent_span_id, s.service_name, s.span_name,
s.duration_ms - coalesce(
(
SELECT sum(z.duration_ms)
FROM ps_trace.span z
WHERE s.trace_id = z.trace_id AND s.span_id = z.parent_span_id
), 0.0) as duration_ms
FROM x
INNER JOIN ps_trace.span s ON (x.trace_id = s.trace_id AND x.span_id = s.parent_span_id)
)
SELECT time, service_name || ' ' || span_name as series, sum(duration_ms) as exec_ms
FROM x
GROUP BY 1, 2 ORDER BY 1
Total execution time by operation over time

51. Operation execution times

52. WITH RECURSIVE x AS
(
SELECT s.trace_id, s.span_id, s.parent_span_id, s.service_name, s.span_name,
s.duration_ms - coalesce(
(
SELECT sum(z.duration_ms)
FROM ps_trace.span z
WHERE s.trace_id = z.trace_id AND s.span_id = z.parent_span_id
), 0.0) as duration_ms,
s.status_code = 'STATUS_CODE_ERROR' as is_err
FROM ps_trace.span s
WHERE $__timeFilter(s.start_time) AND s.service_name = '${service}' AND s.span_name = '${operation}'
UNION ALL
SELECT s.trace_id, s.span_id, s.parent_span_id, s.service_name, s.span_name,
s.duration_ms - coalesce(
(
SELECT sum(z.duration_ms)
FROM ps_trace.span z
WHERE s.trace_id = z.trace_id AND s.span_id = z.parent_span_id
), 0.0) as duration_ms,
s.status_code = 'STATUS_CODE_ERROR' as is_err
FROM x INNER JOIN ps_trace.span s ON (x.trace_id = s.trace_id AND x.span_id = s.parent_span_id)
)
SELECT service_name, span_name as operation,
sum(duration_ms) as total_exec_time,
approx_percentile(0.5, percentile_agg(duration_ms)) as p50,
approx_percentile(0.95, percentile_agg(duration_ms)) as p95,
approx_percentile(0.99, percentile_agg(duration_ms)) as p99,
count(*) FILTER (WHERE x.is_err) as num_errors
FROM x
GROUP BY 1, 2 ORDER BY 3 DESC
Operation execution times

53. 04
Questions?

54. Thank you!
Chat with us 👉 slack.timescale.com
tsdb.co/opentelemetry-demo