Many things can go awry on the journey from pull request (PR) open to merge to production deployment. Issues can arise from the application code, layers of YAML configuration, underlying infrastructure or pipeline logic itself. How can distributed tracing and trace-derived metrics bring developers and operators together for troubleshooting paradise? I’ll unpack a deploy gone bad from both vantage points, gaining an empathy for the engineer who needs to deploy their changes and an ops engineer who is responsible for keeping the system up and running. With signals from OpenTelemetry I will show how increasing the observability of your deploy system can facilitate better collaboration and quicker troubleshooting.

1. @paigerduty
Tracing Adventures
from Pull Request
to Production

2. @paigerduty
2
Can I Deploy?!
Weds, 4:30 pm

3. @paigerduty
3
��🦰
UGH why is pipeline failing?
…better ping Ops

4. @paigerduty
4
#ops-help
#ops-help
:wave: Why does this deploy
keep failing?
<link>
I gotta deploy this before
the end of day
��🦰

5. @paigerduty
5
#ops-help
#ops-help
Have you tried re-running
the pipeline?
��♀

6. @paigerduty
6
#ops-help
#ops-help
Yep. Tried re-running it 3x
before posting here.
Every time it keeps hanging
and spits out this
$BIG_ERROR
��🦰

7. @paigerduty
7
��♀
OF COURSE there’s an urgent
issue to solve with only 30m left
in the day
…I can’t wait to log off

8. @paigerduty
8
#ops-help
#ops-help
Hmmm…OK let me take a
closer look
��♀

9. @paigerduty
9

10. @paigerduty
1.
Dev
10
Ops

11. @paigerduty
Dev and Ops have
distinct perspectives
and data on CI/CD
platform its not
always
straightforward to
know who is
responsible for an
issue
11

12. @paigerduty
12
Dev

13. @paigerduty
13
Ops

14. @paigerduty
14

15. @paigerduty
15
It’s the
platform!

16. @paigerduty
16
It’s the
platform!
It’s your change!

17. @paigerduty
17

18. @paigerduty
18
Dev
Ops

19. @paigerduty
19
Dev
● Application
○ Code
○ Tests
● Deployment Pipeline
● Deploying changes

20. @paigerduty
20
Dev
Ops
● Infrastructure
○ Nodes / Capacity
○ Network
● CI/CD Platform
○ Availability / Reliability
○ Cost Efficiency

21. @paigerduty
21
Dev
Ops
● Infrastructure
○ Nodes / Capacity
○ Network
● CI/CD Platform
○ Availability / Reliability
○ Cost Efficiency
● Application
○ Code
○ Tests
● Deployment Pipeline
● Deploying changes

22. @paigerduty
22

23. @paigerduty
2.
The Path to
Production
23

24. @paigerduty
Changes go on a long
opaque journey from
PR to Production.
Tracing provides a
vantage point both
dev and ops can use
24

25. @paigerduty
a Path to Production
25
1
PR checks

26. @paigerduty
a Path to Production
26
1
2
PR checks
PR Review

27. @paigerduty
a Path to Production
27
1
3
2
PR checks
PR Review
Merge PR

28. @paigerduty
a Path to Production
28
1
3
5
6
4
2
PR checks
PR Review
Merge PR
Deploy to
Integration
Env
Deploy to
Staging Env
Deploy to
Production
Env

29. @paigerduty
29
1
3
5
6
4
2
PR checks
PR Review
Merge PR
Deploy to
Integration
Env
Deploy to
Staging Env
Deploy to
Production
Env
��

30. @paigerduty
30

31. @paigerduty
Integration
Production
Staging
31

32. @paigerduty
Production
Integration
Staging
32
Local
PR

33. @paigerduty
33
Dev & Ops

34. @paigerduty
3.
Tracing
Together
34

35. @paigerduty
Traces bring people
together, just as
traces span service
boundaries they span
team/organization
boundaries.
35

36. @paigerduty
But we have metrics and logs...
36

37. @paigerduty
Why add tracing?
37

38. @paigerduty
What’s a Trace?
38

39. @paigerduty
Build
39
Test Scan Deploy
Pipeline Stages

40. @paigerduty
Checkout
repo
40
Run
linter
Build
Image
Push
image
Pipeline Steps
BUILD

41. @paigerduty
build_image
41
unit_test
scan_image
deploy
Pipeline as Trace Waterfall
integration_deploy

42. @paigerduty
42

43. @paigerduty
Receive
webhook
43
Schedule
Run step
1
Clean up
Pipeline Steps - Platform POV
Run step
2

44. @paigerduty
a Path to Production
44
1
5
6
4
PR checks
Deploy to
Integration
Env
Deploy to
Staging Env
Deploy to
Production
Env
��
��
��
��

45. @paigerduty
45

46. @paigerduty
46

47. @paigerduty
4.
Getting Started
47

48. @paigerduty
48

49. @paigerduty
49

50. @paigerduty
Data Flow
50
O11y
Backend/UI
OTel
Collector*
CI/CD
System
Generate & send Process & route Store & visualize
OTel
SDK/Integration

51. @paigerduty
51
DIY or automatic?

52. @paigerduty
DIY Instrumentation
● Otel-cli
● Tracepusher
● OTel Registry
find your
stack :)
52
OpenTelemetry
Collector

53. @paigerduty
Tracepusher
53

54. @paigerduty
54

55. @paigerduty
55

56. @paigerduty
56

57. @paigerduty
57

58. @paigerduty
58

59. @paigerduty
Integrations
● Ansible Tracing Callback
● Maven Extension
● Pytest
59

60. @paigerduty
60

61. @paigerduty
Native Instrumentation
● Jenkins Plugin
● GitHub Actions Step
● Concourse CI (experimental)
● Zuul OTel Proposal
61

62. @paigerduty
62

63. @paigerduty
63

64. @paigerduty
64

65. @paigerduty
65
🌈
🦋
☀
🌷

66. @paigerduty
Thanks!
You can find me at
- paigerduty@hachyderm.io
- paigerduty@chronosphere.io
Slides from SlidesCarnival, pics from Pixabay
66

67. @paigerduty
BONUS
67

68. @paigerduty
68

69. @paigerduty
69

70. @paigerduty
70
Paige’s tale of how a 1-line config change caused a SEV-1….