1. Continuous Delivery Patterns
for Boring Releases

2. About me
Manuel Pais
MS Software Eng
@manupaisable
manuelpais.net
me@manuelpais.net
DevOps and Delivery Consultant
Focused on teams and flow
2@manupaisable | manuelpais.net

3. @manupaisable | manuelpais.net 3
Today
1. Intro to boring releases
2. Patterns for safer releases
3. Patterns for faster releases
4. Patterns for sustainable delivery

4. @manupaisable | manuelpais.net 4
Today
1. Intro to boring releases
2. Patterns for safer releases
3. Patterns for faster releases
4. Patterns for sustainable delivery

5. 5@manupaisable | manuelpais.net

6. 6
App down,
refuses to restart
@manupaisable | manuelpais.net

7. 7
App down,
refuses to restart
Data migration
completed… partially
@manupaisable | manuelpais.net

8. 8
App down,
refuses to restart…
Missing
libraries…
Data migration
completed… partially
@manupaisable | manuelpais.net

9. 9
App down,
refuses to restart…
The password master…
Data migration
completed… partially
Missing
libraries…
@manupaisable | manuelpais.net

10. 10@manupaisable | manuelpais.net

11. 11@manupaisable | manuelpais.net

12. @manupaisable | manuelpais.net 12

13. @manupaisable | manuelpais.net 13

14. @manupaisable | manuelpais.net 14
Build Test Deploy

15. @manupaisable | manuelpais.net 15
Build Test Deploy
Build Test Deploy

16. @manupaisable | manuelpais.net 16
Build Test Deploy
Build Test Deploy
Build Test Deploy

17. @manupaisable | manuelpais.net 17
Build Test Deploy
Build Test Deploy
Build Test Deploy Monitor

18. 18
Outcomes
@manupaisable | manuelpais.net

19. “ability to get changes of all types,
into production, or into the hands of
users, safely and quickly in a
sustainable way”
–Dave Farley & Jez Humble
continuousdelivery.com
@manupaisable | manuelpais.net 19

20. @manupaisable | manuelpais.net 20
Today
1. Intro to boring releases
2. Patterns for safer releases
3. Patterns for faster releases
4. Patterns for sustainable delivery

21. “ability to get changes of all types,
into production, or into the hands of
users, safely and quickly in a
sustainable way”
–Dave Farley & Jez Humble
continuousdelivery.com
@manupaisable | manuelpais.net 21

22. STEP ONE
map all release activities in a
deployment pipeline
22@manupaisable | manuelpais.net

23. 23@manupaisable | manuelpais.net

24. Gains from
improving visibility to everyone,
and increasing repeatability and
traceability are enormous
24@manupaisable | manuelpais.net

25. Value Stream Mapping
@manupaisable | manuelpais.net 25

26. Value Stream Mapping
is a pen and paper tool that
highlights bottlenecks in a matter of
hours and raises awareness of what
difficulties other teams face
@manupaisable | manuelpais.net 26

27. STEP TWO
measure key metrics on:
-speed (e.g. cycle time)
-quality (e.g. defect rate)
-operability (e.g. MTTR)
27@manupaisable | manuelpais.net

28. 28@manupaisable | manuelpais.net

29. STEP THREE
PUT IN THE WORK !!!
29@manupaisable | manuelpais.net

30. Which work?
Automated builds
in clean environments
30@manupaisable | manuelpais.net

31. Which work?
You can provision a
mini-replica of production
environnment
31@manupaisable | manuelpais.net

32. Which work?
You have application health checks
and fast smoke tests
for sanity checking
32@manupaisable | manuelpais.net

33. Which work?
You have acceptance tests
with reasonable coverage,
including failure scenarios
33@manupaisable | manuelpais.net

34. Which work?
Single source of truth.
Single binary.
Single path to production.
34@manupaisable | manuelpais.net

35. STEP THREE
PUT IN THE WORK !!!
(but in iterative fashion…
walking skeletons FTW!)
35@manupaisable | manuelpais.net

36. 36@manupaisable | manuelpais.net

37. 37
practices first,
tools second
@manupaisable | manuelpais.net

38. Practices for boring releases
•Automated build (and unit tests)
•Provision prod replica in pipeline
•Automated acceptance tests (BDD)
•No “invisible” activities
•One source of truth
•One path to production
@manupaisable | manuelpais.net 38

39. @manupaisable | manuelpais.net 39
Today
1. Intro to boring releases
2. Patterns for safer releases
3. Patterns for faster releases
4. Patterns for sustainable delivery

40. “ability to get changes of all types,
into production, or into the hands of
users, safely and quickly in a
sustainable way”
–Dave Farley & Jez Humble
continuousdelivery.com
@manupaisable | manuelpais.net 40

41. @manupaisable | manuelpais.net 41
can’t auto-
scale people
How to cope with
ever increasing
cognitive load on
teams to build and
run applications?

42. @manupaisable | manuelpais.net 42
CI
Peer
review
Infra Security Comply

43. @manupaisable | manuelpais.net 43
CI
Peer
review
Infra Security Comply
Database Accept UX Deploy

44. @manupaisable | manuelpais.net 44
FixMonitorRun
CI
Peer
review
Infra Security Comply
Database Accept UX Deploy

45. Multiple approaches needed
1. Smarter pipelines
2. Team structures
3. Self-service platforms
4. Resilient delivery system
@manupaisable | manuelpais.net 45

46. Multiple approaches needed
1. Smarter pipelines
2. Well-thought team structures
3. Self-service platforms
4. Resilient delivery system
@manupaisable | manuelpais.net 46

47. Multiple approaches needed
1. Smarter pipelines
2. Well-thought team structures
3. Self-service platforms
4. Resilient delivery system
@manupaisable | manuelpais.net 47

48. Multiple approaches needed
1. Smarter pipelines
2. Well-thought team structures
3. Self-service platforms
@manupaisable | manuelpais.net 49

49. Reduce wait times
Minimum path to production
Risk-based activities
Continuous pruning
50@manupaisable | manuelpais.net
1. Smarter pipelines

50. Waiting for pipeline
@manupaisable | manuelpais.net 51

51. Waiting for pipeline
@manupaisable | manuelpais.net 52

52. Reduce wait for pipeline
@manupaisable | manuelpais.net 53

53. Scalable CI/CD Infrastructure
•Solved problem in cloud systems
•Starts with infrastructure-as-code
•Agent farm (auto scaling if possible)
•Pipelines need to evolve as number of
teams grows@manupaisable | manuelpais.net 54

54. Waiting for dependencies
@manupaisable | manuelpais.net 55

55. Waiting for dependencies
@manupaisable | manuelpais.net 56

56. Flow efficiency =
“teams who aren’t paying attention to this concept generally
have flow efficiencies around the 15% mark - that means that
work normally spends 85% of its lifecycle waiting on something.”
http://leankanban.com/flow-efficiency-a-great-metric-you-probably-arent-using
@manupaisable | manuelpais.net 57

57. issue is not how long it takes to do
something, it's how long we're
waiting for it to get done
@manupaisable | manuelpais.net 58

58. @manupaisable | manuelpais.net 59

59. Initial state: serial pipeline
@manupaisable | manuelpais.net 60

60. Intermediate state: CAB in path to prod
@manupaisable | manuelpais.net 61

61. End state: short and wide pipeline
@manupaisable | manuelpais.net 62

62. Reduce wait times
Minimum path to production
Risk-based activities
Continuous pruning
63@manupaisable | manuelpais.net
1. Smarter pipelines

63. Reduce wait times
Minimum path to production
Risk-based activities
Continuous pruning
64@manupaisable | manuelpais.net
1. Smarter pipelines

64. Reduce wait times
Minimum path to production
Risk-based activities
Continuous pruning
65@manupaisable | manuelpais.net
1. Smarter pipelines

65. Reduce wait times
Minimum path to production
Risk-based activities
Continuous pruning
66@manupaisable | manuelpais.net
1. Smarter pipelines

66. @manupaisable | manuelpais.net 70
Today
1. Intro to boring releases
2. Patterns for safer releases
3. Patterns for faster releases
4. Patterns for sustainable delivery

67. “ability to get changes of all types,
into production, or into the hands of
users, safely and quickly in a
sustainable way”
–Dave Farley & Jez Humble
continuousdelivery.com
@manupaisable | manuelpais.net 71

68. @manupaisable | manuelpais.net 72

69. Delivery system itself should be boring, not just the pipeline.
73@manupaisable | manuelpais.net

70. Resilient delivery system
that enables
fast(er) feedback loops
74@manupaisable | manuelpais.net

71. Delivery system
•CI tool
•Pipeline orchestration tool
•Orchestration plugins / 3rd party tools
•Pipeline definitions
•Source repos
•CI + CD infrastructure
•And ?
@manupaisable | manuelpais.net 75

72. Delivery system
@manupaisable | manuelpais.net 76

73. Delivery system
CI/CD Toolchain
@manupaisable | manuelpais.net 77

74. Delivery system
CI/CD Toolchain
App 1
App 2
@manupaisable | manuelpais.net 78

75. Delivery system
CI/CD Toolchain
Pipelines
App 1
App 2
@manupaisable | manuelpais.net 79

76. Delivery system
CI/CD Toolchain
Pipelines
Infra
App 1
App 2
@manupaisable | manuelpais.net 80

77. How resilient delivery looks like
•Tooling and configuration changes do
not impact regular delivery
•Gracefully handles peak load of
pipeline runs
•Issues with underlying infra/tools
handled swiftly, rollback if needed
•Disaster recovery at a click of a button
@manupaisable | manuelpais.net 81

78. How resilient delivery looks like
•Changes (plugins, configuration, jobs,
etc) do not impact regular delivery
•Gracefully handles peak load of
pipeline runs
•Issues with underlying infra/tools
handled swiftly, rollback if needed
•Disaster recovery at a click of a button
@manupaisable | manuelpais.net 82

79. How resilient delivery looks like
•Changes (plugins, configuration, jobs,
etc) do not impact regular delivery
•Gracefully handles peak load of
pipeline runs
•Issues with underlying infra/tools
handled swiftly, roll backed if needed
•Disaster recovery at a click of a button
@manupaisable | manuelpais.net 83

80. How resilient delivery looks like
•Changes (plugins, configuration, jobs,
etc) do not impact regular delivery
•Gracefully handles peak load of
pipeline runs
•Issues with underlying infra/tools
handled swiftly, rollback if needed
•Disaster recovery at a click of a button
(almost) @manupaisable | manuelpais.net 84

81. Practices for resiliency (1/3)
•Pipeline infrastructure-as-code
•Pipeline configuration-as-code
•Build and release from zero to live
@manupaisable | manuelpais.net 85

82. Practices for resiliency (1/3)
•Pipeline infrastructure-as-code
•Pipeline configuration-as-code
•Build and release from zero to live
@manupaisable | manuelpais.net 86

83. @manupaisable | manuelpais.net 87

84. Bonus Points
pipeline-as-code allows
designing future states of the
value stream and draw evolution
@manupaisable | manuelpais.net 88

85. Practices for resiliency (1/3)
•Pipeline infrastructure-as-code
•Pipeline configuration-as-code
•Build and release from zero to live
@manupaisable | manuelpais.net 89

86. Team A Team B
Team C Team D
@manupaisable | manuelpais.net 90

87. The glitch is believed to have been caused by a power supply issue and there is
no evidence of a cyber-attack, the airline said.
@manupaisable | manuelpais.net 91

88. Team A Team B
Team C Team D
@manupaisable | manuelpais.net 92

89. Team D
Team BTeam A
Team C
@manupaisable | manuelpais.net 93

90. Team A
@manupaisable | manuelpais.net 94
Team D
Team B
Team C

91. Team A
Team C
@manupaisable | manuelpais.net 95
Team D
Team B

92. Team A Team B
Team C
@manupaisable | manuelpais.net 96
Team D

93. Team A Team B
Team C Team D
@manupaisable | manuelpais.net 97

94. Borat’s Law / Caveat
@manupaisable | manuelpais.net 98

95. Practices for resiliency (2/3)
•Pipeline immutable infrastructure
•Blue-green pipeline deployments
@manupaisable | manuelpais.net 99

96. Practices for resiliency (2/3)
•Pipeline immutable infrastructure
•Blue-green pipeline deployments
@manupaisable | manuelpais.net 100

97. @manupaisable | manuelpais.net 101

98. @manupaisable | manuelpais.net 102

99. Practices for resiliency (3/3)
•Aggregated logging
•Monitoring & alerting
•Auto-scaling
@manupaisable | manuelpais.net 103

100. Books
104@manupaisable | manuelpais.net

101. releasabilitybook.com
Book sample out now!
Team Guide to
Software Releasability
by Chris O’Dell & Manuel Pais
105@manupaisable | manuelpais.net

102. @manupaisable | manuelpais.net 106
manuelpais.net

103. thank you
Manuel Pais
MS Software Eng
@manupaisable
manuelpais.net
me@manuelpais.net
DevOps and Delivery Consultant
Focused on teams and flow
107@manupaisable | manuelpais.net