1. La La Land of DevOps Integration
Manuel Pais
@manupaisable | manuelpais.net
Continuous Lifecycle London 2018

2. Continuous Delivery
(Tooling) (Anti-)Patterns
Manuel Pais
@manupaisable | manuelpais.net
Continuous Lifecycle London 2018

3. Today
1. DevOps integration: two camps
2. Pattern #1: Best tool for the job
3. Pattern #2: Pipeline = value stream
4. Pattern #3: Recover quickly
@manupaisable | manuelpais.net 3

4. About me
Manuel Pais
MS Software Eng
@manupaisable
manuelpais.net
me@manuelpais.net
DevOps and Delivery Consultant
Focused on teams and flow
@manupaisable | manuelpais.net 4

5. DevOpsTopologies.com
@manupaisable | manuelpais.net 5

6. Today
1. DevOps integration: two camps
2. Pattern #1: Best tool for the job
3. Pattern #2: Pipeline = value stream
4. Pattern #3: Recover quickly
@manupaisable | manuelpais.net 6

7. “integrating various
tools in software
delivery is very hard”
“a toolchain is a
design pattern
that everybody
can agree on”
https://www.infoq.com/articles/tool-integration-hard-truth
https://www.infoq.com/interviews/interview-damon-edwards-qcon-2014

8. Integrated
DevOps Tool
In the left corner…
@manupaisable | manuelpais.net 8
Integrated
DevOps
Toolchain
In the right corner…

9. Integrated
DevOps
Toolchain
Integrated
DevOps Tool
@manupaisable | manuelpais.net 9

10. @manupaisable | manuelpais.net 10

11. Today
1. DevOps integration: two camps
2. Pattern #1: Best tool for the job
3. Pattern #2: Pipeline = value stream
4. Pattern #3: Recover quickly
@manupaisable | manuelpais.net 11

12. practices first,
tools second
@manupaisable | manuelpais.net 12

13. but some practices
can be quite hard
with integrated tools…
@manupaisable | manuelpais.net 13

14. @manupaisable | manuelpais.net 14

15. “ability to get changes of all types,
into production, or into the hands of
users, safely and quickly in a
sustainable way”
–Dave Farley & Jez Humble
continuousdelivery.com
@manupaisable | manuelpais.net 15

16. CD in the real world
•Focus on individual tools
(Jenkins, VSTS, GoCD, etc)
•Focus on familiarity, cost, ease of
use, security (sometimes…)
•Focus on “scaling”
@manupaisable | manuelpais.net 16

17. CD in the real world
•Focus on individual tools
(Jenkins, VSTS, GoCD, etc)
•Focus on familiarity, cost, ease of
use, security (sometimes…)
•Focus on “scaling”
@manupaisable | manuelpais.net 17

18. CD in the real world
•Focus on individual tools
(Jenkins, VSTS, GoCD, etc)
•Focus on familiarity, cost, ease of
use, security (sometimes…)
•Focus on “scaling”
@manupaisable | manuelpais.net 18

19. CD in the real world
Not enough focus on
expected practices & results
from pipeline / delivery
@manupaisable | manuelpais.net 19

20. Resilient delivery system
that enables
fast(er) feedback loops
20@manupaisable | manuelpais.net

21. Over time issues arise…
•Scalability
•Integration
•Evolvability
•Performance
•Resilience (aka coffee time)
•Bad coding/scripting practices
@manupaisable | manuelpais.net 21

22. Delivery becomes bottleneck
@manupaisable | manuelpais.net 22
ProductionDelivery

23. Today
1. DevOps integration: two camps
2. Pattern #1: Best tool for the job
3. Pattern #2: Pipeline = value stream
4. Pattern #3: Recover quickly
@manupaisable | manuelpais.net 23

24. Integrated DevOps Toolchain
@manupaisable | manuelpais.net 24

25. API-driven tools
are table stakes
for pipeline integration
@manupaisable | manuelpais.net 25

26. @manupaisable | manuelpais.net 26

27. allows for
repeatable mechanisms
and minimizing rework
@manupaisable | manuelpais.net 27

28. @manupaisable | manuelpais.net 28

29. @manupaisable | manuelpais.net 29

30. Integrated DevOps Toolchain
@manupaisable | manuelpais.net 30

31. Integrated DevOps Toolchain
@manupaisable | manuelpais.net 31

32. Integrated DevOps Toolchain
@manupaisable | manuelpais.net 32

33. a flexible toolchain
standardizes practices,
not tools
@manupaisable | manuelpais.net 33

34. tooling should evolve
when YOU need it
@manupaisable | manuelpais.net 34

35. Limit blast radius
@manupaisable | manuelpais.net 35

36. tightly coupled tooling
@manupaisable | manuelpais.net 36

37. tightly coupled tooling
@manupaisable | manuelpais.net 37

38. tightly coupled tooling
@manupaisable | manuelpais.net 38

39. tightly coupled tooling
@manupaisable | manuelpais.net 39

40. @manupaisable | manuelpais.net 40

41. @manupaisable | manuelpais.net 41

42. limit blast radius
@manupaisable | manuelpais.net 42

43. @manupaisable | manuelpais.net 43

44. Today
1. DevOps integration: two camps
2. Pattern #1: Best tool for the job
3. Pattern #2: Pipeline = value stream
4. Pattern #3: Recover quickly
@manupaisable | manuelpais.net 44

45. what’s missing?
@manupaisable | manuelpais.net 45

46. what’s missing?
we tend to care about
technical bits only…
@manupaisable | manuelpais.net 46

47. Integration
Tests
Functional
Tests
Performance
Tests
Operational
Acceptance
PROD
@manupaisable | manuelpais.net 47

48. @manupaisable | manuelpais.net 48

49. @manupaisable | manuelpais.net 49

50. painful bottlenecks
in team interactions
@manupaisable | manuelpais.net 50

51. bottlenecks = silos
time is lost in queues,
not execution
@manupaisable | manuelpais.net 51

52. automating hand-offs
requires trust… which in
turn requires time
@manupaisable | manuelpais.net 52

53. breaking down silos is
HARD!
@manupaisable | manuelpais.net 53

54. we need to design pipelines
with our users in mind,
just like with our apps
@manupaisable | manuelpais.net 54

55. Amy, the Product Owner
@manupaisable | manuelpais.net 55

56. @manupaisable | manuelpais.net 56

57. Amy, the Product Owner
@manupaisable | manuelpais.net 57

58. non-technical stakeholders
are part of delivery too
@manupaisable | manuelpais.net 58

59. pipeline = (real) workflow
highlight bottlenecks => build bridges
trust requires time => automate gradually
not just technical => include everyone
@manupaisable | manuelpais.net 59

60. pipeline = (real) workflow
highlight bottlenecks => build bridges
trust requires time => automate gradually
not just technical => include everyone
@manupaisable | manuelpais.net 60

61. pipeline = (real) workflow
highlight bottlenecks => build bridges
trust requires time => automate gradually
not just technical => include everyone
@manupaisable | manuelpais.net 61

62. Today
1. DevOps integration: two camps
2. Pattern #1: Best tool for the job
3. Pattern #2: Pipeline = value stream
4. Pattern #3: Recover quickly
@manupaisable | manuelpais.net 62

63. Resilient delivery system
that enables
fast(er) feedback loops
63@manupaisable | manuelpais.net

64. time between failures
time to repair
old view of the world
@manupaisable | manuelpais.net 64

65. time between failures
time to repair
new view of the world
@manupaisable | manuelpais.net 65

66. resilient systems also fail
but…
have short time to repair
@manupaisable | manuelpais.net 66

67. time to repair
=
time to diagnose + time to fix
@manupaisable | manuelpais.net 67

68. time to repair
=
time to diagnose (80) + time to fix (20)
@manupaisable | manuelpais.net 68

69. reducing time to repair
requires ownership
@manupaisable | manuelpais.net 69

70. owning delivery system
>
owning errors and logs
@manupaisable | manuelpais.net 70

71. 111.222.333.123 HOME -
[01/Feb/1998:01:08:39 -0800] "GET
/bannerad/ad.htm HTTP/1.0" 200 198
"http://www.referrer.com/bannerad/ba_in
tro.htm" "Mozilla/4.01 (Macintosh; I;
PPC)"
@manupaisable | manuelpais.net 71

72. the problems with logs
@manupaisable | manuelpais.net 72

73. the problems with logs
too much information
information hiding
@manupaisable | manuelpais.net 73

74. the problems with logs
too much information
information hiding
@manupaisable | manuelpais.net 74

75. @manupaisable | manuelpais.net 75

76. log aggregation won’t
help if there’s
no correlation
@manupaisable | manuelpais.net 76

77. https://www.infoq.com/articles/why-test-logging
@manupaisable | manuelpais.net 77

78. information hiding
should not be the reason
why you can’t diagnose
@manupaisable | manuelpais.net 78

79. @manupaisable | manuelpais.net 79

80. without correct error
information you can’t
diagnose/improve
@manupaisable | manuelpais.net 80

81. log aggregation
own your logs
quickly find correlated events
reduce time to diagnose/repair
@manupaisable | manuelpais.net 81

82. Today
1. DevOps integration: two camps
2. Pattern #1: Best tool for the job
3. Pattern #2: Pipeline = value stream
4. Pattern #3: Recover quickly
@manupaisable | manuelpais.net 82

83. More practices for resiliency
•Pipeline infrastructure-as-code
•Pipeline immutable infrastructure
•Blue-green pipeline deployments
•Pipeline configuration-as-code
•Monitoring & alerting
•Auto-scaling
@manupaisable | manuelpais.net 83

84. More practices for fast feedback
•Dynamic test environments
•Dynamic tests scheduling (fail quickly)
•Measure activity times/deviations
•Focus on flow efficiency (remove
largest bottleneck first)
•Parallelize whenever possible
@manupaisable | manuelpais.net 84

85. Take away
delivery system needs to
support evolution of
practices and tools
@manupaisable | manuelpais.net 85

86. Take away
visualize real workflow
to break down silos
@manupaisable | manuelpais.net 86

87. Take away
reducing time to repair
requires ownership
@manupaisable | manuelpais.net 87

88. Sources
• Manuel Pais and Matthew Skelton, 2017, ‘Continuous
Delivery Anti-Patterns’, https://dzone.com/articles/continuous-delivery-
anti-patterns
• Betty Zakheim, 2017, ‘How Difficult Can It Be to Inte-
grate Software Development Tools? The Hard Truth’,
https://www.infoq.com/articles/tool-integration-hard-truth
• Damon Edwards, 2014 ‘DevOps is an Enterprise
Concern’,
https://www.infoq.com/interviews/interview-damon-edwards-qcon-2014
• Don G. Reinertsen, 2009, ‘The Principles of Product Development
Flow: Second Generation Lean Product Development’
https://www.youtube.com/watch?v=L6v6W7jkwok
@manupaisable | manuelpais.net 88

89. @ReleasabilityY
releasabilitybook.com
Team Guide to
Software Releasability
by Chris O’Dell & Manuel Pais
@manupaisable | manuelpais.net 89

90. Resilient delivery system
that enables
fast(er) feedback loops
90@manupaisable | manuelpais.net

91. thank you
Manuel Pais
@manupaisable
me@manuelpais.net
@manupaisable | manuelpais.net 91