The document discusses the inevitable marriage between DevOps and security (DevSecOps). It argues that security must evolve to embrace chaos and resilience in order to remain relevant and aligned with DevOps goals of optimizing software delivery. Specifically, it recommends that security adopt metrics like time-to-recovery over time-between-failures, and embrace distributed, immutable, and ephemeral infrastructure designs that are more secure by default. It also provides examples of how to architect and test for security through controlled chaos engineering. The conclusion is that for security to truly unite with DevOps, it needs to stop resisting chaos and instead focus on outcomes like resilience.

1. CONTROLLED CHAOS
The Inevitable Marriage of
DevOps & Security
Kelly Shortridge (@swagitda_)
Dr. Nicole Forsgren (@nicolefv) Black Hat USA 2019

2. Hi, I’m Kelly
2

3. Hi, I’m Nicole
3

4. “Chaos isn’t a pit. Chaos is a ladder.”
― Petyr Baelish, Game of Thrones
4

5. Software is eating the world.
DevOps drives its devouring.
5

6. Infosec has a choice: marry DevOps
or be rendered impotent & irrelevant
6

7. Infosec isn’t invincible. Denial &
“DevSecOps” won’t save your future.
7

8. How should infosec control chaos &
make a marriage to DevOps last?
8

9. 1. DevOps Dominion
2. The Metamorphosis
3. Time to D.I.E.
4. A Phoenix Rises
5. Marriage Vows
9

10. DevOps Dominion

11. DevOps is not automation or “agile”
11

12. DevOps is a mindset that unifies
responsibility and accountability.
12

13. DevOps has “crossed the chasm” –
the business benefits are too striking
13

14. DevOps integrates once-disparate
roles, encouraging “shifting left”
14

15. Infosec can join DevOps or watch as
DevOps carves its own secure path
15

16. Chaos & resilience is infosec’s future
16

17. Therefore, infosec & DevOps
priorities actually align…
17

18. What are DevOps’s priorities?

19. Optimization of software delivery
performance so tech delivers value
19

20. Stability & speed don’t conflict –
resilience & innovation are bffs
20

21. CI/CD: implement changes in prod
rapidly, sustainably, & safely
21

22. What metrics delineate “elite”
DevOps performers from the rest?
22

23. Lead time for changes: How long does
it take for committed code to
successfully run in production?
23

24. Release frequency: How often is code
deployed to production or released
to end users?
24

25. Time to Recovery (TTR):
How long does it take to restore
service?
25

26. Change failure rate: What percentage
of changes to production degrade
service & require remediation?
26

27. Elite High Medium Low
Lead time for
changes
< One day 1 day - 1 week
1 week –
1 month
1 month –
6 months
Release
frequency
On demand
(>1 daily)
1 per day –
1 per month
1 per week –
1 per month
1 per month –
1 per 6 months
Time to
recovery
< 1 hour < 1 day < 1 day
1 week –
1 month
Change failure
rate
0% – 15% 0% – 15% 0% – 15% 46% – 60%
27

28. The evidence: no tradeoff between
better infosec & DevOps leetness
28

29. Elites conduct security reviews &
implement changes in mere days
29

30. “DevOps doesn’t care about security”
is a lazy straw man. Stop it.
30

31. Security drives stronger DevOps
results. Now infosec must evolve.
31

32. The Metamorphosis

33. Partitioning of responsibility &
accountability engenders conflict
33

34. The real “DevSecOps”: DevOps will be
held accountable for security fixes
34

35. What goals should infosec pursue in
this evolution?
35

36. And… why should infosec goals
diverge from DevOps goals?
36

37. Infosec should support innovation in
the face of change – not add friction
37

38. Infosec has arguably failed, so “this is
how we’ve always done it” is invalid
38

39. Cloud & microservices created the
“Infosec Copernican Revolution”
39

40. But the data doesn’t lie: cloud & PaaS
contribute to elite performance
40

41. The Security of Chaos

42. “Things will fail” naturally extends
into “things will be pwned”
42

43. Security failure is when security
controls don’t operate as intended
43

44. What are the principles of chaotic
security engineering?
44

45. 1. Expect that security controls will
fail & prepare accordingly
45

46. 2. Don’t try to avoid incidents – hone
your ability to respond to them
46

47. What are the benefits of the chaos /
resilience approach?
47

48. Benefits: lowers remediation costs &
stress levels during real incidents
48

49. Benefits: minimizes end-user
disruption & improves confidence
49

50. Benefits: creates feedback loops to
foster understanding of systemic risk
50

51. The ability to automate “toil” away
should also appeal to infosec
51

52. Toil: manual, repetitive, tactical work
that doesn’t provide enduring value
52

53. Manual patching, provisioning 2FA /
ACLs, firewall rule management, etc.
53

54. What other ways can infosec become
more strategic?
54

55. Time to D.I.E.

56. C.I.A. triad: commonly used as a model
to balance infosec priorities
56

57. Confidentiality: withhold info from
people unauthorized to view it
57

58. Integrity: data is a trustworthy
representation of the original info
58

59. Availability: organization’s services
are available to end users
59

60. But these are security values, not
qualities that create security
60

61. We need a model promoting qualities
that make systems more secure
61

62. Instead, use the D.I.E. model:
Distributed, Immutable, Ephemeral
62

63. Distributed: multiple systems
supporting the same overarching goal
63

64. Distributed infrastructure reduces
risk of DoS attacks by design
64

65. Immutable: infrastructure that
doesn’t change after it’s deployed
65

66. Servers are now disposable “cattle”
rather than cherished “pets”
66

67. Immutable infra is more secure by
design – ban shell access entirely
67

68. Lack of control is scary, but unlimited
lives are better than nightmare mode
68

69. Ephemeral: infrastructure with a very
short lifespan (dies after a task)
69

70. Ephemerality creates uncertainty for
attackers (persistence = nightmare)
70

71. Installing a rootkit on a resource that
dies in minutes is a waste of effort
71

72. Optimizing for D.I.E. reduces risk by
design & supports resilience
72

73. A Phoenix Rises

74. What metrics support resilient
security engineering?
74

75. TTR is equally as important for
infosec as it is for DevOps
75

76. Time Between Failure (TBF) will lead
your infosec program astray
76

77. Extended downtime makes users sad,
not more frequent but trivial blips
77

78. Prioritizing failure inhibits innovation
78

79. Instead, harness failure as a tool to
help you prepare for the inevitable
79

80. TTR > TTD – who cares if you detect
quickly if you don’t fix it?
80

81. Game days: like planned firedrills
81

82. Prioritize game days based on
potential business impacts
82

83. Decision trees: start at target asset,
work back to easiest attacker paths
85

84. Determine the attacker’s least-cost
path (hint: it doesn’t involve 0day)
86

85. Architecting chaos

86. Begin with “dumb” testing before
moving to “fancy” testing
88

87. Controlling Chaos: Availability
89

88. Turning security events into
availability events appeals to DevOps
90

89. The existing repertoire of chaos eng
tools primarily covers availability
91

90. Chaos Monkey, Azure Fault Analysis
Service, Chaos-Lambda…
92

91. Kube-monkey, PowerfulSeal, Pod-
reaper, Pumba, Blockade…
93

92. Infosec teams can use these tools but
make attackers the source of failure
94

93. Controlling Chaos: Confidentiality
95

94. Microservices use multiple layers of
auth that preserve confidentiality
96

95. A service mesh is like an on-demand
VPN at the application level
97

96. Attackers are forced to escalate
privileges to access the iptables layer
98

97. Test: inject failure into your service
mesh to test authentication controls
99

98. Controlling Chaos: Integrity
100

99. Test: swap out certs in your ZTNs –
all transactions should fail
101

100. Test: modify encrypted data & see if
your FIM alerts on it
102

101. Test: retrograde libraries, containers,
other resources in CI/CD pipelines
103

102. D.I.E.ing is an art, like everything else

103. Controlling Chaos: Distributed
105

104. Distributed mostly overlaps with
availability in modern infra contexts
106

105. Multi-region services present a fun
opportunity to mess with attackers
107

106. Shuffle IP blocks regularly to change
attackers’ lateral movement game
108

107. Controlling Chaos: Immutable
109

108. Immutable infra is like a phoenix – it
disappears & comes back a lot
110

109. Volatile environments with continually
moving parts raise the cost of attack
111

110. Create rules like, “If there’s ever a
write to disk, crash the node”
112

111. Attackers must stay in-memory,
which hopefully makes them cry
113

112. Metasploit Meterpreter + webshell:
Touch passwords.txt & kaboom
114

113. Infosec teams can build Docker
images with a “bamboozle layer”
115

114. Mark garbage files as “unreadable” to
craft enticing bait for attackers
116

115. A potential goal: architect
immutability turtles all the way down
117

116. Test: inject attempts at writing to
disk to ensure detection & reversion
118

117. Treat changes to disk by adversaries
similarly to failing disks: mercy kill
119

118. Controlling Chaos: Ephemeral
120

119. Most infosec bugs are stated-related
– get rid of state, get rid of bugs
121

120. Reverse uptime: longer host uptime
adds greater security risk
122

121. Test: change API tokens & test if
services still accept old tokens
123

122. Test: inject hashes of old pieces of
data to ensure no data persistence
124

123. Use “arcade tokens” instead of using
direct references to data
125

124. Leverage lessons from toll fraud –
cloud billing becomes security signal
126

125. Test: exfil TBs or run a cryptominer
to inform billing spike detection
127

126. So, how should infosec work with
DevOps to implement all of this?
128

127. Marriage Vows

128. Infosec + DevOps = scalable love

129. How does this scalable love look?

130. Sit in on early design decisions &
demos – but say “No, and…” vs. “No.”

131. Provide input on tests so every
testing suite has infosec’s stamp on it

132. By the last “no” gate in the delivery
process, nearly all issues will be fixed

133. Infosec should focus on outcomes
that are aligned with business goals
135

134. TTR should become the preliminary
anchor of your security metrics
136

135. Security- & performance-related
gamedays can’t be separate species
137

136. Cultivate buy-in together for
resilience & chaos engineering
138

137. Visibility / observability: collecting
system information is essential
139

138. Your DevOps colleagues are likely
already collecting the data you need
140

139. Changing culture: change what
people do, not what they think
141

140. Conclusion

141. Security cannot force itself into
DevOps. It must marry it.
143

142. Chaos/resilience are natural homes
for infosec & represent its future.
144

143. Infosec must now evolve to unify
responsibility & accountability.
145

144. If not, infosec will sit at the kids’ table
until it is uninvited from the business.
146

145. Giving up control isn’t a harbinger of
doom. Resilience is a beacon of hope.
147

146. “You must have chaos within you to
give birth to a dancing star.”
― Friedrich Nietzsche
148

147. @swagitda_
/in/kellyshortridge
kelly@greywire.net
@nicolefv
/in/nicolefv
nicolefv@google.com
149