Talk given for https://www.thesecuredeveloper.com/events/the-new-ways-of-devsecops DevOps and the subsequent move bring security in under the umbrella of DevSecOps has created a new an ethos for security. This is good, however moving security and devops closer together in many organizations leaves us with questions of how this merge works in practice. What happens to security? To developers? And where does chaos engineering fit in? This talk highlights security's place in DevOps and how topics ranging from empathy to chaos to system safety fit in organizations today. The hope is to uncover a new playbook for devs, ops, and security to work together.

2. JAMES WICKETT
Sr. Sec Eng & Dev Advocate @ Verica
Author, LinkedIn Learning
Organizer, DevOps Days Austin, Serverless Days ATX, DevSecOps Days
Austin
Author, DevSecOps Handbook (In progress)
@wickett

4. wickett@verica.io

5. VERICA.IO
An enterprise platform for Continuous Verification,
using Chaos Engineering principles, to take a
proactive and measured approach to preventing
availability and security incidents.
@wickett

6. DEVSECOPS
@wickett

7. credit to Josh Zimmerman, the original DevOps Jack Handy

8. DEVSECOPS
@wickett

9. BUT, WHY?
@wickett

10. FIRST,
UNDERSTAND DEVOPS
AND HOW WE GOT HERE
@wickett

12. TEH CLOUD
@wickett

13. DATASo Big Right Now
@wickett

14. ALL THE WAY
DOWN
@wickett

16. YASSS! OPS (and security)
FOR FREE!@wickett

17. DevOps grew hand-in-hand with cloud
@wickett

18. DEVOPS WAS INEVITABLE
@wickett

19. DevOps is the inevitable result of needing
to do efficient operations in a distributed
computing and cloud environment.
Tom Limoncelli
@wickett

20. DevOps is an epistemological
breakthrough joining disparate people
around a common problem
@wickett

21. DevOps was needed to fix the
inequitable distribution of labor
@wickett

22. 10:1
DEV:OPS
@wickett

23. DevOps is not a technological problem.
DevOps is a business problem.
- Damon Edwards
@wickett

24. DevOps is just another waypoint on
Agile's journey across the business
@wickett

25. DevOps is the application of
Agile methodology to system
administration
— The Practice of Cloud System Administration Book
@wickett

26. Ok DevOps, that's fine.
But why DevSecOps?
@wickett

27. I ASKED MYSELF THIS SAME QUESTION
@wickett

28. @wickett

29. Security finds itself in the same
position that operations did in the
movement of DevOps
@wickett

30. 100:10:1
DEV:OPS:SEC
@wickett

31. SILOIZATION
@wickett

32. Security, like ops struggles to provide
value in most organizations
@wickett

33. Companies are spending a great
deal on security, but we read of
massive computer-related
attacks. Clearly something is
wrong. The root of the problem is
twofold: we’re protecting the
wrong things, and we’re hurting
productivity in the process.
@wickett

34. [Security by risk assessment]
introduces a dangerous fallacy:
that structured inadequacy is
almost as good as adequacy and
that underfunded security
efforts plus risk management are
about as good as properly funded
security work
@wickett

35. While engineering teams are busy deploying
leading-edge technologies, security teams
are still focused on fighting yesterday’s
battles.
SANS 2018 DevSecOps Survey
@wickett

36. 95%OF SECURITY PROFESSIONALS SPEND THEIR
TIME PROTECTING LEGACY APPLICATIONS
@wickett

37. TECH BURDEN CAN ONLY BE
TRANSFERRED
@wickett

38. SECURITY BURDEN IS NOT
CREATED OR DESTROYED,
MERELY TRANSFERRED
@wickett

39. "MANY SECURITY TEAMS
WORK WITH A WORLDVIEW
WHERE THEIR GOAL IS TO
inhibit change AS MUCH
AS POSSIBLE"
@wickett

40. New technology (cloud, k8s,
serverless, ...) and increased
organization focus on software delivery
is why we need DevSecOps.
@wickett

41. A Highly Desireable New Breed:
THE DEVSECOP
@wickett

42. ...not a tool
…not a CI/CD pipeline with security in it
...can’t be bought on an expo floor
@wickett

43. An inclusive person participating in the
movement of security into devops.
@wickett

44. DEVSECOPS
@wickett

45. DEVSECOPS FRAMEWORK:
MEASURE@wickett

46. MEASURE DEVSECOPS
Maker Driven
Experimenting
Automating
Safety Aware
Unrestrained Sharing
Ruggedizing
Empathy First

47. MEASURE
@wickett

48. MAKER DRIVEN
@wickett

49. We are software engineers who
specialize in a specific discipline:
security
@wickett

50. SECURITY MUST BE ABLE TO
WRITE CODE@wickett

51. Why is this considered
a hot take in our industry?
@wickett

52. With all the resources
available today...
@wickett

54. ADOPT THE
MAKER MINDSET
@wickett

55. SECURITY ALREADY USES DSLS
@wickett

57. The Entire Security Team
Must Write Code
Shannon Lietz, Intuit
Aaron Rinehart, United Health Group
@wickett

58. WHY IS THIS IMPORTANT?
▸ Empathy building
▸ Familiarity with tools
▸ Able to move up the pipeline
@wickett

59. A BUG IS A BUG IS A BUG
@wickett

60. Defect Density studies
range from .5 to 10
defects per KLOC
@wickett

61. DEFECT DENSITY
IS NEVER ZERO
@wickett

62. But my application is just
a few lines of code
@wickett

63. 222 Lines of Code
5 Direct Dependencies
54 total deps (including indirect)
(example from snyk.io)
@wickett

64. 460,046 LOC
@wickett

67. You cannot train developers
to write secure code
@wickett

68. INSTEAD, FOCUS ON METHODS DEVELOPERS USE
▸ TDD/BDD/ATDD
▸ Meaningful comments/commits
▸ Code Smells, Refactoring
▸ Instrumentation
@wickett

69. The goal should be to come up
with a set of automated tests
that probe and check security
configurations and runtime
system behavior for security
features that will execute
every time the system is built
and every time it is deployed.

70. Security is connected
with quality
@wickett

73. MAKER DRIVEN means
▸ See security as part of engineering
▸ View quality as a way to bring security in
▸ Use code, not vendors to solve problems
@wickett

74. MEASURE
@wickett

75. EXPERIMENTING(AND
LEARNING)
@wickett

76. BENEFITS TO EXPERIMENTATION
▸ Measured, Repeatable
▸ Results based on your needs
@wickett

77. @wickett

78. DETECT WHAT MATTERS
▸ Account takeover attempts
▸ Areas of the site under attack
▸ Most likely vectors of attack
▸ Business logic flows
▸ Abuse and Misuse
@wickett

79. We can't cede home
field advantage
— Zane Lackey
@wickett

80. EXPERIMENTING NECESSITATES
UNDERSTANDING STEADY STATE
@wickett

81. RESOURCES
▸ Shannon Lietz (@devsecops)
▸ DOES 2018 Talk: youtu.be/
yuOuVC8xljw
@wickett

82. MEASURE
@wickett

83. AUTOMATION OF THINGS
@wickett

84. @wickett

85. @wickett

86. AUTOMATION PROVIDES FEEDBACK
▸ Pre-commit
▸ At build
▸ Deploy
▸ Runtime
@wickett

87. @wickett

88. Continuous Delivery is how
little you can deploy at one
time
— Jez Humble & David Farley
@wickett

89. At Signal Sciences, we optimized
total cycle time--from code
commit to running in prod
@wickett

90. 15,000
DEPLOYS IN 3.5 YEARS
@wickett

91. SECURITY IN THE PIPELINE
▸ Software composition analysis
▸ Lang linters, git-hound, ...
▸ Scanners, gauntlt
▸ Monitoring and telemetry
@wickett

92. [Deploys] can be treated as
standard or routine
changes that have been
pre-approved by
management, and that
don’t require a heavyweight
change review meeting.

93. RESOURCES
@wickett

94. linkedin.com/learning/devsecops-building-
a-secure-continuous-delivery-pipeline
@wickett

95. linkedin.com/learning/devsecops-
automated-security-testing
@wickett

96. MEASURE
@wickett

97. SAFETY FOR COMPLEX
SYSTEMS
@wickett

98. SIMPLE VS. COMPLEX
SYSTEMS
@wickett

99. SIMPLE SYSTEMS
▸ Linear in nature
▸ Easy to Predict
▸ Able to comprehend
@wickett

100. COMPLEX SYSTEMS
▸ Non-linear (bullwhip effect)
▸ Unpredictable
@wickett

101. WE ABSTRACT COMPLEXITY
▸ Human beings
▸ Societial issues
▸ Psychological issues
▸ Cognitive load
@wickett

102. SOFTWARE DEALS WITH COMPLEXITY THROUGH
ABSTRACTION
@wickett

103. ROOT CAUSE (IN A COMPLEX SYSTEM) IS A MYTH
▸ Lacks full picture
▸ Complex systems are not linear
▸ Result of blame culture
▸ Forgets organizational decisions
▸ Puts the focus on the event over situation
@wickett

104. Drifting into failure is a gradual,
incremental decline into
disaster driven by
environmental pressure, unruly
technology and social
proccesses that normalize
growing risk. No organization is
exempt from drifting into failure

105. BOEING 737MAX
▸ Maneuvering Characteristics Augmentation System (MCAS)
▸ Sometimes, the MCAS commands the trim without notifying
the pilots
▸ This is software
@wickett

106. ▸ Events unfolded in minutes
▸ Software was fighting the pilots silently
▸ The "system" was mimicking every 737 they
had ever operated
@jpaulreed

107. HIGH-SPEED DECISION MAKING
SOUNDS FAMILIAR, DOESN'T IT?
@wickett

108. SOFTWARE IS EATING THE
WORLD
@wickett

109. The growth of complexity
in society has got ahead of
our understainding of how
complex systems work and
fail

110. @wickett

111. @wickett

112. Operations and Security's
burden to rationalize
system models
@wickett

113. Failures are a systems
problem because there is
not enough safety margin.
— @adrianco

114. Failure is an inevitable by-
product of a complex
system's normal
functioning

115. WHERE SECURITY FITS
▸ Add safety margin
▸ Telemetry and instrumentation
▸ Blameless retros
▸ ...more to explore in this area
@wickett

116. RESOURCES
▸ Drift into Failure by Dekker
▸ Understanding Human Error Video Series youtu.be/Fw3SwEXc3PU
▸ @jpaulreed coverage of Boeing medium.com/@jpaulreed
▸ Richard Cook paper bit.ly/2ydDQS2
@wickett

117. MEASURE
@wickett

118. UNRESTRAINED SHARING
@wickett

119. Culture is the most
important aspect to devops
succeeding in the enterprise
— Patrick DeBois

120. DevSecOps is the extension of the
DevOps culture for the inclusion of
Security
@wickett

121. A security team who embraces
openness about what it does and
why, spreads understanding.
— Rich Smith

122. SHARING
AFFECTS CULTURE
@wickett

123. Unrestrained Sharing
goes against security's
standard operating procedure
@wickett

124. IT MIGHT FEEL
UNCOMFORTABLE
@wickett

125. SHARING BREAKS
DOWN SILOS@wickett

126. FOUR KEYS TO CULTURE
▸ Mutual Understanding
▸ Shared Language
▸ Shared Views
▸ Collaborative Tooling
@wickett

127. @wickett

128. SECURITY SHARES THROUGH
▸ Making invisible as visible
▸ Security Observability
▸ APIs, webhooks, dev tooling
@wickett

129. Security Observability gives
applications the ability to
expose the attacks that are
happening below the
surface with feedback to
devs, ops, and security.
@wickett

130. A PAVED ROAD APPROACH
▸ Security as normal
▸ Security is "free"
▸ Jason Chan and Netflix

131. THIS INCLUDES THE
AUDITORS@wickett

133. RESOURCES
▸ Phoenix Project
▸ Agile Application Security
▸ dearauditor.org
@wickett

134. MEASURE
@wickett

135. RUGGEDIZATION
@wickett

137. SOFTWARE BILL OF MATERIALS
KNOW WHAT YOU HAVE
@wickett

138. FAVOR SHORT LIVED SYSTEMS
CATTLE NOT PETS
@wickett

139. DIE FRAMEWORK
▸ Distributed
▸ Immutable
▸ Ephemeral
▸ source: @sounilyu
@wickett

140. RUGGEDIZATION IN 2020
▸ Deception
▸ Chaos Engineering
@wickett

141. DECEPTION
▸ Honeypots, Tarpits, Mantraps
▸ Simple to get started (http headers)
▸ HoneyPy, DeceptionLogic
@wickett

142. We’re moving from disaster
recovery to chaos
engineering to resiliency
— @adrianco
@wickett

143. [Chaos Engineering is] empirical rather
than formal. We don’t use models to
understand what the system should do.
We run experiments to learn what it does.
— Michael Nygard, Release It 2nd Ed.
@wickett

144. CHAOS ENGINEERING
▸ Experiments that span eng and security
▸ Manual opt-out
▸ Valuable Learning
▸ ChaosSlingr, CHAP, ChaosMonkey
@wickett

145. RESOURCES
▸ Aaron Rinehart's talk at RSA youtu.be/wLlME4Ve1go
▸ Release It! 2nd ed., Nygard
▸ Phillip Maddux's talk: youtu.be/k81xKjCEeqE
▸ Herb Todd's talk: youtu.be/Cf_XXmRLnRQ
@wickett

146. MEASURE
@wickett

147. EMPATHY BASED TEAMS
@wickett

149. "those stupid
developers"
— Security
@wickett

150. "you want a machine
powered off and unplugged"
— Developer
@wickett

152. DON’T BE A BLOCKER
BE AN ENABLER
@wickett

153. MEASURE DEVSECOPS
Maker Driven
Experimenting
Automating
Safety Aware
Unrestrained Sharing
Ruggedizing
Empathy First

154. SHARE YOUR STORY
book@devsecops.org
@wickett

155. wickett@verica.io