1.
ADevSecOpsTale of
Business,
Engineering,and
People
@wickett

2.
JamesWickett
Sr. Sec Eng & Dev Advocate @ Verica
Author, LinkedIn Learning
Organizer, DevOps Days Austin, Serverless Days ATX
DevSecOps Days Austin
Author, DevSecOps Handbook (In progress)
@wickett

3.
@wickett

4.
Get the slides now
wickett@verica.io
@wickett

5.
verica.io
An enterprise platform for Continuous Verification,
using Chaos Engineering principles, to take a
proactive and measured approach to preventing
availability and security incidents.
@wickett

6.
ATale ofMoney,
Chaos,andWoe
@wickett

7.
September 15
1896@wickett

8.
William Crush
@wickett

9.
Wanted promotion
to growbusiness
@wickett

10.
Demolition Derby, but
forTrains
@wickett

11.
1896 introducedthe
ideaofrunningtrain
crashes for Funand
Profit
@wickett

12.
William Crush's
innovationwas
makingthe event
"free"
@wickett

13.
Crushwentfrom
passengeragentto
big-tentpromoter
@wickett

14.
ACityforaDay
Crush,Texas
@wickett

15.
@wickett

16.
Crush,Texas
Population 40,000
@wickett

17.
200 LawOfficersandaJail
Dozenwaterwells
Watertankers
Concessions from Dallas
Preachersand Politicians
Midwaywith Games
@wickett

18.
Biz Model:
Tickets
Concessions
Advertising
@wickett

19.
@wickett

20.
Crushwas
concernedabout
safety
@wickett

21.
@wickett

22.
Engineers evaluated
boilers
Laid 4 miles oftrack
Crowdat200yards
Pressat100yards
@wickett

23.
Word gotout,
Thomas Edison
wantedto film it
@wickett

24.
@wickett

25.
4pm, September 15th,
1896
@wickett

26.
“The rumble ofthetwo
trains, faintand far offat
firstbut growing nearer
and more distinctwith
each fleeting second,was
likethe gathering force ofa
cyclone”
@wickett

27.
@wickett

28.
Thetrains collided
atcombined speed
between 90and 120
mph
@wickett

29.
@wickett

30.
One secondafter
impactthe boilers
exploded
@wickett

31.
@wickett

32.
@wickett

33.
Steam, Iron,Wood
filledthe sky
@wickett

34.
@wickett

35.
Aftermath:
» 4 people died
» Crush fired
» Widespread injuries during incident
» More injuries after incident
» Town shut down
» Lawyers brought in for settlements
@wickett

36.
@wickett

37.
@wickett

38.
Fallout
@wickett

39.
Days later
Crush rehired
Retired from
the MKT 44
years later
@wickett

40.
Demolition Derbyvia
Trains becamea
national
phenomenon
@wickett

41.
But, Inthe hundreds
ofevents post-
Crush,the boilers
held
@wickett

42.
What I learned:
Chronocentrism exists
Engineering is hard
Blame is easy
@wickett

43.
RootCause isaMyth
@wickett

44.
Breaches or Failures
won'tstopbusiness
@wickett

45.
Experimentationand
Learningare Critical
@wickett

46.
DEVSECOPS
@wickett

47.
credit to Josh Zimmerman, the original DevOps Jack Handy

48.
DEVSECOPS
@wickett

49.
First,
Understand DevOps
and howwe got here
@wickett

50.
Teh Cloud
@wickett

51.
DataSo Big RightNow
@wickett

52.
@wickett

53.
“DevOps is the inevitable
resultofneedingto do
efficient operations in a
distributed computing
and cloud environment.”
Tom Limoncelli
@wickett

54.
“DevOps is nota
technological
problem. DevOps isa
business problem.”
Damon Edwards
@wickett

55.
DevOps isan
epistemological
breakthroughjoining
disparate peoplearounda
common problem
@wickett

56.
DevOpswas needed
to fixthe inequitable
distribution of
labor
@wickett

57.
10:1
DEV:OPS
@wickett

58.
DevOps isjust
anotherwaypointon
Agile'sjourney
acrossthe business
@wickett

59.
Ok DevOps,that's
fine.
ButwhyDevSecOps?
@wickett

60.
Iasked myselfthis same question
@wickett

61.
@wickett

62.
Securityfinds itselfinthe
same positionthat
operations did inthe
movementofDevOps
@wickett

63.
100:10:1
DEV:OPS:SEC
@wickett

64.
Siloization
@wickett

65.
Security, like ops
strugglesto provide
value in most
organizations
@wickett

66.
“Companiesare
spendingagreatdeal
on security, butwe
read ofmassive
computer-related
attacks. Clearly
something iswrong.
The rootofthe problem
istwofold:we’re
protectingthewrong
things,andwe’re
hurting productivity
inthe process.”

67.
“[Securitybyrisk
assessment]
introducesa
dangerous fallacy:
thatstructured
inadequacyis almost
as good asadequacy
and that underfunded
securityefforts plus
risk managementare
aboutas goodas
properlyfunded
securitywork”

68.
“While engineeringteams
are busy deploying
leading-edgetechnologies,
securityteamsare still
focused on fighting
yesterday’s battles.”
SANS 2018 DevSecOps Survey
@wickett

69.
"manysecurity
teamsworkwitha
worldviewwhere
their goalisto
inhibit change
as muchas possible"

70.
Newtechnology(cloud,
k8s, serverless, ...)and
increased organization
focus on software
deliveryiswhywe need
DevSecOps.
@wickett

71.
A Highly Desireable New Breed:
The DevSecOp
@wickett

72.
...notatool
...notaCI/CD pipeline
...can’tbe bought
@wickett

73.
An inclusive person
participating inthe
movementof
securityinto
devops.
@wickett

74.
DevSecOps Framework:
MEASURE
@wickett

75.
Maker Driven
Experimenting
Automating
SafetyAware
Unrestrained Sharing
Ruggedizing
Empathy
@wickett

76.
MEASURE
@wickett

77.
Maker Driven
@wickett

78.
Weare software engineers
who specialize inaspecific
discipline: security
@wickett

79.
Securitymustbeable to
write code
@wickett

80.
Whyisthis considered
ahottake in our industry?
@wickett

81.
Withallthe
resourcesavailable
today...
@wickett

82.
Securityis partof
the making
@wickett

83.
Securityalreadyuses DSLs
@wickett

84.
@wickett

85.
The Entire Security
Team Must
Participate in
Software Delivery
@wickett

86.
Empathybuilding
Familiaritywithtools
Ableto move upthe pipeline
@wickett

87.
Abug isabug isabug
@wickett

88.
DefectDensity
studies range
from .5to 10 defects
per KLOC
@wickett

89.
Defectdensity
is never zero
@wickett

90.
With framework/
deps, 500 LOCyou
write can easilybe
400,000 LOC

91.
Hot take:
You cannottrain
developers
towrite secure code
@wickett

92.
Instead, focus on Methods
Developers use
» TDD/BDD/ATDD
» Meaningful comments/commits
» Code Smells, Patterns, Refactoring
» Instrumentation, Observability
@wickett

93.
“The goalshould beto
come upwithasetof
automatedtests that
probeand check
security
configurations and
runtime system
behavior for
securityfeatures
thatwillexecute
everytimethe system
is builtand every
time itis deployed.”

94.
Securityis
connectedwith
quality
@wickett

95.
Maker Driven means
» See security as part of engineering
» View quality as a way to bring security in
» Use code, not vendors to solve problems
@wickett

96.
MEASURE
@wickett

97.
Experimenting (and
Learning)
@wickett

98.
Benefitsto Experimentation
» Measured, Repeatable
» Results based on your needs
» Actionable Outcomes
@wickett

99.
“Securityincidents
are not effective
measures ofdetection
becauseatthatpoint
it'salreadytoo late”
Aaron Rinehart
@wickett

100.
KnowMostLikelyAttacks
and Howto MeasureAbuse
and Misuse
@wickett

101.
“We can'tcede home
fieldadvantage”
Zane Lackey
@wickett

102.
Experimenting necessitates
understanding steadystate
@wickett

103.
Resources
» Shannon Lietz (@devsecops)
» DOES 2018 Talk: youtu.be/yuOuVC8xljw
@wickett

104.
MEASURE
@wickett

105.
Automation
@wickett

106.
“Continuous
Deliveryis how
littleyou can
deployatonetime”
Jez Humble & David Farley
@wickett

107.
Optimizetotalcycle
time from code
committo running in
prod
@wickett

108.
15,000deploys in 3.5 years
@wickett

109.
Securityinthe Pipeline
» Software composition analysis
» Lang linters, git-hound, ...
» Scanners, gauntlt
» Monitoring and telemetry
@wickett

110.
“[Deploys] can be
treatedas
standard or
routine changes
thathave been
pre-approved by
management,and
thatdon’trequire
a heavyweight
change review
meeting.”

111.
Resources:
@wickett

112.
linkedin.com/learning/devsecops-building-a-secure-
continuous-delivery-pipeline
@wickett

113.
linkedin.com/learning/devsecops-automated-security-
testing
@wickett

114.
MEASURE
@wickett

115.
SafetyAware
@wickett

116.
Simplevs. Complex Systems
@wickett

117.
Simple Systems:
Linear in nature
Easyto Predict
Ableto comprehend
@wickett

118.
Complex Systems:
Non-linear (bullwhipeffect)
Unpredictable in nature
No mentalmodelavailable
@wickett

119.
Weabstractcomplexity
» Human beings
» Societial issues
» Psychological issues
» Cognitive load
@wickett

120.
Software deals with complexitythrough
abstraction
@wickett

121.
RootCause (inacomplex system)
isaMyth
» Lacks full picture
» Complex systems are not linear
» Result of blame culture
» Forgets organizational decisions
» Puts the focus on the event over situation
@wickett

122.
“Drifting into failure is
a gradual, incremental
decline into disaster
driven by
environmental
pressure, unruly
technologyand social
proccessesthat
normalize growing
risk. No organization is
exempt from drifting
into failure”
@wickett

123.
Boeing 737Max
» Maneuvering Characteristics Augmentation System
(MCAS)
» MCAS commands the trim without notifying the
pilots
» This is software
@wickett

124.
Softwarewas
fightingthe pilots
silently
@wickett

125.
High-speed decision
making inan up-
tempo environment
@wickett

126.
Software is eating theworld
@wickett

127.
“The growth of
complexityin
societyhas got
ahead ofour
understaindin
g of how
complex
systemswork
and fail”

128.
@wickett

129.
@wickett

130.
Operationsand
Security's burdento
rationalize system
models
@wickett

131.
“Failures are a
systems problem
because there is not
enough safety
margin. ”
@adrianco
@wickett

132.
“Failure isan
inevitable by-
productofa
complex
system's
normal
functioning”

133.
Where SecurityFits
» Add safety margin
» Telemetry and instrumentation
» Blameless retros
» ...more to explore in this area
@wickett

134.
Resources
» Drift into Failure by Dekker
» Understanding Human Error Video Series youtu.be/
Fw3SwEXc3PU
» @jpaulreed coverage of Boeing medium.com/
@jpaulreed
» Richard Cook paper bit.ly/2ydDQS2
@wickett

135.
MEASURE
@wickett

136.
Unrestrained
Sharing
@wickett

137.
“Culture isthe most
importantaspectto
devops succeeding
inthe enterprise”
Patrick DeBois
@wickett

138.
DevSecOps isthe
extension ofthe
DevOps culture for
the inclusion of
Security
@wickett

139.
“Asecurityteamwho
embraces openness
aboutwhatitdoes
and why, spreads
understanding.”
Rich Smith
@wickett

140.
Unrestrained Sharing
affects culture
@wickett

141.
Unrestrained
Sharing goes
againstsecurity's
standard operating
procedure
@wickett

142.
Itwillfeel
uncomfortable
@wickett

143.
Sharing breaks
down silos
@wickett

144.
Four Keysto Culture
» Mutual Understanding
» Shared Language
» Shared Views
» Collaborative Tooling
@wickett

145.
20% ofdevelopers
don'tknowwhat
securityexpects of
them
@wickett

146.
SecuritySharesThrough
» Making invisible as visible
» Security Observability
» APIs, webhooks, dev tooling
@wickett

147.
This includes the
auditors
@wickett

148.
Resources
» Phoenix Project
» Agile Application Security
» dearauditor.org
@wickett

149.
MEASURE
@wickett

150.
Ruggedization
@wickett

151.
@wickett

152.
Software BillofMaterials
Knowwhatyou have
@wickett

153.
Favor ShortLived Systems
Cattle notPets
@wickett

154.
DIE Framework
Distributed
Immutable
Ephemeral
source: @sounilyu
@wickett

155.
Ruggedization in 2020
1. Deception
2. Chaos Engineering
@wickett

156.
Deception
» Honeypots, Tarpits, Mantraps
» Simple to get started (http headers)
» HoneyPy, DeceptionLogic
@wickett

157.
“We’re moving from
disaster recovery
to chaos engineering
to resiliency”
@adrianco
@wickett

158.
“[Chaos Engineering is]
empiricalratherthan formal.
We don’tuse modelsto
understandwhatthe system
should do.We run experiments
to learnwhat itdoes.”
Michael Nygard, Release It 2nd Ed.
@wickett

159.
“The security discipline of
[chaos] experimentation is
done in orderto build
confidence inthe system’s
abilityto defend against
malicious conditions.”
Aaron Rinehart
@wickett

160.
Chaos Engineering
» Experiments that span eng and security
» Manual opt-out
» Valuable Learning
» Controlled experiment blast radius
@wickett

161.
Resources
» Aaron Rinehart's talk at RSA youtu.be/wLlME4Ve1go
» principlesofchaos.org
» Release It! 2nd ed., Nygard
» Phillip Maddux's talk: youtu.be/k81xKjCEeqE
» Herb Todd's talk: youtu.be/Cf_XXmRLnRQ
@wickett

162.
MEASURE
@wickett

163.
Empathy
@wickett

164.
“those stupid
developers”
Security
@wickett

165.
“youwantamachine
powered offand
unplugged”
Developer
@wickett

166.
Halfofdevelopers
saythatdon'thave
enoughtimeto spend
on security

167.
Don’tbeablocker
be an enabler
@wickett

168.
Maker Driven
Experimenting
Automating
SafetyAware
Unrestrained Sharing
Ruggedizing
Empathy
@wickett

169.
Share your story
book@devsecops.org
@wickett

170.
Get the slides
wickett@verica.io
Questions
@wickett
@wickett