In this presentation, DevOps and DevSecOps expert John Willis dives into how to implement DevSecOps, including:
- Why traditional DevOps has shifted and what this shift means
- How DevSecOps can change the game for your team
- Tips and tricks for getting DevSecOps started within your organization
7. • CAMS
• Culture
• Automation
• Measurement
• Sharing
Devops Taxonomies
• The Three Ways
•The First Way
•The Second Way
•The Third Way
8. Devops Practices and Patterns
• Continuous Delivery
• Everything in version control
• Small batch principle
• Trunk based deployments
• Manage flow (WIP)
• Automate everything
• Culture
• Everyone is responsible
• Done means released
• Stop the line when it breaks
• Remove silos
8
itrevolution.com/devops-handbook
9. Ron Westrum - “A typology of organizational cultures
9
10. 30x 200x
more frequent
deployments
faster lead
times
60x 168x
the change
success rate
faster mean time to
recover (MTTR)
2x 50%
more likely to
exceed profitability,
market share &
productivity goals
higher market
capitalization growth
over 3 years*
High performers compared to their peers…
Data from 2014/2015 State of DevOps Report - https://puppetlabs.com/2015-devops-report
Recent IT Performance Data is Compelling
11. 30x 200x
more frequent
deployments
faster lead
times
60x 168x
the change
success rate
faster mean time to
recover (MTTR)
2x 50%
more likely to
exceed profitability,
market share &
productivity goals
higher market
capitalization growth
over 3 years*
High performers compared to their peers…
Data from 2014/2015 State of DevOps Report - https://puppetlabs.com/2015-devops-report
Recent IT Performance Data is Compelling
Faster
Higher
Quality
More
Effective
2555x
17. 17
Devops Results
Google
• Over 15,000 engineers in over 40 offices
• 4,000+ projects under active development
• 5500+ code submissions per day (20+ p/m)
• Over 75M test cases run daily
• 50% of code changes monthly
• Single source tree
18. 18
Devops Results
Google
• Over 15,000 engineers in over 40 offices
• 4,000+ projects under active development
• 5500+ code submissions per day (20+ p/m)
• Over 75M test cases run daily
• 50% of code changes monthly
• Single source tree
19. 19
Devops Results
Google
• Over 15,000 engineers in over 40 offices
• 4,000+ projects under active development
• 5500+ code submissions per day (20+ p/m)
• Over 75M test cases run daily
• 50% of code changes monthly
• Single source tree
• Over 75M test cases run daily
2016
150 Million automated
tests run daily…
20. 20
Devops Results
Amazon
• 11.6 second mean time between deploys.
• 1079 max deploys in a single hour.
• 10,000 mean number of hosts
simultaneously receiving a deploy.
• 30,000 max number of hosts simultaneously
receiving a deploy
21. 21
Unicorns and Horses (Enterprises)
Unicorns
Enterprise
Shamelessly stolen and repurposed from: Pete Cheslock
22. 22
Devops Results
Enterprise Organizations
• Ticketmaster - 98% reduction in MTTR
• Nordstrom - 20% shorter Lead Time
• Target - Full Stack Deploy 3 months to minutes
• USAA - Release from 28 days to 7 days
• ING - 500 applications teams doing devops
• CSG - From 200 incidents per release to 18
28. 28
Summary
• Agile took us from months to days to deliver software
• Devops took from months to days to deploy software
• Now security is the bottleneck
29. 29
Security Meta Points
• It’s 30 time cheaper to fix a security defect in Dev vs. Prod
• Average data breach incident cost 5.4 million
• High performing organizations include security in the
software delivery process
• 80% to 90% of every modern application consists of open
source components
40. 40
More Security Meta Points
• Have security create templates, recipes, playbook
• Create a Wiki for Security
• All Issues managed in a common issue system
• Create a Github Repo for OWASP code examples
• Create interactive visual environments for security
• Visualize all the things….
• A bug is a bug is a bug….
41. 41
DevSecOps and Cloud Configuration
• IAM and resource policies (S3 Bucket, SQS, etc.)
• Permissive policies (e.g. wildcards)
• Security Group ingress and egress rules
• Liberal rules (e.g. 0.0.0.0/0, port range 1-65535 is open)
• Encryption
• Encryption that is not enabled or enforced for applicable resources
• Automatic Key Rotation
• KMS keys that don't have rotation enabled,
• Invalid SSL configurations
• ELBs with invalid SSL configurations
42. 42
DevSecOps and Containers
• Base Image Policies
• Signed images
• Capabilities policies
• Vulnerability Image Scans
• Port Restrictions
• Secrets Management
43. 43
DevSecOps and Serverless
• OWASP top 10 are still relevant
• Proper Permissions
• Data, Keys and Secrets
• Still can have vulnerable code dependancies
44. 44
Delivery
Team
Version
Control
Build Test Release
DevSecOps Example
Stage
Prod
DevSecOps Basics
Security Training
Security Requirements
Threat Modeling
Architecture Review
OWASP Top 10
IDE Plugins
Code Examples
Fail the Build
Static Code Analysis
Security Policy Testing
Configuration Analysis
Vulnerability Scanning
Code and App Analysis
Automated Pen Testing
Static Code Analysis
Security Policy Testing
Configuration Analysis
Security Monitoring
Configuration Monitoring
45.
46. Best Practices for DevSecOps
• Train development teams to develop secure code
• Track security issues the same as software issues
• If infrastructure is now code, then security should be code.
• Integrate security controls in the software pipeline
• Automate security test in the build process
• Detect known vulnerabilities during the pipeline
• Monitor security in production for known states
• Inject failure to ensure security is hardened
Gene Kim, Jez Humble, Patrick Dubois, and John Willis.
The DevOps Handbook; It Revolution Press, LLC.;2016.
47. Devops Kaizen - Full Life Cycle
1.Key Outcomes
2.Countermeasures
3.Storyboard
4.Kanban Board
5.Post Retrospective
1
2
3
4
5
53. 53
Immutable Service Delivery
Fortune 500 Insurance Company
• Tracks critical and high security defect rate per 10k
lines of code
• Started out with (10/10k)
• After applying Devops practices and principles (4/10k)
• After applying Toyota Supply Chain 4VL (1/10k )
• After Docker with Immutable Delivery (0.1/10k)
54. 54
With Docker
Fortune 500 Insurance Company
• One Service
• One Container
• One Read Only File System
• One Port
Editor's Notes
First a little bit about me…
my first serverless keynote.. yeah..
student of devops…
(5 mins)
So what is Devops then??? My study of…
What part of serverless is in conflict w/that statement…. None…!
The right question is Devops and serverless
not or devops…
I’ll let you be the judge on the question at the end of this…
In 2006 Werner Vogals the CTO of Amazon in an ACM interview said that developers at Amazon don’t throw software over the wall. His famous quote was “You build it, you run it”. This simple phrase became known as one of the battle cries for the Devops movement.
About 8 years ago I co-authored this Devops acronym called CAMS
Culture automation measurement and sharing…
Culture is always the hard stuff… the rest fans in place.
(1:0)
Oh wait… some dudes… put together a book that talk about some of these things..
everything in version control…
automate everything… everyone is responsible… reprised flags and manual deployemnts… really bad juju…
The high performance organizations do tow things well..
They create automated pipelines for software delivery and services (mostly know as Continuous Delivery)
They create collaborate high trust cultures…
Everything in VC .. code, infra, scripts, chef, puppet, Dockerfiles. tests scripts, documentation.
Small batch comes from Toyota/lean.. you break things down into smaller pieces… move away from classic waterfall - how long to get one single line of code change.
Everyone shares the same source tree. one line…
Manage the flow.. reduce waste and eliminate bottlenecks.. things like kanban work very well here. (many compnay’s add buffer time). Google is famous for their 20% time. Actually Google caps SRE support time by 50%
Automate the pipeline… automate the flow,, automate unit tests, integration and acceptance testing..
—
HPO’s think of there service as owned by the whole team… HPO’s think service not project or product..
The team owns the serviice. they don’t just throw things over the wall. In other words done means it’s released in production ..
not uncommon to see devs wear pagers in HPO’s
Werner Vogels (Amazon CTO) .. “You Built it You Run it”
Stop the line if something breaks.. (single source.. we all get it back working..
over the last 5 years we have studied (I lonely say we) I have contribute some questions but Jez Humble, Gene Kim and Nicole forsgren are the primary authors.
25 IT professionals
in fact not only are they faster… they have better quality…
We knew instinctively this was true but now we have sound statical data
(15 mins)
We knew instinctively this was true but now we have sound statical data
(15 mins)
You need to increase speed and you need to be more resilient…
the data shows that you need a generative culture.
dev check in code.. an automated build and tests get run. It fails gets kicked back
Dev fixes.. check in auto build and test succeed. They automated acceptance tests get triggered. fail gets kicked back.
Everyone know about the and on chord right?
Toyota plat in Kentucky… How do you make 2000 cars a day?
You pull the and on chord 5000 times a day…
That’s not technology folks.. that’s optimized human capital…
In Mike Rother’s Toyota Kata he tells a fantastic story about a plant that dips from 1k pulls a shift to 700…
I have ppl say but John we are too big to run a single source tree and have total test coverage…
Look here at google..
Google is that plant in Kentucky..(75M test run daily using a single source tree….
I have ppl say but John we are too big to run a single source tree and have total test coverage…
Look here at google..
Google is that plant in Kentucky..(75M test run daily using a single source tree….
I have ppl say but John we are too big to run a single source tree and have total test coverage…
Look here at google..
Google is that plant in Kentucky..(75M test run daily using a single source tree….
This was three years ago …
This was three years ago …
In devops we talk abut Unicorns vs horses.. Unicorns are companies that have special magical powers.. (Alibaba, google, Amazon)..
Here’s the thing the horses (regualr legacy companies are getting amazing results by applying Devops practices and patterns…
(15Minutes)
Look at the history…
Agile accelerated software delivery
but operations were not really included and this created Devops
But even after Devops Security was still not included..
Look at the history…
Agile accelerated software delivery
but operations were not really included and this created Devops
But even after Devops Security was still not included..
Look at the history…
Agile accelerated software delivery
but operations were not really included and this created Devops
But even after Devops Security was still not included..
Look at the history…
Agile accelerated software delivery
but operations were not really included and this created Devops
But even after Devops Security was still not included..
Even in shops that are practiceg devops are still having to throw it over the wall to queue up static analysis and pen testing.
NIST 2002
Poneman 2014
Devops Study 2016
Verizon report (10 VCE’s and 8 of them were 10 years old)
remember this..
Who knows what this is?
CVE-2017-5638 Struts2
AKA the Equifax debacle..
This is one of the many ways Eqiufax could have been Pwned.
They were not the only one using this. research showed that almost 50k organizations were vulnerable.
Coding helpers
Linting and source analysis
supply chain
config and vulnlaribilty scanning
code analysis
app analysis
continious monitoring in prod
Coding helpers
Linting and source analysis
supply chain
config and vulnlaribilty scanning
code analysis
app analysis
continious monitoring in prod
NIST 2002
Poneman 2014
Devops Study 2016
Verizon report (10 VCE’s and 8 of them were 10 years old)
Coding helpers
Linting and source analysis
supply chain
config and vulnlaribilty scanning
code analysis
app analysis
continious monitoring in prod
15 mins
Even in shops that are practiceg devops are still having to throw it over the wall to queue up static analysis and pen testing.
remember this..
remember I told you could get both (spped and reliability)… (penut butter and chocalte…
Here’s a great story..
Tie it all together (Devops, Docker, and Suppy Chain) = ISD