An experience report on building out DevSecOps at an Ottawa company.

1. DevSecOps
Security and Compliance at the Speed of
Continuous Delivery
Agile Ottawa June 2018

2. Tehama
• Delivers privileged technical services
over the internet with
– Transparency
– Security
– Auditability
• Ensures trust while enabling quick
onboarding and connectivity

3. Tehama and SOC2
• Decided early on that SOC2 was
mandatory to build trust with clients
– Demonstrates robust security practices
via a trusted 3rd party

4. Compliance Basics
• Say what you’re going to do
• Do it
• Prove that you did it
– Easy =)

5. Compliance Basics
• Say what you’re going to do
– Done via Security Controls, plans, and
processes
• Do it
– Hard
• Prove that you did it
– Evidence
• Hardest ... if you haven’t planned for it

6. Control Example
• A vulnerability management plan is
maintained to identify potential threats.
Threats are triaged and mitigated as
required. Remediation plans are developed
to address risks.

7. DevSecOps
• DevOps used to deliver and run systems in a
secure and reliable way
• Bringing in Security and Compliance
increases the focus on Ops
– “You build it, you run it”

8. DevSecOps
• Look at security and compliance as a
– Release problem
– Test problem
– Monitoring and observability problem
• Leverage security and compliance expertise
in building out the system
– This is the secret sauce

9. Tehama and DevSecOps
• Security and compliance can’t be “the office
of no”
• Make it secure in order to demonstrate
compliance
– Keep it valuable
– Don’t make it a checkbox exercise
– E.g. continuous vuln mgmt is much better
than a monthly review cycle

10. There is no Magic
• It’s just mature DevOps
– DevSecOps
– DevOpsSec
– Rugged DevOps
• But it is a good phrase to search with

11. DevOps Patterns
• Infrastructure as Code
• Continuous Delivery
• Continuous Monitoring
• Learning from Failure
• Collaborative Culture

12. Security Patterns
Using DevOps to implement
• Defense in depth
• Access control
• Principle of least privilege

13. Policy Designed for CI/CD
Change Management
• Standard Change
– Pre-approved
– Move most changes here
– High success rate, low MTTR
• High Risk Change
– Classic security approval
• Emergency Change
– Post release approval
– Don’t block an emergency change

14. Policy Requires Vulnerability
Management
Common categorization between tools
• Tools find CVEs
– Common Vulnerabilities and Exposures
• Classified via Common Vulnerability Scoring
System
– CVSS
• Includes remediation timelines (SLA)

15. Implementation
Security is everyone’s job, all the time
• Design it into the system, then it is just how
the software is delivered
• Audit evidence is generated during daily
work
– Not a scramble before an audit

16. Implementation
Secure software supply chain
• All images and OSs are from trusted repos
– Hardened
• All software dependencies are scanned
• Patch management is a priority
– Custom software dependencies
– EC2 OS
– Docker images
– Open source applications

17. Implementation - SDLC
The SDLC is based on a CI/CD pipeline
Automatic
• SAST
– Static Application Security Testing
• DAST
– Dynamic Application Security Testing
• SCA
– Software Component Analysis
• Container vulnerability analysis

18. Implementation - SDLC
Manual
• Prioritization and planning
• Pull requests and code review
– Code review guidelines call out security
concerns with a standard checklist
• PR approval, and release authorization

19. Implementation - Monitoring
Vulnerability plan includes intrusion detection
Requires monitoring and alerting to detect
incidents
• Alerting will launch Incident Response (IR)
• Note, manual detection is still in scope
– Strange system behaviour
– Customer reports
– AWS security
– Law enforcement

20. Implementation - IR
Vulnerability plan includes intrusion detection
Requires monitoring and alerting to detect
incidents
• Alerting will launch Incident Response (IR)
• Manual detection is still in scope
– Noticing strange system behaviour
– Customer reports
– AWS security
– Law enforcement

21. Implementation - IR and Logging
DevOps includes a focus on monitoring and
observability
• This is adds big value
• Enables robust Incident Response and
troubleshooting capabilities

22. Where’s the Evidence?
• Agile planning
• Work ticket workflow
– Pull requests
• CI/CD scan logs
– Remediation tickets
• Release ticket workflow
– Authorization
• Production monitoring
• Incident tickets
• Chat Ops
• Blameless post-mortems
– Remediation tickets

23. Results
• Last pen test had no findings
• Security and compliance dev work is not exceptional
• First audit (Type 1) passed without complications
– Kudos from auditors
• Second audit (Type 2) had no major out of band work
for developers or compliance
• Continuous improvement on logging and monitoring
• IR and post-mortem process well established

24. References
• DevOpsSec: Securing software through continuous delivery
– https://www.safaribooksonline.com/library/view/devopssec/978149197
1413/
• DevOps Audit Defense Toolkit
– https://itrevolution.com/devops-audit-defense-toolkit/
• The DevOps Handbook: How to Create World-Class Agility, Reliability, and
Security in Technology Organizations
– Chapter 19
– Section VI
– Appendix 9
– https://www.amazon.ca/DevOps-Handbook-World-Class-Reliability-
Organizations/dp/1942788002

25. References
• Accelerate: The Science of Lean Software and DevOps: Building and
Scaling High Performing Technology Organizations
– Chapter 6
– https://www.amazon.ca/Accelerate-Software-Performing-Technology-
Organizations/dp/1942788339/
• Incident Management for Operations
– https://www.amazon.ca/Incident-Management-Operations-Rob-
Schnepp/dp/1491917628/
• Pagerduty Incident Response
– https://response.pagerduty.com/
• Incident Response: Trade-offs Under Pressure
– https://www.slideshare.net/InfoQ/incident-response-tradeoffs-under-
pressure

26. References
• Blameless PostMortems and a Just Culture
– https://codeascraft.com/2012/05/22/blameless-postmortems/
• The infinite hows
– https://www.oreilly.com/ideas/the-infinite-hows
• Debriefing Facilitation Guide
– https://extfiles.etsy.com/DebriefingFacilitationGuide.pdf
• Was it technical failure or human error?
– https://www.youtube.com/watch?v=Ygx2AI2RtkI
• AWS Monitoring & Logging
– https://www.slideshare.net/JasonPoley/aws-monitoring-logging
• Container & Microservice Security
– https://www.youtube.com/watch?v=8tDpGyVV8OQ