How to apply DevOps in a regulated organisation

1. © 2017 VERACODE INC. 1© 2017 VERACODE INC.
How to Apply DevOps in a
Regulated Organisation

2. © 2017 VERACODE INC. 2
About This Webinar
Colin Domoney
Consultant Solution Architect
@colindomoney
Topics:
• Benefits of DevOps
• The Problem with DevOps
• Regulation 101
• Best Practices and Recommendations
• DevOps Audit Defence Toolkit
• Case Studies

3. © 2017 VERACODE INC. 3
Further Reading
Kim, Gene, Kevin Behr,
and George Spafford.
2013. The Phoenix Project:
A Novel About IT, DevOps,
and Helping Your Business
Win.
Kim, Gene, Patrick
Debois, and John Willis.
2016. The Devops
Handbook: How to
Create World-Class
Agility, Reliability, and
Security in Technology
Organizations
Bird, Jim. ‘DevOps for
Finance - O’Reilly
Media’. Accessed 19
April 2017.
http://www.oreilly.com/w
ebops-perf/free/devops-
for-finance.csp.
‘2016 State of DevOps Report’. 2017.
Puppet. Accessed January 23.
https://puppet.com/resources/white-
paper/2016-state-of-devops-report.
‘DevOps Audit Defense Toolkit’. Accessed
19 April 2017.
https://plus.google.com/communities/10337
2669680429508474.

4. © 2017 VERACODE INC. 4© 2017 VERACODE INC.
The Benefits of
DevOps

5. © 2017 VERACODE INC. 5
The Benefits of DevOps
• High-performing organizations are decisively outperforming their
lower-performing peers in terms of throughput.
• High performers have better employee loyalty,
as measured by employee Net Promoter Score (eNPS).
• Improving quality is everyone’s job.
• High performers spend 50 percent less time remediating security
issues than low performers.
• Taking an experimental approach to product development can
improve your IT and organizational performance.
• Undertaking a technology transformation initiative can produce
sizeable cost savings for any organization.
Source : ‘2016 State of DevOps Report’. 2017. Puppet. Accessed January
23. https://puppet.com/resources/white-paper/2016-state-of-devops-report.

6. © 2017 VERACODE INC. 6© 2017 VERACODE INC.
The Problem
with DevOps

7. © 2017 VERACODE INC. 7
A Complete Mismatch ?
• “celebrate failure”
• “blameless post-mortem”
• “fail fast, fail often”
• “automated release management”
• “breaking down siloes”

8. © 2017 VERACODE INC. 8
Some Basic Problems
‘DevOps Is Great for Startups, but for
Enterprises It Won’t Work—Yet’. WSJ, 13
May 2014.
https://blogs.wsj.com/cio/2014/05/13/dev
ops-is-great-for-startups-but-for-
enterprises-it-wont-work-yet/.

9. © 2017 VERACODE INC. 9
Some Basic Problems
• Siloed structures and organisational inertia makes changes required by DevOps difficult
and expensive
• DevOps tooling and solutions not always suited to legacy systems
• Building the financial ROI case for a technology-driven business process transformation
across business siloes is problematic
• The cultural transformation necessary may be hard to achieve in large-scale enterprises

10. © 2017 VERACODE INC. 10
And More Problems
• Highly siloed environments present great inertia and resistance to change
• Highly interdependent systems making changes difficult to test
• Legacy systems and technology
• Legacy controls (CMMI, ITIL, ISO standards)
• Compliance Implications
– Separation of Duties
– Severe fines
• Perception that DevOps will make IT less secure

11. © 2017 VERACODE INC. 11© 2017 VERACODE INC.
Regulation 101

12. © 2017 VERACODE INC. 12
What Are We Talking About?

13. © 2017 VERACODE INC. 13
Regulation and Performance
“(the report) found that survey respondents in healthcare,
banking, transportation and manufacturing reported a
disproportionately high percentage of low-performing IT.”
‘But Wait, There’s More: Extra Data from the 2015
DevOps Survey’. Puppet. Accessed 19 April 2017.
https://puppet.com/blog/but-wait-there-s-more-
extra-data-from-2015-devops-survey.

14. © 2017 VERACODE INC. 14
The High Cost of Failure
• On August 1 2012 new code was deployed to all servers
• A single server was missed by an operator
• Alerts were missed by operators before market open
• Checks were bypassed in code
• Rollback made things worse temporarily
• Incident lasted only 45 minutes
• Lost $440 million
• Acquired within four months of incident

15. © 2017 VERACODE INC. 15
Regulators and the Compliance Department
• Many, many regulatory bodies
• Not applicable to all business units
• Often legacy or outdated controls
• Internal compliance organisation creates even more stringent
requirements
• Original intent is lost

16. © 2017 VERACODE INC. 16
Multiple Regulators
IT Revolution. DOES16 London - Robert
Scherrer - DevOps in a Highly Regulated
Financial Infrastructure Company. Accessed
18 April 2017.
https://www.youtube.com/watch?v=SczLjDOd
F1E&t=40s.

17. © 2017 VERACODE INC. 17© 2017 VERACODE INC.
Best Practices and
Recommendations

18. © 2017 VERACODE INC. 18
Understanding Your Environment
1. Find a tame auditor
2. Read the regulations
3. Understand the intent
4. Read the internal regulations
5. Challenge … and simplify
Puppet. Puppet and DevOps in Regulated Environments. Accessed 18
April 2017. https://www.youtube.com/watch?v=yj9PbsNO2uE&t=1503s.

19. © 2017 VERACODE INC. 19
Step by Step
1. Meet the compliance team
2. Categorise risk by severity/prevalence
3. Describe the risk and proposed mitigation
4. Automate the process (if possible)
5. Test and verify the control
6. Communicate the control to stakeholders
DevSecCon. DevOps, Security, and
Compliance: Working in Unison.
Accessed 18 April 2017.
https://www.youtube.com/watch?v=2Rq
3q4C1WaU.

20. © 2017 VERACODE INC. 20
ITIL in a DevOps World
• Nothing in ITIL is in contradiction with DevOps principles
• Issues arise in typical ITIL implementations:
– Slow change management process
– Enforces siloes and departments
“DevOps is basically more about how-to-do, ITIL more
about what-to-do”

21. © 2017 VERACODE INC. 21
Change Management Process
• ITIL change management process defines three types of change:
– Standard (low-risk, follow standard process, can be automated)
– Normal (require approval by CAB, manual process)
– Emergency (high priority CAB)
• Too many changes are classified as ‘normal’
• DevOps best practice suggests:
– Try and make as much as possible ‘standard’ and auto-approve
– Optimise the CAB process for requests that remain as ‘normal’

22. © 2017 VERACODE INC. 22
Two-speed IT Architecture
Two types of IT systems:
1. Slower-changing legacy backend ‘systems of
record’ (where money is kept and counted)
2. More agile frontend ‘systems of engagement’
where money is made or lost
DevOps makes most sense in the latter case

23. © 2017 VERACODE INC. 23
Control Your Source Code Repositories
• Continuous Deployment means any code checked in can
potentially reach production within minutes
• Best practices include:
– Splitting repositories
– Using Perforce for fine grained control
– Performing peer reviews on ‘pull requests’ to critical code

24. © 2017 VERACODE INC. 24
Protect Your Deployment Pipeline
• Continuous Deployment means that your pipeline is a critical piece of
infrastructure
• Best practices include:
– Hardening CI/CD systems to prevent compromise
– Review changes to prevent execution of unwanted code
– Test for suspicious API calls in unit tests or scripts
– Ensure CI/CD runs in isolated containers
– Ensure VCS credentials are ‘read only’

25. © 2017 VERACODE INC. 25
Continuous Deployment … or Delivery ?
• High risk of automated changes to a live, business-critical system
• Live deployments work well for stateless web applications, where users
are unlikely to notice a switch in environment
• Not well suited to high volume, low latency platforms
• Use best practices of Continuous Delivery to ensure changes are
always ready to be deployed
• Schedule deployment using traditional method (out of hours, etc)

26. © 2017 VERACODE INC. 26
How to Change Without Failing
• Minimise the risk of change (automate everything)
• Reduce the batch size of changes
• Identify problems early (measure your MTTD)
• Make sure you can recover (minimise your MTTR)
• Be ready to roll back
• Incident response – always be prepared

27. © 2017 VERACODE INC. 27
Prove It !
DevOps Enterprise Summit. DOES15
- Bill Shinn - Prove It! The Last Mile
for DevOps in Regulated
Organizations. Accessed 18 April
2017.
https://www.youtube.com/watch?v=g
g8gGisl4zM.

28. © 2017 VERACODE INC. 28
Netflix Conformity Monkey
Izrailevsky, Yury. ‘The Netflix Simian Army’. Accessed 19 April 2017.
http://techblog.netflix.com/2011/07/netflix-simian-army.html.

29. © 2017 VERACODE INC. 29
Chef Inspec
‘InSpec | IT Security and Compliance as Code’. Chef. Accessed 19 April
2017. https://www.chef.io/inspec/.
“InSpec is an open-source testing framework for infrastructure with a human-
readable language for specifying compliance, security and other policy
requirements. When compliance is code, you can integrate automated tests that
check for adherence to policy into any stage of your deployment pipeline.”

30. © 2017 VERACODE INC. 30
AWS Inspector
https://aws.amazon.com/inspector/

31. © 2017 VERACODE INC. 31© 2017 VERACODE INC.
DevOps Audit
Defence Toolkit

32. © 2017 VERACODE INC. 32
Parts Unlimited

33. © 2017 VERACODE INC. 33
The “Pull List”

34. © 2017 VERACODE INC. 34
Addressing Concerns

35. © 2017 VERACODE INC. 35
In Detail

36. © 2017 VERACODE INC. 36© 2017 VERACODE INC.
Case Studies

37. © 2017 VERACODE INC. 37
ING Bank
• Used to use heavyweight process frameworks (Prince2,
RUP, and CMMI, and ITIL for operations)
• Changes were made slowly and were costly
• Ad-hoc adoption of Agile with Scrum teams
• Adoption of Continuous Delivery and DevOps followed
• 500 legacy applications decommissioned
• 2 year deployments done in 2 hours

38. © 2017 VERACODE INC. 38
Capital One
• Embraced Agile, DevOps and Cloud from the outset
• Less technical debt and bureaucracy
• DevOps technology operating model, which it calls
“Engineering Excellence.”
“Once you go upstream and have development teams
truly own their code in production there is an
accountability and a quality dynamic that happens that is
a very powerful incentive”
- Rob Alexander, Capital One CIO

39. © 2017 VERACODE INC. 39
Thank You!
© 2017 VERACODE INC.