Brian Henerey discusses how DevOps practices require different controls than traditional IT approaches when conducting a SOC2 audit. He explains that SOC2 reports assess controls related to security, availability, confidentiality, processing integrity, and privacy. While controls aim to mitigate risk, they can be overly optimized for speed in DevOps. The talk provides examples of how to structure controls around common criteria and risks, while still honoring DevOps principles like autonomy, collaboration, and minimizing bureaucracy. Controls should leverage existing DevOps tools and processes to avoid busywork.

1. Brian Henerey
Director of Technology & Operations, OpinionLab
bhenerey@gmail.com | @bhenerey | October 2014
Controlling Devops

2. Me
• Working in tech since 1998
• Discovered Puppet – 2006
• Discovered Devops – 2010
• First Devopsdays – TODAY!

3. Trust us, we’re Engineers!
• Did you delete those users accounts?
• Do those automated backups work?
• Do you know what code was released to Prod?
• Who authorized those changes?
• Who reviewed them?
• Are people ssh-ing into Prod? Did they make
any changes?
• How many people have access to customer
data?

4. Trust, Actually
• Trust is a core element of Devops
• We want to trust and empower our people, that’s
why we hired them
• Yet, we need some controls in place…

5. This Talk...
• I’m going to briefly describe what a SOC2 is and
how it works
• I’m going to tell how to rub some Devops on one

6. SOC(2) it to me
A Report on the Controls at a Service Organization
Relevant to:
– Security
– Availability
– Confidentiality
– Processing Integrity
– Privacy
These are called ‘Trust Services Principles’. Defined
here: http://en.wikipedia.org/wiki/Service_Organization_Controls
American Institute of CPAs (AICPA)

7. Why would you want one?
• To assert that your company meets all the
criteria of the Trust Services Principles
• You want an external auditor to make a
statement that you’ve done this

8. Who has SOC2 reports?
• SaaS’s
• Datacenters
• AWS (http://blogs.aws.amazon.com/security/blog/tag/SOC+2)

9. How does a SOC2 work?
You/Your Org
• Write System Description
• Write Controls
• Write Policies based on Controls
• Write Assertion about your System
Auditor
• Helps you prepare the above
• Performs audit for 3-12 months
• Produces Auditor’s report

10. What does the have to do with
Devops?
1. Accountants are more familiar with traditional IT
practices
– Devops practices require different controls
2. A SOC2 is driven by mitigating risk.
– So is DevOPs

11. What about ITIL?
• ITIL is the baby that got thrown out with the
bathwater.
• This is just my experience.

12. Over-optimized for speed
• There’s a ton more emphasis on speed of
feature delivery than creating operable systems
• You can not wish the operational pain away
• So let’s put some controls in place at the start

13. Getting started with Controls
TSP
Criteria
Risk
Example
Control
Your
Control
Evidence

14. What is a Control?
• Your organization’s practice for mitigating
specific risks
• Tip: You’re better off with multiple controls to
address each Risk

15. What kind of Controls are there?
• AICPA introduced the ‘common criteria’ which
cover:
– Organization and management
– Communications
– Risk management and design and implementation of
controls
– Monitoring of controls
– Logical and physical access of controls
– System Operations
– Change Management

16. Example Control
• Risk: “Breaches and incidents recur because
preventive measures are not implemented after a
previous event. “
• Control: “At least monthly, the Technology
Management Team meet to review the Operations
Review Register and discuss plan for resolution of
issues, or recap of resolved issues.”

17. Evidence of Control Performing Well
• “Monthly meeting minutes OR checklist of categories
of issues/projects discussed at meetings, attendees,
date, and indication if any projects/issues need to be
communicated to other employees.”
• Tip: this is what you want automated or baked into
your processes as much as possible. No one likes
busy work

18. Is a Control a Policy?
No.
• Controls are used only by the auditor
• Policies and Procedures are written for
employees with clear instructions on how to
perform their jobs.

19. How do I write a control?
Auditor
interviews me
Auditor
writes
Review
together

20. How many controls will I need?
• Around 200
– Maybe 20% of these are duplicates. The means the
same control addresses multiple risks.
• I spent 50-60 hours with the auditor writing
controls over 2-3 months

21. These are your Controls!
• Start off by writing what your company actually
does
• The most valuable part of writing these controls
is discovering areas to improve

22. What do Accountants like?
• Clarity
• Hierarchy-based Approval
• Consistency
• Ownership
• Accountability

23. How is Devops different?
• More collaboration
• More autonomy
• Less segregation of responsibilities
• Broader range of impact
– Systems, code, security, networks, databases
• More bottom-up leadership

24. Devops Criteria for writing Controls
• Lightweight
• Minimal impact to workflow
• No bureaucracy
• No busy work
• Automate/bake-in all the evidence
• Team members empowered to do their jobs

25. Up-front versus Back-end Controls
• Up-front control is more powerful, but more
stifling
– People cheat the system in favor of ‘getting things
done’
• Devops controls are on the back-end: The team
is trusted to make changes. We’ll review what
changed after the fact.
– Need multiple controls to increase the effectiveness

26. Don’t fear the policy
• Okay, maybe fear it a bit.

27. Leverage your ticketing system
• Already has history
• Already can create reports
• Easy to add approval to it
• Can automate ticket creation for
weekly/monthly/quarterly tasks
• Easy to add screenshots or attachments as
evidence

28. Leverage your version control
• Name your branches/commits based on Ticket-
ID
• Can review list of commits after deployment to
make sure changes were approved/intended

29. Leverage your tools + logging
• Have your tools create log events
– Who’s using the tool
– Datestamp
– What change was made
– i.e. Chef-handler for logstash
• https://github.com/lusis/logstash_handler

30. Side benefit of doing this
• The more you think about your organization,
you’ll probably find things you should really be
doing but aren’t.
• You’ll also find you aren’t consistent in your
behavior, which is why you need layers of
Control.

31. Example: Alert fatigue
• You probably already have a policy on
responding to alerts
• If you’re not following this process, perhaps it’s
because you have too many alerts, or they’re
un-actionable
• If it’s hard to follow your process, it’s because
you have debt in your system which should be
paid down

32. Detailed example of a Control
Common Criteria 5.6
“Logical access security measures have been
implemented to protect against security,
availability, or confidentiality threats from sources
outside the boundaries of the system.”

33. CC5.6 continued
Risk
“Threats to the system are obtained through
external points of connectivity”

34. CC5.6 Control examples 1
“For VOC system infrastructure components,
logical access is controlled through each
component's native security using group/role
based permissions when possible. The Windows
network is further protected by a 3rd party multi-
factor authentication system.”
Evidence: Network diagram that includes VOC
infrastructure components

35. CC5.6 Control example 2
“For VOC custom developed applications, logical
access is controlled through role-based security
with configurations stored in the applications'
underlying database. For the 3rd party reporting
tool that is integrated with the VOC custom
developed applications, logical access is controlled
through that tool's native security.”
Evidence: Diagram or documentation that
identifies the software/applications that comprise
the VOC system

36. CC5.6 Control example 3
“Firewalls are initially setup as deny all and then
configured to allow access for approved services.
At least quarterly, the configuration of each firewall
is reviewed by the Technology Operations Team
and updated as deemed necessary according to
industry best practices.”
Evidence: Quarterly review of firewall configuration
indicating who reviewed, date reviewed, and steps
taken (e.g. firewall rules updated to…).

37. CC5.7 Control example 4
“Remote access to VOCF systems and
infrastructure components requires single factor
VPN authentication followed by a second factor
authentication at the server or network level.
Certain VOCF system component require two
factor authentication.”
Evidence: VPN configuration screen prints

38. Thanks!!
Questions?
Special thanks to
Mike Becker
Partner-in-charge, Risks and Controls
@ FGMK

39. Judge a man by his questions
rather than his answers
-Voltaire