Keeping The Auditor Away: DevOps Audit Compliance Case Studies


Published on

GenOrganizations and development teams are moving beyond waterfall models to those embracing a continuous delivery/DevOps-style set of processes. The deployment of doing tens, hundreds, or even thousands of deploys per day as 'normal' does not align to the SDLC, separation of duties, and common controls expected by auditors.

In this presentation, we will describe what auditors look for in a compliance audit, how to develop alternate control procedures that fulfill those reporting requirements, how to avoid “red flags” that indicate inadequate controls, and real world case studies and reporting artifacts.

Gene Kim has been studying high performing IT organizations since 1999 and helped develop the SOX scoping guidelines with the Institute of Internal Auditors in 2005. James DeLuccia IV is the leader for the Ernst & Young Americas Certification Services, James oversees all of the audits against common industry standards, and champions several global program implementation roll-outs. Developing and 'translating' the control environment behaviors of clients, such as Google, Amazon, Workday, and others is difficult. This discussion will bridge the needs of auditors with the community of developers by sharing examples, discussing the assurance expectations, and how to communicate to pass an audit.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Keeping The Auditor Away: DevOps Audit Compliance Case Studies

  1. 1. @RealGeneKim @jdeluccia Session ID: Gene Kim James DeLuccia Keeping The Auditor Away: DevOps Audit Compliance Case Studies
  2. 2. @RealGeneKim @jdeluccia OMG. Developers Deploying Code?!?
  3. 3. @RealGeneKim @jdeluccia Introductions Gene Kim ▪ Co-author of "The Phoenix Project” ▪ Founder and CTO of Tripwire, Inc. for 13 years ▪ Worked with Jez Humble (co-author of “Continuous Delivery book) to benchmark 14K technology organizations ▪ Co-chaired SOX-404 Scoping Committee at the Institute of Internal Auditors (2005) James DeLuccia ▪ Author, “IT Compliance & Controls” ▪ Ernst & Young, leader for Americas Certification & Compliance Services ▪ Focus: startups, technology, governance, security ▪ Patent holder - crypto privacy comparison system
  4. 4. @RealGeneKim @jdeluccia Golly, Why Are You Attending This Talk? ▪ How many people have to deal with compliance? ▪ On a scale of 1-10, how painful are your interactions with auditors? (1=delightful, 10=awful beyond words?)
  5. 5. @RealGeneKim @jdeluccia Problem Statement Gene ● DevOps and continuous delivery introduce problems with audit, because the work patterns are so different than traditional SDLC ● Agile also had issues (e.g., testing at end of project, requirements phase at the beginning), but is not as radical as DevOps ○ tens/hundreds of deploys/day (change is risk; can’t rely on change approvals, separation of duty) ● No widespread agreement on what DevOps control requirements should look like James ● Auditors must work off a mature and testable environment ● They must stake their livelihood that what you say is correct, completely ● A partnership is needed between you and them to ensure such an environment exists (of course, it also needs to operate and be amazing .. but that is another talk)
  6. 6. @RealGeneKim @jdeluccia Agenda ▪ The Top-Down, Risk Based Audit Process ▪ What Goes Wrong ▪Scoping ▪Control Testing ▪ Scenarios From The DevOps Audit Defense Toolkit ▪ Ask An Auditor Anything!
  7. 7. @RealGeneKim @jdeluccia The DevOps Audit Defense Toolkit James DeLuccia IV Jeff Gallimore Gene Kim Byron Miller
  8. 8. @RealGeneKim @jdeluccia What Is Audit ▪ Management is defined as those who are there to achieve the goals of the organizations, which includes the officers of the company (e.g., CEO, CFO, etc.), executives and managers, as well as everyone who reports to them. ▪ Includes some board of directors, GRC departments ▪ Audit is defined to be the function inside the organization that resides outside of management to serve as an independent, objective source of assurance that the organization can achieve its goals. ▪ Includes internal auditors, external auditors (regulators, assessors, etc.)
  9. 9. @RealGeneKim @jdeluccia Internal Controls “a process, effected by an organization’s board of directors, management, and other personnel, designed to provide reasonable assurance regarding the achievement of objectives related to operations, reporting, and compliance.” - Operations (effectiveness, efficiency) - Financial Reporting (accuracy of account balances and values) - Compliance (with relevant laws and regulations, contractual obligations: PCI DSS, US Export Law, FEDRAMP, SOC-2) Source: (Committee of Sponsoring Organizations for the Commission on Fraudulent Financial Reporting)
  10. 10. @RealGeneKim @jdeluccia How Audit Plans Are Built And Run ▪ Business objectives ▪ Risks ▪ Control objectives ▪ Control procedures Unfortunately, most contact with auditors start with control procedures… It’s totally appropriate to ask to show work and start from beginning...
  11. 11. @RealGeneKim @jdeluccia The Audit Cycle ▪ Planning ▪ Gaining an understanding of the organization ▪ Scoping ▪ Sampling, reporting period, types of evidence needed, recipient of report ▪ Schedule ▪ Fieldwork ▪ Controls testing ▪ Substantive testing ▪ Reporting ▪ Management responses ▪ Attestation by auditor and delivered to regulator/clients
  12. 12. @RealGeneKim @jdeluccia When Scoping Goes Wrong
  13. 13. @RealGeneKim @jdeluccia When Scoping Goes Wrong ▪ 2001: Enron fails ($63B market cap), Arthur Andersen dissolution ▪ 2002: WorldCom (peak $117B market cap) ▪ Leads to Sarbanes-Oxley Act of 2002
  14. 14. @RealGeneKim @jdeluccia When Scoping Goes Wrong Source: KPMG
  15. 15. @RealGeneKim @jdeluccia Problem: Bottom Up Auditing Source: ISACA
  16. 16. @RealGeneKim @jdeluccia Analysis: Audit control testing work was scoped properly, linking controls to compliance objectives and risk. Control failures must result potentially undetected material financial reporting errors The Problem: Improperly Scoped Audits
  17. 17. @RealGeneKim @jdeluccia Financial Reporting Material Weakness What happens when an audit generates a material weakness?
  18. 18. @RealGeneKim @jdeluccia Under-Scoping Operating Risk
  19. 19. @RealGeneKim @jdeluccia ▪ When we don’t understand why we are being audited ▪ “Why are we doing this audit?” (customers, SOX, regulatory; who is it for?) ▪ When we are asked for something we don’t have (e.g., “evidence of SoD or change approvals) ▪ “What is the control objective? Can we rewrite the control procedure for this asset?” ▪ Do this before the auditor shows up When Auditors Attack Unexpectedly These are delicate conversations, with potentially large impacts on scope, cost, risk...
  20. 20. @RealGeneKim @jdeluccia ▪ If we are reacting to these conversations before we’ve done any of our homework, we may be trouble ▪ Extra work (average time to respond to audit is 40 hours; that’s one Dev sprint) ▪ Audit cost and schedule overages: a 3 hour audit test just turned into a 16 hour audit project ▪ Reduced confidence from auditors, increased visibility from audit and management When Auditors Attack Unexpectedly The DevOps Audit Defense Toolkit
  21. 21. @RealGeneKim @jdeluccia The DevOps Audit Defense Toolkit James DeLuccia IV Jeff Gallimore Gene Kim Byron Miller
  22. 22. @RealGeneKim @jdeluccia Practice: Enabling A Shared Understanding Source: DevOps Audit Defense Toolkit
  23. 23. @RealGeneKim @jdeluccia Practice: Enabling A Shared Understanding Source: DevOps Audit Defense Toolkit
  24. 24. @RealGeneKim @jdeluccia Practice: Enabling A Shared Understanding Source: DevOps Audit Defense Toolkit
  25. 25. @RealGeneKim @jdeluccia Walk Through Of DevOps Risk And Control Strategies What does an effective DevOps control environment look like?
  26. 26. @RealGeneKim @jdeluccia DevOps Orgs Actually Love Process “Facebook values people, tools, and way, way down the list is process.” Jay Parikh VP Infrastructure Engineering, Facebook Not true! They are conflating “process” and “approvals!”
  27. 27. @RealGeneKim @jdeluccia High Performing DevOps Orgs Source: 2014 Puppet Labs State Of DevOps 30xmore frequent deployments 8,000xfaster lead times than their peers
  28. 28. @RealGeneKim @jdeluccia High Performing DevOps Orgs Source: 2014 Puppet Labs State Of DevOps 2xhigher change success rates 12xfaster mean time to recover (MTTR)
  29. 29. @RealGeneKim @jdeluccia High Performing DevOps Orgs Source: 2014 Puppet Labs State Of DevOps more likely to exceed profitability, market share & productivity goals 2x higher market capitalization growth over 3 years* 50%
  30. 30. @RealGeneKim @jdeluccia Top Predictors Of Performance ▪ Version control of all production artifacts ▪ Continuous integration and deployment ▪ Automated acceptance testing ▪ Peer-review of production changes (vs. external change approval) ▪ High trust culture ▪ Proactive monitoring of the production environment ▪ Win-win relationship between Dev and Ops
  31. 31. @RealGeneKim @jdeluccia DevOps Orgs Need Hardcopy DevOps has higher automation and closer monitoring controls than traditional deployment environments and therefore reduced points for human failure The documentation of ephemeral systems, tools, and deployment processes into a hardcopy breakdown will communicate and simplify this management long term.
  32. 32. @RealGeneKim @jdeluccia Practice: Document Risks & Control Strategy Source: DevOps Audit Defense Toolkit
  33. 33. @RealGeneKim @jdeluccia Practice: Document Control Strategy Source: DevOps Audit Defense Toolkit
  34. 34. @RealGeneKim @jdeluccia Practice: Document Control Strategy Source: DevOps Audit Defense Toolkit
  35. 35. @RealGeneKim @jdeluccia Practice: Document Control Strategy Source: DevOps Audit Defense Toolkit
  36. 36. @RealGeneKim @jdeluccia ▪ Gained an understanding of the organization and its objectives ▪ Understood how our service fits in and where we jeopardize those objectives ▪ Designed and documented our control environment so that auditors can share our understanding ▪ Enable auditors to do their work effectively What We Have Done
  37. 37. @RealGeneKim @jdeluccia ▪ Save the date: October 21-23, 2014 ▪ DevOps Enterprise is a conference for horses, by horses ▪ Macy’s, Disney, GE Capital, Blackboard, Telstra, US Citizen and Immigration Services, CSG, Raytheon, Ticketmaster/LiveNation, Capital One, Nordstrom, Union Bank of California ▪ Leaders driving DevOps transformations will talk about ▪ The business problem they set out to solve ▪ The obstacles they had to overcome ▪ The business value they created ▪ Submit talks at: DevOps Enterprise Summit
  38. 38. @RealGeneKim @jdeluccia ▪ We don’t need to wait for auditors to learn about DevOps -- by learning about audit, we can successfully bridge the gap ▪ DevOps control environments can be even more secure than traditional control environments ▪ The DevOps Audit Defense Toolkit might be able to help you! ▪ We’d love your scrutiny and case studies! ▪ DevOps Enterprise Summit: ▪ Emailing us:, Conclusion
  39. 39. @RealGeneKim @jdeluccia Ask An Auditor Anything! ▪ Ask the Auditor and the audience anything: ▪ Separation of Duties? ▪ Security beyond checkboxes and non-contextual requirements? ▪ Governance effects of DevOps and/or Agile? ▪ Integration and dialogues and timing with Management, Auditors, and the effect? ▪ Ask Gene on practical examples ▪ Questions for the audience: ▪ Are you using ISO 27034 as a reference architecture?
  40. 40. @RealGeneKim @jdeluccia Results Of Halving Deployment Interval
  41. 41. @RealGeneKim @jdeluccia Results Of Halving Deployment Interval And customers got the feature in half the time! Source: Scott Prugh, CSG
  42. 42. @RealGeneKim @jdeluccia Results Of Halving Deployment Interval Source: Scott Prugh, CSG
  43. 43. @RealGeneKim @jdeluccia Call to Action ● We're looking for case studies ○ Rough life lessons and smooth successes ○ Submit to: ■ DevOps Audit Defense Toolkit: Google+ Community: ● Look at the DevOps Audit Defense Toolkit ● DevOps Enterprise Summit ○