SlideShare a Scribd company logo
@RealGeneKim
@jdeluccia
Session ID:
Gene Kim
James DeLuccia
Keeping The Auditor Away:
DevOps Audit Compliance Case
Studies
@RealGeneKim
@jdeluccia
OMG. Developers Deploying Code?!?
@RealGeneKim
@jdeluccia
Introductions
Gene Kim
▪ Co-author of "The Phoenix Project”
▪ Founder and CTO of Tripwire, Inc. for
13 years
▪ Worked with Jez Humble (co-author
of “Continuous Delivery book) to
benchmark 14K technology
organizations
▪ Co-chaired SOX-404 Scoping
Committee at the Institute of Internal
Auditors (2005)
James DeLuccia
▪ Author, “IT Compliance & Controls”
▪ Ernst & Young, leader for Americas
Certification & Compliance Services
▪ Focus: startups, technology,
governance, security
▪ Patent holder - crypto privacy
comparison system
@RealGeneKim
@jdeluccia
Golly, Why Are You Attending This Talk?
▪ How many people have to deal with compliance?
▪ On a scale of 1-10, how painful are your
interactions with auditors? (1=delightful,
10=awful beyond words?)
@RealGeneKim
@jdeluccia
Problem Statement
Gene ● DevOps and continuous delivery introduce problems with audit,
because the work patterns are so different than traditional SDLC
● Agile also had issues (e.g., testing at end of project, requirements
phase at the beginning), but is not as radical as DevOps
○ tens/hundreds of deploys/day (change is risk; can’t rely on
change approvals, separation of duty)
● No widespread agreement on what DevOps control requirements
should look like
James ● Auditors must work off a mature and testable environment
● They must stake their livelihood that what you say is correct,
completely
● A partnership is needed between you and them to ensure such an
environment exists (of course, it also needs to operate and be
amazing .. but that is another talk)
@RealGeneKim
@jdeluccia
Agenda
▪ The Top-Down, Risk Based Audit Process
▪ What Goes Wrong
▪Scoping
▪Control Testing
▪ Scenarios From The DevOps Audit Defense Toolkit
▪ Ask An Auditor Anything!
@RealGeneKim
@jdeluccia
The DevOps Audit Defense Toolkit
http://bit.ly/DevOpsAudit
James DeLuccia IV
Jeff Gallimore
Gene Kim
Byron Miller
@RealGeneKim
@jdeluccia
What Is Audit
▪ Management is defined as those who are there to achieve the goals of the
organizations, which includes the officers of the company (e.g., CEO, CFO,
etc.), executives and managers, as well as everyone who reports to them.
▪ Includes some board of directors, GRC departments
▪ Audit is defined to be the function inside the organization that resides
outside of management to serve as an independent, objective source of
assurance that the organization can achieve its goals.
▪ Includes internal auditors, external auditors (regulators, assessors,
etc.)
@RealGeneKim
@jdeluccia
Internal Controls
“a process, effected by an organization’s board of
directors, management, and other personnel,
designed to provide reasonable assurance
regarding the achievement of objectives related to
operations, reporting, and compliance.”
- Operations (effectiveness, efficiency)
- Financial Reporting (accuracy of account
balances and values)
- Compliance (with relevant laws and
regulations, contractual obligations: PCI DSS,
US Export Law, FEDRAMP, SOC-2)
Source: http://coso.org (Committee of Sponsoring Organizations for the Commission on Fraudulent Financial Reporting)
@RealGeneKim
@jdeluccia
How Audit Plans Are Built And Run
▪ Business objectives
▪ Risks
▪ Control objectives
▪ Control procedures
Unfortunately, most contact with auditors start with control procedures…
It’s totally appropriate to ask to show work and start from beginning...
@RealGeneKim
@jdeluccia
The Audit Cycle
▪ Planning
▪ Gaining an understanding of the organization
▪ Scoping
▪ Sampling, reporting period, types of evidence needed, recipient of
report
▪ Schedule
▪ Fieldwork
▪ Controls testing
▪ Substantive testing
▪ Reporting
▪ Management responses
▪ Attestation by auditor and delivered to regulator/clients
@RealGeneKim
@jdeluccia
When Scoping Goes Wrong
@RealGeneKim
@jdeluccia
When Scoping Goes Wrong
▪ 2001: Enron fails ($63B market
cap), Arthur Andersen dissolution
▪ 2002: WorldCom (peak $117B
market cap)
▪ Leads to Sarbanes-Oxley Act of
2002
@RealGeneKim
@jdeluccia
When Scoping Goes Wrong
Source: KPMG
@RealGeneKim
@jdeluccia
Problem: Bottom Up Auditing
Source: ISACA
@RealGeneKim
@jdeluccia
Analysis: Audit control testing work was scoped properly,
linking controls to compliance objectives and risk.
Control failures must result potentially undetected
material financial reporting errors
The Problem: Improperly Scoped Audits
@RealGeneKim
@jdeluccia
Financial Reporting Material Weakness
What happens when an audit generates a material weakness?
@RealGeneKim
@jdeluccia
Under-Scoping Operating Risk
@RealGeneKim
@jdeluccia
▪ When we don’t understand why we are being audited
▪ “Why are we doing this audit?” (customers, SOX, regulatory; who is it
for?)
▪ When we are asked for something we don’t have (e.g., “evidence of SoD or
change approvals)
▪ “What is the control objective? Can we rewrite the control procedure
for this asset?”
▪ Do this before the auditor shows up
When Auditors Attack Unexpectedly
These are delicate conversations, with potentially large
impacts on scope, cost, risk...
@RealGeneKim
@jdeluccia
▪ If we are reacting to these conversations before we’ve done any of our
homework, we may be trouble
▪ Extra work (average time to respond to audit is 40 hours; that’s one
Dev sprint)
▪ Audit cost and schedule overages: a 3 hour audit test just turned into a
16 hour audit project
▪ Reduced confidence from auditors, increased visibility from audit and
management
When Auditors Attack Unexpectedly
The DevOps Audit Defense Toolkit
@RealGeneKim
@jdeluccia
The DevOps Audit Defense Toolkit
http://bit.ly/DevOpsAudit
James DeLuccia IV
Jeff Gallimore
Gene Kim
Byron Miller
@RealGeneKim
@jdeluccia
Practice: Enabling A Shared Understanding
Source: DevOps Audit Defense Toolkit
http://bit.ly/DevOpsAudit
@RealGeneKim
@jdeluccia
Practice: Enabling A Shared Understanding
Source: DevOps Audit Defense Toolkit
http://bit.ly/DevOpsAudit
@RealGeneKim
@jdeluccia
Practice: Enabling A Shared Understanding
Source: DevOps Audit Defense Toolkit
http://bit.ly/DevOpsAudit
@RealGeneKim
@jdeluccia
Walk Through Of DevOps Risk And
Control Strategies
What does an effective DevOps
control environment look like?
@RealGeneKim
@jdeluccia
DevOps Orgs Actually Love Process
“Facebook values people, tools, and way, way
down the list is process.”
Jay Parikh
VP Infrastructure Engineering,
Facebook
Not true! They are conflating “process” and
“approvals!”
@RealGeneKim
@jdeluccia
High Performing DevOps Orgs
Source: 2014 Puppet Labs State Of DevOps
30xmore frequent
deployments
8,000xfaster lead times
than their peers
@RealGeneKim
@jdeluccia
High Performing DevOps Orgs
Source: 2014 Puppet Labs State Of DevOps
2xhigher change
success rates
12xfaster mean time to
recover (MTTR)
@RealGeneKim
@jdeluccia
High Performing DevOps Orgs
Source: 2014 Puppet Labs State Of DevOps
more likely to exceed
profitability,
market share &
productivity goals
2x higher market
capitalization growth
over 3 years*
50%
@RealGeneKim
@jdeluccia
Top Predictors Of Performance
▪ Version control of all production artifacts
▪ Continuous integration and deployment
▪ Automated acceptance testing
▪ Peer-review of production changes (vs. external change
approval)
▪ High trust culture
▪ Proactive monitoring of the production environment
▪ Win-win relationship between Dev and Ops
@RealGeneKim
@jdeluccia
DevOps Orgs Need Hardcopy
DevOps has higher automation and closer monitoring controls than
traditional deployment environments and therefore reduced points
for human failure
The documentation of ephemeral systems, tools, and deployment
processes into a hardcopy breakdown will communicate and
simplify this management long term.
@RealGeneKim
@jdeluccia
Practice: Document Risks & Control
Strategy
Source: DevOps Audit Defense Toolkit
http://bit.ly/DevOpsAudit
@RealGeneKim
@jdeluccia
Practice: Document Control Strategy
Source: DevOps Audit Defense Toolkit
http://bit.ly/DevOpsAudit
@RealGeneKim
@jdeluccia
Practice: Document Control Strategy
Source: DevOps Audit Defense Toolkit
http://bit.ly/DevOpsAudit
@RealGeneKim
@jdeluccia
Practice: Document Control Strategy
Source: DevOps Audit Defense Toolkit
http://bit.ly/DevOpsAudit
@RealGeneKim
@jdeluccia
▪ Gained an understanding of the organization and its
objectives
▪ Understood how our service fits in and where we jeopardize
those objectives
▪ Designed and documented our control environment so that
auditors can share our understanding
▪ Enable auditors to do their work effectively
What We Have Done
@RealGeneKim
@jdeluccia
▪ Save the date: October 21-23, 2014
▪ DevOps Enterprise is a conference for horses, by horses
▪ Macy’s, Disney, GE Capital, Blackboard, Telstra, US Citizen and Immigration Services, CSG,
Raytheon, Ticketmaster/LiveNation, Capital One, Nordstrom, Union Bank of California
▪ Leaders driving DevOps transformations will talk about
▪ The business problem they set out to solve
▪ The obstacles they had to overcome
▪ The business value they created
▪ Submit talks at: http://devopsenterprisesummit.com/
DevOps Enterprise Summit
@RealGeneKim
@jdeluccia
▪ We don’t need to wait for auditors to learn about DevOps -- by learning about audit, we
can successfully bridge the gap
▪ DevOps control environments can be even more secure than traditional control
environments
▪ The DevOps Audit Defense Toolkit might be able to help you! http://bit.ly/DevOpsAudit
▪ We’d love your scrutiny and case studies!
▪ DevOps Enterprise Summit: http://devopsenterprise.io
▪ Emailing us: genek@realgenekim.me, jdeluccia@gmail.com
Conclusion
@RealGeneKim
@jdeluccia
Ask An Auditor Anything!
▪ Ask the Auditor and the audience anything:
▪ Separation of Duties?
▪ Security beyond checkboxes and non-contextual
requirements?
▪ Governance effects of DevOps and/or Agile?
▪ Integration and dialogues and timing with Management,
Auditors, and the effect?
▪ Ask Gene on practical examples
▪ Questions for the audience:
▪ Are you using ISO 27034 as a reference architecture?
@RealGeneKim
@jdeluccia
Results Of Halving Deployment Interval
@RealGeneKim
@jdeluccia
Results Of Halving Deployment Interval
And customers got the
feature in half the time!
Source: Scott Prugh, CSG
@RealGeneKim
@jdeluccia
Results Of Halving Deployment Interval
Source: Scott Prugh, CSG
@RealGeneKim
@jdeluccia
Call to Action
● We're looking for case studies
○ Rough life lessons and smooth successes
○ Submit to:
■ DevOps Audit Defense Toolkit: Google+ Community:
http://bit.ly/DevOpsAudit
● Look at the DevOps Audit Defense Toolkit
● DevOps Enterprise Summit
○ http://devopsenterprise.io/

More Related Content

What's hot

Enterprise Service Management & IT Operations Management Coming Together
Enterprise Service Management & IT Operations Management Coming TogetherEnterprise Service Management & IT Operations Management Coming Together
Enterprise Service Management & IT Operations Management Coming Together
OpsRamp
 
ISO 27001 Training | ISO 27001 Internal Auditor Training | ISMS Internal Audi...
ISO 27001 Training | ISO 27001 Internal Auditor Training | ISMS Internal Audi...ISO 27001 Training | ISO 27001 Internal Auditor Training | ISMS Internal Audi...
ISO 27001 Training | ISO 27001 Internal Auditor Training | ISMS Internal Audi...
himalya sharma
 
Casos de uso de las tecnologías emergentes y la Inteligencia Artificial en la...
Casos de uso de las tecnologías emergentes y la Inteligencia Artificial en la...Casos de uso de las tecnologías emergentes y la Inteligencia Artificial en la...
Casos de uso de las tecnologías emergentes y la Inteligencia Artificial en la...
Comisión de Regulación de Comunicaciones
 
What is GRC – Governance, Risk and Compliance
What is GRC – Governance, Risk and Compliance What is GRC – Governance, Risk and Compliance
What is GRC – Governance, Risk and Compliance
BOC Group
 
Impact of Generative AI in Cybersecurity - How can ISO/IEC 27032 help?
Impact of Generative AI in Cybersecurity - How can ISO/IEC 27032 help?Impact of Generative AI in Cybersecurity - How can ISO/IEC 27032 help?
Impact of Generative AI in Cybersecurity - How can ISO/IEC 27032 help?
PECB
 
Business Transformation Powerpoint Presentation Slides
Business Transformation Powerpoint Presentation SlidesBusiness Transformation Powerpoint Presentation Slides
Business Transformation Powerpoint Presentation Slides
SlideTeam
 
Crafting an End-to-End Pharma GRC Strategy
Crafting an End-to-End Pharma GRC StrategyCrafting an End-to-End Pharma GRC Strategy
Crafting an End-to-End Pharma GRC Strategy
Cognizant
 
Imagining a New Customer Experience
Imagining a New Customer ExperienceImagining a New Customer Experience
Imagining a New Customer Experience
accenture
 
IT Infrastructure Managed Services and RIMS
IT Infrastructure Managed Services and RIMSIT Infrastructure Managed Services and RIMS
IT Infrastructure Managed Services and RIMS
Razak Mohammed Ali
 
Business Value Measurements and the Solution Design Framework
Business Value Measurements and the Solution Design FrameworkBusiness Value Measurements and the Solution Design Framework
Business Value Measurements and the Solution Design Framework
Leo Barella
 
IT Governance - COBIT Perspective
IT Governance - COBIT PerspectiveIT Governance - COBIT Perspective
IT Governance - COBIT Perspective
Sayyed Zakir Ali Rizwe
 
Enterprise Architecture Governance
Enterprise Architecture GovernanceEnterprise Architecture Governance
Enterprise Architecture Governance
Rakesh Sharan
 
2019 Top Lessons Learned Since the Phoenix Project Was Released
2019 Top Lessons Learned Since the Phoenix Project Was Released2019 Top Lessons Learned Since the Phoenix Project Was Released
2019 Top Lessons Learned Since the Phoenix Project Was Released
Gene Kim
 
Basics in IT Audit and Application Control Testing
Basics in IT Audit and Application Control Testing Basics in IT Audit and Application Control Testing
Basics in IT Audit and Application Control Testing
Dinesh O Bareja
 
Supercharge Your Project Management Skills with CHATGPT practical - UK.pdf
Supercharge Your Project Management Skills with CHATGPT practical - UK.pdfSupercharge Your Project Management Skills with CHATGPT practical - UK.pdf
Supercharge Your Project Management Skills with CHATGPT practical - UK.pdf
PMIUKChapter
 
An Introduction to IT Management with COBIT 2019
An Introduction to IT Management with COBIT 2019An Introduction to IT Management with COBIT 2019
An Introduction to IT Management with COBIT 2019
Gregor Polančič
 
ISO 27001:2022 Introduction
ISO 27001:2022 IntroductionISO 27001:2022 Introduction
IT4IT - The Full Story for Digital Transformation - Part 1
IT4IT - The Full Story for Digital Transformation - Part 1IT4IT - The Full Story for Digital Transformation - Part 1
IT4IT - The Full Story for Digital Transformation - Part 1
Mohamed Zakarya Abdelgawad
 
Managed Services - Explained
Managed Services - ExplainedManaged Services - Explained
Managed Services - Explained
Ghassan Chahine
 
It governance
It governanceIt governance
It governance
Mahetab Khan
 

What's hot (20)

Enterprise Service Management & IT Operations Management Coming Together
Enterprise Service Management & IT Operations Management Coming TogetherEnterprise Service Management & IT Operations Management Coming Together
Enterprise Service Management & IT Operations Management Coming Together
 
ISO 27001 Training | ISO 27001 Internal Auditor Training | ISMS Internal Audi...
ISO 27001 Training | ISO 27001 Internal Auditor Training | ISMS Internal Audi...ISO 27001 Training | ISO 27001 Internal Auditor Training | ISMS Internal Audi...
ISO 27001 Training | ISO 27001 Internal Auditor Training | ISMS Internal Audi...
 
Casos de uso de las tecnologías emergentes y la Inteligencia Artificial en la...
Casos de uso de las tecnologías emergentes y la Inteligencia Artificial en la...Casos de uso de las tecnologías emergentes y la Inteligencia Artificial en la...
Casos de uso de las tecnologías emergentes y la Inteligencia Artificial en la...
 
What is GRC – Governance, Risk and Compliance
What is GRC – Governance, Risk and Compliance What is GRC – Governance, Risk and Compliance
What is GRC – Governance, Risk and Compliance
 
Impact of Generative AI in Cybersecurity - How can ISO/IEC 27032 help?
Impact of Generative AI in Cybersecurity - How can ISO/IEC 27032 help?Impact of Generative AI in Cybersecurity - How can ISO/IEC 27032 help?
Impact of Generative AI in Cybersecurity - How can ISO/IEC 27032 help?
 
Business Transformation Powerpoint Presentation Slides
Business Transformation Powerpoint Presentation SlidesBusiness Transformation Powerpoint Presentation Slides
Business Transformation Powerpoint Presentation Slides
 
Crafting an End-to-End Pharma GRC Strategy
Crafting an End-to-End Pharma GRC StrategyCrafting an End-to-End Pharma GRC Strategy
Crafting an End-to-End Pharma GRC Strategy
 
Imagining a New Customer Experience
Imagining a New Customer ExperienceImagining a New Customer Experience
Imagining a New Customer Experience
 
IT Infrastructure Managed Services and RIMS
IT Infrastructure Managed Services and RIMSIT Infrastructure Managed Services and RIMS
IT Infrastructure Managed Services and RIMS
 
Business Value Measurements and the Solution Design Framework
Business Value Measurements and the Solution Design FrameworkBusiness Value Measurements and the Solution Design Framework
Business Value Measurements and the Solution Design Framework
 
IT Governance - COBIT Perspective
IT Governance - COBIT PerspectiveIT Governance - COBIT Perspective
IT Governance - COBIT Perspective
 
Enterprise Architecture Governance
Enterprise Architecture GovernanceEnterprise Architecture Governance
Enterprise Architecture Governance
 
2019 Top Lessons Learned Since the Phoenix Project Was Released
2019 Top Lessons Learned Since the Phoenix Project Was Released2019 Top Lessons Learned Since the Phoenix Project Was Released
2019 Top Lessons Learned Since the Phoenix Project Was Released
 
Basics in IT Audit and Application Control Testing
Basics in IT Audit and Application Control Testing Basics in IT Audit and Application Control Testing
Basics in IT Audit and Application Control Testing
 
Supercharge Your Project Management Skills with CHATGPT practical - UK.pdf
Supercharge Your Project Management Skills with CHATGPT practical - UK.pdfSupercharge Your Project Management Skills with CHATGPT practical - UK.pdf
Supercharge Your Project Management Skills with CHATGPT practical - UK.pdf
 
An Introduction to IT Management with COBIT 2019
An Introduction to IT Management with COBIT 2019An Introduction to IT Management with COBIT 2019
An Introduction to IT Management with COBIT 2019
 
ISO 27001:2022 Introduction
ISO 27001:2022 IntroductionISO 27001:2022 Introduction
ISO 27001:2022 Introduction
 
IT4IT - The Full Story for Digital Transformation - Part 1
IT4IT - The Full Story for Digital Transformation - Part 1IT4IT - The Full Story for Digital Transformation - Part 1
IT4IT - The Full Story for Digital Transformation - Part 1
 
Managed Services - Explained
Managed Services - ExplainedManaged Services - Explained
Managed Services - Explained
 
It governance
It governanceIt governance
It governance
 

Viewers also liked

Leading A DevOps Transformation: Lessons Learned
Leading A DevOps Transformation: Lessons LearnedLeading A DevOps Transformation: Lessons Learned
Leading A DevOps Transformation: Lessons Learned
Gene Kim
 
2014 State Of DevOps Findings! Velocity Conference
2014 State Of DevOps Findings! Velocity Conference2014 State Of DevOps Findings! Velocity Conference
2014 State Of DevOps Findings! Velocity Conference
Gene Kim
 
DevOps: Who Will Create $2.6 Trillion In Business Value Per Year?
DevOps: Who Will Create $2.6 Trillion In Business Value Per Year?DevOps: Who Will Create $2.6 Trillion In Business Value Per Year?
DevOps: Who Will Create $2.6 Trillion In Business Value Per Year?
Gene Kim
 
2013 Velocity DevOps Metrics -- It's Not Just For WebOps Any More!
2013 Velocity DevOps Metrics -- It's Not Just For WebOps Any More!2013 Velocity DevOps Metrics -- It's Not Just For WebOps Any More!
2013 Velocity DevOps Metrics -- It's Not Just For WebOps Any More!
Gene Kim
 
Why Everyone Needs DevOps Now: 15 Year Study Of High Performing Technology Orgs
Why Everyone Needs DevOps Now: 15 Year Study Of High Performing Technology OrgsWhy Everyone Needs DevOps Now: 15 Year Study Of High Performing Technology Orgs
Why Everyone Needs DevOps Now: 15 Year Study Of High Performing Technology Orgs
Gene Kim
 
CampDevOps keynote - DevOps: Using 'Lean' to eliminate Bottlenecks
CampDevOps keynote - DevOps: Using 'Lean' to eliminate BottlenecksCampDevOps keynote - DevOps: Using 'Lean' to eliminate Bottlenecks
CampDevOps keynote - DevOps: Using 'Lean' to eliminate Bottlenecks
Sanjeev Sharma
 
2012 Velocity London: DevOps Patterns Distilled
2012 Velocity London: DevOps Patterns Distilled2012 Velocity London: DevOps Patterns Distilled
2012 Velocity London: DevOps Patterns Distilled
Gene Kim
 
Project controller performance appraisal
Project controller performance appraisalProject controller performance appraisal
Project controller performance appraisal
anniejenny66
 
Don't be Afraid of Scary Code Webcast
Don't be Afraid of Scary Code WebcastDon't be Afraid of Scary Code Webcast
Don't be Afraid of Scary Code Webcast
Compuware
 
Modern Interface to Mainframe - The Compuware Workbench (B. Ebner)
Modern Interface to Mainframe - The Compuware Workbench (B. Ebner)Modern Interface to Mainframe - The Compuware Workbench (B. Ebner)
Modern Interface to Mainframe - The Compuware Workbench (B. Ebner)
NRB
 
Continuous delivery with Jenkins, Docker and Mesos/Marathon - jbcnconf
Continuous delivery with Jenkins, Docker and Mesos/Marathon - jbcnconfContinuous delivery with Jenkins, Docker and Mesos/Marathon - jbcnconf
Continuous delivery with Jenkins, Docker and Mesos/Marathon - jbcnconf
Julia Mateo
 
A Customer's Journey to Mainstreaming the Mainframe Webcast On-demand Replay
A Customer's Journey to Mainstreaming the Mainframe Webcast On-demand ReplayA Customer's Journey to Mainstreaming the Mainframe Webcast On-demand Replay
A Customer's Journey to Mainstreaming the Mainframe Webcast On-demand Replay
Compuware
 
Access by Default
Access by DefaultAccess by Default
Access by Default
Kendra Skeene
 
Leadership Styles Your Team Needs
Leadership Styles Your Team NeedsLeadership Styles Your Team Needs
Leadership Styles Your Team Needs
Joshua Howard
 
Business Models
Business ModelsBusiness Models
Business Models
Morten Gade
 
Agile Experience Design Framework
Agile Experience Design FrameworkAgile Experience Design Framework
Agile Experience Design Framework
Kazumichi (Mario) Sakata
 
Startup Metrics for Pirates (KAUST, Nov 2013)
Startup Metrics for Pirates (KAUST, Nov 2013)Startup Metrics for Pirates (KAUST, Nov 2013)
Startup Metrics for Pirates (KAUST, Nov 2013)
Dave McClure
 
Jesse Robbins Keynote - Hacking Culture @ Cloud Expo Europe 2013
Jesse Robbins Keynote - Hacking Culture @ Cloud Expo Europe 2013Jesse Robbins Keynote - Hacking Culture @ Cloud Expo Europe 2013
Jesse Robbins Keynote - Hacking Culture @ Cloud Expo Europe 2013
Jesse Robbins
 
Doors, Walls and Old Trees: Prioritizing to Get Simple
Doors, Walls and Old Trees: Prioritizing to Get SimpleDoors, Walls and Old Trees: Prioritizing to Get Simple
Doors, Walls and Old Trees: Prioritizing to Get Simple
Jason Ulaszek
 
Operational Auditing
Operational AuditingOperational Auditing
Operational Auditing
ahmad bassiouny
 

Viewers also liked (20)

Leading A DevOps Transformation: Lessons Learned
Leading A DevOps Transformation: Lessons LearnedLeading A DevOps Transformation: Lessons Learned
Leading A DevOps Transformation: Lessons Learned
 
2014 State Of DevOps Findings! Velocity Conference
2014 State Of DevOps Findings! Velocity Conference2014 State Of DevOps Findings! Velocity Conference
2014 State Of DevOps Findings! Velocity Conference
 
DevOps: Who Will Create $2.6 Trillion In Business Value Per Year?
DevOps: Who Will Create $2.6 Trillion In Business Value Per Year?DevOps: Who Will Create $2.6 Trillion In Business Value Per Year?
DevOps: Who Will Create $2.6 Trillion In Business Value Per Year?
 
2013 Velocity DevOps Metrics -- It's Not Just For WebOps Any More!
2013 Velocity DevOps Metrics -- It's Not Just For WebOps Any More!2013 Velocity DevOps Metrics -- It's Not Just For WebOps Any More!
2013 Velocity DevOps Metrics -- It's Not Just For WebOps Any More!
 
Why Everyone Needs DevOps Now: 15 Year Study Of High Performing Technology Orgs
Why Everyone Needs DevOps Now: 15 Year Study Of High Performing Technology OrgsWhy Everyone Needs DevOps Now: 15 Year Study Of High Performing Technology Orgs
Why Everyone Needs DevOps Now: 15 Year Study Of High Performing Technology Orgs
 
CampDevOps keynote - DevOps: Using 'Lean' to eliminate Bottlenecks
CampDevOps keynote - DevOps: Using 'Lean' to eliminate BottlenecksCampDevOps keynote - DevOps: Using 'Lean' to eliminate Bottlenecks
CampDevOps keynote - DevOps: Using 'Lean' to eliminate Bottlenecks
 
2012 Velocity London: DevOps Patterns Distilled
2012 Velocity London: DevOps Patterns Distilled2012 Velocity London: DevOps Patterns Distilled
2012 Velocity London: DevOps Patterns Distilled
 
Project controller performance appraisal
Project controller performance appraisalProject controller performance appraisal
Project controller performance appraisal
 
Don't be Afraid of Scary Code Webcast
Don't be Afraid of Scary Code WebcastDon't be Afraid of Scary Code Webcast
Don't be Afraid of Scary Code Webcast
 
Modern Interface to Mainframe - The Compuware Workbench (B. Ebner)
Modern Interface to Mainframe - The Compuware Workbench (B. Ebner)Modern Interface to Mainframe - The Compuware Workbench (B. Ebner)
Modern Interface to Mainframe - The Compuware Workbench (B. Ebner)
 
Continuous delivery with Jenkins, Docker and Mesos/Marathon - jbcnconf
Continuous delivery with Jenkins, Docker and Mesos/Marathon - jbcnconfContinuous delivery with Jenkins, Docker and Mesos/Marathon - jbcnconf
Continuous delivery with Jenkins, Docker and Mesos/Marathon - jbcnconf
 
A Customer's Journey to Mainstreaming the Mainframe Webcast On-demand Replay
A Customer's Journey to Mainstreaming the Mainframe Webcast On-demand ReplayA Customer's Journey to Mainstreaming the Mainframe Webcast On-demand Replay
A Customer's Journey to Mainstreaming the Mainframe Webcast On-demand Replay
 
Access by Default
Access by DefaultAccess by Default
Access by Default
 
Leadership Styles Your Team Needs
Leadership Styles Your Team NeedsLeadership Styles Your Team Needs
Leadership Styles Your Team Needs
 
Business Models
Business ModelsBusiness Models
Business Models
 
Agile Experience Design Framework
Agile Experience Design FrameworkAgile Experience Design Framework
Agile Experience Design Framework
 
Startup Metrics for Pirates (KAUST, Nov 2013)
Startup Metrics for Pirates (KAUST, Nov 2013)Startup Metrics for Pirates (KAUST, Nov 2013)
Startup Metrics for Pirates (KAUST, Nov 2013)
 
Jesse Robbins Keynote - Hacking Culture @ Cloud Expo Europe 2013
Jesse Robbins Keynote - Hacking Culture @ Cloud Expo Europe 2013Jesse Robbins Keynote - Hacking Culture @ Cloud Expo Europe 2013
Jesse Robbins Keynote - Hacking Culture @ Cloud Expo Europe 2013
 
Doors, Walls and Old Trees: Prioritizing to Get Simple
Doors, Walls and Old Trees: Prioritizing to Get SimpleDoors, Walls and Old Trees: Prioritizing to Get Simple
Doors, Walls and Old Trees: Prioritizing to Get Simple
 
Operational Auditing
Operational AuditingOperational Auditing
Operational Auditing
 

Similar to Keeping The Auditor Away: DevOps Audit Compliance Case Studies

Kim itSMF New England: ITIL at Ludicrous Speeds - Rugged DevOps 6a
Kim itSMF New England: ITIL at Ludicrous Speeds - Rugged DevOps 6aKim itSMF New England: ITIL at Ludicrous Speeds - Rugged DevOps 6a
Kim itSMF New England: ITIL at Ludicrous Speeds - Rugged DevOps 6a
Gene Kim
 
ServiceNow ITIL at Ludicrous Speeds - Rugged DevOps
ServiceNow  ITIL at Ludicrous Speeds - Rugged DevOpsServiceNow  ITIL at Ludicrous Speeds - Rugged DevOps
ServiceNow ITIL at Ludicrous Speeds - Rugged DevOps
Gene Kim
 
Kim IT Pro Forum Eugene: IT at Ludicrous Speeds - rugged dev ops
Kim IT Pro Forum Eugene: IT at Ludicrous Speeds - rugged dev opsKim IT Pro Forum Eugene: IT at Ludicrous Speeds - rugged dev ops
Kim IT Pro Forum Eugene: IT at Ludicrous Speeds - rugged dev ops
Gene Kim
 
2012 SxSW When IT Says No by Gene Kim
2012 SxSW When IT Says No by Gene Kim2012 SxSW When IT Says No by Gene Kim
2012 SxSW When IT Says No by Gene Kim
Gene Kim
 
SecureWorld Kim - Infosec at Ludicrous Speeds - Rugged DevOps 6a
SecureWorld   Kim - Infosec at Ludicrous Speeds - Rugged DevOps 6aSecureWorld   Kim - Infosec at Ludicrous Speeds - Rugged DevOps 6a
SecureWorld Kim - Infosec at Ludicrous Speeds - Rugged DevOps 6a
Gene Kim
 
2019 12 Clojure/conj: Love Letter To Clojure, and A Datomic Experience Report
2019 12 Clojure/conj: Love Letter To Clojure, and A Datomic Experience Report2019 12 Clojure/conj: Love Letter To Clojure, and A Datomic Experience Report
2019 12 Clojure/conj: Love Letter To Clojure, and A Datomic Experience Report
Gene Kim
 
DevOps Kanban Meet Up 3/22/12
DevOps Kanban Meet Up 3/22/12DevOps Kanban Meet Up 3/22/12
DevOps Kanban Meet Up 3/22/12
Gene Kim
 
Fear and Loathing in Agility: Long Live the Accounting Department
Fear and Loathing in Agility: Long Live the Accounting DepartmentFear and Loathing in Agility: Long Live the Accounting Department
Fear and Loathing in Agility: Long Live the Accounting Department
Accenture | SolutionsIQ
 
Proposed Title Fear and Loathing in Agility: Long Live the Accounting Departm...
Proposed Title Fear and Loathing in Agility: Long Live the Accounting Departm...Proposed Title Fear and Loathing in Agility: Long Live the Accounting Departm...
Proposed Title Fear and Loathing in Agility: Long Live the Accounting Departm...
Laszlo Szalvay
 
My Top Five DevOps Learnings
My Top Five DevOps LearningsMy Top Five DevOps Learnings
My Top Five DevOps Learnings
Predix
 
The Unicorn Project and The Five Ideals (Updated Dec 2019)
The Unicorn Project and The Five Ideals (Updated Dec 2019)The Unicorn Project and The Five Ideals (Updated Dec 2019)
The Unicorn Project and The Five Ideals (Updated Dec 2019)
Gene Kim
 
Edit Privacy Settings Analytics FREE Collect Leads Micro Focus DevOps Drive-i...
Edit Privacy Settings Analytics FREE Collect Leads Micro Focus DevOps Drive-i...Edit Privacy Settings Analytics FREE Collect Leads Micro Focus DevOps Drive-i...
Edit Privacy Settings Analytics FREE Collect Leads Micro Focus DevOps Drive-i...
Serena Software
 
Top Lessons Learned While Researching and Writing The DevOps Handbook
Top Lessons Learned While Researching and Writing The DevOps HandbookTop Lessons Learned While Researching and Writing The DevOps Handbook
Top Lessons Learned While Researching and Writing The DevOps Handbook
Dynatrace
 
DevOps and Digital Transformation
DevOps and Digital TransformationDevOps and Digital Transformation
DevOps and Digital Transformation
Omid Shariati
 
Reclaiming Agile Development
Reclaiming Agile Development Reclaiming Agile Development
Reclaiming Agile Development
Centric Consulting
 
Owasp summit debrief v1.0 (jun 2017)
Owasp summit debrief v1.0 (jun 2017)Owasp summit debrief v1.0 (jun 2017)
Owasp summit debrief v1.0 (jun 2017)
owaspsummit
 
How Can We Better Sell DevOps?
How Can We Better Sell DevOps?How Can We Better Sell DevOps?
How Can We Better Sell DevOps?
Gene Kim
 
How Do We Better Sell DevOps? - PuppetConf 2013
How Do We Better Sell DevOps? - PuppetConf 2013How Do We Better Sell DevOps? - PuppetConf 2013
How Do We Better Sell DevOps? - PuppetConf 2013
Puppet
 
Introduction to Recipes for Agile Governance in the Enterprise (RAGE)
Introduction to Recipes for Agile Governance in the Enterprise (RAGE)Introduction to Recipes for Agile Governance in the Enterprise (RAGE)
Introduction to Recipes for Agile Governance in the Enterprise (RAGE)
Cprime
 
6 ways DevOps helped PrepSportswear move from monolith to microservices
6 ways DevOps helped PrepSportswear move from monolith to microservices6 ways DevOps helped PrepSportswear move from monolith to microservices
6 ways DevOps helped PrepSportswear move from monolith to microservices
Dynatrace
 

Similar to Keeping The Auditor Away: DevOps Audit Compliance Case Studies (20)

Kim itSMF New England: ITIL at Ludicrous Speeds - Rugged DevOps 6a
Kim itSMF New England: ITIL at Ludicrous Speeds - Rugged DevOps 6aKim itSMF New England: ITIL at Ludicrous Speeds - Rugged DevOps 6a
Kim itSMF New England: ITIL at Ludicrous Speeds - Rugged DevOps 6a
 
ServiceNow ITIL at Ludicrous Speeds - Rugged DevOps
ServiceNow  ITIL at Ludicrous Speeds - Rugged DevOpsServiceNow  ITIL at Ludicrous Speeds - Rugged DevOps
ServiceNow ITIL at Ludicrous Speeds - Rugged DevOps
 
Kim IT Pro Forum Eugene: IT at Ludicrous Speeds - rugged dev ops
Kim IT Pro Forum Eugene: IT at Ludicrous Speeds - rugged dev opsKim IT Pro Forum Eugene: IT at Ludicrous Speeds - rugged dev ops
Kim IT Pro Forum Eugene: IT at Ludicrous Speeds - rugged dev ops
 
2012 SxSW When IT Says No by Gene Kim
2012 SxSW When IT Says No by Gene Kim2012 SxSW When IT Says No by Gene Kim
2012 SxSW When IT Says No by Gene Kim
 
SecureWorld Kim - Infosec at Ludicrous Speeds - Rugged DevOps 6a
SecureWorld   Kim - Infosec at Ludicrous Speeds - Rugged DevOps 6aSecureWorld   Kim - Infosec at Ludicrous Speeds - Rugged DevOps 6a
SecureWorld Kim - Infosec at Ludicrous Speeds - Rugged DevOps 6a
 
2019 12 Clojure/conj: Love Letter To Clojure, and A Datomic Experience Report
2019 12 Clojure/conj: Love Letter To Clojure, and A Datomic Experience Report2019 12 Clojure/conj: Love Letter To Clojure, and A Datomic Experience Report
2019 12 Clojure/conj: Love Letter To Clojure, and A Datomic Experience Report
 
DevOps Kanban Meet Up 3/22/12
DevOps Kanban Meet Up 3/22/12DevOps Kanban Meet Up 3/22/12
DevOps Kanban Meet Up 3/22/12
 
Fear and Loathing in Agility: Long Live the Accounting Department
Fear and Loathing in Agility: Long Live the Accounting DepartmentFear and Loathing in Agility: Long Live the Accounting Department
Fear and Loathing in Agility: Long Live the Accounting Department
 
Proposed Title Fear and Loathing in Agility: Long Live the Accounting Departm...
Proposed Title Fear and Loathing in Agility: Long Live the Accounting Departm...Proposed Title Fear and Loathing in Agility: Long Live the Accounting Departm...
Proposed Title Fear and Loathing in Agility: Long Live the Accounting Departm...
 
My Top Five DevOps Learnings
My Top Five DevOps LearningsMy Top Five DevOps Learnings
My Top Five DevOps Learnings
 
The Unicorn Project and The Five Ideals (Updated Dec 2019)
The Unicorn Project and The Five Ideals (Updated Dec 2019)The Unicorn Project and The Five Ideals (Updated Dec 2019)
The Unicorn Project and The Five Ideals (Updated Dec 2019)
 
Edit Privacy Settings Analytics FREE Collect Leads Micro Focus DevOps Drive-i...
Edit Privacy Settings Analytics FREE Collect Leads Micro Focus DevOps Drive-i...Edit Privacy Settings Analytics FREE Collect Leads Micro Focus DevOps Drive-i...
Edit Privacy Settings Analytics FREE Collect Leads Micro Focus DevOps Drive-i...
 
Top Lessons Learned While Researching and Writing The DevOps Handbook
Top Lessons Learned While Researching and Writing The DevOps HandbookTop Lessons Learned While Researching and Writing The DevOps Handbook
Top Lessons Learned While Researching and Writing The DevOps Handbook
 
DevOps and Digital Transformation
DevOps and Digital TransformationDevOps and Digital Transformation
DevOps and Digital Transformation
 
Reclaiming Agile Development
Reclaiming Agile Development Reclaiming Agile Development
Reclaiming Agile Development
 
Owasp summit debrief v1.0 (jun 2017)
Owasp summit debrief v1.0 (jun 2017)Owasp summit debrief v1.0 (jun 2017)
Owasp summit debrief v1.0 (jun 2017)
 
How Can We Better Sell DevOps?
How Can We Better Sell DevOps?How Can We Better Sell DevOps?
How Can We Better Sell DevOps?
 
How Do We Better Sell DevOps? - PuppetConf 2013
How Do We Better Sell DevOps? - PuppetConf 2013How Do We Better Sell DevOps? - PuppetConf 2013
How Do We Better Sell DevOps? - PuppetConf 2013
 
Introduction to Recipes for Agile Governance in the Enterprise (RAGE)
Introduction to Recipes for Agile Governance in the Enterprise (RAGE)Introduction to Recipes for Agile Governance in the Enterprise (RAGE)
Introduction to Recipes for Agile Governance in the Enterprise (RAGE)
 
6 ways DevOps helped PrepSportswear move from monolith to microservices
6 ways DevOps helped PrepSportswear move from monolith to microservices6 ways DevOps helped PrepSportswear move from monolith to microservices
6 ways DevOps helped PrepSportswear move from monolith to microservices
 

More from Gene Kim

Speaker Recording Tips For Virtual DevOps Enterprise (And Why We're Pre-Recor...
Speaker Recording Tips For Virtual DevOps Enterprise (And Why We're Pre-Recor...Speaker Recording Tips For Virtual DevOps Enterprise (And Why We're Pre-Recor...
Speaker Recording Tips For Virtual DevOps Enterprise (And Why We're Pre-Recor...
Gene Kim
 
GitHub Universe: 2019: Exemplars, Laggards, and Hoarders A Data-driven Look a...
GitHub Universe: 2019: Exemplars, Laggards, and Hoarders A Data-driven Look a...GitHub Universe: 2019: Exemplars, Laggards, and Hoarders A Data-driven Look a...
GitHub Universe: 2019: Exemplars, Laggards, and Hoarders A Data-driven Look a...
Gene Kim
 
The Unicorn Project and The Five Ideals (older: see notes for newer version)
The Unicorn Project and The Five Ideals (older: see notes for newer version)The Unicorn Project and The Five Ideals (older: see notes for newer version)
The Unicorn Project and The Five Ideals (older: see notes for newer version)
Gene Kim
 
Kevin Behr: Integrating Controls and Process Improvement
Kevin Behr: Integrating Controls and Process ImprovementKevin Behr: Integrating Controls and Process Improvement
Kevin Behr: Integrating Controls and Process Improvement
Gene Kim
 
SecureWorld - Communicating With Your CFO
SecureWorld - Communicating With Your CFOSecureWorld - Communicating With Your CFO
SecureWorld - Communicating With Your CFO
Gene Kim
 
PuppetConf2012GeneKim
PuppetConf2012GeneKimPuppetConf2012GeneKim
PuppetConf2012GeneKim
Gene Kim
 
United2012 Rugged DevOps Rocks
United2012 Rugged DevOps RocksUnited2012 Rugged DevOps Rocks
United2012 Rugged DevOps Rocks
Gene Kim
 
Infosec at Ludicrous Speeds - Rugged DevOps
Infosec at Ludicrous Speeds - Rugged DevOps Infosec at Ludicrous Speeds - Rugged DevOps
Infosec at Ludicrous Speeds - Rugged DevOps
Gene Kim
 
When IT Fails The Business Fails...
When IT Fails The Business Fails...When IT Fails The Business Fails...
When IT Fails The Business Fails...
Gene Kim
 
2012 05 corp fin 1c
2012 05 corp fin 1c2012 05 corp fin 1c
2012 05 corp fin 1c
Gene Kim
 
Winnipeg ISACA Security is Dead, Rugged DevOps
Winnipeg ISACA Security is Dead, Rugged DevOpsWinnipeg ISACA Security is Dead, Rugged DevOps
Winnipeg ISACA Security is Dead, Rugged DevOps
Gene Kim
 
SecureWorld: Security is Dead, Rugged DevOps 1f
SecureWorld:  Security is Dead, Rugged DevOps 1fSecureWorld:  Security is Dead, Rugged DevOps 1f
SecureWorld: Security is Dead, Rugged DevOps 1f
Gene Kim
 
Security is Dead. Long Live Rugged DevOps: IT at Ludicrous Speed
Security is Dead. Long Live Rugged DevOps: IT at Ludicrous SpeedSecurity is Dead. Long Live Rugged DevOps: IT at Ludicrous Speed
Security is Dead. Long Live Rugged DevOps: IT at Ludicrous Speed
Gene Kim
 
2011 09 19 LSPE Dev Ops Cookbook 1a
2011 09 19 LSPE Dev Ops Cookbook 1a2011 09 19 LSPE Dev Ops Cookbook 1a
2011 09 19 LSPE Dev Ops Cookbook 1a
Gene Kim
 

More from Gene Kim (14)

Speaker Recording Tips For Virtual DevOps Enterprise (And Why We're Pre-Recor...
Speaker Recording Tips For Virtual DevOps Enterprise (And Why We're Pre-Recor...Speaker Recording Tips For Virtual DevOps Enterprise (And Why We're Pre-Recor...
Speaker Recording Tips For Virtual DevOps Enterprise (And Why We're Pre-Recor...
 
GitHub Universe: 2019: Exemplars, Laggards, and Hoarders A Data-driven Look a...
GitHub Universe: 2019: Exemplars, Laggards, and Hoarders A Data-driven Look a...GitHub Universe: 2019: Exemplars, Laggards, and Hoarders A Data-driven Look a...
GitHub Universe: 2019: Exemplars, Laggards, and Hoarders A Data-driven Look a...
 
The Unicorn Project and The Five Ideals (older: see notes for newer version)
The Unicorn Project and The Five Ideals (older: see notes for newer version)The Unicorn Project and The Five Ideals (older: see notes for newer version)
The Unicorn Project and The Five Ideals (older: see notes for newer version)
 
Kevin Behr: Integrating Controls and Process Improvement
Kevin Behr: Integrating Controls and Process ImprovementKevin Behr: Integrating Controls and Process Improvement
Kevin Behr: Integrating Controls and Process Improvement
 
SecureWorld - Communicating With Your CFO
SecureWorld - Communicating With Your CFOSecureWorld - Communicating With Your CFO
SecureWorld - Communicating With Your CFO
 
PuppetConf2012GeneKim
PuppetConf2012GeneKimPuppetConf2012GeneKim
PuppetConf2012GeneKim
 
United2012 Rugged DevOps Rocks
United2012 Rugged DevOps RocksUnited2012 Rugged DevOps Rocks
United2012 Rugged DevOps Rocks
 
Infosec at Ludicrous Speeds - Rugged DevOps
Infosec at Ludicrous Speeds - Rugged DevOps Infosec at Ludicrous Speeds - Rugged DevOps
Infosec at Ludicrous Speeds - Rugged DevOps
 
When IT Fails The Business Fails...
When IT Fails The Business Fails...When IT Fails The Business Fails...
When IT Fails The Business Fails...
 
2012 05 corp fin 1c
2012 05 corp fin 1c2012 05 corp fin 1c
2012 05 corp fin 1c
 
Winnipeg ISACA Security is Dead, Rugged DevOps
Winnipeg ISACA Security is Dead, Rugged DevOpsWinnipeg ISACA Security is Dead, Rugged DevOps
Winnipeg ISACA Security is Dead, Rugged DevOps
 
SecureWorld: Security is Dead, Rugged DevOps 1f
SecureWorld:  Security is Dead, Rugged DevOps 1fSecureWorld:  Security is Dead, Rugged DevOps 1f
SecureWorld: Security is Dead, Rugged DevOps 1f
 
Security is Dead. Long Live Rugged DevOps: IT at Ludicrous Speed
Security is Dead. Long Live Rugged DevOps: IT at Ludicrous SpeedSecurity is Dead. Long Live Rugged DevOps: IT at Ludicrous Speed
Security is Dead. Long Live Rugged DevOps: IT at Ludicrous Speed
 
2011 09 19 LSPE Dev Ops Cookbook 1a
2011 09 19 LSPE Dev Ops Cookbook 1a2011 09 19 LSPE Dev Ops Cookbook 1a
2011 09 19 LSPE Dev Ops Cookbook 1a
 

Recently uploaded

BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and DisadvantagesBLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
SAI KAILASH R
 
How UiPath Discovery Suite supports identification of Agentic Process Automat...
How UiPath Discovery Suite supports identification of Agentic Process Automat...How UiPath Discovery Suite supports identification of Agentic Process Automat...
How UiPath Discovery Suite supports identification of Agentic Process Automat...
DianaGray10
 
Discovery Series - Zero to Hero - Task Mining Session 1
Discovery Series - Zero to Hero - Task Mining Session 1Discovery Series - Zero to Hero - Task Mining Session 1
Discovery Series - Zero to Hero - Task Mining Session 1
DianaGray10
 
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
FIDO Alliance
 
Retrieval Augmented Generation Evaluation with Ragas
Retrieval Augmented Generation Evaluation with RagasRetrieval Augmented Generation Evaluation with Ragas
Retrieval Augmented Generation Evaluation with Ragas
Zilliz
 
NVIDIA at Breakthrough Discuss for Space Exploration
NVIDIA at Breakthrough Discuss for Space ExplorationNVIDIA at Breakthrough Discuss for Space Exploration
NVIDIA at Breakthrough Discuss for Space Exploration
Alison B. Lowndes
 
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python CodebaseEuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
Jimmy Lai
 
Acumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptxAcumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptx
BrainSell Technologies
 
Improving Learning Content Efficiency with Reusable Learning Content
Improving Learning Content Efficiency with Reusable Learning ContentImproving Learning Content Efficiency with Reusable Learning Content
Improving Learning Content Efficiency with Reusable Learning Content
Enterprise Knowledge
 
Types of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technologyTypes of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technology
ldtexsolbl
 
Generative AI Reasoning Tech Talk - July 2024
Generative AI Reasoning Tech Talk - July 2024Generative AI Reasoning Tech Talk - July 2024
Generative AI Reasoning Tech Talk - July 2024
siddu769252
 
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
alexjohnson7307
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
alexjohnson7307
 
Mule Experience Hub and Release Channel with Java 17
Mule Experience Hub and Release Channel with Java 17Mule Experience Hub and Release Channel with Java 17
Mule Experience Hub and Release Channel with Java 17
Bhajan Mehta
 
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
bellared2
 
LeadMagnet IQ Review: Unlock the Secret to Effortless Traffic and Leads.pdf
LeadMagnet IQ Review:  Unlock the Secret to Effortless Traffic and Leads.pdfLeadMagnet IQ Review:  Unlock the Secret to Effortless Traffic and Leads.pdf
LeadMagnet IQ Review: Unlock the Secret to Effortless Traffic and Leads.pdf
SelfMade bd
 
Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10
ankush9927
 
Google I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged SlidesGoogle I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged Slides
Google Developer Group - Harare
 
Zaitechno Handheld Raman Spectrometer.pdf
Zaitechno Handheld Raman Spectrometer.pdfZaitechno Handheld Raman Spectrometer.pdf
Zaitechno Handheld Raman Spectrometer.pdf
AmandaCheung15
 
Keynote : AI & Future Of Offensive Security
Keynote : AI & Future Of Offensive SecurityKeynote : AI & Future Of Offensive Security
Keynote : AI & Future Of Offensive Security
Priyanka Aash
 

Recently uploaded (20)

BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and DisadvantagesBLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
 
How UiPath Discovery Suite supports identification of Agentic Process Automat...
How UiPath Discovery Suite supports identification of Agentic Process Automat...How UiPath Discovery Suite supports identification of Agentic Process Automat...
How UiPath Discovery Suite supports identification of Agentic Process Automat...
 
Discovery Series - Zero to Hero - Task Mining Session 1
Discovery Series - Zero to Hero - Task Mining Session 1Discovery Series - Zero to Hero - Task Mining Session 1
Discovery Series - Zero to Hero - Task Mining Session 1
 
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
 
Retrieval Augmented Generation Evaluation with Ragas
Retrieval Augmented Generation Evaluation with RagasRetrieval Augmented Generation Evaluation with Ragas
Retrieval Augmented Generation Evaluation with Ragas
 
NVIDIA at Breakthrough Discuss for Space Exploration
NVIDIA at Breakthrough Discuss for Space ExplorationNVIDIA at Breakthrough Discuss for Space Exploration
NVIDIA at Breakthrough Discuss for Space Exploration
 
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python CodebaseEuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
 
Acumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptxAcumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptx
 
Improving Learning Content Efficiency with Reusable Learning Content
Improving Learning Content Efficiency with Reusable Learning ContentImproving Learning Content Efficiency with Reusable Learning Content
Improving Learning Content Efficiency with Reusable Learning Content
 
Types of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technologyTypes of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technology
 
Generative AI Reasoning Tech Talk - July 2024
Generative AI Reasoning Tech Talk - July 2024Generative AI Reasoning Tech Talk - July 2024
Generative AI Reasoning Tech Talk - July 2024
 
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
 
Mule Experience Hub and Release Channel with Java 17
Mule Experience Hub and Release Channel with Java 17Mule Experience Hub and Release Channel with Java 17
Mule Experience Hub and Release Channel with Java 17
 
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
 
LeadMagnet IQ Review: Unlock the Secret to Effortless Traffic and Leads.pdf
LeadMagnet IQ Review:  Unlock the Secret to Effortless Traffic and Leads.pdfLeadMagnet IQ Review:  Unlock the Secret to Effortless Traffic and Leads.pdf
LeadMagnet IQ Review: Unlock the Secret to Effortless Traffic and Leads.pdf
 
Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10
 
Google I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged SlidesGoogle I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged Slides
 
Zaitechno Handheld Raman Spectrometer.pdf
Zaitechno Handheld Raman Spectrometer.pdfZaitechno Handheld Raman Spectrometer.pdf
Zaitechno Handheld Raman Spectrometer.pdf
 
Keynote : AI & Future Of Offensive Security
Keynote : AI & Future Of Offensive SecurityKeynote : AI & Future Of Offensive Security
Keynote : AI & Future Of Offensive Security
 

Keeping The Auditor Away: DevOps Audit Compliance Case Studies

  • 1. @RealGeneKim @jdeluccia Session ID: Gene Kim James DeLuccia Keeping The Auditor Away: DevOps Audit Compliance Case Studies
  • 3. @RealGeneKim @jdeluccia Introductions Gene Kim ▪ Co-author of "The Phoenix Project” ▪ Founder and CTO of Tripwire, Inc. for 13 years ▪ Worked with Jez Humble (co-author of “Continuous Delivery book) to benchmark 14K technology organizations ▪ Co-chaired SOX-404 Scoping Committee at the Institute of Internal Auditors (2005) James DeLuccia ▪ Author, “IT Compliance & Controls” ▪ Ernst & Young, leader for Americas Certification & Compliance Services ▪ Focus: startups, technology, governance, security ▪ Patent holder - crypto privacy comparison system
  • 4. @RealGeneKim @jdeluccia Golly, Why Are You Attending This Talk? ▪ How many people have to deal with compliance? ▪ On a scale of 1-10, how painful are your interactions with auditors? (1=delightful, 10=awful beyond words?)
  • 5. @RealGeneKim @jdeluccia Problem Statement Gene ● DevOps and continuous delivery introduce problems with audit, because the work patterns are so different than traditional SDLC ● Agile also had issues (e.g., testing at end of project, requirements phase at the beginning), but is not as radical as DevOps ○ tens/hundreds of deploys/day (change is risk; can’t rely on change approvals, separation of duty) ● No widespread agreement on what DevOps control requirements should look like James ● Auditors must work off a mature and testable environment ● They must stake their livelihood that what you say is correct, completely ● A partnership is needed between you and them to ensure such an environment exists (of course, it also needs to operate and be amazing .. but that is another talk)
  • 6. @RealGeneKim @jdeluccia Agenda ▪ The Top-Down, Risk Based Audit Process ▪ What Goes Wrong ▪Scoping ▪Control Testing ▪ Scenarios From The DevOps Audit Defense Toolkit ▪ Ask An Auditor Anything!
  • 7. @RealGeneKim @jdeluccia The DevOps Audit Defense Toolkit http://bit.ly/DevOpsAudit James DeLuccia IV Jeff Gallimore Gene Kim Byron Miller
  • 8. @RealGeneKim @jdeluccia What Is Audit ▪ Management is defined as those who are there to achieve the goals of the organizations, which includes the officers of the company (e.g., CEO, CFO, etc.), executives and managers, as well as everyone who reports to them. ▪ Includes some board of directors, GRC departments ▪ Audit is defined to be the function inside the organization that resides outside of management to serve as an independent, objective source of assurance that the organization can achieve its goals. ▪ Includes internal auditors, external auditors (regulators, assessors, etc.)
  • 9. @RealGeneKim @jdeluccia Internal Controls “a process, effected by an organization’s board of directors, management, and other personnel, designed to provide reasonable assurance regarding the achievement of objectives related to operations, reporting, and compliance.” - Operations (effectiveness, efficiency) - Financial Reporting (accuracy of account balances and values) - Compliance (with relevant laws and regulations, contractual obligations: PCI DSS, US Export Law, FEDRAMP, SOC-2) Source: http://coso.org (Committee of Sponsoring Organizations for the Commission on Fraudulent Financial Reporting)
  • 10. @RealGeneKim @jdeluccia How Audit Plans Are Built And Run ▪ Business objectives ▪ Risks ▪ Control objectives ▪ Control procedures Unfortunately, most contact with auditors start with control procedures… It’s totally appropriate to ask to show work and start from beginning...
  • 11. @RealGeneKim @jdeluccia The Audit Cycle ▪ Planning ▪ Gaining an understanding of the organization ▪ Scoping ▪ Sampling, reporting period, types of evidence needed, recipient of report ▪ Schedule ▪ Fieldwork ▪ Controls testing ▪ Substantive testing ▪ Reporting ▪ Management responses ▪ Attestation by auditor and delivered to regulator/clients
  • 13. @RealGeneKim @jdeluccia When Scoping Goes Wrong ▪ 2001: Enron fails ($63B market cap), Arthur Andersen dissolution ▪ 2002: WorldCom (peak $117B market cap) ▪ Leads to Sarbanes-Oxley Act of 2002
  • 16. @RealGeneKim @jdeluccia Analysis: Audit control testing work was scoped properly, linking controls to compliance objectives and risk. Control failures must result potentially undetected material financial reporting errors The Problem: Improperly Scoped Audits
  • 17. @RealGeneKim @jdeluccia Financial Reporting Material Weakness What happens when an audit generates a material weakness?
  • 19. @RealGeneKim @jdeluccia ▪ When we don’t understand why we are being audited ▪ “Why are we doing this audit?” (customers, SOX, regulatory; who is it for?) ▪ When we are asked for something we don’t have (e.g., “evidence of SoD or change approvals) ▪ “What is the control objective? Can we rewrite the control procedure for this asset?” ▪ Do this before the auditor shows up When Auditors Attack Unexpectedly These are delicate conversations, with potentially large impacts on scope, cost, risk...
  • 20. @RealGeneKim @jdeluccia ▪ If we are reacting to these conversations before we’ve done any of our homework, we may be trouble ▪ Extra work (average time to respond to audit is 40 hours; that’s one Dev sprint) ▪ Audit cost and schedule overages: a 3 hour audit test just turned into a 16 hour audit project ▪ Reduced confidence from auditors, increased visibility from audit and management When Auditors Attack Unexpectedly The DevOps Audit Defense Toolkit
  • 21. @RealGeneKim @jdeluccia The DevOps Audit Defense Toolkit http://bit.ly/DevOpsAudit James DeLuccia IV Jeff Gallimore Gene Kim Byron Miller
  • 22. @RealGeneKim @jdeluccia Practice: Enabling A Shared Understanding Source: DevOps Audit Defense Toolkit http://bit.ly/DevOpsAudit
  • 23. @RealGeneKim @jdeluccia Practice: Enabling A Shared Understanding Source: DevOps Audit Defense Toolkit http://bit.ly/DevOpsAudit
  • 24. @RealGeneKim @jdeluccia Practice: Enabling A Shared Understanding Source: DevOps Audit Defense Toolkit http://bit.ly/DevOpsAudit
  • 25. @RealGeneKim @jdeluccia Walk Through Of DevOps Risk And Control Strategies What does an effective DevOps control environment look like?
  • 26. @RealGeneKim @jdeluccia DevOps Orgs Actually Love Process “Facebook values people, tools, and way, way down the list is process.” Jay Parikh VP Infrastructure Engineering, Facebook Not true! They are conflating “process” and “approvals!”
  • 27. @RealGeneKim @jdeluccia High Performing DevOps Orgs Source: 2014 Puppet Labs State Of DevOps 30xmore frequent deployments 8,000xfaster lead times than their peers
  • 28. @RealGeneKim @jdeluccia High Performing DevOps Orgs Source: 2014 Puppet Labs State Of DevOps 2xhigher change success rates 12xfaster mean time to recover (MTTR)
  • 29. @RealGeneKim @jdeluccia High Performing DevOps Orgs Source: 2014 Puppet Labs State Of DevOps more likely to exceed profitability, market share & productivity goals 2x higher market capitalization growth over 3 years* 50%
  • 30. @RealGeneKim @jdeluccia Top Predictors Of Performance ▪ Version control of all production artifacts ▪ Continuous integration and deployment ▪ Automated acceptance testing ▪ Peer-review of production changes (vs. external change approval) ▪ High trust culture ▪ Proactive monitoring of the production environment ▪ Win-win relationship between Dev and Ops
  • 31. @RealGeneKim @jdeluccia DevOps Orgs Need Hardcopy DevOps has higher automation and closer monitoring controls than traditional deployment environments and therefore reduced points for human failure The documentation of ephemeral systems, tools, and deployment processes into a hardcopy breakdown will communicate and simplify this management long term.
  • 32. @RealGeneKim @jdeluccia Practice: Document Risks & Control Strategy Source: DevOps Audit Defense Toolkit http://bit.ly/DevOpsAudit
  • 33. @RealGeneKim @jdeluccia Practice: Document Control Strategy Source: DevOps Audit Defense Toolkit http://bit.ly/DevOpsAudit
  • 34. @RealGeneKim @jdeluccia Practice: Document Control Strategy Source: DevOps Audit Defense Toolkit http://bit.ly/DevOpsAudit
  • 35. @RealGeneKim @jdeluccia Practice: Document Control Strategy Source: DevOps Audit Defense Toolkit http://bit.ly/DevOpsAudit
  • 36. @RealGeneKim @jdeluccia ▪ Gained an understanding of the organization and its objectives ▪ Understood how our service fits in and where we jeopardize those objectives ▪ Designed and documented our control environment so that auditors can share our understanding ▪ Enable auditors to do their work effectively What We Have Done
  • 37. @RealGeneKim @jdeluccia ▪ Save the date: October 21-23, 2014 ▪ DevOps Enterprise is a conference for horses, by horses ▪ Macy’s, Disney, GE Capital, Blackboard, Telstra, US Citizen and Immigration Services, CSG, Raytheon, Ticketmaster/LiveNation, Capital One, Nordstrom, Union Bank of California ▪ Leaders driving DevOps transformations will talk about ▪ The business problem they set out to solve ▪ The obstacles they had to overcome ▪ The business value they created ▪ Submit talks at: http://devopsenterprisesummit.com/ DevOps Enterprise Summit
  • 38. @RealGeneKim @jdeluccia ▪ We don’t need to wait for auditors to learn about DevOps -- by learning about audit, we can successfully bridge the gap ▪ DevOps control environments can be even more secure than traditional control environments ▪ The DevOps Audit Defense Toolkit might be able to help you! http://bit.ly/DevOpsAudit ▪ We’d love your scrutiny and case studies! ▪ DevOps Enterprise Summit: http://devopsenterprise.io ▪ Emailing us: genek@realgenekim.me, jdeluccia@gmail.com Conclusion
  • 39. @RealGeneKim @jdeluccia Ask An Auditor Anything! ▪ Ask the Auditor and the audience anything: ▪ Separation of Duties? ▪ Security beyond checkboxes and non-contextual requirements? ▪ Governance effects of DevOps and/or Agile? ▪ Integration and dialogues and timing with Management, Auditors, and the effect? ▪ Ask Gene on practical examples ▪ Questions for the audience: ▪ Are you using ISO 27034 as a reference architecture?
  • 41. @RealGeneKim @jdeluccia Results Of Halving Deployment Interval And customers got the feature in half the time! Source: Scott Prugh, CSG
  • 42. @RealGeneKim @jdeluccia Results Of Halving Deployment Interval Source: Scott Prugh, CSG
  • 43. @RealGeneKim @jdeluccia Call to Action ● We're looking for case studies ○ Rough life lessons and smooth successes ○ Submit to: ■ DevOps Audit Defense Toolkit: Google+ Community: http://bit.ly/DevOpsAudit ● Look at the DevOps Audit Defense Toolkit ● DevOps Enterprise Summit ○ http://devopsenterprise.io/