This document summarizes a presentation on helping auditors understand agile development processes. The presentation discusses how traditional audit models are not well-aligned with agile practices. It then outlines a 5-step process to create an agile auditable framework: 1) validate risks and controls, 2) inventory agile practices, 3) create parameters for practices, 4) determine control methods, and 5) establish operational parameters. The framework aims to better align controls and testing with agile ceremonies while reducing paperwork by 50% or more. Challenges include resource intensity and subjectivity, while benefits include improved quality and alignment.

1. AT12
Agile Development Concurrent Session
11/13/2014 1:30 PM
"Dealing with Auditors: Helping
Them Understand Agile"
Presented by:
Steve Nunziata
Independent Consultant
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Steve Nunziata (CSM, PMP, ACP, SAFe SPC) has more than
twenty-five years in IT project management, using waterfall and agile
methodologies—and numerous hybrids in between. Steve’s industry
experience ranges from health care, sporting goods, transportation,
and insurance. For the past ten years, he has focused on agile
practices and teams, fulfilling roles such as ScrumMaster, Product
Owner, agile coach, project manager, and quality assurance
advisor―sometimes in the same day! Steve is very active in the San
Antonio agile community, facilitating monthly meet-ups and
education events. In his spare time, he enjoys playing in his classic
rock band and being with his wonderful family.

3. Dealing with Auditors:
Helping them Understand Agile
CHAOS, CONSISTENCY,
CREATIVITY: A JOURNEY
THROUGH AGILE AUDITABILITY
Steve Nunziata, PMP, PMI-ACP, CSM, SAFe SPC
November 13th, 2014

4. About Steve…
PMP, ACP, CSM, SAFe SPC
EDS, Nike, Adidas, USAA
Agile Trainer & Coach
New Jersey / Oregon
Bassist Extraordinaire
Alamo Agilistas / PMI

5. So… Why Are We Here?
Opportunity:
Educate internal auditors to evolve away
from formal artifacts and accept Agile
tenets of visibility and transparency to
demonstrate adherence to defined
Quality standards.
We will collaborate on an approach to
define an Agile Risk & Control framework
that can start you on your journey.

6. How Would You Like:
 A 50% - or more – reduction
in project ‘paperwork’ to
demonstrate adherence to
compliance processes?
WATERFALL AGILE
59
30
PROJECT
COMPLIANCE
ARTIFACTS
 A framework for
consistent application of
Agile practices and
ceremonies across a large
– and growing –
organization?

7. Background: My Story
Zero to Sixty (Days): Chaos to
Consistency

8. Agenda
Chaos
Failings of
Today’s Risk
Management
Processes
Consistency
Why Audit
Execution
Models Need
to Evolve
Creativity
Creating an
Agile
Auditable
Framework

9. Managing Risk – How Important is it?
 The primary goal of a
business is to… stay in
business.
 It is therefore necessary to continually evaluate,
monitor, and address threats to retain market share.
Otherwise, what would happen?

10. Managing Risk – The Risk Management Process
Risk
Identification
Risk
Assessment
Risk
Response
Risk Review

11. Managing Risk – ISO 9001 Summary
Part 4 – The Company must establish, document, and maintain a
Quality Management System (QMS)
Part 5 – Management commitment in evidence for the QMS
Part 6 – Necessary resources must be determined & provisioned
Part 7 – Plan & Develop processes for product realization. The
processes must produce documents that can be (1) reviewed
for acceptance; and (2) used as proof of conformance
Part 8 – All reports of non-conformances, both of the product or
the process, shall be reported upon, analyzed and lead to
corrective action

12. Managing Risk – Risk & Control
Compliance Framework
Risk
Controls
Control
Tests
Reporting &
Review
Operational
Risks
 Incomplete Requirements
 Ineffective or Incomplete
Software Solution
 Poor User Experience
 Poor Project Execution
Plan
 Formal Requirements
Baseline Process
 Project Execution
Schedule Review
 Code Peer Reviews
 Evidence of
Formal Signoffs
 Published
Meeting Minutes
 Documented
Decisions / Logs
 Formal results of
Audit published for
review; opportunities
for improvements
noted
Auditors

13. Are Risk Management Processes
Inherently anti-Agile?
Source: http://www.devballs.com/wp-content/uploads/2010/02/agilemanifesto.gif

14. SDLC & Process Audit
Execution Models: Challenges
While Agile adoption and evolution has continued unabated
over the past several years, traditional process audits have
largely been unable to keep pace. Why might this be?

15. SDLC & Process Audit Execution Models
Req’s Analysis Design Build Test Deploy
Systems Development Life Cycle – Linear View

16. SDLC & Process Audit Execution Models
Source: http://julianeverett.wordpress.com/
Red Dotted Line: Waterfall
Blue Dotted Line: Agile
RISK
TIMEProject Risk Profile – Agile & Waterfall

17. SDLC & Process Audit Execution Models
Daily
24Hours
Iteration
2-4 Weeks
Release
~3 Months
Closure
~9-12 Months
SDLC Execution – Waterfall, Incremental, & Agile

18. SDLC & Process Audit Execution Models
Process Audit vs. SDLC Execution Gap Analysis
Closure
~9-12 Months
Release
~3 Months
Iteration
2-4 Weeks
Daily
24Hours

19. SDLC & Process Audit Execution Models
Daily
Iteration
2-4 Weeks
Release
~3 Months
Closure
SDLC and Process Audit Execution: Optimal Quality State

20. 5 Steps to Establishing an Agile
Auditable Framework
Risk Validation
Inventory Agile Practices
Create Acceptable Parameters
Determine Method of Control
Establish Operational Parameters
1
2
3
4
5

21. 5 Steps to Evolving an Agile Auditable Framework
Risk Validation
Review and Validate the current Risk & Control Framework,
ensuring traceability from Risks to Controls to Control Tests.
Operational Risk: Risk Control: Control Test:
Failure to Manage
Project Risks
Risk Management
Process
Evidence of a Periodic
Risk Review (Risk Log)
Issue Management
Process
Formal, Complete Issues
Log
1

22. 5 Steps to Evolving an Agile Auditable Framework
Inventory Agile Practices
 Inventory the Agile Practices supported by the organization.
Scrum practices and ceremonies provide a good start.
 Match the Agile ceremonies to the list of Risks in the current
Risk & Control Framework. Can a Ceremony or Practice provide
an acceptable substitute? How / Why?
2

23. 5 Steps to Evolving an Agile Auditable Framework
Inventory Agile Practices
 Introduce the Agile Practice as a Control. Could it work? Could
it be effective? What would be the value of the current control
set – should anything remain, or can they be dismissed?
Operational Risk: Risk Control: Control Test:
Failure to Manage
Project Risks
Risk Management
Process
Evidence of a Periodic
Risk Review
Agile Daily
Standup
2

24. 5 Steps to Evolving an Agile Auditable Framework
Create Acceptable Parameters
 Research Industry standard ‘best practices’ for the ceremonies
or practices you plan on using as a Control (mitigation strategy)
for the Risk. A great example is Version One’s The Agile Checklist
 Create a matrix defining minimally acceptable behaviors, along
with anti-patterns, and radiate the desired outcomes in a
common area
3

25. 5 Steps to Evolving an Agile Auditable Framework
Create Acceptable Parameters
Agile Ceremony: Daily Standup
Best Practice Acceptable Partial Unacceptable
Occurs 5 Days per
Week
Occurs 4 Days per
Week
Occurs 3 Days per
Week
Occurs <3 Days per
Week
3 Core Questions
Addressed
3 Core Questions
Addressed
<3 Core Questions
Addressed
<3 Core Questions
Addressed
…Your
Organization?
…Your
Organization?
…Your
Organization?
….Your
Organization?
3

26. 5 Steps to Evolving an Agile Auditable Framework
Determine Method of Control
 Does the new Control Test require someone observe an Agile
Ceremony, or is there a consistent formal artifact from an Agile
practice that can be viewed?
4

27. 5 Steps to Evolving an Agile Auditable Framework
Establish Operational Parameters
 Review the total number of Control Tests. How
many require observation from an Auditor?
 Establish the Audit cycle & reporting time
(Weekly? Sprint Level? Release Level? Other..?)
 Train and deploy Audit resources
 Execute an Audit cycle… and report to Risk Owners
 Learn… and continue to evolve!
5

28. 5 Steps to Evolving… Creativity
 Host a Retrospective Ceremony with some of the
Agile teams to uncover:
 What may be challenging teams in conforming to
minimal standards?
 What opportunities can they recommend to
evolve to controls?
 Are the audits providing value in holding roles
accountable for their deliverables?
 Finally – when minimal standards are easily
achieved – it’s time to take the next steps in
maturity, and shift the pattern.

29. 5 Steps to Evolving - Going Beyond...
 Challenge: can you evolve traditional, formal artifacts into a
more Agile framework? How can you continuously improve?
Picture Source: http://agile101.wordpress.com/2009/07/27/
agile-risk-management-assessing-risks-step-2-of-4/

30. Positive Outcomes
 Better alignment of Controls and Tests to the project execution model
 Real time, actionable feedback & reporting to teams and Risk owners
 Scalable for future methodologies & practices
 Continual quality assessments; a project can have multiple reviews
 Sets a benchmark for Agile maturity across an Organization
 ‘Humanizes’ the Audit (not ‘check the box’) – gives teams a voice
 Experience – 50% reduction in Controls… while doubling Quality
 Leading – NOT lagging – metric; address problems before they manifest
 Opportunity for two-way communication and learnings

31. Challenges
 Optimal model is labor intensive
 Inherent subjectivity in assessments (‘Auditor Bias’)
 Potential for teams to feel ‘over controlled’
 Oversight and administration of the process
 Communication and support for changes
 Determining boundaries of adherence vs. non-adherence,
and appropriate remedies
 Ever-evolving process; can feel like an ‘arms race’

32. Common Questions
 Does this model Scale?
 How much time per week would this require?
 Isn’t this just the Scrum Master’s… or (insert role here) –
job?
 Could we use Pair Programming as a Control?
 What is the future of Agile Quality Assurance?

33. Objectives Met?
Source: http://www.devballs.com/wp-content/uploads/2010/02/agilemanifesto.gif

34. Remember: Auditors are the
Board of Health!

35. Questions?

36. Thank You!