This document summarizes Steve Nunziata's presentation on establishing an agile auditable framework. It discusses the challenges that traditional audit models pose for agile development and presents a 5-step approach to evolving audit processes: 1) validate risks and controls, 2) inventory agile practices, 3) create parameters for practices, 4) determine audit methods, and 5) establish operational parameters. The goal is to better align audits with agile processes through practices like daily stand-ups in order to reduce paperwork by 50% while improving quality. Both benefits and challenges of the approach are reviewed.

1. Chaos, Consistency,
Creativity:
A Journey Through
Agile Auditability
Steve Nunziata, PMP, PMI-ACP, CSM, SAFe SPC
Agile Austin Monthly Meeting, October 14th, 2014

2. About Steve…
PMP, ACP, CSM, SAFe SPC
EDS, Nike, Adidas, USAA
Agile Trainer & Coach
New Jersey / Oregon
Bassist Extraordinaire
Alamo Agilistas / PMI

3. Background: My Story
Zero to Sixty (Days): Chaos to
Consistency

4. So… Why Are We Here?
Opportunity:
Educate internal auditors to evolve away
from formal artifacts and accept Agile
tenets of visibility and transparency to
demonstrate adherence to defined
Quality standards.
We will collaborate on an approach to
define an Agile Risk & Control framework
that can start you on your journey.

5. How Would You Like:
 A 50% - or more – reduction
in project ‘paperwork’ to
demonstrate adherence to
compliance processes?
59
COMPLIANCE
ARTIFACTS
30
PROJECT
WATERFALL AGILE
 A framework for
consistent application of
Agile practices and
ceremonies across a large
– and growing –
organization?

6. Remember…Use the Force
Remove,
you
must,
Stories
from the
Backlog,
That,
within an
Iteration,
completed,
will not
be…

7. Agenda
Chaos
Failings of
Today’s Risk
Management
Processes
Consistency
Why Audit
Execution
Models Need
to Evolve
Creativity
Creating an
Agile
Auditable
Framework

8. Managing Risk – How Important is it?
 The primary goal of a
business is to… stay in
business.
 It is therefore necessary to continually evaluate,
monitor, and address threats to retain market share.
Otherwise, what would happen?

9. Managing Risk – The Risk Management Process
Risk
Identification
Risk
Assessment
Risk
Response
Risk Review

10. Managing Risk – ISO 9001 Summary
Part 4 – The Company must establish, document, and maintain a
Quality Management System (QMS)
Part 5 – Management commitment in evidence for the QMS
Part 6 – Necessary resources must be determined & provisioned
Part 7 – Plan & Develop processes for product realization. The
processes must produce documents that can be (1) reviewed
for acceptance; and (2) used as proof of conformance
Part 8 – All reports of non-conformances, both of the product or
the process, shall be reported upon, analyzed and lead to
corrective action

11. Managing Risk – Risk & Control
Compliance Framework
Risk
Controls
Control
Tests
Operational
Risks
 Incomplete Requirements
 Ineffective or Incomplete
Reporting &
Review
Software Solution
 Poor User Experience
 Poor Project Execution
Plan
 Formal Requirements
Baseline Process
 Project Execution
Schedule Review
 Code Peer Reviews
 Evidence of
Formal Signoffs
 Published
Meeting Minutes
 Documented
Decisions / Logs
 Formal results of
Audit published for
review; opportunities
for improvements
noted
Auditors

12. Are Risk Management Processes
Inherently anti-Agile?
Source: http://www.devballs.com/wp-content/uploads/2010/02/agilemanifesto.gif

13. SDLC & Process Audit
Execution Models: Challenges
While Agile adoption and evolution has continued unabated
over the past several years, traditional process audits have
largely been unable to keep pace. Why might this be?

14. SDLC & Process Audit Execution Models
Systems Development Life Cycle – Linear View
Req’s Analysis Design Build Test Deploy

15. SDLC & Process Audit Execution Models
Source: http://julianeverett.wordpress.com/
Blue Dotted Line: Agile
Red Dotted Line: Waterfall
RISK
Project Risk Profile – Agile & Waterfall TIME

16. SDLC & Process Audit Execution Models
SDLC Execution – Waterfall, Incremental, & Agile
Daily
24 Hours
Iteration
2-4 Weeks
Release
~3 Months
Closure
~9-12 Months

17. SDLC & Process Audit Execution Models
Process Audit vs. SDLC Execution Gap Analysis
Closure
~9-12 Months
Release
~3 Months
Iteration
2-4 Weeks
Daily
24 Hours

18. SDLC & Process Audit Execution Models
SDLC and Process Audit Execution: Optimal Quality State
Daily
Iteration
2-4 Weeks
Release
~3 Months
Closure

19. 5 Steps to Establishing an Agile
Auditable Framework
Risk Validation
Inventory Agile Practices
Create Acceptable Parameters
Determine Method of Control
Establish Operational Parameters
1
2
3
4
5

20. 5 Steps to Evolving an Agile Auditable Framework
Risk Validation
Review and Validate the current Risk & Control Framework,
ensuring traceability from Risks to Controls to Control Tests.
Operational Risk: Risk Control: Control Test:
Failure to Manage
Project Risks
Risk Management
Process
Evidence of a Periodic
Risk Review (Risk Log)
Issue Management
Process
Formal, Complete Issues
Log
1

21. 5 Steps to Evolving an Agile Auditable Framework
Inventory Agile Practices
2
 Inventory the Agile Practices supported by the organization.
Scrum practices and ceremonies provide a good start.
 Match the Agile ceremonies to the list of Risks in the current
Risk & Control Framework. Can a Ceremony or Practice provide
an acceptable substitute? How / Why?

22. 5 Steps to Evolving an Agile Auditable Framework
Inventory Agile Practices
 Introduce the Agile Practice as a Control. Could it work? Could
it be effective? What would be the value of the current control
set – should anything remain, or can they be dismissed?
Operational Risk: Risk Control: Control Test:
Failure to Manage
Project Risks
Risk Management
Process
Evidence of a Periodic
Risk Review
Agile Daily
Standup
2

23. 5 Steps to Evolving an Agile Auditable Framework
Create Acceptable Parameters
3
 Research Industry standard ‘best practices’ for the ceremonies
or practices you plan on using as a Control (mitigation strategy)
for the Risk. A great example is Version One’s The Agile Checklist
 Create a matrix defining minimally acceptable behaviors, along
with anti-patterns, and radiate the desired outcomes in a
common area

24. 5 Steps to Evolving an Agile Auditable Framework
Create Acceptable Parameters
Agile Ceremony: Daily Standup
Best Practice Acceptable Partial Unacceptable
Occurs 5 Days per
Week
Occurs 4 Days per
Week
Occurs 3 Days per
Week
Occurs <3 Days per
Week
3 Core Questions
Addressed
3 Core Questions
Addressed
<3 Core Questions
Addressed
<3 Core Questions
Addressed
…Your
Organization?
…Your
Organization?
…Your
Organization?
….Your
Organization?
3

25. 5 Steps to Evolving an Agile Auditable Framework
Determine Method of Control
4
 Does the new Control Test require someone observe an Agile
Ceremony, or is there a consistent formal artifact from an Agile
practice that can be viewed?

26. 5 Steps to Evolving an Agile Auditable Framework
Establish Operational Parameters
5
 Review the total number of Control Tests. How
many require observation from an Auditor?
 Establish the Audit cycle & reporting time
(Weekly? Sprint Level? Release Level? Other..?)
 Train and deploy Audit resources
 Execute an Audit cycle… and report to Risk Owners
 Learn… and continue to evolve!

27. 5 Steps to Evolving… Creativity
 Host a Retrospective Ceremony with some of the
Agile teams to uncover:
 What may be challenging teams in conforming to
minimal standards?
 What opportunities can they recommend to
evolve to controls?
 Are the audits providing value in holding roles
accountable for their deliverables?
 Finally – when minimal standards are easily
achieved – it’s time to take the next steps in
maturity, and shift the pattern.

28. 5 Steps to Evolving - Going Beyond...
 Challenge: can you evolve traditional, formal artifacts into a
more Agile framework? How can you continuously improve?
Picture Source: http://agile101.wordpress.com/2009/07/27/
agile-risk-management-assessing-risks-step-2-of-4/

29. Positive Outcomes
 Better alignment of Controls and Tests to the project execution model
 Real time, actionable feedback & reporting to teams and Risk owners
 Scalable for future methodologies & practices
 Continual quality assessments; a project can have multiple reviews
 Sets a benchmark for Agile maturity across an Organization
 ‘Humanizes’ the Audit (not ‘check the box’) – gives teams a voice
 Experience – 50% reduction in Controls… while doubling Quality
 Leading – NOT lagging – metric; address problems before they manifest
 Opportunity for two-way communication and learnings

30. Challenges
 Optimal model is labor intensive
 Inherent subjectivity in assessments (‘Auditor Bias’)
 Potential for teams to feel ‘over controlled’
 Oversight and administration of the process
 Communication and support for changes
 Determining boundaries of adherence vs. non-adherence,
and appropriate remedies
 Ever-evolving process; can feel like an ‘arms race’

31. Common Questions
 Does this model Scale?
 How much time per week would this require?
 Isn’t this just the Scrum Master’s… or (insert role here) –
job?
 Could we use Pair Programming as a Control?
 What is the future of Agile Quality Assurance?

32. Objectives Met?
Source: http://www.devballs.com/wp-content/uploads/2010/02/agilemanifesto.gif

33. Remember: Auditors are the
Board of Health!

34. Questions?

35. Thank You!

36. Information Sources
Malik Imran Ullah & Waqar Ali Zaidi, “Quality
Assurance Activities in Agile – Philosophy to
Practice”. Sep. 2009.
Larry Whittington, “ISO9001:2008 Requirements
Summary in Plain English”.
http://www.whittingtonassociates.com/
Tor Stalhane, Geir Kjetil Hanssen, “The application
of ISO 9001 to Agile Software Development”. 2008.
Buck Kulkami, “Agile Projects: An Emerging
Challenge for IT Auditors”.

37. Information Sources
R. Gopinath, “Guideline: How to Audit and Agile
Project?”
George Schlitz, “Is your Agile Audit and Compliance
Process really Agile?”
Christelle Scharff, “Guiding Global Software
Development Projects using Scrum and Agile with
Quality Assurance”