This presentation covers details on how we can measure that Agile Transformation is providing the intended outcome or not. I presents a research & survey which tries to understand how different people measure value of Agile Transformation

1. How to measure the outcome
of Agile transformation?
Rahul Sudame
CSP, SAFe SPC, PMI-ACP

2. Agenda
 Lets play some game!
 We need to have clear goals
 Transformation Roadmap
 Agile Readiness Assessment Model
 Current State Assessment Model
 ‘Agile’ Metrics
 Agile maturity Model
 Survey on ROI for Agile
 Case Study

3. Lets brainstorm!

4. Agile Innovation Games for Chapter Strategies
Is Agile delivering value to you?
 Innovation Game: Teams
to discuss if Agile is
delivering value to you?
 Are you measuring the
outcome of Agile
Transformation in your
organization?
 Is Agile implementation
providing the desired
impact?

5. Defining goal of going Agile

6. Moving to Agile is not easy!
 Everyone wants to go Agile!
 But Agile does not come for
free, it needs:
◦ Breaking the bones
(waterfall structure) of the
organization
◦ Changing mindset
◦ Investment &
Commitment
 So, how to ensure returns of
this huge investment?
23
35
41
46
52
0 20 40 60
No Mgmt support
Non-willingness
Alignment
Difficult to Transition
No Clear Metrics
Response % for challenges in
achieving its goals with Scrum
Ref: The 2015 State of Scrum Report - Scrum Alliance

7. We need to have clear goals
 Clear identification of
Customer & Business benefits
 Envisioning the goal, based
on type of the organization
 Identification of investment
needs
 Clear identification of ROI
parameters
Business Case of Agile Transformation
Business
Case
Customer
Benefits
Business
Benefits
Project
Benefits
Investment
Needs
ROI
Parameters

8. Building Transformation
Roadmap
 Agile Transformation roadmap
in alignment with
organizational environment
 Pilot to try Agile methodology
and for the groundwork
 Inspect & Adapt at all the
milestones
 Build the right environment
iteratively
Agile Transformation Roadmap

9. Agile Assessment Models

10. Agile Readiness Assessment
Model
 Are we ready for Agile?
 Identification of Agile
readiness using objective
assessment
 Team’s interest / enthusiasm
to be factored in
 Best suited teams to initiate
pilot
 Gap analysis can highlight
organization’s preparedness
Agile Readiness Questionnaire
Question Guidelines Points
Team size • 5 – Team size of 7 to 9
3 – Team size less than 7
2 – Team with 10 to 15
members
• 1 – More than 15
members
2
Team
Composition
• 5 - Completely cross
functional
3 - Cross functional with
some dependency on
external teams
0 - Not Cross functional
2
Business
Value Stream
Coverage
• 5 - Team covers the entire
business value stream
1- Less than 50% of the
value stream
0 - Less than 10% of the
business value stream
1

11. Focus on values, not just
practices
 Coach the team to
implement Agile ‘principles’
and not just the ‘practices’
 Build an environment of
continuous improvement
 Don’t get trapped in ‘Trough
of disillusionment’
 Deliver value periodically
Sprint
Cycle
Define
Roles
Backlog
Grooming
Agile
Estimation
Release
Planning
Sprint
Rituals
Activities in Sprint Cycles

12. Agile Current State Assessment
 Conduct periodic assessment of
team’s Agile practices with
involvement of the entire team
 Assess the effectiveness of
different aspects such as
Release Planning, Sprint
Planning, Product Ownership
 Identify the Gaps and define a
plan of bridging those gaps
Current State Assessment Model

13. ‘Agile’ Metrics
 Identify the metrics that
provide ‘real’ status of
progress
 Define a mechanism to
aggregate required data
periodically
 Do NOT enforce waterfall
metrics on Agile teams 
 Don’t get trapped in ‘Perfect’
burn down chart 
• Earned Business Value / ROI
• Release Cycle
• Cost of Delay (COD)
• Lead / Cycle Time
Business Metrics
• Velocity / Burn down
• Backlog Size
• User Stories Done per Iteration
• User Stories carried forward to next iteration
Process Metrics
• Defects count per story / Defect density
• Acceptance tests per story
• Code Quality & code coverage metrics
• Test Automation metrics
Quality Metrics

14. Adaptability meets Predictability
 Consistent
implementation of Agile
and usage of tools can
provide release
predictability
 Dynamic data
extrapolation can help in
analyzing impact of
scope changes
 Real-time visibility to all
the stakeholders
Release Predictability Chart

15. Agile Maturity Model
 Maturity level of a
team can be
assessed based on
different parameters
 Organizational
benchmarks can be
provided as
guidelines
 Agile Maturity
Model itself should
be lean & aligned
with Agile
philosophy
Level
1
Level
2
Level
3
Level
4
Level
5
Process Group Rating Applicability Score
Maturity
Index
Release Mgmt
Effectiveness
6 7 86%
1.79
Sprint Mgmt Effectiveness 7.5 9 83%
Product Backlog Mgmt 7 7 100%
Frequent Product Delivery 7 8 88%
0.88Quality Assurance
Effectiveness
5.5 6 92%
Retro & Improvements 1 3 33% 0.33
Customer focus 5 5 100% 1.00
Maturity Index 4.12
Agile Maturity Model

16. Reduce Lead Time
 Optimizing Lead time & Cycle
Time would help in reducing
‘waste’ in the system
 ‘Cost of Delay’ identification
would help in delivery of higher
value items sooner
 Focus on business value would
help in reducing delays at all
levels
Deployment
1 month delay for deployment
QA
3 weeks delay for signoff
Development
2 weeks delay for integration
Design
8 days delay for signoff
Requirements
2 weeks delay for clarifications
Delays at various phases

17. Best Agile Team Award
 Healthy competition amongst
the teams can be
encouraged
 Effectiveness of teams on
various parameters related
with Agile implementations
can be considered
 Team’s approach, Scrum
Master / PO’s effectiveness
can also be factored in
Team Retro
Effectiveness
Scrum Master
Effectiveness
Points
Team 1 12 24 73
Team 2 18 27 65
Team 3 14 11 62
Team 4 8 14 62
Team 5 10 13 61
Team 6 9 11 46
Assessment for Best Team Award

18. Are we making progress?
 Agile Project Progress can
be reviewed periodically
 Quantitative as well as
Qualitative measures to be
presented
 Organization-wide changes
across different value-
streams to be reviewed
Project Progress Report
Quantitative
Benefits
• Delivery cycle reduction
• Revenue realization
($$$)
Qualitative
Benefits
• Value delivered to
customer
• Team efficiency
• Customer/Team
satisfaction
Benefits Not
Realized
• Value delivered to
customer
Project Progress Status Repot

19. Survey on Outcome of Agile

20. Survey on Outcome of Agile
Independent Survey Findings – Q2 2015
Highlights parameters used for different people to measure outcome of
Agile
 In Progress:
◦ Details on survey participants
◦ Details on organization, industries, demographics represented
◦ Survey methodology

21. Survey on Outcome of Agile
Independent Survey Findings – Q2 2015
Highlights parameters used for different people to measure outcome of
Agile

22. Survey Findings
 Delivery cycle time - 22%
 Delivering business value - 19%
 Working Software / Customer Satisfaction – 14%
 Not measuring ROI – 11%
 Reduction in defects, waste & risks – 8%
Independent Survey Findings – Q2 2015
Highlights parameters used for different people to measure outcome of
Agile

23. Survey Findings: Parameters
 Parameters for measuring outcome of Agile:
◦ Delivery Cycle Time / WIP Reduction
◦ Achievement of business goals/value
◦ Working Software / Customer Satisfaction
◦ Reduction in defects, waste & risks
◦ Improvement in Business/Engineering processes
◦ On time delivery
◦ Delivery within the budget
◦ Break-even point - bang for your buck
◦ Team satisfaction
Independent Survey Findings – Q2 2015
Highlights parameters used for different people to measure outcome of
Agile

24. Survey Findings: Feedback
Independent Survey Findings – Q2 2015
Highlights parameters used for different people to measure outcome of
Start from what you
expect... what value
did you expect the
transformation to
bring?
If I could choose
just one measure I'd
choose Cycle Time
- time (in days)
Lean up front, big picture
planning, determine
business goals, and,
especially, how those goals
interrelate.
Its not worth to measure
the ROI of Agile.
Reducing WiP can have
up to a 4x impact on
Quality, half your Time-
to-Market, and improve
your Predictability

25. Case Study

26. Case Study
 Product development organization applied Agile enterprise-wide for
product development cycles
 Outcome:
◦ Release Cycle reduced from 1.5 years to 3 months
◦ First release was not appreciated by customers due to low user
experience – Agile helped in Failing Faster!
◦ Manual regression testing efforts (for sanity suite): 5 days
◦ Automated regression testing efforts (for sanity suite): 4 hours
 90% QA effort reduction
◦ Conversion of a short-term customer contract into a long-term
engagement
◦ Time taken for matured environment - 1 year

27. Summary
 Agile Transformation is a journey and involves continuous improvement
 For effectively presenting ROI, build a baseline and set of
parameters/metrics for before and after Agile state
◦ E.g. Pre & Post Agile Delivery Cycles / Durations
◦ Achievement of business / customer goals
 Identify and present both quantitative and qualitative benefits to the
stakeholders periodically
◦ E.g. Customer Satisfaction
◦ Reduction in production hot fixes, change requests
 Retrospect Agile Maturity at all levels of product development and focus
on building effectiveness iteratively
 E.g. Target: Agile maturity Level 3 or Project Maturity Index > 3.5

28. Thank You !
- Rahul Sudame
rahulsudame@yahoo.com