This document discusses measuring success in Agile environments. It begins with the presenter introducing herself and her experience. She then asks the audience about their current measurement practices and Agile experiences. Traditional metrics like budget, scope, productivity, and process are compared to Agile metrics like value, quality, capability, and adaptability/collaboration. Examples of how to measure these Agile metrics are provided, such as tracking customer feedback and rework time. Caveats about measurement are also discussed, such as the observer effect and only measuring what is easy rather than what is important.

1. © 2016 Comtech Services, Inc.© 2016 Comtech Services, Inc.
Adapting Traditional
Metrics to Measure,
Monitor, and Achieve Agile
Success
Dawn Stevens
Vice President
Comtech Services
From Waterfall to Agile:

2. © 2016 Comtech Services, Inc. 2
Comtech and CIDM
About me
• Over 30 years providing
consulting services to
information development
organizations in all industries
around the world
• 27 years in information
development
o 11 years managing a
combined documentation
and training organization
o 16 years at Comtech
Myself

3. © 2016 Comtech Services, Inc. 3
Audience participation:"
About you
• How many are currently in
an Agile environment?
• How many years have you
been in an Agile
environment?
• How many are considering
adopting Agile?

4. © 2016 Comtech Services, Inc. 4
Audience participation: "
Current measurement practices
• Do you gather any
metrics today?
• What are you
measuring?
• Why are you
measuring them?
• What do you do with
them?

5. © 2016 Comtech Services, Inc. 5
Why measure?
• To judge the performance of your team
and the individuals on the team
X

6. © 2016 Comtech Services, Inc. 6
Why measure?
• To demonstrate that promises were met
• To validate or correct assumptions
• To alert the team to irregularities
• To track progress toward a goal

7. © 2016 Comtech Services, Inc. 7
Audience participation:"
Agile expectations
• What promises have been
made about your Agile
implementation?
• What assumptions do you
have about what Agile brings?
• What might go wrong with an
Agile implementation?
• How will you know when you
have reached Agile success?

8. © 2016 Comtech Services, Inc. 8
CIDM survey:"
The promises of Agile
12.2
24.4
31.7
31.7
43.9
46.3
63.4
73.2
0 10 20 30 40 50 60 70 80
Mo-vate employees
Decrease cost of informa-on development
Increase customer sa-sfac-on
Increase technical competence
Improve quality
Receive more customer feedback
Decrease development -me
Increase collabora-on

9. © 2016 Comtech Services, Inc. 9
The Agile difference
That is, while there is value on the items on the right, we value the items on the left more.
Responding to change
Customer collaboration
Working software
Individuals and interactions
over following a plan
over contract negotiation
over comprehensive documentationover processes and tools
We value:

10. © 2016 Comtech Services, Inc. 10
Audience participation: "
The Agile difference
Process/performance vs. Result

11. © 2016 Comtech Services, Inc. 11
Traditional
metrics
The Agile difference
• Budget
• Scope
• Productivity
• Process
• Value
• Quality
• Capability
• Adaptability and
collaboration
Agile
metrics

12. © 2016 Comtech Services, Inc. 12
Budget
The Agile difference
• How much can we / did
we spend?
• How many resources can
be assigned / do we
need?
• What tools can we
acquire to complete this
project?
• Is it needed?
• Is it important to the
customer?
• What will customers pay
for it?
• What affect will it have on
company reputation?
• What difference does it
make on our market
share?
Value

13. © 2016 Comtech Services, Inc. 13
Audience participation: "
Measuring value
What metrics might
indicate the value of
your:
• Agile
implementation?
• product(s)?
• department?

14. © 2016 Comtech Services, Inc. 14
My thoughts: "
Measuring value
Agile implementation
• Number of projects
cancelled before
launch
• Number of launched
products without
expected customer
acceptance

15. © 2016 Comtech Services, Inc. 15
Product
My thoughts: "
Measuring value
• Actual sales and usage
statistics
o Revenue
o # of customers
o Market share
• Growth rate
• Net promoter scores
• % time spent on customer
priorities
• Given $100, how much
would customers assign to:
o product itself
o documentation
o training
o customer support
• Reduction in service calls over
time
• Amount of content never
accessed
Department

16. © 2016 Comtech Services, Inc. 16
Scope
The Agile difference
• What is the size of what
we are creating?
• What specific features
must it include?
• How complex is it?
• Does it meet the need?
• Are customers satisfied
with it?
• Is it usable?
• Are there any defects?
Quality

17. © 2016 Comtech Services, Inc. 17
Audience participation: "
Measuring quality
• What measurements
provide information about
the quality of your work?
• How do your customers
measure quality?
• How do your managers
measure quality?

18. © 2016 Comtech Services, Inc. 18
My thoughts: "
Measuring quality
• % of topics with positive customer feedback
• Time required for users to locate needed information
• # of support calls on topics not covered in content
• # of errors found / reported post release
• Amount of time spent on rework after initial development
effort

19. © 2016 Comtech Services, Inc. 19
Productivity
The Agile difference
• How many man hours will
it / did it take us to
complete?
• How much calendar time
time will it / did it take?
• How much can the team
accomplish within the time
constraints?
• Does the team have the
required skills?
• Is the team able to sustain
the production rate?
Capability

20. © 2016 Comtech Services, Inc. 20
Audience participation: "
Measuring team capability
What measurements provide information about the
capabilities of your team?

21. © 2016 Comtech Services, Inc. 21
My thoughts: "
Measuring team capability
• Team velocity -- # of storypoints / sprint
• On-time delivery rate
• Experience of team
working together
• % work requiring
specialized skill

22. © 2016 Comtech Services, Inc. 22
Beware: "
Velocity ≠ Productivity
Which team is
more
productive?

23. © 2016 Comtech Services, Inc. 23
Beware: "
The observer effect
• People adjust behavior based on the metrics they are held against
• What you measure is often not only what you get, but all you get
• To change behavior, change what you measure

24. © 2016 Comtech Services, Inc. 24
Process
The Agile difference
• How will we
get this done?
• Did we check off all the
boxes?
• What do the rules say?
• How can we get this done?
• What should we do next?
• What needs to change?
• How can we work together
better?
Adaptability / Collaboration

25. © 2016 Comtech Services, Inc. 25
Audience participation: "
Measuring Adaptability / Collaboration
How can you measure
team adaptability and
collaboration?

26. © 2016 Comtech Services, Inc. 26
My thoughts: "
Measuring adaptability / collaboration
• # of changes required during sprint
• Time required to adjust to changes
• # of changes / innovations suggested by
team members
• % team members expressing
satisfaction with communication during
sprint
• % team members expressing that they
are respected on the team
• Time required for team members to get
answers or help

27. © 2016 Comtech Services, Inc. 27
Beware:"
The streetlight effect
What you can
measure
What
you can’t
measure
What you
should
measure
Ask yourself:
• What decisions will be made based
on this measure?
• Is this the RIGHT thing to measure
or an easy thing to measure?
• What might be an unintended
consequence of this measure?
• How can we guard against
unintended consequences?

28. © 2016 Comtech Services, Inc. 28
A final caution:"
Interpreting your measurements
Don’t collect metrics if you aren’t going to use them or believe
them

29. © 2016 Comtech Services, Inc. 29
Questions?
If we didn’t measure things we
wouldn’t know how good we were at
measuring the things that we’re
measuring.
Dawn Stevens
dawn.stevens@comtech-serv.com