XBOSoft runs through the Top 10 Agile Metrics revealing the most fundamental data points Agile methodology requires to work effectively, and will put you on the highly targeted path to successful implementation of your Agile processes. XBOSoft and Go2Group run through the top data points you should be measuring in your Agile Workflow. We’ll show you what to track, when and how often, and most importantly – why. Many believe that metrics are useless, but unless you measure, how can you systematically improve or know how you are doing? And with velocity as an overarching objective in agile, you should be tracking other things so that you know what else you could be impacting by going faster. But, with all the metrics so readily available to us today, how do we filter through to the most meaningful?

1. Agile
Metrics
XBOSoft, Inc. All Rights Reserved. 1

2. XBOSoft
Founded in 2006
Dedicated to software quality
Software QA Consulting
Software Testing
Offices in San Francisco, Beijing and Amsterdam
XBOSoft, Inc. All Rights Reserved. 2

3. House Rules
• Everyone except the speakers is muted
• Questions via gotowebinar control panel OR
via Twitter @XBOSoft using
#Top10AgileMetrics
• Questions can be asked throughout the
webinar, and will be answered if possible
• You’ll receive information for recording after
the webinar
XBOSoft, Inc. All Rights Reserved. 3

4. Meet Our Speakers
• CEO and Founder of XBOSoft
• Relevant specialties and passions
– Software quality process,
evaluation, measurement and
improvement
– Mobile User Experience and
usability
– Cycling and travel
Philip Lew
• Go2Group Vice President
• ALM Expert
• Over 30 years with over 100
client installations of ALM
toolsets in a variety of
development methodologies
Doug Bass
XBOSoft, Inc. All Rights Reserved. 4
Sabrina Gasson
• Marketing Manager of
XBOSoft
• Emails you all regularly
to join our industry hot
topic webinars
• And invites you all to
download our latest
trends in software testing
whitepapers.

5. WHAT IS ONE OF THE
MOST FAMOUS THINGS
HE SAID?
“LIFE IS LIKE A BOX OF
CHOCOLATES, YOU NEVER KNOW
WHAT YOU’RE GOING TO GET”
XBOSoft, Inc. All Rights Reserved. 5

6. Agile Metrics
By using metrics, we can know what
we are going to get!

7. Agenda
1. Agile, Frameworks and GQM
2. Applying GQM to Agile
3. Top 10 Agile Metrics
XBOSoft, Inc. All Rights Reserved. 7

8. Poll
XBOSoft, Inc. All Rights Reserved. 8

9. What We Can Agree On
• Want to minimize mistakes and doing things
over (rework)
• Want to utilize resources effectively
• Want to predict when things could go wrong
• Want to do what we say we will do, when we
say - timelines
• Fix defects early – later is more expensive
• Production defects are bad news
XBOSoft, Inc. All Rights Reserved. 10

10. What We Can Agree On About
Metrics
• Want easy to get/capture
• Should have a valuable purpose toward
improvement
– Evaluate
– Guide
– Insightful
– Actionable
XBOSoft, Inc. All Rights Reserved. 11

11. Agile – How Can We Improve?
• Some say no metrics needed
• Is velocity all you track?
• What is sacrificed for velocity?
• How can we know what and how we are doing
or where to improve?
• How do we know if we’re using resources
effectively?
• How do we know we are spending our
effort/time in the right place?
XBOSoft, Inc. All Rights Reserved. 12

12. Poll
XBOSoft, Inc. All Rights Reserved. 13

13. Using Frameworks to
Organizing Our Thoughts
• Putting metrics into a framework helps us
to categorize and organize
• Helps us to make sure we have all
functions in our organization and process
covered
• Helps us to develop relationships between
different elements in our process

14. Applying Basili’s GQM
• Many software metrics programs have failed because they
had poorly defined, or even non-existent objectives.
Goal
Metric Question

15. Stakeholders and GQM
Goal 1 Goal 2 Goal 3
Question
1
Question
2
Question
3
Question
4
Question
5
Metric
1
Metric
2
Metric
3
Metric
4
Metric
5
Metric
6
Metric
7
CEO CTO Dev Mgr QA Manager

16. GQM - Example
• Goals
– Reduce total cost of development
– Reduce total cost of testing-effort
– Reduce the number of new feature bugs
• Questions
– Which areas have the highest re-work rates?
– Which functional areas have the most defects?
– How long does it take to repair defects?
– How complete are my requirements?
– How much time do I spend testing?

17. Metric Q&A Worksheet
Category Question Metric Tool to Use Needed
Customer
Are we meeting
Schedule
Satisfaction
schedule?
Variance
Project
Management
Tool
None
Software
Testing
Efficiency
Software
Quality
What kinds of
defects are most
common?
Usability Are end users
getting lost,
understand our UI
Software Test
Effectiveness
?

18. Agile Considerations
Waterfall
• Speed
• Quality
• Cost
Agile
• Speed
• Quality
• Cost
Where’s the beef?
XBOSoft, Inc. All Rights Reserved. 20

19. Agile Manifesto
RETURN TO BASICS

20. Excerpts from the Agile Manifesto
1. Working software over comprehensive documentation
2. Our highest priority is to satisfy the customer through early and
continuous delivery of valuable software.
3. Deliver working software frequently, from a couple of weeks to a
couple of months, with a preference to the shorter timescale.
4. Business people and developers must work together daily
throughout the project.
5. Working software is the primary measure of progress.
6. Agile processes promote sustainable development. The sponsors,
developers, and users should be able to maintain a constant pace
indefinitely.
7. Continuous attention to technical excellence and good design
enhances agility.

21. Agile – Understanding Where We Came From?
XBOSoft, Inc. All Rights Reserved. 23

22. Poll
XBOSoft, Inc. All Rights Reserved. 24

23. What are the Goals of Agile?
Speed Quality
Customer-User
Satisfaction

24. Matching Agile Characteristics and
Objectives
Agile
Characteristic
Speed Quality Customer and
End User
Satisfaction
Other
Collaborative x x
Working
x
Software
Adapt to
changing
requirements
x x
From the agile
manifesto

25. What Questions Tell Me If I’m
Reaching Those Objectives?
Speed Quality
Working
Product
Requirements
Owner
Participation
Technical debt
Schedule
Planned versus
unplanned

26. What Questions Tell Me If I’m
Reaching Those Objectives?
Speed
Requirements
Did I get them in time?
Do I understand them?
Did we deliver working product?
Schedule
Am I on schedule?
Can I be more productive?
How much wasted time? Rework?
Unplanned
versus planned
Did I estimate well?
Did I finish what I said I would?
Do I make mistakes that cause more
(unplanned) work?

27. What Questions Tell Me If I’m
Reaching Those Objectives?
Quality
Product
owner
Did my end users find defects?
Did product owner give feedback
regularly?
Are there less defects than before?
Requirements
Are my requirements delivered when
needed?
Did the delivered product match the
requirement? Or reqt. wrong?
Do engineers have trouble
understanding the requirements?
Technical
Debt
Is my software performance getting
worse and worse?
Am I prepared if my lead engineer
leaves?
Do I mess things up when fixing things?

28. ARE YOU READY FOR THE
COUNTDOWN?
METRICS THAT HELP US ANSWER
THESE QUESTIONS!

29. 10.
VELOCITY
Am I on schedule?
Did I estimate well?
XBOSoft, Inc. All Rights Reserved. 32

30. Velocity
• Tracking our velocity over time tells us our progress
and capacity and helps us forecast based on our
capabilities (now and in the future).
• Measurement – Sum of all the approved estimates
of the stories
• One point in time measurement has little meaning,
so need to track over time or wrt another metric
• Metric – Velocity (T)-(T-1)/Velocity (T)

31. Velocity
Velocity Variance
XBOSoft, Inc. All Rights Reserved. 34
30%
25%
20%
15%
10%
5%
0%
-5%
-10%
-15%
-20%
-25%
1 2 3 4 5 6 7 8 9

32. Burn Down – (Distance over Time)
XBOSoft, Inc. All Rights Reserved. 35

33. 9.
DEFECT REMOVAL EFFECTIVENESS
Did my end users find defects?
Are there less defects than before?
XBOSoft, Inc. All Rights Reserved. 36

34. Defect Removal Effectiveness
• We don’t want the end user to find defects
• Any defect found by the user influences
quality
• Metric – Defects in found in production during
(90 day) period post release (DFP)/DFP +
Defects found Pre-release

35. What Should Your Goal Be For DRE
• Estimated (by Capers Jones) that defect
removal effectiveness differs by different
levels of process capability maturity levels:
– Level 1: 85%
– Level 2: 89%
– Level 3: 91%
– Level 4: 93%
– Level 5: 95%
XBOSoft, Inc. All Rights Reserved. 38

36. DRE Distribution
C
A
XBOSoft, Inc. All Rights Reserved. 39

37. 8.
ADDED WORK
Am I on schedule?
Am I overcommitting?
Is my rate of work sustainable?
XBOSoft, Inc. All Rights Reserved. 40

38. Added Work
• This will help us see if we are making
promises that maybe we should not –
overcommitting and unsustainable.
• Measurement – Sum of work (new stories
added) over and above the original plan.
• Metric – New stories added/Original Stories
Planned

39. Piling On
XBOSoft, Inc. All Rights Reserved. 42
35
30
25
20
15
10
5
0
1 2 3 4 5 6 7 8 9
Revised Planned
Original Plan

40. 7.
WORK CATEGORY ALLOCATION
Am I allocating resources optimally?
Do I have trouble understanding requirements?
XBOSoft, Inc. All Rights Reserved. 43

41. Work Category Allocation
• This will help us see where we are spending our time
and adjust our priorities.
• Measurement – Sum of work (for each category)
Metric – Divide by total work to get % allocation
• Categories could include:
– Technical debt – refactor - redo
– Regression testing
– New feature development
– Research
– Requirements review
– Defect fixing
– Other

42. Is Effort in Alignment?
XBOSoft, Inc. All Rights Reserved. 45
Are we spending
enough time
showing our
stakeholders
what we’ve
done?
30%
25%
20%
15%
10%
5%
0%
Are we fixing
things too much
or making lots of
mistakes?

43. 6.
WORK SUSTAINABILITY
Do I have the right people on my team?
Did I estimate work well?
Can I attain quality sustainably?
XBOSoft, Inc. All Rights Reserved. 46

44. Work Sustainability
• “Sustainable” - We can’t sustain our effort if we are
continuously putting in overtime
• Overtime is necessary, but needs to be managed and
causes mistakes
• Mistakes manifest themselves in many ways, such as
technical debt or defects
• Overtime often results from poor work estimates and
directly proportional to technical debt
• Metric – Unplanned Overtime Work/Total Work

45. Unplanned Overtime %
XBOSoft, Inc. All Rights Reserved. 48
16%
14%
12%
10%
8%
6%
4%
2%
0%
1 2 3 4 5 6 7 8 9
Sprint
What caused this?

46. Unplanned Overtime and Defects
XBOSoft, Inc. All Rights Reserved. 49
7
6
5
4
3
2
1
0
1 2 3 4 5 6 7
Defects Found
Unplanned OT

47. 5.
DEFECT REWORK RATE
Is my technical debt accumulating?
Am I wasting time?
XBOSoft, Inc. All Rights Reserved. 50

48. Defect Rework Rate
• We can’t get faster if we are always fixing
things.
• Depends on valid data whereby people are
logging real hours to tasks they take on.
• Metric - Fix Effort Ratio : Time to fix
defects/Total effort expended (hours)

49. Defect ReWork %
XBOSoft, Inc. All Rights Reserved. 52
35%
30%
25%
20%
15%
10%
5%
0%
1 2 3 4 5 6 7 8 9

50. Accumulated Debt
XBOSoft, Inc. All Rights Reserved. 53
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
1 2 3 4 5 6 7 8 9
Defect Rework%
Unfixed %

51. 4.
TOTAL UNEXPECTED WORK
What is throwing me off schedule?
Am I biting off too much?
XBOSoft, Inc. All Rights Reserved. 54

52. Total Unexpected Work
• Measurement – Sum of work done above the
original estimates for those stories in the plan.
• Are we underestimating difficulty?
• Many reasons:
– Incomplete user stories
– Misunderstood user stories
– Bigger and more complex than I thought?
– Not enough time for research
– Too hard…
• Metric – (Total work done – Planned
work)/Planned work

53. Are We Estimating Well?
XBOSoft, Inc. All Rights Reserved. 56
60
50
40
30
20
10
0
1 2 3 4 5 6 7 8 9
Planned
Unplanned
Per Person/Week During the Sprint

54. What are we doing that is
unplanned?
Research
Requirements
Coding
Testing
Other
20%
XBOSoft, Inc. All Rights Reserved. 57
29%
9%
19%
23%

55. 3.
DEFECTS IN PRODUCTION AVG. FIX
TIME
Are my customers/end users happy?
Is the architecture prohibiting me?
XBOSoft, Inc. All Rights Reserved. 58

56. Defect in Production Avg. Fix Time
• If defects make it to production, that’s bad !
• But what is worse if an important defect stays
unfixed.
• Defects that take a long time to fix represent either
technical debt or poor process
• Metric – Time to fix Defects (released to production-usually
P1 defects found) / Total P1 Defects (post
production)

57. Don’t Make Them Wait
Hey, we’re improving! WooHoo!
XBOSoft, Inc. All Rights Reserved. 60
60
50
40
30
20
10
0
1 2 3 4 5 6 7 8 9
#P1 Defects Found
Average Fix Time (Hrs)

58. 2.
DEFECT CATEGORIZATION
ALLOCATION %
Why are things breaking?
Can I get faster?
XBOSoft, Inc. All Rights Reserved. 61

59. Defect Categorization Allocation
• Reduce rework = Increase speed
• Discover when defects are injected
• Reduce and prevent defects from happening
in the first place.
• Drive down the Fix Effort Ratio leading to
better quality and resulting in higher velocity.
• Examine defect rates per category

60. Defect Rework Allocation
Requirements
Coding
Environment
Testing
38%
XBOSoft, Inc. All Rights Reserved. 63
29%
21%
13%
What’s going on here?

61. Rework-Requirements Errors
Requirements
• Change in requirements
• Don’t understand requirements
• Incomplete requirement
• Requirement does not exist
• ? ‘oh, I meant this…’

62. Poll
XBOSoft, Inc. All Rights Reserved. 65

63. 1.
WORKING SOFTWARE DELIVERY
RATE
Am I delivering what I said I would?
Am I getting feedback from the PO?
Will I meet the release date?
XBOSoft, Inc. All Rights Reserved. 67

64. Working Software Delivery Rate
• Does the software work or not?
• Working does not mean perfect
• Working means can show in order to get feedback
• Metric - # story points delivered that ‘work’ / # story
points that were to be shown to the PO but could
not be shown to work
– Many variations

65. Am I Getting Better?
XBOSoft, Inc. All Rights Reserved. 69
25
20
15
10
5
0
1 2 3 4 5 6 7 8 9
Planning Working Visible
Actual Working

66. Am I Delivering So I can Get
Feedback?
% Working
XBOSoft, Inc. All Rights Reserved. 70
100%
80%
60%
40%
20%
0%
1 2 3 4 5 6 7 8 9

67. Be
Agile Is a Way of Thinking
“Be-Do-Have”
People
• Communicative
• Collaborative/Cooperative
• Flexible and willing
• Knowledgeable-multi
• Initiative/responsible
• Responsive/adaptive
Do
Have
Process
• Iterative (sprints)
• Daily standups
• Face to face communication
• Post mortem – end of sprint
• Delivery meeting – end of sprint
• Planning meeting – before sprint
• Self organizing
Results
• Speed
• Quality
XBOSoft, Inc. All Rights Reserved. 71

68. TAKE AWAYS-SUMMARY
• What and
why
• Goals and
objective
s
Understand
Ask
yourself
questions
Evaluate
Measure
to answer
those
questions
and take
action
Improve

69. Q+A
Post your questions on Twitter and we'll answer them @XBOSoft
Join us to keep updated on all our webinars, reports and whitepapers:
facebook.com/xbosoft
+xbosoft
linkedin.com/company/xbosoft
We post regularly on our blog – check us out!
http://xbosoft.com/software-quality-blog/
Why not download our free Whitepapers, available here:
http://xbosoft.com/knowledge-center/
You will receive an email with information on slides and recording. Any
further queries regarding our services or ideas for future webinars please
email us! services@xbosoft.com
www.xbosoft.com
Our latest whitepaper
on Agile Metrics will
be released next week
so check back soon!