"Are we really getting better?" by John Belbute

1. Are we really getting
better?
A critical look at metrics in the software business

2. I hope you come away with
• Some new ideas about metrics
• Know what outcome and diagnostic metrics are
• Perspective about what metrics drive progress
• Hopefully some ideas about how you can
measure more effectively

3. If you can’t measure
something the only reason
to believe you are getting
better is foolish optimism
If you can’t measure
something the only reason
to believe you are getting
better is optimism

4. Think about the
manufacturing business
• We know exactly what a
widget looks like
• All widgets are identical
• Defective widgets are easy to
find
• We know how long it takes to
build a widget
• If we buy another widget
maker we can double our
output
Metrics in manufacturing are easy

5. Now compare that to the
business of software
• We know exactly what a
widget looks like
• All widgets are identical
• Defective widgets are easy
to find
• We know how long it takes to
build a widget
• If we buy another widget
maker we can double our
output

6. Now compare that to the
business of software
• We know exactly what a
widget looks like
• All widgets are identical
• Defective widgets are easy
to find
• We know how long it takes to
build a widget
• If we buy another widget
maker we can double our
output
• We don’t know what our
software will look like

7. Now compare that to the
business of software
• We know exactly what a
widget looks like
• All widgets are identical
• Defective widgets are easy
to find
• We know how long it takes to
build a widget
• If we buy another widget
maker we can double our
output
• We don’t know what our
software will look like
• No 2 classes are identical

8. Now compare that to the
business of software
• We know exactly what a
widget looks like
• All widgets are identical
• Defective widgets are easy
to find
• We know how long it takes to
build a widget
• If we buy another widget
maker we can double our
output
• We don’t know what our
software will look like
• No 2 classes are identical
• Defects are very difficult to
find

9. Now compare that to the
business of software
• We know exactly what a
widget looks like
• All widgets are identical
• Defective widgets are easy
to find
• We know how long it takes to
build a widget
• If we buy another widget
maker we can double our
output
• We don’t know what our
software will look like
• No 2 classes are identical
• Defects are very difficult to
find
• We are very poor at
estimating

10. Now compare that to the
business of software
• We know exactly what a
widget looks like
• All widgets are identical
• Defective widgets are easy
to find
• We know how long it takes to
build a widget
• If we buy another widget
maker we can double our
output
Metrics in software are really hard
• We don’t know what our
software will look like
• No 2 classes are identical
• Defects are very difficult to
find
• We are very poor at
estimating
• Doubling your staff size
probably does not double
your output

11. PD Metrics are definitely hard
but also abundant
• Collected automatically
• Easy to report on
• Easy to export
• Find their way into spreadsheets
• Can lead to incorrect conclusions

12. Metrics are either outcome
focused or diagnostic focused
• Outcome Metrics
• Goals
• Objective
• Project sponsors care about them
• Diagnostic Metrics
• Effort
• Project sponsors could care less

13. It is always a mistake to use a
diagnostic metric as a goal
• Coincidental correlation to outcomes
• Disempowers team
• Goal setter owns responsibility
Diagnostic metrics have a use
they are just bad goals

14. Project Sponsors care about
3 outcomes
1. Are we developing the right things ?
2. Are we developing quickly ?
3. Are we developing with high quality ?
I am going to focus on velocity and quality
Product owners
address this

15. Velocity and quality each
need outcome metrics and
always need to be looked at
together

16. I always ask my teams three
questions
1. Are you better now than you were?
2. Are you going to be better than you are?
3. How do you know?
How do you know leads to our outcome metric

17. We have never found a good
way to measure velocity
• We have tried some interesting things
• Lines of code
• Feature or function points
• Story points
These are arbitrary and meaningless!

18. We typically use defects to
measure quality
• That is problematic
• Hard to define what a defect is
• Some defects never get reported
• Not all defects are equal
• There is a very long feedback loop

19. Diagnostic metrics often
become quality goals
• # of Unit tests
• Unit Test Coverage
• % of code reviewed
• # of UI tests
• # of defects contained
None of these should be objectives
Diagnostic metrics often
become BAD quality goals

20. So are we getting better?
• We want to think we are
• We don’t have the evidence to prove it
• None of the common metrics tell us
• Until we measure the right things we won’t know
Don’t give up! There is one thing we can do

21. Measure Waste
Waste is any time engineers
spend on non-productive
activities, or doing nothing
because they are blocked

22. Teams and organizations
create and can reduce waste
• Team Waste
• Defect rework
• Broken builds
• Organization Waste
• Not investing
• Turnover
• Meetings
It is not uncommon to find that waste
exceeds 50% of team capacity

23. So my answer to the question
“Are we getting better?” is
• We don’t know
• We won’t know until we start measuring waste
• When we start measuring we can start improving
Thanks for Listening
Happy to take questions

24. If you have any questions or want
to discuss this in more detail
email: john.belbute@gmail.com
Twitter: @jbelbute
Linkedin: John Belbute