Lessons learned from measuring software development processes

Software Quality Lab
Markus Unterauer
Consultant and trainer
Lessons Learned from
measuring software development
processes

www.software-quality-lab.com | improve your quality
At the end of this session you know …
Agenda
 Introduction to process measurement
 Experience from practise
 Lessons learned from the project
Slide 2

The Why? and first ideas on the How?
Introduction to process measurement
Slide 3

Measurement as basis for all improvements
Why measure processes
Slide 4
Define goal and
path to reach it
Discover, where we are
Identify gap, to where
we should be
Find corrective measures
Go next step Goal reached 

Proper metrics show the way to our goal
Slide 5
Source:DchicviaFlickr,https://www.flickr.com/photos/agenciamodels/6118171675

Boundaries show, when to act
Slide 6
Upper boundary
Lower boundary
Target value
Measured value
Time

Define proper metrics with G-Q-M
GQM for defining metrics
1. Goal
 What should be done for whom?
 Which aspects do we want to change? What is the context?
e.g. „Improve the quality for all new projects“
2. Question
 Which questions guide us to the measurement goal?
e.g. „Are we finding the right bugs?“
3. Metric
 How can we answer the questions using metrics?
e.g. „Ratio of bugs found per severity class“
Slide 7
Define goals
Find
questions
Answer
questions

Example
GQM for defining metrics
Slide 8
Metric
Question
Goal We want to improve the
quality of our products!
Are we finding
the right bugs?
Ratio of
bugs per severity
No. of bugs
found after release
Do we find the bugs
instead of the users?
Ratio of
bugs found by QA
vs. Users

Question: Are we finding the right bugs?
Goal: Improve quality of our products
 Ratio of bugs per severity class
 Number of bugs found after release
 Per severity class
 Ratio of recurring bugs
 Average number of bugs found per test case
 Number of bugs found by test level
 Unit tests
 Integration tests
 System tests
 User acceptance tests
 Bugs found by type
 Coding bug
 Bad requirements
 UI bug
 …
Slide 9

So much for the theory, let‘s turn to practise …
Experience from practise
Slide 10

Situation at our customer
Slide 11
http://pixabay.com/en/desert-road-sand-forward-winds-609316/

Goals for our customer
 How bad is it really?
 What should we do?
 Set up an improvement cycle
Slide 12
http://en.wikipedia.org/wiki/PDCA#/media/File:PDCA_Cycle.svg

First attempt …
 Our customer wanted to measure …
 ~120 metrics
 in 12 process fields with 21 processes
 on 4 levels
 from 14 different sources and source system
 But …
 for more than half of all metrics we could not get values
 no one really knew, what to do with the metrics
 management did not even look at it
 effort for getting numbers was extremly high
Slide 13

Some sources of values
Slide 14
• Different levels of detail
• Errors and missing data
• Bookings on different categories
Time recording
• Each project with different structure
• Not comparable
• Incomplete
Project
planning
• Only for parts of all sourcecode
• Difficult to query and analyze
Source
Control
• Manually built Excel sheets by Project Managers
• Each project with different structure
• Not comparable
Controlling
sheets
• Very high effort
• Ambiguous questions
Questionaires
and interviews
Good at first glance,
but bad in detail …

Everything very difficult
Slide 15
http://pixabay.com/en/squirrel-reading-books-surprise-304021/

New approach GQM
1. Define two goals
 Ask management, what they would like to achieve
 Define reporting
2. Ask questions
3. Find metrics to answer question
 Only automatically measurable ones
 Only one per question
4. Define target values and boundaries
 Get target values from industry benchmarks
5. Try out and validate
6. Report to management
Slide 16
http://pixabay.com/en/shield-directory-panic-fear-trust-492992/

Step 1: Define Goals
1. We want to improve quality to the customer
2. We want to increase developer productivity
Slide 17
Images from www.pixabay.com

Step 2: Ask questions
 Do we find bugs instead of the customer?
 Are we testing enough?
 Are we testing effectively?
 Do our developers spend too much time on bug fixing?
 Are we developing fast enough?
 Is the quality of our source code high enough?
Slide 18

Step 3: Find metrics
 Do we find bugs instead of the customer?
• Ratio of bugs found by us to bugs found by customer
 Are we testing enough?
• Number of test cases tested per developer day
 Are we testing effectively?
• Number of bugs found per tester day
 Do our developers spend too much time on bug fixing?
• Effort-ratio for bug fixing per developer month
 Are we developing fast enough?
• Number of changed LoC per developer day
 Is the quality of our source code high enough?
• Number of bugs found by customer per developer month
Slide 19

Step 4: Define target values and boundaries
Slide 20
Metric Target Value Current value Conclusion
Changed LoC per
developer day
30 – 40 LoC per day 123 LoC per Day Too high. Results in
poor quality.
Effort-ratio for bug
fixing
25 - 35% 115% Far too high. Low
productivity.
Ratio of bugs found
by customer
1 – 3% of all bugs 68% of all bugs Far too high. Testing
not effective.
… … … …
Target range taken
from literature and
publications

Step 5: Try out and validate
Slide 21
http://pixabay.com/en/calculator-calculation-insurance-385506/

Step 6: Reporting
Slide 22
Trend for past thee
months
Textual comment
from Quality
Manager

Document metrics in a metric handbook
Slide 23
Metric Effort-ratio of bug fixing per developer month
Goal Developers should produce high quality code and spend as much time
for implementing new features as possible
Benefit Fast development of new features
High quality
Formula Ratiobug fixing = Effortbug fixing / Total Effortdevelopment
Method Automatic calculation from timesheets
Datasource Time recording system
Interval Monthly
Responsible Process Manager

From practise
Lessons Learned
Slide 24

Tips for starting your quality metric system
Lessons Learned
 Define metrics top down, not bottom up
 Goal > Question > Metric
 Not: „Hey, look how many things we can measure!“
 Concentrate on 5 - 10 metrics, e.g. the one presented before
 Define lower and upper boundary together with your team
 … or measure without boundaries for some iterations and define boundaries then
 Start with automatically measurable metrics
 When defining metrics, always have in mind, if you have all data necessary in the quality needed
 Define in advance, what to do with the metrics
Slide 25
Image:http://pixabay.com/en/bulb-light-lamp-electric-160207/

Questions?
Lessons Learned from measuring software development processes
Slide 26

Your Partner in Software Quality and Testing
Software Quality Lab GmbH
www.software-quality-lab.com
Consulting | Service | Academy | Tool Expertise
Markus Unterauer
markus.unterauer@software-quality-lab.com

Lessons learned from measuring software development processes

More Related Content

What's hot

Similar to Lessons learned from measuring software development processes

More from Markus Unterauer

Recently uploaded

Lessons learned from measuring software development processes

Editor's Notes