2. What is Software Measurement?
“Quantified observations” about aspect of
software (product,process or project)
Introduction - Process Metrics - Project Metrics – Product Metrics
3. What is Software Measurement?
“It’s easy to get numbers,
what is hard is to know they are right
and understand what they mean.”
William C. Hetzel:
Introduction - Process Metrics - Project Metrics – Product Metrics
4. What is Software Measurement?
“To measure is to know”
“If you cannot measure it, you cannot improve it.”
“The more you understand what is wrong with a figure, the more valuable that figures become.”
Introduction - Process Metrics - Project Metrics – Product Metrics
6. What makes a good measure?
Simple
Objective
Easily collected
Robust
Valid
Introduction - Process Metrics - Project Metrics – Product Metrics
7. What can Measures do for you?
Facilitate estimation
Identify risk areas
Measure testing status
Measure/Predict product quality
Measure test effectiveness
Identify training opportunities
Identify process improvement opportunities
Provide “meters to flag actions”
Introduction - Process Metrics - Project Metrics – Product Metrics
8. 2 sides of Measurement
“… the information will help
me to understand what is
going on and do a better job.”
“… the information may
be used against me.”
Introduction - Process Metrics - Project Metrics – Product Metrics
9. The Human Element
Measure process and products instead of people if possible
Beware of the dark side of the Hawthorne Effect
Introduction - Process Metrics - Project Metrics – Product Metrics
10. Process for defining good netrics
1. Objectives
2. Questions
3. Metrics
4. Goals
5. Improvements
Introduction - Process Metrics - Project Metrics – Product Metrics
11. Process Metrics
MEASURE PROCESS CAPABILITIES
MEASURE THE OVERALL SOFTWARE PROCESS
Introduction - Process Metrics - Project Metrics – Product Metrics
12. Process Metrics
Objective: Finding bugs
Question: What percentage of the bugs present in a system during testing is found by the
testing process?
Metric: Defect Detection Effectiveness (DDE) or Defect Detection Percentage (DDP)
DDE =
𝑑𝑒𝑓𝑒𝑐𝑡𝑠 𝑑𝑒𝑡𝑒𝑐𝑡𝑒𝑑
𝑑𝑒𝑓𝑒𝑐𝑡𝑠 𝑑𝑒𝑡𝑒𝑐𝑡𝑒𝑑 + 𝑝𝑟𝑜𝑑𝑢𝑐𝑡𝑖𝑜𝑛 𝑑𝑒𝑓𝑒𝑐𝑡𝑠
Introduction - Process Metrics - Project Metrics – Product Metrics
13. Process Metrics
Objective: Finding important bugs
Question: What percentage of the bugs present in a system during testing is found by the
testing process?
Metric: Defect Detection Effectiveness (DDE) or Defect Detection Percentage (DDP)
DDE (important bugs) > DDE (all bugs)
Introduction - Process Metrics - Project Metrics – Product Metrics
14. Process Metrics
Objective: Finding bugs
Question: Are there any external factors affecting testing process?
Metric: Bug opened count - the number of times each bug report is opened
Introduction - Process Metrics - Project Metrics – Product Metrics
16. Project Metrics
WHERE WE STAND ON THE PROJECT AND HOW
CLOSE WE ARE TO ACHIEVING PROJECT OBJECTIVES
Introduction - Process Metrics - Project Metrics – Product Metrics
17. Project Metrics
Objective: finding most of the bugs and fixing the most important bugs
Question: Where we are (our progress) in terms of finding and fixing bugs?
Metrics: Total Daily Opened - the cumulative number of bug reports opened as of a given
day on the project
Total Daily Closed - the cumulative number of bug reports resolved (either
confirmed as fixed or deferred for this project) as of a given
day on the project
Average Daily Opened - the average daily number of bug reports opened for a
given week
Average Daily Closed - the average daily number of bug reports resolved for a
given week
Average Daily Backlog - the average daily difference between the total daily
opened and total daily closed bug reports
Introduction - Process Metrics - Project Metrics – Product Metrics
19. Product Metrics
THE CURRENT QUALITY STATUS OF THE SYSTEM
UNDER TESTING
Introduction - Process Metrics - Project Metrics – Product Metrics
20. Product Metrics
95% of the tests have been run
90% of the tests have passed
5% of the tests have failed
4% of the tests are ready to run
1% of the tests are blocked
we are right on schedule in terms of the test execution
schedule
Does this tell us good news or bad news?
Introduction - Process Metrics - Project Metrics – Product Metrics
21. Product Metrics
Introduction - Process Metrics - Project Metrics – Product Metrics
Objective: Risk-based testing
Question: How effectively are we reducing quality risk overall?
Metric: traceability between test results/defects and risk items
23. KISS – Keep It Simple Sir
More is not always better
All metrics are not forever
Automate collection when possible
Introduction - Process Metrics - Project Metrics – Product Metrics
Define objectives for the process.
Consider questions about the effectiveness, efficiency, and elegance with which the process realizes those objectives.
Devise measurable metrics, either direct or surrogate, for each effectiveness, efficiency, and elegance question.
Determine realistic goals for each metric, based on an understanding of industry best practices and current process capability.
As appropriate, implement process improvements that increase effectiveness, efficiency, or elegance as measured by the metrics.
- number of defects found during testing
- effectiveness of defect removal during development
- effectiveness of defect detection
- efforts required for testing
Project metrics are the most commonly used test metrics. project metrics can tell us whether the testing is on target to achieve the plan.
Project metrics provide practical, often immediately-useful insights about your current project status and trends. By benchmarking your project metrics against past projects and industry norms, you can see where your project stands in terms of current status and trends. This can give you good ideas about what kinds of project course-corrections to make to achieve optimal outcomes for your project. When multiple, balanced metrics are used properly, project metrics can present a complete picture of the project that can help you avoid making project decisions that will result in sub-optimal outcomes.
- resources
- cost and schedule
Unfortunately, people often over-rely on project metrics, and often fail to properly balance project metrics. While project metrics tell you the progress made toward project objectives, they might not give a full and balanced picture, especially in terms of product quality.
average daily opened measurement for the week is calculated by totaling the number of bugs reports opened each day during the week, and dividing that total by seven. we would expect the average daily opened measurements to sink gradually toward zero, eventually showing zero new bug reports opened in the final week or two.
Avg Daily Closed. would expect the average daily closed measurements to consistently stand above the average daily opened measurements as the backlog goes to zero, and also expect the average daily closed measurements to move gradually toward zero as well.
average daily opened measurement for the week is calculated by totaling the number of bugs reports opened each day during the week, and dividing that total by seven. we would expect the average daily opened measurements to sink gradually toward zero, eventually showing zero new bug reports opened in the final week or two.
Avg Daily Closed. would expect the average daily closed measurements to consistently stand above the average daily opened measurements as the backlog goes to zero, and also expect the average daily closed measurements to move gradually toward zero as well.
Product metrics are often forgotten, but having good product metrics helps you understand the quality status of the system under test.
- functional size per application
- defect density
- number of pages per document/module
If the 10% of the tests which are failed, queued up for execution, or blocked are relatively unimportant, and the defects associated with the 5% of failed tests are also unimportant, this could be good news. However, if some of those tests relate to important test conditions, or if the defects associated with the failed tests relate to key quality risks for the product, then we could be in serious trouble, especially given that very little time remains.
So, we need product metrics to tell us, throughout test execution, whether the quality is on track for a successful delivery.
So, it’s important to remember—and to remind everyone who sees these metrics—that the role of testing is to measure quality, not to directly improve quality.