Uploaded byDakiry
DaKiRY_BAQ2016_QADay_Оксана Вей "Manage by metrics"
This document discusses software measurement and metrics. It defines software metrics as quantified observations about aspects of software products, processes, or projects. It notes that while it is easy to collect numbers, understanding what they mean is difficult. The document classifies metrics into process, product, and project metrics. It provides examples of metrics that could measure objectives like the effectiveness of the testing process in finding bugs. The document discusses challenges with metrics like the human element and presents a process for defining good metrics. It provides more details on examples of process, project, and product metrics.
More Related Content
Similar to DaKiRY_BAQ2016_QADay_Оксана Вей "Manage by metrics"
DaKiRY_BAQ2016_QADay_Оксана Вей "Manage by metrics"
- 1.
- 2. What is SoftwareMeasurement? “Quantified observations” about aspect of software (product,process or project) Introduction - Process Metrics - Project Metrics – Product Metrics
- 3. What is SoftwareMeasurement? “It’s easy to get numbers, what is hard is to know they are right and understand what they mean.” William C. Hetzel: Introduction - Process Metrics - Project Metrics – Product Metrics
- 4. What is SoftwareMeasurement? “To measure is to know” “If you cannot measure it, you cannot improve it.” “The more you understand what is wrong with a figure, the more valuable that figures become.” Introduction - Process Metrics - Project Metrics – Product Metrics
- 5. Metrics clasification Process metricsProduct metrics Project metrics Introduction - Process Metrics - Project Metrics – Product Metrics
- 6. What makes agood measure? Simple Objective Easily collected Robust Valid Introduction - Process Metrics - Project Metrics – Product Metrics
- 7. What can Measuresdo for you? Facilitate estimation Identify risk areas Measure testing status Measure/Predict product quality Measure test effectiveness Identify training opportunities Identify process improvement opportunities Provide “meters to flag actions” Introduction - Process Metrics - Project Metrics – Product Metrics
- 8. 2 sides ofMeasurement “… the information will help me to understand what is going on and do a better job.” “… the information may be used against me.” Introduction - Process Metrics - Project Metrics – Product Metrics
- 9. The Human Element Measureprocess and products instead of people if possible Beware of the dark side of the Hawthorne Effect Introduction - Process Metrics - Project Metrics – Product Metrics
- 10. Process for defininggood netrics 1. Objectives 2. Questions 3. Metrics 4. Goals 5. Improvements Introduction - Process Metrics - Project Metrics – Product Metrics
- 11. Process Metrics MEASURE PROCESSCAPABILITIES MEASURE THE OVERALL SOFTWARE PROCESS Introduction - Process Metrics - Project Metrics – Product Metrics
- 12. Process Metrics Objective: Findingbugs Question: What percentage of the bugs present in a system during testing is found by the testing process? Metric: Defect Detection Effectiveness (DDE) or Defect Detection Percentage (DDP) DDE = 𝑑𝑒𝑓𝑒𝑐𝑡𝑠 𝑑𝑒𝑡𝑒𝑐𝑡𝑒𝑑 𝑑𝑒𝑓𝑒𝑐𝑡𝑠 𝑑𝑒𝑡𝑒𝑐𝑡𝑒𝑑 + 𝑝𝑟𝑜𝑑𝑢𝑐𝑡𝑖𝑜𝑛 𝑑𝑒𝑓𝑒𝑐𝑡𝑠 Introduction - Process Metrics - Project Metrics – Product Metrics
- 13. Process Metrics Objective: Findingimportant bugs Question: What percentage of the bugs present in a system during testing is found by the testing process? Metric: Defect Detection Effectiveness (DDE) or Defect Detection Percentage (DDP) DDE (important bugs) > DDE (all bugs) Introduction - Process Metrics - Project Metrics – Product Metrics
- 14. Process Metrics Objective: Findingbugs Question: Are there any external factors affecting testing process? Metric: Bug opened count - the number of times each bug report is opened Introduction - Process Metrics - Project Metrics – Product Metrics
- 15. Rework Inefficiency =112*1+26*2+6*3+5*4+1*10=212 Introduction - Process Metrics - Project Metrics – Product Metrics
- 16. Project Metrics WHERE WESTAND ON THE PROJECT AND HOW CLOSE WE ARE TO ACHIEVING PROJECT OBJECTIVES Introduction - Process Metrics - Project Metrics – Product Metrics
- 17. Project Metrics Objective: findingmost of the bugs and fixing the most important bugs Question: Where we are (our progress) in terms of finding and fixing bugs? Metrics: Total Daily Opened - the cumulative number of bug reports opened as of a given day on the project Total Daily Closed - the cumulative number of bug reports resolved (either confirmed as fixed or deferred for this project) as of a given day on the project Average Daily Opened - the average daily number of bug reports opened for a given week Average Daily Closed - the average daily number of bug reports resolved for a given week Average Daily Backlog - the average daily difference between the total daily opened and total daily closed bug reports Introduction - Process Metrics - Project Metrics – Product Metrics
- 18. Introduction - ProcessMetrics - Project Metrics – Product Metrics
- 19. Product Metrics THE CURRENTQUALITY STATUS OF THE SYSTEM UNDER TESTING Introduction - Process Metrics - Project Metrics – Product Metrics
- 20. Product Metrics 95% ofthe tests have been run 90% of the tests have passed 5% of the tests have failed 4% of the tests are ready to run 1% of the tests are blocked we are right on schedule in terms of the test execution schedule Does this tell us good news or bad news? Introduction - Process Metrics - Project Metrics – Product Metrics
- 21. Product Metrics Introduction -Process Metrics - Project Metrics – Product Metrics Objective: Risk-based testing Question: How effectively are we reducing quality risk overall? Metric: traceability between test results/defects and risk items
- 22. Product Metrics Introduction -Process Metrics - Project Metrics – Product Metrics
- 23. KISS – KeepIt Simple Sir More is not always better All metrics are not forever Automate collection when possible Introduction - Process Metrics - Project Metrics – Product Metrics
- 24.
Editor's Notes
- #11 Define objectives for the process. Consider questions about the effectiveness, efficiency, and elegance with which the process realizes those objectives. Devise measurable metrics, either direct or surrogate, for each effectiveness, efficiency, and elegance question. Determine realistic goals for each metric, based on an understanding of industry best practices and current process capability. As appropriate, implement process improvements that increase effectiveness, efficiency, or elegance as measured by the metrics.
- #12 - number of defects found during testing - effectiveness of defect removal during development - effectiveness of defect detection - efforts required for testing
- #17 Project metrics are the most commonly used test metrics. project metrics can tell us whether the testing is on target to achieve the plan. Project metrics provide practical, often immediately-useful insights about your current project status and trends. By benchmarking your project metrics against past projects and industry norms, you can see where your project stands in terms of current status and trends. This can give you good ideas about what kinds of project course-corrections to make to achieve optimal outcomes for your project. When multiple, balanced metrics are used properly, project metrics can present a complete picture of the project that can help you avoid making project decisions that will result in sub-optimal outcomes. - resources - cost and schedule Unfortunately, people often over-rely on project metrics, and often fail to properly balance project metrics. While project metrics tell you the progress made toward project objectives, they might not give a full and balanced picture, especially in terms of product quality.
- #18 average daily opened measurement for the week is calculated by totaling the number of bugs reports opened each day during the week, and dividing that total by seven. we would expect the average daily opened measurements to sink gradually toward zero, eventually showing zero new bug reports opened in the final week or two. Avg Daily Closed. would expect the average daily closed measurements to consistently stand above the average daily opened measurements as the backlog goes to zero, and also expect the average daily closed measurements to move gradually toward zero as well.
- #19 average daily opened measurement for the week is calculated by totaling the number of bugs reports opened each day during the week, and dividing that total by seven. we would expect the average daily opened measurements to sink gradually toward zero, eventually showing zero new bug reports opened in the final week or two. Avg Daily Closed. would expect the average daily closed measurements to consistently stand above the average daily opened measurements as the backlog goes to zero, and also expect the average daily closed measurements to move gradually toward zero as well.
- #20 Product metrics are often forgotten, but having good product metrics helps you understand the quality status of the system under test. - functional size per application - defect density - number of pages per document/module
- #21 If the 10% of the tests which are failed, queued up for execution, or blocked are relatively unimportant, and the defects associated with the 5% of failed tests are also unimportant, this could be good news. However, if some of those tests relate to important test conditions, or if the defects associated with the failed tests relate to key quality risks for the product, then we could be in serious trouble, especially given that very little time remains. So, we need product metrics to tell us, throughout test execution, whether the quality is on track for a successful delivery. So, it’s important to remember—and to remind everyone who sees these metrics—that the role of testing is to measure quality, not to directly improve quality.