Testing Metrics


Published on

Metrics used in Software Testing

Published in: Technology, Business
  • The metrics presented do not say anything about the quality of the product. I think I can say without contradiction that the number of defects found and fixed tell you nothing about the number left unless you use the data to make an estimate. Generally speaking the more defects your test program finds, the more you will ship. I don't have a web site but if anyone wants to know how to make an accurate estimate of the number of defects remaining, send me an email at tom_walton@videotron.ca. If you want to know how to make sure that you don't ship a buggy product and how to reduce your life cycle costs at the same time, send me request and I will send you what you need to know.
    Quality is to testing as Jumbo is to shrimp. Together them make an oxymoron. Testing increases costs and lengthens the project. System testing is always on the critical path. Testing does not significantly improve reliability. For all but the most trivial systems, no economically feasible test program will find enough defects to make the system reliable.
    Quality is not the responsibility of the test team. Quality is the responsibility of the people who designed and built the product.
    Are you sure you want to  Yes  No
    Your message goes here
  • How can I say I covered enough numbers of test cases. in other words how can I tell if this many test cases pass my application is sucess? there are so test cases which I am not covered how to identify those test cases? Is my test coverage is 100% ? please answer me ( dxbsha1@gmail.com)
    Are you sure you want to  Yes  No
    Your message goes here
  • it is very usefull. Thanks much for posting this
    Are you sure you want to  Yes  No
    Your message goes here
  • simple, effective and nice presentation
    Are you sure you want to  Yes  No
    Your message goes here
  • Very informative slides. Its really helpful for me. Thanks for sharing with us. Actually I was in search for some good articles on testing metrics and finally i got one over the internet which is also helped me to complete my task. So I would like to share that post link, please check this url...
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Testing Metrics

  1. 1. SOFTWARE TESTING METRICS Presenter : P.M.Venkatesh Babu
  2. 2. What is a METRIC <ul><li>Metrics can be defined as “STANDARDS OF MEASUREMENT </li></ul><ul><li>Metric is a unit used for describing or measuring an attribute </li></ul><ul><li>Test metrics are the means by which the software quality can be measured </li></ul><ul><li>Test metrics provides the visibility into the readiness of the product, and gives clear measurement of the quality and completeness of the product </li></ul>
  3. 3. Why we need Metrics ? <ul><li>You cannot improve what you cannot measure </li></ul><ul><li>You cannot Control what you cannot measure </li></ul><ul><li>Without measurement it is impossible to tell whether the process implemented is improving or not </li></ul><ul><li>Metrics helps in taking decisions for next phase of activities </li></ul><ul><li>Metrics helps in understanding the type of improvement required and helps in taking decisions on process or technology change </li></ul>
  4. 4. Why Metrics in Software Testing? <ul><li>There will be certain questions during and after testing such as : </li></ul><ul><li>How long would it take to test ? </li></ul><ul><li>How Bad / Good is the product? </li></ul><ul><li>How many bugs still remain in the product? </li></ul><ul><li>Will testing be completed on time? </li></ul><ul><li>Was the testing done effectively? </li></ul><ul><li>How much effort went into testing the product? </li></ul><ul><li>To Answer these questions properly we need some type of measurements and record keeping to justify the answers. </li></ul><ul><li>This is where the testing metrics comes into picture. </li></ul>
  5. 5. Types Of Metrics <ul><li>Base metrics (Direct Measure) </li></ul><ul><li>The Base metrics constitute the raw data gathered by the test Engineers throughout the testing effort </li></ul><ul><li>The Base metrics are used to provide project status reports to the Test lead and to the project manager </li></ul><ul><li>The Base metrics provide the input data to feed into the formulas used to derive Calculated metrics </li></ul><ul><li>Examples of Base metrics are: </li></ul><ul><li># of test cases </li></ul><ul><li># of test cases executed </li></ul>
  6. 6. Types Of Metrics (contd..) <ul><li>Calculated Metrics (Indirect Measure) </li></ul><ul><li>The Calculated Metrics convert the Base metrics data into more useful information </li></ul><ul><li>The Calculated Metrics are generally prepared by the Test lead and is used to track the progress of the project at different levels like at Module level, at Tester level and for the project as a whole </li></ul><ul><li>The Calculated Metrics provide valuable information that when used and implemented often times leads to significant improvements in the Overall SDLC </li></ul>
  7. 7. Base Metrics and Testing Phases Test Reporting Phase Test Execution time Test Reporting Phase Test case Execution time Test Reporting Phase Total Passes and Failures Test Reporting Phase Total Executions Test execution Phase Number of First run Failures Regression Phase Number of Test cases Re-executed Test Execution Phase Number of Test cases Passed Test Execution Phase Number of Test cases Executed Test Execution Phase Number of Test cases Failed Test Dev / Execution Phase Number of Test cases Blocked Test Development Phase Number of Test cases under Investigation Test Development Phase Number of test cases TESTING PHASE TEST METRIC
  8. 8. Calculated Metrics and Phases <ul><li>The Following Calculated metrics are created at Test Reporting Phase or Post Test Analysis Phase : </li></ul><ul><li>% of Test cases Passed </li></ul><ul><li>% of Test Coverage </li></ul><ul><li>% of Defects corrected </li></ul><ul><li>% of Test cases Blocked </li></ul><ul><li>% of Rework </li></ul><ul><li>% of Test Effectiveness </li></ul><ul><li>1 st Run Fail Rate </li></ul><ul><li>Defect discovery rate </li></ul><ul><li>Overall Fail rate </li></ul>
  9. 9. Test case Defect Density <ul><li>The number of errors found in test cases v/s test cases developed and executed </li></ul><ul><li>( Defective Test cases / Total Test cases ) * 100 </li></ul><ul><li>Example : Total no of test cases developed is 1360, total test cases executed is 1280, total no of test cases passed is 1065, total no of test scripts failed is 215 </li></ul><ul><li>So Test case Defect Density is : </li></ul><ul><li>215 X 100 </li></ul><ul><li>------------------------------- = 16.8 % </li></ul><ul><li>1280 </li></ul><ul><li>The 16.8 % value can also be called as Test Case Efficiency % which depends upon the total number of Test cases which found defects </li></ul>
  10. 10. Defect Slippage Ratio <ul><li>No of bugs reported from Production V/S No of defects reported during execution </li></ul><ul><li>No of Defects slipped / ( Number of Defects Raised – Number Defects Withdrawn) * 100 </li></ul><ul><li>Example : Customer reported 21 defects, total no of defects found while testing are, total no of Invalid defects are 17 </li></ul><ul><li>So Slippage ratio is : [ 21 / (267 – 17) ] X 100 = 8.4% </li></ul>
  11. 11. Requirement Volatility Metric <ul><li>This metric ensures that the requirements are normalized or defined properly while estimating </li></ul><ul><li>No of requirements agreed V/S No of requirements changed </li></ul><ul><li>(No of requirements Added + Deleted + Modified) * 100 / No of original requirements </li></ul><ul><li>Example : SVN 1.3 release has 67 requirements initially, later 7 new requirements are added, 3 requirements are deleted from initial requirements and modified 11 requirements </li></ul><ul><li>Hence Requirement volatility is calculated as : </li></ul><ul><li>(7 + 3 + 11 ) X 100 / 67 = 31.34 % </li></ul><ul><li>This means that almost 1/3 of the requirements changed after the initial identification of requirements </li></ul>
  12. 12. Conclusion <ul><li>The Test metrics should be reviewed & interpreted on regular basis throughout the test effort and particularly after the application is released into production </li></ul><ul><li>There are several key factors in implementing and using the metrics in the organization, beginning with determining the goal for developing the metrics, followed by the identification of metrics to be tracked and ending with sufficient analysis of the resulting data to be able to make changes to the software development lifecycle </li></ul><ul><li>- So finally Metrics themselves so not create improvements, they do provide the objective information necessary to understand what changes are necessary to be implemented </li></ul>
  13. 13. <ul><li>Questions ??? </li></ul>