Your SlideShare is downloading. ×
Making Numbers Count: Metrics That Matter
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Making Numbers Count: Metrics That Matter

124

Published on

As testers and test managers, we are frequently asked to report on the progress and results of our testing. The question “How is testing going?” may seem simple enough, but our answer is ultimately …

As testers and test managers, we are frequently asked to report on the progress and results of our testing. The question “How is testing going?” may seem simple enough, but our answer is ultimately based on our ability to extract useful metrics from our work and present them in a meaningful way. This is particularly important in agile environments, where clear, concise, and up-to-date metrics are potentially needed multiple times per day. Mike Trites identifies a number of ways metrics can be used to measure progress during a test cycle and, ultimately, to determine when to consider testing complete. Learn the common pitfalls that metrics misuse can lead to and how you can avoid them by giving proper context when communicating metrics to your stakeholders. Take back key metrics for measuring the effectiveness of your testing and discover how to use what is learned on one project to improve your testing process on future projects.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
124
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
10
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. W18 Special Topics 5/7/2014 3:00:00 PM Making Numbers Count: Metrics That Matter Presented by: Mike Trites Professional Quality Assurance, Ltd. Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
  • 2. Mike Trites Professional Quality Assurance, Ltd. Senior test consultant with Professional Quality Assurance, Ltd. Mike Trites has nearly a decade of experience in the software testing industry and holds multiple certifications, including an Advanced-Level Test Manager Certification from the ISTQB. As a consultant, Mike has worked with clients, both small and large, in a wide range of environments. He has helped deliver projects from web applications and database management software to financial software and video lottery systems. As a test manager, Mike has managed all aspects of the testing lifecycle and works with clients to help define, implement, and improve their internal testing processes. Find Mike on LinkedIn at ca.linkedin.com/in/miketrites.
  • 3. PQA Company Overview 4/26/2014 ©2012 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 1 STAREAST 2014 May 4 – 9, 2014 Orlando, Florida Mike Trites Making Numbers Count Metrics that Matter About Me Mike Trites ‣ Senior Test Consultant ‣ Nearly 10 years of experience in software testing, working with several clients in multiple industry verticals Certifications ‣ Black Box Software Testing Foundations – AST ‣ Certified Tester, Foundation Level – ISTQB ‣ Certified Tester, Advanced Level Test Manager – ISTQB 2©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
  • 4. PQA Company Overview 4/26/2014 ©2012 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 2 Introduction 3©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com How is testing going? “95% of test cases so far have passed.” “We found 15 new defects yesterday.” “We’re on-track.” What I’m here for Illustrate the important role that metrics play in the software testing process. Explain how to give context when communicating metrics. Describe how to use metrics to improve the testing process for future projects. Identify some common pitfalls and how to avoid them. Identify a selection of metrics to: ‣ Measure progress during a test cycle. ‣ Determine when to stop testing. ‣ Measure the effectiveness of testing. 4©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
  • 5. PQA Company Overview 4/26/2014 ©2012 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 3 What I’m NOT here for Provide a “one size fits all” solution for you or your organization. Why not? ‣ Metrics are situation / context specific. ‣ There is no silver bullet. ‣ There is no “right” answer. ‣ They will not solve all your problems. 5©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com Agenda Why Metrics are Important Defining and Capturing Metrics Reporting on Metrics Using Metrics for Process Improvement Misuse of Metrics 6©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
  • 6. PQA Company Overview 4/26/2014 ©2012 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 4 Why Metrics are Important 8©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
  • 7. PQA Company Overview 4/26/2014 ©2012 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 5 Why Metrics are Important 9©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com Metrics play an important role in: • Estimation • Evaluating suspension criteria • Evaluating exit criteria • Status reporting • Process improvement Estimation Metrics captured on one project can help us estimate various activities for future projects. With enough historical data, we can begin to estimate: ‣ Number of test cases ‣ Execution rate ‣ Number of defects ‣ Overall effort 10©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
  • 8. PQA Company Overview 4/26/2014 ©2012 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 6 Suspension Criteria Occasionally, it may be necessary to halt testing of a particular version / build if the quality is found to be too low. How do we make this determination? ‣ Number of defects? ‣ Number of high-severity defects? ‣ Defect detection rate? ‣ Test failure rate? 11©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com Exit Criteria How do we know when to stop testing? ‣ The allotted effort (schedule and / or budget) is expended. ‣ All planned tests have been executed. ‣ All planned tests pass. ‣ The defect detection rate drops below a certain threshold. There are pros and cons for each approach. 12©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
  • 9. PQA Company Overview 4/26/2014 ©2012 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 7 Exit Criteria The best approach is a combination that takes into account multiple criteria, such as: ‣ Effort (schedule / budget) ‣ Defect detection rate ‣ Number of open defects ‣ Completion percentage ‣ Pass rate 13©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com Status Reporting Simple and clear method for communicating the current status of testing to stakeholders Allows stakeholders to: ‣ Draw conclusions ‣ Make decisions Detailed status reports provide a transparent view into the testing effort and show progress toward defined exit criteria. 14©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
  • 10. PQA Company Overview 4/26/2014 ©2012 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 8 Defining and Capturing Metrics Defining Metrics Every phase of the testing process can be measured. Schedule / Budget Test Coverage: ‣ Requirements ‣ Configurations ‣ Code ‣ Automation Test Design Test Execution Defects 16©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
  • 11. PQA Company Overview 4/26/2014 ©2012 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 9 Capturing Metrics 17©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com Expensive commercial tools can be useful, but are not always necessary. • Examples: ‣ Test Case Management Tool ‣ Defect Management Tool ‣ Time Tracking System There are less expensive tools that can be used to achieve the same results: • Examples: ‣ Bugzilla ‣ JIRA ‣ Excel Reporting on Metrics
  • 12. PQA Company Overview 4/26/2014 ©2012 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 10 Metrics Need Context When metrics are used in isolation, their significance is left open to interpretation. Stakeholders can draw the wrong conclusions if no context is provided and they are instead left to interpret the data on their own. Textual summary can provide context, but so can a good report. 19©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com What makes a good report? Accurate – Data is not purposely skewed or omitted. Clear – Information is presented in a way that is unambiguous and easy to interpret at a glance. Concise – Report is not cluttered with unneeded or distracting data. Includes any necessary: ‣ Titles ‣ Labels ‣ Data values ‣ Trend lines ‣ Annotations 20©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
  • 13. PQA Company Overview 4/26/2014 ©2012 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 11 21©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com Medium 68% Low 18% High 14% Medium 68% Low 18% High 14% Colour Conventions Defects by Severity More meaningful with appropriate colours 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Example: Test Progress 22©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
  • 14. PQA Company Overview 4/26/2014 ©2012 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 12 Example: Requirements Coverage 23©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com Example: Configuration Coverage 24©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com Windows XP Windows Vista Windows 7 Windows 8 Mac OSX IE 7 ✓✓✓✓ ✓✓✓✓ IE 8 ✓✓✓✓ ✓✓✓✓ IE 9 ✓✓✓✓ ✓✓✓✓ IE 10 ✓✓✓✓ Chrome ✓✓✓✓ ✓✓✓✓ ✓✓✓✓ ✓✓✓✓ ✓✓✓✓ Firefox ✓✓✓✓ ✓✓✓✓ ✓✓✓✓ ✓✓✓✓ ✓✓✓✓
  • 15. PQA Company Overview 4/26/2014 ©2012 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 13 Example: Automation Coverage 25©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com Total Number of Test Cases Cycle 2 (40%) Cycle 1 (15%) 0 5 10 15 20 25 30 Example: Open vs. Closed Defects 26©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 0 5 10 15 20 25 30
  • 16. PQA Company Overview 4/26/2014 ©2012 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 14 0 2 4 6 8 10 12 14 16 18 0 2 4 6 8 10 12 14 16 18 Example: Open Defects 27©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com Example: Defects by Release 28©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com = 63 (49%) = 56 (44%) = 9 (7%) Fixed / Planned v0.1 v0.2 v0.3 v0.4 v0.5 v0.6 v0.7 Discovered v0.1 1 9 2 v0.2 7 10 15 1 1 3 v0.3 11 17 1 3 v0.4 1 31 10 3 v0.5 1 1
  • 17. PQA Company Overview 4/26/2014 ©2012 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 15 Example: Scorecard 29©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com Execution Progress Pass Rate Blocked Tests Must-Fix Other Component 1 100 % 100 % 0 0 0 Component 2 100 % 78 % 0 0 4 Component 3 87 % 80 % 0 1 1 Component 4 95 % 100 % 0 0 4 Component 5 75 % 92 % 1 1 0 Component 6 98 % 79 % 2 1 4 Overall 93 % 88 % 3 3 13 DefectsTest Execution Execution Progress Pass Rate Blocked Tests Must-Fix Other Component 1 100 % 100 % 0 0 0 Component 2 100 % 78 % 0 0 4 Component 3 87 % 80 % 0 1 1 Component 4 95 % 100 % 0 0 4 Component 5 75 % 92 % 1 1 0 Component 6 98 % 79 % 2 1 4 Overall 93 % 88 % 3 3 13 DefectsTest Execution http://dilbert.com/strips/comic/2007-05-16/ Dashboards 30©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
  • 18. PQA Company Overview 4/26/2014 ©2012 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 16 Process Improvement Process Improvement "Those who cannot remember the past are condemned to repeat it.” - George Santayana What is “better”? How do we know if we’re better now than we were? Estimates: ‣ Comparing planned effort and actual effort ‣ Estimate variance and what factors affected it 32©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
  • 19. PQA Company Overview 4/26/2014 ©2012 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 17 Metrics for Process Improvement 33©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com Closure Period / Turn- around Time • Measures how long it takes for a reported defect to be rectified Defect Ratio • Can be used to predict how many defects may be found based on the number of test cases that remain to be executed Metrics for Process Improvement 34©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com Root Cause Analysis • Allows you to proactively avoid anticipated defects in the future Closed Defects by Resolution • Identifies instances of duplicate or invalid defects Defect Removal Efficiency • Identifies defects that are uncovered after the application is released
  • 20. PQA Company Overview 4/26/2014 ©2012 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 18 Trends One measurement captures status at a point in time. When a series of data points are examined, trends in that data start to emerge. Trends allow us to ‣ Differentiate between systemic behaviour and temporary anomalies. ‣ Make predictions about the future. Accuracy increases with more historical data. 35©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com Misuse of Metrics
  • 21. PQA Company Overview 4/26/2014 ©2012 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 19 Intentional Manipulation “There are three kinds of lies: lies, damned lies, and statistics.” - Benjamin Disraeli Metrics can be manipulated to make any situation look good or bad. 37©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com Misleading Graphs An example of a good graph: 38©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 0 100 200 300 400 500 A B C D 430 455 448 465
  • 22. PQA Company Overview 4/26/2014 ©2012 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 20 Misleading Graphs Inappropriate scale: 39©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 0 200 400 600 800 1000 A B C D Misleading Graphs Scale does not start at 0: 40©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 420 425 430 435 440 445 450 455 460 465 A B C D
  • 23. PQA Company Overview 4/26/2014 ©2012 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 21 Tester 1 Tester 2 Tester 3 Tester 4 41©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com Misleading Graphs Defects by Tester 11% 42% 5%42% Tester 1 Tester 2 Tester 3 Tester 4 Defects by Tester 42©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com Misleading Graphs
  • 24. PQA Company Overview 4/26/2014 ©2012 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 22 Poor Choice of Format 43©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 12% 15% 14% 8%11% 17% 13% 10% Tester 1 Tester 2 Tester 3 Tester 4 Tester 5 Tester 6 Tester 7 Tester 8 Defects by Tester Better Choice of Format 44©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 0% 2% 4% 6% 8% 10% 12% 14% 16% 18% 12% 15% 14% 8% 11% 16% 13% 10% Defects by Tester
  • 25. PQA Company Overview 4/26/2014 ©2012 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 23 Overloading Most managers are interested in defect statistics. Does defect count measure: ‣ Tester effectiveness? ‣ Developer skill? ‣ Product quality? ‣ Effort remaining? One metric does not tell the entire story. 45©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com Surrogate Measures Sometimes, we don’t have a way to easily measure the attribute we are really interested in. Instead, we use another, more convenient metric in its place. But, does the metric we have chosen actually give us the information we are looking for? ‣ To find out, we can use a “five-why” analysis Examples: ‣ Code Coverage ‣ Velocity 46©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
  • 26. PQA Company Overview 4/26/2014 ©2012 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 24 Performance Assessment People optimize what we measure them against, at the expense of what we don’t. For example, if we assess testers based on their defect counts, how much time will testers spend: ‣ Documenting their tests? ‣ Coaching other testers? ‣ Researching and polishing defect reports to make their defect easier to understand, assess and replicate? ‣ Logging variants of the same failure to increase defect count? 47©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
  • 27. PQA Company Overview 4/26/2014 ©2012 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 25 Key Points to Remember Metrics can be captured for just about any testing activity. Metrics play an important role in: ‣ Estimation ‣ Status reporting ‣ Evaluating suspension and exit criteria ‣ Process improvement Metrics, like anything, should be planned in advance. Reporting needs should be identified before setting up tools. To avoid misinterpretation, metrics need context. 49©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com Key Points to Remember Even when properly presented, too much information can be overwhelming. It is best to limit the number of metrics and reports that are presented to stakeholders to only what is essential. Capturing, analysing and reporting on metrics can be a drain or waste of time if those metrics are not being used to provide some value in return. Metrics influence tester performance, but not always in the way you might think. Metrics should be used as a starting point for discussion. One metric does not tell the entire story. 50©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
  • 28. PQA Company Overview 4/26/2014 ©2012 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com 26 Questions or Comments? 51©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com Thanks for coming! Contact Information: 52©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com Mike.Trites@pqatesting.com ca.linkedin.com/in/miketrites @mike_trites www.pqatesting.com

×