MI
AM Tutorial
9/30/2013 8:30:00 AM

"Measurement and Metrics for
Test Managers"
Presented by:
Rick Craig
Software Quality...
Rick Craig
Software Quality Engineering
A consultant, lecturer, author, and test manager, Rick Craig has led numerous team...
© 2013 SQE Training V3.1

1
© 2013 SQE Training V3.1

2
Notice of Rights
Entire contents © 2013 by SQE Training, unless otherwise noted on specific items. All
rights reserved. No...
Please set any pagers or cellular phones to mute.

© 2013 SQE Training V3.1

4
© 2013 SQE Training V3.1

5
© 2013 SQE Training V3.1

6
© 2013 SQE Training V3.1

7
© 2013 SQE Training V3.1

8
Lord Kelvin was apparently a virtual sound-bite machine. There
are entire Web sites devoted to him and to his quotes.

© 2...
© 2013 SQE Training V3.1

10
Rick Craig and Bill Hetzel, Software Measures and Practices
Benchmark Study, 1990. Research Reports TR900-904. SQE,
Orange...
Rick Craig and Bill Hetzel, Software Measures and Practices
Benchmark Study, 1990. Research Reports TR900-904. SQE,
Orange...
Measure: a quantified observation about any aspect of software
Metric: a measure used to compare two or more products, pro...
Simple: a metric that does not include complicated formulas, is intuitive, and easy
to understand. Number of severe bugs i...
Characteristics of metrics and their use:
•Broadly applicable
•Highly visible
•Consistently applied
•Management supported
...
© 2013 SQE Training V3.1

16
© 2013 SQE Training V3.1

17
Have you ever had to collect or submit a metric or report that you knew no
one would ever see? Did you (or at least were y...
© 2013 SQE Training V3.1

19
Unless you’re blessed with a “silver tongue,” buy-in is far from easy and
training helps. Metrics can be a powerful force ...
© 2013 SQE Training V3.1

21
One of the biggest problems with metrics is buy-in. If practitioners feel that the metrics
are not being used—or worse, ar...
The Hawthorne Studies were conducted by Harvard Business School
professor Elton Mayo from 1927 to 1932 at the Western Elec...
© 2013 SQE Training V3.1

24
© 2013 SQE Training V3.1

25
The numbers on the graph above were made up so please don’t Google
them. I do remember seeing a chart something like this ...
© 2013 SQE Training V3.1

27
© 2013 SQE Training V3.1

28
Use a metric to validate a metric: Virtually anything that is important to
measure should have two or more metrics. For ex...
More is not always better/All metrics are not forever: We have visited
companies that have entire teams that collect, slic...
This page left blank

© 2013 SQE Training V3.1

31
© 2013 SQE Training V3.1

32
The well-designed dashboard of a 1958 Corvette

© 2013 SQE Training V3.1

33
Fun fact: The business dashboard originated at General Electric many years
ago.
— Greg Brue, Six Sigma for Managers, Brief...
© 2013 SQE Training V3.1

35
Quality is the totality of features and characteristics of a product or service
that bear on its ability to satisfy stated...
These quality characteristics are sometimes called the “-ility” factors. These
factors are very appealing on first blush, ...
What does the chart above say about the relative risk of each module??
One of the ISTQB principles of test is called defec...
Defect Density: a metric that compares the number of defects to a measure
of size (e.g., defects per KLOC)
— Rick Craig an...
As the complexity of a product increases, so too does the probability of
shipping defects.

© 2013 SQE Training V3.1

40
© 2013 SQE Training V3.1

41
© 2013 SQE Training V3.1

42
© 2013 SQE Training V3.1

43
© 2013 SQE Training V3.1

44
Just because you’ve run 90% of the test cases (raw data) doesn’t mean that
you are 90% done vs. risk, coverage, execution ...
Does a report like this help?

© 2013 SQE Training V3.1

46
© 2013 SQE Training V3.1

47
© 2013 SQE Training V3.1

48
Many organizations use the “defect arrival rate” as an impartial measure to
assist them in predicting when a product will ...
“The defect that is prevented doesn’t need repair, examination, or
explanation. The first step is to examine and adopt the...
© 2013 SQE Training V3.1

51
The purpose of the Test Summary Report is for the test manager to report on
the results of the testing. An added benefit, ...
© 2013 SQE Training V3.1

53
© 2013 SQE Training V3.1

54
© 2013 SQE Training V3.1

55
© 2013 SQE Training V3.1

56
© 2013 SQE Training V3.1

57
© 2013 SQE Training V3.1

58
This is an experience Rick Craig had at his restaurant many years ago:
My head server decided to create a customer satisfa...
© 2013 SQE Training V3.1

60
© 2013 SQE Training V3.1

61
Bug drawing by Catherine Sabourin. From I am a Bug by Rob Sabourin.
Some of the problems associated with counting defects:...
Defect Removal Efficiency (DRE) is a metric similar to DDP. DDP counts the
bugs found (whether fixed or not). DRE only cou...
© 2013 SQE Training V3.1

64
This graph is just an example. The actual hours required to fix a bug will vary
widely from company to company due to comp...
Typically, the longer a defect exists before discovery, the more it costs to fix.
One metric often collected for a defect ...
Latent defect: an existing defect that has not yet caused a failure because
the exact set of conditions has not been met
M...
© 2013 SQE Training V3.1

68
© 2013 SQE Training V3.1

69
Most test methodologies require (at a minimum) a measure of requirements
coverage. A requirements traceability matrix ensu...
© 2013 SQE Training V3.1

71
The table above is a conceptual model of the output of a code coverage tool.
These tools can measure statement, decision, ...
It is our opinion that code coverage is a metric that is most useful at the
developmental test level(s)—unit and/or integr...
© 2013 SQE Training V3.1

74
© 2013 SQE Training V3.1

75
© 2013 SQE Training V3.1

76
© 2013 SQE Training V3.1

77
© 2013 SQE Training V3.1

78
© 2013 SQE Training V3.1

79
© 2013 SQE Training V3.1

80
Please note that Product Quality is a development metric that is normally
measured by the testers. The other instruments a...
© 2013 SQE Training V3.1

82
© 2013 SQE Training V3.1

83
© 2013 SQE Training V3.1

84
This page left blank

© 2013 SQE Training V3.1

85
© 2013 SQE Training V3.1

86
Brooks’s Law: “Adding manpower to a late software project makes it later”
According to Brooks himself, the law is an "outr...
The above information comes from the ISTQB.
If you are going to gain credibility in the software development arena, you mu...
The above information comes from the ISTQB.
Other factors:
Complexity of the process, technology, and organization
Signifi...
The above information comes from the ISTQB.
A common practice is also to estimate the number of test cases required.
Assum...
These figures (both on the slide and in the notes) are for the software
industry as a whole, not specifically testing, but...
Estimation of software and system engineering has long been known to be
fraught with difficulties, both technical and poli...
John McGarry, David Card, Cheryl Jones, Beth Layman, Elizabeth Clark,
Joseph Dean, Fred Hall.
Practical Software Measureme...
© 2013 SQE Training V3.1

94
The cone of uncertainty helps explain why some of our early estimates are
so bad. Using the diagram above, you can see tha...
© 2013 SQE Training V3.1

96
Anecdote by Rick Craig: I confess that I misunderstood the “cone of uncertainty” for
years. Perhaps because I live in Flor...
Manager:

When will you be done testing?

Tester:

Oh, in about four weeks.

Manager:

Any chance it might take five weeks...
There are a number of factors that can affect a schedule; however, they can
all be characterized in one of four categories...
Most of us have been there. You have estimated that you can test a product
in six weeks with four testers. The manager off...
The list in the slide above was adapted from the ISTQB Advanced Syllabus
for Test Management.
Organizational history and m...
Read Karl’s entire article “Estimation Safety Tips” at www.stickyminds.com

© 2013 SQE Training V3.1

102
* This is tongue in cheek. We are suggesting you not use the words pad or
fudge or whatever—but rather “normalize” the val...
Upcoming SlideShare
Loading in...5
×

Measurement and Metrics for Test Managers

250

Published on

To be most effective, test managers must develop and use metrics to help direct the testing effort and make informed recommendations about the software’s release readiness and associated risks. Because one important testing activity is to “measure” the quality of the software, test managers must measure the results of both the development and testing processes. Collecting, analyzing, and using metrics is complicated because many developers and testers are concerned that the metrics will be used against them. Join Rick Craig as he addresses common metrics—measures of product quality, defect removal efficiency, defect density, defect arrival rate, and testing status. Learn the guidelines for developing a test measurement program, rules of thumb for collecting data, and ways to avoid “metrics dysfunction.” Rick identifies several metrics paradigms and discusses the pros and cons of each. Delegates are urged to bring their metrics problems and issues for use as discussion points.

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
250
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
18
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Measurement and Metrics for Test Managers"

  1. 1. MI AM Tutorial 9/30/2013 8:30:00 AM "Measurement and Metrics for Test Managers" Presented by: Rick Craig Software Quality Engineering Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
  2. 2. Rick Craig Software Quality Engineering A consultant, lecturer, author, and test manager, Rick Craig has led numerous teams of testers on both large and small projects. In his twenty-five years of consulting worldwide, Rick has advised and supported a diverse group of organizations on many testing and test management issues. From large insurance providers and telecommunications companies to smaller software services companies, he has mentored senior software managers and helped test teams improve their effectiveness.
  3. 3. © 2013 SQE Training V3.1 1
  4. 4. © 2013 SQE Training V3.1 2
  5. 5. Notice of Rights Entire contents © 2013 by SQE Training, unless otherwise noted on specific items. All rights reserved. No material in this publication may be reproduced in any form without the express written permission of SQE Training Home Office SQE Training 330 Corporate Way, Suite 300 Orange Park, FL 32073 U.S.A. (904) 278-0524 (904) 278-4380 fax www.sqetraining.com Notice of Liability The information provided in this book is distributed on an “as is” basis, without warranty. Neither the author nor SQE Training shall have any liability to any person or entity with respect to any loss or damage caused or alleged to have been caused directly or indirectly by the content provided in this course. © 2013 SQE Training V3.1 3
  6. 6. Please set any pagers or cellular phones to mute. © 2013 SQE Training V3.1 4
  7. 7. © 2013 SQE Training V3.1 5
  8. 8. © 2013 SQE Training V3.1 6
  9. 9. © 2013 SQE Training V3.1 7
  10. 10. © 2013 SQE Training V3.1 8
  11. 11. Lord Kelvin was apparently a virtual sound-bite machine. There are entire Web sites devoted to him and to his quotes. © 2013 SQE Training V3.1 9
  12. 12. © 2013 SQE Training V3.1 10
  13. 13. Rick Craig and Bill Hetzel, Software Measures and Practices Benchmark Study, 1990. Research Reports TR900-904. SQE, Orange Park, FL © 2013 SQE Training V3.1 11
  14. 14. Rick Craig and Bill Hetzel, Software Measures and Practices Benchmark Study, 1990. Research Reports TR900-904. SQE, Orange Park, FL © 2013 SQE Training V3.1 12
  15. 15. Measure: a quantified observation about any aspect of software Metric: a measure used to compare two or more products, processes, or projects Meter: a metric that acts as a trigger or threshold. That is, if some threshold is met, then an action is warranted (e.g., exit criteria) Meta-measure: a measure of a measure. Usually used to measure the effectiveness of a measure (e.g., number of defects discovered per hour of test) © 2013 SQE Training V3.1 13
  16. 16. Simple: a metric that does not include complicated formulas, is intuitive, and easy to understand. Number of severe bugs is relatively simple (if the severity categories are defined). Defect Age (PhAge) is not as simple. Objective: a metric that is measured multiple times with the same result or by different engineers with the same result. The number of test cases executed is more objective than customer satisfaction. Easily collected: metrics that are automatic (LOC is the only one I can think of) or done as a by-product of a task we must do anyway. For example, we collect information about defects so that we can fix them; therefore, it is relatively simple to analyze the trends and patterns of the defects from the defect reporting system. Robust: metrics that can be used to compare across multiple projects and/or organizations. There are no perfectly robust metrics, but some are more robust than others. For example, the number of engineers on a project is much more robust than “quality.” Valid: Not all metrics are valid. In fact, some are totally incorrect, and others may be “off” just a little. We test a metric’s validity by comparing it to another metric. For example, for test effectiveness we may measure coverage and DDP. If the coverage is high but the DDP is low, we may suspect the validity (or at least the value) of the coverage metric or the DDP. © 2013 SQE Training V3.1 14
  17. 17. Characteristics of metrics and their use: •Broadly applicable •Highly visible •Consistently applied •Management supported •Organizational acceptance •Not undermined by politics •Clear identification of responsibility •Historical data available •Must relate to current practices •Facilitate process improvement •Limit the number of metrics •Easily calculated •Readily available •Precisely defined •Tool support Adapted from Dan Paulish and K. H. Moller, Software Metrics. IEEE Press, 1993 © 2013 SQE Training V3.1 15
  18. 18. © 2013 SQE Training V3.1 16
  19. 19. © 2013 SQE Training V3.1 17
  20. 20. Have you ever had to collect or submit a metric or report that you knew no one would ever see? Did you (or at least were you tempted) to put down just any old value so you could get back to doing something important? Congratulations, you too, have falsified a metric. Similarly, have you ever been asked to do something that you thought was a waste of time and so you procrastinated in the hope that it would just go away (and sometimes it does!)? This is an example of lack of buy-in. When Bill Hetzel and Rick Craig conducted metrics research in the 1990s, they learned that one of the main reasons for failure of metrics programs was lack of buy-in by the people asked to collect the metrics. Later Bill went on to develop the “practitioner paradigm of software metrics” based upon the premise that the practitioners themselves should decide what metrics to collect. Interestingly enough, in most teams, the practitioners volunteered to collect most of the metrics that the project or test manager wanted—and since it was their idea, buy-in was not an issue. © 2013 SQE Training V3.1 18
  21. 21. © 2013 SQE Training V3.1 19
  22. 22. Unless you’re blessed with a “silver tongue,” buy-in is far from easy and training helps. Metrics can be a powerful force in obtaining buy-in. If an industry metric shows that 90% of the “best” projects in your industry use inspections*, but they are not used in your organization, that is a powerful incentive to implement inspections. Presenting this information to upper management will almost always command their attention and assistance in obtaining buy-in. * 90% is obviously not really a meaningful value, just an example. © 2013 SQE Training V3.1 20
  23. 23. © 2013 SQE Training V3.1 21
  24. 24. One of the biggest problems with metrics is buy-in. If practitioners feel that the metrics are not being used—or worse, are being used as a “grade card”—they will often refuse to collect the data and/or even falsify the results! © 2013 SQE Training V3.1 22
  25. 25. The Hawthorne Studies were conducted by Harvard Business School professor Elton Mayo from 1927 to 1932 at the Western Electric Hawthorne Works in Chicago IL. Originally, the researchers were interested in determining what (if any) affect light had on worker productivity. After their initial experiments indicated that there was no clear connection between productivity and the amount of illumination, the researchers began to wonder what other factors might increase productivity. The researchers found that their concern for the workers made the workers feel that they were part of a team and subsequently productivity improved. Please let us offer an alternate “Hawthorne Effect”: When you collect metrics that involve people, it will change the way they behave—but not always for the better. What do you think would happen if you “graded” the testers on the number of test cases written? © 2013 SQE Training V3.1 23
  26. 26. © 2013 SQE Training V3.1 24
  27. 27. © 2013 SQE Training V3.1 25
  28. 28. The numbers on the graph above were made up so please don’t Google them. I do remember seeing a chart something like this in the USA Today a few years ago (you know the little charts on each section of the paper). The scale was skewed to make a point. Take a look at the next slide. Note: Everyone taking this class should read How to Lie with Statistics by Darrell Huff. W.W. Norton, Co. 1954 (enjoyable read) Another good reference is The Visual Display of Quantitative Information by Edward R. Tufte. © 2013 SQE Training V3.1 26
  29. 29. © 2013 SQE Training V3.1 27
  30. 30. © 2013 SQE Training V3.1 28
  31. 31. Use a metric to validate a metric: Virtually anything that is important to measure should have two or more metrics. For example, to measure test effectiveness you might choose to use a coverage metric and DDP. If the two metrics agree, you have some confidence in the measurements. If they don’t, you have some more research to do. Consistency sometimes trumps accuracy: All metrics are flawed, but some are useful anyway. For example, suppose your speedometer consistently registers 10 MPH faster than you are actually traveling; as long as you know this, you still can use your speedometer to measure your speed. © 2013 SQE Training V3.1 29
  32. 32. More is not always better/All metrics are not forever: We have visited companies that have entire teams that collect, slice, and dice hundreds of metrics. But no one looks at them. Just because you can measure something doesn’t mean it is worth the effort. Once or twice a year, all metrics should be reviewed to see if they are still valid and useful. It is demoralizing to collect metrics that are never used. © 2013 SQE Training V3.1 30
  33. 33. This page left blank © 2013 SQE Training V3.1 31
  34. 34. © 2013 SQE Training V3.1 32
  35. 35. The well-designed dashboard of a 1958 Corvette © 2013 SQE Training V3.1 33
  36. 36. Fun fact: The business dashboard originated at General Electric many years ago. — Greg Brue, Six Sigma for Managers, Briefcase Books, 1976 An “instrument” refers to a topic such as Quality of product e.g. the dashboard above has 5 instruments. © 2013 SQE Training V3.1 34
  37. 37. © 2013 SQE Training V3.1 35
  38. 38. Quality is the totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs. —ISO 8402 © 2013 SQE Training V3.1 36
  39. 39. These quality characteristics are sometimes called the “-ility” factors. These factors are very appealing on first blush, but it is often problematic to come up with practical ways to measure them. © 2013 SQE Training V3.1 37
  40. 40. What does the chart above say about the relative risk of each module?? One of the ISTQB principles of test is called defect clustering. Modules that have had a lot of defects in earlier releases or in previous levels of test are more likely to have defects in the future. If you have ever been a maintenance programmer, you have no doubt marveled that the same function, module, program, etc., always seems to be same one that breaks. © 2013 SQE Training V3.1 38
  41. 41. Defect Density: a metric that compares the number of defects to a measure of size (e.g., defects per KLOC) — Rick Craig and Stefan Jaskiel, Systematic Software Testing. Artech House, 2002 © 2013 SQE Training V3.1 39
  42. 42. As the complexity of a product increases, so too does the probability of shipping defects. © 2013 SQE Training V3.1 40
  43. 43. © 2013 SQE Training V3.1 41
  44. 44. © 2013 SQE Training V3.1 42
  45. 45. © 2013 SQE Training V3.1 43
  46. 46. © 2013 SQE Training V3.1 44
  47. 47. Just because you’ve run 90% of the test cases (raw data) doesn’t mean that you are 90% done vs. risk, coverage, execution effort or time to execute. © 2013 SQE Training V3.1 45
  48. 48. Does a report like this help? © 2013 SQE Training V3.1 46
  49. 49. © 2013 SQE Training V3.1 47
  50. 50. © 2013 SQE Training V3.1 48
  51. 51. Many organizations use the “defect arrival rate” as an impartial measure to assist them in predicting when a product will be ready to release. When the defect arrival rate begins to drop, it is often assumed (sometimes correctly) that the software is ready to be released. While a declining arrival rate is typically a good sign, remember that other forces (less effort, no new test cases, etc.) may cause the arrival rate to dip. This is why it is normally a good idea to base important decisions on more than one supporting metric. NOTE: The defects may need to be “weighted.” © 2013 SQE Training V3.1 49
  52. 52. “The defect that is prevented doesn’t need repair, examination, or explanation. The first step is to examine and adopt the attitude of defect prevention. This attitude is called, symbolically, Zero Defects.” — Philip Crosby: Quality is Free (1979). The cost to correct production bugs is many times more than bugs discovered earlier in the lifecycle. In some systems the factor may be 10; in others it may be 1,000 or more. A landmark study done by TRW, IBM, and Rockwell in 1974 showed that a requirements bug found in production cost on average 100+ times more than one discovered at the beginning of the lifecycle. Interestingly enough, the graph above was created 20 years later with almost the same result. © 2013 SQE Training V3.1 50
  53. 53. © 2013 SQE Training V3.1 51
  54. 54. The purpose of the Test Summary Report is for the test manager to report on the results of the testing. An added benefit, though, can be its collection and analysis of metrics. © 2013 SQE Training V3.1 52
  55. 55. © 2013 SQE Training V3.1 53
  56. 56. © 2013 SQE Training V3.1 54
  57. 57. © 2013 SQE Training V3.1 55
  58. 58. © 2013 SQE Training V3.1 56
  59. 59. © 2013 SQE Training V3.1 57
  60. 60. © 2013 SQE Training V3.1 58
  61. 61. This is an experience Rick Craig had at his restaurant many years ago: My head server decided to create a customer satisfaction survey (on her own initiative). You have to love employees like that! The survey had two sections: one rated the quality of the food as Outstanding, Excellent, Above Average, Average, and Below Average; and the other section rated service on a scale of Outstanding, Excellent, and Above Average. I asked the server who created the survey about the missing Average and Below Average categories, and she assured me that as long as she was in charge, no one would ever get average or below average service! I realized that the survey designer’s personal bias can (and will) significantly influence the survey results! © 2013 SQE Training V3.1 59
  62. 62. © 2013 SQE Training V3.1 60
  63. 63. © 2013 SQE Training V3.1 61
  64. 64. Bug drawing by Catherine Sabourin. From I am a Bug by Rob Sabourin. Some of the problems associated with counting defects: •When should you start counting defects? •Start of development •Completion of the work product •Formal configuration management •Start of test •Which activities are considered “defect finding”? •Inspections •Test executions •Test design and development •Informal review •Special analysis •What about omissions? •Forgotten requirements •Omitted features •How should you treat severity and impact? •How should you treat confusion and dissatisfaction? TIP: Identifying defect clusters by programmer name can be politically dangerous; a safer approach is to associate defects with team. © 2013 SQE Training V3.1 62
  65. 65. Defect Removal Efficiency (DRE) is a metric similar to DDP. DDP counts the bugs found (whether fixed or not). DRE only counts the bugs that are found and removed. © 2013 SQE Training V3.1 63
  66. 66. © 2013 SQE Training V3.1 64
  67. 67. This graph is just an example. The actual hours required to fix a bug will vary widely from company to company due to complexity of the software, maturity of the organization, automation, etc. © 2013 SQE Training V3.1 65
  68. 68. Typically, the longer a defect exists before discovery, the more it costs to fix. One metric often collected for a defect is called the Phase Age (or PhAge). EXAMPLE: A requirement defect discovered during a high-level design review would be assigned a PhAge of 1. Had this defect not been found until the pilot, it would have been assigned a PhAge of 8. © 2013 SQE Training V3.1 66
  69. 69. Latent defect: an existing defect that has not yet caused a failure because the exact set of conditions has not been met Masked defect: an existing defect that has not yet caused a failure because another defect has prevented that part of the code from being executed © 2013 SQE Training V3.1 67
  70. 70. © 2013 SQE Training V3.1 68
  71. 71. © 2013 SQE Training V3.1 69
  72. 72. Most test methodologies require (at a minimum) a measure of requirements coverage. A requirements traceability matrix ensures that every requirement has been addressed. It does not ensure, however, that each requirement has been fully tested. It also is only as good as the underlying requirements and design documents. NOTE: One test case may address multiple requirements. The same philosophy (and even the same matrix) can show coverage of the design attributes. © 2013 SQE Training V3.1 70
  73. 73. © 2013 SQE Training V3.1 71
  74. 74. The table above is a conceptual model of the output of a code coverage tool. These tools can measure statement, decision, or path coverage. NOTE: A high level of code coverage does not necessarily ensure that a system has been well tested because the code may not address all of the requirements and the uncovered code may be the most risky. © 2013 SQE Training V3.1 72
  75. 75. It is our opinion that code coverage is a metric that is most useful at the developmental test level(s)—unit and/or integration. Code coverage tools have been available for decades but are enjoying a resurgence do to the increasing popularity of agile methods. © 2013 SQE Training V3.1 73
  76. 76. © 2013 SQE Training V3.1 74
  77. 77. © 2013 SQE Training V3.1 75
  78. 78. © 2013 SQE Training V3.1 76
  79. 79. © 2013 SQE Training V3.1 77
  80. 80. © 2013 SQE Training V3.1 78
  81. 81. © 2013 SQE Training V3.1 79
  82. 82. © 2013 SQE Training V3.1 80
  83. 83. Please note that Product Quality is a development metric that is normally measured by the testers. The other instruments are test metrics. Some testers color code the instruments: Red: In trouble Yellow: Keep an eye on it Green: Good to go We have seen dashboards adorned with thermometers (showing status), stoplights, and smiley faces. © 2013 SQE Training V3.1 81
  84. 84. © 2013 SQE Training V3.1 82
  85. 85. © 2013 SQE Training V3.1 83
  86. 86. © 2013 SQE Training V3.1 84
  87. 87. This page left blank © 2013 SQE Training V3.1 85
  88. 88. © 2013 SQE Training V3.1 86
  89. 89. Brooks’s Law: “Adding manpower to a late software project makes it later” According to Brooks himself, the law is an "outrageous oversimplification", but it captures the general rule. Brooks points to two main factors that explain why it works this way: It takes some time for the people added to a project to become productive. Brooks calls this the “ramp up” time. Software projects are complex engineering endeavors, and new workers on the project must first become educated about the work that has preceded them; this education requires diverting resources already working on the project, temporarily diminishing their productivity while the new workers are not yet contributing meaningfully. Each new worker also needs to integrate with a team composed of multiple engineers who must educate the new worker in their area of expertise in the code base, day by day. In addition to reducing the contribution of experienced workers (because of the need to train), new workers may even have negative contributions – for example, possibly introducing bugs that move the project further from completion. Communication overheads increase as the number of people increases. The number of different communication channels increases along with the square of the number of people; doubling the number of people results in four times as many different conversations. Everyone working on the same task needs to keep in sync, so as more people are added they spend more time trying to find out what everyone else is doing. Frederick P. Brooks, Jr., The Mythical Man-Month: Essays on Software Engineering. Addison-Wesley, 1995 © 2013 SQE Training V3.1 87
  90. 90. The above information comes from the ISTQB. If you are going to gain credibility in the software development arena, you must be able to accurately estimate time and resources. It’s important that the schedule reflect how the estimates for the milestones are determined. For example, when the time schedule is very aggressive, estimating becomes even more critical, so that the planning risks, contingencies, and test priorities can be specified. Recording schedules based on estimates also provide the test manager with an audit trail of how the estimates did—and did not—come to pass and forms the basis for better estimating in the future. Test estimation should include all activities involved in the test process: • Test planning and control • Test analysis and design • Test implementation and execution • Test evaluation and reporting • Test closure activities A common practice is also to estimate the number of test cases required. Assumptions made during estimation should always be documented as part of the estimation. © 2013 SQE Training V3.1 88
  91. 91. The above information comes from the ISTQB. Other factors: Complexity of the process, technology, and organization Significant ramp up, training, and orientation needs Assimilation or development of new tools, techniques, custom hardware, number of testware Requirements for a high degree of detailed test specification Complex timing of component arrival Fragile test data © 2013 SQE Training V3.1 89
  92. 92. The above information comes from the ISTQB. A common practice is also to estimate the number of test cases required. Assumptions made during estimation should always be documented as part of the estimation. In most cases, the estimate, once prepared must be delivered to management, along with a justification. Frequently, some negotiation ensues, often resulting in a rework of the estimate. Ideally, the final estimate represents the best-possible balance of organizational and project goals in the areas of quality, schedule budget, and features (scope). © 2013 SQE Training V3.1 90
  93. 93. These figures (both on the slide and in the notes) are for the software industry as a whole, not specifically testing, but I think it would be reasonable to assume that testers face similar challenges. © 2013 SQE Training V3.1 91
  94. 94. Estimation of software and system engineering has long been known to be fraught with difficulties, both technical and political, though project management best practices for estimation are well established. Test estimation is the application of these best practices to the testing activities associated with a project or operation. © 2013 SQE Training V3.1 92
  95. 95. John McGarry, David Card, Cheryl Jones, Beth Layman, Elizabeth Clark, Joseph Dean, Fred Hall. Practical Software Measurement. Addison-Wesley, 2001 In 1986, Professors S. D. Conte, H. E. Dunsmore, and V. Y. Shen (Conte Dunsmore, Shen 1986 Software Engineering Metrics and Models) proposed that a good estimate approach should provide estimates that are within 25% of the actual results 75% of the time. This evaluation standard is the most common standard used to evaluate estimation accuracy (Richard Stuzke, Estimating Software-Intensive Systems., Addison-Wesley, 2005) © 2013 SQE Training V3.1 93
  96. 96. © 2013 SQE Training V3.1 94
  97. 97. The cone of uncertainty helps explain why some of our early estimates are so bad. Using the diagram above, you can see that at project inception our estimates could be off by a factor of 4 above or a factor of 4 below the estimate. As the project progresses, we are learning more about the project and can refine the estimate based on our newly gained knowledge. This is why one of Rick Craig’s estimation tips is not to spend a ton of time on an estimate. Spending more time deliberating with the same information nets more or less the same estimate. With this in mind, another tip is to constantly re-estimate as our knowledge of the project improves and to use planning risks and contingencies to agree on what to do if the estimate is particularly bad. Note: The Cone of Uncertainly was first defined by Barry Boehm more than twenty-five years ago. © 2013 SQE Training V3.1 95
  98. 98. © 2013 SQE Training V3.1 96
  99. 99. Anecdote by Rick Craig: I confess that I misunderstood the “cone of uncertainty” for years. Perhaps because I live in Florida, whenever I heard the term cone of uncertainty, I immediately thought of the forecast cones used by meteorologists to describe the potential path of a hurricane. The forecaster can easily predict where the hurricane will be in the next few hours, but the farther they project into the future, the greater potential there is for error in the forecast. Similarly in the chart above, the upper line shows how a project progresses if everything is nearly perfect (remember this is an anecdote). The middle line represents the estimated path, and the bottom line indicates what happens (all too often) when everything goes wrong. Amusingly enough, this graph which I will jokingly call the “testing” track forecast cone (after the NHS track forecast cone) still makes the same point: estimates that are made early in the project and are not refined can be off by a factor of x (OK, I’m much less precise than Mr. Boehm) above or below the projected track, and the inaccuracy will increase the farther the projection is made into the future. Another way to look at it is to compare it to the path of a bullet (now my Marine Corps roots are showing). If the tip of a gun barrel is off by even a fraction of an inch, the bullet will miss its target by many inches or feet when it lands down range . The longer the range the greater the potential error. © 2013 SQE Training V3.1 97
  100. 100. Manager: When will you be done testing? Tester: Oh, in about four weeks. Manager: Any chance it might take five weeks? Tester: Sure, it might take that long. Manager: Any chance it might actually take eight weeks? Tester: (getting very irritated) Well, I suppose if we get another crappy build like the last one it might. Manager: You’re obviously not a very good estimator. First, you told me it would take four weeks; now you admit it might take up to double that. Any chance you missed it on the downside? Can you deliver it in three weeks? Tester: three. That is so like you. I said four weeks, and now you want it in We call this the fantasy factor. Our managers are always saying “give me your best estimate.” They don’t mean the most accurate they mean if everything were perfect—and in our desire to please, we acquiesce. © 2013 SQE Training V3.1 98
  101. 101. There are a number of factors that can affect a schedule; however, they can all be characterized in one of four categories: •Time •Scope or size •Resources •Quality or risk A change to one of these categories typically requires a counterbalance change to be made to at least one of the others. Unfortunately, many test managers focus on time and do not always consider the other possibilities. © 2013 SQE Training V3.1 99
  102. 102. Most of us have been there. You have estimated that you can test a product in six weeks with four testers. The manager offers to double your staff but insists that you complete the task in three weeks (half the time). This happens, of course, because of pressure to get the product out the door. The problem is exacerbated because time and effort are often measured in the same units (e.g., hours or days). We know that there is a relationship between the schedule and the resources applied. For example, in most cases, ten people can get a task done more quickly than five (although you can probably come up with some examples where ten people actually take longer—like reaching a consensus). Generally as project size grows, productivity drops but not necessarily in a linear fashion. This drop in productivity can be caused by increased complexity, management overhead, Brooks’s Law, etc. © 2013 SQE Training V3.1 100
  103. 103. The list in the slide above was adapted from the ISTQB Advanced Syllabus for Test Management. Organizational history and metrics includes metrics-derived models that estimate the number of defects, the number of test cycles, the number of test cases, each test’s average effort, and the number of regression cycles involved. Industry averages and predictive models include techniques such as: • Test points • Function points • Lines of code • Estimated development effort • Other project parameters © 2013 SQE Training V3.1 101
  104. 104. Read Karl’s entire article “Estimation Safety Tips” at www.stickyminds.com © 2013 SQE Training V3.1 102
  105. 105. * This is tongue in cheek. We are suggesting you not use the words pad or fudge or whatever—but rather “normalize” the value. The more “distant” the knowledge/data used to create an estimate, the less valuable that information is likely to be. Metrics captured from the same project are more valuable than metrics captured from the same department which are more valuable than metrics captured from the same company which are more valuable than metrics captured from the same business sector which are more valuable than metrics captured from the same country which are more valuable than ... global metrics. TIP: The tighter the time scale, the more important the risk assessment. TIP: Remember that it is sometimes necessary to measure resources, size, or quality rather than time. © 2013 SQE Training V3.1 103

×