Do Software Defect Backlogs Really Matter if the Software is Actively Being Used?


Published on

This report concludes that there is no way to know if a software defect backlog matters without further analysis, so it recommends an approach for this further analysis.

The report is available for download here:

To access more Trusted Advisor articles, visit:

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Do Software Defect Backlogs Really Matter if the Software is Actively Being Used?

  1. 1. ©2014 David Consulting Group Page 1 of 6 Do Software Defect Backlogs really matter if the Software is actively being used? March 2014 Scope of this Report The question driving this report assumes that a software defect backlog already exists. Hence, some but perhaps not all defects have been identified and logged. It assumes that the defects described in the software defect backlog are not sufficient to prevent the software from functioning. Hence, this report ignores much of the literature about software defect detection and tracking (which is adequately covered in the Sources listed for interested parties) to focus on the narrower perspective of what, if anything, should be done about those unaddressed defects in the software that have been reported by developers, testers, end-users or code analysis software. The report concludes that there is no way to know if a particular Software Defect Backlog for a particular application or product matters without further analysis. The report recommends an approach for this further analysis. Definitions Defect Nonconformance of a characteristic with specified requirements, or a deficiency in something necessary for an item’s intended, proper use. Latent Defect A defect that is unknown at time of release Patent Defect A defect that is known, reported and logged but not fixed at time of release Defects/function point The average number of defects found per function point Defect Backlog (ideal): The difference between defects created and fixed (may be one or more severities) which is ideally zero at the start and end of each project
  2. 2. ©2014 David Consulting Group Page 2 of 6 Defect Backlog (realistic): New enhancement projects tend to inherit defects in exisitng code and to close without fixing all of the defect sthat they created (as shown in the first chart below). This results in an increase in the defect backlog over the ourse of the project (as shown in the secong chart below). Defect Severity Classification The actual terminologies, and their meaning, can vary depending on people, projects, organizations, or defect tracking tools, but the following is a normally accepted classification: Name label (Typical) Definition (Typical) Example Severity label (Typical) Critical: The defect affects critical functionality or critical data. It does not have a workaround. Unsuccessful installation, complete failure of a feature. 1 (or “Sev1”) Major: The defect affects major functionality or major data. It has a workaround but is not obvious and is difficult. A feature is not functional from one module but the task is doable if 10 complicated indirect steps are followed in another module/s. 2 Minor: The defect affects minor functionality or non-critical data. It has an easy workaround. A minor feature that is not functional in one module but the same task is easily doable from another module. 3 Trivial: The defect does not affect functionality or data. It does not even need a workaround. It does not impact productivity or efficiency. It is merely an inconvenience. Petty layout discrepancies, spelling/grammatical errors. 4
  3. 3. ©2014 David Consulting Group Page 3 of 6 Defect Management Practices To quote Steven Thomas, “In fact defects is an area where software delivery, including agile software development, can get messy.” Thomas goes on to list nine common strategies for managing a growing fault list: 1. Do nothing - Who cares that the product backlog is large? Product Owner? Developers? Project Manager? If nobody cares then it isn’t a problem. 2. Filter out the noise – Why do they care? Can some be labeled as “low impact”? 3. Stop logging low impact defects – If there is a well understood definition of “low impact” then don’t even log them to avoid wasting time discussing them in future. 4. Close off low impact defects – get rid of the “low impacts” already on the list 5. Prune the backlog – every so often remove defects from the backlog that have “aged out” and are no longer relevant. 6. Batch the defects - batch related defects together. They might be high or low impact individually but in combination their overall value will increase. For example you might group together a set of “email” defects and get the product owner to prioritize these groupings rather than the individual items. 7. Blitz the defects - A common tactic is to blitz the defects. That means allocating the whole team to fixing defects for a sprint. This often happens immediately before a major launch. Incidentally, we have seen organizations set up special defect blitz events (maybe out of hours) where they pay developers for the number of defects fixed. This does NOT generate the desired behavior! 8. Fix some every sprint - It is also common to allocate a certain proportion of effort to fixing faults each sprint. Thomas advocates giving the technical team time each sprint to fix technical debt and fixing faults can be rolled into that. 9. Automate the tests – Frankly, this is such a fundamental requirement in today’s software development that the fact that Thomas left it to last says more about the overall attitude of developers to testing. Thomas’s list is more useful and instructive. Useful in that the tips provided are worthwhile, common strategies and instructive in that we would argue that the first five approaches fall into the “don’t bother me with defects, I’m a developer” category. We will dig deeper into strategy #6 when we consider best practices. How many defects are too many? For the answer to this question, we recommend reference to an internal or external benchmark for appropriate levels of defects based on the functional size of the application, the technology and process used and the team capability. For example, a 1000 function point application in Java might be expected to have a range from 32 to 61 defects based on industry data benchmarks. DCG is one source of such benchmark data. If for some reason, you cannot get access to benchmark data, Ian Watts offers the following approach: 1. When asked what the threshold for number of defects each sprint team should carry, the first thought is zero. However this is impractical, so I started to think about how we should figure out the size that was appropriate.
  4. 4. ©2014 David Consulting Group Page 4 of 6 2. If you think about agile you will realize the goal is to be able to stop what you are doing at the end of a sprint and ship the product. So given this goal the defect backlog must always remain low enough that your team can fix and verify the defects within a very short period of time. 3. In my case I desired 1 week (5d) was all I would allow them to have. 4. With this data point I now figured out the correct "fix rate" of a typical developer. On our team that is about 1.5 defects per day. 5. The next data point is how long it takes to verify the defects. In one day, 2 testers can likely verify all the defects the developers have fixed. Add to this that all the code the developers fix has to get into a nightly build to be verified and we take the 5day - 1 day to equal the number of days the development team has to fix defects. 6. So with these data points we can now figure out the "Max" backlog our team should have: Max Backlog = (4d) * 1.5(defects per day) * (number of developers) 7. Typical scrum team has 7 people, with 4 developers, so... Max Backlog = 4*1.5*4= 24 defects Defect Management Best Practices In these days of “big data”, it’s important to remember that, “there’s gold in them thar hills!” More prosaically, there could be important information buried in the Software Defect Backlog that could not only reduce its size but lead to defect prevention strategies that could easily pay back the time taken for some causal analysis. Defect Tracking Details for Causal Analysis Effective causal analysis depends on having sufficient information in the defect backlog to be able to identify batches of similar defects, trends or patterns. Our “Defect Tracking Best Practices” source (see below) recommends the following Defect Report Content:  Originator / Origination Date  Defect Description  Sequence of Events  Supporting Data  Defect Lifecycle History  Subsystem or Domain Kan recommends recording the following key attributes:  Activity – The specific activity that exposed the defect. For example, during system test, a defect occurs when one clicks a button to select a printer. The phase is system test but the activity is function test because the defect surfaced by performing a function-test type activity.  Trigger – The environment or condition that had to exist for the defect to surface.  Impact – This refers to the effect the defect had on the customer in the field (or would have had if it had escaped to the field). Causal Analysis 1. Create a “Possible Cause” List or Chart. Gilb and Grady contain examples but the Fishbone diagram form Grady reproduced here is a good illustration:
  5. 5. ©2014 David Consulting Group Page 5 of 6 USER INTERFACE Guidelines Not Followed Different perspectives Lack of Feedback OOPS (Forgotten) Lack of Guidelines Don’t read them No time No Central Location Lack of Resources Int. customers decide based on functionality No process To provide Early feedback Prototype not enough Users not focused On new product Some panels not used as much Tests don’t model Users environment adequately Too busy to do them Some product parts don’t resemble others No good model Corner cases, particular problems Resource limits Too many combinations of features Decided based on Incomplete Customer base Too many details In a hurry Result of Changes Source: Grady 2. Select the Defects to be Analyzed. As Gilb and Graham note, “The defect rates during software development are generally such that most development organizations cannot analyze every defect, nor is it usually worthwhile analyzing them all.” They suggest the following selection strategies (the order is our recommendation):  Group similar or related defects together into subsets. Pick a typical defect from each subset to do causal analysis.  Analyze every severity 1 defect and a balanced subset of other defects. Do not analyze severity 4 defects.  Select every fifth Major defect. Eliminate any obvious duplicates but make a note that that particular type of defect occurred multiple times. Do not analyze Minor defects. 3. Convene a root cause analysis meeting. Two or more heads are always better than one on this. Involving representative developers facilitates learning for all parties. 4. Capture findings in code inspection and development checklist updates helps to ensure that today’s mistakes are not repeated tomorrow. Static Analysis Tools We agree with Todd Landry’s assertion that a static analysis of your code is a great way to help categorize and prioritize defects in the code. When you run a static analysis product on your code base for the very first time there is a good chance the first analysis is going to list a large number of defects, some that are without question real and on your defect backlog, some that are real and not on your defect backlog and some that perhaps are not real at all. Do not freak out! This is the first time that analysis engine has ‘laid eyes’ upon your code and it is going to flex its muscles and show you any weaknesses it believes exist. So how does one deal with this? Landry suggests a few strategies to help:
  6. 6. ©2014 David Consulting Group Page 6 of 6 1. Don’t boil the ocean. Before you even run that first analysis, don’t have a “wouldn’t it be cool” moment, where you decide to turn on every single rule the analysis engine has. There is a reason why static analysis tools haven’t turned on everything. They are showing the most accurate and critical issues first. So unless you have unlimited time and resources, your best bet is to start with a core set of rules and run the analysis based on that set. 2. Baseline your defects. Consider that first analysis your baseline and choose to ‘park’ them for the time being. Chances are the product that the analysis was run on is one that has already been released to the public, and in good working order. Zero out these defects for now, and start to triage them, which leads into strategy #3. 3. This is going to sound pretty obvious, but when it comes to managing your issue backlog start looking at the most critical issues first. These are the ones that are most likely to cause a failure of some sort, so determine if these issues are real, and if so, fix them immediately. Once you’re done with the most critical issues, move to the next level of severity, and continue on that way. 4. Finally, tune your analysis. Any good vendor will allow you to tune your analysis. The benefits of tuning are twofold; 1) you can find code issues that would otherwise go undetected and, 2) reduce the number of issues that the engine reports incorrectly in the context of your source code. You should think of ways to give the tool more “context” about your code base to increase accuracy. Conclusion There is no way to know if a particular Software Defect Backlog for a particular application or product matters without further analysis of the detail of the defect backlog. Ideally, this analysis should be regular and frequent. There is value in both identifying patterns of development or areas of code that generate the most defects and in removing the spurious reporting of low impact defects that waste analysis effort and deter organizations from tackling the really “valuable” and “useful” defects in the backlog. Sources 1. “Successful Software Process Improvement”, Robert B. Grady, Prentice Hall, 1997 2. “Metrics and Models in Software Quality Engineering”, Stephen H. Kan, Addison-Wesley Professional, 2003 3. “Software Inspection”, Tom Gilb & Dorothy Graham, Addison Wesley, 1993 4. “IT Measurement”, International Function Point Users Group (Various Authors), Addison Wesley, 2002 5. “Project Management Terms” by J. Leroy Ward, ESI International, 2000. 6. “Software Testing Fundamentals” Blog post 7. “It’s a Delivery Thing” Blog post by Steven Thomas managing-a-growing-defect-backlog 8. “Kloctalk” blog post by Todd Landry different-type-of-backlog-your-bug-backlog/ 9. “developerWorks” blog post by Ian Watts _how_to_know_when_you_have_to_many14?lang=en 10. “Defect Tracking Best Practices”, Author unknown,