Successfully reported this slideshow.

Do Bugs Reside in Complex Code?

3,677 views

Published on

Myths In Software Engineering: Does complex code mean there will be more bugs? We have analyzed a number of bug databases (including Eclipse, Mozilla, and various Microsoft projects) and come to surprising conclusions.

Published in: Technology
  • Be the first to comment

Do Bugs Reside in Complex Code?

  1. 1. Myths in Software Engineering “Bugs Reside in Complex Code” Andreas Zeller Saarland University
  2. 2. Obtaining Data
  3. 3. Models Specs Code Traces Profiles Tests e-mail Bugs Effort Navigation Changes Chats
  4. 4. Models Specs Code Traces Profiles Tests e-mail Bugs Effort Navigation Changes Chats
  5. 5. Bugs Changes
  6. 6. Bugs Changes
  7. 7. Bugs Changes
  8. 8. Bugs Changes
  9. 9. Bugs Changes
  10. 10. Bugs Changes
  11. 11. Map bugs to code locations Bugs Changes
  12. 12. Eclipse Bugs
  13. 13. What is the cause Eclipse Bugs of these errors?
  14. 14. Code complexity Past defects
  15. 15. Code complexity Past defects #Lines #Vars #Classes Metrics #Params #Reads #Writes #Arcs #Blocks McCabe Fan In Fan Out …
  16. 16. Projects researched • Internet Explorer 6 • IIS Server • Windows Process Messaging • DirectX • NetMeeting >1,000,000 Lines of Code
  17. 17. Projects researched >1,000,000 Lines of Code
  18. 18. Projects researched ABCDE >1,000,000 Lines of Code
  19. 19. Do metrics correlate with defect density?
  20. 20. Do metrics correlate with defect density? Project Metrics correlated w/ defects A #Classes and 5 derived B almost all C all except MaxInheritanceDepth D only #Lines E #Functions, #Arcs, Complexity
  21. 21. Do metrics correlate with defect density? Project Metrics correlated w/ defects A #Classes and 5 derived B almost all C all except MaxInheritanceDepth D only #Lines E #Functions, #Arcs, Complexity
  22. 22. Do metrics correlate with defect density? Project Metrics correlated w/ defects A #Classes and 5 derived B almost all C all except MaxInheritanceDepth D only #Lines E #Functions, #Arcs, Complexity
  23. 23. Do metrics correlate with defect density? Project Metrics correlated w/ defects A #Classes and 5 derived B almost all C all except MaxInheritanceDepth D only #Lines E #Functions, #Arcs, Complexity
  24. 24. Do metrics correlate with defect density? Project Metrics correlated w/ defects A #Classes and 5 derived B almost all C all except MaxInheritanceDepth D only #Lines E #Functions, #Arcs, Complexity
  25. 25. Do metrics correlate with defect density? Project Metrics correlated w/ defects A #Classes and 5 derived B almost all C all except MaxInheritanceDepth D only #Lines E #Functions, #Arcs, Complexity
  26. 26. Do metrics correlate with defect density? YES Project Metrics correlated w/ defects A #Classes and 5 derived B almost all C all except MaxInheritanceDepth D only #Lines E #Functions, #Arcs, Complexity
  27. 27. Is there a set of metrics that fits all projects? Project Metrics correlated w/ defects A #Classes and 5 derived B almost all C all except MaxInheritanceDepth D only #Lines E #Functions, #Arcs, Complexity
  28. 28. Is there a set of metrics that fits all projects? NO Project Metrics correlated w/ defects A #Classes and 5 derived B almost all C all except MaxInheritanceDepth D only #Lines E #Functions, #Arcs, Complexity
  29. 29. Can we predict defect-prone modules?
  30. 30. Can we predict defect-prone modules? • Basic idea: Combine metrics
  31. 31. Can we predict defect-prone modules? • Basic idea: Combine metrics • Give most weight to most predictive metrics
  32. 32. Can we predict defect-prone modules? • Basic idea: Combine metrics • Give most weight to most predictive metrics • Successful prediction in all five projects –
  33. 33. Can we predict defect-prone modules? • Basic idea: Combine metrics • Give most weight to most predictive metrics • Successful prediction in all five projects – • – but requires history to calibrate
  34. 34. Assistance
  35. 35. Eclipse Bugs
  36. 36. What is the cause Eclipse Bugs of these errors?
  37. 37. Is it the developers?
  38. 38. Is it the developers? Does experience matter?
  39. 39. Is it the developers? Bug density Does experience correlates with matter? experience!
  40. 40. Is it history?
  41. 41. Is it history? I found lots of bugs here. Will there be more?
  42. 42. Is it history? I found lots of Yes! (But where bugs here. Will did these come there be more? from?)
  43. 43. How about metrics?
  44. 44. How about metrics? Do code metrics correlate with bug density?
  45. 45. How about metrics? Do code metrics Sometimes! correlate with bug density?
  46. 46. Uh. Coverage?
  47. 47. Uh. Coverage? Does test coverage correlate with bug density?
  48. 48. Uh. Coverage? Yes – Does test coverage correlate with bug the more coverage, density? the more bugs!
  49. 49. Ah! Language features?
  50. 50. Ah! Language features? Are gotos harmful?
  51. 51. Ah! Language features? Are gotos No correlation! harmful?
  52. 52. Ok. Problem domain?
  53. 53. Ok. Problem domain? Which tokens do matter?
  54. 54. Ok. Problem domain? Which tokens import • extends • implements do matter?
  55. 55. Eclipse imports Joint work with Adrian Schröter • Tom Zimmermann
  56. 56. Eclipse imports import org.eclipse.jdt.internal.compiler.lookup.*; import org.eclipse.jdt.internal.compiler.*; import org.eclipse.jdt.internal.compiler.ast.*; import org.eclipse.jdt.internal.compiler.util.*; ... import org.eclipse.pde.core.*; import org.eclipse.jface.wizard.*; import org.eclipse.ui.*; Joint work with Adrian Schröter • Tom Zimmermann
  57. 57. Eclipse imports 71% of all components importing compiler show a post-release defect import org.eclipse.jdt.internal.compiler.lookup.*; import org.eclipse.jdt.internal.compiler.*; import org.eclipse.jdt.internal.compiler.ast.*; import org.eclipse.jdt.internal.compiler.util.*; ... import org.eclipse.pde.core.*; import org.eclipse.jface.wizard.*; import org.eclipse.ui.*; Joint work with Adrian Schröter • Tom Zimmermann
  58. 58. Eclipse imports 71% of all components importing compiler show a post-release defect import org.eclipse.jdt.internal.compiler.lookup.*; import org.eclipse.jdt.internal.compiler.*; import org.eclipse.jdt.internal.compiler.ast.*; import org.eclipse.jdt.internal.compiler.util.*; ... import org.eclipse.pde.core.*; import org.eclipse.jface.wizard.*; import org.eclipse.ui.*; 14% of all components importing ui show a post-release defect Joint work with Adrian Schröter • Tom Zimmermann
  59. 59. Mozilla Vulnerabilities
  60. 60. And what else?
  61. 61. And what else? So all of this requires earlier defects, right?
  62. 62. And what else? So all of this requires earlier Yes! defects, right?
  63. 63. And what else?
  64. 64. And what else? But are there universal properties?
  65. 65. And what else? ? But are there universal properties?
  66. 66. Defect sources
  67. 67. Defect sources Coding
  68. 68. Defect sources Coding Quality Assurance
  69. 69. Defect sources Design Coding Quality Assurance
  70. 70. Defect sources Design Requirements Coding Quality Assurance
  71. 71. Defect sources • Which properties Design Requirements should we look at? • Which properties can we look at? Coding Quality Assurance
  72. 72. bug density Plugin.java had 5 failures ) before and one failure after release (``post''). The package contains 43 files (``points'') and encountered 16 failures before and one failure after release; on average each file in this package had 0.609 failures before and 0.022 failures after release (``avg'') Bugs • Fixes • Changes
  73. 73. bug density Plugin.java had 5 failures ) before and one failure after release (``post''). The package contains 43 files (``points'') and encountered 16 failures before and one failure after release; on average each file in this package had 0.609 failures before and 0.022 failures after release (``avg'') Bugs • Fixes • Changes
  74. 74. <?xml version=quot;1.0quot; encoding=quot;UTF-8quot;?> <defects project=quot;eclipsequot; release=quot;3.0quot;> <package name=quot;org.eclipse.core.runtimequot;> <counts> <count id=quot;prequot; value=quot;16quot; avg=quot;0.609quot; points=quot;43quot; max=quot;5quot;> <count id=quot;postquot; value=quot;1quot; avg=quot;0.022quot; points=quot;43quot; max=quot;1quot;> </counts> <compilationunit name=quot;Plugin.javaquot;> <counts> <count id=quot;prequot; value=quot;5quot;> Plugin.java had 5 failures ) before and one failure after <count id=quot;postquot; value=quot;1quot;> release (``post''). The package contains 43 files (``points'') and encountered 16 failures before and one failure after release; on average each file in this package had 0.609 failures before and 0.022 failures after release (``avg'') Bugs • Fixes • Changes
  75. 75. Defect sources • Which properties Design Requirements should we look at? • Which properties can we look at? Coding Quality Assurance
  76. 76. Defect sources • Which properties Design Requirements should we look at? • Which properties can we look at? Coding Quality Assurance
  77. 77. Defect sources • Which properties Design Requirements should we look at? • Which properties can we look at? Coding Quality Assurance

×