presented @ Factors Relating Field Failures      and DependenciesThomas Zimmermann ● Nachiappan Nagappan ● Kim Herzig     ...
The Value Of A House
The Value Of A House Condition matters!
The Value Of A HouseAnd so does the neighborhood!
What has this to do with     code quality?
What has this to do with     code quality?     Consider software projects as a city           with classes as houses
What has this to do with     code quality?     Consider software projects as a city           with classes as houses      ...
What has this to do with     code quality?     Consider software projects as a city           with classes as houses      ...
The Quality Of Source Code   /**     * Used by {@link #getThreadInformation()} for a traversal search of     * {@link Thre...
The Quality Of Source Code   /**     * Used by {@link #getThreadInformation()} for a traversal search of     * {@link Thre...
Does code neighborhood influence code quality?
What We Need          for each software artifact•   Determine the value (quality)•   Determine the neighborhood•   Determi...
Quality As ValueWe consider failure-proness as a measure of quality.
Quality As Value     We consider failure-proness as a measure of quality.                                               12...
Quality As Value     We consider failure-proness as a measure of quality.                                               12...
The NeighborhoodArtifacts communicate with each other      (functions/methods, variables)                     B
The NeighborhoodArtifacts communicate with each other      (functions/methods, variables)                     B incoming n...
The NeighborhoodArtifacts communicate with each other      (functions/methods, variables)                     B incoming n...
The NeighborhoodArtifacts communicate with each other      (functions/methods, variables)                     B incoming n...
Code-Metrics As Condition           Code metrics are useful to predict bugs.Churn & Size Metrics   LOC, Churn, Churn Frequ...
Code-Metrics As Condition           Code metrics are useful to predict bugs.Churn & Size Metrics   LOC, Churn, Churn Frequ...
Code-Metrics As Condition           Code metrics are useful to predict bugs.Churn & Size Metrics   LOC, Churn, Churn Frequ...
Code-Metrics As Condition           Code metrics are useful to predict bugs.Churn & Size Metrics   LOC, Churn, Churn Frequ...
Code-Metrics As Condition           Code metrics are useful to predict bugs.Churn & Size Metrics   LOC, Churn, Churn Frequ...
Data Collection             Quality: FP or NFP for eachcomponent             Neighborhood metrics (aggregated)
Data Collection               Quality: FP or NFP                 Is there a relation between            quality and neighb...
Experimental Setup  FP  NFP  FP  NFP  FP
Experimental Setup        m = churn_median  FP  NFP  FP  NFP  FP
Experimental Setup        m = churn_median          FP      NFP  FP  NFP  FP  NFP  FP
Experimental Setup        m = churn_median          FP      NFP  FP      400  NFP             330  FP      450  NFP       ...
Experimental Setup         m = churn_median           FP      NFP   FP      400   NFP              330   FP      450   NFP...
Projects
Projectsfunctional & object oriented         object oriented  metrics on binary level         metrics on class level      ...
Projectsfunctional & object oriented         object oriented  metrics on binary level         metrics on class level      ...
Projectsfunctional & object oriented         object oriented  metrics on binary level         metrics on class level      ...
Survey  We asked 700 Microsoft developers      to share their opinion on:How do dependency characteristics  influence the r...
Survey         110 responded (15.7%)  We asked 700 Microsoft developers      to share their opinion on:How do dependency c...
Survey              Binary has a dependency with another that ...     11%6%                       83%     churned a lot   ...
Survey              Binary has a dependency with another that ...                                           1%     11%    ...
Survey              Binary has a dependency with another that ...                                           1%     11%    ...
Results Vista          indicates medians for           FP are significantly         lower than for NFP         indicates me...
Results Vista  Survey
Results Vista  Survey             Developers were right! High           degree of churn increases failure-                ...
Results Vista  Survey           Code coverage does not matter!
Results Vista  Survey           Many users requires better                    testing!
Results Vista  Survey           Depending on bad code does not                       harm!
Results Eclipse                  Survey
Results Eclipse                  Survey                      as expected!
Results Eclipse                  Survey                     depending on                     components                   ...
Results EclipseResults between Vista and EclipseSurvey a lot!                                  differ                     ...
Contradictions!                   Vista        Eclipse       Survey     churn  complexity   coverage                      ...
Factors Relating Field Failures and Dependencies - ICST 2011
Factors Relating Field Failures and Dependencies - ICST 2011
Factors Relating Field Failures and Dependencies - ICST 2011
Factors Relating Field Failures and Dependencies - ICST 2011
Upcoming SlideShare
Loading in...5
×

Factors Relating Field Failures and Dependencies - ICST 2011

439

Published on

Changing source code in large software systems is complex and requires a good understanding of dependencies between software components. Modification to components with little regard to dependencies may have an adverse impact on the quality of the latter, i.e., increase their risk to fail. We conduct an empirical study to understand the relationship between the quality of components and the characteristics of their dependencies such as their frequency of change, their complexity, number of past failures and the like. Our study has been conducted on two large software systems: Microsoft VISTA and ECLIPSE. Our results show that components that have outgoing dependencies to components with higher object-oriented complexity tend to have fewer field failures for VISTA, but the opposite relation holds for ECLIPSE. Likewise, other notable observations have been made through our study that (a) confirm that certain characteristics of components increase the risk of their dependencies to fail and (b) some of the characteristics are project specific while some were also found to be common. We expect that such results can be leveraged for use to provide new directions for research in defect prediction, test prioritization and related research fields that utilize code dependencies in their empirical analysis. Additionally, these results provide insights to engineers on the potential reliability impacts of new component dependencies based upon the characteristics of the component.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
439
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • [stat] Mann Whitney-U test with the significance level was set to .05\n[stat] the Bonferroni correction is more conservative\n(less likely to reject null hypotheses) and thus less likely to\naccidentally accept incorrect result\n[metrics] 44 on Vista, 252 on Eclipse\n
  • [stat] Mann Whitney-U test with the significance level was set to .05\n[stat] the Bonferroni correction is more conservative\n(less likely to reject null hypotheses) and thus less likely to\naccidentally accept incorrect result\n[metrics] 44 on Vista, 252 on Eclipse\n
  • [stat] Mann Whitney-U test with the significance level was set to .05\n[stat] the Bonferroni correction is more conservative\n(less likely to reject null hypotheses) and thus less likely to\naccidentally accept incorrect result\n[metrics] 44 on Vista, 252 on Eclipse\n
  • Factors Relating Field Failures and Dependencies - ICST 2011

    1. 1. presented @ Factors Relating Field Failures and DependenciesThomas Zimmermann ● Nachiappan Nagappan ● Kim Herzig Rahul Premraj ● Laurie Williams
    2. 2. The Value Of A House
    3. 3. The Value Of A House Condition matters!
    4. 4. The Value Of A HouseAnd so does the neighborhood!
    5. 5. What has this to do with code quality?
    6. 6. What has this to do with code quality? Consider software projects as a city with classes as houses
    7. 7. What has this to do with code quality? Consider software projects as a city with classes as houses The value of a class determined by the number of bugs found!
    8. 8. What has this to do with code quality? Consider software projects as a city with classes as houses The value of a class determined by the number of bugs found! How to determine the condition of a class?
    9. 9. The Quality Of Source Code /** * Used by {@link #getThreadInformation()} for a traversal search of * {@link Thread}s/{@link ThreadGroup}s * * @param group * @param level * @return /** */ * Used by {@link #getThreadInformation()} for a traversal search of private String visit(final ThreadGroup group, * {@link Thread}s/{@link ThreadGroup}s final int level) { * // Get threads in `group /** * @param group * @param level StringBuilder builder = new StringBuilder(); #getThreadInformation()} for a traversal search of * Used by {@link * @return int numThreads = group.activeCount(); Thread}s/{@link ThreadGroup}s * {@link Thread[] threads = new Thread[numThreads * 2]; * */ numThreads = group.enumerate(threads, group * @param false); private String visit(final ThreadGroup group, * @param level final int level) { StringBuilder indent = new StringBuilder(); * @return // Get threads in `group for (int i = 0; i < level; ++i) { */ StringBuilder builder = new StringBuilder(); indent.append(" "); private String visit(final ThreadGroup group, group.activeCount(); int numThreads = } final int level) { Thread[] threads = new Thread[numThreads * 2]; // Get threads in `group numThreads = group.enumerate(threads, false); // Enumerate each thread in `group StringBuilder builder = new StringBuilder(); for (int i = 0; i < numThreads; i++) { StringBuilder indent = new StringBuilder(); int numThreads = group.activeCount(); // Get thread Thread[] threads = new Thread[numThreads * i2]; level; ++i) { for (int i = 0; < Thread thread = threads[i]; numThreads = group.enumerate(threads, false); "); indent.append(" builder.append(indent); } builder.append("|-"); StringBuilder indent = new StringBuilder(); builder.append(thread.getName()).append(" ["); level; // Enumerate each thread in `group for (int i = 0; i < ++i) { builder.append(thread.getClass().getSimpleName()).append("], "); = 0; i < numThreads; i++) { indent.append(" "); for (int i builder.append(thread.getPriority()).append(", "); } // Get thread builder.append(thread.getState().name()); Thread thread = threads[i]; builder.append(indent); builder.append(FileUtils.lineSeparator); each thread in `group // Enumerate builder.append("|-"); for (StackTraceElement element : thread.getStackTrace()) { i++) { for (int i = 0; i < numThreads; builder.append(indent); // Get thread builder.append(thread.getName()).append(" ["); builder.append("| "); Thread thread = threads[i]; builder.append(thread.getClass().getSimpleName()).append("], "); builder.append(element.toString()); builder.append(indent); builder.append(thread.getPriority()).append(", "); builder.append(FileUtils.lineSeparator); builder.append("|-"); builder.append(thread.getState().name()); } builder.append(FileUtils.lineSeparator); builder.append(thread.getName()).append(" ["); // builder.append(FileUtils.lineSeparator); for (StackTraceElement element : thread.getStackTrace()) { builder.append(thread.getClass().getSimpleName()).append("], "); } builder.append(indent); builder.append(thread.getPriority()).append(", "); builder.append("| "); builder.append(thread.getState().name()); // Get thread subgroups of `group builder.append(element.toString()); builder.append(FileUtils.lineSeparator); int numGroups = group.activeGroupCount(); builder.append(FileUtils.lineSeparator); for (StackTraceElement element : thread.getStackTrace()) { builder.append(indent); } ThreadGroup[] groups = new ThreadGroup[numGroups * 2]; numGroups = group.enumerate(groups, false); builder.append("| "); // builder.append(FileUtils.lineSeparator); } builder.append(element.toString()); // Recursively visit each subgroup builder.append(FileUtils.lineSeparator); for (int i = 0; i < numGroups; i++) { } // Get thread subgroups of `group builder.append(indent); int numGroups = group.activeGroupCount(); // builder.append(FileUtils.lineSeparator); builder.append(visit(groups[i], level + 1)); } ThreadGroup[] groups = new ThreadGroup[numGroups * 2]; } numGroups = group.enumerate(groups, false); // Get thread subgroups of `group return builder.toString(); int numGroups = group.activeGroupCount(); visit each subgroup // Recursively } ThreadGroup[] groups = new for (int i = 0; i < numGroups; i++) { ThreadGroup[numGroups * 2]; builder.append(indent); numGroups = group.enumerate(groups, false); builder.append(visit(groups[i], level + 1)); } // Recursively visit each subgroup for (int i = 0; i < numGroups; i++) { builder.append(indent); return builder.toString(); } builder.append(visit(groups[i], level + 1)); } return builder.toString(); } Code Metrics
    10. 10. The Quality Of Source Code /** * Used by {@link #getThreadInformation()} for a traversal search of * {@link Thread}s/{@link ThreadGroup}s * * @param group * @param level * @return /** */ * Used by {@link #getThreadInformation()} for a traversal search of private String visit(final ThreadGroup group, * {@link Thread}s/{@link ThreadGroup}s final int level) { * // Get threads in `group /** * @param group * @param level StringBuilder builder = new StringBuilder(); #getThreadInformation()} for a traversal search of * Used by {@link * @return int numThreads = group.activeCount(); Thread}s/{@link ThreadGroup}s * {@link Thread[] threads = new Thread[numThreads * 2]; * */ numThreads = group.enumerate(threads, group * @param false); private String visit(final ThreadGroup group, * @param level final int level) { StringBuilder indent = new StringBuilder(); * @return // Get threads in `group for (int i = 0; i < level; ++i) { */ StringBuilder builder = new StringBuilder(); indent.append(" "); private String visit(final ThreadGroup group, group.activeCount(); int numThreads = } final int level) { Thread[] threads = new Thread[numThreads * 2]; // Get threads in `group numThreads = group.enumerate(threads, false); // Enumerate each thread in `group StringBuilder builder = new StringBuilder(); for (int i = 0; i < numThreads; i++) { StringBuilder indent = new StringBuilder(); int numThreads = group.activeCount(); // Get thread Thread[] threads = new Thread[numThreads * i2]; level; ++i) { for (int i = 0; < Thread thread = threads[i]; numThreads = group.enumerate(threads, false); "); indent.append(" builder.append(indent); } builder.append("|-"); StringBuilder indent = new StringBuilder(); builder.append(thread.getName()).append(" ["); level; // Enumerate each thread in `group for (int i = 0; i < ++i) { builder.append(thread.getClass().getSimpleName()).append("], "); = 0; i < numThreads; i++) { indent.append(" "); for (int i builder.append(thread.getPriority()).append(", "); } // Get thread builder.append(thread.getState().name()); Thread thread = threads[i]; builder.append(indent); builder.append(FileUtils.lineSeparator); each thread in `group // Enumerate builder.append("|-"); for (StackTraceElement element : thread.getStackTrace()) { i++) { for (int i = 0; i < numThreads; builder.append(indent); // Get thread builder.append(thread.getName()).append(" ["); builder.append("| "); Thread thread = threads[i]; builder.append(thread.getClass().getSimpleName()).append("], "); builder.append(element.toString()); builder.append(indent); builder.append(thread.getPriority()).append(", "); builder.append(FileUtils.lineSeparator); builder.append("|-"); builder.append(thread.getState().name()); } builder.append(FileUtils.lineSeparator); builder.append(thread.getName()).append(" ["); // builder.append(FileUtils.lineSeparator); for (StackTraceElement element : thread.getStackTrace()) { builder.append(thread.getClass().getSimpleName()).append("], "); } builder.append(indent); builder.append(thread.getPriority()).append(", "); builder.append("| "); builder.append(thread.getState().name()); // Get thread subgroups of `group builder.append(element.toString()); builder.append(FileUtils.lineSeparator); int numGroups = group.activeGroupCount(); builder.append(FileUtils.lineSeparator); for (StackTraceElement element : thread.getStackTrace()) { builder.append(indent); } ThreadGroup[] groups = new ThreadGroup[numGroups * 2]; numGroups = group.enumerate(groups, false); builder.append("| "); // builder.append(FileUtils.lineSeparator); } builder.append(element.toString()); // Recursively visit each subgroup builder.append(FileUtils.lineSeparator); for (int i = 0; i < numGroups; i++) { } // Get thread subgroups of `group builder.append(indent); int numGroups = group.activeGroupCount(); // builder.append(FileUtils.lineSeparator); builder.append(visit(groups[i], level + 1)); } ThreadGroup[] groups = new ThreadGroup[numGroups * 2]; } numGroups = group.enumerate(groups, false); // Get thread subgroups of `group return builder.toString(); int numGroups = group.activeGroupCount(); visit each subgroup // Recursively } ThreadGroup[] groups = new for (int i = 0; i < numGroups; i++) { ThreadGroup[numGroups * 2]; builder.append(indent); numGroups = group.enumerate(groups, false); builder.append(visit(groups[i], level + 1)); } // Recursively visit each subgroup for (int i = 0; i < numGroups; i++) { builder.append(indent); return builder.toString(); } builder.append(visit(groups[i], level + 1)); } return builder.toString(); } Code Metrics Neighborhood
    11. 11. Does code neighborhood influence code quality?
    12. 12. What We Need for each software artifact• Determine the value (quality)• Determine the neighborhood• Determine the condition (metrics)
    13. 13. Quality As ValueWe consider failure-proness as a measure of quality.
    14. 14. Quality As Value We consider failure-proness as a measure of quality. 12 bugs• determine all post- release bug 1 bugs 0 bugs 2 bugs
    15. 15. Quality As Value We consider failure-proness as a measure of quality. 12 bugs• determine all post- FP release bug 1 bugs FP• failure-prone <=> at 0 bugs least one post- NFP release bug 2 bugs FP
    16. 16. The NeighborhoodArtifacts communicate with each other (functions/methods, variables) B
    17. 17. The NeighborhoodArtifacts communicate with each other (functions/methods, variables) B incoming neighborhood
    18. 18. The NeighborhoodArtifacts communicate with each other (functions/methods, variables) B incoming neighborhood outgoing neighborhood
    19. 19. The NeighborhoodArtifacts communicate with each other (functions/methods, variables) B incoming neighborhood outgoing neighborhood } neighborhood
    20. 20. Code-Metrics As Condition Code metrics are useful to predict bugs.Churn & Size Metrics LOC, Churn, Churn FrequencyCode Metrics FanIn, FanOut Cyclomatic Complexity #methods, inheritance, couplingCode Coverage Block Coverage, Arc CoveragePeople Metrics Org. level, #authors and more ...
    21. 21. Code-Metrics As Condition Code metrics are useful to predict bugs.Churn & Size Metrics LOC, Churn, Churn FrequencyCode Metrics FanIn, FanOut B Cyclomatic Complexity #methods, inheritance, couplingCode Coverage Block Coverage, Arc CoveragePeople Metrics Org. level, #authors and more ...
    22. 22. Code-Metrics As Condition Code metrics are useful to predict bugs.Churn & Size Metrics LOC, Churn, Churn Frequency Compute theCode Metrics metrics for FanIn, FanOut B neighbors Cyclomatic Complexity #methods, inheritance, couplingCode Coverage Block Coverage, Arc CoveragePeople Metrics Org. level, #authors and more ...
    23. 23. Code-Metrics As Condition Code metrics are useful to predict bugs.Churn & Size Metrics LOC, Churn, Churn Frequency Compute theCode Metrics metrics for FanIn, FanOut B neighbors Cyclomatic Complexity #methods, inheritance, couplingCode Coverage Aggregate Block Coverage, Arc Coverage metricsPeople Metrics (med,max,total) Org. level, #authors and more ...
    24. 24. Code-Metrics As Condition Code metrics are useful to predict bugs.Churn & Size Metrics LOC, Churn, Churn Frequency Compute theCode Metrics metrics for FanIn, FanOut B neighbors Cyclomatic Complexity #methods, inheritance, couplingCode Coverage Aggregate Block Coverage, Arc Coverage Assign to B metricsPeople Metrics (med,max,total) Org. level, #authors and more ...
    25. 25. Data Collection Quality: FP or NFP for eachcomponent Neighborhood metrics (aggregated)
    26. 26. Data Collection Quality: FP or NFP Is there a relation between quality and neighborhood condition? for eachcomponent Neighborhood metrics (aggregated)
    27. 27. Experimental Setup FP NFP FP NFP FP
    28. 28. Experimental Setup m = churn_median FP NFP FP NFP FP
    29. 29. Experimental Setup m = churn_median FP NFP FP NFP FP NFP FP
    30. 30. Experimental Setup m = churn_median FP NFP FP 400 NFP 330 FP 450 NFP 405 FP 500
    31. 31. Experimental Setup m = churn_median FP NFP FP 400 NFP 330 FP 450 NFP 405 FP 500 Are the values in FP significantlygreater/lower that values in NFP?
    32. 32. Projects
    33. 33. Projectsfunctional & object oriented object oriented metrics on binary level metrics on class level slightly different set of metrics
    34. 34. Projectsfunctional & object oriented object oriented metrics on binary level metrics on class level slightly different set of metrics
    35. 35. Projectsfunctional & object oriented object oriented metrics on binary level metrics on class level slightly different set of metrics
    36. 36. Survey We asked 700 Microsoft developers to share their opinion on:How do dependency characteristics influence the risk of failures?
    37. 37. Survey 110 responded (15.7%) We asked 700 Microsoft developers to share their opinion on:How do dependency characteristics influence the risk of failures?
    38. 38. Survey Binary has a dependency with another that ... 11%6% 83% churned a lot No Effect Increases Decreases No Opinion risk of failure of binary
    39. 39. Survey Binary has a dependency with another that ... 1% 11% 18%6% 4% 83% 77% churned a lot has high test coverage No Effect Increases Decreases No Opinion risk of failure of binary
    40. 40. Survey Binary has a dependency with another that ... 1% 11% 2% 18%6% 6% 1% 4% 77% 91% 83% churned a lot has high test failed many times in coverage the past No Effect Increases Decreases No Opinion risk of failure of binary
    41. 41. Results Vista indicates medians for FP are significantly lower than for NFP indicates medians for FP are significantly higher than for NFP
    42. 42. Results Vista Survey
    43. 43. Results Vista Survey Developers were right! High degree of churn increases failure- proness.
    44. 44. Results Vista Survey Code coverage does not matter!
    45. 45. Results Vista Survey Many users requires better testing!
    46. 46. Results Vista Survey Depending on bad code does not harm!
    47. 47. Results Eclipse Survey
    48. 48. Results Eclipse Survey as expected!
    49. 49. Results Eclipse Survey depending on components with fewer interfaces more bugs
    50. 50. Results EclipseResults between Vista and EclipseSurvey a lot! differ depending on components with fewer interfaces more bugs
    51. 51. Contradictions! Vista Eclipse Survey churn complexity coverage n/a post-release failuresContradiction raises questions regarding the relationship between dependencies and code quality!

    ×