Quantifying the Value of Static Analysis
Upcoming SlideShare
Loading in...5
×
 

Quantifying the Value of Static Analysis

on

  • 185 views

During the past ten years, static analysis tools have become a vital part of software development for many organizations. However, the question arises, “Can we quantify the benefits of static ...

During the past ten years, static analysis tools have become a vital part of software development for many organizations. However, the question arises, “Can we quantify the benefits of static analysis?” William Oliver presents the results of a Lawrence Livermore National Laboratory study that first measured the cost of finding software defects using formal testing on a system without static analysis; then, they integrated a static analysis tool into the process and, over a period of time, recalculated the cost of finding software defects. Join William as he shares the results of their study and discusses the value and benefits of static testing. Learn how commercial and open source analysis tools can perform sophisticated source code analysis over large code bases. Take back proof that employing static analysis can not only reduce the time and cost of finding defects and their subsequent debugging but ultimately can reduce the number of defects making their way into your releases.

Statistics

Views

Total Views
185
Views on SlideShare
184
Embed Views
1

Actions

Likes
0
Downloads
1
Comments
0

1 Embed 1

http://www.stickyminds.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Perform Static Analysis: Check out code and build with static analysis tool (Usually automated via nightly run after all code has been checked into the repository) Supports continuous integrationAnalyze Defects: Developers review results and mark defects as either False Positives or Defects Needed to be Fixed and sets prioritiesFix Defects: Developers fix defects from the analysis phaseTest Fixes: Developers perform necessary unit tests to verify that the code provides the required functionality. Add Functionality: Developers continue the development process. Code check-in occurs here.
  • Less Wasted Time: By applying static analysis the code is more testable the first time it is delivered to the test team. Allowing the test team to test more functionality early on in the test cycle. Also by fixing structural defects during software development some functional defects get fixed.Allows more time for test case development: Testers can now focus on what they do best, develop test cases.Better Test cases: Testers have more time to add test cases that improve code coverage.
  • Time Per Defect varied considerable among developers from a little over 52 minutes per defect to as low as 3.58 minutes per defect. Attributable to the learning curve.Above number reflect the top three severity levels (Critical, Severe, Error) only

Quantifying the Value of Static Analysis Quantifying the Value of Static Analysis Presentation Transcript

  • Lawrence Livermore National Laboratory Quantifying the Value of Static Analysis Date 5/19/2011 William B. Oliver Lawrence Livermore National Laboratory, P. O. Box 808, Livermore, CA 94551 This work performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344 LLNL-PRES-490136
  • What is Static Analysis  The use of tools during pre-testing to remove structural defects Software Developer Lawrence Livermore National Laboratory Option:UCRL# Option:Additional Information 2
  • What is Static Analysis  Static analysis tools provide an in-depth analysis of source code to find defects. Lawrence Livermore National Laboratory Option:UCRL# Option:Additional Information 3
  • What is Static Analysis  Finds Defects that compilers and traditional testing miss Lawrence Livermore National Laboratory Option:UCRL# Option:Additional Information 4
  • What is Static Analysis  Defect types include but not limited to • Use of uninitialized variables Lawrence Livermore National Laboratory Option:UCRL# Option:Additional Information 5
  • What is Static Analysis  Defect types include but not limited to • Use of uninitialized variables • Memory leaks Lawrence Livermore National Laboratory Option:UCRL# Option:Additional Information 6
  • What is Static Analysis  Defect types include but not limited to • Use of uninitialized variables • Memory leaks • Null Pointer dereferences Lawrence Livermore National Laboratory Option:UCRL# Option:Additional Information 7
  • What is Static Analysis  Defect types include but not limited to • Use of uninitialized variables • Memory leaks • Null Pointer dereferences • Array Bounds Overflows (and many others) Lawrence Livermore National Laboratory Option:UCRL# Option:Additional Information 8
  • Why Incorporate Static Analysis  Static Analysis finds additional defects  Better Code Coverage  Reduced Developer Debug Time  Advanced Tools find Defects Inter-Procedurally  Uncovers structural defects that can cause Functional Defects  Finds defects missed during code reviews/walk thrus Lawrence Livermore National Laboratory Option:UCRL# Option:Additional Information 9
  • Inter-Procedural Analysis File A foo(x) File B bar(y) File C foobar(z) Lawrence Livermore National Laboratory Option:UCRL# Option:Additional Information 10
  • Why Incorporate Static Analysis Enhances Dynamic Testing • Dynamic testing does not generally uncover memory leaks and other structural defects • Static Analysis provides 100 % code coverage • Structural defects such as Array Bounds Overflows can cause Functional Defects “They are best used in combination with traditional dynamic testing techniques, and can even reduce the cost to create and manage test cases for stringent run-time coverage..” Dr. Paul Anderson PhD Lawrence Livermore National Laboratory Option:UCRL# Option:Additional Information 11
  • Structural Defects vs Functional Defects Relate to conformance to the Programming Language rules and syntax SD FD Uninitialized Data Memory/Resource Leaks Array Bounds Overflows Null Pointer Dereferences Lawrence Livermore National Laboratory Option:UCRL# Option:Additional Information 12
  • Structural Defects vs Functional Defects Associated with Features, Performance, Availability etc. SD FD Found During Dynamic Testing Some Causes Include: Solving the wrong problem Code Logic Errors System Integration Issues Lawrence Livermore National Laboratory Option:UCRL# Option:Additional Information 13
  • Uninitialized Data Correct Set of Values Set of Right Answers y z x Random Set of Values Lawrence Livermore National Laboratory Option:UCRL# Option:Additional Information 14
  • Memory/Resource Leaks Lawrence Livermore National Laboratory Option:UCRL# Option:Additional Information 15
  • Array Bounds Overflow Lawrence Livermore National Laboratory Option:UCRL# Option:Additional Information 16
  • NULL Pointer Dereference This memory location contains the address of this memory location Pointer Value of the contents of address pointed to by the pointer Lawrence Livermore National Laboratory Option:UCRL# Option:Additional Information 17
  • NULL Pointer Dereference Pointer = NULL Lawrence Livermore National Laboratory Option:UCRL# Option:Additional Information 18
  • Typical Static Analysis Work Flow Perform Static Analysis Add Features Test Fixes Analyze Defects Fix Defects Lawrence Livermore National Laboratory Option:UCRL# Option:Additional Information 19
  • Advantages for Testers o Less Wasted Time o Allows more time for test case development o Better Test Cases Lawrence Livermore National Laboratory Option:UCRL# Option:Additional Information 20
  • Assumptions About Time to Find Defects One Million Lines of Code Static Analysis 1000 Defects 20 Percent False Positives 800 Valid Defects Time to Run Code Thru Tool Negligable Ten Minutes Per Defect to Triage Dynamic Testing Automated Testing: 1 hour per defect Includes Test Case Development Test Evaluation Test Report Generation Manual Testing: 2 hours per Defect 1000 Defects Lawrence Livermore National Laboratory Option:UCRL# Option:Additional Information 21
  • Automated Testing TD = Total Defects = SD + FD Time = Time to Find SD + Time to find FD TD = 800 + 1000 = 1800 SD Time = (1000 Defects * 10 min/defect)/60min/hour = 166.67 hours Time = 166.67 + 1000 = 1166.67 hours Time/TD = 1166.67/1800 = .65 hours/defect = 39 minutes per defect Lawrence Livermore National Laboratory Option:UCRL# Option:Additional Information 22
  • Test Case: Automated Testing Code Type: Programming Language: Number of Developers: Source Lines of Code Analyzed: Scientific Simulation C++ 4 161,880 Total Number SD found: 528 Total Number SD Analyzed: 190 Number of False Positives: 55 Average Analysis Time/Defect: 8.9 minutes Lawrence Livermore National Laboratory Option:UCRL# Option:Additional Information 23
  • Test Case: Automated Testing TD = Total Defects = SD + FD FD = 297 for dynamic testing and 1 hour per defect TD = 135 + 297 = 432 SD Time = (190 Defects * 8.9 min/defect)/60min/hour = 28 hours Time = 28 + 297 = 325 hours Time/TD = 325/432 = .75 hours/defect = 45 minutes per defect Lawrence Livermore National Laboratory Option:UCRL# Option:Additional Information 24
  • Just For Fun What If All 528 defects were triaged Assuming 28 % False Positive Rate Total Number SD found: 528 Total Number SD Analyzed: 528 Number of False Positives: 148 Average Analysis Time/Defect: 8.9 minutes Estimated number of real defects = 380 Estimated Time = (528 * 8.9) / 60 = 78 hours TD = 380 + 297 = 677 Time = 78 + 297 = 375 hours Time/TD = 375/677 = .55 hours/defect = 33 minutes per defect Lawrence Livermore National Laboratory Option:UCRL# Option:Additional Information 25
  • Manual Testing Code Type: Programming Language: Security Access C# Number of Developers: Total Number SD found: 76 Total Number SD Analyzed: 35 Number of False Positives: 0 Average Analysis Time/Defect: 3.4 minutes Lawrence Livermore National Laboratory Option:UCRL# Option:Additional Information 26
  • Manual Testing FD = 339 for dynamic testing and 5 hours per defect TD = 35 + 339 = 374 SD Time = (35 Defects * 3.4 min/defect)/60min/hour = 2 hours Time = 2 + 1695 = 1697 hours Time/TD = 1697/374 = 4.5 hours/defect Lawrence Livermore National Laboratory Option:UCRL# Option:Additional Information 27
  • Bottom Line Combined with dynamic testing Static Analysis results in finding more Defects And the organization spends less time per defect in the process Lawrence Livermore National Laboratory Option:UCRL# Option:Additional Information 28
  • Summary For Static Analysis the time to Find a defect is less than or equal to 10 minutes Dynamic Testing: Automated: 1 hour per Defect Manual: 4 – 5 hours per Defect Lawrence Livermore National Laboratory Option:UCRL# Option:Additional Information 29
  • New Breed of Tester Perform Static Analysis Analyze Defects Lawrence Livermore National Laboratory Option:UCRL# Option:Additional Information 30
  • Questions??? Lawrence Livermore National Laboratory Option:UCRL# Option:Additional Information 31