Leveraging Automated .NET Analysis Techniques to Find Bugs
Outline <ul><li>Examples from real life </li></ul><ul><li>Pattern-based static analysis </li></ul><ul><li>Flow-based stati...
Real Life Example #1 <ul><li>Exceptions are thrown from event handlers  </li></ul><ul><li>This bug is from the .TEST code ...
Points to Note <ul><li>Exception is possibly thrown by code written by third party </li></ul><ul><li>Exception is possibly...
Real Life Example #2 <ul><li>Another example from the old .TEST code base </li></ul><ul><ul><ul><li>~ILReader() </li></ul>...
Points to Note <ul><li>The garbage collector runs in a separate thread </li></ul><ul><li>The objects can get freed in a no...
Pattern-Based Static Analysis <ul><li>Checks for the presence or absence of code patterns </li></ul><ul><li>Eliminates sev...
Flow-Based Static Analysis <ul><li>Deep analysis involving symbolic simulation </li></ul><ul><ul><li>Analyzes a large numb...
Demo – Static Analysis <ul><li>Results from pattern-based static analysis </li></ul><ul><li>Results from flow-based static...
Testing – Main Types <ul><li>Application-level tests – manual </li></ul><ul><li>Application-level tests – automated </li><...
Application-Level Automatic Tests <ul><li>Pros </li></ul><ul><ul><li>Absolutely necessary for most products </li></ul></ul...
Unit Testing <ul><li>Pros </li></ul><ul><ul><li>Tightly coupled to code base—can assert on important internal state </li><...
Application-Hosted Testing <ul><li>Combines the strengths of application level-testing and unit testing </li></ul><ul><li>...
Demo – Application-Hosted Tests <ul><li>Configuring the tests </li></ul><ul><li>Running the tests </li></ul><ul><li>Review...
Other Important Features for Testing <ul><li>Stubs—test-specific and global </li></ul><ul><ul><li>Call Foo instead of Bar ...
Creating Tests <ul><li>Quick creation of simple test suites to get started </li></ul><ul><li>Easier test creation via appl...
Upcoming SlideShare
Loading in …5
×

Beyond Static Analysis: Integrating .NET Static Analysis with Unit Testing and More

999 views

Published on

Learn the strengths and weaknesses of .NET static analysis—and how a comprehensive development testing strategy that also includes unit testing, code review, and runtime error detection can pick up where development testing leaves off.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
999
On SlideShare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Beyond Static Analysis: Integrating .NET Static Analysis with Unit Testing and More

  1. 1. Leveraging Automated .NET Analysis Techniques to Find Bugs
  2. 2. Outline <ul><li>Examples from real life </li></ul><ul><li>Pattern-based static analysis </li></ul><ul><li>Flow-based static analysis </li></ul><ul><li>Testing methodologies </li></ul>
  3. 3. Real Life Example #1 <ul><li>Exceptions are thrown from event handlers </li></ul><ul><li>This bug is from the .TEST code base </li></ul><ul><ul><li>public delegate void ChosenItemChanged( </li></ul></ul><ul><ul><li>IChooserItem selectedItem); </li></ul></ul><ul><ul><li>public event ChosenItemChanged ChosenItemChanged; </li></ul></ul><ul><ul><li>private void lowerListSelectionChanged( </li></ul></ul><ul><ul><li>object sender, System.EventArgs e) </li></ul></ul><ul><li>{ </li></ul><ul><li>if (ChosenItemChanged != null) </li></ul><ul><li>{ </li></ul><ul><li> ChosenItemChanged(SelectedItem); </li></ul><ul><li> } </li></ul><ul><li>} </li></ul>
  4. 4. Points to Note <ul><li>Exception is possibly thrown by code written by third party </li></ul><ul><li>Exception is possibly thrown by code written much later—i.e. after this component is tested and certified </li></ul><ul><li>Exception is possibly thrown from different threads </li></ul><ul><li>Crashes are not easy to reproduce… so hard to debug and fix </li></ul>
  5. 5. Real Life Example #2 <ul><li>Another example from the old .TEST code base </li></ul><ul><ul><ul><li>~ILReader() </li></ul></ul></ul><ul><ul><ul><li>{ </li></ul></ul></ul><ul><ul><ul><li>Dispose(); </li></ul></ul></ul><ul><ul><ul><li>} </li></ul></ul></ul><ul><ul><ul><li>public void Dispose() </li></ul></ul></ul><ul><ul><ul><li>{ </li></ul></ul></ul><ul><ul><ul><li>if (stream != null) stream.Close(); </li></ul></ul></ul><ul><ul><ul><li>stream = null; </li></ul></ul></ul><ul><ul><ul><li>reader = null; </li></ul></ul></ul><ul><ul><ul><li>header = null; </li></ul></ul></ul><ul><ul><ul><li>metaData = null; </li></ul></ul></ul><ul><ul><ul><li>} </li></ul></ul></ul>
  6. 6. Points to Note <ul><li>The garbage collector runs in a separate thread </li></ul><ul><li>The objects can get freed in a non-deterministic order… so managed resources used in the dtor may already be freed </li></ul><ul><li>The bug can involve multiple methods </li></ul><ul><li>Crashes are non-deterministic… so hard to debug and fix </li></ul>
  7. 7. Pattern-Based Static Analysis <ul><li>Checks for the presence or absence of code patterns </li></ul><ul><li>Eliminates several classes of bugs completely—and with very little effort </li></ul><ul><ul><li>Prevents bugs from entering the code base </li></ul></ul><ul><li>Enhances code readability and maintainability </li></ul><ul><ul><li>Source level analysis is critical for this </li></ul></ul><ul><li>Customization of rules </li></ul><ul><ul><li>Parameterization </li></ul></ul><ul><ul><li>Creation of new rules </li></ul></ul><ul><li>Features and strategies for successful deployment </li></ul><ul><ul><li>Nightly runs </li></ul></ul><ul><ul><li>Automatic task assignment </li></ul></ul><ul><ul><li>Suppressions </li></ul></ul>
  8. 8. Flow-Based Static Analysis <ul><li>Deep analysis involving symbolic simulation </li></ul><ul><ul><li>Analyzes a large number of potential execution paths </li></ul></ul><ul><ul><li>Works across methods and across assemblies </li></ul></ul><ul><li>Points out critical bugs that are hard to reproduce and escape normal testing </li></ul><ul><li>Provides details of the path leading to the bug— making it easy to understand and fix </li></ul>
  9. 9. Demo – Static Analysis <ul><li>Results from pattern-based static analysis </li></ul><ul><li>Results from flow-based static analysis </li></ul>
  10. 10. Testing – Main Types <ul><li>Application-level tests – manual </li></ul><ul><li>Application-level tests – automated </li></ul><ul><li>Unit tests (individual class level) </li></ul>
  11. 11. Application-Level Automatic Tests <ul><li>Pros </li></ul><ul><ul><li>Absolutely necessary for most products </li></ul></ul><ul><ul><li>Many manual tests can be automated at this level </li></ul></ul><ul><ul><li>These usually represent the way the product gets used by the end user </li></ul></ul><ul><li>Cons </li></ul><ul><ul><li>Often takes more time to run </li></ul></ul><ul><ul><li>Loosely coupled with the code—assertions have to be based on program outputs </li></ul></ul><ul><ul><li>Failures are hard to debug </li></ul></ul>
  12. 12. Unit Testing <ul><li>Pros </li></ul><ul><ul><li>Tightly coupled to code base—can assert on important internal state </li></ul></ul><ul><ul><li>Efficient—can be run multiple times during development </li></ul></ul><ul><ul><li>Bugs are found early and are easy to fix when they are found </li></ul></ul><ul><li>Cons </li></ul><ul><ul><li>Hard to create and maintain </li></ul></ul><ul><ul><li>Some classes are hard to test; they require the entire application context for creation and proper functioning </li></ul></ul><ul><ul><li>Classes get tested in artificial environments created solely for testing </li></ul></ul>
  13. 13. Application-Hosted Testing <ul><li>Combines the strengths of application level-testing and unit testing </li></ul><ul><li>Industry examples: </li></ul><ul><ul><li>Plugin test frameworks for IDEs </li></ul></ul><ul><ul><li>Host adapters </li></ul></ul><ul><li>Unit tests are kicked off from different points inside the application </li></ul><ul><ul><li>The user controls the point from which tests are run </li></ul></ul><ul><ul><li>Little or no change is made to the code base </li></ul></ul><ul><ul><li>Can be used with any kind of application </li></ul></ul><ul><ul><ul><li>Plugins for MS project, VS, MS Office </li></ul></ul></ul><ul><ul><ul><li>Windows forms </li></ul></ul></ul><ul><ul><ul><li>ASP.NET </li></ul></ul></ul>
  14. 14. Demo – Application-Hosted Tests <ul><li>Configuring the tests </li></ul><ul><li>Running the tests </li></ul><ul><li>Reviewing one test - OpenPicTest </li></ul>
  15. 15. Other Important Features for Testing <ul><li>Stubs—test-specific and global </li></ul><ul><ul><li>Call Foo instead of Bar </li></ul></ul><ul><ul><li>Application level or individual test level </li></ul></ul><ul><li>Automatic task assignment </li></ul><ul><li>Nightly runs to keep tests under control </li></ul><ul><li>Coverage to measure and guide testing </li></ul><ul><li>Flexible data sources </li></ul><ul><ul><li>Pump data through the same test to cover different kinds of data </li></ul></ul>
  16. 16. Creating Tests <ul><li>Quick creation of simple test suites to get started </li></ul><ul><li>Easier test creation via application tracing </li></ul><ul><ul><li>Unit tests via application execution </li></ul></ul><ul><ul><li>Realistic values (example – long strings, correlated integers, etc.) </li></ul></ul><ul><li>Automatic assertion generation (value based, behavior based) </li></ul><ul><li>Good reports/graphs for tracking progress </li></ul>

×