0
Directed Test Generation for Effective Fault Localization   Shay Artzi,  Julian Dolby ,  Frank Tip,  Marco Pistoia   IBM T...
Overview <ul><li>Automatically generate tests for Web applications  </li><ul><li>Synthesize test suite to maximize code co...
Presented at [ISSTA 08] </li></ul><li>Fault localization finds bugs given a set of tests </li><ul><li>Statistical techniqu...
Presented at [ICSE'10] </li></ul><li>Can tune generation of tests to aid fault localization? </li></ul>
Outline <ul><li>Example
Enhanced Fault Localization
Directed Test Generation for Fault Localization
Evaluation
Related Work
Conclusion </li></ul>
Running PHP Example 1  if (isset($_REQUEST['query']) ){ 2  3  switch ($_REQUEST['query']) { 4  case &quot;model&quot;: 5  ...
Running Example – Success 1   if (isset($_REQUEST['query']) ){ 2   3   switch ($_REQUEST['query']) { 4  case &quot;model&q...
Running Example – Failure 1   if (isset($_REQUEST['query']) ){ 2  3   switch ($_REQUEST['query']) { 4  case &quot;model&qu...
Fault Localization <ul><li>Given a  failure , find responsible  bug </li><ul><li>Statistical techniques on set of passing,...
Correlate executed statements with manifestations of bug </li></ul><li>Specific formula:  Ochiai  [Abreu et al, ISDC’06] <...
Fault Localization Example <ul><li>Apply Ochiai to the two example executions </li><ul><li>Correctly find two blameless st...
But cannot isolate fault precisely </li></ul></ul>statement 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 passing 1 1 1 1 1 1 1 1 fa...
<ul>Enhanced Fault Localization </ul><ul><li>Some faults caused by  absence  of code
Not captured by statistics over statements
E.g. don't handle all values from a function (new) </li><ul><li>Extend “statement” for call to include abstraction of result
Refine blame to kind of result
Abstract value, e.g. 0, non-zero, null, non-null, etc </li></ul><li>E.g. missing 'else' or switch case (ICSE 2010) </li><u...
Upcoming SlideShare
Loading in...5
×

ISSTA 2010 Presentation

392

Published on

This is the presentation I gave at the International Symposium on Software Testing and Analysis in 2010

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
392
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • non-conditional statements are handled the same as before conditional statements are assigned the maximum of the suspiciousness of each of their branches
  • Transcript of "ISSTA 2010 Presentation"

    1. 1. Directed Test Generation for Effective Fault Localization Shay Artzi, Julian Dolby , Frank Tip, Marco Pistoia IBM T.J. Watson Research Center ISSTA – July 14, 2010 – Trento, Italy
    2. 2. Overview <ul><li>Automatically generate tests for Web applications </li><ul><li>Synthesize test suite to maximize code coverage
    3. 3. Presented at [ISSTA 08] </li></ul><li>Fault localization finds bugs given a set of tests </li><ul><li>Statistical techniques based on executed statements </li></ul><li>Fault localization can work over generated tests </li><ul><li>Apply localization on the set of tests generated for coverage
    4. 4. Presented at [ICSE'10] </li></ul><li>Can tune generation of tests to aid fault localization? </li></ul>
    5. 5. Outline <ul><li>Example
    6. 6. Enhanced Fault Localization
    7. 7. Directed Test Generation for Fault Localization
    8. 8. Evaluation
    9. 9. Related Work
    10. 10. Conclusion </li></ul>
    11. 11. Running PHP Example 1 if (isset($_REQUEST['query']) ){ 2 3 switch ($_REQUEST['query']) { 4 case &quot;model&quot;: 5 $kf=&quot;model_id&quot;; break; 6 case &quot;make&quot;: 7 $kf=&quot;make_id&quot;; break; 8 } 9 $v =_REQUEST['key']; 10 11 $sql=&quot;SELECT * FROM VEHICLES WHERE &quot; . $kf . &quot; = '” . $v . “'”; 12 $result=mysql_query($sql); 13 $n = mysql_numrows($result); 14 15 } else if (isset($_REQUEST['update']) ) { … }
    12. 12. Running Example – Success 1 if (isset($_REQUEST['query']) ){ 2 3 switch ($_REQUEST['query']) { 4 case &quot;model&quot;: 5 $kf=&quot;model_id&quot;; break; 6 case &quot;make&quot;: 7 $kf=&quot;make_id&quot;; break; 8 } 9 $v =_REQUEST['key']; 10 11 $sql=&quot;SELECT * FROM VEHICLES WHERE &quot; . $kf . &quot; = '” . $v . “'”; 12 $result=mysql_query($sql); 13 $n = mysql_numrows($result); 14 15 } else if (isset($_REQUEST['update']) ) { … } query -> 'make' key -> 'Ford' 'query' is set in request $kf set to 'make_id' SELECT * FROM VEHICLES WHERE make_id = 'Ford'; returns 2 result rows $n is assigned 2
    13. 13. Running Example – Failure 1 if (isset($_REQUEST['query']) ){ 2 3 switch ($_REQUEST['query']) { 4 case &quot;model&quot;: 5 $kf=&quot;model_id&quot;; break; 6 case &quot;make&quot;: 7 $kf=&quot;make_id&quot;; break; 8 } 9 $v =_REQUEST['key']; 10 11 $sql=&quot;SELECT * FROM VEHICLES WHERE &quot; . $kf . &quot; = '” . $v . “'”; 12 $result=mysql_query($sql); 13 $n = mysql_numrows($result); 14 15 } else if (isset($_REQUEST['update']) ) { … } 'query' is set in request switch falls through $kf is never assigned SELECT * FROM VEHICLES WHERE = 'Ford'; returns null due to invalid sql mysql_numrows(): supplied argument is not a valid MySQL result resource query -> 'company' key -> 'Ford'
    14. 14. Fault Localization <ul><li>Given a failure , find responsible bug </li><ul><li>Statistical techniques on set of passing, failing tests
    15. 15. Correlate executed statements with manifestations of bug </li></ul><li>Specific formula: Ochiai [Abreu et al, ISDC’06] </li></ul>
    16. 16. Fault Localization Example <ul><li>Apply Ochiai to the two example executions </li><ul><li>Correctly find two blameless statements
    17. 17. But cannot isolate fault precisely </li></ul></ul>statement 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 passing 1 1 1 1 1 1 1 1 failing 1 1 1 1 1 1 suspiciousness .7 .7 0 0 .7 .7 .7 .7
    18. 18. <ul>Enhanced Fault Localization </ul><ul><li>Some faults caused by absence of code
    19. 19. Not captured by statistics over statements
    20. 20. E.g. don't handle all values from a function (new) </li><ul><li>Extend “statement” for call to include abstraction of result
    21. 21. Refine blame to kind of result
    22. 22. Abstract value, e.g. 0, non-zero, null, non-null, etc </li></ul><li>E.g. missing 'else' or switch case (ICSE 2010) </li><ul><li>Extend conditional and switch with the outcome
    23. 23. Refine blame to specific control flow outcome
    24. 24. if statement s become (s, true) or (s, false)
    25. 25. switch statement s becomes (s, case number taken ) </li></ul></ul>
    26. 26. Enhanced Fault Localization Example <ul><li>Enhanced technique isolates key issues </li><ul><li>Fall through of switch leaves the key field unset
    27. 27. Return of null result on sql error not checked </li></ul></ul>statement 1 (3,2) (3,0) 6 7 9 11 (12,obj) (12,null) 13 passing 1 1 1 1 1 1 1 1 failing 1 1 1 1 1 1 suspiciousness .7 0 1 0 0 .7 .7 0 1 .7
    28. 28. Combined Concrete And Symbolic Execution <ul><li>Executions that explore different paths </li><ul><li>For new executions, find input that forces different path </li></ul><li>Path constraint captured during execution </li><ul><li>Based on conditionals true for the given execution
    29. 29. E.g. failing example run has path constraint: </li></ul><li>Negate portion of constraint, solve for new input </li><ul><li>Multiple choices for what to negate
    30. 30. Direct choices to aid fault localization </li></ul></ul>
    31. 31. <ul>Directed Test Generation </ul><ul><li>If failure occurs, how to localize the fault? </li><ul><li>Assume no relevant test suite is available </li></ul></ul><ul><ul><li>How best to generate tests for fault-localization?
    32. 32. What kind of tests work best given a known failure? </li></ul></ul><ul><li>Intuition: “similar” tests to localize fault </li><ul><li>Minimize differences to narrow possible bug causes </li></ul></ul><ul><ul><li>Investigate “directed” test generation methods </li></ul></ul><ul><ul><ul><li>Based on similarity metrics </li></ul></ul></ul>
    33. 33. Directed Test Generation Example <ul><li>Original failing test:
    34. 34. Negate a tail:
    35. 35. Each new input explores a different execution </li><ul><li>Intuition: most 'similar' execution best to focus on bug
    36. 36. Note second path will yield successful run </li></ul></ul>
    37. 37. Example Generated Execution 1 1 if (isset($_REQUEST['query']) ){ 2 3 switch ($_REQUEST['query']) { 4 case &quot;model&quot;: 5 $kf=&quot;model_id&quot;; break; 6 case &quot;make&quot;: 7 $kf=&quot;make_id&quot;; break; 8 } 9 $v =_REQUEST['key']; 10 11 $sql=&quot;SELECT * FROM VEHICLES WHERE &quot; . $kf . &quot; = '” . $v . “'”; 12 $result=mysql_query($sql); 13 $n = mysql_numrows($result); 14 15 } else if (isset($_REQUEST['update']) ) { … }
    38. 38. Example Generated Execution 2 1 if (isset($_REQUEST['query']) ){ 2 3 switch ($_REQUEST['query']) { 4 case &quot;model&quot;: 5 $kf=&quot;model_id&quot;; break; 6 case &quot;make&quot;: 7 $kf=&quot;make_id&quot;; break; 8 } 9 $v =_REQUEST['key']; 10 11 $sql=&quot;SELECT * FROM VEHICLES WHERE &quot; . $kf . &quot; = '” . $v . “'”; 12 $result=mysql_query($sql); 13 $n = mysql_numrows($result); 14 15 } else if (isset($_REQUEST['update']) ) { … }
    39. 39. Example Generated Execution 3 1 if (isset($_REQUEST['query']) ){ 2 3 switch ($_REQUEST['query']) { 4 case &quot;model&quot;: 5 $kf=&quot;model_id&quot;; break; 6 case &quot;make&quot;: 7 $kf=&quot;make_id&quot;; break; 8 } 9 $v =_REQUEST['key']; 10 11 $sql=&quot;SELECT * FROM VEHICLES WHERE &quot; . $kf . &quot; = '” . $v . “'”; 12 $result=mysql_query($sql); 13 $n = mysql_numrows($result); 14 15 } else if (isset($_REQUEST['update']) ) { … }
    40. 40. Similarity Metrics <ul><li>Intuitively, find similar but different executions </li><ul><li>Ideally find a similar execution without the bug
    41. 41. Similar buggy executions also helpful to localize </li></ul><li>Two heuristics: </li><ul><li>Path constraint similarity (PCS) </li><ul><li>Number of identical path constraint elements </li></ul><li>Input similarity (IS) </li><ul><li>Number of identical input values </li></ul></ul></ul>
    42. 42. Directed Test Generation Example <ul><li>Original failing test:
    43. 43. Generated subsequent executions: </li><ul><li>PCS: 3, IS: 1 </li></ul><ul><li>PCS: 1, IS: 1 </li></ul><ul><li>PCS: 0, IS: 0 </li></ul></ul>
    44. 44. <ul>Evaluation of Directed Test Generation </ul><ul><li>35 manually localized execution faults in four programs
    45. 45. Localize each fault
    46. 46. 100 tests generated using: </li><ul><li>Base : tests generated by original run (maximize branch coverage)
    47. 47. Starting generation from the failing test </li><ul><li>Coverage : tests generated using the original Apollo
    48. 48. PCS : Directed test generation using path-constraint similarity
    49. 49. IS : Directed test generation using input similarity </li></ul></ul><li>Measure effort needed to find the bug </li></ul><ul><ul><li>assuming statements examined in order of decreasing suspiciousness
    50. 50. measure % of well-localized faults </li></ul></ul>
    51. 51. <ul>Directed Test Generation Results </ul>
    52. 52. Related Work <ul><li>Foundations </li><ul><li>Localization: Tarantula [Jones et al, 02] , Ochiai [Abreu et al., 06]
    53. 53. Concolic testing [Sen et al., 05], DART [Gotfroid et al., 05] </li></ul><li>Other directed dynamic, e.g. [Burnim, Sen, 08] </li><ul><li>Aims to improve coverage, not fault localization </li></ul><li>Delta Debugging [Zeller, 99] </li><ul><li>Focuses on minimizing inputs for single executions </li></ul><li>Other fault localization, e.g. [Wang, Roychoudhury, 05] </li><ul><li>Focus on localization for a single failing test input </li></ul></ul>
    54. 54. Conclusions <ul><li>Directed generation strategy can aid localization </li><ul><li>Generate small, effective test suite
    55. 55. Reduce time and number of tests by > 85%
    56. 56. Enhancements needed to fault localization </li></ul></ul>
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×