ISSTA 2010 Presentation
Upcoming SlideShare
Loading in...5
×
 

ISSTA 2010 Presentation

on

  • 505 views

This is the presentation I gave at the International Symposium on Software Testing and Analysis in 2010

This is the presentation I gave at the International Symposium on Software Testing and Analysis in 2010

Statistics

Views

Total Views
505
Views on SlideShare
504
Embed Views
1

Actions

Likes
0
Downloads
2
Comments
0

1 Embed 1

http://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as OpenOffice

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • non-conditional statements are handled the same as before conditional statements are assigned the maximum of the suspiciousness of each of their branches

ISSTA 2010 Presentation ISSTA 2010 Presentation Presentation Transcript

  • Directed Test Generation for Effective Fault Localization Shay Artzi, Julian Dolby , Frank Tip, Marco Pistoia IBM T.J. Watson Research Center ISSTA – July 14, 2010 – Trento, Italy
  • Overview
    • Automatically generate tests for Web applications
      • Synthesize test suite to maximize code coverage
      • Presented at [ISSTA 08]
    • Fault localization finds bugs given a set of tests
      • Statistical techniques based on executed statements
    • Fault localization can work over generated tests
      • Apply localization on the set of tests generated for coverage
      • Presented at [ICSE'10]
    • Can tune generation of tests to aid fault localization?
  • Outline
    • Example
    • Enhanced Fault Localization
    • Directed Test Generation for Fault Localization
    • Evaluation
    • Related Work
    • Conclusion
  • Running PHP Example 1 if (isset($_REQUEST['query']) ){ 2 3 switch ($_REQUEST['query']) { 4 case "model": 5 $kf="model_id"; break; 6 case "make": 7 $kf="make_id"; break; 8 } 9 $v =_REQUEST['key']; 10 11 $sql="SELECT * FROM VEHICLES WHERE " . $kf . " = '” . $v . “'”; 12 $result=mysql_query($sql); 13 $n = mysql_numrows($result); 14 15 } else if (isset($_REQUEST['update']) ) { … }
  • Running Example – Success 1 if (isset($_REQUEST['query']) ){ 2 3 switch ($_REQUEST['query']) { 4 case "model": 5 $kf="model_id"; break; 6 case "make": 7 $kf="make_id"; break; 8 } 9 $v =_REQUEST['key']; 10 11 $sql="SELECT * FROM VEHICLES WHERE " . $kf . " = '” . $v . “'”; 12 $result=mysql_query($sql); 13 $n = mysql_numrows($result); 14 15 } else if (isset($_REQUEST['update']) ) { … } query -> 'make' key -> 'Ford' 'query' is set in request $kf set to 'make_id' SELECT * FROM VEHICLES WHERE make_id = 'Ford'; returns 2 result rows $n is assigned 2
  • Running Example – Failure 1 if (isset($_REQUEST['query']) ){ 2 3 switch ($_REQUEST['query']) { 4 case "model": 5 $kf="model_id"; break; 6 case "make": 7 $kf="make_id"; break; 8 } 9 $v =_REQUEST['key']; 10 11 $sql="SELECT * FROM VEHICLES WHERE " . $kf . " = '” . $v . “'”; 12 $result=mysql_query($sql); 13 $n = mysql_numrows($result); 14 15 } else if (isset($_REQUEST['update']) ) { … } 'query' is set in request switch falls through $kf is never assigned SELECT * FROM VEHICLES WHERE = 'Ford'; returns null due to invalid sql mysql_numrows(): supplied argument is not a valid MySQL result resource query -> 'company' key -> 'Ford'
  • Fault Localization
    • Given a failure , find responsible bug
      • Statistical techniques on set of passing, failing tests
      • Correlate executed statements with manifestations of bug
    • Specific formula: Ochiai [Abreu et al, ISDC’06]
  • Fault Localization Example
    • Apply Ochiai to the two example executions
      • Correctly find two blameless statements
      • But cannot isolate fault precisely
    statement 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 passing 1 1 1 1 1 1 1 1 failing 1 1 1 1 1 1 suspiciousness .7 .7 0 0 .7 .7 .7 .7
    • Enhanced Fault Localization
    • Some faults caused by absence of code
    • Not captured by statistics over statements
    • E.g. don't handle all values from a function (new)
      • Extend “statement” for call to include abstraction of result
      • Refine blame to kind of result
      • Abstract value, e.g. 0, non-zero, null, non-null, etc
    • E.g. missing 'else' or switch case (ICSE 2010)
      • Extend conditional and switch with the outcome
      • Refine blame to specific control flow outcome
      • if statement s become (s, true) or (s, false)
      • switch statement s becomes (s, case number taken )
  • Enhanced Fault Localization Example
    • Enhanced technique isolates key issues
      • Fall through of switch leaves the key field unset
      • Return of null result on sql error not checked
    statement 1 (3,2) (3,0) 6 7 9 11 (12,obj) (12,null) 13 passing 1 1 1 1 1 1 1 1 failing 1 1 1 1 1 1 suspiciousness .7 0 1 0 0 .7 .7 0 1 .7
  • Combined Concrete And Symbolic Execution
    • Executions that explore different paths
      • For new executions, find input that forces different path
    • Path constraint captured during execution
      • Based on conditionals true for the given execution
      • E.g. failing example run has path constraint:
    • Negate portion of constraint, solve for new input
      • Multiple choices for what to negate
      • Direct choices to aid fault localization
    • Directed Test Generation
    • If failure occurs, how to localize the fault?
      • Assume no relevant test suite is available
      • How best to generate tests for fault-localization?
      • What kind of tests work best given a known failure?
    • Intuition: “similar” tests to localize fault
      • Minimize differences to narrow possible bug causes
      • Investigate “directed” test generation methods
        • Based on similarity metrics
  • Directed Test Generation Example
    • Original failing test:
    • Negate a tail:
    • Each new input explores a different execution
      • Intuition: most 'similar' execution best to focus on bug
      • Note second path will yield successful run
  • Example Generated Execution 1 1 if (isset($_REQUEST['query']) ){ 2 3 switch ($_REQUEST['query']) { 4 case "model": 5 $kf="model_id"; break; 6 case "make": 7 $kf="make_id"; break; 8 } 9 $v =_REQUEST['key']; 10 11 $sql="SELECT * FROM VEHICLES WHERE " . $kf . " = '” . $v . “'”; 12 $result=mysql_query($sql); 13 $n = mysql_numrows($result); 14 15 } else if (isset($_REQUEST['update']) ) { … }
  • Example Generated Execution 2 1 if (isset($_REQUEST['query']) ){ 2 3 switch ($_REQUEST['query']) { 4 case "model": 5 $kf="model_id"; break; 6 case "make": 7 $kf="make_id"; break; 8 } 9 $v =_REQUEST['key']; 10 11 $sql="SELECT * FROM VEHICLES WHERE " . $kf . " = '” . $v . “'”; 12 $result=mysql_query($sql); 13 $n = mysql_numrows($result); 14 15 } else if (isset($_REQUEST['update']) ) { … }
  • Example Generated Execution 3 1 if (isset($_REQUEST['query']) ){ 2 3 switch ($_REQUEST['query']) { 4 case "model": 5 $kf="model_id"; break; 6 case "make": 7 $kf="make_id"; break; 8 } 9 $v =_REQUEST['key']; 10 11 $sql="SELECT * FROM VEHICLES WHERE " . $kf . " = '” . $v . “'”; 12 $result=mysql_query($sql); 13 $n = mysql_numrows($result); 14 15 } else if (isset($_REQUEST['update']) ) { … }
  • Similarity Metrics
    • Intuitively, find similar but different executions
      • Ideally find a similar execution without the bug
      • Similar buggy executions also helpful to localize
    • Two heuristics:
      • Path constraint similarity (PCS)
        • Number of identical path constraint elements
      • Input similarity (IS)
        • Number of identical input values
  • Directed Test Generation Example
    • Original failing test:
    • Generated subsequent executions:
      • PCS: 3, IS: 1
      • PCS: 1, IS: 1
      • PCS: 0, IS: 0
    • Evaluation of Directed Test Generation
    • 35 manually localized execution faults in four programs
    • Localize each fault
    • 100 tests generated using:
      • Base : tests generated by original run (maximize branch coverage)
      • Starting generation from the failing test
        • Coverage : tests generated using the original Apollo
        • PCS : Directed test generation using path-constraint similarity
        • IS : Directed test generation using input similarity
    • Measure effort needed to find the bug
      • assuming statements examined in order of decreasing suspiciousness
      • measure % of well-localized faults
    • Directed Test Generation Results
  • Related Work
    • Foundations
      • Localization: Tarantula [Jones et al, 02] , Ochiai [Abreu et al., 06]
      • Concolic testing [Sen et al., 05], DART [Gotfroid et al., 05]
    • Other directed dynamic, e.g. [Burnim, Sen, 08]
      • Aims to improve coverage, not fault localization
    • Delta Debugging [Zeller, 99]
      • Focuses on minimizing inputs for single executions
    • Other fault localization, e.g. [Wang, Roychoudhury, 05]
      • Focus on localization for a single failing test input
  • Conclusions
    • Directed generation strategy can aid localization
      • Generate small, effective test suite
      • Reduce time and number of tests by > 85%
      • Enhancements needed to fault localization