Are Automated Debugging Techniques Actually Helping Programmers?<br />Chris Parnin<br />    Georgia Tech<br />@chrisparnin...
Finding bugs can be hard…<br />
Automated debugging to the rescue!<br />I’ll help you find location of bug!<br />
How it works (Ranking-Based)<br />I have calculated most likely location of bug!<br />Give me a failing program.<br />Calc...
How it works (Ranking-Based)<br />I have calculated most likely location of bug!<br />Give me input.<br />But how does a p...
Conceptual Model<br />1)<br />2)<br />3)<br />4)<br />…<br />Here is a list of places to check out<br />Ok, I will check o...
Conceptual Model<br />1)<br />2)<br />3)<br />4)<br />…<br />Found the bug!<br />
A Skeptic<br />Does the conceptual model make sense?<br />Have we evaluated it?<br />
Let’s see…<br />Over 50 years of researchonautomated debugging.<br />2001. Statistical Debugging<br />1999. Delta Debuggin...
 Did you see anything?<br />
Only 5 papers have evaluated <br />automated debugging techniques <br />with actual programmers.<br /><ul><li>Most find no...
Most done on programs < 100 LOC</li></li></ul><li>More generally, two points<br />Techniques rely on <br />two strong assu...
Assumption #1: Perfect bug understanding must also exist when using automated tool.<br />Do you see a bug?<br />
Assumption #2<br />Programmer inspects statements linearly and exhaustively until finding bug.<br />Is this realistic?<br />
How can we evaluate the conceptual model?<br />How could we measure the benefit of an automated tool?<br />
Conceptual model: What if gave a developer a list of statements to inspect?<br />How would they use the list?<br />Would t...
Benefit: What if we evaluate programmers with and without automated debuggers?<br />><br />?<br />We also could observe wh...
Study Setup<br />34 Developers<br />2 Debugging Tasks<br />Automated debugging tool<br />
Study Setup<br />Participants:<br />34 developers<br />MS/Phd Students<br />Different levels of expertise 	(low,medium,hig...
Study Setup<br />Software subjects:<br />Tetris (2.5 kloc)<br />NanoXML (4.5 kloc)<br />
Study Setup<br />21<br />Tools:<br />	Traditional debugger <br />	Eclipse ranking plugin<br />(logged activity)<br />
Study Setup<br />Tasks:<br />	Debugging fault<br />	30 minutes per task<br />Questionnaire at end<br />
Bugs<br />Bug #1: Pressing rotate key causes <br />square figure to move up!<br />
Bugs<br />When running the NanoXML program (main is in class Parser1_vw_v1), the following exception is thrown:<br />Excep...
Study Setup: Groups<br />
Study Setup: Groups<br />26<br />A<br />B<br />
Study Setup: Groups<br />C<br />D<br />Rank<br />Rank<br />
Results<br />
How do developers use a ranked list?<br />Lowperformers did follow list.<br />Survey says searched through statements.<br ...
Is perfect bug understanding realistic?<br />Only 1 out of 10 programmers who clicked on bug stopped investigation.<br />T...
Are automated toolsspeeding up debugging?<br />=<br />Traditional<br />Automated group<br />No<br />✘<br />
Are automated toolsspeeding up debugging?<br />No<br />✘<br />=<br />Traditional<br />Automated group<br />Rank<br />No<br...
Are automated toolsspeeding up debugging?<br />=<br />Traditional<br />Automated group<br />No<br />✘<br />
But… Stratifying Participants<br />High Performers<br />Medium Performers<br />Low Performers<br />✔<br />✔<br />✘<br />✔<...
Significant difference for “experts”<br />High Performers<br />On average, 5 minutes faster<br />✔<br />✔<br />
Are automated toolsspeeding up debugging?<br />No<br />✘<br />=<br />Traditional<br />Automated group<br />Yes!<br />✔<br ...
Observations<br />Developers searched through statements.<br />Developers without tool fixed symptoms (not problem).<br />...
Future directions<br />
Moving beyond fault space reduction<br />39<br />We can keep building better tools.<br />But we can’t keep abstracting awa...
Performing further studies<br />Does different granularity work better for inspection?  Documents?  Methods?<br />How does...
How do developers use a ranked list?<br />Is perfect bug understanding realistic?<br />Are Automated Debugging Tools Helpf...
64,000,000 miles<br />800,000 miles<br />35 years of <br />Scientific Progress<br />1969<br />2004<br />42<br />
Upcoming SlideShare
Loading in …5
×

Are Automated Debugging Techniques Actually Helping Programmers

2,744 views
2,621 views

Published on

Published in: Technology
1 Comment
2 Likes
Statistics
Notes
No Downloads
Views
Total views
2,744
On SlideShare
0
From Embeds
0
Number of Embeds
20
Actions
Shares
0
Downloads
59
Comments
1
Likes
2
Embeds 0
No embeds

No notes for slide
  • Just ask this guy.
  • Maybe you have some questions…
  • Maybe you have some questions…
  • Cons
  • 58,300 papers on automated debugging…  1,170 this year! (google scholar)
  • Both linear and exhaustive?
  • Do two things
  • Intutively, mental model etc
  • 37% of the visits jumped more than one position and, on average, each jump skipped 10 positionsEach participant zigzagged, with an average of 10.3 zigzags,with an overall range between 1 and 36 zigzags.would scan the ranked list to find a statement that might confirm theirhypothesis about the cause of the failure, whereas othertimes they skipped statements that did not appear relevant.
  • Or are we blind to the bugs in front of us.
  • Info graphics, N=X
  • Can we leveragehypothesises
  • Does perfect bug understanding exist?Are Automated Debugging Tools Faster?
  • Siemens 138346299297402483512SIR 62 subjects (229)
  • Are Automated Debugging Techniques Actually Helping Programmers

    1. 1. Are Automated Debugging Techniques Actually Helping Programmers?<br />Chris Parnin<br /> Georgia Tech<br />@chrisparnin (twitter)<br />Alessandro (Alex) Orso<br />Georgia Tech<br />@alexorso(twitter)<br />
    2. 2. Finding bugs can be hard…<br />
    3. 3. Automated debugging to the rescue!<br />I’ll help you find location of bug!<br />
    4. 4. How it works (Ranking-Based)<br />I have calculated most likely location of bug!<br />Give me a failing program.<br />Calculating…<br />Here is your ranked list of statements. <br />
    5. 5. How it works (Ranking-Based)<br />I have calculated most likely location of bug!<br />Give me input.<br />But how does a programmer usea ranked list of statements?<br />Calculating…<br />Here is your rankedlist of statements. <br />
    6. 6. Conceptual Model<br />1)<br />2)<br />3)<br />4)<br />…<br />Here is a list of places to check out<br />Ok, I will check out your suggestions one by one.<br />
    7. 7. Conceptual Model<br />1)<br />2)<br />3)<br />4)<br />…<br />Found the bug!<br />
    8. 8. A Skeptic<br />Does the conceptual model make sense?<br />Have we evaluated it?<br />
    9. 9. Let’s see…<br />Over 50 years of researchonautomated debugging.<br />2001. Statistical Debugging<br />1999. Delta Debugging<br />1981. Weiser. Program Slicing<br />1962. Symbolic Debugging (UNIVAC FLIT)<br />
    10. 10. Did you see anything?<br />
    11. 11. Only 5 papers have evaluated <br />automated debugging techniques <br />with actual programmers.<br /><ul><li>Most find no benefit
    12. 12. Most done on programs < 100 LOC</li></li></ul><li>More generally, two points<br />Techniques rely on <br />two strong assumptions<br />
    13. 13. Assumption #1: Perfect bug understanding must also exist when using automated tool.<br />Do you see a bug?<br />
    14. 14. Assumption #2<br />Programmer inspects statements linearly and exhaustively until finding bug.<br />Is this realistic?<br />
    15. 15. How can we evaluate the conceptual model?<br />How could we measure the benefit of an automated tool?<br />
    16. 16. Conceptual model: What if gave a developer a list of statements to inspect?<br />How would they use the list?<br />Would they be able to see the bug after visiting it?<br />Is ranking important?<br />
    17. 17. Benefit: What if we evaluate programmers with and without automated debuggers?<br />><br />?<br />We also could observe what works and what doesn’t.<br />
    18. 18. Study Setup<br />34 Developers<br />2 Debugging Tasks<br />Automated debugging tool<br />
    19. 19. Study Setup<br />Participants:<br />34 developers<br />MS/Phd Students<br />Different levels of expertise (low,medium,high)<br />
    20. 20. Study Setup<br />Software subjects:<br />Tetris (2.5 kloc)<br />NanoXML (4.5 kloc)<br />
    21. 21. Study Setup<br />21<br />Tools:<br /> Traditional debugger <br /> Eclipse ranking plugin<br />(logged activity)<br />
    22. 22. Study Setup<br />Tasks:<br /> Debugging fault<br /> 30 minutes per task<br />Questionnaire at end<br />
    23. 23. Bugs<br />Bug #1: Pressing rotate key causes <br />square figure to move up!<br />
    24. 24. Bugs<br />When running the NanoXML program (main is in class Parser1_vw_v1), the following exception is thrown:<br />Exception in thread "main" net.n3.nanoxml.XMLParseException: <br />XML Not Well-Formed at Line 19: Closing tag does not match opening tag: `ns:Bar' != `:Bar'<br />at net.n3.nanoxml.XMLUtil.errorWrongClosingTag(XMLUtil.java:497)<br />at net.n3.nanoxml.StdXMLParser.processElement(StdXMLParser.java:438)<br /> at net.n3.nanoxml.StdXMLParser.scanSomeTag(StdXMLParser.java:202)<br /> at net.n3.nanoxml.StdXMLParser.processElement(StdXMLParser.java:453)<br /> at net.n3.nanoxml.StdXMLParser.scanSomeTag(StdXMLParser.java:202)<br /> at net.n3.nanoxml.StdXMLParser.scanData(StdXMLParser.java:159)<br /> at net.n3.nanoxml.StdXMLParser.parse(StdXMLParser.java:133)<br /> at net.n3.nanoxml.Parser1_vw_v1.main(Parser1_vw_v1.java:50)<br /> <br />The input, testvm_22.xml, contains the following input xml document:<br /><Foo a=”test”><br /> <ns:Bar><br /> <Blah x=”1” ns:x=”2”/><br /> </ns:Bar><br /></Foo><br />Bug #2: Exception on input xml document.<br />
    25. 25. Study Setup: Groups<br />
    26. 26. Study Setup: Groups<br />26<br />A<br />B<br />
    27. 27. Study Setup: Groups<br />C<br />D<br />Rank<br />Rank<br />
    28. 28. Results<br />
    29. 29. How do developers use a ranked list?<br />Lowperformers did follow list.<br />Survey says searched through statements.<br />37% of visits jumped avg. 10.<br />Navigation pattern zig-zagged (avg. 10 zigzags)<br />
    30. 30. Is perfect bug understanding realistic?<br />Only 1 out of 10 programmers who clicked on bug stopped investigation.<br />The others spent on average ten minutes continuing investigation.<br />
    31. 31. Are automated toolsspeeding up debugging?<br />=<br />Traditional<br />Automated group<br />No<br />✘<br />
    32. 32. Are automated toolsspeeding up debugging?<br />No<br />✘<br />=<br />Traditional<br />Automated group<br />Rank<br />No<br />=<br />✘<br />Traditional<br />Automated group<br />
    33. 33. Are automated toolsspeeding up debugging?<br />=<br />Traditional<br />Automated group<br />No<br />✘<br />
    34. 34. But… Stratifying Participants<br />High Performers<br />Medium Performers<br />Low Performers<br />✔<br />✔<br />✘<br />✔<br />✘<br />✘<br />
    35. 35. Significant difference for “experts”<br />High Performers<br />On average, 5 minutes faster<br />✔<br />✔<br />
    36. 36. Are automated toolsspeeding up debugging?<br />No<br />✘<br />=<br />Traditional<br />Automated group<br />Yes!<br />✔<br />Experts<br />Experts<br />><br />Traditional<br />Automated group<br />
    37. 37. Observations<br />Developers searched through statements.<br />Developers without tool fixed symptoms (not problem).<br />Developers wanted explanations rather than recommendations.<br />
    38. 38. Future directions<br />
    39. 39. Moving beyond fault space reduction<br />39<br />We can keep building better tools.<br />But we can’t keep abstracting away the human.<br />
    40. 40. Performing further studies<br />Does different granularity work better for inspection? Documents? Methods?<br />How does different interfaces or visualizations impact technique?<br />Do other automated debugging techniques fare any better?<br />40<br />
    41. 41. How do developers use a ranked list?<br />Is perfect bug understanding realistic?<br />Are Automated Debugging Tools Helpful?<br />Human studies, human studies, human studies!<br />
    42. 42. 64,000,000 miles<br />800,000 miles<br />35 years of <br />Scientific Progress<br />1969<br />2004<br />42<br />
    43. 43. 352 LOC<br />(median 8 programs)<br />63.5 LOC<br />(median 4 programs)<br />30 years<br />30 years of <br />Scientific Progress<br />1981<br />2011<br />43<br />

    ×