Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Are Automated Debugging Techniques Actually Helping Programmers?<br />Chris Parnin<br />    Georgia Tech<br />@chrisparnin...
Finding bugs can be hard…<br />
Automated debugging to the rescue!<br />I’ll help you find location of bug!<br />
How it works (Ranking-Based)<br />I have calculated most likely location of bug!<br />Give me a failing program.<br />Calc...
How it works (Ranking-Based)<br />I have calculated most likely location of bug!<br />Give me input.<br />But how does a p...
Conceptual Model<br />1)<br />2)<br />3)<br />4)<br />…<br />Here is a list of places to check out<br />Ok, I will check o...
Conceptual Model<br />1)<br />2)<br />3)<br />4)<br />…<br />Found the bug!<br />
A Skeptic<br />Does the conceptual model make sense?<br />Have we evaluated it?<br />
Let’s see…<br />Over 50 years of researchonautomated debugging.<br />2001. Statistical Debugging<br />1999. Delta Debuggin...
 Did you see anything?<br />
Only 5 papers have evaluated <br />automated debugging techniques <br />with actual programmers.<br /><ul><li>Most find no...
Most done on programs < 100 LOC</li></li></ul><li>More generally, two points<br />Techniques rely on <br />two strong assu...
Assumption #1: Perfect bug understanding must also exist when using automated tool.<br />Do you see a bug?<br />
Assumption #2<br />Programmer inspects statements linearly and exhaustively until finding bug.<br />Is this realistic?<br />
How can we evaluate the conceptual model?<br />How could we measure the benefit of an automated tool?<br />
Conceptual model: What if gave a developer a list of statements to inspect?<br />How would they use the list?<br />Would t...
Benefit: What if we evaluate programmers with and without automated debuggers?<br />><br />?<br />We also could observe wh...
Study Setup<br />34 Developers<br />2 Debugging Tasks<br />Automated debugging tool<br />
Study Setup<br />Participants:<br />34 developers<br />MS/Phd Students<br />Different levels of expertise 	(low,medium,hig...
Study Setup<br />Software subjects:<br />Tetris (2.5 kloc)<br />NanoXML (4.5 kloc)<br />
Study Setup<br />21<br />Tools:<br />	Traditional debugger <br />	Eclipse ranking plugin<br />(logged activity)<br />
Study Setup<br />Tasks:<br />	Debugging fault<br />	30 minutes per task<br />Questionnaire at end<br />
Bugs<br />Bug #1: Pressing rotate key causes <br />square figure to move up!<br />
Bugs<br />When running the NanoXML program (main is in class Parser1_vw_v1), the following exception is thrown:<br />Excep...
Study Setup: Groups<br />
Study Setup: Groups<br />26<br />A<br />B<br />
Study Setup: Groups<br />C<br />D<br />Rank<br />Rank<br />
Results<br />
How do developers use a ranked list?<br />Lowperformers did follow list.<br />Survey says searched through statements.<br ...
Is perfect bug understanding realistic?<br />Only 1 out of 10 programmers who clicked on bug stopped investigation.<br />T...
Are automated toolsspeeding up debugging?<br />=<br />Traditional<br />Automated group<br />No<br />✘<br />
Are automated toolsspeeding up debugging?<br />No<br />✘<br />=<br />Traditional<br />Automated group<br />Rank<br />No<br...
Are automated toolsspeeding up debugging?<br />=<br />Traditional<br />Automated group<br />No<br />✘<br />
But… Stratifying Participants<br />High Performers<br />Medium Performers<br />Low Performers<br />✔<br />✔<br />✘<br />✔<...
Significant difference for “experts”<br />High Performers<br />On average, 5 minutes faster<br />✔<br />✔<br />
Are automated toolsspeeding up debugging?<br />No<br />✘<br />=<br />Traditional<br />Automated group<br />Yes!<br />✔<br ...
Observations<br />Developers searched through statements.<br />Developers without tool fixed symptoms (not problem).<br />...
Future directions<br />
Moving beyond fault space reduction<br />39<br />We can keep building better tools.<br />But we can’t keep abstracting awa...
Performing further studies<br />Does different granularity work better for inspection?  Documents?  Methods?<br />How does...
How do developers use a ranked list?<br />Is perfect bug understanding realistic?<br />Are Automated Debugging Tools Helpf...
64,000,000 miles<br />800,000 miles<br />35 years of <br />Scientific Progress<br />1969<br />2004<br />42<br />
Upcoming SlideShare
Loading in …5
×

Are Automated Debugging Techniques Actually Helping Programmers

2,906 views

Published on

Published in: Technology

Are Automated Debugging Techniques Actually Helping Programmers

  1. 1. Are Automated Debugging Techniques Actually Helping Programmers?<br />Chris Parnin<br /> Georgia Tech<br />@chrisparnin (twitter)<br />Alessandro (Alex) Orso<br />Georgia Tech<br />@alexorso(twitter)<br />
  2. 2. Finding bugs can be hard…<br />
  3. 3. Automated debugging to the rescue!<br />I’ll help you find location of bug!<br />
  4. 4. How it works (Ranking-Based)<br />I have calculated most likely location of bug!<br />Give me a failing program.<br />Calculating…<br />Here is your ranked list of statements. <br />
  5. 5. How it works (Ranking-Based)<br />I have calculated most likely location of bug!<br />Give me input.<br />But how does a programmer usea ranked list of statements?<br />Calculating…<br />Here is your rankedlist of statements. <br />
  6. 6. Conceptual Model<br />1)<br />2)<br />3)<br />4)<br />…<br />Here is a list of places to check out<br />Ok, I will check out your suggestions one by one.<br />
  7. 7. Conceptual Model<br />1)<br />2)<br />3)<br />4)<br />…<br />Found the bug!<br />
  8. 8. A Skeptic<br />Does the conceptual model make sense?<br />Have we evaluated it?<br />
  9. 9. Let’s see…<br />Over 50 years of researchonautomated debugging.<br />2001. Statistical Debugging<br />1999. Delta Debugging<br />1981. Weiser. Program Slicing<br />1962. Symbolic Debugging (UNIVAC FLIT)<br />
  10. 10. Did you see anything?<br />
  11. 11. Only 5 papers have evaluated <br />automated debugging techniques <br />with actual programmers.<br /><ul><li>Most find no benefit
  12. 12. Most done on programs < 100 LOC</li></li></ul><li>More generally, two points<br />Techniques rely on <br />two strong assumptions<br />
  13. 13. Assumption #1: Perfect bug understanding must also exist when using automated tool.<br />Do you see a bug?<br />
  14. 14. Assumption #2<br />Programmer inspects statements linearly and exhaustively until finding bug.<br />Is this realistic?<br />
  15. 15. How can we evaluate the conceptual model?<br />How could we measure the benefit of an automated tool?<br />
  16. 16. Conceptual model: What if gave a developer a list of statements to inspect?<br />How would they use the list?<br />Would they be able to see the bug after visiting it?<br />Is ranking important?<br />
  17. 17. Benefit: What if we evaluate programmers with and without automated debuggers?<br />><br />?<br />We also could observe what works and what doesn’t.<br />
  18. 18. Study Setup<br />34 Developers<br />2 Debugging Tasks<br />Automated debugging tool<br />
  19. 19. Study Setup<br />Participants:<br />34 developers<br />MS/Phd Students<br />Different levels of expertise (low,medium,high)<br />
  20. 20. Study Setup<br />Software subjects:<br />Tetris (2.5 kloc)<br />NanoXML (4.5 kloc)<br />
  21. 21. Study Setup<br />21<br />Tools:<br /> Traditional debugger <br /> Eclipse ranking plugin<br />(logged activity)<br />
  22. 22. Study Setup<br />Tasks:<br /> Debugging fault<br /> 30 minutes per task<br />Questionnaire at end<br />
  23. 23. Bugs<br />Bug #1: Pressing rotate key causes <br />square figure to move up!<br />
  24. 24. Bugs<br />When running the NanoXML program (main is in class Parser1_vw_v1), the following exception is thrown:<br />Exception in thread "main" net.n3.nanoxml.XMLParseException: <br />XML Not Well-Formed at Line 19: Closing tag does not match opening tag: `ns:Bar' != `:Bar'<br />at net.n3.nanoxml.XMLUtil.errorWrongClosingTag(XMLUtil.java:497)<br />at net.n3.nanoxml.StdXMLParser.processElement(StdXMLParser.java:438)<br /> at net.n3.nanoxml.StdXMLParser.scanSomeTag(StdXMLParser.java:202)<br /> at net.n3.nanoxml.StdXMLParser.processElement(StdXMLParser.java:453)<br /> at net.n3.nanoxml.StdXMLParser.scanSomeTag(StdXMLParser.java:202)<br /> at net.n3.nanoxml.StdXMLParser.scanData(StdXMLParser.java:159)<br /> at net.n3.nanoxml.StdXMLParser.parse(StdXMLParser.java:133)<br /> at net.n3.nanoxml.Parser1_vw_v1.main(Parser1_vw_v1.java:50)<br /> <br />The input, testvm_22.xml, contains the following input xml document:<br /><Foo a=”test”><br /> <ns:Bar><br /> <Blah x=”1” ns:x=”2”/><br /> </ns:Bar><br /></Foo><br />Bug #2: Exception on input xml document.<br />
  25. 25. Study Setup: Groups<br />
  26. 26. Study Setup: Groups<br />26<br />A<br />B<br />
  27. 27. Study Setup: Groups<br />C<br />D<br />Rank<br />Rank<br />
  28. 28. Results<br />
  29. 29. How do developers use a ranked list?<br />Lowperformers did follow list.<br />Survey says searched through statements.<br />37% of visits jumped avg. 10.<br />Navigation pattern zig-zagged (avg. 10 zigzags)<br />
  30. 30. Is perfect bug understanding realistic?<br />Only 1 out of 10 programmers who clicked on bug stopped investigation.<br />The others spent on average ten minutes continuing investigation.<br />
  31. 31. Are automated toolsspeeding up debugging?<br />=<br />Traditional<br />Automated group<br />No<br />✘<br />
  32. 32. Are automated toolsspeeding up debugging?<br />No<br />✘<br />=<br />Traditional<br />Automated group<br />Rank<br />No<br />=<br />✘<br />Traditional<br />Automated group<br />
  33. 33. Are automated toolsspeeding up debugging?<br />=<br />Traditional<br />Automated group<br />No<br />✘<br />
  34. 34. But… Stratifying Participants<br />High Performers<br />Medium Performers<br />Low Performers<br />✔<br />✔<br />✘<br />✔<br />✘<br />✘<br />
  35. 35. Significant difference for “experts”<br />High Performers<br />On average, 5 minutes faster<br />✔<br />✔<br />
  36. 36. Are automated toolsspeeding up debugging?<br />No<br />✘<br />=<br />Traditional<br />Automated group<br />Yes!<br />✔<br />Experts<br />Experts<br />><br />Traditional<br />Automated group<br />
  37. 37. Observations<br />Developers searched through statements.<br />Developers without tool fixed symptoms (not problem).<br />Developers wanted explanations rather than recommendations.<br />
  38. 38. Future directions<br />
  39. 39. Moving beyond fault space reduction<br />39<br />We can keep building better tools.<br />But we can’t keep abstracting away the human.<br />
  40. 40. Performing further studies<br />Does different granularity work better for inspection? Documents? Methods?<br />How does different interfaces or visualizations impact technique?<br />Do other automated debugging techniques fare any better?<br />40<br />
  41. 41. How do developers use a ranked list?<br />Is perfect bug understanding realistic?<br />Are Automated Debugging Tools Helpful?<br />Human studies, human studies, human studies!<br />
  42. 42. 64,000,000 miles<br />800,000 miles<br />35 years of <br />Scientific Progress<br />1969<br />2004<br />42<br />
  43. 43. 352 LOC<br />(median 8 programs)<br />63.5 LOC<br />(median 4 programs)<br />30 years<br />30 years of <br />Scientific Progress<br />1981<br />2011<br />43<br />

×