Your SlideShare is downloading. ×
0
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Are Automated Debugging Techniques Actually Helping Programmers

2,336

Published on

Published in: Technology
1 Comment
1 Like
Statistics
Notes
No Downloads
Views
Total Views
2,336
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
48
Comments
1
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Just ask this guy.
  • Maybe you have some questions…
  • Maybe you have some questions…
  • Cons
  • 58,300 papers on automated debugging…  1,170 this year! (google scholar)
  • Both linear and exhaustive?
  • Do two things
  • Intutively, mental model etc
  • 37% of the visits jumped more than one position and, on average, each jump skipped 10 positionsEach participant zigzagged, with an average of 10.3 zigzags,with an overall range between 1 and 36 zigzags.would scan the ranked list to find a statement that might confirm theirhypothesis about the cause of the failure, whereas othertimes they skipped statements that did not appear relevant.
  • Or are we blind to the bugs in front of us.
  • Info graphics, N=X
  • Can we leveragehypothesises
  • Does perfect bug understanding exist?Are Automated Debugging Tools Faster?
  • Siemens 138346299297402483512SIR 62 subjects (229)
  • Transcript

    • 1. Are Automated Debugging Techniques Actually Helping Programmers?
      Chris Parnin
      Georgia Tech
      @chrisparnin (twitter)
      Alessandro (Alex) Orso
      Georgia Tech
      @alexorso(twitter)
    • 2. Finding bugs can be hard…
    • 3. Automated debugging to the rescue!
      I’ll help you find location of bug!
    • 4. How it works (Ranking-Based)
      I have calculated most likely location of bug!
      Give me a failing program.
      Calculating…
      Here is your ranked list of statements.
    • 5. How it works (Ranking-Based)
      I have calculated most likely location of bug!
      Give me input.
      But how does a programmer usea ranked list of statements?
      Calculating…
      Here is your rankedlist of statements.
    • 6. Conceptual Model
      1)
      2)
      3)
      4)

      Here is a list of places to check out
      Ok, I will check out your suggestions one by one.
    • 7. Conceptual Model
      1)
      2)
      3)
      4)

      Found the bug!
    • 8. A Skeptic
      Does the conceptual model make sense?
      Have we evaluated it?
    • 9. Let’s see…
      Over 50 years of researchonautomated debugging.
      2001. Statistical Debugging
      1999. Delta Debugging
      1981. Weiser. Program Slicing
      1962. Symbolic Debugging (UNIVAC FLIT)
    • 10. Did you see anything?
    • 11. Only 5 papers have evaluated
      automated debugging techniques
      with actual programmers.
      • Most find no benefit
      • 12. Most done on programs < 100 LOC
    • More generally, two points
      Techniques rely on
      two strong assumptions
    • 13. Assumption #1: Perfect bug understanding must also exist when using automated tool.
      Do you see a bug?
    • 14. Assumption #2
      Programmer inspects statements linearly and exhaustively until finding bug.
      Is this realistic?
    • 15. How can we evaluate the conceptual model?
      How could we measure the benefit of an automated tool?
    • 16. Conceptual model: What if gave a developer a list of statements to inspect?
      How would they use the list?
      Would they be able to see the bug after visiting it?
      Is ranking important?
    • 17. Benefit: What if we evaluate programmers with and without automated debuggers?
      >
      ?
      We also could observe what works and what doesn’t.
    • 18. Study Setup
      34 Developers
      2 Debugging Tasks
      Automated debugging tool
    • 19. Study Setup
      Participants:
      34 developers
      MS/Phd Students
      Different levels of expertise (low,medium,high)
    • 20. Study Setup
      Software subjects:
      Tetris (2.5 kloc)
      NanoXML (4.5 kloc)
    • 21. Study Setup
      21
      Tools:
      Traditional debugger
      Eclipse ranking plugin
      (logged activity)
    • 22. Study Setup
      Tasks:
      Debugging fault
      30 minutes per task
      Questionnaire at end
    • 23. Bugs
      Bug #1: Pressing rotate key causes
      square figure to move up!
    • 24. Bugs
      When running the NanoXML program (main is in class Parser1_vw_v1), the following exception is thrown:
      Exception in thread "main" net.n3.nanoxml.XMLParseException:
      XML Not Well-Formed at Line 19: Closing tag does not match opening tag: `ns:Bar' != `:Bar'
      at net.n3.nanoxml.XMLUtil.errorWrongClosingTag(XMLUtil.java:497)
      at net.n3.nanoxml.StdXMLParser.processElement(StdXMLParser.java:438)
      at net.n3.nanoxml.StdXMLParser.scanSomeTag(StdXMLParser.java:202)
      at net.n3.nanoxml.StdXMLParser.processElement(StdXMLParser.java:453)
      at net.n3.nanoxml.StdXMLParser.scanSomeTag(StdXMLParser.java:202)
      at net.n3.nanoxml.StdXMLParser.scanData(StdXMLParser.java:159)
      at net.n3.nanoxml.StdXMLParser.parse(StdXMLParser.java:133)
      at net.n3.nanoxml.Parser1_vw_v1.main(Parser1_vw_v1.java:50)
       
      The input, testvm_22.xml, contains the following input xml document:
      <Foo a=”test”>
      <ns:Bar>
      <Blah x=”1” ns:x=”2”/>
      </ns:Bar>
      </Foo>
      Bug #2: Exception on input xml document.
    • 25. Study Setup: Groups
    • 26. Study Setup: Groups
      26
      A
      B
    • 27. Study Setup: Groups
      C
      D
      Rank
      Rank
    • 28. Results
    • 29. How do developers use a ranked list?
      Lowperformers did follow list.
      Survey says searched through statements.
      37% of visits jumped avg. 10.
      Navigation pattern zig-zagged (avg. 10 zigzags)
    • 30. Is perfect bug understanding realistic?
      Only 1 out of 10 programmers who clicked on bug stopped investigation.
      The others spent on average ten minutes continuing investigation.
    • 31. Are automated toolsspeeding up debugging?
      =
      Traditional
      Automated group
      No

    • 32. Are automated toolsspeeding up debugging?
      No

      =
      Traditional
      Automated group
      Rank
      No
      =

      Traditional
      Automated group
    • 33. Are automated toolsspeeding up debugging?
      =
      Traditional
      Automated group
      No

    • 34. But… Stratifying Participants
      High Performers
      Medium Performers
      Low Performers






    • 35. Significant difference for “experts”
      High Performers
      On average, 5 minutes faster


    • 36. Are automated toolsspeeding up debugging?
      No

      =
      Traditional
      Automated group
      Yes!

      Experts
      Experts
      >
      Traditional
      Automated group
    • 37. Observations
      Developers searched through statements.
      Developers without tool fixed symptoms (not problem).
      Developers wanted explanations rather than recommendations.
    • 38. Future directions
    • 39. Moving beyond fault space reduction
      39
      We can keep building better tools.
      But we can’t keep abstracting away the human.
    • 40. Performing further studies
      Does different granularity work better for inspection? Documents? Methods?
      How does different interfaces or visualizations impact technique?
      Do other automated debugging techniques fare any better?
      40
    • 41. How do developers use a ranked list?
      Is perfect bug understanding realistic?
      Are Automated Debugging Tools Helpful?
      Human studies, human studies, human studies!
    • 42. 64,000,000 miles
      800,000 miles
      35 years of
      Scientific Progress
      1969
      2004
      42
    • 43. 352 LOC
      (median 8 programs)
      63.5 LOC
      (median 4 programs)
      30 years
      30 years of
      Scientific Progress
      1981
      2011
      43

    ×