Valentina Grigoreanu, Margaret Burnett, George Robertson<br />Oregon State University & Microsoft Corporation<br />04.13.1...
Agenda<br />Problem + Theoretical Findings<br />StratCel: New Strategy-Centric Debugging Tool<br />Tactics+Moves<br />Stra...
Research Methods: Eight Studies<br />Factors to consider in future studies (Moves and Tactics)<br />Generalizing across en...
Problem Addressed<br />End-user programmers’ code is buggy<br /><ul><li>E.g., [Powell et al. 2008; EuSpRIG 2006]</li></ul>...
Problem Addressed<br />Not only is EUP code buggy…<br />Problem amplified by:<br />Tens of millions of end-user programmer...
Related Work<br />Previous research / tools to improve end-user debugging:<br />Automatic bug detection<br />Mapping techn...
CHI Paper/Dissertation RQ<br />How can we design tools such that they support end-user programmers’ existingeffectivedebug...
Definitions<br />Has Impact on…<br />Iteratative improvements to Existing Features.<br />New feature innovation.<br />Over...
Outline<br />Valentina Grigoreanu (M.A.X.)	9<br />
The Stratagems<br />Code Inspection<br />Control Flow<br />Dataflow<br />Feedback Following<br />Help<br />Proceeding as i...
Outline<br />Valentina Grigoreanu (M.A.X.)	11<br />
Outline<br />Valentina Grigoreanu (M.A.X.)	12<br />
StratCel<br />21 implications for design<br />At 3 levels of abstraction:<br />Strategies (high)<br />Stratagems (medium)<...
StratCel<br />Valentina Grigoreanu (M.A.X.)	14<br />
1. Strategies Implication<br />Observed two strategies<br />Comprehensive strategy:<br />Thorough, found easy bugs quickly...
Strategies Implication<br />Sounds a lot like support for the to-do listing stratagem might help:<br />Observed in Forms/3...
StratCel:Strategies and To Dos<br />Does not impose an order<br />Valentina Grigoreanu (M.A.X.)	17<br />
2. Stratagems Implication<br />Support all strategems in the context of each to-do item (specification checking, testing, ...
StratCel:Multi-Stratagem Support<br />Valentina Grigoreanu (M.A.X.)	19<br />
3. Moves & Tactics Implications<br />Highlighting formula cells<br />Too many warnings is as bad as none at all<br />Inter...
Good Tactics, Bad MovesImplication 3<br />Following dependencies between spreadsheets<br />How many of our participants do...
StratCel: Highlight Formulas<br />Valentina Grigoreanu (M.A.X.)	22<br />
StratCel: Error Checking<br />Valentina Grigoreanu (M.A.X.)	23<br />
StratCel: Interworksheet Dependencies<br />Valentina Grigoreanu (M.A.X.)	24<br />
Evaluation: Study Setup<br />8 Control participants / 5 Treatment participants<br />Experience with formulas<br />Think-al...
StratCel Results:Triangulation<br />Why?<br /><ul><li> Too much feedback is as bad as no feedback.
 Support for selective and comprehensive strats:</li></ul>Selective: used status borders to easily find areas on which to ...
StratCel Results:Triangulation<br />Why?<br /><ul><li> Found significantly more bugs.
 Contextually-provided information on additional stratagems:
 Specification
 Recently used function library
 Each feature used by at least one participant</li></ul>Bugs Fixed<br />p<0.002<br />The time factor:<br />- Debugging is ...
 Time to first bug find: p<0.005
Upcoming SlideShare
Loading in …5
×

StratCel: A Strategy-Centric Approach to the Design of End-User Debugging Tools

537 views

Published on

Presented by Valentina Grigoreanu at CHI2010. Demonstrates how tools can be designed around users' strategies. StratCel is an add-in for Excel that supports spreadsheet users' debugging strategies. Includes evaluation and design guidelines for supporting spreadsheet debugging.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

StratCel: A Strategy-Centric Approach to the Design of End-User Debugging Tools

  1. 1. Valentina Grigoreanu, Margaret Burnett, George Robertson<br />Oregon State University & Microsoft Corporation<br />04.13.10 – CHI 2010 Talk<br />Strategy-Centric Approach:Design of End-User Debugging Tools <br />Valentina Grigoreanu (M.A.X.) 1<br />
  2. 2. Agenda<br />Problem + Theoretical Findings<br />StratCel: New Strategy-Centric Debugging Tool<br />Tactics+Moves<br />Stratagems<br />Strategies<br />Valentina Grigoreanu (M.A.X.) 2<br />
  3. 3. Research Methods: Eight Studies<br />Factors to consider in future studies (Moves and Tactics)<br />Generalizing across environments, populations, & study setups (Stratagems)<br />A Sensemaking perspective on end-user debugging (Strategies)<br />Designing end-user debugging tools (All four levels)<br />Valentina Grigoreanu (M.A.X.) 3<br />
  4. 4. Problem Addressed<br />End-user programmers’ code is buggy<br /><ul><li>E.g., [Powell et al. 2008; EuSpRIG 2006]</li></ul>Valentina Grigoreanu (M.A.X.) 4<br />
  5. 5. Problem Addressed<br />Not only is EUP code buggy…<br />Problem amplified by:<br />Tens of millions of end-user programmers [Scaffidi et al. 2005]<br />Lack of training in software engineering techniques [Study Participants]<br />Increasing size and complexity of spreadsheets [Whittaker 1999]<br />Valentina Grigoreanu (M.A.X.) 5<br />
  6. 6. Related Work<br />Previous research / tools to improve end-user debugging:<br />Automatic bug detection<br />Mapping techniques from professional SE<br />Feature usage<br />A critical stone has been left unturned…<br />Valentina Grigoreanu (M.A.X.) 6<br />
  7. 7. CHI Paper/Dissertation RQ<br />How can we design tools such that they support end-user programmers’ existingeffectivedebugging strategies?<br />What are they?<br />How are they used?<br />Are there gender differences?<br />Do the strategies generalize across EUP environments?<br />How can tools be designed around strategies?<br />What impact would strategy support have on success?<br />Valentina Grigoreanu (M.A.X.) 7<br />
  8. 8. Definitions<br />Has Impact on…<br />Iteratative improvements to Existing Features.<br />New feature innovation.<br />Overview. Stumbling Blocks. Checks.<br />Note: All three of these are needed for strategy-centric design.<br />Valentina Grigoreanu (M.A.X.) 8<br />
  9. 9. Outline<br />Valentina Grigoreanu (M.A.X.) 9<br />
  10. 10. The Stratagems<br />Code Inspection<br />Control Flow<br />Dataflow<br />Feedback Following<br />Help<br />Proceeding as in Prior Experience<br />Spatial<br />Specification Checking<br />Testing<br />To-do Listing<br />Valentina Grigoreanu (M.A.X.) 10<br />
  11. 11. Outline<br />Valentina Grigoreanu (M.A.X.) 11<br />
  12. 12. Outline<br />Valentina Grigoreanu (M.A.X.) 12<br />
  13. 13. StratCel<br />21 implications for design<br />At 3 levels of abstraction:<br />Strategies (high)<br />Stratagems (medium)<br />Moves/tactics (low)<br />Valentina Grigoreanu (M.A.X.) 13<br />
  14. 14. StratCel<br />Valentina Grigoreanu (M.A.X.) 14<br />
  15. 15. 1. Strategies Implication<br />Observed two strategies<br />Comprehensive strategy:<br />Thorough, found easy bugs quickly, lots of fixes within 30 minutes<br />But… Needed help keeping track of cells to return to<br />Selective strategy:<br />Fixed the hardest bug, found some of the easier ones in the process<br />But… Needed help keeping track of cells not checked<br />Valentina Grigoreanu (M.A.X.) 15<br />
  16. 16. Strategies Implication<br />Sounds a lot like support for the to-do listing stratagem might help:<br />Observed in Forms/3, PowerShell, professional devs.<br />Poorly supported in all: often requires repurposing<br />Shows promise for improving debugging success<br />Valentina Grigoreanu (M.A.X.) 16<br />
  17. 17. StratCel:Strategies and To Dos<br />Does not impose an order<br />Valentina Grigoreanu (M.A.X.) 17<br />
  18. 18. 2. Stratagems Implication<br />Support all strategems in the context of each to-do item (specification checking, testing, code inspection, etc.).<br />Each provides additional relevant information for debugging.<br />Valentina Grigoreanu (M.A.X.) 18<br />
  19. 19. StratCel:Multi-Stratagem Support<br />Valentina Grigoreanu (M.A.X.) 19<br />
  20. 20. 3. Moves & Tactics Implications<br />Highlighting formula cells<br />Too many warnings is as bad as none at all<br />Interworksheet dependencies<br />Valentina Grigoreanu (M.A.X.) 20<br />
  21. 21. Good Tactics, Bad MovesImplication 3<br />Following dependencies between spreadsheets<br />How many of our participants do you think understood this icon when they saw it? <br />Implication for design: at least a tooltip, would be nice. Though a method for navigating to that/those cell(s) would be ideal.<br />Valentina Grigoreanu (M.A.X.) 21<br />
  22. 22. StratCel: Highlight Formulas<br />Valentina Grigoreanu (M.A.X.) 22<br />
  23. 23. StratCel: Error Checking<br />Valentina Grigoreanu (M.A.X.) 23<br />
  24. 24. StratCel: Interworksheet Dependencies<br />Valentina Grigoreanu (M.A.X.) 24<br />
  25. 25. Evaluation: Study Setup<br />8 Control participants / 5 Treatment participants<br />Experience with formulas<br />Think-aloud study<br />Debugged a grade-book spreadsheet in Excel<br />Seeded with 10 bugs<br />Or so we thought... There were actually 12!<br />Our StratCel participants found 2 more bugs that the researchers, the spreadsheet’s creator, and the Excel-only participants all had overlooked!<br />Valentina Grigoreanu (M.A.X.) 25<br />
  26. 26. StratCel Results:Triangulation<br />Why?<br /><ul><li> Too much feedback is as bad as no feedback.
  27. 27. Support for selective and comprehensive strats:</li></ul>Selective: used status borders to easily find areas on which to focus attention.<br />Comprehensive: walked through the entire to-do list item-by-item.<br />Bugs Found<br />p<0.005<br />Valentina Grigoreanu (M.A.X.) 26<br />
  28. 28. StratCel Results:Triangulation<br />Why?<br /><ul><li> Found significantly more bugs.
  29. 29. Contextually-provided information on additional stratagems:
  30. 30. Specification
  31. 31. Recently used function library
  32. 32. Each feature used by at least one participant</li></ul>Bugs Fixed<br />p<0.002<br />The time factor:<br />- Debugging is often a time-sensitive activity.<br /><ul><li> On average, Treatment participants found and fixed bugs faster.
  33. 33. Time to first bug find: p<0.005
  34. 34. Time to first bug fix: p<0.003</li></ul>Valentina Grigoreanu (M.A.X.) 27<br />
  35. 35. StratCel Results:Triangulation<br />Found<br />CF: 4.5, TF: 9.0<br />CM: 5.0, TM: 9.0<br />Fixed:<br />CF: 3.5, TF: 9<br />CM: 2.5, TM: 8.5<br />“I feel like [StratCel] would be extra useful for someone like me who, well, I can use Excel and I can figure it out, but I'm definitely not an expert at Excel. […] I think the only problems I had were with the Excel functions I hadn't learned. This is like a really good way of helping me keep track of what I've done and not get lost."<br />Valentina Grigoreanu (M.A.X.) 28<br />
  36. 36. Outline<br />Valentina Grigoreanu (M.A.X.) 29<br />
  37. 37. Parallels to IR Research<br />- This design approach can be used beyond end-user debugging.<br />- [T]he goal has been formed in much IR research to have the system do the searching for us, but that is not the only alternative. It should also be possible to design search interfaces that harmonize with and make easy the prosecution of good search strategy, systems that make it easy for novices to move quickly into good searching because the system promotes it” [Bates 1990].<br />Valentina Grigoreanu (M.A.X.) 30<br />
  38. 38. StratCel Results:Usability<br />Was StratCel intuitive – a.k.a. did we really support end-user programmers’ debugging strategies?<br />"And this is straight-forward and makes a lot of sense. When you look at it, you know what it is. There are lots of tools, where you can tell that people said, 'well… there's just a workaround and you can just do it this way'. But this one, it just seemed very straightforward and it builds on everything from Excel.”<br />“This was really helpful because it has a way to say these are all your formulas… These are the ones you need to go look at. And I like this part [the address field] which shows me where I can find all of the formulas, so I can see them. For example, on this one, I could see there was a gap for E16and I could go back and look specifically at that cell, because I expect it to be the same, and see what's going on.”<br />Valentina Grigoreanu (M.A.X.) 31<br />
  39. 39. StratCel Results:Real-World Applications<br />Applicable to real-world tasks:<br />“I think this would be useful for my complex accounting spreadsheets. If you would like to share the tool, I would love to try it on those.”<br />“Looking at [StratCel], I was thinking I have to have a way of tracking my [knitting] patterns. So things that… Ok. I have a pattern and I have steps I have to go through. And I need a way to track them.”<br />"So, can I use your tool? You should sell this and make a million dollars!”<br />Valentina Grigoreanu (M.A.X.) 32<br />
  40. 40. Validated Design Guidelines<br />Automatic error detection tools default: Value quality (low number of false-positives) over quantity (detecting more possible types of errors). <br />Important information about cells (e.g., to-do status) should be overlaid onto the spreadsheet to give the user a quick overview of the to-do status of both individual cells and of the overall spreadsheet. <br />Some users are comprehensive, whereas others will start by trying to fix apparent bugs right away. Support both, since both have pros and cons.<br />Strategy-based tools should provide explicit support for to-do listing. <br />To improve debugging of end-user programs, it helps to automatically generate a list of items to check so that all areas of the code are given equal attention. <br />Information about the remaining strategems should be provided in the context of each to-do item to provide more information on which to base a bug fix. <br />Viewing formulas related to an item (e.g., the consistent formulas, recently used formulas, or formulas used in files in a certain directory) may be particularly useful for improving debugging success.<br />33<br />

×