Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Reducing Test Case Bloat2 1


Published on

Lanette Creamer's presentation on Reducing Test Case Bloat (version 2.1) from QASIG, March 10, 2010

Published in: Technology, Business
  • Want to preview some of our plans? You can get 50 Woodworking Plans and a 440-Page "The Art of Woodworking" Book... Absolutely FREE ➤➤
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Reducing Test Case Bloat2 1

  1. 1. ReducingTest Case Bloat<br />Lanette Creamer<br />Adobe Systems, Inc<br />QASIG March 2010<br />
  2. 2. Introduction<br />Tester and Test Lead at Adobe since 2000<br />September 2009-First Agile project<br /> Python scripts, collaboration, user focus, testing the whole system, cat photos.<br />Dysfunctional fiefdoms, clip art, spiders, waiting, retesting.<br />
  3. 3. Agenda<br />What is a Test Case? What is Test Case Bloat?<br />Methods to reduce test case bloat in practice.<br />Barriers to reducing bloat.<br />How did it go?<br />Q&A<br />
  4. 4. Section One: Why reduce at all?<br />“I have no help to send, therefore I must go myself.”<br />J.R.R. Tolkien<br />
  5. 5. We like our bloat<br />How is that working for you?<br />
  6. 6. What is a Test?<br />A test is an action which produces discoveries that can be used to evaluate product quality.<br />
  7. 7. Test Case<br />Is every test a test case?<br />
  8. 8. What is Bloat?<br />Test Case Bloat-More test cases than you can effectively manage* in the time you have.<br />Is all bloat test cases?<br />No.<br />*Manage may imply some or all of the following: cover, update, execute, report on, allocate, run, automate, respond to.<br />
  9. 9. Section Two: Find the bloat<br />“All we have to decide is what to do with the time that is given to us.”<br />J.R.R. Tolkien<br />The Fellowship of the Ring 1954<br />
  10. 10. Is it worth keeping?<br />Cost versus Risk<br />If it doesn’t result in a bug is it bloat?<br />If it does result in a bug is it always the most important bug?<br />Does reduce mean delete?<br />Is it worth keeping all test cases just in case?<br /><ul><li>Time
  11. 11. Culture
  12. 12. Set up
  13. 13. Trust</li></li></ul><li>Section 3: Ideas used in practice with success<br />“Their plans were improved with the best advice.”<br />J.R.R. Tolkien<br />
  14. 14. Ways to Reduce Test Case Bloat<br />Mathematical Combinations<br /><ul><li>Scatterplot
  15. 15. Pairwise
  16. 16. Random Selection
  17. 17. Rotation of Environmental Factors</li></li></ul><li>Ways to Reduce Test Case Bloat<br />Tiered by Importance<br /><ul><li>Priority 1, 2, 3
  18. 18. High, Medium, Low
  19. 19. Daily, Build, Weekly, Once</li></ul>Stakeholders Priorities<br /><ul><li>Poker Chips
  20. 20. Backlog</li></li></ul><li>Ways to Reduce Test Case<br />Change Based<br /><ul><li>Bug fixes
  21. 21. Features added
  22. 22. Components integrated
  23. 23. New visualization tools</li></ul>Timing Based<br /><ul><li>Covered later elsewhere
  24. 24. Stop running at ______</li></li></ul><li>Ways to Reduce Test Case Bloat<br />Test Case Find Results<br /><ul><li>Areas of breakage
  25. 25. Date of last failure
  26. 26. Auto retire?</li></ul>Model Based<br /><ul><li>Interactions with more risk
  27. 27. Changes to the model</li></li></ul><li>Ways to Reduce Test Case Bloat<br />Code Coverage Based<br /><ul><li>Reconcile duplication
  28. 28. Which code is covered</li></li></ul><li>Ways to Reduce Test Case Bloat<br />Customer Workflow Based<br /><ul><li>Usage data
  29. 29. Sales info
  30. 30. Meeting users
  31. 31. Studies
  32. 32. Surveys</li></li></ul><li>Section 4: Letting Go<br />“The world changes, and all that once was strong now proves unsure.”<br />J.R.R. Tolkien<br />
  33. 33. Does it still matter?<br />Risk:<br />High defect density.<br />Developers are worried about it.<br />Area of integration.<br />Area undergoing change.<br />Provides code coverage.<br />Data showing it is used.<br />Bottom line impact.<br />
  34. 34. Does it still matter?<br />Protecting Your Peeps:<br />Impacts a customer workflow.<br />Barrier to adoption.<br />Top customer issue.<br />Silent failure is catastrophic.<br />Stakeholders still care about it.<br />
  35. 35. Letting Go<br />When I let go of what I am, I become what I might be.- Lao Tzu <br />You do not need a parachute to skydive. You only need a parachute to skydive twice.<br />-Unknown (likely a tester)<br />
  36. 36. Section 5: Human Elements and Change Management<br />“Valour needs first strength, then a weapon.”<br />J.R.R. Tolkien<br />
  37. 37. Why it may be hard<br />What if we choose wrong?<br />Management would rather hear that we ran out of time for the low priority tests rather than that we decided not to test them.<br />Not everyone agrees on what is important.<br />Some are ethically opposed to this idea.<br />No one wants to lose value.<br />
  38. 38. Why it may be hard<br />Feels safer to just keep them.<br />We paid good money for these test cases!<br />We’ve invested too much.<br />No time to deal with this!<br />It would be wasteful.<br />We may need these some day.<br />
  39. 39. Other Ideas<br />More Automation<br />Ok, but is it maintainable?<br />Will it have value in time?<br />You are making it instead of?<br />Who runs, isolates, owns, maintains it?<br />What about changes to what it relies on?<br />More Testers<br />Ok, but are they trained?<br />Do they save or cost time?<br />Do they come with more tests implied?<br />More or less risk?<br />
  40. 40. Other Ideas<br />Do nothing<br /><ul><li>Status Quo may be ok.
  41. 41. Powerless to change process.
  42. 42. Quality not culturally important.
  43. 43. Bloat is acceptable.
  44. 44. Change not worth the political cost.
  45. 45. Acceptance may be enough.</li></ul>Bloat is the norm for the majority of people and now over 50% of all household pets. It is hard to change. Long term success is the most difficult part.<br />
  46. 46. Section 6: “Go forth and be spectacular!”-Gretchen Sidener<br />“Still round the corner there may wait, A new road or a secret gate.”<br />J.R.R. Tolkien<br />
  47. 47. Now What? Now success!<br />The hard part<br />There is no after. Only during.<br />Evaluate what worked.<br />Identify what didn’t apply.<br />Test cases usually shouldn’t live forever.<br />Pruning test cases is part of test planning.<br />Know when to start over.<br />Share your own ideas with others!<br />The good news<br />There is no failure, only “another fine learning opportunity”.<br />
  48. 48. Section 7: Disclaimer and Cautionary Note<br />“...for advice is a dangerous gift, even from the wise to the wise, and all courses may run ill.”<br />J.R.R. Tolkien<br />
  49. 49. Choose Wisely<br />The same level of risk is not appropriate for all situations.<br />Failure to choose gives you fewer options. This experience report is intended to expand your options, not as a directive.<br />These ideas have been tested in one context and worked well for my team. Use at your own discretion. No warrantees implied.<br />
  50. 50. Summary<br />Bloat happens to good people, projects, and pets.<br />With courage, information, and your own plan you can deal with test case bloat on your own terms.<br />A whole community of testers is here to help.<br />Each iteration is a new chance.<br />
  51. 51. Followup-Technical Details<br />VARIATIONS ON BLOAT<br /><ul><li>Jean Hartmann’s work (Microsoft) see
  52. 52. Reducing Test Case Bloat Technical Paper, Version 1.0
  53. 53. Visual Representations of Data</li></li></ul><li>Q& Twitter @lanettecreamBlog at<br />