Lanette Creamer's presentation on Reducing Test Case Bloat (version 2.1) from QASIG, March 10, 2010

1. ReducingTest Case Bloat Lanette Creamer Adobe Systems, Inc QASIG March 2010

2. Introduction Tester and Test Lead at Adobe since 2000 September 2009-First Agile project Python scripts, collaboration, user focus, testing the whole system, cat photos. Dysfunctional fiefdoms, clip art, spiders, waiting, retesting.

3. Agenda What is a Test Case? What is Test Case Bloat? Methods to reduce test case bloat in practice. Barriers to reducing bloat. How did it go? Q&A

4. Section One: Why reduce at all? “I have no help to send, therefore I must go myself.” J.R.R. Tolkien

5. We like our bloat How is that working for you?

6. What is a Test? A test is an action which produces discoveries that can be used to evaluate product quality.

7. Test Case Is every test a test case?

8. What is Bloat? Test Case Bloat-More test cases than you can effectively manage* in the time you have. Is all bloat test cases? No. *Manage may imply some or all of the following: cover, update, execute, report on, allocate, run, automate, respond to.

9. Section Two: Find the bloat “All we have to decide is what to do with the time that is given to us.” J.R.R. Tolkien The Fellowship of the Ring 1954

11. Culture

12. Set up

15. Pairwise

16. Random Selection

18. High, Medium, Low

21. Features added

22. Components integrated

25. Date of last failure

29. Sales info

30. Meeting users

31. Studies

33. Does it still matter? Risk: High defect density. Developers are worried about it. Area of integration. Area undergoing change. Provides code coverage. Data showing it is used. Bottom line impact.

34. Does it still matter? Protecting Your Peeps: Impacts a customer workflow. Barrier to adoption. Top customer issue. Silent failure is catastrophic. Stakeholders still care about it.

35. Letting Go When I let go of what I am, I become what I might be.- Lao Tzu You do not need a parachute to skydive. You only need a parachute to skydive twice. -Unknown (likely a tester)

36. Section 5: Human Elements and Change Management “Valour needs first strength, then a weapon.” J.R.R. Tolkien

37. Why it may be hard What if we choose wrong? Management would rather hear that we ran out of time for the low priority tests rather than that we decided not to test them. Not everyone agrees on what is important. Some are ethically opposed to this idea. No one wants to lose value.

38. Why it may be hard Feels safer to just keep them. We paid good money for these test cases! We’ve invested too much. No time to deal with this! It would be wasteful. We may need these some day.

39. Other Ideas More Automation Ok, but is it maintainable? Will it have value in time? You are making it instead of? Who runs, isolates, owns, maintains it? What about changes to what it relies on? More Testers Ok, but are they trained? Do they save or cost time? Do they come with more tests implied? More or less risk?

41. Powerless to change process.

42. Quality not culturally important.

43. Bloat is acceptable.

44. Change not worth the political cost.

45. Acceptance may be enough.Bloat is the norm for the majority of people and now over 50% of all household pets. It is hard to change. Long term success is the most difficult part.

46. Section 6: “Go forth and be spectacular!”-Gretchen Sidener “Still round the corner there may wait, A new road or a secret gate.” J.R.R. Tolkien

47. Now What? Now success! The hard part There is no after. Only during. Evaluate what worked. Identify what didn’t apply. Test cases usually shouldn’t live forever. Pruning test cases is part of test planning. Know when to start over. Share your own ideas with others! The good news There is no failure, only “another fine learning opportunity”.

48. Section 7: Disclaimer and Cautionary Note “...for advice is a dangerous gift, even from the wise to the wise, and all courses may run ill.” J.R.R. Tolkien

49. Choose Wisely The same level of risk is not appropriate for all situations. Failure to choose gives you fewer options. This experience report is intended to expand your options, not as a directive. These ideas have been tested in one context and worked well for my team. Use at your own discretion. No warrantees implied.

50. Summary Bloat happens to good people, projects, and pets. With courage, information, and your own plan you can deal with test case bloat on your own terms. A whole community of testers is here to help. Each iteration is a new chance.

52. Reducing Test Case Bloat Technical Paper, Version 1.0 http://www.pnsqc.org/conference09/Paper_25_slides.pdf