Testing Does Not Equal Quality


Published on

Presentation for SQuAD on May 12, 2009.


In an agile environment, testing alone is not sufficient to ensure quality. Many other factors come into play to ensure true quality. This session will explore the meaning of "quality" and give some examples of how it can be enhanced. Richard and Bob will also explain the principles behind an agile process which achieves a quality product as well as a potential workflow to implement the process. This will include how QA integrates with the team to avoid creating a "mini-waterfall" situation.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Photo: Tug o’ war between speed/value and quality
  • This is the structure/agenda for the rest of the presentation
  • Briefly dig into “valuable” and “now and in the future.” Big idea: quality is about ensuring business value and speed/sustainability.
  • Explain the cumulative cash flow chart, what it means. Wave at how quality might change its shape.
  • The impact of quality on value is clearer if we just look at value over time non-cumulatively. Here’s an example of a value flow for a feature that works as intended, the counterpart to the cumulatively cash flow in the previous slide.
  • straight line at 0 – there’s never any value
  • Less value than the original value flow, and goes up and down over time.
  • Value flow with gaps
  • Increasing value flow with a cliff when the system gets overloaded.
  • We’ve been looking at the value over time of a feature. We can also look at the value produced over time by a team. We normally call this “velocity.”
  • Slightly increasing, pretty stable velocity
  • Decreasing velocity, as tech debt increases the cost of new features over time.
  • Quality is not something you add on top of value. It only makes sense in terms of value now and in the future, as revealed by feature value flows and team value flows. (Flip side of this: Anything we would do in the name of quality that doesn’t ensure value now and in the future is probably waste.)
  • Inspect for quality: PO asks for something. Dev builds what they think PO wants. Test inspects for what they think PO wants. Loops ensue. Sometimes it works. More often, it doesn’t. Build quality in: Use tests as specs. Engage QA/testers with PO and dev at the start of a feature instead of just at the end.
  • Moving testing to the front is good. But if our testing is all or mostly manual, we have a problem after just a few iterations.
  • If you automate your tests as you build them, this iteration’s new tests become next iteration’s regression tests, but you only need to build them once. Also, you give your devs something they can run to ensure that their code passes all the tests before they call their work done. This reduces the loops we mentioned a moment ago.Note, however, automation can’t be much more expensive than manual testing or the whole thing falls apart. This drives tool choice. So does the need to write automated tests against not-yet-existent features. No time to talk about it here, but there’s a reason certain test tools are more popular in the agile community and others are more popular in the waterfall world: your tool can make or break your success with agile testing.
  • Remember, features are in priority order, so the most valuable feature is the most tested feature.
  • We’ve moved testing to the front and started automating our tests. But if it still takes a day or two to get code built and into a test environment, completing a story can take twice as long. If the process is manual and error-prone or if you regularly find defects post-deployment, now you have loops. Add in multitasked RM people who aren’t available when you need them, and your 2 day story easily takes 6-8. In a two week sprint, you’re not getting much done.
  • Testing Does Not Equal Quality

    1. 1. Testing != Quality<br />Rethinking quality and how to achieve it in an agile world<br />Bob Hartman<br />Richard Lawrence<br />
    2. 2. “How does QA fit into all of this?” <br />
    3. 3. Speed/value<br />and <br />quality <br />are conflicting goals<br />
    4. 4. Or are they?<br />
    5. 5. What is quality?<br />How do we ensure <br />our software has it?<br />How can quality enable speed and value?<br />
    6. 6. What is quality?<br />
    7. 7. The goal of a software org:Produce valuable software now and in the future.<br />
    8. 8. A feature can be modeled with a cumulative cash flow chart<br />
    9. 9. Or, we can remove the investment and just look at value flow<br />
    10. 10. Value flow for a feature that never works right<br />
    11. 11. Value flow for a feature that fails for some users some of the time<br />
    12. 12. Value flow for a feature with downtimes<br />
    13. 13. Value flow for a feature that doesn’t scale<br />
    14. 14. Quality is the foundation of value<br />
    15. 15. A team’s productivity can be measured with a velocity chart<br />
    16. 16. Velocity of a team with a well-designed, high quality system<br />
    17. 17. Velocity of a team with a poorly-designed, low quality system<br />
    18. 18. Quality is the foundation of speed<br />
    19. 19. Photo credit: woodsy on sxc.hu<br />Quality is what ensures: Value now and in the future<br />
    20. 20. How do we ensure our software has it?<br />
    21. 21. Testing at the end doesn’t work<br />Iteration Start<br />Iteration End<br />Plan<br />Dev<br />Test<br />The Plan<br />The Reality<br />
    22. 22. What’s really going on?<br />Dev<br />Test<br />The Plan<br />Dev to test is really a loop<br />Dev<br />Test<br />The Reality<br />Dev expands to fill the available time<br />
    23. 23. Build quality in vs. inspecting for it at end<br />Dev<br />Test<br />Plan<br />Inspect for quality<br />Build quality in<br />
    24. 24. Manual testing doesn’t scale<br />(or happens in overtime)<br />Testing that doesn’t happen, but should<br />Regression testing<br />Testing capacity<br />New feature testing<br />Sprint 1<br />Sprint 2<br />Sprint 3<br />Sprint 4<br />Sprint 5<br />
    25. 25. Automate and get regression tests for free<br />Automated tests that are now regression tests<br />Testing capacity<br />New feature testing<br />Sprint 1<br />Sprint 2<br />Sprint 3<br />Sprint 4<br />Sprint 5<br />
    26. 26. What is really being tested?<br />Iterations<br />Features<br />
    27. 27. Which features need to work best?<br />Usage of features in a typical software product<br />
    28. 28. But without regression it is different<br />Iterations<br />Features<br />
    29. 29. Minimal regression is a little better<br />Iterations<br />Features<br />
    30. 30. But what will happen where we don’t have regression tests running?<br />Iterations<br />Features<br /> = Missed tests, which Murphy’s Law says will lead to defects being missed!<br />
    31. 31. Build and deploy processescan kill speed<br />DONE<br />The Plan<br />Define acceptance criteria and automate tests<br />Develop<br />Package<br />Deploy to test env<br />Test<br />Loops, waits, errors…more time<br />The Reality<br />
    32. 32. To ensure quality in an agile world:<br />Move testing to the front of the process<br />Collaborate<br />Automate tests (using agile-friendly tools)<br />Automate/accelerate build and deploy<br />Work on user stories one at a time, all the way to DONE<br />
    33. 33. Questions?<br />www.agilebob.com<br />bob.hartman@agileforall.com<br />www.richardlawrence.info<br />richard@humanizingwork.com<br />