Downloaded 52 times
Testing Does Not Equal Quality
- 1. Testing != QualityRethinkingquality and how to achieve it in an agile worldBob HartmanRichard Lawrence
- 2. “How does QAfit into all of this?”
- 3. Speed/valueand quality areconflicting goals
- 4.
- 5. What is quality?Howdo we ensure our software has it?How can quality enable speed and value?
- 6.
- 7. The goal ofa software org:Produce valuable software now and in the future.
- 8. A feature canbe modeled with a cumulative cash flow chart
- 9. Or, we canremove the investment and just look at value flow
- 10. Value flow fora feature that never works right
- 11. Value flow fora feature that fails for some users some of the time
- 12. Value flow fora feature with downtimes
- 13. Value flow fora feature that doesn’t scale
- 14. Quality is thefoundation of value
- 15. A team’s productivitycan be measured with a velocity chart
- 16. Velocity of ateam with a well-designed, high quality system
- 17. Velocity of ateam with a poorly-designed, low quality system
- 18. Quality is thefoundation of speed
- 19. Photo credit: woodsyon sxc.huQuality is what ensures: Value now and in the future
- 20. How do weensure our software has it?
- 21. Testing at theend doesn’t workIteration StartIteration EndPlanDevTestThe PlanThe Reality
- 22. What’s really goingon?DevTestThe PlanDev to test is really a loopDevTestThe RealityDev expands to fill the available time
- 23. Build quality invs. inspecting for it at endDevTestPlanInspect for qualityBuild quality in
- 24. Manual testing doesn’tscale(or happens in overtime)Testing that doesn’t happen, but shouldRegression testingTesting capacityNew feature testingSprint 1Sprint 2Sprint 3Sprint 4Sprint 5
- 25. Automate and getregression tests for freeAutomated tests that are now regression testsTesting capacityNew feature testingSprint 1Sprint 2Sprint 3Sprint 4Sprint 5
- 26. What is reallybeing tested?IterationsFeatures
- 27. Which features needto work best?Usage of features in a typical software product
- 28. But without regressionit is differentIterationsFeatures
- 29. Minimal regression isa little betterIterationsFeatures
- 30. But what willhappen where we don’t have regression tests running?IterationsFeatures = Missed tests, which Murphy’s Law says will lead to defects being missed!
- 31. Build and deployprocessescan kill speedDONEThe PlanDefine acceptance criteria and automate testsDevelopPackageDeploy to test envTestLoops, waits, errors…more timeThe Reality
- 32. To ensure qualityin an agile world:Move testing to the front of the processCollaborateAutomate tests (using agile-friendly tools)Automate/accelerate build and deployWork on user stories one at a time, all the way to DONE
- 33.
Editor's Notes
- #4 Photo: Tug o’ war between speed/value and quality
- #6 This is the structure/agenda for the rest of the presentation
- #8 Briefly dig into “valuable” and “now and in the future.” Big idea: quality is about ensuring business value and speed/sustainability.
- #9 Explain the cumulative cash flow chart, what it means. Wave at how quality might change its shape.
- #10 The impact of quality on value is clearer if we just look at value over time non-cumulatively. Here’s an example of a value flow for a feature that works as intended, the counterpart to the cumulatively cash flow in the previous slide.
- #11 straight line at 0 – there’s never any value
- #12 Less value than the original value flow, and goes up and down over time.
- #13 Value flow with gaps
- #14 Increasing value flow with a cliff when the system gets overloaded.
- #16 We’ve been looking at the value over time of a feature. We can also look at the value produced over time by a team. We normally call this “velocity.”
- #17 Slightly increasing, pretty stable velocity
- #18 Decreasing velocity, as tech debt increases the cost of new features over time.
- #20 Quality is not something you add on top of value. It only makes sense in terms of value now and in the future, as revealed by feature value flows and team value flows. (Flip side of this: Anything we would do in the name of quality that doesn’t ensure value now and in the future is probably waste.)
- #24 Inspect for quality: PO asks for something. Dev builds what they think PO wants. Test inspects for what they think PO wants. Loops ensue. Sometimes it works. More often, it doesn’t. Build quality in: Use tests as specs. Engage QA/testers with PO and dev at the start of a feature instead of just at the end.
- #25 Moving testing to the front is good. But if our testing is all or mostly manual, we have a problem after just a few iterations.
- #26 If you automate your tests as you build them, this iteration’s new tests become next iteration’s regression tests, but you only need to build them once. Also, you give your devs something they can run to ensure that their code passes all the tests before they call their work done. This reduces the loops we mentioned a moment ago.Note, however, automation can’t be much more expensive than manual testing or the whole thing falls apart. This drives tool choice. So does the need to write automated tests against not-yet-existent features. No time to talk about it here, but there’s a reason certain test tools are more popular in the agile community and others are more popular in the waterfall world: your tool can make or break your success with agile testing.
- #27 Remember, features are in priority order, so the most valuable feature is the most tested feature.
- #32 We’ve moved testing to the front and started automating our tests. But if it still takes a day or two to get code built and into a test environment, completing a story can take twice as long. If the process is manual and error-prone or if you regularly find defects post-deployment, now you have loops. Add in multitasked RM people who aren’t available when you need them, and your 2 day story easily takes 6-8. In a two week sprint, you’re not getting much done.