Efficient agile organizations focus on defect prevention rather than downstream defect discovery, because discovering defects during or after testing adds to development costs. Delaying discovery and repair of defects can make an agile team feel like they are operating in a mini-waterfall. Sharing his experience with Scrum/ Kanban teams, Leveraging the lean concept of limiting work in progress, David Jellison describes how grouping defects into two major categories—work-in-progress defects and escaping defects—reduces development costs and improves reliability in the field. Dave illustrates how to manage problem discovery early and minimize the existence of escaping defects. Treating escaping defects as the exception rather than the norm results in a much smaller defect backlog and increased customer satisfaction. This approach encourages increased collaboration between quality engineers and developers, and shifts the focus of team measures from defect counts to product delivery velocity and cycle time, with increased confidence in quality as the work is completed.
StarWest 2012 - Agile Defect Management: Focus On Prevention
1. Agile Defect Management
Focus On Prevention
David Jellison
Director, Quality Engineering
djellison@constantcontact.com
Twitter: davidjellison
2. You Will Learn
Categorizing defects drives defect context
management
Testing is part of delivering a story or feature
Detecting and resolving defects as the work is
delivered prevents backlog
Attaching an in-progress defect to a current story
encapsulates the defect in the current work
Escaping defects become the exception rather than
the norm
2 Twitter: davidjellison
3. The One Finding The Most Bugs Wins!
Downstream Testing Approach
Lord of the Rings: Legolas & Gimli counting kills
Twitter: davidjellison
4. Preventing Bugs is Craftsmanship
upstream Testing Approach
Twitter: davidjellison
5. Quest for Zero Defects
You can only manage what is known
Crosby’s “Zero Defect” standard manages to requirements
(http://en.wikipedia.org/wiki/Zero_Defects)
Agile changes the game, managing business value to
the customer frequently
Need a pattern to manage defects that we inevitably
discover during and after release
Never really reach 0 Defects
5 Twitter: davidjellison
8. Late Defect Discovery Costs
Opportunity cost
Not working on other things
Developer Context switching
Time to get acclimated and then back again
Rebuild and test time
More cycles to deliver the fix
Damage control
Support time and reputation
Other hidden costs…
8 Twitter: davidjellison
9. Traditional Defect Discovery Trends
Defect debt accumulation
Delay in resolution
Defects resolved
Dev/QE aligned
9 Twitter: davidjellison
10. Defect Discovery Improvement Plan
Stop defect debt accumulation
Move defect discovery up in the development cycle
Embed QE into teams with Dev/QE alignment
Prevent defects through design review & test automation
Write failing regression tests as defects are discovered
Test in sandbox
Don’t wait for final integration environment to test
Continuous build and continuous integration patterns
Whole-team responsibility for quality
Clear acceptance criteria for all new work
Everyone tests
Automated test code reviews with developer
10 Twitter: davidjellison
17. Categorizing Defects
Escaping Defects
Discovered defects allowed into the field
Undiscovered defects found in the field
17 Twitter: davidjellison
18. Categorizing Defects
WIP (work in progress) Defects
Defects discovered in new code not yet delivered to field
Becomes additional acceptance criteria for delivery
Treat as subtasks of feature work
Caged by the parent work in progress
18 Twitter: davidjellison
19. Managing Defect Discovery
Its OK to not write
defects if there are
failing automated tests
to manage
Promote failing tests to
WIP defects if need to
manage separately
Promote WIP defects
that are still failing at
time of delivery to
escaping defects and
added to the backlog
19 Twitter: davidjellison
20. Focus on Prevention
Develop failing
regression tests when
discovered manually
Continuous feedback
through failing test run
reports
Associate WIP defects
with feature work in
progress
20
21. Managing Escaping Defects
Delivering business
value over no defects
No unknown failures in
regression tests
Coordinate planned
escaping defects with
customer support
Reduce escaping
defects to less than you
can count on two hands
21
24. Escaping Defect Reduction Plan
Set expectation in the teams that fixing WIP defects
is part of the acceptance criteria
Add escaping defects in the field to the backlog and
rank along with other work
Plan to fix some number of escaping defects from
the backlog in each development cycle
Treat escaping defects as the exception rather than
the norm
24 Twitter: davidjellison
26. Example: WIP Defects/Escaping Defects
Shifting the focus to WIP defects results in far fewer
escaping defects in the field, even with increased
discovery
WIP Defects [blue]
Escaping Defects [red]
26 Twitter: davidjellison
27. Example: WIP Defects/Escaping Defects
Each team may be in a different place with business
risk and can manage escaping defects differently
Develop a theme strategy
WIP Defects [ red & blue]
Escaping Defects [green]
27 Twitter: davidjellison
28. What We Covered
Categorizing defects drives defect context
management
Testing is part of delivering a story or feature
Detecting and resolving defects as the work is
delivered prevents backlog
Attaching an in-progress defect to a current story
encapsulates the defect in the current work
Escaping defects become the exception rather than
the norm
28 Twitter: davidjellison
29. Contact:
David Jellison
Dir, QE
Constant Contact, Inc.
1601 Trapelo Road
Waltham, MA 01451
djellison@constantcontact.com
P: 339-222-5712
Blog: http://davidjellison.wordpress.com
Twitter: davidjellison
http://ConstantContact.com/Careers
29 Twitter: davidjellison
Editor's Notes
Phillip Crosby’s “Absolute Defect Management” with a “Zero Defect” quality standard can only be met when validating against very clear and precise requirements and standards.
The relative cost of defect discovery at least doubles at each stage of .