Quality Tree Software, Inc.
Copyright (c) 2004 - 2005, Quality Tree Software, Inc. 1
Poll: How Agile Is Your Testing Now?
How many of you…
Would call your test practices “Agile”?
Participate in defining the acceptance criteria?
Routinely test incomplete software to provide feedback to
your stakeholders sooner?
Have adopted a lightweight documentation style?
Have found that your test artifacts are relatively easy to
update even when there are radical changes to the code?
Have access to the source code and to the same set of
tools as the programmers?
Have ensured the programmers have full access to all the
artifacts and tools you use?
Are co-located with the project team?
Copyright (c) 2004 - 2005, Quality Tree Software, Inc. 2
Agile is more than a buzzword. It is a relentless
focus on providing business value, usually by
adopting one or more Agile methodologies such as
Scrum or XP.
– See the Agile Manifesto:
– And the Agile Alliance:
Copyright (c) 2004 - 2005, Quality Tree Software, Inc. 3
Examples of Agile Methodologies
Lean manufacturing Lightweight
concepts applied to management
Lightweight set of Programming (XP)
development Rigorous set of
practices. practices designed to
keep both the code
and team agile.
Copyright (c) 2004 - 2005, Quality Tree Software, Inc. 4
• Communication and • Feedback
collaboration • Whole team thinking
• Visible indicators • Short iterations
• Disciplined development • Low overhead, high
Copyright (c) 2004 - 2005, Quality Tree Software, Inc. 5
Calling It “Agile” Doesn’t Make It So
This is NOT Agile:
1. Compress the schedule
2. Toss out the
3. Code up to the last minute
The organization may gain
short term speed but at the cost of
long term pain.
Copyright (c) 2004 - 2005, Quality Tree Software, Inc. 6
How Traditional Test Practices Evolved
With great optimism and the best of
intentions, The Project Plan is announced:
Analyze Design Code Test/Bug Fix
Requirements Completed Code
handed off to Dev handed off to Test
Copyright (c) 2004 - 2005, Quality Tree Software, Inc. 7
How Traditional Test Practices Evolved
Inevitably, The Project Plan is revised:
Analyze, Design, & Code Test/Bug Fix
handed off to Test
Copyright (c) 2004 - 2005, Quality Tree Software, Inc. 8
The Result: Practices Intended to Control Chaos
Traditional test practices attempt to manage the chaos (or
at least avoid the blame):
• “Last Defender of Quality” stance
• Strict change management
• Detailed preparation and up front planning
• Heavyweight documentation suitable for outsourcing
the test effort
• Strict entrance and exit criteria with signoffs
• Heavyweight test automation focused on regression
Copyright (c) 2004 - 2005, Quality Tree Software, Inc. 9
Becoming Agile: Shifting Roles
from last line of
defense… …to team support
Copyright (c) 2004 - 2005, Quality Tree Software, Inc. 10
Testing Directions: Marick’s Model
Copyright (c) 2004 - 2005, Quality Tree Software, Inc. 11
Traditional & Agile Testing Contrasted
Traditional Testing Agile Testing
Change Manage & control it. Accept it.
Planning Comprehensive up front test Plan as you go.
Documentation Verbose. Only as much as absolutely
Handoffs Formal entrance and exit criteria It’s not a relay race.
with signoffs. Collaborate.
Automation System-level, built by tool All levels, built by anyone,
specialists, created after the code an integral part of the project.
Copyright (c) 2004 - 2005, Quality Tree Software, Inc. 12
Change Embrace Change: Plan Tests for Maintainability
When creating test artifacts…
• Minimize duplication.
Thought exercise: if a feature were
removed from your software, how many
test artifacts would have to be updated?
• Use tools designed for change.
Hint: if the vendor says “stabilize the
interface first,” run away!
Copyright (c) 2004 - 2005, Quality Tree Software, Inc. 13
Change How Not to Plan a Test Effort
Unfortunately, this is common practice:
• Download a generic, standard, formal test plan
template from the Internet.
• Fill in every section in the template even if you
don’t understand it and have to make stuff up.
• Kill the maximum possible number of trees by
distributing copies to every team member.
The result is a large and mostly irrelevant plan
that must now be maintained. Yuck.
Copyright (c) 2004 - 2005, Quality Tree Software, Inc. 14
Plan as Little as Possible
A little planning is good. More is not better.
• Know who you serve (primarily).
Technology-facing or business-facing?
• Plan for the current iteration.
Speculative planning means rework.
• Have a strategy that fits on one page.
If it is still relevant in 6 months, it’s probably at
the right level of detail.
• Keep an up-to-date, prioritized risks list.
What kind of information are the testers looking
for? The risks list covers it.
Copyright (c) 2004 - 2005, Quality Tree Software, Inc. 15
Change Monitor Documentation Costs
How Much Time Are We Spending on Test Documentation?
# Testers Reporting
0 10 20 30 40 50 60 70 80 90 100
How Much Does Documentation Cost?
Informal polling of 162 software testers from
65 companies revealed that most spend more
than 33% of their time documenting tests.
Copyright (c) 2004 - 2005, Quality Tree Software, Inc. 17
Change Keep the Documentation Simple
• Capture the essence, not the details.
Step-by-step instructions cost time without
providing value (usually).
• Point to other project documents.
If it’s in the user guide, requirements,
specs, etc., leave it there.
• Centralize generic tests in a checklist.
Try this: count the number of times you find
a common condition, like invalid dates or
null strings, in the test docs. More than
once is too many.
Copyright (c) 2004 - 2005, Quality Tree Software, Inc. 18
Watch Verbose Wording
Which would you rather maintain?
Choose the Import option from the File menu.
A dialog titled “Import File” appears. Navigate
to xyzlong.dat and click Open. A dialog
titled “Importing…” appears with the current
status, a progress bar, and a button labeled
“Cancel.” Click on the Cancel button. Choose
OK on the confirmation dialog. Verify that the
Start a long import. Cancel it in the
middle. Verify it stops.
Copyright (c) 2004 - 2005, Quality Tree Software, Inc. 19
Change A Lightweight Example
Copyright (c) 2004 - 2005, Quality Tree Software, Inc. 20
Change Do Away with the Signoff Sheet
Hint: when people joke about
signing the handoff form in
blood, the signoff process is
more about blame than about
producing good software.
Copyright (c) 2004 - 2005, Quality Tree Software, Inc. 21
Testing is not a phase. It’s a way of life.
• Agile teams are test infected.
The question, “How will we test it?” is as
important as “How will we build it?”
• Co-locate testers and programmers. But
sitting side by side does not ensure
• Track testing status and programming
status all together.
Show tests run-passed-failed together with
features/stories done and left to do.
Copyright (c) 2004 - 2005, Quality Tree Software, Inc. 22
Leverage Automation Investments
Automate everything you can, but invest wisely.
• Collaborate with programmers on test
The programmers have already automated the
unit tests. Why not reuse the investment where
• Use different types of automated tests for
Automated system tests should cover end-to-end
sequences. Unit tests detect unintended change.
Don’t substitute one for the other.
Copyright (c) 2004 - 2005, Quality Tree Software, Inc. 23
Change Automation & Exploratory Testing
Use test automation to support early
Traditional test wisdom says we can’t start
testing a feature until it’s accessible from
an external interface (like a GUI). But we
don’t have to wait. Test automation can
facilitate manual exploration.
Copyright (c) 2004 - 2005, Quality Tree Software, Inc. 24
Agile Testing Synthesized
• Communication and • Early involvement
collaboration • Leveraged automation
• Whole team thinking • Focus on providing rapid
• Low overhead, high feedback to key stakeholders
Copyright (c) 2004 - 2005, Quality Tree Software, Inc. 25
Necessary Conditions for Agile Testing
1. The organization is willing to embrace agility as
defined by the Agile Manifesto.
Saying “Be More Agile” or “Test Faster” isn’t enough.
2. The whole team is responsible for quality, not just the
testers or people with “Quality” in their title.
Which are you more likely to hear: “How did you miss
that bug?!?” or “How did we not catch that?”
3. Everyone tests, not just designated testers.
Agile teams are “test infected.”
4. Managers focus on fixing problems, not blame. Agile
practices don’t provide CYA paper trails and are
unlikely to succeed in a high-blame, high-fear
Copyright (c) 2004 - 2005, Quality Tree Software, Inc. 26
Agile Testers Wanted: A Job Description
• Perform manual/exploratory tests on early-stage code
• Create automated acceptance tests
• Advise the team about overall risks and trends
• Assist the business stakeholders define acceptance criteria
• Facilitate communication between the technical & business
• Experience designing and executing tests
• Scripting skills in at least one of the following languages:
programming in Java, C#, etc. a plus.
• Strong analysis skills
• Ability to work in a team (bullpen) environment
Copyright (c) 2004 - 2005, Quality Tree Software, Inc. 27
Test Teams that Embrace Agility…
• Focus on providing value to our key
stakeholders (both business-facing and
• Shift from being the last line of defense to
providing an information service.
• Aggressively reduce time and resources
spent on anything that does not directly
contribute to providing information.
• Collaborate with programmers to improve
testability and leverage test automation
Copyright (c) 2004 - 2005, Quality Tree Software, Inc. 28
Beck, K. (1999). Extreme Programming Explained:
Embrace Change. Addison-Wesley.
Cockburn, A. (2004). Crystal Clear: A Human-
Powered Methodology for Small Teams. Addison-
Crispin, L., & House, T. (2002). Testing Extreme
Poppendieck, M. & Poppendieck, T. (2003). Lean
Software Development. Addison-Wesley.
Schwaber, K. & Beedle, M. (2001). Agile Software
Development with SCRUM. Prentice Hall.
Copyright (c) 2004 - 2005, Quality Tree Software, Inc. 29
Many thanks to early reviewers of the ideas
presented here: Brian Marick, William Wake,
Jonathan Kohl, Jeffrey Fredrick, Daniel Knierim,
Marc Kellogg, Danny Faught, Ron Jeffries,
Hubert Smits, Rob Mee, Sherry Erskine, Amy Jo
Esser, Gunjan Doshi, Dave Liebreich, Janet
Gregory, Chris McMahon
Copyright (c) 2004 - 2005, Quality Tree Software, Inc. 30