The document discusses an Agile test strategy approach for testing projects using Agile methodologies like Scrum. It proposes using project profiling to understand the context of a project and identify suitable testing interventions. Some key interventions discussed are story challenge and definition during sprint planning, daily stand-ups to report progress, story refinement throughout sprints, developer testing of code, and integration, system and user acceptance testing after each sprint. The strategy aims to blend testing activities seamlessly into each sprint and increase testing scope and coverage over time as the product is developed iteratively.

1. AgileTest Strategy
@paul_gerrard
Paul Gerrard
paul@gerrardconsulting.com
gerrardconsulting.com

2. Overview
• What is AgileTest Strategy?
• Project Profiling
• (Test Strategy as) Agile Interventions
• Test Automation
• What’s Left?
• Summary
• Q&A
Intelligent Definition and Assurance Slide 2

3. What is AgileTest
Strategy?
Agile Strategy – an oxymoron?

4. Agile Test Strategy
• What do we mean by this?
1. AGILE Test Strategy – how to create a test
strategy in an Agile way?
2. AGILETest Strategy – a test strategy for an
Agile project?
• We’ll look at how we created an Agile
approach to strategy, but we’ll spend more
time on strategy for an Agile project.
Intelligent Definition and Assurance Slide 4

5. Google “Agile Test Strategy”
• There are plenty of recipes out there
• Most offer a selection of techniques but don’t
provide much guidance on how to blend them
• We need to know how to make choices, not
just know what choices exist
• Strategy is a thought process, not a document
– Although you might document the ideas for reuse, as
a reminder or ‘for the record’.
Intelligent Definition and Assurance Slide 5

6. Agile governance
• Governance is the act of governing. It relates
to decisions that define expectations,
grant power, or verify performance
Wikipedia
• Define expectations – DEFINITION of need
• Grant power – DELEGATION to a project
team
• Verify performance – ASSURANCE of solution.
Intelligent Definition and Assurance Slide 6

7. Strategy helps you decide what to do
A. The strategy presents some decisions that can be
made ahead of time
B. Defines the process or method or information that
will allow decisions to be made (in project)
C. Sets out the principles (or process) to follow for
uncertain situations or unplanned events
• In a structured/waterfall environment, most questions
answered off-the-shelf – “A-style, ready for anything”
• In an Agile environment – might have some ready-made
policies but we manage scope and adapt (mostly C?)
Intelligent Definition and Assurance Slide 7

8. Contexts of Test Strategy
Test
Strategy
Risks
Goals
Constraints
Human
resource
Environment
Timescales
Process
(lack of?)
Contract
Culture
Opportunities
User
involvement
Automation
De-
Duplication
Early
Testing
Skills
Communication
Axioms
Artefacts

9. Traditional v Agile test strategy
• Traditional – structured, goal/risk-driven
– Identify stakeholders; what are their goals?
– Product risk analysis
– Allocate risks/goals to test stages
– Formulate test stage definitions (entry/exit criteria,
environments, tools etc. etc.
• Agile – interventionist, consensus-driven
– Project profiling to set the testing theme
– Identify testing interventions in the Agile process
– Test policy overlays the process; catches exceptions
Intelligent Definition and Assurance Slide 9

10. Project Profiling
Select a profile for your project first,
then choose the aspects of test
strategy that suite your project

11. Template-driven? Bah!
• So this is just a template copy and edit process?
• Won’t you always end up with the same document?
• Profiling doesn’t need to be prescriptive
– No need to write a document if you don’t need to
– But if company policy or common sense dictates certain
approaches, save yourself some time
– Create a set of deeper, more detailed questions to be
answered (Pocketbook)
• Profilers are really just checklists: heuristic guidelines
designed to help you make choices and trade-offs.
Intelligent Definition and Assurance Slide 11

12. Cerise
Orange
Green
Test Plan
Items
Product
Risks
Project Profiler
Risk Profiler
Project Plan
Test
Assurance
Project Manager
Business, Project Team
and Boards
Consultation
Blue
Unknowns
Using a Project Profiler to Derive
a Test Strategy and Project Plan
(A government client example)
The Project Profiler (with Test Assurance) helps
Project Managers to:
• Select a project style that fits (Waterfall or Agile)
• Identify the product risks that need testing
• Identify test activities to include in project plans
• Carefully define the scope of the project
Environment
Story Guideline
Tools
Incident Mgt.
Waterfall
Test Strategy
SCRUM/Agile
Test Strategy
Intelligent Definition and Assurance 12

13. Project profiling process
Task
1 Have the Information you need to hand
2 Project Profiler (section 3):
Select the statements that best match your project context. The Blue column indicates
that you need more information – consult your stakeholders, team or relevant Board(s).
3 Generic Risk Profiler (section 4):
Consider the generic project risks – which are significant for your project? Can testing
help?
4 Product Risk Profiler (Section 5):
Consider the functional and non-functional risks that most concern your stakeholders –
should they be tested?
5 Actions and Test Activities (Section 6):
Consider the actions that prompt your ‘next steps’ and the test activities that should be
incorporated into your project plan.
6 Create your Test Strategy from the Test Strategy Framework Template
7 Incorporate the activities from stage 5 and identified in 6 into your Project
Plan.
Intelligent Definition and Assurance 13

14. Project Profiler (part of section 3)
Project Aspect Cerise Orange Green Blue
Responsibility for
Acceptance
Users will take
responsibility for UAT;
they have UAT experience
Users will be responsible
for UAT but have no test
experience
Users will take part in UAT
or witness tests at critical
periods, and will review
the outcome
Users are unwilling/unable
to take part in UAT;
reluctant to make the
acceptance decision or not
known
Requirements (Sources
of Knowledge)
New system replaces a
well-understood existing
system; users have clear
vision of system goals and
prefer to document their
requirements up-front
Users want to collaborate
to jointly define
requirements and meet
them incrementally or
iteratively
Users put the onus of
requirements elicitation on
the project; requirements
and the solution will evolve
Inexperienced users who
are unable or unwilling to
collaborate with
requirements gathering
Requirements Stability New system is a functional
replacement of an existing
system or a well-defined
process (requirements can
be fixed early on)
New system replaces an
existing system with
enhancements or an
established (but not
necessarily documented
process)
New system supports a
new business need;
business process exists
but will change/evolve;
users have experience of
requirements
New system supports a
new business need;
business process is not
yet known; users have no
experience or
requirements
Visibility, Formality High visibility/risk to
general public; formal
progress reporting
required at board level;
fixed scope and
deliverables; formal
approvals and sign-offs
High visibility/risk to
business; formal progress
reporting required; some
defined deliverables, some
deliverables will
emerge/evolve; some
approvals and sign-offs
Relatively low business-
risk; informal progress
reporting is acceptable;
partial solution may
suffice,
incremental/iterative
delivery
Potentially, high visibility,
high risk project, uncertain
impact on the business
External Dependencies More than one or new
external suppliers
responsible for
development (and supplier
testing)
Single, known supplier
responsible for
development (and supplier
testing)
In-house development, no
external dependencies
Dependencies on external
suppliers, their
responsibilities or
competence not yet known
Etc. Etc. Etc. Etc. Etc.
Intelligent Definition and Assurance 14

15. Project types - examples
Cerise Structured, waterfall style of project (and includes
COTS projects)
Orange Iterative/prototyping style of project using SCRUM in
a formal way and having dedicated resources for the
Business Analyst and Tester roles.
Green A project using SCRUM in a less formal way but not
having dedicated resources for the Business Analyst
and/or the Tester roles.
Blue Blue column statements describe where there is
insufficient information available to identify the style
of project and the recommendation must be that
some further investigation is required.
Intelligent Definition and Assurance 15

16. (Test Strategy as)
Agile Interventions
I’m using Scrum/Sprint terminology,
but you don’t have to of course

17. Interventions (a government client example)
• On the following
slides, we highlight
8 interventions
• Some are test
phases, but some
aren’t
No. Activity When?
1 Story Challenge As stories are added to the
Product Backlog
2 Story Definition As stories are added to a
Sprint Backlog
These
activities
are
repeated
for
each
Sprint
iteration
3 Daily Stand-Up Once per day during the
Sprint
4 Story Refinement Occurs throughout the Sprint
as new information emerges
5 Developer Testing Occurs throughout the Sprint
as the developer codes the
stories
6 Integration (and
incremental
System) Testing
During and at the end of
each sprint, including the
final sprint
7 System Testing At the end of each sprint,
including the final sprint
8 User Acceptance
Testing
At the end of each sprint,
including the final sprint
9 Non-functional
Testing and Pre-
Production Testing
Expected to take place on an
as-needs basis.
Intelligent Definition and Assurance Slide 17

18. Integration into
Existing Code base
Automated testing
New Code
8. User Test
7. System Test
Sprint 1
Developed Stories Developed Stories Developed Stories
Sprint 3
Sprint 2
Sprint Backlog Sprint Backlog Sprint Backlog
1. Story Challenge
Suggest ‘what-ifs’ to
challenge new stories
and define story
headlines
Increasing
Scope of
Sys. Test
and UAT
Increasing Scope of Integration,
System and Users Testing
2. Story Definition
Introduce scenarios
to enhance the
Acceptance Criteria
Complete Tests after
Final Sprint
Project Level Test Activities
(This diagram shows three sprints, but there could be more or fewer)
6. Integration Test
6. Integration Test
6. Integration Test

19. Integration into
Existing Code base
Automated testing
New Code
8. User Test
7. System Test
Sprint 1
Developed Stories Developed Stories Developed Stories
Sprint 3
Sprint 2
Sprint Backlog Sprint Backlog Sprint Backlog
1. Story Challenge
Suggest ‘what-ifs’ to
challenge new stories
and define story
headlines
Increasing
Scope of
Sys. Test
and UAT
Increasing Scope of Integration,
System and Users Testing
2. Story Definition
Introduce scenarios
to enhance the
Acceptance Criteria
Complete Tests after
Final Sprint
Project Level Test Activities
(This diagram shows three sprints, but there could be more or fewer)
6. Integration Test
6. Integration Test
6. Integration Test

20. Integration into
Existing Code base
Automated testing
New Code
8. User Test
7. System Test
Sprint 1
Developed Stories Developed Stories Developed Stories
Sprint 3
Sprint 2
Sprint Backlog Sprint Backlog Sprint Backlog
1. Story Challenge
Suggest ‘what-ifs’ to
challenge new stories
and define story
headlines
Increasing
Scope of
Sys. Test
and UAT
Increasing Scope of Integration,
System and Users Testing
2. Story Definition
Introduce scenarios
to enhance the
Acceptance Criteria
Complete Tests after
Final Sprint
Project Level Test Activities
(This diagram shows three sprints, but there could be more or fewer)
6. Integration Test
6. Integration Test
6. Integration Test

21. Integration into
Existing Code base
Automated testing
New Code
8. User Test
7. System Test
Sprint 1
Developed Stories Developed Stories Developed Stories
Sprint 3
Sprint 2
Sprint Backlog Sprint Backlog Sprint Backlog
1. Story Challenge
Suggest ‘what-ifs’ to
challenge new stories
and define story
headlines
Increasing
Scope of
Int. Sys.
and UAT
Increasing Scope of Integration,
System and Users Testing
2. Story Definition
Introduce scenarios
to enhance the
Acceptance Criteria
Complete Tests after
Final Sprint
Project Level Test Activities
(This diagram shows three sprints, but there could be more or fewer)
6. Integration Test
6. Integration Test
6. Integration Test

22. Integration into
Existing Code base
Automated testing
New Code
8. User Test
7. System Test
Sprint 1
Developed Stories Developed Stories Developed Stories
Sprint 3
Sprint 2
Sprint Backlog Sprint Backlog Sprint Backlog
1. Story Challenge
Suggest ‘what-ifs’ to
challenge new stories
and define story
headlines
Increasing
Scope of
Int. Sys.
and UAT
Increasing Scope of Integration,
System and Users Testing
2. Story Definition
Introduce scenarios
to enhance the
Acceptance Criteria
Complete Tests after
Final Sprint
Project Level Test Activities
(This diagram shows three sprints, but there could be more or fewer)
6. Integration Test
6. Integration Test
6. Integration Test

23. Daily Scrum
Stand-Up
Meeting 24
Hours
2-4 Weeks
Backlog tasks
expanded
by team
Potentially Shippable
Product increment
Product backlog
As prioritised by Product Owner
Sprint Backlog
4. Story Refinement
Refine scenarios to
enhance story definition,
create system tests as
stories, as required
6) Integration/System Testing
Incorporate automated unit
tests into the CI regime.
On weekly basis and at end of
Sprint, deploy to System test
environment and tester runs
system tests.
3. Daily Stand-Up
Report anomalies found,
stories tested, amended,
created
5) Developer Testing
Private ad-hoc tests and
build/run automated unit
tests
Test Activities in the Sprint
Intelligent Definition and Assurance 23

24. Daily Scrum
Stand-Up
Meeting 24
Hours
2-4 Weeks
Backlog tasks
expanded
by team
Potentially Shippable
Product increment
Product backlog
As prioritised by Product Owner
Sprint Backlog
4. Story Refinement
Refine scenarios to
enhance story definition,
create system tests as
stories, as required
6) Integration/System Testing
Incorporate automated unit
tests into the CI regime.
On weekly basis and at end of
Sprint, deploy to System test
environment and tester runs
system tests.
3. Daily Stand-Up
Report anomalies found,
stories tested, amended,
created
5) Developer Testing
Private ad-hoc tests and
build/run automated unit
tests
Test Activities in the Sprint
Intelligent Definition and Assurance 24

25. Daily Scrum
Stand-Up
Meeting 24
Hours
2-4 Weeks
Backlog tasks
expanded
by team
Potentially Shippable
Product increment
Product backlog
As prioritised by Product Owner
Sprint Backlog
4. Story Refinement
Refine scenarios to
enhance story definition,
create system tests as
stories, as required
6) Integration/System Testing
Incorporate automated unit
tests into the CI regime.
On weekly basis and at end of
Sprint, deploy to System test
environment and tester runs
system tests.
3. Daily Stand-Up
Report anomalies found,
stories tested, amended,
created
5) Developer Testing
Private ad-hoc tests and
build/run automated unit
tests
Test Activities in the Sprint
Intelligent Definition and Assurance 25

26. Daily Scrum
Stand-Up
Meeting 24
Hours
2-4 Weeks
Backlog tasks
expanded
by team
Potentially Shippable
Product increment
Product backlog
As prioritised by Product Owner
Sprint Backlog
4. Story Refinement
Refine scenarios to
enhance story definition,
create system tests as
stories, as required
6) Integration/System Testing
Incorporate automated unit
tests into the CI regime.
On weekly basis and at end of
Sprint, deploy to System test
environment and tester runs
system tests.
3. Daily Stand-Up
Report anomalies found,
stories tested, amended,
created
5) Developer Testing
Private ad-hoc tests and
build/run automated unit
tests
Test Activities in the Sprint
Intelligent Definition and Assurance 26

27. 4. Story Refinement (example definition)
Objectives  To define acceptance criteria for all stories that are included in a Sprint as they are worked
on by development
 To define scenarios that describe the tests and expected behaviours of the System
 Improve understanding of the requirement and communicate anomalies to developers
 To identify System Tests that exercise functionality of multiple stories that can be system
tested in this sprint
 To assure the completeness for stories in the current Sprint
What’s being tested?  Stories to be included in the current Sprint
Deliverables  Refined story definitions with defined acceptance criteria and scenarios, where appropriate
 System tests
Responsibilities (Orange)  Tester – challenges stories by suggesting potential scenarios, new stories, story merges and
splits; performs ad-hoc testing with/on behalf of developers; assures completeness of
stories.
 Developers – considers stories, evaluates impact on development
 Product Owner or Analyst – collates feedback and decisions on stories
 Product Owner – approves changes to stories, accepts completeness of stories
 Scrum Master – monitors progress; evaluates impact on resources and schedules
Responsibilities (Green)  Not performed in Green projects
Baseline  Story Guideline (reference 3)
Entry Criteria  On commencement of the Sprint
Exit Criteria  When all stories within a Sprint are considered complete
 Queries, anomalies, discrepancies and inconsistencies have been eliminated or explained
 System Tests appropriate to the Sprint have been defined
 Definition of acceptance is agreed with Product Owner
Intelligent Definition and Assurance 27

28. Test Automation
Could you create an AgileTest
Strategy without automation?

29. Brian Marick’sTesting quadrants
Intelligent Definition and Assurance 29

30. Test Automation Pyramid – Lisa
Crispin’s version (Google others)
• Pyramid reflects the
relative numbers of tests
• Focus on
unit/component
– Acceptance of “Services”
• GUI are end-to-end
• Manual checking the
exception?
Intelligent Definition and Assurance 30

31. GUI Test Framework
GUI Test Tool
Browser
Inter/Intranet
HTTP/S
Web Server
App. Server
DB Server
HTTP/S
HTTP Driver
Stories/
Scenarios
Unit Test Framework
Test Code
HTTP/S
Unit Test Framework
Test Code
API
4. Programmers
write low level HTTP
GET/POST calls
through a driver that
simulates a browser
3. Non-Technical
testers write scripts
Tools Experts write
interface
2. Technical Testers
code scripts directly
1. Programmers
write unit tests or
execute embedded
unit tests using a
unit test framework
to test components
Where do you automate?

32. Distributed testing
• Use business stories and scenarios/acceptance
criteria to validate requirements
• Reuse those stories to feed ‘Acceptance-
Driven Development’ BDD/TDD process
• Automated tests are an Anti-Regression tactic
• Automated tests don’t replicate manual tests;
think of them as a ‘change trip-wire’ that
triggers an alarm, if tripped.
Intelligent Definition and Assurance 32

33. Deriving scenarios to test
• To understand feature scope?
• To get stakeholder to accept?
• To validate the requirement?
• To estimate the work to build this feature?
• To system test this feature?
• To unit test this feature?
Scenarios are created to meet several goals
Story Challenge
Story Refinement
Story Definition
Iteration Planning
SystemTesting
DeveloperTesting
Intelligent Definition and Assurance 33

34. What’s Left?

35. Other aspects of test policy
• Definitions (of done etc.)
• Incident management
• Test automation
• Story format e.g. Gherkin
• Environment request and management
• Regression testing (at what levels)
• Test deliverables and documentation.
Intelligent Definition and Assurance Slide 35

36. The Three Amigos
• Business Analyst
– Liaises and manages stakeholders and their needs
– Transforms business requirements into
specification (at multiple levels)
• Developer
– Scopes, designs, builds, tests and delivers features
• Tester
– Challenges the thinking on the project
– Performs ‘Assurance in the small’.
Intelligent Definition and Assurance Slide 36

37. The tester’s contribution to testing
• _________ feature/story acceptance criteria
• _________ the developers to unit test (auto)
• _________ feature/story acceptance
• _________ acceptance test
• _________ service/UI level automation
• Scope: from low-level detail to system integration
• Liaison with integration testers and feedback
• Fill in the blanks yourself; negotiate with your
team.
Intelligent Definition and Assurance Slide 37

38. Summary

39. Close
• Agile test strategy has its place and many aspects
of test can be pre-defined
• Importantly, we use a principled approach to deal
with the unexpected
• Project profiling can help
• Consider testing as interventions, rather than test
phases
• Test automation in the context of Specification by
Example, requirements validation, BDD,TDD.
Intelligent Definition and Assurance Slide 39

40. AgileTest Strategy
@paul_gerrard
Paul Gerrard
paul@gerrardconsulting.com
gerrardconsulting.com