Planning Your Agile Testing: A Practical Guide

1/22/2013

Star Canada 2013
Janet Gregory, DragonFire Inc.
Copyright 2013

With material from Lisa Crispin

My experience comes …
As a tester, working on agile teams
Coaching and training, learning
Agile Testing: A Practical Guide for Testers and Agile Teams;
Addison Wesley 2009
Copyright 2013 DragonFire Inc.

1

1/22/2013

How many years experience with agile?
◦
◦
◦
◦
◦

At least 2 years
Less than 2 years
Less than 1 year
Have done some reading, starting in a team
Completely new to the concept

Are you a ...
◦
◦
◦
◦

Tester
Test Manager
Programmer
Other


3

10 minutes ~1 minute / person
Name
Where you are from?
Why you are here?
Let everyone have a chance to introduce themselves


4

2

1/22/2013


•
•
•
•
•
•

5

have short iterations
encourage active customer participation
demand whole team collaboration
test features & stories as they are coded
deliver business value at regular intervals
adapt their processes based on feedback


6

3

1/22/2013

Sequential, phased – eg. waterfall

Requirements
Specifications

Code
Testing
Release

Time

D
C

C
A
It 0

B

It 1

A

B
It 2

A

Agile: iterative, incremental

F
E
D
C
B

It 3

A

– Each story is expanded, coded and
tested
– Possible release after each iteration
B

It 4

7

What concerns you
about agile
and
test planning?


4

1/22/2013

Test Approach – The Agile Way
Project Initiation

Release/Project
Planning
Each Iteration
1
….
X

System Test /
End Game

Release to Prod/
Support

Get an understanding of the project (business)

Participate in sizing stories

Create Test Plan

Estimate tasks, Run regression tests
Collaborate with customers on acceptance tests
Write, automate and execute new story tests
Pair test with other testers, developers
Perform exploratory testing
Perform Final Load Test
Complete Final Regression Test
Perform UAT
Perform Mock Deploy
Participate in Release Readiness
Participate in Release to Prod
Participate in Retrospectives


9

Project
Test Plan

Release
Test Plan

Story
Tests

10

5

1/22/2013

• Product Roadmap
◦ High level feature ideas

• Release Planning
◦ Backlog of sized and prioritized stories

• Iteration Planning
◦ Backlog of estimated tasks
◦ Estimations are in hours: how long will it take


11

• Types of testing needed
• Automation
• Complexity
• Size

More details later …


12

6

1/22/2013

Instead of saying NO,
or being the gatekeeper
Be the information provider so
business can make the decisions
Questions… on agile approach to planning

13


14

7

1/22/2013

Let’s talk a little
testing ….

What is enough
to keep you out
of trouble?

15

• Find bugs
• Check correctness
• Test reliability
• Check usability
• Answer “Is it done?”
• Learn about the application
• Feedback into future stories

What else???

16

8

1/22/2013

• Each group, collaborate
• Write different types of testing that you do
• One type per sticky
• 5 minutes


17

• Model for classifying tests
• Looks at the purpose of the tests – the ‘why’


18

9

1/22/2013

Agile Testing Quadrants

Brain child of Brian Marick

19

• Take your tests and put them in the quadrant
you think they belong in.
• Were you able to get them all classified?
• Add tests as you think about them.


20

10

1/22/2013

Agile Testing Quadrants

(Brian Marick)

21

• Can be used as a communication tool
o

to explain testing in a common language

• Emphasize whole-team responsibility
o
o

focus on collaboration
whole team participation

• Help plan what tests to automate
o

and tools and infrastructure needed


11

1/22/2013

For each story / feature
◦ No story is done until tested
◦ Customer needs captured as passing tests
◦ Automated regression tests
For release readiness
◦ Delivers value
◦ “Doneness” in all quadrants


23


24

12

1/22/2013

Unit Tests
o
o
o
o

Tests developer intent - program design
Tests a small piece of code
Makes sure it does what it should
TDD

Component Tests
o
o

Tests architecture intent – system design
Tests that components work together correctly

Connectivity Tests

• Builds quality in
• Focus on internal code quality
• Builds testability into code
• Go faster, do more
o

Unit tests provide refactoring support

• Provides instant feedback
• TDD increases confidence in design
• Forms the foundation of automation suite

13

1/22/2013

Can you think
of anything else
for Quadrant 1?

27


28

14

1/22/2013

• Use to elicit requirements
• Acceptance Test (or Example) Driven Development
o

Allows developers to code until the tests pass

• Capture examples, express as executable tests
• User experience
o

wire frames; mock-ups / prototypes


• Help customers achieve advance clarity
• Gain shared common understanding of stories
• Drive development with business-facing tests
• Obtain enough requirements to start coding
• Focus is external quality
• Customer – developer – tester collaboration
◦ Power of Three


15

1/22/2013

Can you think
of anything else
for Quadrant 3?

Discussions?

31


32

16

1/22/2013

• Exploratory testing
o

what didn’t we think about

• Test for usability
o

understand end users, personas

• Tours
• User Acceptance Testing (UAT)


• Iteration reviews / demos
o

Builds confidence

o

Quick feedback loop

• Pair test with customers
• Informal demos
o

Pair exploratory testing with customer


17

1/22/2013

• Provide feedback ….
o

Turn learnings into tests that drive new features

o

Change process as needed

• Evaluation of the product
• Recreate actual user experiences
• Realistic use


• Know your customers
• Make them real
• Plan your exploratory
testing using them

• Picture – from Jeff Patton’s
Pragmatic Personas weekly column
on Sticky Minds (1/25/2010)


18

1/22/2013

• Think of some personas or characters and
devise exploratory scenarios which that role
might get into, for example:
Shopping website - Amazon
o

Senior citizen who never shopped on the web

o

Hacker looking to cause trouble

o

Working mom in a rush

o

Internet-savvy teen

37


37

Any questions about
Quadrant 3?

38

19

1/22/2013

39


• Non-functional tests
• Performance, scalability, stress, load
• “ility” testing
• Automated deployments
• Memory management
• Infrastructure testing
• Security testing
o

Roles & permissions , system ‘hacking’

• Data migration
• Recovery

20

1/22/2013

• May be higher priority than functional req’ts
• Makes the ‘finished’ product
• Transfer specialized expertise

Consider the four quadrants
• Would you add anything?
• Consider your quality attributes
• Do you need to move any tests?
• What tests are you missing?
• What is your team is not doing now?


42

21

1/22/2013

Automate at
the feature
level

push
the
tests
lower

Automate
at the story
level

Automate
at the task
level

Sequential, phased – eg. waterfall

Requirements

All
automation
done here

Specifications
Code
Testing

Release

Time

D
C

C
A
It 0

B

It 1

A

B
It 2

A

Agile: iterative, incremental

F
E
D
C
B

It 3

A

– Each story is expanded, coded and
tested
– Possible release after each iteration
B

Automation
here
It 4
Automation
Automation here
Automation
here
here
44

22

1/22/2013

Questions… on types of testing?

45


46

23

1/22/2013

Release Level Test Planning


47

• Consider scope, priorities, risks
• Document only what is useful
• Consider all types of testing
• Budget time for:
o
o
o

infrastructure
tools
automation needs

• Strive for simplicity


48

24

1/22/2013

49

• Sizing of stories
◦ think about impacts to system

• Ask questions
◦ that may impact the ‘bigness’ of the story
◦ to uncover hidden assumptions
◦ such as ‘what if?’, or “what happens if?”

• Not the time for details
• Create a project “test plan”

50

25

1/22/2013

• Are we working with a vendor?
◦ If so, how do we coordinate testing?

• Are there dependencies between …
◦ features?
◦ teams?
◦ Stories?


51

26

1/22/2013

• Should be project specific
• High-light critical information
◦ risks
◦ assumptions
◦ constraints

• Focus on what is really needed
• Put static information –in a Quality
Management Strategy document


53


54

27

1/22/2013


55

A test matrix (one example)
◦ provides a view of the release
◦ can also be used as visible progress
But, remember
◦ the value is in the planning
Let`s look at an example


56

28

1/22/2013

Copyright 2009 Janet
Gregory, DragonFire

57

Let’s do some collaboration,
& work together to find a solution


58

29

1/22/2013

rules
Sub topic

rules

user name

save

password
Sub topic

first time
Sub topic

encryption
MAIN
new account
TOPIC

Sub topic
change
Sub topic


59

Mind map testing ideas for a feature
How many ideas can you generate
in 5 minutes?


30

1/22/2013

Feature A
• As an on-line shopper, I want to provide my
shipping address when I check out so that my
order goes to the right location
Feature B
• As an on-line shopper, I want to provide
payment information when I check out so I am
billed correctly.


• Create a test matrix for the stories on the next
slide.
• Functionality down the left side
• Test conditions across the top
o

Think heuristics

o

Use mnemonics (ex. SFDPOT)

• Gray out the squares that are not applicable.


31

1/22/2013

Feature A
• As an on-line shopper, I want to provide my
shipping address when I check out so that my
order goes to the right location
Feature B
• As an on-line shopper, I want to provide
payment information when I check out so I am
billed correctly.


• What value do you see in something like this?
• What did you learn?
• How would you use it?


32

1/22/2013

Questions on test planning at the release level?

65


66

33

1/22/2013

Iteration Planning:
• Stories tell us about the tip of the iceberg
• What don’t we know?
• What questions should we ask to find out?

68

34

1/22/2013

Create a
user story

Write
Customer
(Q2) Tests

Expand
tests –
Story Tests
Automate
Q2 Tests
Start
thinking
how to code

Pair,
“Show Me”

TDD

Product owner
Product owner/ Tester
Tester
Tester/Programmer
Programmer

Exploratory
testing

Customer
User
Acceptance


69

• Be proactive – preplanning
• Try specifications workshops – Gojko Adzic
• Help customer achieve ‘advance clarity’ on
stories
◦
◦
◦
◦

Customers speak with “one voice”
Testable stories
Steel threads
Create acceptance tests


70

35

1/22/2013

• Define high level story tests or examples
• Find hidden assumptions
• Define and estimate testing tasks – consider:
o
o
o
o

automation needs
test data,
exploratory testing
Q4 tests (‘ilities’, security, performance, etc.)


71

Ask questions
• What's the business goal?
• Can the user mess up?
• What’s the best thing?
• What’s the worst thing that can happen?
• Watch for scope creep or ‘bling’
• Is this story testable?


72

36

1/22/2013

To
Review


Picture from
Mike Cohn’s
website

73
73

•
•
•
•

Express the intent of the story
Use examples as specific instances of a scenario
Think expected and unexpected behaviors
Gives a shared common understanding of the
story
• Feeds into TDD (Test Driven Development)


74

37

1/22/2013

As a new user, I want to create an
account with a user name and
password so that only I can access
my information.
===============================


75

75

• To create my account, I enter a valid user name
and password, my information is saved and I am
logged into the system and on the home page.
• If I enter an invalid user name, I get an error
message “Invalid User Name”, and I am able to
try again.
• If I enter an invalid password, I get an error
message “Invalid Password”, and I am able to try
again.

76

38

1/22/2013

BDD – Behavioural Driven Development
Given the user has no existing account
When she requests to create a new account,
Then she enters a valid user name and valid
password (rules defined)
And the information is saved upon submitting .

77


77

returnValue TestLogin ( userName, password)
User Name

Password

Expected result

comments

JanetGregory

Password

Access system as
JanetGregory

Valid combo saved

Janet Gregory

Password

Error

Space in user name

JanetGregory

Abc

Error

Invalid password


78

39

1/22/2013

• Write Acceptance Test(s) for one of the two
stories – any format you like
• To help, try mind-mapping or draw a flow
diagram
• Ask your customer for examples.
• Think “Amazon”
• (~15 min)

Story a – Part of Feature A
• As the company shipper, I need to verify the
city, state and postal code so that the order
goes to the right location.
Story b – Part of Feature B
• As the company accountant, I need to ensure
the credit card information is correct so the
on-line shopper is billed correctly.


40

1/22/2013

We walk out of the iteration planning
meeting ....
Now what?


82

41

1/22/2013

• Start simple with the high level acceptance tests
◦ Add boundary, edge conditions, etc.
◦ Add non-functional tests

• Experiment to find the right detail level
• Review with programmers
◦ Then automate

• Keep adding until story is complete
• Consider exploratory test scenarios


83

Acceptance Tests – Fit Automation Style
-- Third thread
User Name

Password

Expected result

Comments

JanetGregory

Password

Access system as
JanetGregory

Valid combo saved

Janet Gregory

Password

Error

Space in user name

Janet#Gregory

Password

Error

Special char not
allowed

Password

Error

Blank user name

JanetGregory

Password

Error

User already exists

JanetGregory

Abc

Error

Not enough char in
password

Error

Blank password

JanetGregory


84

42

1/22/2013

How do you know
you are done?


Description

85

Completed

1

Story tests reviewed, automated and pass

Yes

2

Exploratory testing complete

Yes

3

Unit tests reviewed and pass

Yes

4

Q4 tests complete

Yes

5

Acceptance tests pass

Yes


86

43

1/22/2013

Questions on test planning at the story level?

87


88

44

1/22/2013

• Understand your context
• Understand the purpose
• Consider ROI (return on investment)
• Push the tests lower
• Automate the repetitive and boring tests
• Plan, but document simply
• Plan, but plan for the appropriate level


• How much is too much?
• What is not enough?
• Why do you need them?
• What is the right information?
• Who looks at them?
• What do they use them for?
• What is the simplest way you can capture
them?


90

45

1/22/2013

•
•
•
•
•
•
•
•

What problem are you trying to solve?
What measurements make sense?
What is the simplest way to gather them?
Who needs to see them?
What is the simplest way to make them visible?
How are you going to use them?
When will you know to stop?
Be Aware!!!
o
o

The wrong measurements can be deadly
Metrics can be misinterpreted


• How are you going to know when you are done?
• What is enough?
• Collaborate to decide what coverage you want
• Think risk – include the customer
• Functional coverage
• Lines of code? … be careful
• What tools can you use?
o
o

Heuristics
Mnemonics


46

1/22/2013

• Understand the level of precision you need
◦ Think big picture in release planning
◦ Think tasks during iteration planning
◦ Think tests during story test planning

• Use ATDD for feature and story testing
• Make it visible
• Make it simple
• Make it valuable
• You want to be testing, not documenting

Are there still unanswered questions.. Concerns?

94

47

1/22/2013

Now Available
Agile Testing: A Practical Guide for
Testers and Agile Teams
By Lisa Crispin and Janet Gregory
www.agiletester.ca
My contact info
www.janetgregory.ca
http://janetgregory.blogspot.com/
Email: janet@agiletester.ca
www.lisacrispin.com
http://lisacrispin.com
Email: lisa@agiletester.ca


95

•
•
•
•
•
•
•
•

www.lisacrispin.com
agile-testing@yahoogroups.com
http://www.exampler.com - Brian Marick’s web site
www.testobsessed.com (heuristics cheat sheet) – Elisabeth Hendrickson
http://www.shino.de/blog/ - Markus Gärtner’s blog
Gojko Adzic, Bridging the Communication Gap, 2009
Gojko Adzic, Executable Specifications, Manning, 2011
vwww.mountaingoatsoftware.com – Mike Cohn’s web site (and all his
books)
• http://www.developsense.com/2009/04/of-testing-tours-anddashboards.html
• James Whittaker, Exploratory Software Testing, 2009 Addison-Wesley
• http://www.stickyminds.com/ Pragmatic Personas - Jeff Patton’s weekly
column 1/25/2010
• Jean Tabaka, Collaboration Explained, 2006 Addison-Wesley
• Copyright 2013Manifesto: http://agilemanifesto.org/
Agile DragonFire Inc.

96

48

Planning Your Agile Testing: A Practical Guide

More Related Content

What's hot

Viewers also liked

Similar to Planning Your Agile Testing: A Practical Guide

More from TechWell

Recently uploaded

Planning Your Agile Testing: A Practical Guide