The document discusses some of the challenges of being a tester in an Agile project. It notes that while Agile teams are more effective at delivering functionality, they tend to have lower quality than traditional teams. It suggests that testers can help improve quality by taking a "whole team approach" where they work closely with developers and business representatives. This includes collaborating at daily stand-up meetings to ensure quality across all test levels. However, the document also points out some potential issues, noting that unit tests alone have limited effectiveness at finding bugs, and independent testing by a separate team may be needed to achieve high bug detection rates.

1. Why I do not like to be a
tester in Agile project
Lucjan Stapp
Warsaw University of Technology
Stowarzyszenie Jakości Systemów Informatycznych
L.Stapp@mini.pw.edu.pl
L.Stapp@sjsi.org

2. Lucjan Stapp
Scientific worker at Warsaw University of Technology
ISTQB® CTAL – TM, ISTQB® CTAL - TA
Author of more than 40 papers, more then 10 are
connected with testing;
Acting vice-president of Stowarzyszenia Jakości
Systemów Informatycznych (Polish Testing Board);
Member of ISTQB® Dictionary Working Group;
Member of ISTQB® Exam Working Group;
I believe(d) in Agile methodologies.
Lviv November 7th 2013
2/37

3. Agile manifesto
In February 2001, 17 software developers met at
the Snowbird, Utah resort, to discuss lightweight
development methods.
They published the Agile Manifesto (Manifesto
for Agile Software Development) to define the
approach now known as agile software
development.
Lviv November 7th 2013
3/37

4. Agile manifesto
We are uncovering better ways of developing software by
doing it and helping others do it.
Through this work we have come to value:
Individuals and interactions over
Working software
over
Customer collaboration
Responding to change
over
over
processes and tools
comprehensive
documentation
contract negotiation
following a plan
That is, while there is value in the items on the right, we value
the items on the left more.
agilemanifesto.org
Lviv November 7th 2013
4/37

5. Agile manifesto
Only iterative
incremental
approach
As a result of each iteration we
obtain the working software
Lviv November 7th 2013
5/37

6. Agile manifesto
Basic Agile principles:

Working software is the primary measure of progress

Informal (face-to-face) communication,

Self-organization and motivation,

Inspect and adapt,

the team reflects on how to become more effective, then tunes and adjusts
its behavior accordingly

Customer collaboration:

welcome changing requirements, even late in development.

Improving the chances of understanding what the customer wants
Lviv November 7th 2013
6/37

7. Agile manifesto
Agile methodologies (between others):

 eXtreme Programming XP
XP,
 Scrum,
 Kanban,
 Dynamic Systems Development Method,
 Adaptive Software Development,
 …...
Lviv November 7th 2013
7/37

8. How it works?
Main aspects
0.7
0.6
0.6
0.6
0.6
0.5
0.5
0.4
0.4
0.4
0.4
0.3
0.3
0.3
0.2
0.2
0.2
0.2
0.1
0.1
0.1 0.1
0.0
0.0
Quality
Functionality
Ad-hoc
Waterfall
ROI (money)
Classical Iterative
Schedule
Agile
Agile teams are more
effective in desired
functionality then
“traditional” teams.
But less in quality.
However testers are
responsible for quality.
What can we do to
change this aspect, to
obtain higher quality
in Agile teams?
Results of Dr. Dobbs Journal Investigations(2008)
Lviv November 7th 2013
8/37

9. How improve quality?
Higher ROI.
Agile teams work in more effective way (but not harder) and
should propose needed functionality earlier, hence there is a
shorter path to the market and greater profit.
Faith
DDJ found also that people believe that Agile teams product higher
quality (however it is NOT true) then traditional teams, hence one
can observe higher satisfaction of stakeholders.
Lviv November 7th 2013
9/37

10. How improve quality?
Power of Three.

“Whole Team Approach”



Whole Team usage means involving all of the people with the knowledge
and skills necessary to ensure project success.
quality is the responsibility of the Whole Team
Testers work closely with both developers and the business
representatives to ensure quality across all test levels
Lviv November 7th 2013
10/37

11. How improve quality?
Power of Three.


From the individual agile tester perspective “Whole Team
approach” means daily collaboration with both developers and
business representatives.
Testers should



transfer testing knowledge to non-testing team members and
influence the development of the product.
ensure the right level of integration with developers and the right
level of collaboration with customers.
How to do it?:

the daily stand-up meeting, involving all members of the team, at
which project status is discussed and any impediments to progress
are highlighted.
Lviv November 7th 2013
11/37

12. Scrum
Daily scrum
Lviv November 7th 2013
12/37

13. Teams
 Teamwork is one of the fundamental principles in agile

development.
It is important to select the right people that work well
together to successfully create the desired product
Lviv November 7th 2013
13/37

14. Team
Galleys vs. Viking boats
Propelled mostly by oars
Lviv November 7th 2013
14/37

15. Team
Viking expansion from 8th till 11th centuries
Lviv November 7th 2013
15/37

16. Team
But
• Viking expansion from late 8th till 12th century
• Galleys are used from the 9th century BC until development
of advanced sailing warships in the 16th century
• Trade and transport used galleys, not Viking knörr
Lviv November 7th 2013
16/37

17. Team
Team complement
There is no division on developers and testers.
Testers should be build in team.
But:
• How many team members do concentrate on quality
problems? (1/10, 2/10 ???);
• “Group thinking” – typically only positive;
• Not enough “business knowledge” (power of three?)
• Problems with product owner.
Lviv November 7th 2013
17/37

18. User Story
User Story the basis of requirements
 Elements of User Story
 As… (concrete type of user)
 I want… (problems to be solved)
 because… (desired results).
 Definition of user satisfaction criteria


Typically as set of acceptance tests (Done)
Test priority and story criticality can be based on the users
category.
Lviv November 7th 2013
18/37

19. User story
INVEST
A good user story is:
 Independent
 Negotiable
 Valuable
 Estimable
 Sized Appropriately
 Testable
Lviv November 7th 2013
19/37

20. Testing in Agile Projects
INVEST –Testable
• Testing in Agile should be iterative
• Testers in Agile must work without complete
documentation
• Testers in Agile should be flexible.
Lviv November 7th 2013
20/37

21. User Story
INVEST –Testable
• Tests concentrate on “simple” problems
Agile team does not expect very specific complicated
conditions.
World wide transaction system for an international bank
A fish trade company in Japan makes a payment to a vendor
on Iceland. It should have been a payment in Icelandic
Kronur, but it was done in Yen instead. The error is
discovered after 9 days and the payment is revised and
corrected, however, the interest calculation (value dating)…
From a talk by Hans Buwalda
Lviv November 7th 2013
21/37

22. Testing in Agile
Beginning of the project
Understanding of the project principles
Release planning
Stories estimation; questions: „What happens, if…?”
Sprint planning
Each sprint
Validation of satisfaction conditions , adding new ones
Creation and testing in
threes:
Developer, business and
tester
• Write and execute tests for
each story,
• Write and execute functional
tests,
• Automate test,
• Make exploration tests.
Basic testers activities in Scrum
Lviv November 7th 2013
22/37

23. Testing in Agile
•
•
•
•
Write and execute tests for each story,
Write and execute functional tests,
Automate test,
Make exploration tests.
Agile Tester should
 be able to get quickly acquainted with the product under test
 get familiar with the products domain to understand product
requirements
 have sufficient product and/or customer related domain knowledge
 be able to distinguish between critical and less critical
requirements
 be able to design appropriate test cases
 be able to evaluate test outcomes
Lviv November 7th 2013
23/37

24. Testing in Agile
•
•
•
•
Write and execute tests for each story,
Write and execute functional tests,
Automate test,
Make exploration tests.
Not only Test Driven Development (MUST)
But also
Acceptance Test Driven Development
Inverse test pyramid
Behaviour Driven Development
Lviv November 7th 2013
24/37

25. Testing in Agile
•
•
•
•
focus of unit testing
L
I
k
e
l
I
h
o
o
d
Could Test
H
Must Test
I
II
focus of acceptance
testing
M
III
L
Write and execute tests for each story,
Write and execute functional tests,
Automate test,
Make exploration tests.
L
IV
“Won’t Test”
Should Test
M
Impact
H
Tester should be able to
distinguish between
critical and less critical
requirements.
Lviv November 7th 2013
25/37

26. Testing in Agile
•
•
•
•
Write and execute tests for each story,
Write and execute functional tests,
Automate test,
Make exploration tests.
• Continuous feedback is obtainable and can be achieved by
creating and maintaining an appropriate set of automated
tests.
• A tester in an Agile team should understand basics of test
automation to be able to execute automated tests and to (on
his own or with help of the programmers in his team) automate
test cases
Lviv November 7th 2013
26/37

27. Testing in Agile
•
•
•
•
Write and execute tests for each story,
Write and execute functional tests,
Automate test,
Make exploration tests.
 It is not possible to instantly automate each test case.
 Automated testing has to be supplemented by techniques for
quick manual testing.
 Exploratory Testing is a technique accomplishing that.
Lviv November 7th 2013
27/37

28. Testing in Agile
What it means for Agile testers?
 “fanatic”, „maniac”, „zealot”
 continuous learning
by experience ?
 when?
 Fail fast
 Fail often
 Fail cheap
tools, tools, tools
 exploratory testing


Learn quickly
Learn continually
Learn inexpensively *
*d’Amico V. The state of Agile
Lviv November 7th 2013
28/37

29. But ……
Problems
…
Module tests (also TDD) are limited in finding bugs:
Capers Jones1 found that average effectiveness for unit test is between
25 - 30%.
And Rex Black2 (for American market ) found that good system tests
done by independent test team achieves 85% effectiveness in
founding bugs.
1Capers Jones: MEASURING DEFECT POTENTIALS AND DEFECT REMOVAL EFFICIENCY http://www.rbcs-
us.com/images/documents/Measuring-Defect-Potentials-and-Defect-Removal-Efficiency.pdf
2
http://www.rbcs-us.com/images/documents/
Lviv November 7th 2013
29/37

30. Typical solution
Solution:
Two levels of testing:
• Internal testing – build in Agile team
• External testing – independent test team
•ATDD facilitates the use of external testing teams to
perform functional testing
Lviv November 7th 2013
30/37

31. Typical solution
New stories
(changes +
defects)
New stories
(changes +
defects)
Release no. k
Release no. k+1
Independent test team
Lviv November 7th 2013
Internal tests
External tests
• Acceptance
• UAT
• Exploratory
• Nonfunctional
• Scenario based
31/37

32. Typical solution
But:
It means at least 3 - 4 weeks delay between founding the failure
and the new version ready for re - tests.
Lviv November 7th 2013
32/37

33. Examples
AGILE == SUCCESS??
Lviv November 7th 2013
33/37

34. Examples
Example no.1
• Only f2f communication
• No written requirements
• Based on old existing
systems
• Product owner
RESULT:
• Concentrate on specific
• 6 month delay
financial modules
• Business problems
• No business representative in
•No working application
team
• Tests
• Mostly user acceptance
• No written information about
incidents
Lviv November 7th 2013
34/37

35. Examples
Example no. 2 NEW FUNCTIONALITY in bank system
• Only f2f communication
• No written requirements
• Frequent changes
• Without info to test team
• Product owner
• Less of business knowledge
• Concentrate on security
problems
RESULT:
• disaster
• Tests
• Long time before bugs closure
Lviv November 7th 2013
35/37

36. Examples
Example no. 3 Telecom application connected with
EURO 2012
• 90% written requirements
• Product owner
• Concentrate on performance
problems
RESULT:
• Ready June 1st 2012
• Success??
• Tests
• 2 performance testers in team
• 60% time – performance tests
Lviv November 7th 2013
36/37

37. Thanks for your attention
Lviv November 7th 2013
37/37