A case about how a company (Sectra) is using Exploratory Testing in their agile development organization where testers and developers are sitting together in cross functional teams using Scrum.

1. Case: Exploratory Testing
in an agile development organization
Johan Åtting, Sectra
@johanatting
IT Quality & Test Management, Copenhagen 12 June 2013

2. Developer, Tester,
Project Manager,
Test Manager,
Support Manager,
Operations Manager,
Quality Manager,
Line Manager, @johanatting
Founded EAST: Östergötlands Association for Software Testers
Conference speaker
Currently: Chief Quality Officer at Sectra
Transport, Bank
& Finance,
Telecom and
MedTech.
19 years in the software business

3. WHY & HOW
we use exploratory testing
in our context today

4. CONTEXT

5. 13%
87%Radiology IT - RIS/PACS
Orthopedic Imaging
Rheumathology
Our mission is to increase effectiveness of healthcare, while
maintaining or increasing quality in patient care.

6. 10 Agile development teams
2 Testers + 4 Programmers per team
Development & Testing - One department

7. Release TestDevelopment
...
...
...
.
.
.
Exploratory Testing
- During Sprints
- Between Sprints (Cross Team Testing)
- During Release Test

8. What is
Exploratory Testing?

9. Exploration
Wandering

10. Explore: The jungles of ...
Using: Map, local guides, whip ...
To: Find lost treasures

11. Explore: export function
To: see how file size effects performance
Using: The image data db, ...

12. Explore: export function
To: collect info about design consistency
Using: the import function as Oracle

13. Mission / Plan / Charters
Test...
Adjust Observe
use feedback from the previous
test to inform the next

15. WHY

16. Benefits
of exploratory testing
• It’s agile, flexible and responsive
• Focus & time is spent on testing and not
on writing test scripts
• Learning about the system step by step
• We find more bugs

17. HOW

18. 1 2 3 4 5 6 7 8 9 10
Two week sprintSprintPlanning
SprintDemo
Testing
Initial test
planning
Continuous test
re-planning

19. Initial test planning:
1) Brainstorm test ideas based on:
• Sprint objective
• Quality characteristics (safety, security,
performance, reliability, capability, usability…)
• Risks (safety, business, technical)
• ...or any other basis or source for test ideas
=> Documented as e.g. mind maps

20. Initial test planning:
2) Organize test ideas (mind map) into charters:
• Explore: Area, function, feature...
• To: discover information about...
• Using: resources, oracles, heuristics

21. Testing
Test...
Adjust Observe
Adjust: next step in the charter

22. Continuous test re-planning
Test...
Adjust Observe
Adjust: add charters & re-prioritize

23. When are we done?
We use test levels!
(to discuss test coverage)

24. 0 Not tested
1 Sanity Check
2 Common & Critical
3 Reasonable
4 Complex
5 Can’t think of
anything more
that could give us
valuble
information
Test Levels
Todays testing tasks progress board
Target area

25. Test Owner Area Status
Anna Reporting

Ben Import

Chris Export

David Printing

Ellen User roles

Fred Archiving (not started)
Greg Regression (not started)
Weekly test status meeting with all testers
Possible problems* or we suspect there will be
No problems* found or suspicions of any
Known problems*
*Problems = Preventing release or preventing further testing

26. Output:
(from a team sprint)
• Exploratory test records
• Updated (or new) regression tests
• Test cases for the Release Test phase
• Bug reports (only unfixed bugs)
But only if needed

27. Release TestDevelopment
...
...
...
.
.
.
Exploratory Testing
- During Sprints
- Between Sprints (Cross Team Testing)
- During Release Test

28. Cross Team Testing
Gather all testers to test each others test
objects after every sprint.
We need to get fresh, unbiased, independent
eyes on what is being developed.

29. Sprint Sprint Sprint Sprint
CrossTeamTesting
CrossTeamTesting
CrossTeamTesting
Sprint Sprint Sprint SprintSprint Sprint Sprint SprintSprint Sprint Sprint Sprint

30. 1 2 3 4 5 6 7 8 9 10
In the beginning of the next two week sprintSprintPlanning
SprintDemo
Planning(1-2hours)
Testing(6hours)
Cross Team Testing

31. Planning (1-2 hours)
1. High level presentation of test objects
2. Planning sessions in smaller groups

32. Testing (6 hours)
• Exploratory group testing sessions (*3)
• Debrief to the Test Owner
• Bug reporting

33. Time Slot Activity
9:20 Test Session #1 70min
10:30 Debriefing Session #1 15min
10:45 FIKA
11:00 Test Session #2 75min
12:15 Debriefing Session #2 15min
12:30 LUNCH
13:30 Test Session #3 75min
14:45 Debriefing Session #3 15min
15:00 -> Bug Reporting / Optional additional
testing

34. Takeaways
Cross Team Testing
• Everyone is biased
• There is a need for independent eyes
• Cross Team Testing is one solution
More info:
http://vimeo.com/53218576#at=0 (Video from ÖreDev 2012)
http://www.slideshare.net/Johantting/cross-team-testing-at-dev-lin2013 (Slides from DevLin 2013)

35. Release TestDevelopment
...
...
...
.
.
.
Exploratory Testing
- During Sprints
- Between Sprints (Cross Team Testing)
- During Release Test

36. Release Testing
Before we start Relese Testing all areas need
to have a ...


37. Test Owner Area Status
Anna Reporting

Ben Import

Chris Export

David Printing

Ellen User roles

Fred Archiving

Greg Regression

... at the end of each row.

38. Test Owner Area Status
Anna Reporting
Ben Import
Chris Export
David Printing
Ellen User roles
Fred Archiving
Greg Regression
Start of Release Test Phase

39. Release Testing
• Planned by each test owner
• Coordinated by a test project manager
• Mix of: Retest of new features, workflow
based tests, regression tests...
• Mix of exploratory tests & scripted tests
• Test environment is freezed

40. Test Owner Area Status
Anna Reporting

Ben Import

Chris Export

David Printing

Ellen User roles

Fred Archiving

Greg Regression

Continue Release Testing until

41. Summary

42. Benefits
of exploratory testing
• It’s agile, flexible and responsive
• Focus & time is spent on testing and not
on writing test scripts
• Learning about the system step by step
• We find more bugs

43. We do Exploratory Testing
- During Sprints
- Between Sprints (Cross Team Testing)
- During Release Test 

44. Twitter: @JohanAtting
Linkedin: Johan Åtting
E-mail: johan.atting@sectra.com