2. • What is exploratory testing?
• Strategy and techniques of ET
• SBTM: plan, structure & traceability
• How more exploration can boost the value of
your testing
• How to master exploratory testing?
3. Exploratory Testing
“a style of software testing that
emphasizes the personal freedom
and responsibility of the individual
tester to continually optimize the quality of
his/her work by treating test-related learning,
test design, test execution, and test result
interpretation as mutually supportive activities
that run in parallel throughout the project”
4. Exploratory Testing
“An approach to software testing that emphasizes
the personal freedom and responsibility of each
tester to continually optimize the value of his work
by treating learning, test design and test execution as
mutually supportive activities that run in parallel
throughout the project."
"Simultaneously designing and executing tests to
learn about the system, using your insights from the
last experiment to inform the next."
5. Exploratory Testing
“An approach to software testing that emphasizes
the personal freedom and responsibility of each
tester to continually optimize the value of his work
by treating learning, test design and test execution as
mutually supportive activities that run in parallel
throughout the project."
"Simultaneously designing and executing tests to
learn about the system, using your insights from the
last experiment to inform the next."
6. freestyle
fragmentary test
pure scripted exploratory
case (scenarios)
vague scripts charters roles
Freedom
Pre-
scription
Source: Telling Your Exploratory Story by Jon Bach
7. Fairy tales
Do not apply ET if:
• Higher demands on demonstrability / reporting
(eg. imposed standards)
• Critical functionality (failure = big damage)
• Inexperienced testers
• Test cases must be able to be executed by a different tester
• Test cases have to be re-usable
• No direct feedback from test execution (the results are not
immediately available)
• Tests require a lot of preparation
• Testing needs to be on the critical path as short as possible
Source: TMap Next, chapter 14
8. Fairy tales
Source: Fan booklet Exploratory testing (Microsoft)
10. • What is exploratory testing?
• Strategy and techniques of ET
• SBTM: plan, structure & traceability
• How more exploration can boost the value of
your testing
• How to master exploratory testing?
11. Exploration: searching for value and risk
execution discovery
design
investigation
reporting learning
Source original picture: Two futures of testing by Michael Bolton
12. Source: Rapid Software Testing by James Bach & Michael Bolton
This slide is used with kind permission
13. Source: Rapid Software Testing by James Bach & Michael Bolton
This slide is used with kind permission
14. Source: Rapid Software Testing by James Bach & Michael Bolton
This slide is used with kind permission
15. Source: Rapid Software Testing by James Bach & Michael Bolton
This slide is used with kind permission
16.
17.
18. Source: Rapid Software Testing by James Bach & Michael Bolton
This slide is used with kind permission
19. Stories
Product Story:
How the product works and how it fails...
Testing Story:
What we have tested and how, what we do not
test and how good our testing was...
20. • What is exploratory testing?
• Strategy and techniques of ET
• SBTM: plan, structure & traceability
• How more exploration can boost the value of
your testing
• How to master exploratory testing?
21. Heuristics Test Ideas Risk Catalog Issues Bugs Session
General
Sheets
Daily Test Sessions
Test Log
Test Wrap-up /
Planning
Ongoing
Session Debrief
(weekly) Test
planning Charters Test Log
Test Wrap-up /
session Session Debrief
Test Log
Test Wrap-up /
Session Debrief
Test Plan Coverage Risks Low Tech Dashboard
Strategy
Project
22.
23. Source: Ruud Cox, Testing in a Medical Device Context, EuroStar 2012
24. Heuristics Test Ideas Risk Catalog Issues Bugs Session
General
Sheets
Daily Test Sessions
Test Log
Test Wrap-up /
Planning
Ongoing
Session Debrief
(weekly) Test
planning Charters Test Log
Test Wrap-up /
session Session Debrief
Test Log
Test Wrap-up /
Session Debrief
Test Plan Coverage Risks Low Tech Dashboard
Strategy
Project
25. Test Charters
Explore (target) Explore Application X import
With (resources) menu. Identify important
features with the goal of
To discover (information) developing a coverage outline
and a risk list.
My mission is to test
Read Chapter 4 of the product
<insert risk here> to specification. Prepare a mind map, and
<insert coverage here> discuss it with Peter (programmer) and
David (architect).
The charter is a one- to three-sentence mission for a testing session
26. Heuristics Test Ideas Risk Catalog Issues Bugs Session
General
Sheets
Daily Test Sessions
Test Log
Test Wrap-up /
Planning
Ongoing
Session Debrief
(weekly) Test
planning Charters Test Log
Test Wrap-up /
session Session Debrief
Test Log
Test Wrap-up /
Session Debrief
Test Plan Coverage Risks Low Tech Dashboard
Strategy
Project
27. Test log: my notes
Preparation: talk to product owner
Test Strategy & notes on a bugs
Issue/question & coverage outline
Debrief using PROOF
28. Session sheet
• Charter
• Coverage areas (product areas, product elements, quality criteria or test techniques)
• Start Time + Session duration (long, normal, or short)
• Tester Name(s)
• Charter/opportunity time , expressed as a time breakdown:
– Test design and execution (as a percentage of the total on-charter time)
– Bug investigation and reporting (as a percentage of the total on-charter time)
– Session setup (as a percentage of the total on-charter time)
• Data Files
• Test Log (notes the tester made during testing)
• Bugs (a problem that threatens the value of the product)
• Issues (a problem that threatens the value of the testing process)
Source: An Exploratory Tester’s Notebook by Michael Bolton
29. • What is exploratory testing?
• Strategy and techniques of ET
• SBTM: plan, structure & traceability
• How more exploration can boost the value of
your testing
• How to master exploratory testing?
30. More exploration can boost
the value of testing
• Do what really needs to be done
• Create engagement: help people use their
brains
• Take advantage of tacit knowledge and skill
• Use insights from experiments to inform the
next
• Beware of the creative power of exploration
31. • What is exploratory testing?
• Strategy and techniques of ET
• SBTM: plan, structure & traceability
• How more exploration can boost the value of
your testing
• How to master exploratory testing?
32. Challenges?
Exploratory Testing = not easy and
needs (a lot) training and practice
Often seen as most difficult:
– Note taking
– Test ideas
– Coverage reporting
– Managing ET
33. How to master exploratory testing?
• Just do it!
• Practice, practice and practice some more…
• Pair with colleagues
• Testing Dojo
• Debrief & retrospect
• Train creating test ideas fast
• Learn more about observation, experiments,
biases, social science, ....
34. Source: Gitte Ottosen – Many flavours of exploratory testing (Agile Testing days 2012)
37. “A tester knows
that things can be
“Testing is about
different”
questioning & learning
under conditions of Jerry Weinberg
fundamental
uncertainty.”
“I've participated and organized
RST many testing competitions. I've
never seen someone win such a
competition by writing down
tests. Doesn't that tell us
If you cannot trust your
testers, you do not make them
something?”
write more detailed test case.
But you train them! James Bach
Rikard Edgren – EuroStar 2012
Gitte Ottosen – ATD 2012
Image: http://corkboard.me
http://www.developsense.com/blog/2010/01/exploratory-testing-is-accountable/How More Exploration Can Boost the Value of Your Testing”“How Becoming More Conscious of Your Exploration Can Boost the Value of Your Testing
Scripted testing is not the "normal" way of testing. People think it is only because they are lying to themselves and lying to us. Or else they are just ignorant about testing. Anyone who is truly and only following a script is NOT TESTING. They are FAKING IT.
TMAP zegt:In welk geval kun je Exploratory Testing (beter) niet toepassen,Wanneer er hogere eisen aan de aantoonbaarheid/verslaglegging van testen worden gesteld, bijvoorbeeld door opgelegde standaards.Bij kritische functionaliteit waarvan het falen grote schade kan veroorzaken.Wanneer testgevallen moeten kunnen worden uitgevoerd door een andere tester of een testtool.Voor onervaren testers, omdat deze niet zonder ondersteuning van een expliciete techniek testgevallen kunnen opstellen.Wanneer er geen rechtstreekse feedback van de testuitvoering is, zodat testresultaten niet direct beschikbaar zijn.Bij tests die veel voorbereiding vergenWanneer het testen zich zo kort mogelijk op het kritieke pad van het project moet bevinden.http://www.softwaretesten.net/testmethodieken/tmap-next-te/samenvatting-tmap-next-hoofdstuk-14/
TMAP zegt:In welk geval kun je Exploratory Testing (beter) niet toepassen,Wanneer er hogere eisen aan de aantoonbaarheid/verslaglegging van testen worden gesteld, bijvoorbeeld door opgelegde standaards.Bij kritische functionaliteit waarvan het falen grote schade kan veroorzaken.Wanneer testgevallen moeten kunnen worden uitgevoerd door een andere tester of een testtool.Voor onervaren testers, omdat deze niet zonder ondersteuning van een expliciete techniek testgevallen kunnen opstellen.Wanneer er geen rechtstreekse feedback van de testuitvoering is, zodat testresultaten niet direct beschikbaar zijn.Bij tests die veel voorbereiding vergenWanneer het testen zich zo kort mogelijk op het kritieke pad van het project moet bevinden.http://www.softwaretesten.net/testmethodieken/tmap-next-te/samenvatting-tmap-next-hoofdstuk-14/
An Exploratory Tester’s Notebook
is that SKILLED and VISIBLE exploratory testing boosts the power of the UNSKILLED and SECRETIVE exploratory testing that everyone is already doing
Trish Khooexploratory testing as a set of learnable techniques?It’s like learning the guitar. You can learn the chords if you want, but that’s not how you become a good guitarist. At some point you have to start figuring out how to make the guitar work for you, how to get it to do what you want. Trish & piano: lessons and practice the same songs over and over again, but never enjoyed playing the pianountil I started experimenting with my own methods of learning. Music is not a mindless, step-following activity. Nor is programming. Nor is testing.
