Talk for Santa Barbara Agile on Exploratory Testing

1. Exploratory Testing
Explained and
ExperiencedMaaret Pyhäjärvi
Email: <maaret@iki.fi> | Twitter: maaretp
Maaret Pyhäjärvi
Nimeä | Attribution (Finland)
http://creativecommons.org/licenses/by/1.0/fi/
http://creativecommons.org/licenses/by/1.0/fi/deed.en
1

2. TESTING IS ABOUT
FEEDBACK:
QUALITY-RELATED
INFORMATION
2

3. Realizations about Nature of
Testing
20
16
1639
5±2
3

4. 20DYNAMICALLY ADAPT
FOR LIMITED BUDGET
OPPORTUNITY COST
4

5. 16 EXPECT THE
UNEXPECTED
5

6. 1639ROUTES ARE RELEVANT
NOT ALL BUGS ARE EQUAL
6

7. 5±2
TAKE NOTES
CREATE CHECKLISTS
7

8. Things Can Look Different from
Different Perspectives
8

9. Exploratory Testing:
Better tests, better testers!
• An approach, not a technique
• Find unknown unknowns
• Disciplined
• Test is a performance, not artifact
– Artifacts support human
memory
– Many forms: e.g. checklists and
automation
• Exploratory performance testing,
Exploratory test automation,
Exploratory regression testing
Test-related
learning
Design of new
tests
Test execution
Result
interpretation
9

10. There’s a Difference!
• A comment I hear often on the ET
course I give:
– ”I’ve always been doing this just did not
give it a name”
• I must emphasize that:
– I require more out of exploratory testing
than just ”going where you feel like
while testing”
• Ask yourself:
– Can you defend your choices of
strategy and tactics?
– Can you explain what you’ve done in
words that don’t just explain numbers of
bugs found?
– How do you know if you’re done or not?
10
”My testing would
be unsystematic
ad hoc testing if I
could not tell the
story of my tests,
remember what
I’ve tested or
what my strategy
was, or relate
that back to my
mission” –James
Bach

11. Exploratory Testing Self-
Management
Vision (“Sandbox”) Current Charter
Other Charters Details
11

12. Exploratory Testing:
Frame of Management
”A day’s work”
Vision (“Sandbox”) Current Charter
Other Charters Details
Bug
Reports
Perception of
quality and
coverage
Quality
ReportDebriefing
Tester
Test
Manager
Past
Results
Obstacles
Outlook
Feelings
?
#
xCharter backlog of the future
testing
Out of
budget
Next in
importance!
#, ?, x, +
20:20:60
Session sheets of the past testing
Idea of
exploration
Metrics
summary
Coachin
g
12
Playbooks
Coverage outlines

13. Test Ideas / Quick-and-Dirty
Download the full 2-page Cheat Sheet with ideas from Elisabeth Hendrickson,
James Lyndsay, and Dale Emery on Qualitytree.com 14

14. (Exploratory) Testing Dynamics
Source: Adapted from James Bach, Jon Bach, Michael Bolton. Exploratory Testing
Dynamics. v.2.2. 2009
Evolving
work
products
Skills
and
tactics
Testing
polarities
Test
strategy
”A set of
considerations designed
to help you test robustly
or evaluate someone
else’s testing.”
” To develop ideas or
search a complex space
quickly yet thoroughly,
not only must you look
at the
world from many points
of view and perform
many kinds of activities”
”Exploratory testing
spirals upward toward a
complete and
professional set of test
artifacts”
” …skills that comprise
professional and cost
effective exploration of
technology. Each is
distinctly observable and
learnable, and each is
necessary to exploratory
work.”
15

15. Exploration Skills
Source: Adapted from James Bach, Jon Bach, Michael Bolton. Exploratory Testing
Dynamics. v.2.2. 2009
Self-
managemen
t
Developing
ideas
Examining
product
Done
To Do
Issues
Coverage
All sources available
Best use of time – effective and efficient work
Making models
Tool support – creative solutions
Risk-based testing – scientific approach
Keeping one’s eyes open
16

16. Test Automation / Tools in ET
• Any form of acquiring quality-related
information fits into exploratory testing
• It’s not manual, it’s brain-engaged – and for
making that happen, you need to be smart
with automation and tools!
• Example: you might not at first know what
you’re looking for...
Search "<ns1:Koodi>" (443 hits in 169 files) in VE
Y:ELLULAPATestausKokonaiseläketurvan-oteAnsaintatiedot-VEHaeAnsaintatiedot-VE__
20110307 14-22-15.xml (4 hits)
Line 1313: <ns1:Koodi>YL130I</ns1:Koodi>
Line 1317: <ns1:Koodi>LAPA_172_011</ns1:Koodi>
Line 1321: <ns1:Koodi>67</ns1:Koodi>
Line 1324: <ns1:Koodi>67</ns1:Koodi>
A lot of text cut away from here… 17

17. 18
Favoring Mindmaps
(Mindmup, Xmind) and
Notetaking (Rapid Reporter,
iTester) tools

18. Testers don’t break your
code, they break your
illusions about the
code.
-- adapted from James Bach
19

19. Empirical Evidence over
Speculation
VALU
E
ILLUSIONS
20

20. Serendipity, Perseverance
…and Love of Testing
• Serendipity = Lucky
accident
• Just my luck?
– Luck favors the ones
who intentionally vary
their actions
• ”The more I practice,
the luckier I get”
• Perseverance = Keep
trying
• Testing takes time –
keep trying with more
ideas, stop giving up so
easily
• “It's not that I'm so
smart, it's just that I
stay with problems
longer.”

21. The World Has Already
Changed
COMMODITY
TESTERS
• Manual checkers
• Tests are an
artifact
SKILLED
TESTERS
• Explorers of
products and
businesses
• Testing is a
performance
22

23. Question, Thoughts?
Feedback?