A 45 minute presentation delivered in Turku Agile Days for a mixed audience of mostly non-testers.

1. ExploratoryTesting –Explained and Experienced Maaret Pyhäjärvi Email: <maaret@iki.fi> | Gsm: 040-8233777 "The purpose of testing is largely to force everyone to see the detailed implications of their vague daydreams." -- James Bach Turku Agile Day 2011 (2011-03-17) Erkki Pöyhönen & Maaret Pyhäjärvi Attribution (Tekijä mainittava) http://creativecommons.org/licenses/by/3.0/

2. About the Speaker: Experiences in testing and being a tester on varied basis Started with testing -95: localization testing for greek version of an English software system Subcontractor-side on various projects: moving from localizations to functional testing Tried being in a developer role after thinking testers are not respected: the problem is not per role Teaching testing at HUT, giving public presentations and courses on testing Researcher in testing, consulting various organizations, teaching, being a test manager and a tester, with and without test automation At F-Secure with product business 2005 – 2008 Moved to insurance pension sector, first with subcontractor side, now on customer side (Ilmarinen) as test manager / test specialist. Doing testing-related training on the side of a normal day-to-day job to get to meet brilliant testers in Finland. TESTER AND A WALKING TESTING DICTIONARY AGILE PROJECTS @ F-SECURE PLAN-DRIVEN PROJECTS @ ILMARINEN

3. RealizationsaboutNature of Testing 20 14 1639 5±2 3

5. http://www.dothetest.co.uk/whodunnit.htmlThe samemaynotbe the sameifyouchangeanyvariable. Testing is notaboutcoveringall of itagainst a specification, butthinkingabout ”is there a problem, is itrelevant”. Human memory is limited – ”human envelope”. We need documentation as support, especially when working under pressure. Documentation is huge expense, which only passes small portion of information. Why not create the documentation at the time we understand the needs best – towards the end of project.

6. Exploratory Testing:Better tests, better testers! An approach to software testing Emphasized freedom and responsibility of an individual in a process where continuous optimization of value of information is important “Any testing process that involves simultaneous learning, test design, and execution.” – James Bach & Cem Kaner Disciplined, planned and controlled testing that emphasizes continuous learning Research in Finland, Itkonen et al. 2007 No significant difference in results for preplanned test cases and exploratory testing More false alarms with test cases Comparing overall effort: significantly more in test-case based testing Unknown territory Test-related learning Design of new tests Test execution Result interpretation 5

7. There’s a Difference! A comment I hear often on the ET course I give: ”I’ve always been doing this just did not give it a name” I must emphasize that: I require more out of exploratory testing than just ”going where you feel like while testing” Ask yourself: Can you defend your choices of strategy and tactics? Can you explain what you’ve done in words that don’t just explain numbers of bugs found? How do you know if you’re done or not? 6 ”My testing would be unsystematic ad hoc testing if I could not tell the story of my tests, remember what I’ve tested or what my strategy was, or relate that back to my mission” –James Bach

8. The Scale of Tester’s Freedom Role-based sandboxes Vague scripts Fragment test cases (scenarios) Pure scripted Freestyle Charters ”To what extent am I in control of the test and from where did the idea originate?” 7

9. Two Sides of Exploratory Testing For the tester For the manager Value Fun Freedom Controllability Flexibility Reliability Professionalism Visibility Respect 8

10. Framework of Management Vision (“Sandbox”) Current Charter Other Charters Details Coaching Quality Report Perception of quality and coverage Debriefing Charter backlog of the future testing ”A day’s work” Past Results Obstacles Outlook Feelings Next in importance! Test Manager Metrics summary # ? x Out of budget #, ?, x, + 20:20:60 Session sheets of the past testing Bug Reports Idea of exploration Tester 9

11. The Pieces in Management Framework A disciplined tester replanning on various levels Session with charter that provides a report Classification of information created as metrics Prioritizing of what test idea comes next Supporting reporting by debriefing Supporting skills development by coaching Creating a combined judgement of quality by quality reporting 10

12. Testautomation / tools in ET Anyform of acquiringquality-relatedinformationfits into exploratorytesting It’snotmanual, it’sbrain-engaged – and for makingthathappen, youneed to besmart with automation and tools! Example: youmightnot at firstknowwhatyou’relooking for... Search "<ns1:Koodi>" (443 hits in 169 files) in VE Y:LLUAPAestausokonaiseläketurvan-otensaintatiedot-VEaeAnsaintatiedot-VE__ 20110307 14-22-15.xml (4 hits) Line 1313: <ns1:Koodi>YL130I</ns1:Koodi> Line 1317: <ns1:Koodi>LAPA_172_011</ns1:Koodi> Line 1321: <ns1:Koodi>67</ns1:Koodi> Line 1324: <ns1:Koodi>67</ns1:Koodi> A lot of textcutawayfromhere…

13. Potentially Useful Tools for Exploratory Testing SnapzPro http://www.ambrosiasw.com/utilities/snapzprox/ HyperSnap http://www.hyperionics.com/hsdx/ TestExplorer http://www.sirius-sqa.com/ BBTestAssistant http://www.bbsoftware.co.uk/ Spector http://www.spector.com/spectresoft.html Cygwin http://www.cygwin.com/ UnixUtils http://unxutils.sourceforge.net/ Regular Expressions http://www.ilovejackdaniels.com/cheat-sheets/regular-expressions-cheat-sheet/ Process Monitor http://www.microsoft.com/technet/sysinternals/processesandthreads/processmonitor.mspx Process Explorer http://www.microsoft.com/technet/sysinternals/processesandthreads/processexplorer.mspx Holodeck http://www.securityinnovation.com/holodeck/index.shtml perlclip http://www.satisfice.com/tools.shtml DataGraph http://www.visualdatatools.com/DataPlot/index.html Vermont High Test http://www.vtsoft.com/ Firebug http://www.getfirebug.com/ iMacros http://www.iopus.com/imacros/firefox/ JMeter http://jakarta.apache.org/jmeter/ VirtualPC http://www.microsoft.com/windows/products/winfamily/virtualpc/default.mspx VMWare http://www.vmware.com/ Parallels http://www.parallels.com/

14. TestIdeas / Quick-and-Dirty Download the full 2-page CheatSheet with ideasfromElisabeth Hendrickson, James Lyndsay, and Dale Emery on Qualitytree.com

15. (Exploratory) Testing Dynamics Source: Adapted from James Bach, Jon Bach, Michael Bolton. Exploratory Testing Dynamics. v.2.2. 2009 ”…skills that comprise professional and cost effective exploration of technology. Each is distinctly observable and learnable, and each is necessary to exploratory work.” ”Exploratory testing spirals upward toward a complete and professional set of test artifacts” ”A set of considerations designed to help you test robustly or evaluate someone else’s testing.” ” To develop ideas or search a complex space quickly yet thoroughly, not only must you look at the world from many points of view and perform many kinds of activities”

16. Exploration SkillsSource: Adapted from James Bach, Jon Bach, Michael Bolton. Exploratory Testing Dynamics. v.2.2. 2009 Done To Do Issues Coverage All sources available Best use of time – effective and efficient work Making models Tool support – creative solutions Risk-based testing – scientific approach Keeping one’s eyes open

17. To EndThis With: ExploratoryTestingStrenghts and BlindSpots Strengths Value-focused: stakeholders, risks, good-enoughquality and knowledge of quality Leverageseachtester’sstrengths Makestestersbettercontinuously Enablesadapting to changes and surprises Revealsinformationthatwasnotavailablewithoutlearningwhileusing the system Avoidnon-valuableduplicativework in testingthroughfocus on results Highbugfindingrates BlindSpots Requirescalendartime for learning with the software: the lessweknow, the morewe’relikely to miss Limited byeachtester’sweaknessesifnotmitigatedbyoverlappingassignmentsbased on individual’scompetencies Skilled and disciplinedwork – canbelearned with mentoring, nottransferredthroughdocumentation

18. WHY do testers look both ways when crossing a one-way street? It may be one-way on this corner, two ways on that corner. You may only *think* it's a one-way street. Someone might be backing up Ears don't warn you when a bicycle is coming the wrong way. They realize people can be fooled by preconceptions. When helping someone, you've extra duty to seek improbables. Source: Various tweets by testers in 2010.