Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Tips for Writing Better Charters for Exploratory Testing Sessions by Michael D Kelly

11,344 views

Published on

We will look at some common pitfalls encountered when chartering your testing for session-based exploratory testing. After a brief overview of the session-based test management process we will jump into specific practices and techniques to help you and the rest of your team achieve better coverage and find better bugs. A presentation for the EuroSTAR Software Testing Community from September 2012.

Published in: Technology

Tips for Writing Better Charters for Exploratory Testing Sessions by Michael D Kelly

  1. 1. Today’s webinar is presented by Michael Kelly. He will discussTips for Writing Better Charters for Exploratory TestingSessions.Welcome to the EuroSTAR WebinarSerieswww.eurostarconferences.comThis webinar is due to start at 2pm. Make sure you stick around at the end for theQ&A session and continue the conversation with the speaker on Twitter after theshow!@esconfs#esconfs
  2. 2. Tips for Writing Better Charters forExploratory Testing SessionsMichael Kelly, DeveloperTownMike Kelly is a managing partner at DeveloperTown, aventure development firm. Mike is a contributing author to“How to Reduce the Cost of Software Testing”, is a pastdirector and president for the Association for SoftwareTesting, and a co-founder of the Indianapolis Workshops onSoftware Testing, a series of ongoing meetings on topics insoftware testing.www.eurostarconferences.com
  3. 3. Join the conversation on Twitter#esconfs@esconfs@michael_d_kellywww.eurostarconferences.comContinue the conversation with the speaker on Twitter after the show!
  4. 4. MICHAEL KELLYManaging Partner, DeveloperTownwww.DeveloperTown.comPast President, Association for Software Testingwww.AssociationForSoftwareTesting.orgArticles and Blogswww.MichaelDKelly.comwww.eurostarconferences.com@esconfs#esconfs
  5. 5. “The tester, the way they work,and their ability to work in thefuture, is as important as theproduct being tested.”www.eurostarconferences.com@esconfs#esconfs“One of the most difficult aspects ofsoftware testing is coming up with goodtest ideas. It doesnt matter how youredoing your testing: scripted vs.exploratory, manual vs. automated, orperformance vs. functional.”
  6. 6. EXPLORATORY TESTING“Simultaneous learning, test design, and test execution.” – James Bachwww.eurostarconferences.com@esconfs#esconfs
  7. 7. www.eurostarconferences.com@esconfs#esconfspure scripted freestyle exploratorychartersvague scriptsfragmentarytest cases(scenarios)roles-image from Jon Bach
  8. 8. SESSION BASED TESTMANAGEMENTA method for managing exploratory testing. Created by Jon and JamesBach, refined by others.www.eurostarconferences.com@esconfs#esconfs
  9. 9. • charters• time-boxed• session notes• debriefs• team prioritization• ad-hoc test documentation• ad-hoc test automation• dynamic metrics and reportingwww.eurostarconferences.com@esconfs#esconfs
  10. 10. CHARTERINGMaking your own decisions about what you will work on and how youwill work. Understanding your client’s needs, the problems you mustsolve, and assuring that your work is on target.www.eurostarconferences.com@esconfs#esconfs
  11. 11. What the tester thinks• What am I suppose to bedoing?• I need more time!• This is boring…• What data should I be usingfor this?• Didn’t someone already testthis?• Is this really the mostimportant thing for me to befocused on?What happens on the project• Areas of the product don’tget covered.• Entire categories of riskdon’t get tested.• Duplication of effort.• The team does not workbased on priority.• Testers get frustrated.• Poor visibility into actualprogress.www.eurostarconferences.com@esconfs#esconfsBAD CHARTERING
  12. 12. ESSENTIAL ELEMENTS OF ACHARTERwww.eurostarconferences.com@esconfs#esconfsRisk, Coverage, and Timeframe
  13. 13. COVERAGEwww.eurostarconferences.com@esconfs#esconfsWhat am I testing? Which features, screens, artifacts, or areas ofthe product should I be touching while testing?RISKWhy am I running this test? What could go wrong? What am Ilooking for?TIMEFRAMEHow long should it take to perform this testing? Are there setup tasks Ishould do before I officially start? What do I need to do when I’m done?
  14. 14. When you look at a list of 10 charters,you should see 10 distinct testing missions,equating to around eight to 10 hoursof heads-down testing.www.eurostarconferences.com@esconfs#esconfs
  15. 15. BETTER CHARTERINGwww.eurostarconferences.com@esconfs#esconfsList specific risks and coverage targetsLeverage mnemonics for risk and coverage ideasDevelop a knowledge base for risk/coverage summariesCompare missionsTry developing a couple of templatesCharter for smaller sessions, then affinity map to create larger sessionsThumb vote for priorityUse testing polaritiesLet charters emerge over timeTrack metrics
  16. 16. LIST SPECIFIC RISK ANDCOVERAGE TARGETSTest the portal for reporting accuracy.Which reports?What do you mean by accuracy?Test reports X, Y, and Z for errors related to startand end time selection criteria, summing,totaling, and rounding.www.eurostarconferences.com@esconfs#esconfs
  17. 17. MNEMONICS FOR RISKAND COVERAGE IDEAS• Application Touring: FFC CUTS VIDS• Coverage: SFDPOT• Coverage and Risk: FIBLOTS• Quality Criteria: CRUSSPICSTMPL• Test Techniques: FDSFSCURA• Oracles: HICCUPPS• Session Notes: MCOASTERwww.eurostarconferences.com@esconfs#esconfs
  18. 18. RISK AND COVERAGEKNOWLEDGE BASEAs a team, build out a wiki detailing the nuance ofvarious coverage areas and specific areas of risk.Commonly captured items will include:• Detailed feature listings/checklists• Integration points and data flow• Example and common test data / test beds• Known issues/gotchas• Application-specific or feature-specific testtechniqueswww.eurostarconferences.com@esconfs#esconfs
  19. 19. COMPAREMISSIONS• Test feature X• Stress test feature X• Performance test feature XWhat’s the difference?www.eurostarconferences.com@esconfs#esconfs
  20. 20. My mission is to test<insert risk here> for<insert coverage here>Some examples:• My mission is to test forvarious boundary errors forMicrosoft Word’s bullets andnumbering feature.• My mission is to test foraccurate error messagingpop-ups for Ford MotorVehicles Build and Pricewebsite.• My mission is to test for SQLinjection vulnerabilities forapplication login andadministration screens.www.eurostarconferences.com@esconfs#esconfsCHARTERTEMPLATE
  21. 21. KiwanisInternationalApp(possible10to15mincharters)1. Test for boundary related issues related to projects2. Test for boundary related issues related to project reports3. Test for deliverability related issues related to project emails4. Test for data quality issues with clubs available on the app5. Test for data accuracy issues related to calculations on project reports6. Test for data accuracy issues related to calculations on service andfundraising summary stats7. Test for data accuracy issues related to Piggy Bank commitmentcalculations8. Perform basic link validation for items in the news feed9. Perform a copy review of the various descriptive dialogs in the applooking for typos and grammar issues10. Test for performance related issues related to mapping capabilities11. Stress test the application to identify conditions that crash the appwww.eurostarconferences.com@esconfs#esconfsCHARTER FORSMALLER SESSIONS
  22. 22. Walk your charters quickly as a teamUse thumb voting to see if there isagreement around clarity and scopeThe particularly contentious charterswill lead the team to discussion anddebate (and new charters)Thumb Vote 101High Priority“We need to run thischarter.”Medium Priority“If we have time, weshould run thischarter.”Low Priority“It’s a test, and wecould run it, but likelythere are better usesof our time.”www.eurostarconferences.com@esconfs#esconfsTHUMBVOTE
  23. 23. TESTINGPOLARITIESExample Polarities:• Warming up vs. cruising vs. cooling down• Doing vs. describing• Careful vs. quick• Data gathering vs. data analysis• Working with the product vs. reading about the product• Working with the product vs. working with the developer• Training (or learning) vs. performing• Product focus vs. project focus• Solo work vs. team effort• Your ideas vs. other peoples’ ideas• Lab conditions vs. field conditions• Current version vs. old versions• Feature vs. feature• Requirement vs. requirement• Coverage vs. oracles• Testing vs. touring• Individual tests vs. general lab procedures and infrastructure• Testing vs. resting• Playful vs. seriousThe Process:• When you charter your tests, includepolarities explicitly in your mission.• This practice is particularly effective forshorter test charters (20-30 minutes).• Or, when you charter, use the polarities tohelp you identify new charter ideaswww.eurostarconferences.com@esconfs#esconfsSource: http://www.satisfice.com/blog/wp-content/uploads/2009/10/et-dynamics22.pdfIN YOUR CHARTER:• Testing vs. Touring• Feature vs. Feature• Requirement vs. RequirementWHILE CHARTERING:Individual Tests vs. General Lab ProceduresCoverage vs. OraclesLab Conditions vs. Field Conditions
  24. 24. LET CHARTEREMERGE OVER TIMEwww.eurostarconferences.com@esconfs#esconfs1. Pull down acharter for testing2. Execute thatcharter3. Debrief4. After yourdebrief, add newcharters andreprioritizeexisting charters5. Repeat
  25. 25. Time management is critical to chartering.Metrics help provide visibility into what’sreally happening. Tagging allows for easiermetrics gathering.I regularly tag the following:• bugs or issues for isolation (b)• possible new charters (c)• open questions (?)• tasks for automation (a)• tasks for documentation (doc)• setup, testing, investigation #...Some examples:• Actual execution time vs.estimated time• Confidence factor of testeron their coverage of thestated risks• Number of follow upcharters resulting from lackof time• Percentage of session timespent setting up for testing• Percentage of session timespent testing• Percentage of session timespent investigatingproblemswww.eurostarconferences.com@esconfs#esconfsTRACKMETRICS
  26. 26. BETTER CHARTERINGwww.eurostarconferences.com@esconfs#esconfsList specific risks and coverage targetsLeverage mnemonics for risk and coverage ideasDevelop a knowledge base for risk/coverage summariesCompare missionsTry developing a couple of templatesCharter for smaller sessions, then affinity map to create larger sessionsThumb vote for priorityUse testing polaritiesLet charters emerge over timeTrack metrics
  27. 27. EuroSTAR Webinar ArchiveAccess almost 50 on-demand software testing webinars by the worlds leadingtesting experts. Topics include….www.eurostarconferences.comAgile, Automation, Test Management, Process Improvements, Risk-based Testing, PerformanceTesting, Test Documentation, Cloud Testing, Session-based Testing, Weekend TestersVisit the archive!http://www.eurostarconferences.com/community/member/webinar-archive@esconfs#esconfs
  28. 28. The EuroSTAR Community.Are you a member?Did you know we have an online resource area with almost …www.eurostarconferences.com600 software testing presentations, 60 topicalwebinars, over 40 videos and podcasts and 20 experteBooksDo you know that membership is completely free and you can access thesepremium testing resources anytime?Join today!http://www.eurostarconferences.com/community/member@esconfs#esconfs

×