Exploratory Testing Is Now in Session

596 views

Published on

The nature of exploration, coupled with the ability of testers to rapidly apply their skills and experience, make exploratory testing a widely used test approach—especially when time is short. Unfortunately, exploratory testing often is dismissed by project managers who assume that it is not reproducible, measurable, or accountable. If you have these concerns, you may find a solution in a technique called session-based test management (SBTM), developed by Jon Bach and his brother James to specifically address these issues. In SBTM, testers are assigned areas of a product to explore, and testing is time boxed in “sessions” that have mission statements called “charters” to create a meaningful and countable unit of work. Jon discusses—and you practice—the skills of exploration using the SBTM approach. He demonstrates a freely available, open source tool to help manage your exploration and prepares you to implement SBTM in your test organization.

1 Comment
1 Like
Statistics
Notes
No Downloads
Views
Total views
596
On SlideShare
0
From Embeds
0
Number of Embeds
15
Actions
Shares
0
Downloads
12
Comments
1
Likes
1
Embeds 0
No embeds

No notes for slide

Exploratory Testing Is Now in Session

  1. 1. MM PM Tutorial 9/30/2013 1:00:00 PM "Exploratory Testing Is Now in Session" Presented by: Jon Bach eBay Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
  2. 2. Jon Bach eBay, Inc. With more than eighteen years of experience in software testing, Jon Bach has held technical and managerial positions in companies including Hewlett-Packard and Microsoft. In his current role as director of Live Site Quality for eBay, Jon is dedicated to building “end-to-end” tests (activity flows) in eBay’s core sites to discover important bugs that threaten its core business. He is most notable for creating, with his brother James, Session-Based Test Management, a method to manage and report exploratory testing.
  3. 3. Exploratory Testing: Now in Session Jon Bach QE Director, eBay jobach@ebay.com eBay Inc. confidential STAR West 2013 Testing is about DISCOVERY… eBay Inc. confidential 1
  4. 4. …finding emerging context eBay Inc. confidential enhanced by conference… eBay Inc. confidential 2
  5. 5. ...affected by inference eBay Inc. confidential …framed by reference eBay Inc. confidential 3
  6. 6. Key Idea Exploratory testing is a combination of Imagination and Discovery eBay Inc. confidential } } } } } How to Discover with Your Imagination CIDTESTDSFDPOTCRUSSPICSTMPLFDSFSCURA Project Environment Product Elements Quality Criteria Development Criteria General Test Techniques Customers Information Developer relations Team Equipment & tools Schedule Test Items Deliverables Structure Function Data Platform Operations Time Capability Reliability Usability Security Scalability Performance Installability Compatibility Supportability Testability Maintainability Portability Localizability Function testing Domain testing Stress testing Flow testing Scenario testing Claims testing User testing Risk testing Automatic testing eBay Inc. confidential 4
  7. 7. “The” ET Definition A style of software testing… that emphasizes the personal freedom… and responsibility of the individual tester… to continually optimize the quality of his/her work… by treating test-related learning… test design… test execution… and test result interpretation… as mutually supportive activities… that run in parallel… throughout the project. -- Cem Kaner, 2006 eBay Inc. confidential The agile nature of exploration and the ability of testers to rapidly apply their skills and experience make exploratory testing a widely used test approach—especially when time is short. But exploratory testing is often dismissed by project managers who assume that exploratory testing is not reproducible, measurable, or accountable. eBay Inc. confidential 5
  8. 8. Why this tutorial? 1) Exploratory testers want respect: When testers explore during testing, they find great bugs. However, since they often don’t know how to describe their thinking, it’s considered to be dismissed as “playing around”. 2) The documentation dilemma: Project managers may insist that all testing be documented, but Agile does not emphasize that. So how to balance time spent documenting with time spent testing? 3) Your work might be scrutinized: You may have to give a report someday about something you did that was exploratory – like attending this conference. eBay Inc. confidential My missions for you 1) Discuss — and practice — a way to manage and measure exploratory testing. 2) See some free tools that support sessionbased exploration. 3) Learn why exploratory testing can be thoughtful, structured, and measurable. 4) Have some fun. eBay Inc. confidential 6
  9. 9. A structure for exploration Mission Tactics Story eBay Inc. confidential Exploratory exercise Test this… eBay Inc. confidential 7
  10. 10. Scrutiny What’s being asked What they may be thinking What was your mission? Remind me what I told you to do. Why did you do that? How did it go? Were you careful or reckless? What should I be worried about? How far did you get? Are we closer to shipping? Can you help me know our status? Need anything? Can I speed this along? Do you need more of my time? When will you be done? Will I get my bonus? I have a new task for you… eBay Inc. confidential How could you accurately report your testing? eBay Inc. confidential 8
  11. 11. Session-Based Exploration The “Session” 1) 2) 3) Time Box Reviewable Result Debriefing eBay Inc. confidential Time Box: focused effort, fixed duration Short: 60 minutes (+-15) Normal: 90 minutes (+-15) Long: 120 minutes (+-15) – Brief enough: • for accurate reporting • to allow flexible scheduling • to allow course correction – Long enough: • to get solid testing done • for efficient debriefings eBay Inc. confidential 9
  12. 12. The deliverable: a session report • Charter – #AREAS CHARTER ----------------------------------------------Analyze MapMaker’s View menu functionality and report on areas of potential risk. • Notes #AREAS OS | Windows 2000 Menu | et-jsb-010417-c.sesView Strategy | Function Testing Strategy | Functional Analysis Sample START ----------------------------------------------5/30/00 03:20 pm TESTER ----------------------------------------------Jonathan Bach TASK BREAKDOWN ----------------------------------------------#DURATION short   Metrics – – – – – Bugs – #TEST DESIGN AND EXECUTION 65 #BUG INVESTIGATION AND REPORTING 25 #BUG #SESSION SETUP 20 #DURATION #TEST DESIGN AND EXECUTION #SESSION SETUP #BUG INVESTIGATION AND REPORTING #CHARTER / OPPORTUNITY  Issues – #ISSUE eBay Inc. confidential Anticipating Scrutiny Agenda: “PROOF” Past Results Obstacles Outlook Feelings Past: What kinds of things did you do? Results: What were your findings? Obstacles: What slowed your mission? Outlook: Where do we go from here? Feelings: How are you feeling about what happened? eBay Inc. confidential 10
  13. 13. Session report My session write-up from exploratory testing for Agile2010… CHARTER ----------------------------------------------Analyze MapMaker’s View menu functionality and report on areas of potential risk. #AREAS .LINK OS | Windows 2000 Menu | Strategy | Function Testing Strategy | Functional Analysis START ----------------------------------------------5/30/00 03:20 pm TESTER ----------------------------------------------Jonathan Bach TASK BREAKDOWN ----------------------------------------------#DURATION short #TEST DESIGN AND EXECUTION 65 #BUG INVESTIGATION AND REPORTING 25 #SESSION SETUP 20 eBay Inc. confidential eBay-related sample CHARTER ----------------------------------------------Analyze MapMaker’s View menu functionality and report on areas of potential risk. #AREAS LINK Menu | et-jsb-010417-c.sesView Strategy | Function Testing Strategy | Functional Analysis START ----------------------------------------------5/30/00 03:20 pm TESTER ----------------------------------------------Jonathan Bach TASK BREAKDOWN ----------------------------------------------#DURATION short #TEST DESIGN AND EXECUTION 65 #BUG INVESTIGATION AND REPORTING 25 #SESSION SETUP 20"C:Documents and SettingsjobachDesktopLive Siteisrat_sessionsSession_Listing_Promo.txt" eBay Inc. confidential 11
  14. 14. Exercise: An “exploration and analysis” session eBay Inc. confidential When to use SBTM Getting support: “We have thousands of test cases and a huge regression suite, but some of our best bugs come from ad hoc testing. Why is that?” Improving skill: “How she finds those great bugs without test cases, I’ll never know. I guess some people are just natural explorers -- you either have it or you don’t -- and I just don’t have her knack for it.” Earning trust and autonomy: “The triage team needs more information about these 3 bugs, can you give me that by the end of the day?” eBay Inc. confidential 12
  15. 15. First: what’s there to test? eBay Inc. confidential Second: ideas for charters • What item for sale is nearest to you location? Submitted the most and sold the least? What item appears most with no bids? What are the most recent items posted for any given category? Which category has the most items? Which search strings create null queries? What are 3 examples of broad queries vs 3 examples of ambiguous queries? What queries create invalid results? How do you find "sold" items? What is the most common item? What is the most bizarre item? What is the most expensive item? Is it possible to restrict results to one category without the categories field? Which seller has listed the most items which have not sold? What search string takes the longest to return results? Is there a search that works differently on different browsers? What was the highest bid for an item in support of any charity? Find an item that the seller has no business selling! Find an item you find only on eBay, not Amazon. What item could you find that had the most bids? How many eBay stores are there? What is the oldest object (not item) for sale on ebay? Are results different if you are logged in? Try a search on ebay.com and then try the same on on your mobile. What are the 3 major differences? What item has the greatest difference between the highest and second highest bids? What user has bid the most but won the least? What item has been on eBay the longest? Is there a way to tell the age of the item ID? Who has sold the most items? Is there a way to find the most commonly misspelled item? eBay Inc. confidential What item can be found in the most categories? 13
  16. 16. Sources of charters • Documentation / Specifications • Web forums • Previous products • Team members • Competing products • Your expertise • Help files • Manuals eBay Inc. confidential High-level sources Test Plan Evaluation Model http://www.satisfice.com/tools/tpe-model.pdf Test Planning Checklist http://www.satisfice.com/tools/build-the-plan.pdf Heuristic Test Strategy Model http://www.satisfice.com/tools/satisfice-tsm-4p.pdf eBay Inc. confidential 14
  17. 17. Charters • General charters may be necessary at first: • “Analyze the Insert Picture function” • Specific charters provide better focus, but take more effort to design: • “Test clip art insertion. Focus on stress and flow techniques, and make sure to insert into a variety of documents. We’re concerned about resource leaks or anything else that might degrade performance over time.” eBay Inc. confidential The power of collaboration As a London Underground rider, I want to enter my ticket into the kiosk, Because I want to go to Heathrow. What are the possible outcomes? http://www.faqs.org/faqs/uk/transport-london/section-6.html eBay Inc. confidential 15
  18. 18. The session report • Charter – #AREAS CHARTER ----------------------------------------------Analyze MapMaker’s View menu functionality and report on areas of potential risk. • Notes #AREAS OS | Windows 2000 Menu | et-jsb-010417-c.sesView Strategy | Function Testing Strategy | Functional Analysis Sample START ----------------------------------------------5/30/00 03:20 pm TESTER ----------------------------------------------Jonathan Bach TASK BREAKDOWN ----------------------------------------------#DURATION short  #TEST DESIGN AND EXECUTION 65  Metrics – – – – – #BUG INVESTIGATION AND REPORTING 25 Bugs – #BUG #SESSION SETUP 20 #DURATION  #TEST DESIGN AND EXECUTION #SESSION SETUP #BUG INVESTIGATION AND REPORTING #CHARTER / OPPORTUNITY Issues – #ISSUE eBay Inc. confidential A boring (Boeing) story eBay Inc. confidential 16
  19. 19. On October 30, 2006, Alaska Airlines flight 61, a Boeing 737-200, N740AS, took off from runway 34 right at Seattle-Tacoma International Airport, Seattle, Washington. There were no injuries to the 71 passengers or 5 crewmembers and there was no damage to the airplane. The airplane was operating under the provisions of 14 CFR Part 121 and continued uneventfully to its destination of Juneau International Airport, Juneau, Alaska. eBay Inc. confidential The Report eBay Inc. confidential 17
  20. 20. Why the report? 1. The government is collecting statistics on runway incursions, which are potentially deadly. 2. Since the pilots or ground controllers are subject to reprimand or dismissal for this sort of thing, a legal record that it happened is necessary. 3. Ground controllers or signage may have caused the problem. You need a record of the event, and investigation, to discover if there is another accident waiting to happen. eBay Inc. confidential Styles I’ve seen 1) Novel Narrative 2) Flight Data Recorder 3) ESPN Highlight eBay Inc. confidential 18
  21. 21. Clark’s notes… Captain Clark, River Dubois opposite the mouth of the Missouri River, 13 May 1804 “I dispatched an express this morning to Captain Lewis at St. Louis. All our provisions, goods, and equipage on board of a boat of 22 oars [party], a large pirogue of 71 oars [in which 8 French], a second pirogue of 6 oars [soldiers], complete with sails, &c. Men completed with powder cartridges and 100 balls each, all in health and readiness to set out. Boats and everything complete, with the necessary stores of provisions and such articles of merchandise as we thought ourselves authorized to procure-though not as much as I think necessary for the multitude of Indians through which we must pass on our road across the continent.” eBay Inc. confidential What to write while exploring Historical Explorer Tester Observations • drawings of flora / fauna descriptions of indigenous people • landmarks • (To the degree you think they are relevant to stakeholders) • • feature model text from log files • text from dialogs Conjectures • what is this thing? where should we go today? • how do we get there? • new orders from HQ? • are those people hostile? • (Inferences based on experiences. After I test, I think I know something) • • Project information • mission supplies and staff • latitude / longitude • death and disease • supply status • (Independent of observer) • • test ideas questions • product and project issues • concerns • risks charter test actions • config info • build details • tools used eBay Inc. confidential 19
  22. 22. Scrutiny: what happened here? “(Orderly Book) Camp Mouth of the Kansies June 29, 1804 Ordered --- A Court Martiall will Set this day at 11 oClock... for the trial of John Collins and Hugh Hall... John Collins Charged "with getting drunk on his post this Morning out of whiskey put under his charge as a Sentinal, and for suffering Hugh Hall to draw whiskey out of the Said Barrel intended for the party." ...The commanding Officers approve of the Sentence of the Court and orders that Punishment take place at half past three this evening, at which time the party will Parrade for inspection.” June 29, 1804 William Clark eBay Inc. confidential Jon Bach, 2001 Sample test notes from actual exploratory testing : et-jsb-010416-a.ses et-jsb-010416-b.ses (for Issues) et-jsb-010416-c.ses (a map for more ideas) et-jsb-010417-a.ses (Claims Testing) et-jsb-010417-b.ses (for Bugs) et-jsb-010417-c.ses (for referenced notes) et-jsb-010418-c.ses (more Claims) eBay Inc. confidential 20
  23. 23. How much detail is enough? 1) Sufficient benefits 2) No critical problems 3) Benefits outweigh problems 4) All things being equal, further documentation (or time, effort, testing, editing, etc.) is more harmful than helpful All 4 conditions must be met to know when you’re done, (From brother James’ Rapid Software Testing course) But ask: “good enough for who, what and when?” eBay Inc. confidential Metrics and Mechanics Gathering empirical data about the object of your study; collecting different kinds of data, or data about different aspects of the object. Designing experiments and establishing lab procedures. eBay Inc. confidential 21
  24. 24. The session report • Charter – #AREAS CHARTER ----------------------------------------------Analyze MapMaker’s View menu functionality and report on areas of potential risk. • Notes #AREAS OS | Windows 2000 Menu | et-jsb-010417-c.sesView Strategy | Function Testing Strategy | Functional Analysis Sample START ----------------------------------------------5/30/00 03:20 pm TESTER ----------------------------------------------Jonathan Bach TASK BREAKDOWN ----------------------------------------------#DURATION short  #TEST DESIGN AND EXECUTION 65  Metrics – – – – – #BUG INVESTIGATION AND REPORTING 25 Bugs – #BUG #SESSION SETUP 20 #DURATION  #TEST DESIGN AND EXECUTION #SESSION SETUP #BUG INVESTIGATION AND REPORTING #CHARTER / OPPORTUNITY Issues – #ISSUE eBay Inc. confidential What to measure Bug Investigation Test Design and Execution (and Reporting) Session Setup eBay Inc. confidential 22
  25. 25. Reporting the metrics • Test, Bug, and Setup are orthogonal categories, but all we’re only tracking interruptions to testing • Nearest 5% or 10% is good enough • If activities are done simultaneously, report the highest precedence activity: T, B, then S eBay Inc. confidential TBS Report Test Design & Execution Bug Investigation Environment Config Automation (BLOCKERS) Learning-Gathering Requirements 15% 35% 15% 10% eBay Inc. confidential 25% 23
  26. 26. Sample project HTML status report eBay Inc. confidential Tools 1) Scan Tool 2) Session Tester (Jonathan Kohl) 3) Rapid Reporter (Shmuel Gershon) (James & Jon Bach) eBay Inc. confidential 24
  27. 27. Diagnosing productivity Test 28% • Do these proportions make sense? Bug 4% • How do they change over time? • Is the reporting protocol being followed? Setup 6% Non-Session 61% Opportunity 1% 300.0 250.0 200.0 150.0 100.0 50.0 0.0 Inc. confidential eBay 5/26 6/9 6/23 7/7 7/21 8/4 8/18 Estimating a Test Cycle 1. How many perfect sessions (100% on-charter testing) does it take to do a cycle? (let’s say 40) 2. How many sessions can the team (of 4 testers) do per day? (let’s say 3 per day, per tester = 12) 3. How productive are the sessions? (let’s say 66% is on-charter test design and execution) 4. Estimate: 40 / (12 * .66) = 5 days 5. We base the estimate on the data we’ve collected. When any conditions or assumptions behind this estimate change, we will update the estimate. eBay Inc. confidential 25
  28. 28. Testing is like looking for words… ??? (10 minutes) eBay Inc. confidential Debriefing Mission A: What happened during the session? Mission B: How many words did you find? Mission C: What obstacles did you face? Mission D: Do you think you found all the words? All groups: How did you feel about your mission? Agenda: “PROOF” Past Results Obstacles Outlook Feelings eBay Inc. confidential 26
  29. 29. Next steps? 1) Tell the “tactics” story: Practice using terminology for the skills used during exploration. It has helped managers understand the sapience and discipline of exploration. 2) Try a three-week SBTM pilot: Frame your explorations in sessions (time-boxed effort with a report) and see how your stakeholders react. 3) Use PROOF to anticipate scrutiny: Testing ourselves is just as important as testing software. It has won testers more credibility, autonomy, and respect. eBay Inc. confidential More info • ET Dynamics: http://www.satisfice.com/articles/et-dynamics.pdf • Test Heuristics and Planning http://www.satisfice.com • Context-Driven Software Testing http://groups.yahoo.com/group/software-testing • Center for Software Testing Education and Research http://www.testingeducation.org/BBST • Books related to Exploratory Testing skills and tactics http://www.testingreflections.com/node/view/3190 • Scenario testing examples http://www.testingeducation.org/a/scenario2.pdf eBay Inc. confidential 27

×