The nature of exploration, coupled with the ability of testers to rapidly apply their skills and experience, make exploratory testing a widely used test approach—especially when time is short. Unfortunately, exploratory testing often is dismissed by project managers who assume that it is not reproducible, measurable, or accountable. If you have these concerns, you may find a solution in a technique called session-based test management (SBTM), developed by Jon Bach and his brother James to specifically address these issues. In SBTM, testers are assigned areas of a product to explore, and testing is time boxed in “sessions” that have mission statements called “charters” to create a meaningful and countable unit of work. Jon discusses—and you practice—the skills of exploration using the SBTM approach. He demonstrates a freely available, open source tool to help manage your exploration and prepares you to implement SBTM in your test organization.
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Exploratory Testing Is Now in Session
1. TI
AM Tutorial
4/30/13 8:30AM
Exploratory Testing Is
Now in Session
Presented by:
Jon Bach
eBay, Inc.
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
2. Jon Bach
Jon Bach works for eBay (San Jose) as a QA director for the Buyer Experience team. Formerly a manager
of corporate intellect and senior test consultant at Quardev, Jon has been in testing since 1995 with
experience that includes managing teams at Microsoft, HP, and LexisNexis. The co-inventor (with his
brother James) of session-based test management, Jon is an award-winning speaker on test
management and exploratory testing. He’s mostly known for his “half-baked” ideas about how to inspire
innovation and create a learning culture for test teams. Find Jon on Facebook and Twitter @jbtestpilot.
View his presentations at quardev.com/articles and his blog at jonbox.wordpress.com.
3. Exploratory Testing:
Now in Session
Jon Bach
QE Director, eBay
jobach@ebay.com
STAR East 2013
eBay Inc. confidential
11. Why this tutorial?
1) Exploratory testers want respect: When testers explore during testing,
they find great bugs. However, since they often don’t know how to
describe their thinking, it’s considered to be dismissed as “playing around”.
2) The documentation dilemma: Project managers may insist that all testing
be documented, but Agile does not emphasize that. So how to balance
time spent documenting with time spent testing?
3) Your work might be scrutinized: You may have to give a report someday
about something you did that was exploratory – like attending this
conference.
eBay Inc. confidential
16. Key Idea
If you’re not free to think,
or learn,
and adapt
while you test,
it’s not exploration.
eBay Inc. confidential
17. My missions for you
1) Discuss — and practice — a way to manage and
measure exploratory testing.
2) See some free tools that support sessionbased
exploration.
3) Learn why exploratory testing can be thoughtful,
structured, and measurable.
4) Have some fun.
eBay Inc. confidential
18. A structure for exploration
Mission
Tactics
Story
eBay Inc. confidential
19. Session report
My session writeup from exploratory testing for
Agile2010…
CHARTER
---------------------------------------------Analyze MapMaker’s View menu functionality and
report on areas of potential risk.
#AREAS
.LINK
OS | Windows 2000
Menu |
Strategy | Function Testing
Strategy | Functional Analysis
START
---------------------------------------------5/30/00 03:20 pm
TESTER
---------------------------------------------Jonathan Bach
TASK BREAKDOWN
---------------------------------------------
#DURATION
short
#TEST DESIGN AND EXECUTION
65
#BUG INVESTIGATION AND REPORTING
25
#SESSION SETUP
20
eBay Inc. confidential
25. Time Box: focused effort, fixed
duration
Short: 60 minutes (+Normal: 90 minutes (+15)
Long: 120 minutes (+15)
15)
– Brief enough:
• for accurate reporting
• to allow flexible scheduling
• to allow course correction
– Long enough:
• to get solid testing done
• for efficient debriefings
eBay Inc. confidential
26. The deliverable: a session report
• Charter
– #AREAS
Metrics
–
–
–
–
–
CHARTER
---------------------------------------------Analyze MapMaker’s View menu functionality and
report on areas of potential risk.
#AREAS
OS | Windows 2000
Menu | et-jsb-010417-c.sesView
Strategy | Function Testing
Strategy | Functional Analysis
Sample
START
---------------------------------------------5/30/00 03:20 pm
TESTER
---------------------------------------------Jonathan Bach
TASK BREAKDOWN
---------------------------------------------
#DURATION
short
#TEST DESIGN AND EXECUTION
65
#BUG INVESTIGATION AND REPORTING
25
#SESSION SETUP
20
#DURATION
#TEST DESIGN AND EXECUTION
#SESSION SETUP
#BUG INVESTIGATION AND REPORTING
#CHARTER / OPPORTUNITY
eBay Inc. confidential
• Notes
Bugs
–
#BUG
Issues
–
#ISSUE
27. Debriefing: a talk with the lead
The tester answers questions
Session metrics are checked
Charter may be adjusted
Session may be extended
New sessions may be chartered
eBay Inc. confidential
29. Are you accountable?
How did you spend your time?
What did you find?
Did you need some help / tools?
Do you think there’s more to do
here?
Was this mission
reasonable?
Agenda: “PROOF”
Past
Results
Obstacles
Outlook
Feelings
eBay Inc. confidential
30. The real message
What’s being asked
What they may be thinking
What was your mission?
Remind me what I told you to do.
Why did you do that?
How did it go?
Were you careful or reckless?
What should I be worried about?
How far did you get?
Are we closer to shipping?
Can you help me know our status?
Need anything?
Can I speed this along?
Do you need more of my time?
When will you be done?
eBay Inc. confidential
Will I get my bonus?
I have a new task for you…
31. eBay sample
CHARTER
----------------------------------------------Analyze MapMaker’s View menu functionality and report on areas of
potential risk.
#AREAS
L
INK
Menu | et-jsb-010417-c.sesView
Strategy | Function Testing
Strategy | Functional Analysis
START
----------------------------------------------5/30/00 03:20 pm
TESTER
----------------------------------------------Jonathan Bach
TASK BREAKDOWN
----------------------------------------------
#DURATION
short
#TEST DESIGN AND EXECUTION
65
#BUG INVESTIGATION AND REPORTING
25
#SESSION SETUP
20"C:Documents and SettingsjobachDesktopLive
Siteisrat_sessionsSession_Listing_Promo.txt"
eBay Inc. confidential
33. When to use SBTM
Getting support: “We have thousands of test cases and a huge regression
suite, but some of our best bugs come from ad hoc testing. Why is that?”
Improving skill: “How she finds those great bugs without test cases, I’ll never
know. I guess some people are just natural explorers you either have it or
you don’t and I just don’t have her knack for it.”
Earning trust and autonomy: “The triage team needs more information
about these 3 bugs, can you give me that by the end of the day?”
eBay Inc. confidential
34. Are you accountable?
How did you spend your time?
What did you find?
Did you need some help / tools?
Do you think there’s more to do
here?
Was this mission
reasonable?
eBay Inc. confidential
37. Sources of charters
• Documentation / Specifications
• Web forums
• Previous products
• Team members
• Competing products
• Your expertise
• Help files
• Manuals
eBay Inc. confidential
41. Charters
• General charters may be necessary at first:
• “Analyze the Insert Picture function”
• Specific charters provide better focus, but take
more effort to design:
• “Test clip art insertion. Focus on stress and flow
techniques, and make sure to insert into a variety of
documents. We’re concerned about resource leaks or
anything else that might degrade performance over time.”
eBay Inc. confidential
42. Some sample session charters
• Installation: When installed, does Triangle! put any files in the wrong places? Does it leave any files
for the uninstall? Check the registry keys, use InCtrl to see what changes are made. Installation is new,
so we want to be sure it’s clean.
• Boundary testing: We got word from customer support that there are runtime errors when using
integers over 32000 but no one can repro it. Best recon is on Win XP Pro with Office 2003 running in
the background. Sam K. in CSS says you can use his machine, and he also has customer specs.
• Ship drill: Start Triangle! right out of the box. For example, is the readme ready to go? We’re waiting
from word on Legal as to the License Agreement, but that shouldn’t hold you up. Also make sure you hit
Vista and see what issues arise there.
• Claims testing: Triangle is meant for first graders, but we plan to ship a version to General Dynamics in
a few months. Try some usability profiles or personas to see what functions become more or less risky.
Also, discover the algorithm by which Triangle! reports its results. Is it way off from what a user would
expect. Does it cause the user to lower their confidence?
eBay Inc. confidential
43. Open-Book Charters for eBay
• What item for sale is nearest to you location?
Submitted the most and sold the least?
What item appears most with no bids?
What are the most recent items posted for any given category?
Which category has the most items?
Which search strings create null queries?
What are 3 examples of broad queries vs 3 examples of ambiguous queries?
What queries create invalid results?
How do you find "sold" items?
What is the most common item?
What is the most bizarre item?
What is the most expensive item?
Is it possible to restrict results to one category without the categories field?
Which seller has listed the most items which have not sold?
What search string takes the longest to return results?
Is there a search that works differently on different browsers?
What was the highest bid for an item in support of any charity?
Find an item that the seller has no business selling!
Find an item you find only on eBay, not Amazon.
What item could you find that had the most bids?
How many eBay stores are there?
What is the oldest object (not item) for sale on ebay?
Are results different if you are logged in?
Try a search on ebay.com and then try the same on on your mobile. What are the 3 major differences?
What item has the greatest difference between the highest and second highest bids?
What user has bid the most but won the least?
What item has been on eBay the longest? Is there a way to tell the age of the item ID?
Who has sold the most items?
Is there a way to find the most commonly misspelled item?
What item can be found in the most categories?
eBay Inc. confidential
44. The session report
• Charter
– #AREAS
Metrics
–
–
–
–
–
CHARTER
---------------------------------------------Analyze MapMaker’s View menu functionality and
report on areas of potential risk.
#AREAS
OS | Windows 2000
Menu | et-jsb-010417-c.sesView
Strategy | Function Testing
Strategy | Functional Analysis
Sample
START
---------------------------------------------5/30/00 03:20 pm
TESTER
---------------------------------------------Jonathan Bach
TASK BREAKDOWN
---------------------------------------------
#DURATION
short
#TEST DESIGN AND EXECUTION
65
#BUG INVESTIGATION AND REPORTING
25
#SESSION SETUP
20
#DURATION
#TEST DESIGN AND EXECUTION
#SESSION SETUP
#BUG INVESTIGATION AND REPORTING
#CHARTER / OPPORTUNITY
eBay Inc. confidential
• Notes
Bugs
–
#BUG
Issues
–
#ISSUE
45. Notes from a recent conference
eBay Inc. confidential
46. Styles I’ve seen
1) Novel Narrative
2) Flight Data Recorder
3) ESPN Highlight
eBay Inc. confidential
48. What to write while exploring
Historical Explorer
Tester
•
(To the degree you think they are
relevant to stakeholders)
drawings of flora / fauna
• descriptions of indigenous
people
• landmarks
Conjectures
•
(Inferences based on experiences.
After I test, I think I know
something)
•
Observations
•
•
•
•
Project
information
•
(Independent of observer)
•
•
•
•
eBay Inc. confidential
•
•
what is this thing?
where should we go today?
how do we get there?
new orders from HQ?
are those people hostile?
•
mission
supplies and staff
latitude / longitude
death and disease
supply status
•
feature model
text from log files
text from dialogs
test ideas
• questions
• product and
project issues
• concerns
• risks
•
•
•
•
charter
test actions
config info
build details
tools used
49. Scrutiny: what happened here?
“(Orderly Book) Camp Mouth of the Kansies June 29, 1804
Ordered A Court Martiall will Set this day at 11 oClock... for the trial of John
Collins and Hugh Hall...
John Collins Charged "with getting drunk on his post this Morning out of whiskey
put under his charge as a Sentinal, and for suffering Hugh Hall to draw whiskey out
of the Said Barrel intended for the party."
...The commanding Officers approve of the Sentence of the Court and orders that
Punishment take place at half past three this evening, at which time the party will
Parrade for inspection.”
June 29, 1804
William Clark
eBay Inc. confidential
51. How much detail is enough?
1) Sufficient benefits
2) No critical problems
3) Benefits outweigh problems
4) All things being equal, further documentation (or time, effort, testing,
editing, etc.) is more harmful than helpful
All 4 conditions must be met to know when you’re done,
(From brother James’ Rapid Software Testing course)
But ask: “good enough for who, what and when?”
eBay Inc. confidential
53. The session report
• Charter
– #AREAS
Metrics
–
–
–
–
–
CHARTER
---------------------------------------------Analyze MapMaker’s View menu functionality and
report on areas of potential risk.
#AREAS
OS | Windows 2000
Menu | et-jsb-010417-c.sesView
Strategy | Function Testing
Strategy | Functional Analysis
Sample
START
---------------------------------------------5/30/00 03:20 pm
TESTER
---------------------------------------------Jonathan Bach
TASK BREAKDOWN
---------------------------------------------
#DURATION
short
#TEST DESIGN AND EXECUTION
65
#BUG INVESTIGATION AND REPORTING
25
#SESSION SETUP
20
#DURATION
#TEST DESIGN AND EXECUTION
#SESSION SETUP
#BUG INVESTIGATION AND REPORTING
#CHARTER / OPPORTUNITY
eBay Inc. confidential
• Notes
Bugs
–
#BUG
Issues
–
#ISSUE
54. What to measure
Bug Investigation
Test Design and Execution
Session Setup
eBay Inc. confidential
(and Reporting)
55. Reporting the metrics
• Test, Bug, and Setup are orthogonal categories, but all we’re only
tracking interruptions to testing
• Nearest 5% or 10% is good enough
• If activities are done simultaneously, report the highest
precedence activity: T, B, then S
eBay Inc. confidential
59. Diagnosing productivity
T est
28%
• Do these proportions make sense?
• How do they change over time?
Bug
4%
• Is the reporting protocol being
followed?
No n S e s s io n
61%
S e tu p
6%
O p p o rtu n ity
1%
3 0 0 .0
2 5 0 .0
2 0 0 .0
1 5 0 .0
1 0 0 .0
5 0 .0
0eBay Inc. confidential
.0
5 /2 6
6 /9
6 /2 3
7 /7
7 /2 1
8 /4
8 /1 8
60. Estimating a Test Cycle
1. How many perfect sessions (100% on-charter
testing) does it take to do a cycle? (let’s say
40)
2. How many sessions can the team (of 4 testers)
do per day? (let’s say 3 per day, per tester =
12)
3. How productive are the sessions? (let’s say
66% is on-charter test design and execution)
4. Estimate: 40 / (12 * .66) = 5 days
5. We base the estimate on the data we’ve
collected. When any conditions or assumptions
eBay Inc. confidential
61. Testing is like looking for
words…
???
(10 minutes)
eBay Inc. confidential
62. Debriefing
Mission A: What happened during the
session?
Mission B: How many words did you find?
Mission C: What obstacles did you face?
Mission D: Do you think you found all the words?
All groups: How did you feel about your
mission?
Agenda: “PROOF”
Past
Results
Obstacles
Outlook
Feelings
eBay Inc. confidential
67. More info
• ET Dynamics:
http://www.satisfice.com/articles/etdynamics.pdf
• Test Heuristics and Planning
http://www.satisfice.com
• ContextDriven Software Testing
http://groups.yahoo.com/group/softwaretesting
• Center for Software Testing Education and Research
http://www.testingeducation.org/BBST
• Books related to Exploratory Testing skills and tactics
http://www.testingreflections.com/node/view/3190
• Scenario testing examples
http://www.testingeducation.org/a/scenario2.pdf
eBay Inc. confidential