Stakeholders always want to release when they think we’ve finished testing. They believe we have discovered “all of the important problems” and “verified all of the fixes”—and now it’s time to reap the rewards. However, as testers we still can assist in improving software by learning about problems after code has rolled live—especially if it’s a website. Jon Bach explores why and how at eBay they have a post-ship site quality mindset in which testers continue to learn from live A/B testing, operational issues, customer sentiment analysis, discussion forums, and customer call patterns—just to name a few. Jon explains what eBay’s Live Site Quality team learns every day about what they just released to production. Take away new ideas on what you can do to test and improve value—even after you’ve shipped.

1. T6
Special Topics
5/2/2013 9:45:00 AM
Testing - After You’ve Finished Testing
Presented by:
Jon Bach
eBay, Inc.
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Jon Bach
Jon Bach works for eBay (San Jose) as a QA director for the Buyer Experience team. Formerly a manager
of corporate intellect and senior test consultant at Quardev, Jon has been in testing since 1995 with
experience that includes managing teams at Microsoft, HP, and LexisNexis. The co-inventor (with his
brother James) of session-based test management, Jon is an award-winning speaker on test
management and exploratory testing. He’s mostly known for his “half-baked†ideas about how to
inspire innovation and create a learning culture for test teams. Find Jon on Facebook and Twitter
@jbtestpilot. View his presentations at quardev.com/articles and his blog at jonbox.wordpress.com.

3. Testing… (After You’ve
Finished Testing)
Jon Bach
QE Director, Live Site Quality
jobach@ebay.com
STAREast 2013
What’s the quality right NOW?
1

4. 2

5. Is it good enough?
Too Much Quality
Further time and effort
is a waste of resources.
floating line
Good enough quality bar
Further time and effort
is crucial.
Unacceptable Quality
3

6. What I do
7
10 Testing-related activities after release
•
•
•
•
•
•
•
•
•
•
Monitoring
Experimentation
Surveys
Scenario
Follow-up
Investigation
Competitive
Events
Reputation
Dogfooding
8
4

7. Monitoring (Checking)
9
Experimentation (A/B testing)
A
B
10
5

8. Surveys
When site visitors click this
11
Surveys
we see this
We see this
12
6

9. Scenario
13
Follow-up
I go on eBay Radio (a weekly podcast for customers) as “The Bug
Hunter” to elicit stories about technical problems in Production.
------------------------------------------------------------------------------“I have attached the screen shot for you. Try as I might, I can't get
the selling reminders to go to the top. The arrows don't work, but I
can move sold, unsold and active selling. I even set the default to
summary, but no luck. FYI- I am using Safari on my trusty Mac.
Also, the default font size is too big. When I zoom it out, it's too
small. It was a better size the way it was before. I sent a message
to the team through the 'what do you think' for that page.”
14
7

10. Investigative Testing
This is a dip in bids we saw in Production.
What questions do you have to troubleshoot this?
15
My brother’s ideas
16
8

11. Topics for investigation (FORCOSTTTIP)
•
•
•
•
•
•
•
•
•
•
•
Feature – Do we know what happens when people bid?
Origin -- When did this start happening?
Recent – Did we just push code to Production?
Consistency / Correlation -- What happened yesterday,
tomorrow? Are there other graphs and metrics?
Overlap – Is there any other process happening?
Security – Is it malicious? Intentional?
Terminology -- What kinds of “bids”? US, UK, International?
Third-Party / Silent Partner – What’s not in the room? What sort
of information have we NOT heard from?
Truth is out there – does someone already know about this?
Instrumentation – is the graph wrong? Can I get more data?
Prediction – can we test a conjecture with a prediction
17
Competitive Testing
18
9

12. Capability
A large seller on eBay wants to have a “Flash Sale”.
Are we ready?
19
Reputation
20
10

13. Dogfooding
Video of a flow test:
21
Ask my brother to test
Video of a flow test:
22
11

14. Most Important Message
JON BACH
I AM LIVE SITE QUALITY
(AND SO CAN YOU!)
23
Back at your desk on Monday
•
•
•
•
•
•
•
•
•
•
•
Run Experimentation???
Create monitoring???
What to investigate
Make surveys???
How to make scenarios
What’s our reputation???
Follow-up on what???
Dogfood a medical device?
Capabilities? What???
Competitive what now?
Usability, but no lab?
Try crowdsourced charters
Take an Ops guy to lunch
Customer Support’s Most Wanted
Interview a Business Analyst
Try a flow exercise with legos
Scan user groups
Talk to an early adopter
Watch a live study or trial
See a marketing roadmap
Explore a competitor URL
Try Open-Book Testing
24
12