Your SlideShare is downloading. ×
Keynote: Selling (and Buying) “Live Site Quality” at eBay
Keynote: Selling (and Buying) “Live Site Quality” at eBay
Keynote: Selling (and Buying) “Live Site Quality” at eBay
Keynote: Selling (and Buying) “Live Site Quality” at eBay
Keynote: Selling (and Buying) “Live Site Quality” at eBay
Keynote: Selling (and Buying) “Live Site Quality” at eBay
Keynote: Selling (and Buying) “Live Site Quality” at eBay
Keynote: Selling (and Buying) “Live Site Quality” at eBay
Keynote: Selling (and Buying) “Live Site Quality” at eBay
Keynote: Selling (and Buying) “Live Site Quality” at eBay
Keynote: Selling (and Buying) “Live Site Quality” at eBay
Keynote: Selling (and Buying) “Live Site Quality” at eBay
Keynote: Selling (and Buying) “Live Site Quality” at eBay
Keynote: Selling (and Buying) “Live Site Quality” at eBay
Keynote: Selling (and Buying) “Live Site Quality” at eBay
Keynote: Selling (and Buying) “Live Site Quality” at eBay
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Keynote: Selling (and Buying) “Live Site Quality” at eBay

234

Published on

In the February Fortune magazine, eBay made the cover with the title “eBay is Back!” The article cited improvements in the look and feel of the site, strategic investments in fulfillment, and …

In the February Fortune magazine, eBay made the cover with the title “eBay is Back!” The article cited improvements in the look and feel of the site, strategic investments in fulfillment, and technology partnerships with retailers to establish it as more than just an online auction service. Jon Bach joined just as eBay was making big bets to make notable and visible gains with this strategy. Jon recounts his two and a half years as a quality engineering director and introduces a concept he calls Live Site Quality. It means the value your customers get while experiencing different activity flows through your online product or service. It’s the impression they’re left with as they try to solve a problem or meet a need—whether it’s a simple task or a complex exploration of your site’s capabilities. Jon’s main idea is selling stakeholders on different Live Site Quality perspectives and getting eBay colleagues from different parts of the organization to buy into the idea that bugs that might most affect their Live Site Quality may lie between their team and another.

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
234
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
7
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. K4 Keynote 10/3/2013 8:30:00 AM "Selling (and Buying) "Live Site Quality" at eBay" Presented by: Jon Bach eBay Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
  • 2. Jon Bach eBay, Inc. With more than eighteen years of experience in software testing, Jon Bach has held technical and managerial positions in companies including Hewlett-Packard and Microsoft. In his current role as director of Live Site Quality for eBay, Jon is dedicated to building “end-to-end” tests (activity flows) in eBay’s core sites to discover important bugs that threaten its core business. He is most notable for creating, with his brother James, Session-Based Test Management, a method to manage and report exploratory testing.
  • 3. 9/23/2013 Selling (and Buying) Live Site Quality) at Jon Bach QE Director, Live Site Quality jobach@ebay.com STAR West 2013 February issue 1
  • 4. 9/23/2013 What’s the quality right NOW? Selling and Buying Live Site Quality Is it good enough? Too Much Quality Further time and effort is a waste of resources. floating line Good enough quality bar Further time and effort is crucial. Unacceptable Quality Selling and Buying Live Site Quality 2
  • 5. 9/23/2013 Heuristics of quality Bad Buyer Experiences Gross Merchandise Volume (+) Net Promoter Score Earnings Per Share Social Media chatter Morning Mission Report eWatch’s Most Wanted Re-rolls and Rollbacks Call volume Mean Time to Detect Mean Time to Mitigate Selling and Buying Live Site Quality 5 Heuristics of quality Overdue bug fixes Survey results Tell Us What You Think Metrics: DoD / Wow / MoM / YoY Key Flow Test Results eCommerce Bytes #ebaysux “Be The Customer” Discussion Forum topics Calls to eBay Radio Emails from VOICES Selling and Buying Live Site Quality 6 3
  • 6. 9/23/2013 What I do (“Director of Live Site Quality”) Selling and Buying Live Site Quality 7 “Director of Live Site Quality” Jon the Bug Hunter on eBay Radio Call customers who participate in our NPS callback program Speak at VOICES meet-ups Selling and Buying Live Site Quality 8 4
  • 7. 9/23/2013 “Director of Live Site Quality” Live Site Ledger P1 Bugs found this week (with Root Cause Analysis) Overdue bugs that need fixing What bad things happened in the last 24 hours? Teach New Hires How to Explore Selling and Buying Live Site Quality 9 “Director of Live Site Quality” Make sure eWatch is in the loop on issues Find out who *really* owns this bug Drive issues that threaten the notion of “site readiness” Selling and Buying Live Site Quality 10 5
  • 8. 9/23/2013 “Director of Live Site Quality” Coordinate teams in end-to-end testing Host bug bashes that focus on exploration Manage crowdsourced testing efforts in production Selling and Buying Live Site Quality 11 What WE do … especially after release Monitoring Experimentation Surveys Scenario Follow-up Investigation Competitive Events Reputation Dogfooding Selling and Buying Live Site Quality 12 6
  • 9. 9/23/2013 Monitoring (Checking) Selling and Buying Live Site Quality 13 Experimentation (A/B testing) A B Selling and Buying Live Site Quality 14 7
  • 10. 9/23/2013 Surveys When site visitors click this… Selling and Buying Live Site Quality 15 Selling and Buying Live Site Quality 16 …we see this: 8
  • 11. 9/23/2013 Selling and Buying Live Site Quality 17 Follow-up on “UFO reports” ------------------------------------------------------------------------------“I have attached the screen shot for you. Try as I might, I can't get the selling reminders to go to the top. The arrows don't work, but I can move sold, unsold and active selling. I even set the default to summary, but no luck. FYI- I am using Safari on my trusty Mac. Also, the default font size is too big. When I zoom it out, it's too small. It was a better size the way it was before. I sent a message to the team through the 'what do you think' for that page.” Selling and Buying Live Site Quality 18 9
  • 12. 9/23/2013 Investigative Testing What questions would you have to troubleshoot a dip in metrics like this? Selling and Buying Live Site Quality 19 Topics for investigation (FORCOSTTTIP) •Feature – Do we know what happens when people bid? •Origin -- When did this start happening? •Recent – Did we just push code to Production? •Consistency / Correlation -- What happened yesterday, tomorrow? Are there other graphs and metrics? •Overlap – Is there any other process happening? •Security – Is it malicious? Intentional? •Terminology -- What kinds of “bids”? US, UK, International? •Third-Party / Silent Partner – What’s not in the room? What sort of information have we NOT heard from? •Truth is out there – does someone already know about this? •Instrumentation – is the graph wrong? Can I get more data? •Prediction – can we test a conjecture with a prediction Selling and Buying Live Site Quality 20 10
  • 13. 9/23/2013 Competitive Testing Testing… After You’ve Finished Testing 21 “Site Readiness” A large seller on eBay wants to have a “Flash Sale”. Are we ready? Selling and Buying Live Site Quality 22 11
  • 14. 9/23/2013 Reputation Selling and Buying Live Site Quality 23 Dogfooding Video of a flow test: Selling and Buying Live Site Quality 24 12
  • 15. 9/23/2013 Ask my brother to round up a posse… Rapid Testing Intensive July 2012 • 80 testers from around the world – in-person and online * 4 days of testing eBay Motors: Tire Finder / Wheel Center •How to jump into a product and find bugs while you learn all about it, while working toward "deep coverage" testing. •Charter-based exploration, modeling, usability testing, intense coverage to deliver notions of quality and risk Selling and Buying Live Site Quality 25 What I’ve learned Once they see it, they may want it There is no sale without a buyer A buyer can return an item being sold Selling and Buying Live Site Quality 26 13
  • 16. 9/23/2013 Most Important Message… JON BACH I AM LIVE SITE QUALITY (AND SO CAN YOU!) Selling and Buying Live Site Quality Selling and Buying Live Site Quality 27 Back at your desk on Monday •Run Experimentation??? •Create monitoring??? •What to investigate… •Make surveys??? •How to make scenarios… •What’s our reputation??? •Follow-up on what??? •Dogfood a medical device? •Capabilities? What??? •Competitive what now? •Usability, but no lab? Try crowdsourced charters Take an Ops guy to lunch Customer Support’s Most Wanted Interview a Business Analyst Try a flow exercise with legos Scan user groups Talk to an early adopter Watch a live study or trial See a marketing roadmap Explore a competitor URL Try Open-Book Testing Selling and Buying Live Site Quality 28 14

×