Your SlideShare is downloading. ×
Testing in the Wild: Practices for Testing Beyond the Lab
Testing in the Wild: Practices for Testing Beyond the Lab
Testing in the Wild: Practices for Testing Beyond the Lab
Testing in the Wild: Practices for Testing Beyond the Lab
Testing in the Wild: Practices for Testing Beyond the Lab
Testing in the Wild: Practices for Testing Beyond the Lab
Testing in the Wild: Practices for Testing Beyond the Lab
Testing in the Wild: Practices for Testing Beyond the Lab
Testing in the Wild: Practices for Testing Beyond the Lab
Testing in the Wild: Practices for Testing Beyond the Lab
Testing in the Wild: Practices for Testing Beyond the Lab
Testing in the Wild: Practices for Testing Beyond the Lab
Testing in the Wild: Practices for Testing Beyond the Lab
Testing in the Wild: Practices for Testing Beyond the Lab
Testing in the Wild: Practices for Testing Beyond the Lab
Testing in the Wild: Practices for Testing Beyond the Lab
Testing in the Wild: Practices for Testing Beyond the Lab
Testing in the Wild: Practices for Testing Beyond the Lab
Testing in the Wild: Practices for Testing Beyond the Lab
Testing in the Wild: Practices for Testing Beyond the Lab
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Testing in the Wild: Practices for Testing Beyond the Lab

178

Published on

The stakes in the mobile app marketplace are very high, with thousands of apps vying for the limited space on users’ mobile devices. Organizations must ensure that their apps work as intended from day …

The stakes in the mobile app marketplace are very high, with thousands of apps vying for the limited space on users’ mobile devices. Organizations must ensure that their apps work as intended from day one and to do that must implement a successful mobile testing strategy leveraging in-the-wild testing. Matt Johnston describes how to create and implement a tailored in-the-wild testing strategy to boost app success and improve user experience. Matt provides strategies, tips, and real-world examples and advice on topics ranging from fragmentation issues, to the different problems inherent in web and mobile apps, to deciding what devices you must test vs. those you should test. After hearing real-world examples of how testing in the wild affects app quality, leave with an understanding of and actionable information about how to launch apps that perform as intended in the hands of end-users—from day one.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
178
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. W14 Test Techniques 5/7/2014 3:00:00 PM Testing in the Wild: Practices for Testing Beyond the Lab Presented by: Matt Johnston Applause Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
  • 2. Matt Johnston Applause With more than a decade of experience at companies ranging from early-stage startups to publicly-traded enterprises, Matt Johnston leads the uTest and Applause organizational strategy efforts. Since joining uTest as employee #8 in late 2008, Matt has provided a technical sensibility to uTest’s engagement of testers and engineers. As CMO/CSO, he leads uTest’s large-scale initiatives, such as partnership formation and helping to define the company’s expansion strategy. Matt is a frequent speaker at top technology and mobile conferences, including STARWEST, CTIA, and Mobile App World. A renowned expert on crowdsourcing, Matt’s work has appeared in the bestseller Crowdsourcing for Dummies.
  • 3. 4/26/2014 1 | Testing In the Wild: Practices for Testing Beyond the Lab Matt Johnston | CMO/CSO, uTest | matt@utest.com | @matjohnston April, 2014 The Challenge || App Quality Then vs. Now
  • 4. 4/26/2014 2 | The Way Things Were You Tell Them What To Think | The Way Things Are TODAY They Tell You What They Think
  • 5. 4/26/2014 3 | Mobile App Quality Is Judged More Harshly • Users less tolerant of spotty quality • Social & app stores give every user a megaphone • Switching costs lower than ever • Cost of imperfect app quality has spiked | So What Could Possibly Go Wrong?
  • 6. 4/26/2014 4 | Cars E-ReadersHousehold Appliances Fragmentation of users, devices, OSes, browsers, locations Oh, And It’s Not Just Mobile Phones & Tablets Laptops & Desktops Connected TVs Gaming Consoles | • Inside-out quality perspective is valuable but incomplete – Regardless of mobile app type or dev & test methodologies • Replaced by outside-in quality worldview – Quality signals come in many forms, and from USERS - Bug reports - Test cases executed - Crashes - User feedback - Analytics – And at many times (continuous) No Turning Back Now
  • 7. 4/26/2014 5 | • External Signals Inform Internal Decisions: – What you think matters less than what users think – How YOU measure app quality matters less – What users feel and say matters MORE – Court of public opinion rules - App stores - Social media What’s That Mean? The Challenge || 10 Ummmm So What Do We Do?
  • 8. 4/26/2014 6 | Stuff Gets Built; Then It Breaks 1. Marketing identifies a need 2. Product specifies a solution 3. Engineering builds the product 4. QA tests the application 5. Help desk listens to complaints 6. Sales & execs blame everyone 7. Then we do it all over again and hope for better results 11 Vicious Cycle | 12 Lab-Based Testing Innovation Agile vs. Waterfall Manual vs. Automation Managing vs. Leading Teams Exploratory vs. Test Cases Testers vs. Engineers In-House vs. Outsourced Offshore vs. Nearshore
  • 9. 4/26/2014 7 | Mob Mentality BUT crowds often look (and act) like unruly mobs 13 | From Mob to Community And mobs don’t work in every category Example: a skilled service like software testing or requires an orderly and professional “community” capable of consistently delivering the desired results 14
  • 10. 4/26/2014 8 | The challenge is to turn this – 15 | Into this 16
  • 11. 4/26/2014 9 The Challenge || 17 The Crowd: Testing In-the-Wild | Community Management 101 • What Good CM Isn’t – It’s not customer service – It’s not marketing or social media • What Good CM Is – 1st gen reactive & narrowcast (customer service) – 2nd gen proactive & broadcast (marketing) – 3rd gen independent & self-serve (program management) – 4th gen interdependent & vetting, teaching and policing one another (community management) 18
  • 12. 4/26/2014 10 | • CM goals – to build relationships, develop loyalty and nurture desired behavior – Get beyond 1:1 discussions – Get beyond reactive corrections • CM Challenges – to do the following at scale – Recruiting Vetting Engaging Incenting Managing • Tool Kit – Forums & message boards > email responses – Help topics & tutorials > repeatedly answering same questions – Webinars > phone calls – Profiles – Matching – Ratings – Reputations – Payments 19 Community Management Tool Kit | • Top c-sourcing firms use two forms of compensation – Monetary – Reputation • Performance-based ratings based upon a dozen factors: – Participation Level: - Lifetime & recent participation: # active test cycles, # reported issues – Quality of Participation - Approval percentage for bugs, test cases and usability surveys - Accuracy of bug type & severity classifications - Ability to write test cases & create automated test scripts 20 Rating & Reputation System
  • 13. 4/26/2014 11 The Challenge || uTest Case Study: USA Today | • Timeline of USA Today Pioneering – 1982: 1st nat’l paper to use full color – 1996: Early adopter of web – 2008: 1st major news brand with native apps - iOS, Android, Windows, Kindle, web TVs, gaming consoles – 2012: Re-designed & re-launched entirely new UI/UX across all - Print + website + 14 native apps - First web exceeded print; now apps exceed web At the Forefront of Innovation
  • 14. 4/26/2014 12 | • Great for USA Today’s readers • Great for USA Today’s sales & marketing • Vast pressure on tech execs to deliver quality across: – Device makers & models – OSes & browsers – Carriers – Languages (localized content) – Locations (geo-based intelligence) Impact on Testing | USA Today’s Testing Portfolio In-House Manual Testing Outsourced Manual Testing In-House Test Automation Outsourced Test Automation In-The-Lab Testing
  • 15. 4/26/2014 13 The Challenge || The Goal: High Bar For App Quality | • Maintain Quality – Launch nothing less than a 4.5 star app – Each new version, across 14 native apps • Increase Test Coverage – Real-world conditions – Cost-effective – Scalable • Meet Tight Deadlines – Must have apps ready to win - For each new platform that gains traction - For each new device that gains traction The Goals
  • 16. 4/26/2014 14 The Challenge || The Approach: Achieve In-the-Wild Testing | • Traditional Outsourcing – Lives in a lab - Further removed from users - Geo + demo – Cheaper vs. in-house – Hidden costs - Time zones & language - Scaling concerns and costs – High noise-to-signal ratio - No transparency Two Distinct Choices • Crowdsourcing & expertsourcing – Lives beyond the lab, in the wild - Professional testers + real-world conditions – Mirrors USA Today’s user base - Real-world conditions - Tech + geo + demo – Blended team - Recurring testers (incl on-site) + crowd testers - Dedicated uTest project manager – Scales to match shifting needs
  • 17. 4/26/2014 15 | • In-the-wild testing solution for USA Today – Provide on-demand complement to in-house QA team – Reduce escape rate by testing beyond the lab - Across OS, browsers, devices & carriers - Across locations, languages, industries & hobbies – Blended test team built into USA Today’s dev & test lifecycle - Dedicated uTest project manager – Tester management – Triaging services - Team of dedicated testers (remote or on-site) + large team of crowd testers Their Choice: Partner with uTest The Challenge || Results: Apps That Win
  • 18. 4/26/2014 16 | • Reduced escape rate – ID’d and documented 1,000+ bugs across 6 apps in first 90 days • Improved usability – Several iterations of usability testing & analysis • Crowd details – Testers from US, Canada, Brazil, UK, France, Germany, China & Japan – Some projects (eg: brand new iPad app) involved 80+ testers – Others with tighter scope required smaller teams of 10-12 testers • Hit the ground running – Launching Kindle Fire app same day the device launched – Hand-picked testers pre-purchased various models of device – Tested their app with all over the world, almost instantly • Bottom line: better apps – Enabled USA Today to keep innovating, while maintaining superior quality – USA Today able to consistently meet 4.5 star app-store rating goal Results | Customer Satisfaction The Final Word “uTest gives our organization a much better sense of where we stand in terms of application stability and performance. For a group that iterates as often as we do, that type of insight is critical” -- Tim Carlson, Director of Mobile Product Development, USA Today
  • 19. 4/26/2014 17 The Challenge || In-the-Wild Testing Specific Takeaways | Key Takeaways: In-the-Wild Testing • Apps universe has forever changed testing – Exponentially more diverse user environments – Devices & configurations – Locations – User demographics • In-the-wild testing is a necessity – Some testing must be done beyond the lab – In-the-wild where app users live, work & play • Agility is crucial to app success – Test team size and tester location enterprise-friendly mobile testing • Partner selection is vital – Not all created equal – Alignment of capabilities, needs and philosophy – Track record of relevant enterprise success
  • 20. 4/26/2014 18 The Challenge || 35 Questions? Answers Matt Johnston | CMO @ uTest mattj@utest.com | @matjohnston

×