Get Testing Help from the Crowd
Upcoming SlideShare
Loading in...5

Get Testing Help from the Crowd



Crowdsourcing has become widely acknowledged as a productivity solution across numerous industries. However, for companies incorporating crowdsourcing into existing business practices, specific issues ...

Crowdsourcing has become widely acknowledged as a productivity solution across numerous industries. However, for companies incorporating crowdsourcing into existing business practices, specific issues must be addressed: What problem are we trying to solve? How do we control the process? How do we incentivize people to achieve our goals? Ultimately, the key to successfully employing a crowdsourcing model is to move beyond the realm of the “mob” to create an engaged, interactive community of diverse and skilled professionals. In the world of quality assurance, crowdsourcing has the potential to effectively solve emerging challenges and take your testing to new heights. Using real-world examples, Matt Johnston explains how you can leverage the crowd to complement your internal systems, ensure systems work as intended under real-world conditions, and effectively manage the scalability of testing efforts.



Total Views
Views on SlideShare
Embed Views



1 Embed 6 6



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    Get Testing Help from the Crowd Get Testing Help from the Crowd Document Transcript

    • T18 Concurrent Class 10/3/2013 1:30:00 PM "Get Testing Help from the Crowd" Presented by: Matt Johnston uTest Inc. Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ ∙
    • Matt Johnston uTest With more than a decade of marketing experience at companies ranging from early-stage startups to publicly traded enterprises, Matt Johnston leads uTest's marketing and community management efforts. Since joining uTest as employee #8 in late 2008, Matt has provided a technical sensibility to uTest’s engagement of testers and engineers. As CMO, he leads uTest’s strategic initiatives, such as partnership formation and helping to define uTest’s expansion strategy.
    • 9/19/2013 Getting Testing Help From The Crowd Matt Johnston | CMO/CSO, uTest | | @matjohnston October 2013 | The Challenge App Quality Then vs. Now | 1
    • 9/19/2013 The Way Things Were You Tell Them What To Think | The Way Things Are TODAY They Tell You What They Think | 2
    • 9/19/2013 Mobile App Quality Is Judged More Harshly • Users less tolerant of spotty quality • Social & app stores give every user a megaphone • Switching costs lower than ever • Cost of imperfect app quality has spiked | So What Could Possibly Go Wrong? | 3
    • 9/19/2013 Oh, And It’s Not Just Mobile Fragmentation of users, devices, OSes, browsers, locations Cars Household Appliances E-Readers Connected TVs Laptops & Desktops Gaming Consoles Phones & Tablets | No Turning Back Now • Inside-out quality perspective is valuable but incomplete – Regardless of mobile app type or dev & test methodologies • Replaced by outside-in quality worldview – Quality signals come in many forms, and from USERS - Bug reports - Test cases executed - Crashes - User feedback - Analytics – And at many times (continuous) | 4
    • 9/19/2013 What’s That Mean? • External Signals Inform Internal Decisions: – What you think matters less than what users think – How YOU measure app quality matters less – What users feel and say matters MORE – Court of public opinion rules - App stores - Social media | Tools For Outside-In App Quality In-House InApps Test That Leaders Win | 5
    • 9/19/2013 The Challenge Ummmm So What Do We Do? | 11 Stuff Gets Built; Then It Breaks 1. Marketing identifies a need 2. Product specifies a solution 3. Engineering builds the product 4. QA tests the application 5. Help desk listens to complaints 6. Sales & execs blame everyone 7. Then we do it all over again and hope for better results Vicious Cycle | 12 6
    • 9/19/2013 Lab-Based Testing Innovation Manual vs. Automation Managing vs. Leading Teams In-House vs. Outsourced Agile vs. Waterfall Exploratory vs. Test Cases Testers vs. Engineers Offshore vs. Nearshore | 13 Mob Mentality BUT crowds often look (and act) like unruly mobs | 14 7
    • 9/19/2013 From Mob to Community And mobs don’t work in every category Example: a skilled service like software testing or requires an orderly and professional “community” capable of consistently delivering the desired results | 15 The challenge is to turn this – | 16 8
    • 9/19/2013 Into this | 17 The Challenge The Crowd: Testing In-the-Wild | 18 9
    • 9/19/2013 Community Management 101 • What Good CM Isn’t – It’s not customer service – It’s not marketing or social media • What Good CM Is – – – – 1st gen reactive & narrowcast (customer service) nd gen 2 proactive & broadcast (marketing) rd gen 3 independent & self-serve (program management) 4th gen interdependent & vetting, teaching and policing one another (community management) | 19 Community Management Tool Kit • CM goals – to build relationships, develop loyalty and nurture desired behavior – Get beyond 1:1 discussions – Get beyond reactive corrections • CM Challenges – to do the following at scale • Tool Kit – Recruiting – – – – – – – – Vetting Engaging Incenting Managing Forums & message boards > email responses Help topics & tutorials > repeatedly answering same questions Webinars > phone calls Profiles Matching Ratings Reputations Payments | 20 10
    • 9/19/2013 Rating & Reputation System • Top c-sourcing firms use two forms of compensation – Monetary – Reputation • Performance-based ratings based upon a dozen factors: – Participation Level: - Lifetime & recent participation: # active test cycles, # reported issues – Quality of Participation - Approval percentage for bugs, test cases and usability surveys - Accuracy of bug type & severity classifications - Ability to write test cases & create automated test scripts | 21 The Challenge uTest Case Study: USA Today | 11
    • 9/19/2013 At the Forefront of Innovation • Timeline of USA Today Pioneering – 1982: 1st nat’l paper to use full color – 1996: Early adopter of web – 2008: 1st major news brand with native apps - iOS, Android, Windows, Kindle, web TVs, gaming consoles – 2012: Re-designed & re-launched entirely new UI/UX across all - Print + website + 14 native apps - First web exceeded print; now apps exceed web | Impact on Testing • • • Great for USA Today’s readers Great for USA Today’s sales & marketing Vast pressure on tech execs to deliver quality across: – – – – – Device makers & models OSes & browsers Carriers Languages (localized content) Locations (geo-based intelligence) | 12
    • 9/19/2013 USA Today’s Testing Portfolio In-House Manual Testing Outsourced Manual Testing In-The-Lab Testing In-House Outsourced Test Automation Test Automation | The Challenge The Goal: High Bar For App Quality | 13
    • 9/19/2013 The Goals • Maintain Quality – Launch nothing less than a 4.5 star app – Each new version, across 14 native apps • Increase Test Coverage – Real-world conditions – Cost-effective – Scalable • Meet Tight Deadlines – Must have apps ready to win - For each new platform that gains traction - For each new device that gains traction | The Challenge The Approach: Achieve In-the-Wild Testing | 14
    • 9/19/2013 Two Distinct Choices • Traditional Outsourcing – Lives in a lab - Further removed from users - Geo + demo • Crowdsourcing & expertsourcing – Lives beyond the lab, in the wild - Professional testers + real-world conditions – Mirrors USA Today’s user base – Cheaper vs. in-house – Hidden costs - Time zones & language - Scaling concerns and costs - Real-world conditions - Tech + geo + demo – Blended team - Recurring testers (incl on-site) + crowd testers - Dedicated uTest project manager – High noise-to-signal ratio - No transparency – Scales to match shifting needs | Their Choice: Partner with uTest • In-the-wild testing solution for USA Today – Provide on-demand complement to in-house QA team – Reduce escape rate by testing beyond the lab - Across OS, browsers, devices & carriers - Across locations, languages, industries & hobbies – Blended test team built into USA Today’s dev & test lifecycle - Dedicated uTest project manager – Tester management – Triaging services - Team of dedicated testers (remote or on-site) + large team of crowd testers | 15
    • 9/19/2013 The Challenge Results: Apps That Win | Results • Reduced escape rate • Improved usability • Crowd details – ID’d and documented 1,000+ bugs across 6 apps in first 90 days – Several iterations of usability testing & analysis – Testers from US, Canada, Brazil, UK, France, Germany, China & Japan – Some projects (eg: brand new iPad app) involved 80+ testers – Others with tighter scope required smaller teams of 10-12 testers • Hit the ground running – Launching Kindle Fire app same day the device launched – Hand-picked testers pre-purchased various models of device – Tested their app with all over the world, almost instantly • Bottom line: better apps – Enabled USA Today to keep innovating, while maintaining superior quality – USA Today able to consistently meet 4.5 star app-store rating goal – 2012 Mobile Excellence Award – Best User Experience | 16
    • 9/19/2013 Customer Satisfaction The Final Word “uTest gives our organization a much better sense of where we stand in terms of application stability and performance. For a group that iterates as often as we do, that type of insight is critical” -- Tim Carlson, Director of Mobile Product Development, USA Today | The Challenge Crowdsourced Testing Specific Takeaways | 17
    • 9/19/2013 Key Takeaways: Crowd Testing • Apps universe has forever changed testing – Exponentially more diverse user environments – Devices & configurations – Locations – User demographics • In-the-wild testing is a necessity – Some testing must be done beyond the lab – In-the-wild where app users live, work & play • Agility is crucial to app success – Test team size and tester location • enterprise-friendly mobile testing Partner selection is vital – Not all created equal – Alignment of capabilities, needs and philosophy – Track record of relevant enterprise success | The Challenge Crowdsourcing OverarchingTakeaways | 18
    • 9/19/2013 Understand The Job To Be Done | The Challenge Questions? Answers Matt Johnston | CMO @ uTest | @matjohnston | 38 19