1. |
Crowdsourced Testing
Idea to implementation, complete with stories
John Montgomery | uTest VP of Product Delivery | johnm@utest.com | September, 2013

2. |
Agenda
• Intro
• What is Crowdsourcing?
• Why do we need it for testing?
• Which Crowdsourced testing approach is
for you?
• Myths
• Our Approach
• Q&A
2

3. The Challenge
|| 3
uTest
Who we are, what we do

4. | 4
Who We Are
uTest is the world’s largest provider of
in-the-wild testing services

5. | 5
Types of Testing Services
 Scalable, on-demand access to professional testers with the geographies and
real-world infrastructure used by your customers and end users
 On-demand test subjects and usability experts to provide comprehensive UX
and user interaction testing services across most demographics and configs
 Live and synthetic load and performance testing and analysis
 White hat security experts providing automated and manual penetration
testing from anywhere in the world
 Real-world localization validation using native speakers from over 190
countries

6. | 6
Since Our 2008 Launch
1,500+
customers
100,000+ testers from 200 countries
& territories
Enterprises, SMBs & universities
Functional, load,
usability, L10N &
security
Web, mobile
& desktop apps
30,000+
test cycles

7. The Challenge
|| 7
Crowdsourcing
A Primer

8. |
Crowdsourcing Defined
“Taking a service traditionally performed by
a person or team and sourcing it to a large
online group of people as an open project”
- Jeff Howe
Wired journalist & author of Crowdsourcing
8

9. |
Who Offers Crowdsourcing?
9

10. |
Myriad Potential Benefits:
• Specialized skill sets
• Scale capacity to meet variable demand
• Increases fluidity of workforce
• Keeps fixed cost off balance sheet
What’s The Upside?
And Why Wouldn’t They?

11. The Challenge
|| 11
In-the-Wild
Why Crowdsource Testing

12. |
Users Do Not Live In Labs
Your Test Lab Is Here Your Users Are Here
Testing inside the lab is vital. But your customers and their
challenges do not necessarily live there.
Does this look familiar?

13. |
Test Matrix Are More Complex
• This is Android devices only
• Doesn’t account for OS versions, carriers or locations

14. |
A Lot More Complex
14
• Android alone (and that’s only by OS version)

15. |
Users Are More Vocal
• Users less tolerant of
spotty quality
• Social & app stores
give every user a
megaphone
• Switching costs
vanishing to zero
• Cost of poor app
quality has spiked

16. |
• Thousands of companies have moved a portion of testing
closer to where users work, live and play. Why?
1. Apps consumed under endless range of environments
- Exponential spike in devices, OSes & browsers
- Imperfect connectivity
- Infinite number of locations
2. Users have less patience than ever for buggy apps
- Switching costs lower
- App quality issues carry higher costs
3. In-the-lab testing alone cannot meet the demands of modern apps
Crowdsource Testing Can Help

17. |
Crowdsourcing Is In-the-Wild
18
In-House
Manual
Testing
Outsourced
Manual
Testing
In-House
Test
Automation
Outsourced
Test
Automation
In-The-Lab
Testing

18. The Challenge
|| 19
Crowdsourced Testing
Which is best for you?

19. |
How Do You Get From This
20
Crowds often look (and act) like unruly mobs

20. |
To This
21
Delivering a skilled service predictably requires
a community capable of producing desired results

21. | 22
Choose Your Weapon
uTest Lives Here

22. |
Know Thy Task
• Automation
• Performance
• Usability
• Management
• Architecture
• Functional
• Localization
• Security
• Combination
of Group &
Specialist
• User panels
• Beta testing
• Data validation
• Ad quality

23. |
• What are your competitors doing to differentiate?
• What is your risk profile for quality?
• How do you compare against your competition?
Know Thy Competition

24. |
• Is your culture highly cautious & risk-averse?
• Are you in a highly regulated industry?
– Defense industry
– Pharma
– Banking
• Do you have an appetite for innovation?
• Do you have the budget to do it correctly?
Know Thyself

25. |
Which One Is Best?
26
Self managed White glove
Skills may vary
Any participant
Results may vary Exact results
Exact skills
Vetted
participant

26. The Challenge
|| 27
Lessons Learned
Myths, Misconceptions

27. |
Less In-House Staff = Less Overhead
TRUTH:
• Complement & scale “as needed”…not replace
• Only employees will fully understand strategy,
company position, internal processes
BEST PRACTICES:
• Build strengths around employees’ core competencies
• Where does the crowd ‘fit’ into your team?
– Fresh eyes, fresh perspectives, and fresh ideas
– Rote or repetitive tasks
– Outside of the lab

28. |
TRUTH:
• Intellectual property (IP) is a legit issue, esp. for the new &
innovative
– Copywriting
– Design
– Animation
– Code writing
• IP policies vary within crowdsourcing orgs
BEST PRACTICES:
• Establish the ground rules: What’s their IP policy?
– How is it communicated to the crowd?
– How is it enforced?
But we need to own the results

29. |
Tell the crowd? But we’re in stealth
mode!
TRUTH:
• Confidentiality is a serious matter with material consequence.
• Crowdsourcing is often used for pre-launch products
BEST PRACTICES:
• Ask upfront about preventative measures
– Precautions… NDAs? What else?
– Is the policy all-inclusive? e.g. incl. social media, message boards,
etc.
– What are the consequences for breaches?

30. |
Selection criteria are vital:
• Referenceable customer successes
– By company size
– By industry
• Ability to adapt to your legacy systems and processes
• Ability to satisfy legal requirements
– IP protection
– NDA
• Ability to do the job
– Consistent
– Predictable
– Professional
All Vendors Are Equal

31. The Challenge
|| 32
uTest & the Crowd:
Testing In-the-Wild

32. |
Focus On Your Needs
1. Community profiling
– Technical: OS, browser, mobile devices, carriers
– Geographic: City, country, languages
– Demographic: Age, gender, education, hobbies
2. Community ratings & micro-ratings
– By testing type
– By industry
3. Precise matching
– Between each project & each tester
4. Services layer for clients
– Onboarding
– Maximize throughput
– Minimize overhead

33. |
Not Restricted By Location
Diverse
Community
Trained
Community
Dedicated
Offsite
Dedicated
Onsite
34
• Expertise On Demand
– Who you need
– When you want
– Where you need

34. | 35
How It Works
Customers
uTest pays testers
Specify testing needs
- OS, Browser, Carrier, Device
- Location, Language & Industry
Customer approves
testing results
Selected testers
invited to project
Testers report
in real-time
Testing Community

35. |
• Top c-sourcing firms use two forms of compensation
– Monetary
– Reputation
• Performance-based ratings based upon a dozen factors:
– Participation Level:
- Lifetime & recent participation: # active test cycles, # reported issues
– Quality of Participation
- Approval percentage for bugs, test cases and usability surveys
- Accuracy of bug type & severity classifications
- Ability to write test cases & create automated test scripts
36
Rating & Reputation System

36. | 37
• Better Applications
– In-the-wild testing – live testers, real devices, real-world conditions
– Unmatched coverage across large matrix of OS/browser/location
– Testers rated by customers
• Faster Time to Market
– Test cycles created in minutes; underway in hours; completed in days
– Test cycle can be executed overnight or during weekends
• Lower Total Cost of Testing
– Subscription based and project based contracts
– Lower total cost of testing than near-shore boutiques or offshore shops
• Scalability
– 100K + professional testers from around the world
– Scale size of test team up or down at will
How Our Customers Benefit

37. The Challenge
|| 38
Questions?
Answers