Find out how crowd testing works. Learn more about the benefits of crowd testing.

1. How Crowd Testing Works
February, 2016

2. Overview
Crowd testing is a form of testing where in, a crowd sourced platform is used to connect testers from
around the world to testing projects. When it comes to user-centric apps like games and e-commerce
apps, crowd testing is the best solution. That’s why corporations like Google & Microsoft rely on crowd
testing to test their apps.
Crowd testing is apt for user-centric apps that is used by a diverse crowd. It is particularly useful in
scenarios where you want to test the application on multiple devices, with many testers from different
locations.As it is more fast and cost effective, it is a boon for clients who want to meet their release
deadlines faster & also for those who do not have the QA resources to execute testing. It is also helpful in
case of testing apps that are in the early stages of development. The fresh insights given by crowd testers
can help development teams identify and fix defects in time before releasing the app to the public.
Highlights Of Crowd Testing Platform:
Run longer and more through testing without straining internal resources.
Access a flexible, on-demand workforce that can handle peak demands.
Transparent output based pricing that allows more predictability.
Enables testing in various real world conditions.
Option to select crowd testers for project, based on your target demographics.
Faster results and time to market.
More accountability and performance.
Lab Testing Vs Real World Testing
In-house testing involves the use of emulators and in-house devices for creating a test environment. It is
not possible to procure all the devices & create the fluctuating conditions that exist in the real world inside
a lab. Further, many factors out in the open can create unusual scenarios. For example, internet
bandwidth impacts app quality. Based on how good or bad the internet connection is, application
behaviour will be different for different users. Also, you also need to ensure that the updates you release
work well in different OS versions. While ensuring the releases are fast, you have to make sure that they
are easy to upgrade as well.
Critical issues that arise in real world scenarios can be found and fixed with crowd testing. By relying on
crowd testers, you can test your application in various real world scenarios with different combinations of

3. devices, operating systems and software versions. While an internal QA check may help out in weeding
some of the defects found during a lab-based testing, they can never reproduce the harsh and varying
conditions that your app will face out in the open.Unlike a lab, the real world is far from user friendly.
Analyzing how real users react to the app is crucial for identifying and fixing bugs that can affect app
quality.
Benefits Of Crowd Testing
1) Scalability
Whether you are time pressed for the release of a product or need to upgrade or improve a product
feature within a set deadline to beat the competition, crowd testing can give you the flexibility you need to
scale your testing efforts with ease. You can access crowd testers on demand for your project and
increase the capability of your development team to build your products faster.
2) Speed
One of the biggest advantages of crowd testing is the flexibility you get for running multiple test cycles at
the same time. This will not only help you to get testing insights faster but also help in reducing the time
to market.
3) Cost
The best part about crowd testing is that you don’t have to invest in full time QA hires or money on
acquiring devices for your testing project. You get a comprehensive test coverage and access to many
testers from different locations without a major financial investment.
4) Output based Pricing
Crowd testing is an extremely result driven form of testing where you are charged according to the bugs
found by crowd testers. The pay for performance model not only drives the crowd testers to deliver their
best, but also enables you to get the best value for money unlike other forms of testing which follows a
rigid head count form of pricing.
5) Device Coverage
With crowd testing, you can leave behind the hassles of configuring emulators and the high costs of
investing in device farms. You can get coverage across multiple devices and browsers easily without
major investments.

4. 6) Geographic Coverage
When you rely on a crowd sourced platform, your QA team is truly global. You can access testers from
anywhere in the world. This is especially useful in case of localization testing and other forms of targeted
testing based on specific target demographics.
7) Testing in The Real World
Ever wondered, how real users will react to your app? Crowd testing enables you to target your testing
efforts with testers who match the interests & demographics of your target group. This will help you in
gaining insights to improve app features to delight your users. Crowd testing is thus ideal for usability,
functional & exploratory testing.
8) Access Testing Experts
One of the biggest constraints of a traditional QA approach is the limited access to testing talent. By
tapping into a global pool of talented testers you can access testing experts in the domain you require
easily. Expert support means that your testing efforts can be more effective and simpler.
9) Get A Fresh Perspective
It is possible to unearth new issues and gain fresh app insights with crowd testing as crowd testers would
be more impartial & would not have any intrinsic bias like an internal QA team. When you follow new
permutations and combinations, the chances of finding issues and bottlenecks are higher.
10) Align testing efforts with development process
Crowd testing gives you the ability to get complete coverage in terms of devices and locations with the
flexibility to scale your testing efforts, making it easier for you to plan and align your testing strategy with
the requirement of your development team. You can give your development team the timely inputs they
need to address critical issues in the application.
The Crowd testing community
The strength of a crowd testing platform lies in its diverse testing community. A crowd sourced testing
community consists of many testers from all over the world, with diverse backgrounds, domain
knowledge and expertise. This provides clients the opportunity to test their application in diverse test
environments.

5. Apart from allowing you to see how your app works in real life conditions, crowd testing platforms enable
you to scale your QA efforts as per needs.
How crowd testing works
1) Requirement Analysis
Every app is unique. The testing requirement for an app depends on the business goals of the company.
The first step in crowd testing is to analyze the requirements of the client and the testing project. A
testing plan in line with the product’s quality goals is prepared. This testing plan serves as the basis for
fixing the price, assigning the crowd testers and finalizing other aspects like number of test cycles etc.
Usually a dedicated project manager will be assigned by the crowd testing company for creating and
managing the test strategy. By understanding the app, its environment and its quality goals, the plan and
deliverables are set by the project manager.
2) Project Launch
Once the project is initiated, crowd testers who match the requirement in terms of demographics and
domain knowledge are assigned for testing. The test environment covering the configurations mentioned
is set up.
3) Project Execution
Crowd testers execute the project by testing the app on different devices from different locations. The
crowd sourced testing experts will help uncover all the issues in your app.
4) Monitoring & Evaluation
Clients can monitor the bug logging and testing coverage. The bugs logged by the crowd testers is
evaluated either by crowd validators or designated experts from the crowd testing company.
5) Project Completion
All the bug reports, test summary and recommendations are shared with the client once the test cycles
are executed. Many platforms have added bug tracking features that makes it easy for development
teams to not only find the bugs but also to fix them. The crowd testers assigned to the project are also

6. rated as per the quality of bugs they log. This ensures that the top testers who log the most difficult bugs
are paid the most, creating a reward based system that encourages testers to perform well.
While crowd testing can never replace an internal QA team, it definitely supplements the efforts of an
In-house QA team. A testing strategy that encompasses the inputs of the In-house QA & the real world
insights of crowd testing gives product managers a bird’s eye view of their software, making it easy to
access & fix risks in time.Crowd testing lets you test your app on real devices by real users, giving you
accurate insights on how well your app works. It can add value in addition to traditional testing methods
to give you greater coverage and a fresh insight on your app quality.
About 99tests
99tests is a crowd testing platform to get your mobile & web apps tested by top notch testers. We provide companies with
economical, and effective crowd-sourced testing and automation services.​ You can integrate the 99tests crowd sourced
automation testing platform with your existing resources easily without much investment. We provide a full-service solution for
automation testing for mobile and web apps. You will get all the resources you need to map out test scenarios, execute the test
runs and monitor the test runs. You can empower existing QA teams and developers with the productivity tools and analytics they
need to take your software to the next level.​Learn how automation through 99tests gives the insights you need for building
stunning software products. Mail us at contact@99tests.com.
www.99tests.com