Crowd-sourced testing is a relatively new testing approach where an anonymous crowd of testers take on different testing tasks for web-based and mobile applications. We have used this approach to regression test our responsive web application on desktop and mobile devices with great results: it increased our test coverage while drastically reducing the time for regression testing. This talk introduces the concept of crowd-sourced testing, talks about our journey of implementing crowd-sourced testing, and our experience after more than a year of using it.

1. Solution Proposal 1
Crowd Sourced Testing
May 11, 2018
Marc Fasel
Technical Director, Performio
Using Crowd-sourced Testing to Regression-Test Responsive Web Applications

2. Crowd SourcedTesting 2
SetThe Stage

3. Crowd SourcedTesting
Crowd Sourcing
3
•"Practice of engaging a 'crowd' or group for a common goal"
•Modern Concept
•Collaborative Future
•Distributed problem solving
•Micro-tasking
•Not completely new
•Kickstarter Crowdfunding
•Wikipedia
•Open Source software development
•BetaTesting

4. Crowd SourcedTesting
What is Crowd Testing?
4
•Crowd: large number of testers
•Parallel execution of tests
•Testing-as-a-Service
•Typical use case:
•web-based application testing
•mobile web/ application testing

5. Crowd SourcedTesting
Types of Testing available via Crowd-Testing
5
•Functional exploratory
•Device/ bandwidth compatibility, e.g. laptop, desktop, tablet, smartphones etc
•Localisation/ Multi-language support
•RegressionTesting
•PerformanceTesting
•Security/ ethical hacking

6. Crowd SourcedTesting
Crowd-Testing Platform Differences
6
Semi-qualified Specialised QA Engineers
Large crowd Small group
Real Devices Virtual Devices
DIY Managed
International Local
Size
Localisation
Device Type
Service Type
Vetting

7. Crowd SourcedTesting
Where do Testers come from?
7
•Sign up to crowdsourcing platform
•Send in CV to crowdsourcing company
•Qualified QA engineers with tertiary degree
•Managed crowd: testers can be selected based on their skills
•Human IntelligenceTasks (HIT) Marketplace, e.g.Amazon MechanicalTurk
•Unqualified or semi-qualified users
•Earn qualifications through training
•Earn reputation through successful HITs

8. Crowd SourcedTesting
Crowd Testing Advantages
8
•Scalability - talent on demand
•Execution time
•Better coverage/ higher detection rate
•Pay-as-you-go

9. Crowd SourcedTesting
Crowd Testing Disadvantages
9
•No application-specific knowledge
•Less control
•Tester Quality
•Testing approach
•Training
•Intellectual property protection
•Issue duplication

10. Crowd SourcedTesting 10
Real World Experience

11. Crowd SourcedTesting
Our Problem: Lengthy Regression Test
11
•Web application with > 300 pages
•Agile development
•2 week sprints
•deploy after each sprint
•Regression test
•manual execution before every deployment
•200 written test cases, average10 test steps
•execution time: 2 person weeks
•NOT AGILE

12. Crowd SourcedTesting
Solution: Crowd Regression Testing
12
•Written tests are entered into platform
•select tests you want to run, click: run now
•Tests are executed immediately
•Tests run in parallel
• REGRESSION TEST TIME: 1 HOUR

13. Crowd SourcedTesting
Solution: Crowd Testing
13
Semin-qualified Specialised QA Engineers
Large Crowd Small group
Real Devices Virtual Devices
DIY Managed
International Local
Size
Localisation
Device Type
Service Type
Vetting

14. Crowd SourcedTesting
PlatformView
14

15. Crowd SourcedTesting
Individual Test
15

16. Crowd SourcedTesting
Test Step
16

17. Crowd SourcedTesting
Results
17

18. Crowd SourcedTesting
Individual Failure Result
18

19. Crowd SourcedTesting
Responsive Design:Adding Mobile Tests
19
•Application uses responsive design for mobile
•Adding mobile tests
•Copy-paste desktop test cases
•Change screenshots
•Very few tests had to be modified or completely rewritten
•Choice between real and virtual devices (cost factor)

20. Crowd SourcedTesting
Cost
20
•Cost is per test step: e.g. $0.70/ test step
•200Tests * 10 Steps * 2 browsers * US $0.70 = US $2800 per regression test
•2 regression tests per month = US $5600
•Mobile testing on small subset of tests, otherwise costed would have exploded (iPhone, iPad,Android,
tablet, landscape/ portrait)

21. Crowd SourcedTesting
Advantages Of Our Crowd-Sourced Regression Test
21
•Test results in an hour: super agile!
•Results
•7 failures
•5 false positives
•2 true positives
•0 false negatives
•Maintenance is a breeze:
•1 hour to add new tests per sprint
•5 tests broken per cycle, and it takes <30mins to fix
•0 management overhead
•Easy testing against different browsers, mobile devices
•Test crowd works 24/7
•Issues easy to reproduce

22. Crowd SourcedTesting
Disadvantages
22
•Cost
•Monthly cost US$5600 for 2 regression tests @1 hr
•Worst case you could have 200 failed tests: cost US$2800
•Adding a browser, another test cycle, or another device costs a lot
•The tester look only for the answer of each test step

23. Crowd SourcedTesting
Conclusion
23
•CrowdTesting: part of our collaborative future
•Main advantages is scalability, execution time, coverage, detection rates
•Crowd RegressionTesting has been a great success for us
•Quick results
•Easy scalability to new browsers and devices
•Well-maintained test suite
•Few false-positives
•Main drawback: cost
•Agile QA
•Release the day after the sprint ends
•Regression-tested hot fixes in 4 hours
•Retain small team

24. Crowd SourcedTesting
Contact Us
24
Measure Sales - Reward Success
80 Market Stree, South Melbourne
+61 3 8685 1200
info@performio.co
linkedin.com/company/performio
www.performio.co