Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Crowd Sourced Testing - Real-World Experience Using Rainforest QA Crowd-Sourced Testing Platform


Published on

Crowd sourced application testing is a hot new way of testing! It is estimated that by 2018 20% of testing will be done using crowd sourcing.

Instead of keeping testers in-house crowd sourcing gives you access to an unlimited number of testers, at any time and any place. It also allows you to test using a previously unattainable number of combinations of devices and software.

Marc Fasel is the Technical Director of the Melbourne-based SaaS company PerformanceCentre and will present to us on how they leveraged Crowd Sourced Testing using the Rainforest QA platform within his organisation. Using real world case study examples Marc will explain how after a failed attempt to automate their regression test suite, they successfully managed to reduce the two week regression testing effort to less than a single day! Marc is amazed by the reliability and results being produced by their Crowd Sourced Testing, not to mention the great bugs being found.

Published in: Software
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website!
    Are you sure you want to  Yes  No
    Your message goes here

Crowd Sourced Testing - Real-World Experience Using Rainforest QA Crowd-Sourced Testing Platform

  1. 1. Solution Proposal 1 Crowd Sourced Testing March 22, 2017 Marc Fasel Technical Director Real-World Experience Using Rainforest QA Crowd-Sourced Testing Platform
  2. 2. Crowd SourcedTesting PerformanceCentre 2 •PerformanceCentre is a SaaS platform •We calculate commissions for sales people •Clients: •Companies with large sales teams: 50-5000 •Commissions cannot be accurately calculated using Excel •Example Suncorp, Optus, Bankwest … •Headquarters: South Melbourne, team of 17 •Cater to the Australian- and US- market •I am the technical director, lead an agile team of 4 developers, 1 tester
  3. 3. Crowd SourcedTesting Problem: Our Regression is not Agile 3 •Agile development: 2 week sprints, deploy after each sprint •Lots of functionality built over the last 10 years •Web application with > 300 pages •Regression test •Before every deployment •Written Regression test: 300 test cases, average10 test steps •Execution time: 2 person weeks •Not agile!
  4. 4. Crowd SourcedTesting Test Automation 4 •High initial development effort; estimate 2 hours per test; Every time we added a new feature there was a new problem to overcome with Selenium •High maintenance effort; many tests broken with each run •Tests were flaky (timeouts etc) •We could hire a single person only doing Selenium •The tests never found anything (or it was drowned out by the false-positives)
  5. 5. Crowd SourcedTesting The Search 5 •Add local test engineer •Management Overhead •Expensive •Regression test time still too long: 1 week •Outsourcing •Recruit tester in the Philippines •Testing is not on-demand: Gotta keep person busy 40hrs/wk •Management overhead •Regression test time still too long: 1 week
  6. 6. Crowd SourcedTesting The Solution: Crowd-Sourced Testing 6 •Emerging trend in testing •Gartner report “Market Guide for Crowdsourced ApplicationTesting Services” Oct 2016: •By 2018 20% of testing will be done using crowdsourcing •Testing-as-a-Service •Instead of using in-house test engineers for QA the testing is outsourced to a company •Crowd: Large number of testers, parallel execution of tests •Typically use case: web-based or mobile testing •Web-based SaaS platform to manage the testing •Testers can be qualified or unqualified •Either virtual machines or use their own devices
  7. 7. Crowd SourcedTesting Solution: Crowd Testing 7 •We ended up using Rainforest QA SaaS platform •Written tests are entered into platform •Run now button: select tests you want to run •Tests are executed by human testers •Testers get a virtual machine spun up with the correct OS and browser •Tests run in parallel •Time to run all tests is determined by the time to run the longest test •On-demand •Regression test time:1 hour
  8. 8. Crowd SourcedTesting PlatformView 8
  9. 9. Crowd SourcedTesting Test Details 9 •Typical test consists of 5-10 steps •Each step is an instruction followed by aYes/No question •Any qualified tester can take on the test •Tester getsVM with correct OS and browser, and the step description shown. •Tester executes the step and answersYes/No question.
  10. 10. Crowd SourcedTesting Individual Test 10
  11. 11. Crowd SourcedTesting Test Step 11
  12. 12. Crowd SourcedTesting Results 12
  13. 13. Crowd SourcedTesting Individual Failure Result 13
  14. 14. Crowd SourcedTesting Where do Testers come from? 14 •In our case they come from HIT market places •Example:Amazon MechanicalTurk •Human IntelligenceTasks (HIT): Micro-jobs that cannot be automated •Marketplace: requestors and workers •Platform is requestor, workers are the testers •Each test becomes a HIT •Workers have to earn qualifications and have reputation to be allowed to take test HITs •Earn qualifications through training •Earn reputation through successful HITs
  15. 15. Crowd SourcedTesting Amazon Mechanical Turk 15
  16. 16. Crowd SourcedTesting HIT Example 16
  17. 17. Crowd SourcedTesting Cost 17 •Cost is per test step: US $0.66-$0.77 depending on pre-paid number of credits •Example: US $0.70/ test step •200Tests * 10 Steps * 2 browsers * US $0.70 = US $2800 per regression test •2 regression tests per month = US $5600 •Cheaper than an localTest Engineer, more expensive than offshoreTester
  18. 18. Crowd SourcedTesting Advantages 18 •Test results in an hour: super agile! •Maintenance is a breeze: •1 hour to add new tests per sprint •5 tests broken per cycle, and it takes <30mins to fix •0 management overhead •Easy testing against different browsers, mobile devices •Test crowd works 24/7 •Issues easy to reproduce •They actually found real bugs!
  19. 19. Crowd SourcedTesting Disadvantages 19 •Cost •Monthly cost US$4000 for 2 regression tests @1 hr •You could have 200 failed tests: cost US$2000 •Tester Skill •Testers do not know anything about your application •The testers are not particularly skilled in testing •They know how to use a web app •The tests have to be simple to execute •Testers sign NDA: how do you enforce that? •Tests run in parallel •The tester look only for the answer of each test step
  20. 20. Crowd SourcedTesting Platform Choice 20 •Popular vendor Options •Rainforest QA ( •Applause •Testbirds •Lionbridge •Mob4Hire •Space is still very young, and vendors have different approaches •Exploratory testing vs. functional/ regression testing •Testers from HIT marketplaces vs. fixed test teams •Some vendors even provide test automation for you
  21. 21. Crowd SourcedTesting Conclusion 21 •CrowdTesting has been a great success for us •Best of both worlds of test automation and manual testing •Quick results •No management overhead •Easy scalability to new browsers and devices •Well-maintained test suite •Few false-positives •Main drawback: Cost •Agile QA •Release the day after the sprint ends •Regression-tested hot fixes in 4 hours •Retain Small team •Very young and dynamic space, lots of changes still happening
  22. 22. Crowd SourcedTesting Contact Us 22 Measure Sales - Reward Success 11Yarra Street, South MelbourneVIC +61 3 8685 1200 info@performancecentre.c www.performancecentre.c