Is Crowd Testing (relevant) for Software Engineers?

1,413 views

Published on

This presentation was performed at AST 2014 (tech.brookes.ac.uk/AST2014/) the 9th International Workshop on Automation of Software Test (AST’14)
on June 1, 2014, in Hyderabad, India.

It introduces CrowdTesting and provides some initial thoughts on how Crowd Testing and Software Engineering can be combined to get even better results.

Published in: Software
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,413
On SlideShare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
55
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Is Crowd Testing (relevant) for Software Engineers?

  1. 1. Università degli Studi dell’Aquila Henry Muccini DISIM, University of L’Aquila henry.muccini@univaq.it, @muccinihenry, www.henrymuccini.com @AST 2014, Hyderabad, India – June 01, 2014
  2. 2. Researcher at the University of L’Aquila, Italy Research 2
  3. 3. How did I get to Crowd Testing? 3 @AST 2012 @MobileSoft2014
  4. 4. Acknowledgment and Disclaimer This work is conducted with the collaboration of Fabio Guaiani (University of L’Aquila) and the CLARITER company (Rome). This presentation represents my group initial effort towards the understanding of Software Engineering needs and opportunities in Crowd Testing. 4
  5. 5. Agenda 5 Software Testing Trends Crowd Testing Software Engineering for Crowd Testing
  6. 6. PREFACE: SOFTWARE TESTING TRENDS 6 Software Testing Trends Crowd Testing Software Engineering for Crowd Testing
  7. 7. Software Testing Evolution 7 Software Testing No dedicated testing team Dedicated (in-house) testing team Outsourced testing (in-house) resources resources in the cloud Software development
  8. 8. Investment in QA 8 [World Quality Report 2013-2014] [1191 respondents] «Right first time» apps > $50 billion/year on applications testing and quality assurance [Top six trends to drive market for software testing in 2012,” TechJournal, citing Pierre Audoin Consultants, Nov 2011] 18% 23% 28% 0% 5% 10% 15% 20% 25% 30% 1 IT budget allocated to QA 2015 2013 2012
  9. 9. 9 Software Testing No dedicated testing team Dedicated (in-house) testing team Outsourced testing [1500 respondents] 0% 20% 40% 60% 51% 41% Internal/In-house Testing 2013 2012 [1500 respondents] 0% 10% 20% 13% 20% External/Managed Services 2013 2012 0% 20% 40% 8% 26% "Centralized" Testing 2013 2012 [World Quality Report 2013-2014]
  10. 10. 10 Software Testing No dedicated testing team Dedicated (in-house) testing team Outsourced testing [1500 respondents] 0% 20% 40% 60% 51% 41% Internal/In-house Testing 2013 2012 [1500 respondents] 0% 10% 20% 13% 20% Crowd Sourced Testing (External/Managed Services) 2013 2012 0% 20% 40% 8% 26% "Centralized" Testing 2013 2012 [World Quality Report 2013-2014] “IDC and Forrester indicate … an increase in outsourcing of 15% annually over the next several years.” [Forrester]
  11. 11. Testing Mobile App 11 Taken from [World Quality Report 2013-2014], page 27
  12. 12. Testing Mobile App 12 Taken from [World Quality Report 2013-2014], page 27 Must be «right first time» Even «for free» apps Must get into the market «first»
  13. 13. Summing up… CROWD TESTING QA needs Mobile and Web Outsourci ng
  14. 14. CROWD TESTING 14 Software Testing Trends Crowd Testing Software Engineering for Crowd Testing
  15. 15. (the Crowd) [Oxford Dictionaries] “The mass or multitude of ordinary people”
  16. 16. Crowdsourcing [CrowdSource] The word is a combination of the words 'crowd' and 'outsourcing'. Work Work Work Work Work Work Work Work Crowdsourcing is the process of getting work or funding, usually online, from a crowd of people.
  17. 17. Crowd Testing…Glossary … (to be refined) 17 Crowd Crowd Sourcing Outsourcing Crowd Testing
  18. 18. What is Crowd Testing? “ It is, therefore, a form of outsourcing testing tasks to a mass of internet users (the crowd).” [Stein_TestExp14] “If you are testing software that all kinds of strangers are going to use, then why not use a bunch of strangers to test it.” (Paul Herzlich) “ Crowtesting puts software to test under real-world conditions by real users, allowing companies to gather real insight and feedback and to identify defects quickly and cheaply” [CSTE2013] 18 HM5 HM6
  19. 19. Diapositiva 18 HM5 “It’s called the power of the people. I asked thousands of friends to spend 5 minutes testing rather than requiring five of our employees to spend thousands of hours testing. It got us quicker results on more devices than we typically consider.” [Mike Sparks article] Henry Muccini; 30/05/2014 HM6 It consists in asking 1000s of crowd testers to spend 5 minutes testing rather than requiring 5 employees to spend 1000s of hours testing. Henry Muccini; 30/05/2014
  20. 20. Examples 19 Tested by a crowd of 9 million people 2 million valuable comments and insights [Stein_TestExp14] GAP had to scrap their logo redesign after protests on Facebook and Twitter [www.pass.ch]
  21. 21. 20 diversity/coverage/realistic independent and test as a user payment system immediate results Principles
  22. 22. 21 Crowd Testing
  23. 23. The «players» and the «process» 22 Crowd Testers Crowd Testing Company Customers
  24. 24. Who are the crowd testers? [RV_TestExp14] The testing crowd can be a pool of: ▪ Inter-company testers ▪ Intra/inter-project testers in a company ▪ Strategic partners ▪ Domain knowledge experts ▪ Technology experts ▪ Tool experts ▪ Network of end users ▪ Social network users 23 Certified Specialist Recreational
  25. 25. Crowd Testers expertise and infrastructure ⇒⇒⇒⇒ Crowd Testing types 24 Crowd Testing Exploratory Crowd Testing Enterprise Crowd Testing
  26. 26. Exploratory Crowd Testing + Effective to cover all the many configurations + Good for non critical systems + Simulating real load and testing - Outcome not predictable - Discovering basic failures To help making it more effective: Show them the big picture of what they are doing Formal process A clear report 25
  27. 27. Enterprise cloud infrastruct ure Enterprise Crowd Testing 26 Large enterprises utilizing their in-house excess capacity for on-demand crowdsourced testing • Secured private cloud infrastructure • Testing Managers Organized • Crowd testers screening • Project management, tracking, effort planning Professionalized • Standardized Process and ReportingStandardized
  28. 28. Enterprise («Managed») Crowd Testing «Enterprise crowd testing, although still in its nascent and emerging stage, promises to be the next wave in the social, mobility and localization era.» [28+_TestExp14] 27 Crowdtesting Platform Test Scheduling Testers Profiles Knowle dge center Comm. and collaboration Test Managem ent Reporting
  29. 29. A typical Process 28 Preparation Initiation Execution Evaluation Completion Identify customer’s needs Fix price Define report template Select crowdtesters Prepare the test environment Initiate testing Supervise and monitor Validate results Prepare client deliverables Testers payment Presents results to client Crowdtesters’ ratings [TestExp14]
  30. 30. Crowd Testing…Glossary 29 Crowd Crowd Sourcing Outsourcing Crowd Testing Exploratory Crowd T. Managed Crowd T. Beta Testing Enterprise Crowd T.
  31. 31. 30
  32. 32. - 1.5 months of full-time job for one mobile developer novice + 0.5 months for an expert mobile developer (client side) - 13.000 products in the DB - 400 MB of storage into the device - DB update 31
  33. 33. - The «annulla» feature (when the button is visualized during the data download) does not work on IOs - In case the product image is missing, a default image needs to be loaded - The «loading %» screen is not fluid - On IOs, if I download a product with an already existing image, I get an error - In the «product list», when tapping on a product imagine, while looking at the magnified image, I can still scroll the page (shall be disabled). 32
  34. 34. SOFTWARE ENGINEERING FOR CROWD TESTING 33 Software Testing Trends Crowd Testing Software Engineering for Crowd Testing
  35. 35. Benefits 34 Availability Coverage Cost- effectiveness Impartiality Real scenarios Speed Diversity
  36. 36. Most of the reports come from CrowdTesting companies, so there could be a «BIAS»!! 35
  37. 37. Still… a number of big companies working on it [BenchmarkQA] 36 are 55.000 testers better than 1.600 ? What types of testing can be run with the crowd? Big companies!
  38. 38. Data from Utest (now, «Applause») 37 Randy Perry. “Trulia Leverages uTest's "Crowdsourced" Testing Solution to Speed Deployment of High-Quality Mobile Apps.” Sponsored by uTest. January 2013
  39. 39. Goldman Sachs Bets Millions On uTest [http://www.forbes.com/sites/petercohan/2014/01/22/goldman-sachs-bets-millions-on-utest/] 38
  40. 40. Challenges in Crowd Testing 39 Security breaches Duplicate defect management Measurement Coverage critical features coverage No control Lack of ownership Reward mechanism Communication and mgmt. Lack of standards Continuous involvement
  41. 41. Opportunities for SE research in crowd testing: COMMUNITIES Crowdsourcing in SE Global Software Engineering Social Software Engineering Empirical studies Software Testing 40
  42. 42. 1. Opportunities for SE research Effectiveness of different testing methods «When»shall we test with the crowd, when not to? 41 Crowd Testing Cloud Testing Lab Testing Which sw. domain Which type of tests Ongoing Work
  43. 43. 2. Opportunities for SE research Complementing different testing methods 42 Crowd Testing Cloud Testing Lab Testing How «crowd and lab testing» can be used «together»? Ongoing Work Need: clients’ want to get the best from the combination of both Efficiency
  44. 44. 2. Complementing different testing methods (cont.) 43 Systematic Mobile Testing Crowd Testing Solution to ?
  45. 45. 2. Complementing different testing methods (cont.) 44 Systematic Structural Testing Crowd Testing Solution to ? Coverage Issues Test Selection Test Execution
  46. 46. 2. Complementing different testing methods (cont.) 45 Model- based Mobml Testing Crowd Testing Model- based Mobile
  47. 47. Principle 46 Device independent tests (in LAB) + device dependent in crowd Test HTML 5 PhoneGap (in LAB) + test Phone Gap app in Android, iOS, Microsoft Test MobML mobile (in LAB) + test Phone Gap app in Android, iOS, Microsoft Impl. 1 Impl. With MBT 2. Complementing different testing methods (cont.)
  48. 48. 3. Opportunities for SE research Engineering the CrowdTesting Platform 47 Crowdtesting Platform Test Scheduling Testers Profiles Knowle dge center Comm. and collaboration Test Managem ent Reporting
  49. 49. 4. Opportunities for SE research «Crowd» Coverage Criteria 48 Measurement Coverage critical features coverage No control System Coverage Crowd Coverage … Metrics
  50. 50. 5. Opportunities for SE research Social Software Engineering How to keep Crowd Testers motivated How to keep them involved Appropriate reward mechanisms 49
  51. 51. In summary 50 Crowd Testing is growing exponentially Crowd Testing may bring big benefits to QA Still, there a number of challenges to be managed
  52. 52. If you are interested to this topic, please 51 Stop by after the presentation Contact me at henry.muccini@univaq.it Tweet @muccinihenry Skype me at henry.muccini Call me ☺☺☺☺ H. Muccini “Is Crowd Testing (relevant) for Software Engineers?” @AST20114
  53. 53. References «Testing Experience, 25/2014» issue on Crowd testing [TestExp14] «Testing Experience, 25/2014» issue on Crowd testing [RV_TestExp14] Rama Murari & Venkat Ramesh Atigadda, “Panorama of Crowd Testing”, in Testing Experience, 25/2014 [Stein_TestExp14] Markus Steinhauser “Crowd Testing – An Addition to Traditional Software Testing”, in Testing Experience, 25/2014 [34+_TestExp14] Dieter Speidel, Mayank Mittal & Mithun Sridharan “Enterprise Crowd Testing”, in Testing Experience, 25/2014, page 34+ [28+_TestExp14] Saisubramanian Sivasailem & Dilipkarthik Jayamohan, “The Next Wave in the Social, Mobile and Localization Era - Enterprise Crowd Testing”, in Testing Experience, 25/2014, page 28+ 52
  54. 54. References [BenchmarkQA] http://www.benchmarkqa.com/ [CSTE2013] Crowdsourced Software Testing for Enterprises, August 2013 (by passbrains.com) [World Quality Report 2013-2014] By Capgemini, Sogeti, Hp, 5th edition Crowdsourced Software Testing for Enterprises, August 2013 (passbrain) Dagstuhl Seminar 13362, Cloud-based Software Crowdsourcing, Edited by Michael N. Huhns, Wei Li, and Wei-Tek Tsai 53

×