SlideShare a Scribd company logo
Maja Schreiner
OnlineTestConf, June 2017
Senior Test Master @ Swisscom,
Switzerland
maja.schreiner@gmail.com
testmotion.wordpress.com
Twitter: majaschreiner
Crowdsourcing is “Taking a service traditionally
performed by a person or team and sourcing it to a large
online group of people as an open project”.
A unique way of involving crowd i.e. the real users/testers,
into software testing under real world conditions.
* Definition by Jeff Howe, Wired journalist & author of the book Crowdsourcing
 user-centric software -> success determined by users feedback.
 testers experience and interact with the app without preconceptions, like their
customers do, they don’t suffer from the confirmation biases that an in-house team
would.
 end users do not live in testing labs, but in the whole world.
 Test coverage for
 mobile iOS and Android devices
 operating systems
 internet browsers.
 manual functional testing, cross-browser testing, and iOS and Android testing.
 scaling the testing resources in a very short time period (testers test concurrently)
 CT typically saves about 30% - 40 % of your budget
 4 Scrum development teams
 test lead + in the meantime 3 junior testers employed temporarily
 CT company: 2 test leads, lots of testers
 Process:
 Preparation
 Trigger the test cycle
 Analyze the results
 Approve
 Reject
 Reward the testers (star the best ones)
inherited the CT contract + team
time frame / deadline problems
Not possible to scale the testing resources while having
only 1 internal resource (tester / test lead)
Developers liked the idea of having lots of testers
Another big project in our company already used the
same crowdsourced testing company
Problems:
Developers rejecting lots of bugs
Developers generally rejecting the value of CT
Distrusting CT
Communication taking lots of time
Lots of administrative and coordination time needed
Impacted everybody in the team
Myself feeling a bit stressed and frustrated
 Decide where to start
 Track improvements
 Inspect and adapt
 Never stop explaining and learning
 The result
 Improved process
 Communication with the testers and their leads
 Build trust
 Add value
 Input
 Test scope / products
 Test plan
 Internal release notes
 Sprint review video demo & presentation
 Roadmap
 All other needed information
 Output
-> test results including featuring screenshots, video recordings, and crash reports
-> test report (dashboard: results summary per Risk, Devices,
Browsers, technical or functional etc.)
HOW TO SUCCEED
 Know your testing goal & purpose
 What
 How
 When do you want to test (every day, middle of sprint, end of sprint)
 Integrate developers in the test cycle and even in the weekly sync calls
 Foster the communication with crowdsourced testers
 Star and reward the good testers
 Keep the best testers in every cycle
 Invite new testers – fresh point of view
 Smoke test your products before giving them further to CT
FURTHER TIPS
 Don’t expect CT to solve all your testing, QA or software development
problems
 Don’t forget to test internally as well
 Beware the costs
Pros for crowdsourced testing
Specialized skill sets
Scale capacity to meet variable demand
Increases fluidity of workforce
Keeps fixed cost off balance sheet
Good for functional end-user testing of mobile and web
apps
Contra for crowdsourced testing
Not so applicable for the applications which require
extensive business know-how
Why and how
Strategy
Problems
How to succeed
What to keep in mind
Please contact me for any further questions:
maja.schreiner@gmail.com
testmotion.wordpress.com
Twitter: majaschreiner

More Related Content

More from PractiTest

Oren rubin statistical element locator
Oren rubin   statistical element locatorOren rubin   statistical element locator
Oren rubin statistical element locator
PractiTest
 
Testing fundamentals in a changing world
Testing fundamentals in a changing worldTesting fundamentals in a changing world
Testing fundamentals in a changing world
PractiTest
 
Automation is not the answer... unless you WANT it to be
Automation is not the answer... unless you WANT it to beAutomation is not the answer... unless you WANT it to be
Automation is not the answer... unless you WANT it to be
PractiTest
 
Testing Web Apps and API's
Testing Web Apps and API'sTesting Web Apps and API's
Testing Web Apps and API's
PractiTest
 
Test beyond the obvious- Root Cause Analysis
Test beyond the obvious- Root Cause AnalysisTest beyond the obvious- Root Cause Analysis
Test beyond the obvious- Root Cause Analysis
PractiTest
 
Testing Metrics and why Managers like them
Testing Metrics and why Managers like themTesting Metrics and why Managers like them
Testing Metrics and why Managers like them
PractiTest
 
The Risk Questionnaire - by: Adam Knight
  The Risk Questionnaire - by: Adam Knight  The Risk Questionnaire - by: Adam Knight
The Risk Questionnaire - by: Adam Knight
PractiTest
 
S.M.A.R.T & F.O.C.U.S Testing - Increasing the value provided by your testing...
S.M.A.R.T & F.O.C.U.S Testing - Increasing the value provided by your testing...S.M.A.R.T & F.O.C.U.S Testing - Increasing the value provided by your testing...
S.M.A.R.T & F.O.C.U.S Testing - Increasing the value provided by your testing...
PractiTest
 
Rob lambert10 Behaviors of Effective Employees" at OnlineTestConf.
Rob lambert10 Behaviors of Effective Employees" at OnlineTestConf.Rob lambert10 Behaviors of Effective Employees" at OnlineTestConf.
Rob lambert10 Behaviors of Effective Employees" at OnlineTestConf.
PractiTest
 
10+ Testing Pitfalls and How to Avoid them
10+ Testing Pitfalls and How to Avoid them 10+ Testing Pitfalls and How to Avoid them
10+ Testing Pitfalls and How to Avoid them
PractiTest
 
Communication skills for testers
Communication skills for testersCommunication skills for testers
Communication skills for testers
PractiTest
 
Software testing - Risk management
Software testing - Risk managementSoftware testing - Risk management
Software testing - Risk management
PractiTest
 
Managing agile testing
Managing agile testing Managing agile testing
Managing agile testing
PractiTest
 
How to create a 'Master Test Plan'
How to create a 'Master Test Plan'How to create a 'Master Test Plan'
How to create a 'Master Test Plan'
PractiTest
 
Mixing testing types to improve your testing results
Mixing testing types to improve your testing resultsMixing testing types to improve your testing results
Mixing testing types to improve your testing results
PractiTest
 
Developer testing webinar
Developer testing webinar Developer testing webinar
Developer testing webinar
PractiTest
 
Agile testing webinar
Agile testing webinarAgile testing webinar
Agile testing webinar
PractiTest
 
Testing metrics webinar
Testing metrics webinarTesting metrics webinar
Testing metrics webinar
PractiTest
 
Reporting principles for every QA manager
Reporting principles for every QA manager Reporting principles for every QA manager
Reporting principles for every QA manager
PractiTest
 
10 signs you have outgrown
10 signs you have outgrown 10 signs you have outgrown
10 signs you have outgrown PractiTest
 

More from PractiTest (20)

Oren rubin statistical element locator
Oren rubin   statistical element locatorOren rubin   statistical element locator
Oren rubin statistical element locator
 
Testing fundamentals in a changing world
Testing fundamentals in a changing worldTesting fundamentals in a changing world
Testing fundamentals in a changing world
 
Automation is not the answer... unless you WANT it to be
Automation is not the answer... unless you WANT it to beAutomation is not the answer... unless you WANT it to be
Automation is not the answer... unless you WANT it to be
 
Testing Web Apps and API's
Testing Web Apps and API'sTesting Web Apps and API's
Testing Web Apps and API's
 
Test beyond the obvious- Root Cause Analysis
Test beyond the obvious- Root Cause AnalysisTest beyond the obvious- Root Cause Analysis
Test beyond the obvious- Root Cause Analysis
 
Testing Metrics and why Managers like them
Testing Metrics and why Managers like themTesting Metrics and why Managers like them
Testing Metrics and why Managers like them
 
The Risk Questionnaire - by: Adam Knight
  The Risk Questionnaire - by: Adam Knight  The Risk Questionnaire - by: Adam Knight
The Risk Questionnaire - by: Adam Knight
 
S.M.A.R.T & F.O.C.U.S Testing - Increasing the value provided by your testing...
S.M.A.R.T & F.O.C.U.S Testing - Increasing the value provided by your testing...S.M.A.R.T & F.O.C.U.S Testing - Increasing the value provided by your testing...
S.M.A.R.T & F.O.C.U.S Testing - Increasing the value provided by your testing...
 
Rob lambert10 Behaviors of Effective Employees" at OnlineTestConf.
Rob lambert10 Behaviors of Effective Employees" at OnlineTestConf.Rob lambert10 Behaviors of Effective Employees" at OnlineTestConf.
Rob lambert10 Behaviors of Effective Employees" at OnlineTestConf.
 
10+ Testing Pitfalls and How to Avoid them
10+ Testing Pitfalls and How to Avoid them 10+ Testing Pitfalls and How to Avoid them
10+ Testing Pitfalls and How to Avoid them
 
Communication skills for testers
Communication skills for testersCommunication skills for testers
Communication skills for testers
 
Software testing - Risk management
Software testing - Risk managementSoftware testing - Risk management
Software testing - Risk management
 
Managing agile testing
Managing agile testing Managing agile testing
Managing agile testing
 
How to create a 'Master Test Plan'
How to create a 'Master Test Plan'How to create a 'Master Test Plan'
How to create a 'Master Test Plan'
 
Mixing testing types to improve your testing results
Mixing testing types to improve your testing resultsMixing testing types to improve your testing results
Mixing testing types to improve your testing results
 
Developer testing webinar
Developer testing webinar Developer testing webinar
Developer testing webinar
 
Agile testing webinar
Agile testing webinarAgile testing webinar
Agile testing webinar
 
Testing metrics webinar
Testing metrics webinarTesting metrics webinar
Testing metrics webinar
 
Reporting principles for every QA manager
Reporting principles for every QA manager Reporting principles for every QA manager
Reporting principles for every QA manager
 
10 signs you have outgrown
10 signs you have outgrown 10 signs you have outgrown
10 signs you have outgrown
 

Recently uploaded

Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Pitangent Analytics & Technology Solutions Pvt. Ltd
 
"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota
Fwdays
 
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptxThe Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
operationspcvita
 
Day 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio FundamentalsDay 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio Fundamentals
UiPathCommunity
 
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptxPRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
christinelarrosa
 
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid ResearchHarnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Neo4j
 
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectorsConnector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
DianaGray10
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Hiroshi SHIBATA
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
Antonios Katsarakis
 
Christine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptxChristine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptx
christinelarrosa
 
A Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's ArchitectureA Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's Architecture
ScyllaDB
 
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users",  Kevin Goedecke"Scaling RAG Applications to serve millions of users",  Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
Fwdays
 
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and StandardsLeveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
Neo4j
 
What is an RPA CoE? Session 1 – CoE Vision
What is an RPA CoE?  Session 1 – CoE VisionWhat is an RPA CoE?  Session 1 – CoE Vision
What is an RPA CoE? Session 1 – CoE Vision
DianaGray10
 
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and BioinformaticiansBiomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Neo4j
 
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSFAppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
Ajin Abraham
 
What is an RPA CoE? Session 2 – CoE Roles
What is an RPA CoE?  Session 2 – CoE RolesWhat is an RPA CoE?  Session 2 – CoE Roles
What is an RPA CoE? Session 2 – CoE Roles
DianaGray10
 
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-EfficiencyFreshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
ScyllaDB
 
Y-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PPY-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PP
c5vrf27qcz
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
Miro Wengner
 

Recently uploaded (20)

Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
 
"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota
 
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptxThe Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
 
Day 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio FundamentalsDay 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio Fundamentals
 
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptxPRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
 
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid ResearchHarnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
 
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectorsConnector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
 
Christine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptxChristine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptx
 
A Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's ArchitectureA Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's Architecture
 
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users",  Kevin Goedecke"Scaling RAG Applications to serve millions of users",  Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
 
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and StandardsLeveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
 
What is an RPA CoE? Session 1 – CoE Vision
What is an RPA CoE?  Session 1 – CoE VisionWhat is an RPA CoE?  Session 1 – CoE Vision
What is an RPA CoE? Session 1 – CoE Vision
 
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and BioinformaticiansBiomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
 
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSFAppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
 
What is an RPA CoE? Session 2 – CoE Roles
What is an RPA CoE?  Session 2 – CoE RolesWhat is an RPA CoE?  Session 2 – CoE Roles
What is an RPA CoE? Session 2 – CoE Roles
 
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-EfficiencyFreshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
 
Y-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PPY-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PP
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
 

Crowdsourced Testing Magic

  • 2. Senior Test Master @ Swisscom, Switzerland maja.schreiner@gmail.com testmotion.wordpress.com Twitter: majaschreiner
  • 3. Crowdsourcing is “Taking a service traditionally performed by a person or team and sourcing it to a large online group of people as an open project”. A unique way of involving crowd i.e. the real users/testers, into software testing under real world conditions. * Definition by Jeff Howe, Wired journalist & author of the book Crowdsourcing
  • 4.  user-centric software -> success determined by users feedback.  testers experience and interact with the app without preconceptions, like their customers do, they don’t suffer from the confirmation biases that an in-house team would.  end users do not live in testing labs, but in the whole world.  Test coverage for  mobile iOS and Android devices  operating systems  internet browsers.  manual functional testing, cross-browser testing, and iOS and Android testing.  scaling the testing resources in a very short time period (testers test concurrently)  CT typically saves about 30% - 40 % of your budget
  • 5.
  • 6.  4 Scrum development teams  test lead + in the meantime 3 junior testers employed temporarily  CT company: 2 test leads, lots of testers  Process:  Preparation  Trigger the test cycle  Analyze the results  Approve  Reject  Reward the testers (star the best ones)
  • 7. inherited the CT contract + team time frame / deadline problems Not possible to scale the testing resources while having only 1 internal resource (tester / test lead) Developers liked the idea of having lots of testers Another big project in our company already used the same crowdsourced testing company
  • 8. Problems: Developers rejecting lots of bugs Developers generally rejecting the value of CT Distrusting CT Communication taking lots of time Lots of administrative and coordination time needed Impacted everybody in the team Myself feeling a bit stressed and frustrated
  • 9.  Decide where to start  Track improvements  Inspect and adapt  Never stop explaining and learning  The result  Improved process  Communication with the testers and their leads  Build trust  Add value
  • 10.  Input  Test scope / products  Test plan  Internal release notes  Sprint review video demo & presentation  Roadmap  All other needed information  Output -> test results including featuring screenshots, video recordings, and crash reports -> test report (dashboard: results summary per Risk, Devices, Browsers, technical or functional etc.)
  • 11. HOW TO SUCCEED  Know your testing goal & purpose  What  How  When do you want to test (every day, middle of sprint, end of sprint)  Integrate developers in the test cycle and even in the weekly sync calls  Foster the communication with crowdsourced testers  Star and reward the good testers  Keep the best testers in every cycle  Invite new testers – fresh point of view  Smoke test your products before giving them further to CT
  • 12. FURTHER TIPS  Don’t expect CT to solve all your testing, QA or software development problems  Don’t forget to test internally as well  Beware the costs
  • 13. Pros for crowdsourced testing Specialized skill sets Scale capacity to meet variable demand Increases fluidity of workforce Keeps fixed cost off balance sheet Good for functional end-user testing of mobile and web apps Contra for crowdsourced testing Not so applicable for the applications which require extensive business know-how
  • 14. Why and how Strategy Problems How to succeed What to keep in mind
  • 15. Please contact me for any further questions: maja.schreiner@gmail.com testmotion.wordpress.com Twitter: majaschreiner