Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Training Workers for Improving Performance in Crowdsourcing Microtasks

557 views

Published on

With the advent and growing use of crowdsourcing labor markets for a variety of applications, optimizing the quality of results produced is of prime importance. The quality of the results produced is typically a function of the performance of crowd workers. In this paper, we investigate the notion of treating crowd workers as ‘learners’ in a novel learning environment. This learning context is characterized by a short-lived learning phase and immediate application of learned concepts. We draw motivation from the desire of crowd workers to perform well in order to maintain a good reputation, while attaining monetary rewards successfully. Thus, we delve into training workers in specific microtasks of different types. We exploit (i) implicit training, where workers are provided training when they provide erraneous responses to questions with priorly known answers, and (ii) explicit training, where workers are required to go through a training phase before they attempt to work on the task itself. We evaluated our approach in 4 different types of microtasks with a total of 1200 workers, who were subjected to either one of the proposed training strategies or baseline case of no training. The results show that workers who undergo training depict an improvement in performance upto 5 %, and a reduction in the task completion time upto 41 %. Additionally, crowd training led to the elimination of malicious workers and a costs-benefit gain upto nearly 15%.

Published in: Education
  • Be the first to comment

  • Be the first to like this

Training Workers for Improving Performance in Crowdsourcing Microtasks

  1. 1. Training Workers for Improving Performance in Crowdsourcing Microtasks Ujwal Gadiraju, Besnik Fetahu, and Ricardo Kawase ECTEL’15 Toledo, Spain 16th September, 2015
  2. 2. Source: altamartv
  3. 3. Source: http://www.mission4636.org/
  4. 4. Dalila: I need Thomassin Apo please Apo: Kenscoff Route: Lat: 18.495746829274168, Long:-72.31849193572998 Apo: This Area after Petion-Ville and Pelerin 5 is not on Google Map. We have no streets name Apo: I know this place like my pocket Dalila: thank God u was here “just got emergency SMS, child delivery, USCG are acting, and the GPS coordinates of the location we got from the translators were 100% accurate!”
  5. 5. ● People from over 50 countries participated in relief efforts ● Free phone number 4636 ● Maps about aid stations and food distribution centers ● Sustainability: Created 100 jobs Ahead of the curve in all relief efforts! Mission 4636 HOW ?! A triumph of people working together and doing their small bits.
  6. 6. Crowdsourcing-A Brief Introduction “The whole is greater than the sum of its parts.” - Aristotle ● Accumulating small contributions from each crowd worker to solve a bigger problem.
  7. 7. Crowdsourcing-A Brief Introduction “The whole is greater than the sum of its parts.” - Aristotle ● Accumulating small contributions from each crowd worker to solve a bigger problem.
  8. 8. Some Remarkable Outcomes
  9. 9. Crowdsourcing Platforms PAID MICROTASK CROWDSOURCING
  10. 10. Outline ● Introduction ● Motivation ● Methodology ● Analysis & Results ● Conclusions 13 Training Workers for Improving Performance in Crowdsourcing Microtasks
  11. 11. Motivation ● Crowdsourcing labor markets are being widely used nowadays. ● Optimizing the quality of results produced is of prime importance. ● At the same time, crowd workers aim to maintain high reputation scores. ○ A worker’s good reputation gives access to more tasks. ○ A worker consequently can earn more money.
  12. 12. Research Questions ❏ How can crowd workers be trained within crowdsourcing microtasks? ❏ What is the impact of training on the performance of crowd workers? ❏ How does the impact of training differ across various types of microtasks?
  13. 13. Crowd Workers as Learners We recognize a crowd worker as a learner in an atypical learning environment : ● There is no information regarding the background, knowledge, or skills of a crowd worker. ● Given the short nature of crowdsourced microtasks, workers face an ‘on-the-fly’ learning situation. ○ Comparable to experiential learning and microlearning. ● In many cases, workers have no time to apply their gained experience. ● Often for single use, high % of new requesters.
  14. 14. Training Strategies Implicit Training Explicit Training ● Workers are prompted with training information on providing erroneous responses to questions with priorly known answers. ● Workers are prompted with training information before they begin responding to questions within the actual task.
  15. 15. Taxonomy of Microtasks Information Finding Verification & Validation Interpretation & Analysis Content Creation Surveys Content Access A Taxonomy of Microtasks on the Web. Ujwal Gadiraju, Ricardo Kawase and Stefan Dietze. In Proceedings of the 25th ACM Conference on Hypertext and Social Media. 2014. (IF) Finding Middle-names (VV) Spam Detection (IA) Sentiment Analysis (CC) Image Transcription
  16. 16. Experimental Setup and Tasks Design For each type of task: ● 3 Experiments ○ No-Training (NT) ○ Implicit Training (IT) ○ Explicit Training (ET) ● Tasks deployed on CrowdFlower ● Responses from 100 independent crowd workers for each training setup ● Monetary compensation of 5 USD cents on successful task completion ● Randomized order of units, clear instructions ● Task lengths to beget similar task completion time for different types
  17. 17. Information Finding Verification & Validation ● Find middle-names ● 20 units in each task ● No-Training ● Implicit Training on erroneous responses ● 5 units of Explicit Training ● Identify spam SMS from a set of 5 SMSes ● 30 units in each task ● No-Training ● Implicit Training on erroneous responses ● 5 units of Explicit Training Error Reason: Seemingly fake advertisements or meaningless offers are good candidates of being spam messages.
  18. 18. HAM or SPAM Example Unit
  19. 19. Interpretation & Analysis Content Creation ● Assess sentiment of tweets ● 30 units in each task ● No-Training ● Implicit Training on erroneous responses ● 5 units of Explicit Training ● Transcribe the text from images (CAPTCHAs) ● 30 units in each task ● No-Training ● Implicit Training on erroneous responses ● 5 units of Explicit Training
  20. 20. ● No Training: 92% ● Implicit Training: 96% ● Explicit Training: 98% Results 1/4 Worker Performance (accuracy)
  21. 21. ● No Training: 82% ● Implicit Training: 65% ● Explicit Training: 77% Results 2/4 Worker Performance (accuracy)
  22. 22. ● No Training: 86% ● Implicit Training: 89% ● Explicit Training: 92% Results 3/4 Worker Performance (accuracy)
  23. 23. ● No Training: 91.6% ● Implicit Training: 96.37% ● Explicit Training: 98.14% Results 4/4 Worker Performance (accuracy)
  24. 24. Impact of Training on Elite Workers
  25. 25. Impact of Training on Unit Completion Time
  26. 26. Impact of Training on Malicious Workers Percentage of Malicious Workers across different types of Tasks “workers with ulterior motives, who either simply sabotage a task, or provide poor responses in an attempt to quickly attain task completion for monetary gains”
  27. 27. Costs-Benefit Analysis
  28. 28. ● Information Finding : Explicit Training improves the costs-benefit ratio by 8.5%. ● Verification & Validation : Explicit Training improves the costs-benefit ratio by 1.4%. ● Interpretation & Analysis : Explicit Training improves the costs-benefit ratio by 14.6%. ● Content Creation : Explicit Training improves the costs-benefit ratio by 8.5%. Costs-Benefit Analysis
  29. 29. Caveats, Limitations and Future Work ● Quantum of Training We need further experiments to quantify the ideal duration/length of training excerpts (both implicit and explicit). ● Quality of Training Excerpts Experiments that assess the impact of varying training ‘quality’ on crowd worker performance. ● Cost overheads due to training ● Impact of Task Complexity
  30. 30. Conclusions ● Training crowd workers in microtasks can lead to substantial improvements in their performance. ● Explicit Training leads to the most improvement in individual worker performance and overall quality of the results. ● We studied the effect of training across different types of microtasks. RQ #1 RQ #2 RQ #3 Training workers in crowdsourcing microtasks makes for an interesting learning environment with opportunities galore!
  31. 31. Contact Details : gadiraju@l3s.de http://www.L3S.de SLIDES: http://www. slideshare.net/ujwal07/ 34
  32. 32. Geographical Distribution of Workers ● No restrictions on worker participation. ○ Information Finding: 50% from India, 25% from Turkey ○ Verification and Validation: 53% from Spain, 24% from Italy ○ Interpretation and Analysis: 37% from Bosnia and Herzegovina, 31% from Turkey ○ Content Creation: 40% from Philipines, 30% from Great Britain

×