A brief introduction to crowdsourcing for data collection

775 views
659 views

Published on

Examples of crowdsourcing experiments for research data forensics, talk at the USEWOD workshop at ESWC2014

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
775
On SlideShare
0
From Embeds
0
Number of Embeds
41
Actions
Shares
0
Downloads
5
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

A brief introduction to crowdsourcing for data collection

  1. 1. A BRIEF INTRODUCTION TO CROWDSOURCED DATA COLLECTION ELENA SIMPERL UNIVERSITY OF SOUTHAMPTON 25-May-14 USEWOD@ESWC2014 1
  2. 2. EXECUTIVE SUMMARY • Crowdsourcing can help with research data forensics • But • There are things computers do better than humans  hybrid approaches are the ultimate solution • There is crowdsourcing and crowdsourcing  pick your faves and mix them • Human intelligence is a valuable resource  experiment design is key 2
  3. 3. CROWDSOURCING: PROBLEM SOLVING VIA OPEN CALLS "Simply defined, crowdsourcing represents the act of a company or institution taking a function once performed by employees and outsourcing it to an undefined (and generally large) network of people in the form of an open call. This can take the form of peer-production (when the job is performed collaboratively), but is also often undertaken by sole individuals. The crucial prerequisite is the use of the open call format and the large network of potential .“ [Howe, 2006] 25-May-14 3
  4. 4. CROWDSOURCING COMES IN DIFFERENT FORMS AND FLAVORS 25-May-14 4
  5. 5. 25-May-14 5 DIMENSIONS OF CROWDSOURCING
  6. 6. DIMENSIONS OF CROWDSOURCING WHAT IS OUTSOURCED • Tasks based on human skills not easily replicable by machines • Visual recognition • Language understanding • Knowledge acquisition • Basic human communication • ... WHO IS THE CROWD • Open call (crowd accessible through a platform) • Call may target specific skills and expertise (qualification tests) • Requester typically knows less about the ‘workers’ than in other ‘work’ environments 25-May-14 6 See also [Quinn & Bederson, 2012]
  7. 7. USEWOD EXPERIMENT: TASK AND CROWD WHAT IS OUTSOURCED • Annotating research papers with data set information. • Alternative representations of the domain • Bibliographic reference • Abstract + title • Paragraph • Full paper • What if the domain is not known in advance or is infinite? • Do we know the list of potential answers? • Is there only one correct solution to each atomic task? • How many people would solve the same task? WHO IS THE CROWD • People who know the papers or the data sets • Experts in the (broader ) field • Casual gamers • Librarians • Anyone (knowledgeable of English, with a computer/cell phone…) • Combinations thereof… 25-May-14 7
  8. 8. 25-May-14 Tutorial@ISWC2013 CROWDSOURCING AS ‚HUMAN COMPUTATION‘ Outsourcing tasks that machines find difficult to solve to humans 8
  9. 9. DIMENSIONS OF CROWDSOURCING (2) HOW IS THE TASK OUTSOURCED • Explicit vs. implicit participation • Tasks broken down into smaller units undertaken in parallel by different people • Coordination required to handle cases with more complex workflows • Partial or independent answers consolidated and aggregated into complete solution 25-May-14 9 See also [Quinn & Bederson, 2012]
  10. 10. EXAMPLE: CITIZEN SCIENCE WHAT IS OUTSOURCED • Object recognition, labeling, categorization in media content WHO IS THE CROWD • Anyone HOW IS THE TASK OUTSOURCED • Highly parallelizable tasks • Every item is handled by multiple annotators • Every annotator provides an answer • Consolidated answers solve scientific problems 25-May-14 10
  11. 11. Users aware of how their input contributes to the achievement of application’s goal (and identify themselves with it) vs. Tasks are hidden behind the application narratives. Engagement ensured through other incentives 25-May-14 11 EXPLICIT VS. IMPLICIT CONTRIBUTION - AFFECTS MOTIVATION AND ENGAGEMENT
  12. 12. USEWOD EXPERIMENT: TASK DESIGN HOW IS THE TASK OUTSOURCED: ALTERNATIVE MODELS • Use the data collected here to train a IE algorithm • Use paid microtask workers to go a first screening, then expert crowd to sort out challenging cases • What if you have very long documents potentially mentioning different/unknown data sets? • Competition via Twitter • ‘Which version of DBpedia does this paper use?’ • One question a day, prizes • Needs golden standard to bootstrap and redundancy • Involve the authors • Use crowdsourcing to find out Twitter accounts, then launch campaign on Twitter • Write an email to the authors… • Change the task • Which papers use Dbpedia 3.X? • Competition to find all papers 25-May-14 12
  13. 13. EXAMPLE: SOYLENT AND COMPLEX WORKFLOWS 25-May-14 13 http://www.youtube.com/watch?v=n_miZqsPwsc WHAT IS OUTSOURCED • Text shortening, proof- reading, open editing WHO IS THE CROWD • MTurk HOW IS THE TASK OUTSOURCED • Text divided into paragraphs • Select-fix-verify pattern • Multiple workers in each step See also [Bernstein et al., 2010]
  14. 14. DIMENSIONS OF CROWDSOURCING (3) HOW ARE THE RESULTS VALIDATED • Solutions space closed vs. open • Performance measurements/ground truth • Statistical techniques employed to predict accurate solutions • May take into account confidence values of algorithmically generated solutions HOW CAN THE PROCESS BE OPTIMIZED • Incentives and motivators • Assigning tasks to people based on their skills and performance (as opposed to random assignments) • Symbiotic combinations of human- and machine- driven computation, including combinations of different forms of crowdsourcing 25-May-14 14 See also [Quinn & Bederson, 2012]
  15. 15. USEWOD EXPERIMENT: VALIDATION • Domain is fairly restricted • Spam and obvious wrong answers can be detected easily • When are two answers the same? Can there be more than one correct answer per question? • Redundancy may not be the final answer • Most people will be able to identify the data set, but sometimes the actual version is not trivial to reproduce • Make educated version guess based on time intervals and other features 25-May-14 15
  16. 16. ALIGNING INCENTIVES IS ESSENTIAL Motivation: driving force that makes humans achieve their goals Incentives: ‘rewards’ assigned by an external ‘judge’ to a performer for undertaking a specific task • Common belief (among economists): incentives can be translated into a sum of money for all practical purposes. Incentives can be related to both extrinsic and intrinsic motivations. Extrinsic motivation if task is considered boring, dangerous, useless, socially undesirable, dislikable by the performer. Intrinsic motivation is driven by an interest or enjoyment in the task itself. 16
  17. 17. EXAMPLE: DIFFERENT CROWDS FOR DIFFERENT TASKS Contest Linked Data experts Difficult task Final prize Find Verify Microtasks Workers Easy task Micropayments TripleCheckMate [Kontoskostas2013] MTurk Adapted from [Bernstein2010] http://mturk.com See also [Acosta et al., 2013] 17
  18. 18. IT‘S NOT ALWAYS JUST ABOUT MONEY 25-May-14 18 http://www.crowdsourcing.org/editorial/how-to-motivate-the-crowd-infographic/ http://www.oneskyapp.com/blog/tips-to-motivate-participants-of-crowdsourced- translation/ [Kaufmann, Schulze, Viet, 2011]
  19. 19. USEWOD EXPERIMENT: OTHER INCENTIVES MODELS • Twitter-based contest • ‘Which version of DBpedia does this paper use?’ • One question a day, prizes • If question is not answered correctly, increase the prize • If low participation, re-focus the audience or change the incentive. • Altruism: for each ten papers annotated we send a student to ESWC… 25-May-14 19
  20. 20. PRICING ON MTURK: AFFORDABLE, BUT SCALE OF EXPERIMENTS DOES MATTER 25-May-14 20 [Ipeirotis, 2008]
  21. 21. USEWOD EXPERIMENT: HYBRID APPROACH • Use IE algorithm to select best candidates • Use different types of crowds • Publish results as Linked Data 25-May-14 21 See also [Demartini et al., 2012]
  22. 22. 25-May-14 22 SUMMARY
  23. 23. SUMMARY AND FINAL REMARKS • There are things computers do better than humans  hybrid approaches are the ultimate solution • There is crowdsourcing and crowdsourcing  pick your faves and mix them • Human intelligence is a valuable resource  experiment design is key 25-May-14 23

×