Pick a Crowd

470 views
286 views

Published on

Published in: Business, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
470
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Pick a Crowd

  1. 1. Pick-A-Crowd: Tell Me What You Like, and I’ll Tell You What to Do A Crowdsourcing Platform for Personalized Human Intelligence Task Assignment Based on Social Networks Djellel E. Difallah, GianlucaDemartini, Philippe Cudré-Mauroux eXascaleInfolab University of Fribourg, Switzerland 15th May 2013, WWW 2013 - Rio De Janeiro, Brazil 1
  2. 2. Crowdsourcing • Exploit human intelligence to solve tasks that are simple for Humans and complex for machines • Examples: – Wikipedia, reCaptcha, Duolingo • Incentives – Financial, fun, visibility 2
  3. 3. Motivation • The Pull Methodology is suboptimal Actual workers Max Overlap Effective workers 3
  4. 4. Motivation • The Push Methodology is a Task-to-Worker Recommender System. 4
  5. 5. Contribution and Claim • Pick-A-Crowd: A system architecture that uses Task-to-Worker matching: – The worker’s social profile – The task context • Workers can provide higher quality answers on tasks they relate to 5
  6. 6. Worker Social Profiling “YouAreWhatYouLike” 7
  7. 7. Problem Definition (1)The Human Intelligence Task (HIT) Categorization Survey Image Tagging Data Collection Batch of Tasks: Title Batch Instruction Specific task instruction* Task data: - Text. - Options. - Additional data (image, Url) List of categories* 8
  8. 8. ProblemDefinition (2)The Worker Completed HITs: 256 Approval Rate: 96% Qualification Types Generic Qualifications Page: Page: Page: - -Title Title - Title - -Category Category - Category - -Description Description - Description - -Feed, etc. Feed, etc. - Feed, etc. 9
  9. 9. Problem Definition (3) – Task-to-Worker Matching Batch of Tasks: Title Batch Instruction Specific task instruction* Task data: - Text. - Options. - Additional data (image, Url) List of categories* Page: Page: Page: - -Title Title - Title - -Category Category - Category - -Description Description - Description - -Feed, etc. Feed, etc. - Feed, etc. 1- Task-to-Page Matching Function - Category - Expert finding - Semantic 2- Worker Ranking 10
  10. 10. Matching Models (1/3)– Category Based • The requester provides a list of categories related to the batch • We create a subset of pages whose category is in the category list of the batch • Rank the workers by the number of liked pages in the subset 11
  11. 11. Matching Models (2/3) – Expert Finding • • • Build an inverted index on the pages’ titles and description Use the title/description of the tasks as a key word query on the inverted index and get a subset of pages Rank the workers by the number of liked pages in the subset 12
  12. 12. Matching Models (3/3) – Semantic Based • • Link the context to an external knowledge base (e.g., DBPedia) Exploit the underlying graph structure to determine the Hits and Pages similarity – Assumption that a worker who likes a page is able to answer questions about related entities – Worker who likes a page is able to answer questions about entities of the same type • Rank the workers by the number of liked pages in the subset Similarity Relatedness HIT FB Pages Type-Similarity 13
  13. 13. Pick-A-Crowd Architecture 15
  14. 14. Experimental Evaluation • The Facebook app OpenTurkimplements part of the Pick-A-Crowd architecture: – More than 170 registered workers participated – Over 12k pages crawled • Covered both multiple answer questions as well as open-ended questions – 50 images with multiple choice question and 5 candidate answers (Soccer, Actors, Music, Authors,Movies, Animes) – Answer 20 open-ended questions related to the topic (Cricket) 16
  15. 15. OpenTurk app 18
  16. 16. Evaluation - WORKER PRECISION Correlation between the crowd accuracy and the number of relevant likes (Category Based) NUMBER OF RELEVANT LIKES 19
  17. 17. Evaluation (Baseline) – Amazon Mechanical Turk (AMT) AMT 3 = Majority vote of 3 workers AMT 5 = Majority vote of 5 workers 20
  18. 18. Evaluation – HIT Assignment Models CATEGORY APPROACH 21
  19. 19. Evaluation – HIT Assignment Models EXPERT FINDING BASED TITLE/INSTRUCTION CONTENT 22
  20. 20. Evaluation – HIT Assignment Models SEMANTIC BASED TYPE RELATEDNESS 23
  21. 21. PICK-A-CROWD AMT Evaluation Comparison With Mechanical Turk 24
  22. 22. Conclusions and Future Work • Pull vs. Pushmethodologies in Crowdsourcing • Pick-A-Crowd system architecture with Taskto-Worker recommendation • Experimental comparison with AMT shows a consistent quality improvement “Workers Know what they Like” • Exploit more of the social activity, and handle content-less tasks 25
  23. 23. Next Step • We are building a Crowdsourcing platform for the research community • Pre-register on: www.openturk.com Thank You! 26

×