This document presents Pick-A-Crowd, a crowdsourcing platform that uses a task-to-worker recommender system to match workers to human intelligence tasks based on their social profiles and task contexts. It describes common crowdsourcing approaches, and proposes using workers' social media likes to determine their expertise and assign them to relevant tasks. An evaluation of Pick-A-Crowd showed it improved task quality over traditional methods by better targeting workers. The authors invite others to pre-register for their new OpenTurk platform to further develop these crowd-powered techniques.

1. Pick-A-Crowd: Tell Me What You Like,
and I’ll Tell You What to Do
A Crowdsourcing Platform for Personalized
Human Intelligence Task Assignment Based on Social
Networks
Djellel E. Difallah, GianlucaDemartini, Philippe Cudré-Mauroux
eXascaleInfolab
University of Fribourg, Switzerland
15th May 2013, WWW 2013 - Rio De Janeiro, Brazil
1

2. Crowdsourcing
• Exploit human intelligence to solve tasks that
are simple for Humans and complex for
machines
• Examples:
– Wikipedia, reCaptcha, Duolingo
• Incentives
– Financial, fun, visibility
2

3. Motivation
• The Pull Methodology is suboptimal
Actual workers
Max Overlap
Effective workers
3

4. Motivation
• The Push Methodology is a Task-to-Worker
Recommender System.
4

5. Contribution and Claim
• Pick-A-Crowd: A system architecture that uses
Task-to-Worker matching:
– The worker’s social profile
– The task context
• Workers can provide higher quality answers
on tasks they relate to
5

6. Worker Social Profiling
“YouAreWhatYouLike”
7

7. Problem Definition (1)The Human Intelligence Task (HIT)
Categorization
Survey
Image Tagging
Data Collection
Batch of Tasks:
Title
Batch Instruction
Specific task instruction*
Task data:
- Text.
- Options.
- Additional data (image, Url)
List of categories*
8

8. ProblemDefinition (2)The Worker
Completed HITs: 256
Approval Rate: 96%
Qualification Types
Generic Qualifications
Page:
Page:
Page:
- -Title
Title
- Title
- -Category
Category
- Category
- -Description
Description
- Description
- -Feed, etc.
Feed, etc.
- Feed, etc.
9

9. Problem Definition (3) –
Task-to-Worker Matching
Batch of Tasks:
Title
Batch Instruction
Specific task instruction*
Task data:
- Text.
- Options.
- Additional data (image, Url)
List of categories*
Page:
Page:
Page:
- -Title
Title
- Title
- -Category
Category
- Category
- -Description
Description
- Description
- -Feed, etc.
Feed, etc.
- Feed, etc.
1- Task-to-Page Matching Function
- Category
- Expert finding
- Semantic
2- Worker Ranking
10

10. Matching Models (1/3)–
Category Based
• The requester provides a list of categories related to the batch
• We create a subset of pages whose category is in the category
list of the batch
• Rank the workers by the number of liked pages in the subset
11

11. Matching Models (2/3) –
Expert Finding
•
•
•
Build an inverted index on the pages’ titles and description
Use the title/description of the tasks as a key word query on the
inverted index and get a subset of pages
Rank the workers by the number of liked pages in the subset
12

12. Matching Models (3/3) –
Semantic Based
•
•
Link the context to an external knowledge base (e.g., DBPedia)
Exploit the underlying graph structure to determine the Hits and Pages similarity
– Assumption that a worker who likes a page is able to answer questions about related entities
– Worker who likes a page is able to answer questions about entities of the same type
•
Rank the workers by the number of liked pages in the subset
Similarity
Relatedness
HIT
FB Pages
Type-Similarity
13

13. Pick-A-Crowd Architecture
15

14. Experimental Evaluation
• The Facebook app OpenTurkimplements part
of the Pick-A-Crowd architecture:
– More than 170 registered workers participated
– Over 12k pages crawled
• Covered both multiple answer questions as
well as open-ended questions
– 50 images with multiple choice question and 5 candidate answers
(Soccer, Actors, Music, Authors,Movies, Animes)
– Answer 20 open-ended questions related to the topic (Cricket)
16

15. OpenTurk app
18

16. Evaluation -
WORKER PRECISION
Correlation between the crowd accuracy and
the number of relevant likes (Category Based)
NUMBER OF RELEVANT LIKES
19

17. Evaluation (Baseline) –
Amazon Mechanical Turk (AMT)
AMT 3 = Majority vote of 3 workers
AMT 5 = Majority vote of 5 workers
20

18. Evaluation –
HIT Assignment Models
CATEGORY APPROACH
21

19. Evaluation –
HIT Assignment Models
EXPERT FINDING BASED
TITLE/INSTRUCTION
CONTENT
22

20. Evaluation –
HIT Assignment Models
SEMANTIC BASED
TYPE
RELATEDNESS
23

21. PICK-A-CROWD
AMT
Evaluation Comparison With Mechanical Turk
24

22. Conclusions and Future Work
• Pull vs. Pushmethodologies in Crowdsourcing
• Pick-A-Crowd system architecture with Taskto-Worker recommendation
• Experimental comparison with AMT shows a
consistent quality improvement
“Workers Know what they Like”
• Exploit more of the social activity, and handle
content-less tasks
25

23. Next Step
• We are building a Crowdsourcing platform for
the research community
• Pre-register on:
www.openturk.com
Thank You!
26