https://www.insight-centre.org/content/capability-requirements-approach-predicting-worker-performance-crowdsourcing Presented at CollaborateCom 2013 Abstract: Assigning heterogeneous tasks to workers is an important challenge of crowdsourcing platforms. Current approaches to task assignment have primarily focused on contentbased approaches, qualifications, or work history. We propose an alternative and complementary approach that focuses on what capabilities workers employ to perform tasks. First, we model various tasks according to the human capabilities required to perform them. Second, we capture the capability traces of the crowd workers performance on existing tasks. Third, we predict performance of workers on new tasks to make task routing decisions, with the help of capability traces. We evaluate the effectiveness of our approach on three different tasks including fact verification, image comparison, and information extraction. The results demonstrate that we can predict worker’s performance based on worker capabilities. We also highlight limitations and extensions of the proposed approach.

1. Digital Enterprise Research Institute
www.deri.ie
A CAPABILITY REQUIREMENTS APPROACH FOR
PREDICTING WORKER PERFORMANCE IN
CROWDSOURCING
Umair ul Hassan, Edward Curry
Digital Enterprise Research Institute
National University of Ireland, Galway
9th IEEE International Conference on Collaborative Computing:
Networking, Applications and Worksharing
Austin, Texas, United States
October 20–23, 2013
Copyright 2010 Digital Enterprise Research Institute. All rights reserved.

2. Agenda
Digital Enterprise Research Institute



Motivation
Background
Task Modelling



Capability Requirements


www.deri.ie
Capabilities Taxonomy
Capability Tracing
Experiment
Summary
2

3. Motivation: Heterogeneity
Digital Enterprise Research Institute
www.deri.ie
3

4. Motivation: Task Routing
Digital Enterprise Research Institute

www.deri.ie
Assigning heterogeneous tasks to heterogeneous workers
TASK MODELLING
Models
Models
Models
TASK ROUTING
WORKER PROFILING
Matching
Profiles
Profiles
Profiles
Task↔Worker
4

5. Proposal: Performance Prediction
Digital Enterprise Research Institute

www.deri.ie
Predict performance of workers on new tasks based on the
capabilities required for tasks and assign tasks accordingly
TASK MODELLING
Models
Models
Models
Capability
Requirements
Approach
TASK ROUTING
WORKER PROFILING
Matching
Profiles
Profiles
Profiles
Task↔Worker
Performance
Prediction
Capability Tracing
Model
5

6. Background: Micro tasks
Digital Enterprise Research Institute

www.deri.ie
When micro tasks are crowd sourced


Single person cannot do the task


Computers cannot do the task
Work can be split into smaller tasks
Some online microtask platforms
6

7. Background: Micro tasks
Digital Enterprise Research Institute

www.deri.ie
Most common tasks in Amazon Mechanical Turk (AMT)
and CrowdFlower (CFL)
7

8. Background: Micro tasks
Digital Enterprise Research Institute

www.deri.ie
Example of information extraction task in AMT
8

9. Background: Micro tasks
Digital Enterprise Research Institute

www.deri.ie
Example of video transcription task in AMT
9

10. Task Modelling
Digital Enterprise Research Institute
www.deri.ie

Appropriate models are needed to compare and contrast
micro tasks.

Capability Requirements approach

Capability is defined as the ability of humans to do things in
terms of both the capacity and the opportunity.

Four types of capabilities
–
–
–
–
Knowledge,
Skill,
Ability,
Other characteristics (e.g. motivation, price, etc)
10

11. Capability Requirements
Digital Enterprise Research Institute

www.deri.ie
Taxonomies have be used to study human task
performance, e.g.


Bloom’s taxonomy of classification of learning objectives


Fleishman’s taxonomy of human abilities
O*NET-SOC taxonomy of occupational classification
We are interested in taxonomy that

Describes tasks in terms of human capabilities

Helps in comparing tasks in terms of differences and similarities
of capabilities
11

12. Capabilities Taxonomy
Digital Enterprise Research Institute


www.deri.ie
Based on Fleishman’s abilities taxonomy
Selected abilities relevant to micro tasks

Comprehension (C): The ability to understand the meaning or importance of
something

Bilingualism (B): The ability to speak and understand two languages

Writing (W): The ability or capacity to write text in a given language

Comparison (M): The ability or capacity to compare things based on some
criteria

Judgment (J): The act or process of judging; the formation of an opinion after
consideration

Perception (P): The ability or capacity to perceive items visually or phonetically

Identification (I): The process of recognizing something

Reasoning (R): The ability to draw conclusions from
facts, evidence, relationships, etc.
12

13. Requirements of Micro Tasks
Digital Enterprise Research Institute
www.deri.ie
13

14. Capability Tracing
Digital Enterprise Research Institute


www.deri.ie
How to model worker’s capabilities?
Capability tracing



Inspired by Knowledge Tracing*
Estimates probability of a worker knowing a capability given
worker’s responses to test tasks
Worker Profile constrains

Set of binary variables representing capabilities

Probability estimates of each variable being in a state
* A. T. Corbett and J. R. Anderson, “Knowledge tracing: Modeling the acquisition of procedural knowledge,” User Modeling and User-Adapted
Interaction, vol. 4, no. 4, pp. 253–278, 1994.
14

15. Capability Tracing
Digital Enterprise Research Institute

www.deri.ie
Probabilistic network of a capability and four parameters of
capability tracing model
States of
Capability
Variable
Not
Learned
p(T): Probability of
transition between states
p(T)
Learned
p(L)
p(G)
Values of
Response
Variable
p(L): Probability of a
worker learning to
employ the capability
p(S)
p(G): Probability of
guess
Correct
Incorrect
15
p(S): Probability of slip

16. Experiment
Digital Enterprise Research Institute

www.deri.ie
Objective



Solicit capability requirements of tasks from crowds
Evaluation of capability tracing for performance prediction
Three types of micro tasks with manually created ground
truth data


Image comparison


Fact verification
Information Extraction
37 crowd workers including

University students

Workers from Shorttask.com
16

17. Crowdsourcing
Digital Enterprise Research Institute


www.deri.ie
Custom web application for gathering data
Example of fact verification task
17

18. Capability Requirements of Tasks
Digital Enterprise Research Institute

www.deri.ie
Objective 1: Solicit capability requirements of tasks
from crowds
(a) fact verification
(b) image comparison
18
(c) information extraction

19. Crowd Performance
Digital Enterprise Research Institute

www.deri.ie
How the crowd performed on each type of task?
Fact Verification task
• 37 workers
• Best workers perform with
both precision and recall
above 0.8
• More variation in recall means
some workers were could not
spot the incorrect facts
• Ideally tasks should be
assigned to workers that lie in
the top-right quadrant of the
plot
19

20. Crowd Performance
Digital Enterprise Research Institute

www.deri.ie
Image Comparison (20 workers) and Information Extraction (17 workers)
20

21. Performance Prediction
Digital Enterprise Research Institute


www.deri.ie
Objective 2: Evaluation of capability tracing for
performance prediction
Two phases

Build model with observation tasks

Predict performance on new tasks
AC: Consider previous
Accuracy as prediction
of future performance
CT: Capability Tracing
21

22. Summary
Digital Enterprise Research Institute


Capabilities taxonomy is first steps towards modelling of
micro tasks based on human factors
Capability tracing is effective in predicting future
performance



www.deri.ie
Even across tasks if there are similar capabilities
Predicted performance can be used to make right task
routing decisions
Future Work

Evaluate on more types of tasks

Evaluate capabilities such as domain knowledge and skills

Define standard tests for measuring worker capabilities
22

23. Further Reading
Digital Enterprise Research Institute
www.deri.ie
9th IEEE International Conference on Collaborative Computing:
Networking, Applications and Worksharing
Austin, Texas, United States
October 20–23, 2013
U. Ul Hassan and E. Curry, “A Capability Requirements Approach for Predicting
Worker Performance in Crowdsourcing,” in 9th IEEE International Conference on
Collaborative Computing: Networking, Applications and Worksharing, 2013.
http://deri.ie/users/umair-ul-hassan
23

24. Capability Tracing
Digital Enterprise Research Institute
www.deri.ie

Conditional probability of worker learning to employ
capability p(Ln|On) is calculated

When evidence On is positive

When evidence On is negative
24

25. Capability Tracing
Digital Enterprise Research Institute
www.deri.ie

Probability of worker learning to employ capability

Performance of worker on next task
25