Digital Enterprise Research Institute

www.deri.ie

A CAPABILITY REQUIREMENTS APPROACH FOR
PREDICTING WORKER PERFORMANCE I...
Agenda
Digital Enterprise Research Institute





Motivation
Background
Task Modelling





Capability Requirements
...
Motivation: Heterogeneity
Digital Enterprise Research Institute

www.deri.ie

3
Motivation: Task Routing
Digital Enterprise Research Institute



www.deri.ie

Assigning heterogeneous tasks to heterogen...
Proposal: Performance Prediction
Digital Enterprise Research Institute



www.deri.ie

Predict performance of workers on ...
Background: Micro tasks
Digital Enterprise Research Institute



www.deri.ie

When micro tasks are crowd sourced



Sin...
Background: Micro tasks
Digital Enterprise Research Institute



www.deri.ie

Most common tasks in Amazon Mechanical Turk...
Background: Micro tasks
Digital Enterprise Research Institute



www.deri.ie

Example of information extraction task in A...
Background: Micro tasks
Digital Enterprise Research Institute



www.deri.ie

Example of video transcription task in AMT
...
Task Modelling
Digital Enterprise Research Institute

www.deri.ie



Appropriate models are needed to compare and contras...
Capability Requirements
Digital Enterprise Research Institute



www.deri.ie

Taxonomies have be used to study human task...
Capabilities Taxonomy
Digital Enterprise Research Institute




www.deri.ie

Based on Fleishman’s abilities taxonomy
Sel...
Requirements of Micro Tasks
Digital Enterprise Research Institute

www.deri.ie

13
Capability Tracing
Digital Enterprise Research Institute




www.deri.ie

How to model worker’s capabilities?
Capability...
Capability Tracing
Digital Enterprise Research Institute



www.deri.ie

Probabilistic network of a capability and four p...
Experiment
Digital Enterprise Research Institute



www.deri.ie

Objective





Solicit capability requirements of tas...
Crowdsourcing
Digital Enterprise Research Institute




www.deri.ie

Custom web application for gathering data
Example o...
Capability Requirements of Tasks
Digital Enterprise Research Institute



www.deri.ie

Objective 1: Solicit capability re...
Crowd Performance
Digital Enterprise Research Institute



www.deri.ie

How the crowd performed on each type of task?

Fa...
Crowd Performance
Digital Enterprise Research Institute



www.deri.ie

Image Comparison (20 workers) and Information Ext...
Performance Prediction
Digital Enterprise Research Institute




www.deri.ie

Objective 2: Evaluation of capability trac...
Summary
Digital Enterprise Research Institute




Capabilities taxonomy is first steps towards modelling of
micro tasks ...
Further Reading
Digital Enterprise Research Institute

www.deri.ie

9th IEEE International Conference on Collaborative Com...
Capability Tracing
Digital Enterprise Research Institute

www.deri.ie



Conditional probability of worker learning to em...
Capability Tracing
Digital Enterprise Research Institute

www.deri.ie



Probability of worker learning to employ capabil...
Upcoming SlideShare
Loading in …5
×

A Capability Requirements Approach for Predicting Worker Performance in Crowdsourcing

725 views

Published on

https://www.insight-centre.org/content/capability-requirements-approach-predicting-worker-performance-crowdsourcing

Presented at CollaborateCom 2013

Abstract:
Assigning heterogeneous tasks to workers is an important challenge of crowdsourcing platforms. Current approaches to task assignment have primarily focused on contentbased approaches, qualifications, or work history. We propose an
alternative and complementary approach that focuses on what capabilities workers employ to perform tasks. First, we model various tasks according to the human capabilities required to perform them. Second, we capture the capability traces of the
crowd workers performance on existing tasks. Third, we predict performance of workers on new tasks to make task routing decisions, with the help of capability traces. We evaluate the effectiveness of our approach on three different tasks including fact
verification, image comparison, and information extraction. The results demonstrate that we can predict worker’s performance based on worker capabilities. We also highlight limitations and extensions of the proposed approach.

Published in: Data & Analytics, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
725
On SlideShare
0
From Embeds
0
Number of Embeds
36
Actions
Shares
0
Downloads
16
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • AMT is amazonmechincalturk
  • AMT is amazonmechincalturk
  • AMT is amazonmechincalturk
  • AMT is amazon mechanical turk
  • Probability estimates are generated while learning the model through capability tracing
  • As can be seen, more than majority of workers believed that identification capability is essential for all three types of tasks. Majority of workers also agreed that the judgment and comprehension capabilities are important for the Fact Verification task. In the case of the Image Comparison task most workers specified that comparison and perception are important as well. There is general consensus between workers that comprehension, judgment and reasoning capabilities are also useful for Information Extraction tasks. We selected top-3 capabilities for each type of task for building capability tracing models.
  • Interestingly, no worker achieved the highest recall for the information extraction task, which highlights the difference between workers and ground truth in terms of the entities extracted from the Wikipedia articles. Nevertheless, these distributions emphasize that in order to achieve high accuracy tasks should be assigned to workers that lie in the top-right quadrant of the plots.
  • Results show that the capability tracing approach is comparable to the baseline approach in general and achieves better accuracy of prediction between similar tasks. The Fact Verification and Information Extraction tasks have similar capabilities requirements, therefore capability tracingcan better predict the performance of workers between them. The drop in prediction quality of capability tracing for Image Comparison task can be attributed to the little variation is the performance of workers on this task.
  • Hide these
  • Hide these
  • A Capability Requirements Approach for Predicting Worker Performance in Crowdsourcing

    1. 1. Digital Enterprise Research Institute www.deri.ie A CAPABILITY REQUIREMENTS APPROACH FOR PREDICTING WORKER PERFORMANCE IN CROWDSOURCING Umair ul Hassan, Edward Curry Digital Enterprise Research Institute National University of Ireland, Galway 9th IEEE International Conference on Collaborative Computing: Networking, Applications and Worksharing Austin, Texas, United States October 20–23, 2013 Copyright 2010 Digital Enterprise Research Institute. All rights reserved.
    2. 2. Agenda Digital Enterprise Research Institute    Motivation Background Task Modelling    Capability Requirements   www.deri.ie Capabilities Taxonomy Capability Tracing Experiment Summary 2
    3. 3. Motivation: Heterogeneity Digital Enterprise Research Institute www.deri.ie 3
    4. 4. Motivation: Task Routing Digital Enterprise Research Institute  www.deri.ie Assigning heterogeneous tasks to heterogeneous workers TASK MODELLING Models Models Models TASK ROUTING WORKER PROFILING Matching Profiles Profiles Profiles Task↔Worker 4
    5. 5. Proposal: Performance Prediction Digital Enterprise Research Institute  www.deri.ie Predict performance of workers on new tasks based on the capabilities required for tasks and assign tasks accordingly TASK MODELLING Models Models Models Capability Requirements Approach TASK ROUTING WORKER PROFILING Matching Profiles Profiles Profiles Task↔Worker Performance Prediction Capability Tracing Model 5
    6. 6. Background: Micro tasks Digital Enterprise Research Institute  www.deri.ie When micro tasks are crowd sourced   Single person cannot do the task   Computers cannot do the task Work can be split into smaller tasks Some online microtask platforms 6
    7. 7. Background: Micro tasks Digital Enterprise Research Institute  www.deri.ie Most common tasks in Amazon Mechanical Turk (AMT) and CrowdFlower (CFL) 7
    8. 8. Background: Micro tasks Digital Enterprise Research Institute  www.deri.ie Example of information extraction task in AMT 8
    9. 9. Background: Micro tasks Digital Enterprise Research Institute  www.deri.ie Example of video transcription task in AMT 9
    10. 10. Task Modelling Digital Enterprise Research Institute www.deri.ie  Appropriate models are needed to compare and contrast micro tasks.  Capability Requirements approach  Capability is defined as the ability of humans to do things in terms of both the capacity and the opportunity.  Four types of capabilities – – – – Knowledge, Skill, Ability, Other characteristics (e.g. motivation, price, etc) 10
    11. 11. Capability Requirements Digital Enterprise Research Institute  www.deri.ie Taxonomies have be used to study human task performance, e.g.   Bloom’s taxonomy of classification of learning objectives   Fleishman’s taxonomy of human abilities O*NET-SOC taxonomy of occupational classification We are interested in taxonomy that  Describes tasks in terms of human capabilities  Helps in comparing tasks in terms of differences and similarities of capabilities 11
    12. 12. Capabilities Taxonomy Digital Enterprise Research Institute   www.deri.ie Based on Fleishman’s abilities taxonomy Selected abilities relevant to micro tasks  Comprehension (C): The ability to understand the meaning or importance of something  Bilingualism (B): The ability to speak and understand two languages  Writing (W): The ability or capacity to write text in a given language  Comparison (M): The ability or capacity to compare things based on some criteria  Judgment (J): The act or process of judging; the formation of an opinion after consideration  Perception (P): The ability or capacity to perceive items visually or phonetically  Identification (I): The process of recognizing something  Reasoning (R): The ability to draw conclusions from facts, evidence, relationships, etc. 12
    13. 13. Requirements of Micro Tasks Digital Enterprise Research Institute www.deri.ie 13
    14. 14. Capability Tracing Digital Enterprise Research Institute   www.deri.ie How to model worker’s capabilities? Capability tracing    Inspired by Knowledge Tracing* Estimates probability of a worker knowing a capability given worker’s responses to test tasks Worker Profile constrains  Set of binary variables representing capabilities  Probability estimates of each variable being in a state * A. T. Corbett and J. R. Anderson, “Knowledge tracing: Modeling the acquisition of procedural knowledge,” User Modeling and User-Adapted Interaction, vol. 4, no. 4, pp. 253–278, 1994. 14
    15. 15. Capability Tracing Digital Enterprise Research Institute  www.deri.ie Probabilistic network of a capability and four parameters of capability tracing model States of Capability Variable Not Learned p(T): Probability of transition between states p(T) Learned p(L) p(G) Values of Response Variable p(L): Probability of a worker learning to employ the capability p(S) p(G): Probability of guess Correct Incorrect 15 p(S): Probability of slip
    16. 16. Experiment Digital Enterprise Research Institute  www.deri.ie Objective    Solicit capability requirements of tasks from crowds Evaluation of capability tracing for performance prediction Three types of micro tasks with manually created ground truth data   Image comparison   Fact verification Information Extraction 37 crowd workers including  University students  Workers from Shorttask.com 16
    17. 17. Crowdsourcing Digital Enterprise Research Institute   www.deri.ie Custom web application for gathering data Example of fact verification task 17
    18. 18. Capability Requirements of Tasks Digital Enterprise Research Institute  www.deri.ie Objective 1: Solicit capability requirements of tasks from crowds (a) fact verification (b) image comparison 18 (c) information extraction
    19. 19. Crowd Performance Digital Enterprise Research Institute  www.deri.ie How the crowd performed on each type of task? Fact Verification task • 37 workers • Best workers perform with both precision and recall above 0.8 • More variation in recall means some workers were could not spot the incorrect facts • Ideally tasks should be assigned to workers that lie in the top-right quadrant of the plot 19
    20. 20. Crowd Performance Digital Enterprise Research Institute  www.deri.ie Image Comparison (20 workers) and Information Extraction (17 workers) 20
    21. 21. Performance Prediction Digital Enterprise Research Institute   www.deri.ie Objective 2: Evaluation of capability tracing for performance prediction Two phases  Build model with observation tasks  Predict performance on new tasks AC: Consider previous Accuracy as prediction of future performance CT: Capability Tracing 21
    22. 22. Summary Digital Enterprise Research Institute   Capabilities taxonomy is first steps towards modelling of micro tasks based on human factors Capability tracing is effective in predicting future performance    www.deri.ie Even across tasks if there are similar capabilities Predicted performance can be used to make right task routing decisions Future Work  Evaluate on more types of tasks  Evaluate capabilities such as domain knowledge and skills  Define standard tests for measuring worker capabilities 22
    23. 23. Further Reading Digital Enterprise Research Institute www.deri.ie 9th IEEE International Conference on Collaborative Computing: Networking, Applications and Worksharing Austin, Texas, United States October 20–23, 2013 U. Ul Hassan and E. Curry, “A Capability Requirements Approach for Predicting Worker Performance in Crowdsourcing,” in 9th IEEE International Conference on Collaborative Computing: Networking, Applications and Worksharing, 2013. http://deri.ie/users/umair-ul-hassan 23
    24. 24. Capability Tracing Digital Enterprise Research Institute www.deri.ie  Conditional probability of worker learning to employ capability p(Ln|On) is calculated  When evidence On is positive  When evidence On is negative 24
    25. 25. Capability Tracing Digital Enterprise Research Institute www.deri.ie  Probability of worker learning to employ capability  Performance of worker on next task 25

    ×