Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Copyright 2010 Digital Enterprise Research Institute. All rights reserved.Digital Enterprise Research Institute www.deri.i...
Digital Enterprise Research Institute www.deri.ieAgenda Paper Overview Motivation Human Computation Task Routing Chal...
Digital Enterprise Research Institute www.deri.iePaper Overview Motivation People have differing levels of expertise Ef...
Digital Enterprise Research Institute www.deri.ieHuman Computation Solve computationally hard problems with help of human...
Digital Enterprise Research Institute www.deri.ieHuman Computation5* Edith Law and Luis von Ahn, Human Computation - Core ...
Digital Enterprise Research Institute www.deri.ieTask Routing Pull Routing System provides an interface to support worke...
Digital Enterprise Research Institute www.deri.ieTask Routing Push Routing System has complete control over assignment o...
Digital Enterprise Research Institute www.deri.ieChallenges of Push Routing Workers have different domain knowledge and e...
Digital Enterprise Research Institute www.deri.ieRoutingProfilingKnowledgeProfileTasksPerformanceProfile3.TestTasks1.Conce...
Digital Enterprise Research Institute www.deri.ieUse case: Verification Tasks Data quality in DBpedia Verification of ne...
Digital Enterprise Research Institute www.deri.ieUse case: Verification Tasks11Update: Missing Valuedbpedia-owl:writer =db...
Digital Enterprise Research Institute www.deri.ieUse case: Verification Tasks Datasets based on filmsrelated entities fro...
Digital Enterprise Research Institute www.deri.ieProfiling13Conceptc1: Buddy films 0.6 0.2 0.2c2: Gang films 0.6 0.2 0.6c...
Digital Enterprise Research Institute www.deri.ieAssessment14
Digital Enterprise Research Institute www.deri.ieAssessment15
Digital Enterprise Research Institute www.deri.ieRouting16
Digital Enterprise Research Institute www.deri.ieKnowledge workers Volunteers having varyingknowledge about films Hollyw...
Digital Enterprise Research Institute www.deri.ieEvaluation Metrics Quality (for routed tasks during routing phase)– Res...
Digital Enterprise Research Institute www.deri.ieResults: Costs Combined assessment Filtering assessment tasks based on ...
Digital Enterprise Research Institute www.deri.ieResults: Quality of Routing Likelihood of response and accuracy of respo...
Digital Enterprise Research Institute www.deri.ieSummary Conclusion Effective push routing depends on worker expertise ...
Digital Enterprise Research Institute www.deri.ieFurther ReadingU. Ul Hassan, S. O’Riain, and E. Curry, “Effects of Expert...
Upcoming SlideShare
Loading in …5
×

Effects of Expertise Assessment on the Quality of Task Routing in Human Computation

683 views

Published on

https://www.insight-centre.org/content/effects-expertise-assessment-quality-task-routing-human-computation

Presented at SoHuman'12

Abstract:
Human computation systems are characterized by the use of human workers to solve computationally difficult problems. Expertise profiling involves assessment and representation of a worker’s expertise, in order to route human computation tasks to appropriate workers. This paper studies the relationship between the assessment workload on workers and the quality of task routing. Three expertise assessment approaches were compared with the help of a user study, using two different groups of human workers. The first approach requests workers to provide self-assessment of their knowledge. The second approach measures the knowledge of workers through their performance against tasks with known responses. We propose a third approach based on a combination of self-assessment and task-assessment. The results suggest that the self-assessment approach requires minimum assessment workload from workers during expertise profiling. By comparison, the task-assessment approach achieved the highest response rate and accuracy. The proposed approach requires less assessment workload, while achieving the response rate and accuracy similar to the task-assessment approach.

  • Be the first to comment

Effects of Expertise Assessment on the Quality of Task Routing in Human Computation

  1. 1. Copyright 2010 Digital Enterprise Research Institute. All rights reserved.Digital Enterprise Research Institute www.deri.ieEFFECTS OF EXPERTISE ASSESSMENT ON THEQUALITY OF TASK ROUTING IN HUMANCOMPUTATIONUmair ul Hassan, Sean O’Riain, Edward CurryDigital Enterprise Research InstituteNational University of Ireland, GalwayInternational Workshop on Social Media for Crowdsourcingand Human Computation - SoHuman’13, Paris, France
  2. 2. Digital Enterprise Research Institute www.deri.ieAgenda Paper Overview Motivation Human Computation Task Routing Challenges of Push Routing Experiment Use case Methodology Results Summary2
  3. 3. Digital Enterprise Research Institute www.deri.iePaper Overview Motivation People have differing levels of expertise Effective task routing requires expertise information Expertise profiling involves assessment Problem How to assess worker’s expertise for generating profiles? How to reduce costs of expertise assessment while attaining higherquality of task routing? Contribution Comparison of self-assessment and task-assessment approaches A hybrid approach, based on combination of self-assessment and taskassessment, for cost reduction3
  4. 4. Digital Enterprise Research Institute www.deri.ieHuman Computation Solve computationally hard problems with help of humans Algorithms control human workers Computation is carried out by Humans4* Barowy et al, “AutoMan: a platform for integrating human-based and digital computation,” OOPSLA ’12Define ComputeAlgorithmDeveloperWorkers
  5. 5. Digital Enterprise Research Institute www.deri.ieHuman Computation5* Edith Law and Luis von Ahn, Human Computation - Core Research Questions and State of the ArtInput OutputTask Routerbefore computationOutput Aggregationafter computationTask Designduring computationOur Focus
  6. 6. Digital Enterprise Research Institute www.deri.ieTask Routing Pull Routing System provides an interface to support workers Workers actively seek tasks and assign to themselves6WorkersTasks SelectResultAlgorithmSearch & Browse Interface* www.mtruk.comResult
  7. 7. Digital Enterprise Research Institute www.deri.ieTask Routing Push Routing System has complete control over assignment of tasks– Based on criteria such as expertise, cost, and latency Workers passively receive tasks7WorkersTasksAssignResultAssignAlgorithmTask Interface* www.mobileworks.comResult
  8. 8. Digital Enterprise Research Institute www.deri.ieChallenges of Push Routing Workers have different domain knowledge and expertise1. How to define the expertise requirements of a task? And howto model the expertise profile of a worker?2. How to profile the expertise of human workers, via suitableexpertise assessment methods with minimum cost?3. How to leverage the expertise profiles of workers for effectivelyrouting tasks , resulting in quality responses?8
  9. 9. Digital Enterprise Research Institute www.deri.ieRoutingProfilingKnowledgeProfileTasksPerformanceProfile3.TestTasks1.ConceptsRoutingModel5.NewTasks2.SelfAssessment4.TaskAssessment6.RoutedTasksWorkersTwo phase process Steps of push routing using worker profiles9Cost of assessmentfor profilingQuality of profiles forrouting
  10. 10. Digital Enterprise Research Institute www.deri.ieUse case: Verification Tasks Data quality in DBpedia Verification of new facts for DBpedia10Conceptrelated tothe task
  11. 11. Digital Enterprise Research Institute www.deri.ieUse case: Verification Tasks11Update: Missing Valuedbpedia-owl:writer =dbpedia:Akiva_GoldsmanSKOS Concepts:American_biographical_filmsFilms_set_in_the_1950sWorker ExpertiseSKOS Concepts:Films_set_in_the_1950s (Good)Films_about_psychiatry (Poor)American_drama_films (Fair)Data Quality AlgorithmWorkers & Expertise ModelEntity: A Beautiful MindSKOS Concepts:American_biographical_filmsFilms_set_in_the_1950sProperty & Values:dbpedia-owl:Work/runtime135.0dbpedia-owl:directordbpedia:Ron_Howarddbpedia-owl:producerdbpedia:Ron_Howarddbpedia:Brian_Grazedbpedia-owl:starringdbpedia:Ed_Harrisdbpedia:Russell_CroweSource DataTask: Confirm Missing ValueDid Akiva Goldsman wrote themovie "A Beautiful Mind"?SKOS Concepts:American_biographical_filmsFilms_set_in_the_1950sTask RoutingMatchAmerican_biographical_filmsAmerican_drama_films (Fair)Task ModelRouting Model* SKOS = Simple KnowledgeOrganization System
  12. 12. Digital Enterprise Research Institute www.deri.ieUse case: Verification Tasks Datasets based on filmsrelated entities from hollywoodand bollywood Distribution of tasks againstnumber of concepts per task12Dataset CharacteristicsMoviesDatasetActorsDatasetTotal entities 724 14Total concepts 42 14Total tasks 230 120Avg. tasks per concept 9 8.6Avg. concepts per task 1.64 1 0204060801001201401601 2 3 4 5No.ofTasksNo. of Concepts per Task
  13. 13. Digital Enterprise Research Institute www.deri.ieProfiling13Conceptc1: Buddy films 0.6 0.2 0.2c2: Gang films 0.6 0.2 0.6c3: Horror films 0.8 0.4 0.4c4: Comedy films 0.8 0.6 0.6
  14. 14. Digital Enterprise Research Institute www.deri.ieAssessment14
  15. 15. Digital Enterprise Research Institute www.deri.ieAssessment15
  16. 16. Digital Enterprise Research Institute www.deri.ieRouting16
  17. 17. Digital Enterprise Research Institute www.deri.ieKnowledge workers Volunteers having varyingknowledge about films Hollywood vs. Bollywood Survey before and afterparticipation17MoviesDatasetActorsDatasetNo. of knowledgeworkers (volunteers) 11 26No. of knowledgeconcepts 42 14No. of test tasks(profiling phase) 100 56No. of new tasks(routing phase) 130 64 012345678910Interest Knowledge Expertise ConfidenceAverageLevelBeforeAfterOnly significantdifference
  18. 18. Digital Enterprise Research Institute www.deri.ieEvaluation Metrics Quality (for routed tasks during routing phase)– Response Rate: percentage of routed tasks with agree or disagreeresponses– Accuracy: percentage of routed tasks with correct responses Cost (for assessments during profiling phase)– Workload: number of decisions for self-rating of conceptualknowledge or responding to test task Hypothesis The quality of CA strategy approaches the quality of TAstrategy during routing phase, while requiringcomparatively less assessment cost during profiling phase.18
  19. 19. Digital Enterprise Research Institute www.deri.ieResults: Costs Combined assessment Filtering assessment tasks based on highly self-rated conceptsreduces assessment cost190%20%40%60%80%100%120%140%160%RND SA TA CA CA (P+) CA (F+) CA (G+) CA (Ex)%wokrloadcomparedtoTAMovies Dataset Actors DatasetFor examplesfilter tasks withconcepts ofGood or higherself-rating
  20. 20. Digital Enterprise Research Institute www.deri.ieResults: Quality of Routing Likelihood of response and accuracy of responseremains near maximum during routing stage200%20%40%60%80%100%RND SA TA CA CA(P+)CA(F+)CA(G+)CA(Ex)%AccuracyMovies Dataset Actors Dataset0%20%40%60%80%100%RND SA TA CA CA(P+)CA(F+)CA(G+)CA(Ex)%ResponseRateMovies Dataset Actors Dataset
  21. 21. Digital Enterprise Research Institute www.deri.ieSummary Conclusion Effective push routing depends on worker expertise Concepts are effective for expertise profiling Combining task-assessment with self-assessment is effective inreducing assessment cost Future Directions Task routing under constraints– Cost, Latency, Expertise, Utility Complex workflows in data quality management21
  22. 22. Digital Enterprise Research Institute www.deri.ieFurther ReadingU. Ul Hassan, S. O’Riain, and E. Curry, “Effects of Expertise Assessment on theQuality of Task Routing in Human Computation,” in 2nd International Workshop onSocial Media for Crowdsourcing and Human Computation, 2013.http://www.deri.ie/about/team/member/umair_ul_hassan/222nd International Workshop on Social Media for Crowdsourcing andHuman ComputationParis, 1 May 2013

×