Copyright 2010 Digital Enterprise Research Institute. All rights reserved.Digital Enterprise Research Institute www.deri.i...
Digital Enterprise Research Institute www.deri.ieAgenda Paper Overview Motivation Human Computation Task Routing Chal...
Digital Enterprise Research Institute www.deri.iePaper Overview Motivation People have differing levels of expertise Ef...
Digital Enterprise Research Institute www.deri.ieHuman Computation Solve computationally hard problems with help of human...
Digital Enterprise Research Institute www.deri.ieHuman Computation5* Edith Law and Luis von Ahn, Human Computation - Core ...
Digital Enterprise Research Institute www.deri.ieTask Routing Pull Routing System provides an interface to support worke...
Digital Enterprise Research Institute www.deri.ieTask Routing Push Routing System has complete control over assignment o...
Digital Enterprise Research Institute www.deri.ieChallenges of Push Routing Workers have different domain knowledge and e...
Digital Enterprise Research Institute www.deri.ieRoutingProfilingKnowledgeProfileTasksPerformanceProfile3.TestTasks1.Conce...
Digital Enterprise Research Institute www.deri.ieUse case: Verification Tasks Data quality in DBpedia Verification of ne...
Digital Enterprise Research Institute www.deri.ieUse case: Verification Tasks11Update: Missing Valuedbpedia-owl:writer =db...
Digital Enterprise Research Institute www.deri.ieUse case: Verification Tasks Datasets based on filmsrelated entities fro...
Digital Enterprise Research Institute www.deri.ieProfiling13Conceptc1: Buddy films 0.6 0.2 0.2c2: Gang films 0.6 0.2 0.6c...
Digital Enterprise Research Institute www.deri.ieAssessment14
Digital Enterprise Research Institute www.deri.ieAssessment15
Digital Enterprise Research Institute www.deri.ieRouting16
Digital Enterprise Research Institute www.deri.ieKnowledge workers Volunteers having varyingknowledge about films Hollyw...
Digital Enterprise Research Institute www.deri.ieEvaluation Metrics Quality (for routed tasks during routing phase)– Res...
Digital Enterprise Research Institute www.deri.ieResults: Costs Combined assessment Filtering assessment tasks based on ...
Digital Enterprise Research Institute www.deri.ieResults: Quality of Routing Likelihood of response and accuracy of respo...
Digital Enterprise Research Institute www.deri.ieSummary Conclusion Effective push routing depends on worker expertise ...
Digital Enterprise Research Institute www.deri.ieFurther ReadingU. Ul Hassan, S. O’Riain, and E. Curry, “Effects of Expert...
Upcoming SlideShare
Loading in …5
×

Effects of Expertise Assessment on the Quality of Task Routing in Human Computation

483
-1

Published on

Presented at SoHuman'12

Published in: Technology, Health & Medicine
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
483
On Slideshare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
1
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Other sources of expertise information such as Social network, publications, etc. are part for future work.
  • Manually created ground truth for the tasks (test and new)
  • Effects of Expertise Assessment on the Quality of Task Routing in Human Computation

    1. 1. Copyright 2010 Digital Enterprise Research Institute. All rights reserved.Digital Enterprise Research Institute www.deri.ieEFFECTS OF EXPERTISE ASSESSMENT ON THEQUALITY OF TASK ROUTING IN HUMANCOMPUTATIONUmair ul Hassan, Sean O’Riain, Edward CurryDigital Enterprise Research InstituteNational University of Ireland, GalwayInternational Workshop on Social Media for Crowdsourcingand Human Computation - SoHuman’13, Paris, France
    2. 2. Digital Enterprise Research Institute www.deri.ieAgenda Paper Overview Motivation Human Computation Task Routing Challenges of Push Routing Experiment Use case Methodology Results Summary2
    3. 3. Digital Enterprise Research Institute www.deri.iePaper Overview Motivation People have differing levels of expertise Effective task routing requires expertise information Expertise profiling involves assessment Problem How to assess worker’s expertise for generating profiles? How to reduce costs of expertise assessment while attaining higherquality of task routing? Contribution Comparison of self-assessment and task-assessment approaches A hybrid approach, based on combination of self-assessment and taskassessment, for cost reduction3
    4. 4. Digital Enterprise Research Institute www.deri.ieHuman Computation Solve computationally hard problems with help of humans Algorithms control human workers Computation is carried out by Humans4* Barowy et al, “AutoMan: a platform for integrating human-based and digital computation,” OOPSLA ’12Define ComputeAlgorithmDeveloperWorkers
    5. 5. Digital Enterprise Research Institute www.deri.ieHuman Computation5* Edith Law and Luis von Ahn, Human Computation - Core Research Questions and State of the ArtInput OutputTask Routerbefore computationOutput Aggregationafter computationTask Designduring computationOur Focus
    6. 6. Digital Enterprise Research Institute www.deri.ieTask Routing Pull Routing System provides an interface to support workers Workers actively seek tasks and assign to themselves6WorkersTasks SelectResultAlgorithmSearch & Browse Interface* www.mtruk.comResult
    7. 7. Digital Enterprise Research Institute www.deri.ieTask Routing Push Routing System has complete control over assignment of tasks– Based on criteria such as expertise, cost, and latency Workers passively receive tasks7WorkersTasksAssignResultAssignAlgorithmTask Interface* www.mobileworks.comResult
    8. 8. Digital Enterprise Research Institute www.deri.ieChallenges of Push Routing Workers have different domain knowledge and expertise1. How to define the expertise requirements of a task? And howto model the expertise profile of a worker?2. How to profile the expertise of human workers, via suitableexpertise assessment methods with minimum cost?3. How to leverage the expertise profiles of workers for effectivelyrouting tasks , resulting in quality responses?8
    9. 9. Digital Enterprise Research Institute www.deri.ieRoutingProfilingKnowledgeProfileTasksPerformanceProfile3.TestTasks1.ConceptsRoutingModel5.NewTasks2.SelfAssessment4.TaskAssessment6.RoutedTasksWorkersTwo phase process Steps of push routing using worker profiles9Cost of assessmentfor profilingQuality of profiles forrouting
    10. 10. Digital Enterprise Research Institute www.deri.ieUse case: Verification Tasks Data quality in DBpedia Verification of new facts for DBpedia10Conceptrelated tothe task
    11. 11. Digital Enterprise Research Institute www.deri.ieUse case: Verification Tasks11Update: Missing Valuedbpedia-owl:writer =dbpedia:Akiva_GoldsmanSKOS Concepts:American_biographical_filmsFilms_set_in_the_1950sWorker ExpertiseSKOS Concepts:Films_set_in_the_1950s (Good)Films_about_psychiatry (Poor)American_drama_films (Fair)Data Quality AlgorithmWorkers & Expertise ModelEntity: A Beautiful MindSKOS Concepts:American_biographical_filmsFilms_set_in_the_1950sProperty & Values:dbpedia-owl:Work/runtime135.0dbpedia-owl:directordbpedia:Ron_Howarddbpedia-owl:producerdbpedia:Ron_Howarddbpedia:Brian_Grazedbpedia-owl:starringdbpedia:Ed_Harrisdbpedia:Russell_CroweSource DataTask: Confirm Missing ValueDid Akiva Goldsman wrote themovie "A Beautiful Mind"?SKOS Concepts:American_biographical_filmsFilms_set_in_the_1950sTask RoutingMatchAmerican_biographical_filmsAmerican_drama_films (Fair)Task ModelRouting Model* SKOS = Simple KnowledgeOrganization System
    12. 12. Digital Enterprise Research Institute www.deri.ieUse case: Verification Tasks Datasets based on filmsrelated entities from hollywoodand bollywood Distribution of tasks againstnumber of concepts per task12Dataset CharacteristicsMoviesDatasetActorsDatasetTotal entities 724 14Total concepts 42 14Total tasks 230 120Avg. tasks per concept 9 8.6Avg. concepts per task 1.64 1 0204060801001201401601 2 3 4 5No.ofTasksNo. of Concepts per Task
    13. 13. Digital Enterprise Research Institute www.deri.ieProfiling13Conceptc1: Buddy films 0.6 0.2 0.2c2: Gang films 0.6 0.2 0.6c3: Horror films 0.8 0.4 0.4c4: Comedy films 0.8 0.6 0.6
    14. 14. Digital Enterprise Research Institute www.deri.ieAssessment14
    15. 15. Digital Enterprise Research Institute www.deri.ieAssessment15
    16. 16. Digital Enterprise Research Institute www.deri.ieRouting16
    17. 17. Digital Enterprise Research Institute www.deri.ieKnowledge workers Volunteers having varyingknowledge about films Hollywood vs. Bollywood Survey before and afterparticipation17MoviesDatasetActorsDatasetNo. of knowledgeworkers (volunteers) 11 26No. of knowledgeconcepts 42 14No. of test tasks(profiling phase) 100 56No. of new tasks(routing phase) 130 64 012345678910Interest Knowledge Expertise ConfidenceAverageLevelBeforeAfterOnly significantdifference
    18. 18. Digital Enterprise Research Institute www.deri.ieEvaluation Metrics Quality (for routed tasks during routing phase)– Response Rate: percentage of routed tasks with agree or disagreeresponses– Accuracy: percentage of routed tasks with correct responses Cost (for assessments during profiling phase)– Workload: number of decisions for self-rating of conceptualknowledge or responding to test task Hypothesis The quality of CA strategy approaches the quality of TAstrategy during routing phase, while requiringcomparatively less assessment cost during profiling phase.18
    19. 19. Digital Enterprise Research Institute www.deri.ieResults: Costs Combined assessment Filtering assessment tasks based on highly self-rated conceptsreduces assessment cost190%20%40%60%80%100%120%140%160%RND SA TA CA CA (P+) CA (F+) CA (G+) CA (Ex)%wokrloadcomparedtoTAMovies Dataset Actors DatasetFor examplesfilter tasks withconcepts ofGood or higherself-rating
    20. 20. Digital Enterprise Research Institute www.deri.ieResults: Quality of Routing Likelihood of response and accuracy of responseremains near maximum during routing stage200%20%40%60%80%100%RND SA TA CA CA(P+)CA(F+)CA(G+)CA(Ex)%AccuracyMovies Dataset Actors Dataset0%20%40%60%80%100%RND SA TA CA CA(P+)CA(F+)CA(G+)CA(Ex)%ResponseRateMovies Dataset Actors Dataset
    21. 21. Digital Enterprise Research Institute www.deri.ieSummary Conclusion Effective push routing depends on worker expertise Concepts are effective for expertise profiling Combining task-assessment with self-assessment is effective inreducing assessment cost Future Directions Task routing under constraints– Cost, Latency, Expertise, Utility Complex workflows in data quality management21
    22. 22. Digital Enterprise Research Institute www.deri.ieFurther ReadingU. Ul Hassan, S. O’Riain, and E. Curry, “Effects of Expertise Assessment on theQuality of Task Routing in Human Computation,” in 2nd International Workshop onSocial Media for Crowdsourcing and Human Computation, 2013.http://www.deri.ie/about/team/member/umair_ul_hassan/222nd International Workshop on Social Media for Crowdsourcing andHuman ComputationParis, 1 May 2013
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×