Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Measuring the Effectiveness of Gamesourcing Expert Oil Painting Annotations

1,674 views

Published on

Presentation at ECIR 2014 in Amsterdam

Published in: Science, Business, Education
  • Be the first to comment

Measuring the Effectiveness of Gamesourcing Expert Oil Painting Annotations

  1. 1. Myriam C. Traub, Jacco van Ossenbruggen, Jiyin He, Lynda Hardman Centrum Wiskunde & Informatica Measuring the Effectiveness of Gamesourcing Expert Oil Painting Annotations
  2. 2. Research Problem • Expert tasks are hard to crowdsource • too complex for non- experts • experts difficult to target and engage • Approach: • simplify the task by suggesting potential answers • help non-experts to learn by providing expert feedback 2
  3. 3. Chosen Task Subject type annotation of paintings • about 100 different subject types in Art & Architecture Thesaurus • subtle differences Marine? Seascape? History painting? Kacho?3
  4. 4. 4 http://sealincmedia.project.cwi.nl/artgame/
  5. 5. Research Questions Non-Experts: • How well do they compare with experts, both individually and as a crowd? • Do they memorize the correct subject type? • Can they generalize what they have learned to new paintings? Task: • How does the presence or absence of a correct answer influence a user's performance? Data: • Are there features of images or subject types that can predict high or low agreement? Game / UI / Backend system • None
  6. 6. Experimental Setup - Data • Subset of Steve Tagger data set: 125 images of paintings with … • … subject type annotations by 4 experts from Rijksmuseum Amsterdam: 168 expert annotations • Limitation: only one expert per painting 6
  7. 7. Experimental Setup - Query Image Selection • first 10: random, no repetition • after 10: random, but repetition probability of 50%
  8. 8. Experimental Setup - Candidate Selection • total 5 candidates + „other“: • Perfect: 1 correct; 
 Imperfect: 1 correct (75%) or a related but incorrect (25%), • at most 3 related but incorrect, • at least 1 incorrect 8
  9. 9. Users - Performance •Perfect: 21 users, 5640 judgements (range: 26 -1250), correct: 12% - 83% •Imperfect: 17 users, 1929 judgements (range: 19 - 350), correct: 22% - 54% 9 020406080100 users percentageofcorrectannotations perfect % imperfect % random behavior
  10. 10. Users - Performance 10 perfect # imperfect # 200 400 600 800 1000 1200 numberofannotations(bars) 020406080100 users percentageofcorrectannotations(dots) perfect % imperfect % •Perfect: correlation of .71 and p < 0.001 •Imperfect: correlation of .32 and p=0.215
  11. 11. perfect 11 48 6 4 8 5 48 4 1 26 6 5 5 5 6 26 38 164 2 27 1 12 39 35 5 1 129 51 34 3 1 1 11 49 2 1 1 29 13 47 1 1 1 3 1 107 3 1 2 2 1 1 286 1 8 16 1 1 2 6 105 2 86 1 2 20 2 203 3 12 1 3 2 53 7 1 1 2 9 6 11 1 1 27 5 1 1 3 1 2 3 846 5 23 8 4 58 1 16 3 1 2 95 2 1 2 77 32 15 15 1 1 2 30 980 4 16 1 27 10 5 9 1 86 6 2 1 9 2 3 6 1 4 20 2 3 136 3 1 6 18 9 3 2 355 18 2 28 4 13 2 5 2 1 86 1 17 6 132 29 86 1 2 3 45 2 21 12 18 1 13 1 5 3 164 1 14 2 7 1 figu land full port alle half genr hist kach city seas stil anim town flow mari maes othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes Non−Experts Experts 0 25 50 75 Percent baseline condition − individual annotations 2 1 7 2 1 1 2 1 3 3 8 3 2 5 3 1 30 1 3 1 1 1 37 1 4 8 12 1 6 7 1 1 8 figu land full port alle half genr hist kach city seas stil anim town flow mari maes othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes Non−Experts Experts 0 25 50 75 100 Percent baseline condition − aggregated annotations 291 63 8 7 5 52 10 9 6 34 4 29 14 8 13 65 7 3 1 59 2 20 10 9 2 7 2 1 8 3 4 2 13 2 9 5 32 8 2 1 1 60 1 1 1 1 2 6 12 1 1 10 35 1 8 2 2 1 1 1 10 4 1 1 3 3 4 1 6 1 7 5 1 1 1 176 20 1 3 3 30 6 1 6 166 3 7 1 6 1 7 18 6 38 1 4 1 1 3 4 6 3 1 10 4 1 89 1 1 6 1 2 1 62 3 1 7 23 10 4 1 1 1 3 26 3 1 25 2 9 2 5 4 5 31 25 2 1 4 2 othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes Non−Experts Experts 0 25 50 75 Percent imperfect condition − individual annotations imperfect 96 11 4 1 6 3 1 1 6 1 3 1 1 3 9 3 7 1 2 1 2 1 3 2 6 1 2 1 4 2 1 1 1 23 2 3 19 3 3 1 1 12 1 11 5 1 1 5 1 1 4 6othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes Non−Experts Experts 0 25 50 75 100 Percent imperfect condition − aggregated annotations aggregated individual
  12. 12. 48 6 4 8 5 48 4 1 26 6 5 5 5 6 26 38 164 2 27 1 12 39 35 5 1 129 51 34 3 1 1 11 49 2 1 1 29 13 47 1 1 1 3 1 107 3 1 2 2 1 1 286 1 8 16 1 1 2 6 105 2 86 1 2 20 2 203 3 12 1 3 2 53 7 1 1 2 9 6 11 1 1 27 5 1 1 3 1 2 3 846 5 23 8 4 58 1 16 3 1 2 95 2 1 2 77 32 15 15 1 1 2 30 980 4 16 1 27 10 5 9 1 86 6 2 1 9 2 3 6 1 4 20 2 3 136 3 1 6 18 9 3 2 355 18 2 28 4 13 2 5 2 1 86 1 17 6 132 29 86 1 2 3 45 2 21 12 18 1 13 1 5 3 164 1 14 2 7 1 figu land full port alle half genr hist kach city seas stil anim town flow mari maes othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes Non−Experts Experts 0 25 50 75 Percent baseline condition − individual annotations 2 1 7 2 1 1 2 1 3 3 8 3 2 5 3 1 30 1 3 1 1 1 37 1 4 8 12 1 6 7 1 1 8 figu land full port alle half genr hist kach city seas stil anim town flow mari maes othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes Non−Experts Experts 0 25 50 75 100 Percent baseline condition − aggregated annotations 291 63 8 7 5 52 10 9 6 34 4 29 14 8 13 65 7 3 1 59 2 20 10 9 2 7 2 1 8 3 4 2 13 2 9 5 32 8 2 1 1 60 1 1 1 1 2 6 12 1 1 10 35 1 8 2 2 1 1 1 10 4 1 1 3 3 4 1 6 1 7 5 1 1 1 176 20 1 3 3 30 6 1 6 166 3 7 1 6 1 7 18 6 38 1 4 1 1 3 4 6 3 1 10 4 1 89 1 1 6 1 2 1 62 3 1 7 23 10 4 1 1 1 3 26 3 1 25 2 9 2 5 4 5 31 25 2 1 4 2 othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes Non−Experts Experts 0 25 50 75 Percent imperfect condition − individual annotations 96 11 4 1 6 3 1 1 6 1 3 1 1 3 9 3 7 1 2 1 2 1 3 2 6 1 2 1 4 2 1 1 1 23 2 3 19 3 3 1 1 12 1 11 5 1 1 5 1 1 4 6othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes Non−Experts Experts 0 25 50 75 100 Percent imperfect condition − aggregated annotations Perfect condition, individual: substantial agreement (κ = 0.65) 12
  13. 13. 2 1 7 2 1 1 2 1 3 3 8 3 2 5 3 1 30 1 3 1 1 1 37 1 4 8 12 1 6 7 1 1 8 figu land full port alle half genr hist kach city seas stil anim town flow mari maes othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes Non−Experts Experts 0 25 50 75 100 Percent baseline condition − aggregated annotations 48 6 4 8 5 48 4 1 26 6 5 5 5 6 26 38 164 2 27 1 12 39 35 5 1 129 51 34 3 1 1 11 49 2 1 1 29 13 47 1 1 1 3 1 107 3 1 2 2 1 1 286 1 8 16 1 1 2 6 105 2 86 1 2 20 2 203 3 12 1 3 2 53 7 1 1 2 9 6 11 1 1 27 5 1 1 3 1 2 3 846 5 23 8 4 58 1 16 3 1 2 95 2 1 2 77 32 15 15 1 1 2 30 980 4 16 1 27 10 5 9 1 86 6 2 1 9 2 3 6 1 4 20 2 3 136 3 1 6 18 9 3 2 355 18 2 28 4 13 2 5 2 1 86 1 17 6 132 29 86 1 2 3 45 2 21 12 18 1 13 1 5 3 164 1 14 2 7 1 figu land full port alle half genr hist kach city seas stil anim town flow mari maes othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes Non−Experts Experts 0 25 50 75 Percent baseline condition − individual annotations 291 63 8 7 5 52 10 9 6 34 4 29 14 8 13 65 7 3 1 59 2 20 10 9 2 7 2 1 8 3 4 2 13 2 9 5 32 8 2 1 1 60 1 1 1 1 2 6 12 1 1 10 35 1 8 2 2 1 1 1 10 4 1 1 3 3 4 1 6 1 7 5 1 1 1 176 20 1 3 3 30 6 1 6 166 3 7 1 6 1 7 18 6 38 1 4 1 1 3 4 6 3 1 10 4 1 89 1 1 6 1 2 1 62 3 1 7 23 10 4 1 1 1 3 26 3 1 25 2 9 2 5 4 5 31 25 2 1 4 2 othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes Non−Experts Experts 0 25 50 75 Percent imperfect condition − individual annotations 96 11 4 1 6 3 1 1 6 1 3 1 1 3 9 3 7 1 2 1 2 1 3 2 6 1 2 1 4 2 1 1 1 23 2 3 19 3 3 1 1 12 1 11 5 1 1 5 1 1 4 6othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes Non−Experts Experts 0 25 50 75 100 Percent imperfect condition − aggregated annotations Perfect condition, aggregated: almost perfect agreement (κ = 0.87) 13
  14. 14. “… dramatic bird’s eye view of Broadway and Wall Street…” Church Street El, 1920 by Charles Sheeler 14 Townscape Cityscapes
  15. 15. 291 63 8 7 5 52 10 9 6 34 4 29 14 8 13 65 7 3 1 59 2 20 10 9 2 7 2 1 8 3 4 2 13 2 9 5 32 8 2 1 1 60 1 1 1 1 2 6 12 1 1 10 35 1 8 2 2 1 1 1 10 4 1 1 3 3 4 1 6 1 7 5 1 1 1 176 20 1 3 3 30 6 1 6 166 3 7 1 6 1 7 18 6 38 1 4 1 1 3 4 6 3 1 10 4 1 89 1 1 6 1 2 1 62 3 1 7 23 10 4 1 1 1 3 26 3 1 25 2 9 2 5 4 5 31 25 2 1 4 2 othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes Non−Experts Experts 0 25 50 75 Percent imperfect condition − individual annotations 48 6 4 8 5 48 4 1 26 6 5 5 5 6 26 38 164 2 27 1 12 39 35 5 1 129 51 34 3 1 1 11 49 2 1 1 29 13 47 1 1 1 3 1 107 3 1 2 2 1 1 286 1 8 16 1 1 2 6 105 2 86 1 2 20 2 203 3 12 1 3 2 53 7 1 1 2 9 6 11 1 1 27 5 1 1 3 1 2 3 846 5 23 8 4 58 1 16 3 1 2 95 2 1 2 77 32 15 15 1 1 2 30 980 4 16 1 27 10 5 9 1 86 6 2 1 9 2 3 6 1 4 20 2 3 136 3 1 6 18 9 3 2 355 18 2 28 4 13 2 5 2 1 86 1 17 6 132 29 86 1 2 3 45 2 21 12 18 1 13 1 5 3 164 1 14 2 7 1 figu land full port alle half genr hist kach city seas stil anim town flow mari maes othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes Non−Experts Experts 0 25 50 75 Percent baseline condition − individual annotations 2 1 7 2 1 1 2 1 3 3 8 3 2 5 3 1 30 1 3 1 1 1 37 1 4 8 12 1 6 7 1 1 8 figu land full port alle half genr hist kach city seas stil anim town flow mari maes othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes Non−Experts Experts 0 25 50 75 100 Percent baseline condition − aggregated annotations 96 11 4 1 6 3 1 1 6 1 3 1 1 3 9 3 7 1 2 1 2 1 3 2 6 1 2 1 4 2 1 1 1 23 2 3 19 3 3 1 1 12 1 11 5 1 1 5 1 1 4 6othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes Non−Experts Experts 0 25 50 75 100 Percent imperfect condition − aggregated annotations Imperfect condition, individual: moderate agreement (κ = 0.47) 15
  16. 16. AllegoryOther Lucretia Calypso
  17. 17. 96 11 4 1 6 3 1 1 6 1 3 1 1 3 9 3 7 1 2 1 2 1 3 2 6 1 2 1 4 2 1 1 1 23 2 3 19 3 3 1 1 12 1 11 5 1 1 5 1 1 4 6othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes Non−Experts Experts 0 25 50 75 100 Percent imperfect condition − aggregated annotations 48 6 4 8 5 48 4 1 26 6 5 5 5 6 26 38 164 2 27 1 12 39 35 5 1 129 51 34 3 1 1 11 49 2 1 1 29 13 47 1 1 1 3 1 107 3 1 2 2 1 1 286 1 8 16 1 1 2 6 105 2 86 1 2 20 2 203 3 12 1 3 2 53 7 1 1 2 9 6 11 1 1 27 5 1 1 3 1 2 3 846 5 23 8 4 58 1 16 3 1 2 95 2 1 2 77 32 15 15 1 1 2 30 980 4 16 1 27 10 5 9 1 86 6 2 1 9 2 3 6 1 4 20 2 3 136 3 1 6 18 9 3 2 355 18 2 28 4 13 2 5 2 1 86 1 17 6 132 29 86 1 2 3 45 2 21 12 18 1 13 1 5 3 164 1 14 2 7 1 figu land full port alle half genr hist kach city seas stil anim town flow mari maes othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes Non−Experts Experts 0 25 50 75 Percent baseline condition − individual annotations 2 1 7 2 1 1 2 1 3 3 8 3 2 5 3 1 30 1 3 1 1 1 37 1 4 8 12 1 6 7 1 1 8 figu land full port alle half genr hist kach city seas stil anim town flow mari maes othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes Non−Experts Experts 0 25 50 75 100 Percent baseline condition − aggregated annotations 291 63 8 7 5 52 10 9 6 34 4 29 14 8 13 65 7 3 1 59 2 20 10 9 2 7 2 1 8 3 4 2 13 2 9 5 32 8 2 1 1 60 1 1 1 1 2 6 12 1 1 10 35 1 8 2 2 1 1 1 10 4 1 1 3 3 4 1 6 1 7 5 1 1 1 176 20 1 3 3 30 6 1 6 166 3 7 1 6 1 7 18 6 38 1 4 1 1 3 4 6 3 1 10 4 1 89 1 1 6 1 2 1 62 3 1 7 23 10 4 1 1 1 3 26 3 1 25 2 9 2 5 4 5 31 25 2 1 4 2 othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes Non−Experts Experts 0 25 50 75 Percent imperfect condition − individual annotations Imperfect condition, aggregated: moderate agreement (κ = 0.55) 17
  18. 18. 18 Cityscapes Other Genre Landscape
  19. 19. Learning Memorizing 19 2 4 6 8 10 020406080100 number of repetitions percentageofcorrectannotations random behavior
  20. 20. Learning Memorizing 20 2 4 6 8 10 020406080100 number of repetitions percentageofcorrectannotations perfect % imperfect %
  21. 21. Learning Generalizing 21 sequence number of new images percentageofcorrectannotations [1,20] (60,80] (120,140] (200,220] (280,300] (360,380] 020406080100
  22. 22. Learning Generalizing 22 sequence number of new images percentageofcorrectannotations perfect % imperfect % [1,20] (60,80] (120,140] (200,220] (280,300] (360,380] 020406080100
  23. 23. Conclusions Agreement between experts and non-experts: • substantial in perfect, moderate in imperfect condition • aggregation reduces deviations Strong disagreement may indicate: • need for additional metadata • incorrect or incomplete expert judgements Learning: • users memorize and generalize • users need a training phase on high quality data ! Results in line with He, J., van Ossenbruggen, J., de Vries, A.P.: Do you need experts in the crowd?: a case study in image annotation for marine biology. OAIR 2013
  24. 24. Future Work - scarce + high quality data - need training data + high quantity data - lack expertise + high quality when trained & assisted 24
  25. 25. Thank you for your attention! Myriam C. Traub myriam.traub@cwi.nl ! ! http://sealincmedia.project.cwi.nl/artgame/ On the beach of Trouville Eugène Boudin

×