Presentation at ECIR 2014 in Amsterdam

1. Myriam C. Traub, Jacco van Ossenbruggen, Jiyin He, Lynda Hardman
Centrum Wiskunde & Informatica
Measuring the Effectiveness of
Gamesourcing Expert Oil Painting Annotations

2. Research Problem
• Expert tasks are hard to
crowdsource
• too complex for non-
experts
• experts difficult to target
and engage
• Approach:
• simplify the task by
suggesting potential
answers
• help non-experts to learn by
providing expert feedback
2

3. Chosen Task
Subject type annotation of
paintings
• about 100 different
subject types in Art &
Architecture Thesaurus
• subtle differences
Marine?
Seascape?
History painting?
Kacho?3

4. 4
http://sealincmedia.project.cwi.nl/artgame/

5. Research Questions
Non-Experts:
• How well do they compare with experts, both individually and as a crowd?
• Do they memorize the correct subject type?
• Can they generalize what they have learned to new paintings?
Task:
• How does the presence or absence of a correct answer influence a user's
performance?
Data:
• Are there features of images or subject types that can predict high or low
agreement?
Game / UI / Backend system
• None

6. Experimental Setup - Data
• Subset of Steve Tagger data set: 125 images of paintings with …
• … subject type annotations by 4 experts from Rijksmuseum
Amsterdam: 168 expert annotations
• Limitation: only one expert per painting
6

7. Experimental Setup - Query Image Selection
• first 10: random, no repetition
• after 10: random, but repetition probability of 50%

8. Experimental Setup - Candidate Selection
• total 5 candidates + „other“:
• Perfect: 1 correct;
Imperfect: 1 correct (75%) or a related but incorrect (25%),
• at most 3 related but incorrect,
• at least 1 incorrect
8

9. Users - Performance
•Perfect: 21 users, 5640 judgements (range: 26 -1250), correct: 12% - 83%
•Imperfect: 17 users, 1929 judgements (range: 19 - 350), correct: 22% - 54%
9
020406080100
users
percentageofcorrectannotations
perfect %
imperfect %
random behavior

10. Users - Performance
10
perfect #
imperfect #
200
400
600
800
1000
1200
numberofannotations(bars)
020406080100
users
percentageofcorrectannotations(dots)
perfect %
imperfect %
•Perfect: correlation of .71 and p < 0.001
•Imperfect: correlation of .32 and p=0.215

11. perfect
11
48
6
4
8
5
48
4
1
26
6
5
5
5
6
26
38
164
2
27
1
12
39
35
5
1
129
51
34
3
1
1
11
49
2
1
1
29
13
47
1
1
1
3
1
107
3
1
2
2
1
1
286
1
8
16
1
1
2
6
105
2
86
1
2
20
2
203
3
12
1
3
2
53
7
1
1
2
9
6
11
1
1
27
5
1
1
3
1
2
3
846
5
23
8
4
58
1
16
3
1
2
95
2
1
2
77
32
15
15
1
1
2
30
980
4
16
1
27
10
5
9
1
86
6
2
1
9
2
3
6
1
4
20
2
3
136
3
1
6
18
9
3
2
355
18
2
28
4
13
2
5
2
1
86
1
17
6
132
29
86
1
2
3
45
2
21
12
18
1
13
1
5
3
164
1
14
2
7
1
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes
Non−Experts
Experts
0
25
50
75
Percent
baseline condition − individual annotations
2
1
7
2
1
1
2
1
3
3
8
3
2
5
3
1
30
1
3
1
1
1
37
1
4
8
12
1
6
7
1
1
8
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes
Non−Experts
Experts
0
25
50
75
100
Percent
baseline condition − aggregated annotations
291
63
8
7
5
52
10
9
6
34
4
29
14
8
13
65
7
3
1
59
2
20
10
9
2
7
2
1
8
3
4
2
13
2
9
5
32
8
2
1
1
60
1
1
1
1
2 6
12
1
1
10
35
1
8
2
2
1
1
1
10
4
1
1
3
3
4
1
6
1
7
5
1
1
1
176
20
1
3
3
30
6
1
6
166
3
7
1
6
1
7
18
6
38
1
4
1
1
3
4
6
3
1
10
4
1
89
1
1
6
1
2
1
62
3
1
7
23
10
4
1
1
1
3 26
3
1
25
2
9
2
5
4
5
31
25
2
1
4
2
othe
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes
Non−Experts
Experts
0
25
50
75
Percent
imperfect condition − individual annotations
imperfect
96
11
4
1
6
3
1
1
6
1
3
1
1
3
9
3
7
1
2
1
2
1
3
2
6
1
2
1
4
2
1
1
1
23
2
3
19
3
3
1
1
12
1
11
5
1
1
5
1
1
4
6othe
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes
Non−Experts
Experts
0
25
50
75
100
Percent
imperfect condition − aggregated annotations
aggregated
individual

12. 48
6
4
8
5
48
4
1
26
6
5
5
5
6
26
38
164
2
27
1
12
39
35
5
1
129
51
34
3
1
1
11
49
2
1
1
29
13
47
1
1
1
3
1
107
3
1
2
2
1
1
286
1
8
16
1
1
2
6
105
2
86
1
2
20
2
203
3
12
1
3
2
53
7
1
1
2
9
6
11
1
1
27
5
1
1
3
1
2
3
846
5
23
8
4
58
1
16
3
1
2
95
2
1
2
77
32
15
15
1
1
2
30
980
4
16
1
27
10
5
9
1
86
6
2
1
9
2
3
6
1
4
20
2
3
136
3
1
6
18
9
3
2
355
18
2
28
4
13
2
5
2
1
86
1
17
6
132
29
86
1
2
3
45
2
21
12
18
1
13
1
5
3
164
1
14
2
7
1
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes
Non−Experts
Experts
0
25
50
75
Percent
baseline condition − individual annotations
2
1
7
2
1
1
2
1
3
3
8
3
2
5
3
1
30
1
3
1
1
1
37
1
4
8
12
1
6
7
1
1
8
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes
Non−Experts
Experts
0
25
50
75
100
Percent
baseline condition − aggregated annotations
291
63
8
7
5
52
10
9
6
34
4
29
14
8
13
65
7
3
1
59
2
20
10
9
2
7
2
1
8
3
4
2
13
2
9
5
32
8
2
1
1
60
1
1
1
1
2 6
12
1
1
10
35
1
8
2
2
1
1
1
10
4
1
1
3
3
4
1
6
1
7
5
1
1
1
176
20
1
3
3
30
6
1
6
166
3
7
1
6
1
7
18
6
38
1
4
1
1
3
4
6
3
1
10
4
1
89
1
1
6
1
2
1
62
3
1
7
23
10
4
1
1
1
3 26
3
1
25
2
9
2
5
4
5
31
25
2
1
4
2
othe
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes
Non−Experts
Experts
0
25
50
75
Percent
imperfect condition − individual annotations
96
11
4
1
6
3
1
1
6
1
3
1
1
3
9
3
7
1
2
1
2
1
3
2
6
1
2
1
4
2
1
1
1
23
2
3
19
3
3
1
1
12
1
11
5
1
1
5
1
1
4
6othe
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes
Non−Experts
Experts
0
25
50
75
100
Percent
imperfect condition − aggregated annotations
Perfect condition, individual: substantial agreement (κ = 0.65)
12

13. 2
1
7
2
1
1
2
1
3
3
8
3
2
5
3
1
30
1
3
1
1
1
37
1
4
8
12
1
6
7
1
1
8
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes
Non−Experts
Experts
0
25
50
75
100
Percent
baseline condition − aggregated annotations
48
6
4
8
5
48
4
1
26
6
5
5
5
6
26
38
164
2
27
1
12
39
35
5
1
129
51
34
3
1
1
11
49
2
1
1
29
13
47
1
1
1
3
1
107
3
1
2
2
1
1
286
1
8
16
1
1
2
6
105
2
86
1
2
20
2
203
3
12
1
3
2
53
7
1
1
2
9
6
11
1
1
27
5
1
1
3
1
2
3
846
5
23
8
4
58
1
16
3
1
2
95
2
1
2
77
32
15
15
1
1
2
30
980
4
16
1
27
10
5
9
1
86
6
2
1
9
2
3
6
1
4
20
2
3
136
3
1
6
18
9
3
2
355
18
2
28
4
13
2
5
2
1
86
1
17
6
132
29
86
1
2
3
45
2
21
12
18
1
13
1
5
3
164
1
14
2
7
1
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes
Non−Experts
Experts
0
25
50
75
Percent
baseline condition − individual annotations
291
63
8
7
5
52
10
9
6
34
4
29
14
8
13
65
7
3
1
59
2
20
10
9
2
7
2
1
8
3
4
2
13
2
9
5
32
8
2
1
1
60
1
1
1
1
2 6
12
1
1
10
35
1
8
2
2
1
1
1
10
4
1
1
3
3
4
1
6
1
7
5
1
1
1
176
20
1
3
3
30
6
1
6
166
3
7
1
6
1
7
18
6
38
1
4
1
1
3
4
6
3
1
10
4
1
89
1
1
6
1
2
1
62
3
1
7
23
10
4
1
1
1
3 26
3
1
25
2
9
2
5
4
5
31
25
2
1
4
2
othe
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes
Non−Experts
Experts
0
25
50
75
Percent
imperfect condition − individual annotations
96
11
4
1
6
3
1
1
6
1
3
1
1
3
9
3
7
1
2
1
2
1
3
2
6
1
2
1
4
2
1
1
1
23
2
3
19
3
3
1
1
12
1
11
5
1
1
5
1
1
4
6othe
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes
Non−Experts
Experts
0
25
50
75
100
Percent
imperfect condition − aggregated annotations
Perfect condition, aggregated: almost perfect agreement (κ = 0.87)
13

14. “… dramatic bird’s eye view
of Broadway and Wall
Street…”
Church Street El, 1920
by Charles Sheeler
14
Townscape
Cityscapes

15. 291
63
8
7
5
52
10
9
6
34
4
29
14
8
13
65
7
3
1
59
2
20
10
9
2
7
2
1
8
3
4
2
13
2
9
5
32
8
2
1
1
60
1
1
1
1
2 6
12
1
1
10
35
1
8
2
2
1
1
1
10
4
1
1
3
3
4
1
6
1
7
5
1
1
1
176
20
1
3
3
30
6
1
6
166
3
7
1
6
1
7
18
6
38
1
4
1
1
3
4
6
3
1
10
4
1
89
1
1
6
1
2
1
62
3
1
7
23
10
4
1
1
1
3 26
3
1
25
2
9
2
5
4
5
31
25
2
1
4
2
othe
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes
Non−Experts
Experts
0
25
50
75
Percent
imperfect condition − individual annotations
48
6
4
8
5
48
4
1
26
6
5
5
5
6
26
38
164
2
27
1
12
39
35
5
1
129
51
34
3
1
1
11
49
2
1
1
29
13
47
1
1
1
3
1
107
3
1
2
2
1
1
286
1
8
16
1
1
2
6
105
2
86
1
2
20
2
203
3
12
1
3
2
53
7
1
1
2
9
6
11
1
1
27
5
1
1
3
1
2
3
846
5
23
8
4
58
1
16
3
1
2
95
2
1
2
77
32
15
15
1
1
2
30
980
4
16
1
27
10
5
9
1
86
6
2
1
9
2
3
6
1
4
20
2
3
136
3
1
6
18
9
3
2
355
18
2
28
4
13
2
5
2
1
86
1
17
6
132
29
86
1
2
3
45
2
21
12
18
1
13
1
5
3
164
1
14
2
7
1
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes
Non−Experts
Experts
0
25
50
75
Percent
baseline condition − individual annotations
2
1
7
2
1
1
2
1
3
3
8
3
2
5
3
1
30
1
3
1
1
1
37
1
4
8
12
1
6
7
1
1
8
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes
Non−Experts
Experts
0
25
50
75
100
Percent
baseline condition − aggregated annotations
96
11
4
1
6
3
1
1
6
1
3
1
1
3
9
3
7
1
2
1
2
1
3
2
6
1
2
1
4
2
1
1
1
23
2
3
19
3
3
1
1
12
1
11
5
1
1
5
1
1
4
6othe
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes
Non−Experts
Experts
0
25
50
75
100
Percent
imperfect condition − aggregated annotations
Imperfect condition, individual: moderate agreement (κ = 0.47)
15

16. AllegoryOther
Lucretia Calypso

17. 96
11
4
1
6
3
1
1
6
1
3
1
1
3
9
3
7
1
2
1
2
1
3
2
6
1
2
1
4
2
1
1
1
23
2
3
19
3
3
1
1
12
1
11
5
1
1
5
1
1
4
6othe
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes
Non−Experts
Experts
0
25
50
75
100
Percent
imperfect condition − aggregated annotations
48
6
4
8
5
48
4
1
26
6
5
5
5
6
26
38
164
2
27
1
12
39
35
5
1
129
51
34
3
1
1
11
49
2
1
1
29
13
47
1
1
1
3
1
107
3
1
2
2
1
1
286
1
8
16
1
1
2
6
105
2
86
1
2
20
2
203
3
12
1
3
2
53
7
1
1
2
9
6
11
1
1
27
5
1
1
3
1
2
3
846
5
23
8
4
58
1
16
3
1
2
95
2
1
2
77
32
15
15
1
1
2
30
980
4
16
1
27
10
5
9
1
86
6
2
1
9
2
3
6
1
4
20
2
3
136
3
1
6
18
9
3
2
355
18
2
28
4
13
2
5
2
1
86
1
17
6
132
29
86
1
2
3
45
2
21
12
18
1
13
1
5
3
164
1
14
2
7
1
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes
Non−Experts
Experts
0
25
50
75
Percent
baseline condition − individual annotations
2
1
7
2
1
1
2
1
3
3
8
3
2
5
3
1
30
1
3
1
1
1
37
1
4
8
12
1
6
7
1
1
8
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes
Non−Experts
Experts
0
25
50
75
100
Percent
baseline condition − aggregated annotations
291
63
8
7
5
52
10
9
6
34
4
29
14
8
13
65
7
3
1
59
2
20
10
9
2
7
2
1
8
3
4
2
13
2
9
5
32
8
2
1
1
60
1
1
1
1
2 6
12
1
1
10
35
1
8
2
2
1
1
1
10
4
1
1
3
3
4
1
6
1
7
5
1
1
1
176
20
1
3
3
30
6
1
6
166
3
7
1
6
1
7
18
6
38
1
4
1
1
3
4
6
3
1
10
4
1
89
1
1
6
1
2
1
62
3
1
7
23
10
4
1
1
1
3 26
3
1
25
2
9
2
5
4
5
31
25
2
1
4
2
othe
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maes
Non−Experts
Experts
0
25
50
75
Percent
imperfect condition − individual annotations
Imperfect condition, aggregated: moderate agreement (κ = 0.55)
17

18. 18
Cityscapes
Other Genre Landscape

19. Learning Memorizing
19
2 4 6 8 10
020406080100
number of repetitions
percentageofcorrectannotations
random behavior

20. Learning Memorizing
20
2 4 6 8 10
020406080100
number of repetitions
percentageofcorrectannotations
perfect %
imperfect %

21. Learning Generalizing
21
sequence number of new images
percentageofcorrectannotations
[1,20] (60,80] (120,140] (200,220] (280,300] (360,380]
020406080100

22. Learning Generalizing
22
sequence number of new images
percentageofcorrectannotations
perfect %
imperfect %
[1,20] (60,80] (120,140] (200,220] (280,300] (360,380]
020406080100

23. Conclusions
Agreement between experts
and non-experts:
• substantial in perfect,
moderate in imperfect
condition
• aggregation reduces
deviations
Strong disagreement may
indicate:
• need for additional
metadata
• incorrect or incomplete
expert judgements
Learning:
• users memorize and
generalize
• users need a training phase
on high quality data
!
Results in line with
He, J., van Ossenbruggen,
J., de Vries, A.P.: Do you
need experts in the
crowd?: a case study in
image annotation for
marine biology. OAIR 2013

24. Future Work
- scarce
+ high quality data - need training data
+ high quantity data
- lack expertise
+ high quality when
trained & assisted
24

25. Thank you
for your attention!
Myriam C. Traub
myriam.traub@cwi.nl
!
!
http://sealincmedia.project.cwi.nl/artgame/
On the beach of Trouville
Eugène Boudin