Slides prepared by Jasper Oosterman (j.e.g.oosterman@tudelft.nl). Find more information at our website: http://sealincmedia.wordpress.com Large datasets such as Cultural Heritage collections require detailed annotations when digitised and made available online. Annotating dierent aspects of such collections requires a variety of knowledge and expertise which is not always possessed by the collection curators. Artwork annotation is an example of a knowledge intensive image annotation task, i.e. a task that demands annotators to have domain-specic knowledge in order to be successfully completed. Today, Lora Aroyo will present WebSci2014 conference the results of a study aimed at investigating the applicability of crowdsourcing techniques to knowledge intensive image annotation tasks. We observed a clear relationship between the annotation difficulty of an image, in terms of number of items to identify and annotate, and the performance of the recruited workers. Here you can see the poster and the slides of the presentation.

1. Crowdsourcing
Knowledge-­‐Intensive
Tasks
In
Cultural
Heritage
J.
Oosterman,
A.
Bozzon,
G.J.
Houben
A.
NoCamkandath,
C.
Dijkshoorn,
L.
Aroyo

2. AnnotaIon
of
digiIzed
Cultural
Heritage
with
specific
elements.
For
example:
Flowers,
Birds,
Castles

3. Botanical
name
depicted
element
Rosa
banksiae
What
is
the
relaIon
between
enIty
idenIficaIon
difficulty
and
crowd
annotaIon
behavior?

4. Complica0ons
Varying
amount
of
elements,
different
sizes
and
prominence.
Possibly
overlapping
or
lacking
detail.

5. Examples
of
prints
in
the
collecIon

6. AnnotaIon
task
template
Number
of
flowers
Number
of
flower
types
Up
to
three
flower
labels
Certainty
of
the
answers
References

7. # Unable
# Fantasy
# Flower Name
#Prints
1
10
20
30
40
Worker ID
0 10 20 30 40
#
Workers
opened
task
732
#
Workers
passed
test
quesIons
84
#
Selected
workers
44
#
AnnotaIon
tasks
performed
488
#
“Fantasy”
task
58
#
“Unable”
task
70
#
“Flowers”
task
360
#
Flower
labels
465
Tasks
performed
by
workers
LiCle
agreement
on
“unable”
or
“fantasy”
Tasks
requiring
domain
knowledge
not
popular

8. # Instances
# Prints
10
100
R
ose
Lilium
Tulip
Sunflow
er
D
ianthus
B
ellis
Iris
Viola
O
rchidaceaePaeonia
Papaver
Lotus
B
ellis
Perennis
N
arcissus
H
yacinth
%WrongAnswer
0
0.2
0.4
0.6
0.8
1.0
# of Flowers
NP P
Flower Types
NP P
Flower
idenIficaIon
not
dependent
on
prominence
Most
workers
provide
simple
common
names,
but
also
workers
with
excepIonal
detail.

9. Local
idenIficaIon
Outlook
Local
labeling

10. hCp://sealincmedia.wordpress.com
sealincmedia-­‐all@lists.tudelj.nl
J.
Oosterman,
A.
Bozzon,
G.J.
Houben
A.
NoCamkandath,
C.
Dijkshoorn,
L.
Aroyo