Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Crowdsourcing Knowledge-Intensive Tasks In Cultural Heritage: SealincMedia Accurator demonstrator @WebSci2014

Slides prepared by Jasper Oosterman ( Find more information at our website:

Large datasets such as Cultural Heritage collections require detailed annotations when digitised and made available online. Annotating dierent aspects of such collections requires a variety of knowledge and expertise which is not always possessed by the collection curators. Artwork annotation is an example of a knowledge intensive image annotation task, i.e. a task that demands annotators to have domain-specic knowledge in order to be successfully completed.

Today, Lora Aroyo will present WebSci2014 conference the results of a study aimed at investigating the applicability of crowdsourcing techniques to knowledge intensive image annotation tasks. We observed a clear relationship between the annotation difficulty of an image, in terms of number of items to identify and annotate, and the performance of the recruited workers. Here you can see the poster and the slides of the presentation.

  • Login to see the comments

  • Be the first to like this

Crowdsourcing Knowledge-Intensive Tasks In Cultural Heritage: SealincMedia Accurator demonstrator @WebSci2014

  1. 1. Crowdsourcing  Knowledge-­‐Intensive  Tasks  In   Cultural  Heritage   J.  Oosterman,  A.  Bozzon,  G.J.  Houben   A.  NoCamkandath,  C.  Dijkshoorn,  L.  Aroyo  
  2. 2. AnnotaIon  of  digiIzed  Cultural     Heritage  with  specific  elements.   For  example:  Flowers,  Birds,  Castles  
  3. 3. Botanical  name  depicted  element   Rosa  banksiae   What  is  the  relaIon  between  enIty   idenIficaIon  difficulty  and  crowd  annotaIon   behavior?  
  4. 4. Complica0ons   Varying  amount  of  elements,  different  sizes   and  prominence.  Possibly  overlapping  or   lacking  detail.  
  5. 5. Examples  of  prints  in  the  collecIon  
  6. 6. AnnotaIon  task  template   Number  of  flowers   Number  of  flower  types   Up  to  three  flower  labels   Certainty  of  the  answers   References  
  7. 7. # Unable # Fantasy # Flower Name #Prints 1 10 20 30 40 Worker ID 0 10 20 30 40 #  Workers  opened  task   732   #  Workers  passed  test  quesIons   84   #  Selected  workers   44   #  AnnotaIon  tasks  performed   488   #  “Fantasy”  task   58   #  “Unable”  task   70   #  “Flowers”  task     360   #  Flower  labels   465   Tasks  performed  by  workers   LiCle  agreement  on     “unable”  or  “fantasy”   Tasks  requiring  domain     knowledge  not  popular    
  8. 8. # Instances # Prints 10 100 R ose Lilium Tulip Sunflow er D ianthus B ellis Iris Viola O rchidaceaePaeonia Papaver Lotus B ellis Perennis N arcissus H yacinth %WrongAnswer 0 0.2 0.4 0.6 0.8 1.0 # of Flowers NP P Flower Types NP P Flower  idenIficaIon  not   dependent  on  prominence     Most  workers  provide  simple  common  names,   but  also  workers  with  excepIonal  detail.  
  9. 9. Local  idenIficaIon   Outlook   Local  labeling  
  10. 10. hCp://     sealincmedia-­‐       J.  Oosterman,  A.  Bozzon,  G.J.  Houben   A.  NoCamkandath,  C.  Dijkshoorn,  L.  Aroyo