Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

CrowdTruth for User-Centric Relevance

1,159 views

Published on

Presented at Mini-workshop on Multiple Dimensions of Relevance: https://www.facebook.com/events/293331494186432/

Published in: Technology
  • I think you could try to characterize the workers, as some kind of analysis of the contributors. However, I think the volume is the most important aspect there - to have enough people to annotate (especially when it is potentially vague and with multiple perspectives) - so that there are enough votes for each perspective to get its partial agreement. Everything is in the collective 'decision making' - each of them will have their own perspective, but together, they will annotate each event and event property from every perspective possible. Does this make sense?
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • As I looked at slides 24-25 etc, I was wondering: won't people (workers) interpret and annotate within (or wrt) their respective contexts, which are informed by their own knowledge and experiences? Of all the things, degree of agreement among the crowd members (annotators), captured along different properties characterizing an event (time, location, topic, ...or other properties you choose to model events) becomes the central objective. So would it make sense to characterize that? And how would one do that? Would you characterize say by classifying people (eg, a 10 ear with one class in cultural history, a graduate in arts, an expert curator)? And that would be quite hard, right? Just some questions for which I have few answers. I recall an old example: In a task asking just two educators to annotate text from 8th grade textbook, they had ONLY 62% agreement! Anyways, an important topic to explore and try to understand...I do like the point of trying to understand disagreement (as that is part of understanding agreement) in the conclusion.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

CrowdTruth for User-Centric Relevance

  1. 1. CrowdTruth Human-assisted computing for understanding semantic interpretation & user-centric relevance Lora Aroyo http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  2. 2. “The Gallery of Cornelis van der Geest” (Willem van Haecht) http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  3. 3. hun<ng vs. dogs http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  4. 4. events vs. people http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  5. 5. DIGITAL HERMENEUTICS theory of interpretation: relation parts of wholes events as context for interpretation of online collections intersection of hermeneutics & Web technology http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  6. 6. Linking to Events http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  7. 7. Generating Events Narrative http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  8. 8. So far so good, but …. L. Aroyo, C. Welty: Truth is a Lie: 7 Myths about Human Annota;on, AI Magazine 2014 (in press). http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  9. 9. Events are Vague people have no clear notion of what events are http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  10. 10. Events have Perspec+ves and people don’t always agree http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  11. 11. “Event can refer to many things such as: An observable occurrence, phenomenon or an extraordinary occurrence.” “an event is an incident that's very important or monumental” “A planned public or social get together or occasion.” “An event is something occurring at a specific time and/or date to celebrate or recognize a particular occurrence.” “a location where something like a function is held. you could tell if something is an event if there people gathering for a purpose.” If you ask the crowd ... http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  12. 12. “an event is the exemplifica;on of a property by a substance at a given ;me” Jaegwon Kim, 1966 “events are changes that physical objects undergo” Lawrence Lombard, 1981 “events are proper;es of spa;otemporal regions”, David Lewis, 1986 under30ceo.com If you ask the experts ... http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  13. 13. under30ceo.com People are the ones who search & determine relevance http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  14. 14. Experts vs. Crowd? Medical Relation Extraction Task (Aroyo, Welty 2014) • 91% of expert annotations covered by the crowd • expert annotators agree only in 30% • popular crowd vote covers 95% of expert agreement Waisda? Video Tagging (Gligorov et al. 2011) • 14% tags in search logs are in professional vocab (GTAA) • huge gap between expert & lay users’ views on what’s important Steve.Museum Project (Leason 2009) • 14% user tags are in expert-curated documentation under30ceo.com http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  15. 15. CrowdTruth Based on annotator disagreement as an indication of the variation in human semantic interpretation of signs, and can indicate ambiguity, vagueness, over-generality, etc. crowdtruth.org L. Aroyo, C. Welty: Crowd Truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard. ACM WebSci 2013. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  16. 16. CrowdTruth Framework for News Event Extraction O.Inel, K.Khamkham, T.Cristea, A.Rutjes, J.van der Ploeg, L.Aroyo, R. Sips, A.Dumitrache, L.Romaszko: CrowdTruth: Machine- Human Computation Framework for Harnessing Disagreement in Gathering Annotated Data. ISWC 2014. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  17. 17. News Event Extraction The police came to Apple’s glass cube on Fifth Avenue on Tuesday to enforce order after activists released black balloons inside the cube to [protest] the company’s environmental policies. The police came to Apple’s glass cube on Fifth Avenue on Tuesday [to enforce] order after activists released black balloons inside the cube to protest the company’s environmental policies. The police came to Apple’s glass cube on Fifth Avenue on Tuesday [to enforce order] after activists released black balloons inside the cube to protest the company’s environmental policies. The police came to Apple’s glass cube on Fifth Avenue on Tuesday to enforce order after activists [released] black [balloons] inside the cube to protest the company’s environmental policies. The police [came]to Apple’s glass cube on Fifth Avenue on Tuesday [to enforce] order after activists released black balloons inside the cube to protest the company’s environmental policies. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  18. 18. Video Event Extraction Following the grandeur of Baroque, Rococo art is often dismissed as frivolous and unserious, but Waldemar Januszczak disagrees. […] The first episode is about travel in the 18th century and how it impacted greatly on some of the finest art ever made. The world was getting smaller and took on new influences shown in the glorious Bavarian pilgrimage architecture, Canaletto's romantic Venice and the blossoming of exotic designs and tastes all over Europe. Rococo: Travel, pleasure, madness http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  19. 19. Events have multiple DIMENSIONS Micro-task Template http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  20. 20. Each DIMENSION has different GRANULARITY Micro-task Template http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  21. 21. People have different POINTS OF VIEWS Micro-task Template http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  22. 22. Triangle of Reference Sign Reference Observer http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  23. 23. Triangle of Reference to Capture Disagreement Sentence Annotation Task Worker http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  24. 24. CrowdTruth Metrics Event Extraction Three parts to understand human interpretations: Sentence How good is a sentence for the event extraction task? Workers How well does a worker understand the sentence? Relations Is the meaning of the event type clear? How ambiguous/confusable is it? Aroyo, L., Welty, C.: (2014) The Three Sides of CrowdTruth. Journal of Human Computation Lora Aroyo Crowd Truth for Cognitive Computing Chris Welty
  25. 25. Crowd Truth Metrics based on the Triangle of Reference Three parts to understand human interpretations: Sign How good is a sign for conveying information? People How well does a person understand the sign? Ontology Are the distinctions of the ontology clear? How ambiguous/confusable are they? Aroyo, L., Welty, C.: (2014) The Three Sides of CrowdTruth. Journal of Human Computation Lora Aroyo Crowd Truth for Cognitive Computing Chris Welty
  26. 26. Disagreement Analytics • sentence metrics: sentence clarity, sentence-relation score • annotation task metrics: event clarity, type similarity, relation ambiguity • worker metrics: o worker-sentence disagreement o worker-worker disagreement o avg number of annotations per sentence o valid words in explanation text o same explanation across contributions o “[OTHER]” + different type o time to complete, number of sentences, etc. Aroyo, L., Welty, C.: (2013) Measuring crowd truth for medical relation extraction. AAAI Fall Symposium on Semantics for Big Data http://www.americanprogress.org/wp-content/uploads/2012/12/multiple_http://lora-aroyo.org http://slideshare.net/laroyo @laroyo measures_onpage.jpg
  27. 27. Spam Detection o filter sentences on their clarity score: to avoid penalizing workers for contributing on ambiguous sentences o bad sentences are removed = increase of accuracy on spam detection o apply worker metrics to analyze worker agreement: workers who systematically disagree o with majority (worker-sentence disagreement) o with rest of co-workers (worker-worker disagreement) o spammers annotations are removed = improvement of accuracy of sentence metrics http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  28. 28. Annotation Example Around 2:30 p.m., as if delivering birthday greetings, several Greenpeace demonstrators [ENTERED] the cube clutching helium-filled balloons, which were the shape and color of charcoal briquettes. Overall annotation & granularity distribution: http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  29. 29. Event Type Disagreement Around 2:30 p.m., as if delivering birthday greetings, several Greenpeace demonstrators [ENTERED] the cube clutching helium-filled balloons, which were the shape and color of charcoal briquettes. [ENTERED] ACTION (18.2%) MOTION (9.1%) ARRIVING_OR_ DEPARTING (54.5%) PURPOSE (18.2%) type type Sentence type type Ontology Worker http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  30. 30. Event Location Disagreement Around 2:30 p.m., as if delivering birthday greetings, several Greenpeace demonstrators [ENTERED] the cube clutching helium-filled balloons, which were the shape and color of charcoal briquettes. [ENTERED] the cube (38.5%) cube (38.5%) none (23%) OTHER (100%) NOT APPLICABLE (100%) type type COMMERCIAL (40%) OTHER (40%) INDUSTRIAL (20%) type type type Sentence Ontology Worker http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  31. 31. Around 2:30 p.m., as if delivering birthday greetings, several Greenpeace demonstrators [ENTERED] the cube clutching helium-filled balloons, which were the shape and color of charcoal briquettes. [ENTERED] Event Time Disagreement Around (9.1%) Around 2:30 p.m. (45.45%) 2:30 p.m. (45.45%) TIMESTAM P (100%) TIMESTAM P (100%) type type TIMESTAM P (100%) type Sentence Ontology Worker http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  32. 32. Event Participant Disagreement Around 2:30 p.m., as if delivering birthday greetings, several Greenpeace demonstrators [ENTERED] the cube clutching helium-filled balloons, which were the shape and color of charcoal briquettes. [ENTERED] Greenpeace (15.39%) demonstrators (15.39%) Greenpeace demonstrators (69.23%) ORGANIZATION (100%) PERSON (100%) type type ORGANIZATION (77.77%) PERSON (22.22%) type type Sentence Ontology Worker http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  33. 33. Comparative Annotation Distribution Event Type Distribution Time Type Distribution Sentence Ontology Worker The high disagreement for event type across all sentences likely indicates problems with the ontology. These event types are difficult to distinguish between. The event classes may overlap, be confusable, too vague, etc. http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  34. 34. Comparative Annotation Distribution Location Type Distribution Participant Type Distribution Sentence Ontology Worker http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  35. 35. Challenges ● Defining relevance, e.g. relevant or related events, entities, videos ● Depicted vs. associated relevance, e.g. in video, in audio ● Deal with reliability, e.g. provenance ● Visualize quality analytics, e.g. multidimensionality http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  36. 36. Events Cultural Heritage Exploration http://dive.beeldengeluid.nl/ http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  37. 37. Events @ • Agora: Historical Events in Cultural Heritage Collections – http://agora.cs.vu.nl/ • Extractivism: Activist Events in Newspapers – http://mona-project.org/ • Semantics of History – http://www2.let.vu.nl/oz/cltl/semhis/ • BiographyNet: Events Change in Perspective over Time • NewsReader: Multilingual Events & Storylines in Newspapers – http://www.newsreader-project.eu/ http://lora-aroyo.org http://slideshare.net/laroyo @laroyo
  38. 38. Conclusions ● Events are just one example for diversity of human interpretations ● Understanding crowd disagreement helps understand event semantics ● Considering the interdependence of the different aspects of the annotations improves their quality ● Disagreement metrics adaptable across domains - helped us to understand the vagueness and the clarity of a sentence/ putative event http://lora-aroyo.org http://slideshare.net/laroyo @laroyo Sentence Annotation task Worker
  39. 39. Questions? http://lora-aroyo.org http://slideshare.net/laroyo @laroyo

×