0
Harnessing Disagreement for     Event Semantics                  Lora Aroyo                  Chris Welty
Objects vs. Events	  events perdure = their parts exist at different time points	objects endure = they have all their part...
Extraction i n NLPEvent•  Find Events mentioned in text	•  Type them (communication, bombing, …)	•  Find the role fillers (...
Measuring•    Define the task for human annotators	•    Annotate by multiple people	•    Measure agreement	•    While k<.6 ...
Position                                                             eir semantics                                        ...
Events are Vague Humans have no clear notion of what            events are
event is a significant                             "happening" or                             gathering of people. I       ...
Event is a happening,                     which can be scheduled                     or unscheduled. An                   ...
An event would be any                    occurrence where physical action                    has taken place. It may be a ...
Event can refer to many things such as: An observable occurrence, phenomenon or an extraordinary occurrence.              ...
What do Experts think an EVENT is?   an event is the exemplification of a property by a substance at a given time          ...
What do Experts think an EVENT is?   an event is the exemplification of a property by a substance at a given time          ...
Why is eventsemantics hard?
the World is Open     1. events have multiple dimensions2. each dimension has levels of granularity   3. people have differ...
and our goal is ...                   1. not to enforce agreement               2. to capture different view points      3....
Artificially                          Hypothesis                      restricting                                  humans d...
What do PeopleDisagree on?      are sub-events always mere parts?    are mentions meaningful for events? are events corefe...
the bombing targeted a housing development in      Baghdad, killing 3 and injuring 13         indistinguishable by people,...
Disagreement Framework•    ontology: disagreements on the basic status of events     themselves as referents of linguistic...
Granularity Disagreement•  spatial, temporal, participants•  compositional, classificational
Event Participants                  Disagreement                                     Israeli       Prime minister50%	     ...
Temporal Disagreement       Prime minister50%	    Benjamin                    Sunday	      50%	         Netanyahu	        ...
Spatial Disagreement                                          Southern                                                    ...
Approach Principles  1.  tolerate, capture & exploit disagreement  2.  understand the range of disagreements      by creat...
Position                                                             eir semantics                                        ...
Artificially                          Conclusion                      restricting                                  humans d...
Harnessing Disagreement for Event Semantics
Upcoming SlideShare
Loading in...5
×

Harnessing Disagreement for Event Semantics

748

Published on

http://ceur-ws.org/Vol-902/paper_4.pdf

The focus of this paper is on how events can be detected & extracted
from natural language text, and how those are represented for use on the semantic web. We draw an inspiration from the similarity between crowdsourcing approaches for tagging and text annotation task for ground truth of events. Thus,
we propose a novel approach that harnesses the disagreement between the human annotators by defining a framework to capture and analyze the nature of the
disagreement. We expect two novel results from this approach. On the one hand,
achieving a new way of measuring ground truth (performance), and on the other
hand identifying a new set of semantic features for learning in event extraction.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
748
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
4
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Harnessing Disagreement for Event Semantics"

  1. 1. Harnessing Disagreement for Event Semantics Lora Aroyo Chris Welty
  2. 2. Objects vs. Events events perdure = their parts exist at different time points objects endure = they have all their parts at all points in time objects are wholly present at any point in time, events unfold over time Flickr: vanilllaph
  3. 3. Extraction i n NLPEvent•  Find Events mentioned in text •  Type them (communication, bombing, …) •  Find the role fillers (location, date, participants, …) •  State of the art is extremely low: .10 F
  4. 4. Measuring•  Define the task for human annotators •  Annotate by multiple people •  Measure agreement •  While k<.6 refine definition and repeat ... for events this process is long, disagreement is high, agreement becomes forced
  5. 5. Position eir semantics are par t of th t about events en man di sagreem HuFlickr: elkabong
  6. 6. Events are Vague Humans have no clear notion of what events are
  7. 7. event is a significant "happening" or gathering of people. I would define a "happening" as an event if the group of people gathered were united in one common goal. We Asked the Crowd What an EVENT isFlickr: massimo vitali
  8. 8. Event is a happening, which can be scheduled or unscheduled. An earthquake or fire happens (unscheduled). A wedding or birthday party (scheduled). It is an occasion that is unusual and tends to be memorable.We Asked the Crowd What an EVENT is
  9. 9. An event would be any occurrence where physical action has taken place. It may be a single, momentary instance (I sneezed), or it may span a period of time (the festival ran for four hours). An event may also be made up of a number of smaller events, such as a day at school is an event, but each individual class is also an event itself. Basically an event must have a physical action over any delimited time span.We Asked the Crowd What an EVENT is
  10. 10. Event can refer to many things such as: An observable occurrence, phenomenon or an extraordinary occurrence. an event is an incident thats very important or monumental A planned public or social get together or occasion. An event is something occurring at a specific time and/or date to celebrate or recognize a particular occurrence. a location where something like a function is held. you could tell if something is an event if there people gathering for a purpose.We Asked the Crowd What an EVENT is
  11. 11. What do Experts think an EVENT is? an event is the exemplification of a property by a substance at a given time Jaegwon Kim, 1966 events are changes that physical objects undergo Lawrence Lombard, 1981 events are properties of spatiotemporal regions , David Lewis, 1986under30ceo.com
  12. 12. What do Experts think an EVENT is? an event is the exemplification of a property by a substance at a given time Jaegwon Kim, 1966 events are changes that physical objects undergo Lawrence Lombard, 1981 nothing everything events are properties of spatiotemporal regions , David Lewis, 1986under30ceo.com
  13. 13. Why is eventsemantics hard?
  14. 14. the World is Open 1. events have multiple dimensions2. each dimension has levels of granularity 3. people have different views on both all this leads to very complex semantics
  15. 15. and our goal is ... 1. not to enforce agreement 2. to capture different view points 3. to teach machines to reason in the disagreement spaceFlickr: elkabong
  16. 16. Artificially Hypothesis restricting humans d Machines oes not h will learn elp mach from dive ines to le rsity arn.Flickr: elkabong
  17. 17. What do PeopleDisagree on? are sub-events always mere parts? are mentions meaningful for events? are events coreferential across documents? (e.g. perspectives, observations)
  18. 18. the bombing targeted a housing development in Baghdad, killing 3 and injuring 13 indistinguishable by people, confusable: is bombing part of killing, or killing part of bombing? What about targeting? merelogically extensional (i.e arbitrary): container bursting into fragments as a result of explosion some events don t exist: an action by military forces prevented the bombing.
  19. 19. Disagreement Framework•  ontology: disagreements on the basic status of events themselves as referents of linguistic utterances, e.g. are people events or do events exist at all.•  granularity: disagreements that result from issues of granularity, e.g. the location being a country, region, or city, the time being a day, week, month, etc.•  interpretation: disagreements that result from (non- granular) ambiguity, differences in perspective, or error in interpreting an expression, e.g. classifying a person as a terrorist/hero, October Revolution took place in September.
  20. 20. Granularity Disagreement•  spatial, temporal, participants•  compositional, classificational
  21. 21. Event Participants Disagreement Israeli Prime minister50% Government 10% Benjamin Netanyahu Israeli Cabinet 15% his Cabinet 15% 35% Benjamin {TOLD} Netanyahu Benjamin Israeli Prime Netanyahu s 5% 15% minister Cabinet Cabinet 45%
  22. 22. Temporal Disagreement Prime minister50% Benjamin Sunday 50% Netanyahu March 1, 1998 25% 35% Benjamin {TOLD} March 1998 15% Netanyahu Spring 1998 5% Israeli Prime15% minister
  23. 23. Spatial Disagreement Southern 35% 30% Israel Lebanon {WILLING TO WITHDRAW}65% Israels Northern Lebanon 45% Frontier Middle East 10%
  24. 24. Approach Principles 1.  tolerate, capture & exploit disagreement 2.  understand the range of disagreements by creating a space of possibilities with frequencies & similarities 3.  score machine output based on where it falls in this space 4.  adaptable to new annotation tasksFlickr: auroille
  25. 25. Position eir semantics are par t of th t about events en man di sagreem HuFlickr: elkabong
  26. 26. Artificially Conclusion restricting humans d Machines oes not h will learn elp mach from dive ines to le rsity arn.Flickr: elkabong
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×