http://ceur-ws.org/Vol-902/paper_4.pdf The focus of this paper is on how events can be detected & extracted from natural language text, and how those are represented for use on the semantic web. We draw an inspiration from the similarity between crowdsourcing approaches for tagging and text annotation task for ground truth of events. Thus, we propose a novel approach that harnesses the disagreement between the human annotators by defining a framework to capture and analyze the nature of the disagreement. We expect two novel results from this approach. On the one hand, achieving a new way of measuring ground truth (performance), and on the other hand identifying a new set of semantic features for learning in event extraction.

1. Harnessing Disagreement for
Event Semantics
Lora Aroyo
Chris Welty

2. Objects vs. Events
events perdure = their parts exist at different time points
objects endure = they have all their parts at all points in time
objects are wholly present at any point in time, events unfold over time
Flickr: vanilllaph

3. Extraction i n NLP
Event
• Find Events mentioned in text
• Type them (communication, bombing, …)
• Find the role fillers (location, date,
participants, …)
• State of the art is extremely low: .10 F

4. Measuring
• Define the task for human annotators
• Annotate by multiple people
• Measure agreement
• While k<.6 refine definition and repeat
... for events this
process is long, disagreement
is high, agreement becomes
forced

5. Position
eir semantics
are par t of th
t about events
en
man di sagreem
Hu
Flickr: elkabong

6. Events are Vague
Humans have no clear notion of what
events are

7. event is a significant
"happening" or
gathering of people. I
would define a
"happening" as an event
if the group of people
gathered were united in
one common goal.
We Asked the Crowd What an EVENT is
Flickr: massimo vitali

8. Event is a happening,
which can be scheduled
or unscheduled. An
earthquake or fire
happens (unscheduled). A
wedding or birthday
party (scheduled). It is an
occasion that is unusual
and tends to be
memorable.
We Asked the Crowd What an EVENT is

9. An event would be any
occurrence where physical action
has taken place. It may be a single,
momentary instance (I sneezed),
or it may span a period of time
(the festival ran for four hours). An
event may also be made up of a
number of smaller events, such
as a day at school is an event, but
each individual class is also an
event itself. Basically an event must
have a physical action over any
delimited time span.
We Asked the Crowd What an EVENT is

10. Event can refer to many things such as: An observable
occurrence, phenomenon or an extraordinary occurrence.
an event is an incident that's very important or monumental
A planned public or social get together or occasion.
An event is something occurring at a specific time and/or
date to celebrate or recognize a particular occurrence.
a location where something like a function is held. you could tell
if something is an event if there people gathering for a purpose.
We Asked the Crowd What an EVENT is

11. What do Experts think an EVENT is?
an event is the exemplification of a property by a substance at a given time
Jaegwon Kim, 1966
events are changes that physical objects undergo Lawrence Lombard, 1981
events are properties of spatiotemporal regions , David Lewis, 1986
under30ceo.com

12. What do Experts think an EVENT is?
an event is the exemplification of a property by a substance at a given time
Jaegwon Kim, 1966
events are changes that physical objects undergo Lawrence Lombard, 1981
nothing everything
events are properties of spatiotemporal regions , David Lewis, 1986
under30ceo.com

13. Why is event
semantics hard?

14. the World is Open
1. events have multiple dimensions
2. each dimension has levels of granularity
3. people have different views on both
all this leads to very complex semantics

15. and our goal is ...
1. not to enforce agreement
2. to capture different view points
3. to teach machines to reason in the disagreement
space
Flickr: elkabong

16. Artificially
Hypothesis
restricting
humans d
Machines oes not h
will learn elp mach
from dive ines to le
rsity arn.
Flickr: elkabong

17. What do People
Disagree on?
are sub-events always mere parts?
are mentions meaningful for events?
are events coreferential across documents?
(e.g. perspectives, observations)

18. the bombing targeted a housing development in
Baghdad, killing 3 and injuring 13
indistinguishable by people, confusable:
is bombing part of killing, or killing part of bombing?
What about targeting?
merelogically extensional (i.e arbitrary):
container bursting into fragments as a result of explosion
some events don t exist:
an action by military forces prevented the bombing.

19. Disagreement Framework
• ontology: disagreements on the basic status of events
themselves as referents of linguistic utterances, e.g. are
people events or do events exist at all.
• granularity: disagreements that result from issues of
granularity, e.g. the location being a country, region, or
city, the time being a day, week, month, etc.
• interpretation: disagreements that result from (non-
granular) ambiguity, differences in perspective, or error in
interpreting an expression, e.g. classifying a person as a
terrorist/hero, October Revolution took place in
September.

20. Granularity Disagreement
• spatial, temporal, participants
• compositional, classificational

22. Event Participants
Disagreement
Israeli
Prime minister
50%
Government
10%
Benjamin
Netanyahu
Israeli Cabinet
15%
his Cabinet
15%
35%
Benjamin {TOLD}
Netanyahu
Benjamin
Israeli Prime Netanyahu s 5%
15%
minister
Cabinet
Cabinet
45%

23. Temporal Disagreement
Prime minister
50%
Benjamin Sunday
50%
Netanyahu
March 1, 1998
25%
35%
Benjamin {TOLD} March 1998
15%
Netanyahu
Spring 1998
5%
Israeli Prime
15%
minister

24. Spatial Disagreement
Southern
35%
30%
Israel
Lebanon
{WILLING TO
WITHDRAW}
65%
Israel's Northern Lebanon
45%
Frontier
Middle East
10%

25. Approach Principles
1. tolerate, capture & exploit disagreement
2. understand the range of disagreements
by creating a space of possibilities with
frequencies & similarities
3. score machine output based on where it
falls in this space
4. adaptable to new annotation tasks
Flickr: auroille

26. Position
eir semantics
are par t of th
t about events
en
man di sagreem
Hu
Flickr: elkabong

27. Artificially
Conclusion
restricting
humans d
Machines oes not h
will learn elp mach
from dive ines to le
rsity arn.
Flickr: elkabong