1. Dial E
for
Events
Lora Aroyo
Monday, September 10, 12
2. Observation
events are important
events are omni-present
in events carry different points of view
th
in ew
ou orl
r p d, e
er .g.
so
na new
ne l li s,
tw ve sc
or s, ien
kin e.g ce
g .s
oc etc
ial .
Flickr: elkabong iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
3. Position
ts are p art of
s
ue ne sof even
greeme nt & vag
human disa vent se mantics
The the e
Flickr: elkabong iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
4. Objects vs. Events
events perdure = their parts exist at different time points
objects endure = they have all their parts at all points in time
objects are wholly present at any point in time, events unfold over time
Flickr: vanilllaph iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
5. Events are important
create context for objects, e.g. people, locations, organizations, etc.
Lora Aroyo @laroyo iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
6. Events are important
create meaning for objects, e.g. artifacts, pictures, videos.
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
7. Events are important
link concepts, objects, and stories.
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
8. Events in the World
events anchor the information we
consume daily
Flickr: craftydogma iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
9. Events @ Google News
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
10. Events @ Google News
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
17. The Arab Spring
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
18. The Arab Spring
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
19. Events @ Social Web
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
20. Events @ Social Web
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
21. Events @ Social Web
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
22. Events @ Social Web
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
23. Events @ Social Web
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
24. Events are Vague
Humans have no clear notion of what events are
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
25. “event is a significant
"happening" or
gathering of people. I
would define a
"happening" as an event
if the group of people
gathered were united in
one common goal.”
We Asked the Crowd What an EVENT is
Flickr: massimo vitali iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
26. Event is a happening,
which can be scheduled or
unscheduled. An
earthquake or fire
happens (unscheduled). A
wedding or birthday
party (scheduled). It is an
occasion that is unusual
and tends to be
memorable.
We Asked the Crowd What an EVENT is
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
27. “An event would be any
occurrence where physical action
has taken place. It may be a single,
momentary instance (I sneezed),
or it may span a period of time
(the festival ran for four hours). An
event may also be made up of a
number of smaller events, such
as a day at school is an event, but
each individual class is also an
event itself. Basically an event must
have a physical action over any
delimited time span.”
We Asked the Crowd What an EVENT is
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
28. “Event can refer to many things such as: An observable
occurrence, phenomenon or an extraordinary occurrence.”
“an event is an incident that's very important or monumental”
“A planned public or social get together or occasion.”
“An event is something occurring at a specific time and/or
date to celebrate or recognize a particular occurrence.”
“a location where something like a function is held. you could tell
if something is an event if there people gathering for a purpose.”
We Asked the Crowd What an EVENT is
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
29. What do Experts think an EVENT is?
“an event is the exemplification of a property by a substance at a given time” Jaegwon Kim, 1966
“events are changes that physical objects undergo” Lawrence Lombard, 1981
“events are properties of spatiotemporal regions”, David Lewis, 1986
under30ceo.com iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
30. Event-centric Projects
how events can be detected & extracted from
natural language text
how those extracted events are represented for use
on the semantic web
how to identify the same events in different sources
how to capture different perspectives
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
31. Activists
prominent on
the new web
through
different
channels
by nature
multi-
perspective,
biased &
emotional
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
32. Activists
prominent on
the new web
through
different
channels
by nature
multi-
perspective,
biased &
emotional
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
34. • build visualizations
• create appropriate analytics“All protest events
• answer questions of end users & social in.”
Greenpeace participated
scientists
Mapping Online blogs, news, activists websites
Networks of Activism
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
35. Extracting
Historical Events
• objects (digitized • What happened
artworks and before/after?
artifacts)
• Who does what,
• events (concrete when, and where?
particulars)
• All bomb attacks in
• entities (actors, the 1950s
locations, periods) • In what events did
• narratives Indonesia
participate?
(organization of
events) • ‘Grand narratives’
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
36. iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
37. • generate meaningful event sequences
• capture the different perspectives
• serve both end users & history researchers
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
38. Timelines
from Text
“al-Qaeda activities in Syria”
4 right, 2 wrong, 3 missing events
two have no explicit times & are in the
wrong order
One involved al-Qaeda but took place in
Jordan on the Syrian border
does a fuzzy task require fuzzier metrics?
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
39. Timelines
from Text
“al-Qaeda activities in Syria”
4 right, 2 wrong, 3 missing events
two have no explicit times & are in the
wrong order
One involved al-Qaeda but took place in
Jordan on the Syrian border
does a fuzzy task require fuzzier metrics?
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
40. Machine Reading
build event timelines from text in 2 example
domains
• NFL: news articles on football;
• Ontology: 4 classes, 20 relations
• Intel: news articles on terrorist events;
• Ontology: 20 Classes, 50 relations
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
41. Why is event
semantics hard?
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
42. According to NLP tradition
Gather Extract
1 2
your source material events and properties
Analyze Visualize
3 4
statistics, timelines, etc.
find links between events
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
43. but for events we stumble
Gather Extract
1 2
your source material events and properties
Analyze Visualize
3 4
statistics, timelines, etc.
find links between events
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
44. but for events we stumble
Gather Extract
experts typically:
1 2
define a problem
annotate ground truth
your source material events and properties
train
Analyze Visualize
3 evaluate 4
statistics, timelines, etc.
find links between events
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
45. Closed World Dictatorship
1. domain experts define the meaning
2. using limited vocabulary
3. aim for agreement
to fix the problem of high disagreement for events
experts enforce more tyranny - stricter rules
comparatively little annotated data for training & evaluation of event
detection systems
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
46. But the World is Open
1. events have multiple dimensions
2. each dimension has levels of granularity
3. people have different views on both
all this leads to very complex semantics
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
47. and our goal is ...
1. not to enforce agreement
2. to capture different view points
3. to teach machines to reason in the disagreement space
Flickr: elkabong iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
48. Position
Artificially
restricting
humans d
Machines oes not h
will learn elp mach
from dive ines to le
arn.
rsity
Flickr: elkabong iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
49. Professional Dictatorship of the Closed World
• Museum, libraries, archives & researchers have been dominating the views.
• Controlled vocabularies & annotation schemes were leading.
• Professionals enforced agreement among themselves.
• End-users needs & tasks are not considered.
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
50. there is a tiny overlap between end-user terminology & professional annotations
the latter are typically coarse-grained & refer to entire object / topic
Flickr: ganzelka iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
51. only 1,900 tags (32,200 in total) match in vocabularies
257 in people (83 validated) 1,661 in geo (666 validated)
9,796 validated, but no match in professional vocabulary
8% professional vocab
23 % lexical vocab
63% meaningful Google matches
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
52. Amateur democracy of the Open World
• Once the Web opened the world of information, professional
dictatorship clashed with end-users democracy.
• What professionals consider interesting, relevant or important
does not match what users think of it.
• Amateurs cannot find what they were searching for.
(c) banksy iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
53. people are interested in different annotation
categories than the professionals
Flickr: ganzelka iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
54. Tag sample:
1,343 verified tags of 5 random video fragments
Abstract General Specific Total
Video aspects that are
Who 10 166 177 31%
described by those tags: 5 12
What 73 563 12 57%
non-visual (0) Where 0 68 8 7%
perceptual (11), e.g. color When 4 31
31 6 5%
conceptual (1,332) Total 7% 74% 9%
Object tags (1,313)
Scene tags (30)
195 tags
(adverbs & adjectives)
couldn’t be classified
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
55. Harnessing the Crowd
• we include the user’s opinion as first class citizens.
• this brings the need to combine all these (different)
opinions into a system of opinions that makes sense.
• new solutions are needed, e.g. crowdsourcing of
perspectives on events that exploit disagreement
Flickr: AmyJanelle iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
56. What do People
Disagree on?
are sub-events always mere parts?
are “mentions” meaningful for events?
are events coreferential across documents?
(e.g. perspectives, observations)
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
57. the bombing targeted a housing development
in Baghdad, killing 3 and injuring 13
indistinguishable by people, confusable:
is bombing part of killing, or killing part of bombing?
What about targeting?
“merelogically extensional” (i.e arbitrary):
container bursting into fragments as a result of explosion
some events don’t exist:
an action by military forces prevented the bombing.
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
58. Disagreement Framework
• ontology: disagreements on the basic status of events
themselves as referents of linguistic utterances, e.g. are
people events or do events exist at all.
• granularity: disagreements that result from issues of
granularity, e.g. the location being a country, region, or city,
the time being a day, week, month, etc.
• interpretation: disagreements that result from (non-
granular) ambiguity, differences in perspective, or error in
interpreting an expression, e.g. classifying a person as a
terrorist/hero, ”October Revolution” took place in September.
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
59. Disagreement Framework
• ontology: disagreements on the basic status of events
themselves as referents of linguistic utterances, e.g. are
people events or do events exist at all.
• granularity: disagreements that result from issues of
granularity, e.g. the location being a country, region, or city,
the time being a day, week, month, etc.
• interpretation: disagreements that result from (non-
granular) ambiguity, differences in perspective, or error in
interpreting an expression, e.g. classifying a person as a
terrorist/hero, ”October Revolution” took place in September.
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
61. iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
62. iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
63. Event Participants
Disagreement
Israeli
Prime minister 10%
50% Government
Benjamin
Netanyahu Israeli Cabinet 15%
his Cabinet 15%
35%
Benjamin {TOLD}
Netanyahu Benjamin
Israeli Prime Netanyahu’s 5%
15% minister Cabinet
Cabinet 45%
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
64. Temporal Disagreement
Prime minister
50% Benjamin 50%
Sunday
Netanyahu
March 1, 1998 25%
35%
Benjamin {TOLD} March 1998 15%
Netanyahu
Spring 1998 5%
Israeli Prime
15% minister
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
65. Spatial Disagreement
Southern
35%
30% Israel Lebanon
{WILLING TO
WITHDRAW} Lebanon 45%
65% Israel's Northern
Frontier
Middle East 10%
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
66. Approach Principles
1. tolerate, capture & exploit disagreement
2. understand the range of disagreements by creating a space of possibilities with frequencies
& similarities
3. score the machine output based on where it falls in this space
4. adaptable to new annotation tasks
Flickr: auroille iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
67. it
seems
to
refer
to
an
Top
Israeli
officials
SENT
strong
does not inference
or
new
SIGNALS
Sunday
that
Israel
refer to communicated
feeling
wants
to
withdraw
from
southern
an event more
than
specific
Lebanon,
...
event.
a
group
of
people
did
refers to
something
specific
at
a
an event
specific
point
in
6me.
the
actors
in
ques6on
(top
Israeli
officials)
refers to
performed
an
ac6on
an event
during
a
specified
6me
(Sunday).
it
refers
to
what
the
israelis
did
on
sunday,
a
specific
6me.
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
68. it
is
not
a
par6cular
That
1978
resolu6on
calls
for
movement
that
has
or
is
Israel's
uncondi6onal
does not going
on
but
a
request
that
WITHDRAWAL
from
the
self-‐ refer to the
country
of
Israel
declared
security
zone
it
an event remove
their
forces
from
occupies
in
south
Lebanon,
...
the
zone
they
occupy.
does not
refer to
an event the
sentence
is
speaking
of
a
demand
for
a
withdrawal
that
had
not
yet
occurred.
refers to
an event
Because
it
is
describing
a
historical
issue
concerning
the
resolu6on
of
1978
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
69. The Dark Side of Crowdsourcing
Disagreement
• disagreement is beautiful, except when it results from spamming
• crowdsourcing has to account for people that want to get paid for
not doing any work
• spammers generate disagreement for the wrong reasons
• most spam detection requires gold standard
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
70. Spam or not?
• cut & paste from text
• identical to other explanations
• much shorter time than the average
• low trust value of the worker
• shorter than 5-6 words
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
71. Spam or not?
+ low worker trust
Prime
Minister
Benjamin
does not Because
being
told
Netanyahu
TOLD
his
Cabinet
on
refer to something
doesn't
Sunday
that
Israel
was
willing
to
... an event seem
like
an
event.
Top
Israeli
officials
sent
strong
new
+ low worker trust
signals
Sunday
that
Israel
wants
to
refers to Because
the
WAR
is
withdraw
from
southern
Lebanon,
an event being
described
as
a
where
a
costly
WAR
of
aTri6on
costly
event.
has
been
claiming
soldiers'
lives.
+ low worker trust
Top
Israeli
officials
sent
strong
new
Because
Israel
WANTS
signals
Sunday
that
Israel
WANTS
refers to
TO
WITHDRAW
from
TO
WITHDRAW
from
southern
an event
Lebanon.
Lebanon,
...
Top
Israeli
officials
sent
strong
new
+ short time
refers to Because
WANTS
TO
signals
Sunday
that
Israel
WANTS
an event WITHDRAW
is
an
ac6on.
TO
WITHDRAW
...
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
72. Motivation-Verification
Method
• 2-stage method:
• disagreement collection + motivation
• spam filtering = motivation judgement
• Additionally:
• sample the motivation stage to manually
extract gold standard for stage 2
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
73. Extraction of
Putative Events • a new way of
measuring ground
truth
input: Manual selection of
putative events Gold Questions
input:
•
output of A
Phase I:
Phase I:
a new set of semantic
features for learning
A. Collect event input:
annotations + output of A C. Filtering spam
motivations event annotations
in event extraction
input: input: input:
list of events list of events list of events
Phase III: Phase IV:
Phase II:
A. Collect event A. Collect event
A. Collect event types modalities +
+ motivations role fillers +
motivations motivations
input: input: input:
output of A output of A output of A
input: input: input:
output of A Manual output of A Manual output of A Manual
selection of selection of selection of
Gold Questions Gold Questions Gold Questions
Phase II: Phase IV:
Phase III:
B. Filtering spam B. Filtering spam B. Filtering spam
event types event modalities event role fillers
iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
74. Position
Artificially
restricting
humans d
Machines oes not h
will learn elp mach
from dive ines to le
arn.
rsity
Flickr: elkabong iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
75. Position
ts are p art of
s
ue ne sof even
greeme nt & vag
human disa vent se mantics
The the e
Flickr: elkabong iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
76. finally ...
n ny l
ty ra ifu
ut
t he be
a
n d is
e t
en
re em
isag
d
Flickr: elkabong iSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
77. Acknowledgements
Roxane
Segers
Iina Hellsten
Chiel van
den Akker Geertje Jacobs Frank de Bakker Marteen
Brinkerink
Bibiana
Armenta
Piek Vossen
Thomas Johan
Chris Welty Michiel
Guus Schreiber Ploeger Oomen
Hildebrand
Riste
Gligorov
Marieke
Lourens van Susan Jacco van Geert-Jan Lotte Belice
van Erp
der Meij Legêne Ossenbruggen Houben Baltussen
Monday, September 10, 12
78. Questions?
@laroyo
http://lora-aroyo.org
Monday, September 10, 12