I-Semantics 2012 Keynote

Dial E
for
Events

Lora Aroyo

Monday, September 10, 12

Observation
events are important
events are omni-present
in events carry diﬀerent points of view
th
in ew
ou orl
r p d, e
er .g.
so
na new
ne l li s,
tw ve sc
or s, ien
kin e.g ce
g .s
oc etc
ial .
Flickr: elkabong iSemantics2012 Lora Aroyo @laroyo

Position
ts are p art of
s
ue ne sof even
greeme nt & vag
human disa vent se mantics
The the e


Objects vs. Events

events perdure = their parts exist at different time points
objects endure = they have all their parts at all points in time

objects are wholly present at any point in time, events unfold over time
Flickr: vanilllaph iSemantics2012 Lora Aroyo @laroyo

Events are important
create context for objects, e.g. people, locations, organizations, etc.
Lora Aroyo @laroyo iSemantics2012 Lora Aroyo @laroyo

create meaning for objects, e.g. artifacts, pictures, videos.
iSemantics2012 Lora Aroyo @laroyo

link concepts, objects, and stories.

Events in the World
events anchor the information we
consume daily

Flickr: craftydogma iSemantics2012 Lora Aroyo @laroyo

Events @ Google News


Julian Assange’s Extradition Row


Lance Armstrong’s Doping Fight


The Arab Spring


Events @ Social Web


Events are Vague
Humans have no clear notion of what events are


“event is a signiﬁcant
"happening" or
gathering of people. I
would deﬁne a
"happening" as an event
if the group of people
gathered were united in
one common goal.”

We Asked the Crowd What an EVENT is
Flickr: massimo vitali iSemantics2012 Lora Aroyo @laroyo

Event is a happening,
which can be scheduled or
unscheduled. An
earthquake or ﬁre
happens (unscheduled). A
wedding or birthday
party (scheduled). It is an
occasion that is unusual
and tends to be
memorable.


“An event would be any
occurrence where physical action
has taken place. It may be a single,
momentary instance (I sneezed),
or it may span a period of time
(the festival ran for four hours). An
event may also be made up of a
number of smaller events, such
as a day at school is an event, but
each individual class is also an
event itself. Basically an event must
have a physical action over any
delimited time span.”


“Event can refer to many things such as: An observable
occurrence, phenomenon or an extraordinary occurrence.”

“an event is an incident that's very important or monumental”

“A planned public or social get together or occasion.”

“An event is something occurring at a speciﬁc time and/or
date to celebrate or recognize a particular occurrence.”

“a location where something like a function is held. you could tell
if something is an event if there people gathering for a purpose.”


What do Experts think an EVENT is?

“an event is the exempliﬁcation of a property by a substance at a given time” Jaegwon Kim, 1966

“events are changes that physical objects undergo” Lawrence Lombard, 1981

“events are properties of spatiotemporal regions”, David Lewis, 1986

under30ceo.com iSemantics2012 Lora Aroyo @laroyo

Event-centric Projects
how events can be detected & extracted from
natural language text

how those extracted events are represented for use
on the semantic web

how to identify the same events in diﬀerent sources

how to capture diﬀerent perspectives


Activists

prominent on
the new web
through
diﬀerent
channels

by nature
multi-
perspective,
biased &
emotional


“All protest events
Greenpeace participated in.”

Mapping Online blogs, news, activists websites

Networks of Activism

• build visualizations
• create appropriate analytics“All protest events
• answer questions of end users & social in.”
Greenpeace participated
scientists

Mapping Online blogs, news, activists websites

Networks of Activism

Extracting
Historical Events
• objects (digitized • What happened
artworks and before/after?
artifacts)
• Who does what,
• events (concrete when, and where?
particulars)
• All bomb attacks in
• entities (actors, the 1950s
locations, periods) • In what events did

• narratives Indonesia
participate?
(organization of
events) • ‘Grand narratives’


• generate meaningful event sequences
• capture the diﬀerent perspectives
• serve both end users & history researchers


Timelines
from Text
“al-Qaeda activities in Syria”

4 right, 2 wrong, 3 missing events
two have no explicit times & are in the
wrong order
One involved al-Qaeda but took place in
Jordan on the Syrian border

does a fuzzy task require fuzzier metrics?


Machine Reading
build event timelines from text in 2 example
domains

• NFL: news articles on football;
• Ontology: 4 classes, 20 relations

• Intel: news articles on terrorist events;
• Ontology: 20 Classes, 50 relations


Why is event
semantics hard?


According to NLP tradition
Gather Extract
1 2

your source material events and properties

Analyze Visualize
3 4

statistics, timelines, etc.
ﬁnd links between events


but for events we stumble
Gather Extract
1 2


Analyze Visualize
3 4



but for events we stumble
Gather Extract
experts typically:
1 2

deﬁne a problem
annotate ground truth

train
Analyze Visualize
3 evaluate 4



Closed World Dictatorship

1. domain experts deﬁne the meaning
2. using limited vocabulary
3. aim for agreement

to ﬁx the problem of high disagreement for events
experts enforce more tyranny - stricter rules
comparatively little annotated data for training & evaluation of event
detection systems

But the World is Open

1. events have multiple dimensions
2. each dimension has levels of granularity
3. people have diﬀerent views on both

all this leads to very complex semantics


and our goal is ...

1. not to enforce agreement
2. to capture diﬀerent view points
3. to teach machines to reason in the disagreement space


Position
Artiﬁcially
restricting
humans d
Machines oes not h
will learn elp mach
from dive ines to le
arn.
rsity


Professional Dictatorship of the Closed World

• Museum, libraries, archives & researchers have been dominating the views.
• Controlled vocabularies & annotation schemes were leading.
• Professionals enforced agreement among themselves.
• End-users needs & tasks are not considered.

there is a tiny overlap between end-user terminology & professional annotations
the latter are typically coarse-grained & refer to entire object / topic
Flickr: ganzelka iSemantics2012 Lora Aroyo @laroyo

only 1,900 tags (32,200 in total) match in vocabularies
257 in people (83 validated) 1,661 in geo (666 validated)
9,796 validated, but no match in professional vocabulary
8% professional vocab
23 % lexical vocab
63% meaningful Google matches


Amateur democracy of the Open World

• Once the Web opened the world of information, professional
dictatorship clashed with end-users democracy.

• What professionals consider interesting, relevant or important
does not match what users think of it.

• Amateurs cannot ﬁnd what they were searching for.
(c) banksy iSemantics2012 Lora Aroyo @laroyo

people are interested in diﬀerent annotation
categories than the professionals
Flickr: ganzelka iSemantics2012 Lora Aroyo @laroyo

Tag sample:
1,343 verified tags of 5 random video fragments

Abstract General Specific Total
Video aspects that are
Who 10 166 177 31%
described by those tags: 5 12
What 73 563 12 57%
non-visual (0) Where 0 68 8 7%
perceptual (11), e.g. color When 4 31
31 6 5%
conceptual (1,332) Total 7% 74% 9%

Object tags (1,313)
Scene tags (30)

195 tags
(adverbs & adjectives)
couldn’t be classified


Harnessing the Crowd

• we include the user’s opinion as ﬁrst class citizens.

• this brings the need to combine all these (diﬀerent)
opinions into a system of opinions that makes sense.

• new solutions are needed, e.g. crowdsourcing of
perspectives on events that exploit disagreement
Flickr: AmyJanelle iSemantics2012 Lora Aroyo @laroyo

What do People
Disagree on?
are sub-events always mere parts?
are “mentions” meaningful for events?
are events coreferential across documents?
(e.g. perspectives, observations)


the bombing targeted a housing development
in Baghdad, killing 3 and injuring 13

indistinguishable by people, confusable:
is bombing part of killing, or killing part of bombing?
What about targeting?

“merelogically extensional” (i.e arbitrary):
container bursting into fragments as a result of explosion

some events don’t exist:
an action by military forces prevented the bombing.


Disagreement Framework
• ontology: disagreements on the basic status of events
themselves as referents of linguistic utterances, e.g. are
people events or do events exist at all.

• granularity: disagreements that result from issues of
granularity, e.g. the location being a country, region, or city,
the time being a day, week, month, etc.

• interpretation: disagreements that result from (non-
granular) ambiguity, diﬀerences in perspective, or error in
interpreting an expression, e.g. classifying a person as a
terrorist/hero, ”October Revolution” took place in September.


Granularity Disagreement

spatial, temporal, participants
compositional, classiﬁcational

Event Participants
Disagreement
Israeli
Prime minister 10%
50% Government
Benjamin
Netanyahu Israeli Cabinet 15%
his Cabinet 15%
35%
Benjamin {TOLD}
Netanyahu Benjamin
Israeli Prime Netanyahu’s 5%
15% minister Cabinet
Cabinet 45%

Temporal Disagreement

Prime minister
50% Benjamin 50%
Sunday
Netanyahu
March 1, 1998 25%
35%
Benjamin {TOLD} March 1998 15%
Netanyahu
Spring 1998 5%
Israeli Prime
15% minister


Spatial Disagreement

Southern
35%
30% Israel Lebanon
{WILLING TO
WITHDRAW} Lebanon 45%
65% Israel's Northern
Frontier
Middle East 10%


Approach Principles
1. tolerate, capture & exploit disagreement
2. understand the range of disagreements by creating a space of possibilities with frequencies
& similarities
3. score the machine output based on where it falls in this space
4. adaptable to new annotation tasks
Flickr: auroille iSemantics2012 Lora Aroyo @laroyo

it
seems
to
refer
to
an

Top
Israeli
officials
SENT
strong

does not inference
or

new
SIGNALS
Sunday
that
Israel

refer to communicated
feeling

wants
to
withdraw
from
southern

an event more
than
specific

Lebanon,
...
event.

a
group
of
people
did

refers to
something
specific
at
a

an event
specific
point
in
6me.

the
actors
in
ques6on

(top
Israeli
officials)

refers to
performed
an
ac6on

an event
during
a
specified
6me

(Sunday).

it
refers
to
what
the

israelis
did
on
sunday,

a
specific
6me.

it
is
not
a
par6cular

That
1978
resolu6on
calls
for

movement
that
has
or
is

Israel's
uncondi6onal
does not going
on
but
a
request
that

WITHDRAWAL
from
the
self-‐ refer to the
country
of
Israel

declared
security
zone
it
an event remove
their
forces
from

occupies
in
south
Lebanon,
...
the
zone
they
occupy.
does not
refer to
an event the
sentence
is
speaking
of

a
demand
for
a
withdrawal

that
had
not
yet
occurred.

refers to
an event
Because
it
is
describing
a

historical
issue
concerning

the
resolu6on
of
1978


The Dark Side of Crowdsourcing
Disagreement

• disagreement is beautiful, except when it results from spamming
• crowdsourcing has to account for people that want to get paid for
not doing any work
• spammers generate disagreement for the wrong reasons
• most spam detection requires gold standard

Spam or not?

• cut & paste from text
• identical to other explanations
• much shorter time than the average
• low trust value of the worker
• shorter than 5-6 words

Spam or not?
+ low worker trust
Prime
Minister
Benjamin
does not Because
being
told

Netanyahu
TOLD
his
Cabinet
on
refer to something
doesn't

Sunday
that
Israel
was
willing
to
... an event seem
like
an
event.

Top
Israeli
officials
sent
strong
new
+ low worker trust
signals
Sunday
that
Israel
wants
to
refers to Because
the
WAR
is

withdraw
from
southern
Lebanon,
an event being
described
as
a

where
a
costly
WAR
of
aTri6on
costly
event.
has
been
claiming
soldiers'
lives.
+ low worker trust
Top
Israeli
officials
sent
strong
new

Because
Israel
WANTS

signals
Sunday
that
Israel
WANTS
refers to
TO
WITHDRAW
from

TO
WITHDRAW
from
southern
an event
Lebanon.
Lebanon,
...

Top
Israeli
officials
sent
strong
new
+ short time
refers to Because
WANTS
TO

signals
Sunday
that
Israel
WANTS

an event WITHDRAW
is
an
ac6on.
TO
WITHDRAW

...

Motivation-Veriﬁcation
Method
• 2-stage method:
• disagreement collection + motivation
• spam ﬁltering = motivation judgement
• Additionally:
• sample the motivation stage to manually
extract gold standard for stage 2


Extraction of
Putative Events • a new way of
measuring ground
truth
input: Manual selection of
putative events Gold Questions
input:

•
output of A
Phase I:
Phase I:
a new set of semantic
features for learning
A. Collect event input:
annotations + output of A C. Filtering spam
motivations event annotations
in event extraction
input: input: input:
list of events list of events list of events

Phase III: Phase IV:
Phase II:
A. Collect event A. Collect event
A. Collect event types modalities +
+ motivations role ﬁllers +
motivations motivations

output of A output of A output of A
output of A Manual output of A Manual output of A Manual
selection of selection of selection of
Gold Questions Gold Questions Gold Questions

Phase II: Phase IV:
Phase III:
B. Filtering spam B. Filtering spam B. Filtering spam
event types event modalities event role ﬁllers


ﬁnally ...

n ny l
ty ra ifu
ut
t he be
a
n d is
e t
en
re em
isag
d

Acknowledgements
Roxane
Segers
Iina Hellsten
Chiel van
den Akker Geertje Jacobs Frank de Bakker Marteen
Brinkerink
Bibiana
Armenta

Piek Vossen
Thomas Johan
Chris Welty Michiel
Guus Schreiber Ploeger Oomen
Hildebrand

Riste
Gligorov
Marieke
Lourens van Susan Jacco van Geert-Jan Lotte Belice
van Erp
der Meij Legêne Ossenbruggen Houben Baltussen

Questions?

@laroyo
http://lora-aroyo.org

I-Semantics 2012 Keynote

Recommended

Recommended

More Related Content

More from Lora Aroyo

More from Lora Aroyo (20)

Recently uploaded

Recently uploaded (20)

I-Semantics 2012 Keynote