I-Semantics 2012 Keynote

Dial E for EventsLora AroyoMonday, September 10, 12

Observation events are important events are omni-present in events carry diﬀerent points of view th in ew ou orl r p d, e er .g. so na new ne l li s, tw ve sc or s, ien kin e.g ce g .s oc etc ial . Flickr: elkabong iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Position ts are p art of s ue ne sof even greeme nt & vag human disa vent se mantics The the e Flickr: elkabong iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Objects vs. Events events perdure = their parts exist at different time points objects endure = they have all their parts at all points in time objects are wholly present at any point in time, events unfold over time Flickr: vanilllaph iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Events are important create context for objects, e.g. people, locations, organizations, etc. Lora Aroyo @laroyo iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Events are important create meaning for objects, e.g. artifacts, pictures, videos. iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Events are important link concepts, objects, and stories. iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Events in the World events anchor the information we consume daily Flickr: craftydogma iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Events @ Google News iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Julian Assange’s Extradition Row iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Lance Armstrong’s Doping Fight iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

The Arab Spring iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Events @ Social Web iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Events are Vague Humans have no clear notion of what events are iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

“event is a signiﬁcant "happening" or gathering of people. I would deﬁne a "happening" as an event if the group of people gathered were united in one common goal.” We Asked the Crowd What an EVENT is Flickr: massimo vitali iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Event is a happening, which can be scheduled or unscheduled. An earthquake or ﬁre happens (unscheduled). A wedding or birthday party (scheduled). It is an occasion that is unusual and tends to be memorable. We Asked the Crowd What an EVENT is iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

“An event would be any occurrence where physical action has taken place. It may be a single, momentary instance (I sneezed), or it may span a period of time (the festival ran for four hours). An event may also be made up of a number of smaller events, such as a day at school is an event, but each individual class is also an event itself. Basically an event must have a physical action over any delimited time span.” We Asked the Crowd What an EVENT is iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

“Event can refer to many things such as: An observable occurrence, phenomenon or an extraordinary occurrence.” “an event is an incident thats very important or monumental” “A planned public or social get together or occasion.” “An event is something occurring at a speciﬁc time and/or date to celebrate or recognize a particular occurrence.” “a location where something like a function is held. you could tell if something is an event if there people gathering for a purpose.” We Asked the Crowd What an EVENT is iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

What do Experts think an EVENT is? “an event is the exempliﬁcation of a property by a substance at a given time” Jaegwon Kim, 1966 “events are changes that physical objects undergo” Lawrence Lombard, 1981 “events are properties of spatiotemporal regions”, David Lewis, 1986under30ceo.com iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Event-centric Projects how events can be detected & extracted from natural language text how those extracted events are represented for use on the semantic web how to identify the same events in diﬀerent sources how to capture diﬀerent perspectives iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Activists prominent on the new web through diﬀerent channels by nature multi- perspective, biased & emotional iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

“All protest events Greenpeace participated in.” Mapping Online blogs, news, activists websites Networks of Activism iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

• build visualizations • create appropriate analytics“All protest events • answer questions of end users & social in.” Greenpeace participated scientists Mapping Online blogs, news, activists websites Networks of Activism iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Extracting Historical Events • objects (digitized • What happened artworks and before/after? artifacts) • Who does what, • events (concrete when, and where? particulars) • All bomb attacks in • entities (actors, the 1950s locations, periods) • In what events did • narratives Indonesia participate? (organization of events) • ‘Grand narratives’ iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

• generate meaningful event sequences • capture the diﬀerent perspectives • serve both end users & history researchers iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Timelines from Text “al-Qaeda activities in Syria” 4 right, 2 wrong, 3 missing events two have no explicit times & are in the wrong order One involved al-Qaeda but took place in Jordan on the Syrian border does a fuzzy task require fuzzier metrics? iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Machine Reading build event timelines from text in 2 example domains • NFL: news articles on football; • Ontology: 4 classes, 20 relations • Intel: news articles on terrorist events; • Ontology: 20 Classes, 50 relations iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Why is event semantics hard? iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

According to NLP tradition Gather Extract 1 2 your source material events and properties Analyze Visualize 3 4 statistics, timelines, etc. ﬁnd links between events iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

but for events we stumble Gather Extract 1 2 your source material events and properties Analyze Visualize 3 4 statistics, timelines, etc. ﬁnd links between events iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

but for events we stumble Gather Extract experts typically: 1 2 deﬁne a problem annotate ground truth your source material events and properties train Analyze Visualize 3 evaluate 4 statistics, timelines, etc. ﬁnd links between events iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Closed World Dictatorship 1. domain experts deﬁne the meaning 2. using limited vocabulary 3. aim for agreement to ﬁx the problem of high disagreement for events experts enforce more tyranny - stricter rules comparatively little annotated data for training & evaluation of event detection systems iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

But the World is Open 1. events have multiple dimensions 2. each dimension has levels of granularity 3. people have diﬀerent views on both all this leads to very complex semantics iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

and our goal is ... 1. not to enforce agreement 2. to capture diﬀerent view points 3. to teach machines to reason in the disagreement space Flickr: elkabong iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Position Artiﬁcially restricting humans d Machines oes not h will learn elp mach from dive ines to le arn. rsity Flickr: elkabong iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Professional Dictatorship of the Closed World • Museum, libraries, archives & researchers have been dominating the views. • Controlled vocabularies & annotation schemes were leading. • Professionals enforced agreement among themselves. • End-users needs & tasks are not considered. iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

there is a tiny overlap between end-user terminology & professional annotations the latter are typically coarse-grained & refer to entire object / topic Flickr: ganzelka iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

only 1,900 tags (32,200 in total) match in vocabularies 257 in people (83 validated) 1,661 in geo (666 validated) 9,796 validated, but no match in professional vocabulary 8% professional vocab 23 % lexical vocab 63% meaningful Google matches iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Amateur democracy of the Open World • Once the Web opened the world of information, professional dictatorship clashed with end-users democracy. • What professionals consider interesting, relevant or important does not match what users think of it. • Amateurs cannot ﬁnd what they were searching for. (c) banksy iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

people are interested in diﬀerent annotation categories than the professionals Flickr: ganzelka iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Tag sample: 1,343 verified tags of 5 random video fragments Abstract General Specific Total Video aspects that are Who 10 166 177 31% described by those tags: 5 12 What 73 563 12 57% non-visual (0) Where 0 68 8 7% perceptual (11), e.g. color When 4 31 31 6 5% conceptual (1,332) Total 7% 74% 9% Object tags (1,313) Scene tags (30) 195 tags (adverbs & adjectives) couldn’t be classified iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Harnessing the Crowd • we include the user’s opinion as ﬁrst class citizens. • this brings the need to combine all these (diﬀerent) opinions into a system of opinions that makes sense. • new solutions are needed, e.g. crowdsourcing of perspectives on events that exploit disagreementFlickr: AmyJanelle iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

What do People Disagree on? are sub-events always mere parts? are “mentions” meaningful for events? are events coreferential across documents? (e.g. perspectives, observations) iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

the bombing targeted a housing development in Baghdad, killing 3 and injuring 13 indistinguishable by people, confusable: is bombing part of killing, or killing part of bombing? What about targeting? “merelogically extensional” (i.e arbitrary): container bursting into fragments as a result of explosion some events don’t exist: an action by military forces prevented the bombing. iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Disagreement Framework • ontology: disagreements on the basic status of events themselves as referents of linguistic utterances, e.g. are people events or do events exist at all. • granularity: disagreements that result from issues of granularity, e.g. the location being a country, region, or city, the time being a day, week, month, etc. • interpretation: disagreements that result from (non- granular) ambiguity, diﬀerences in perspective, or error in interpreting an expression, e.g. classifying a person as a terrorist/hero, ”October Revolution” took place in September. iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Granularity Disagreement spatial, temporal, participants compositional, classiﬁcational iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Event Participants Disagreement Israeli Prime minister 10% 50% Government Benjamin Netanyahu Israeli Cabinet 15% his Cabinet 15% 35% Benjamin {TOLD} Netanyahu Benjamin Israeli Prime Netanyahu’s 5% 15% minister Cabinet Cabinet 45% iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Temporal Disagreement Prime minister 50% Benjamin 50% Sunday Netanyahu March 1, 1998 25% 35% Benjamin {TOLD} March 1998 15% Netanyahu Spring 1998 5% Israeli Prime 15% minister iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Spatial Disagreement Southern 35%30% Israel Lebanon {WILLING TO WITHDRAW} Lebanon 45%65% Israels Northern Frontier Middle East 10% iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Approach Principles 1. tolerate, capture & exploit disagreement 2. understand the range of disagreements by creating a space of possibilities with frequencies & similarities 3. score the machine output based on where it falls in this space 4. adaptable to new annotation tasks Flickr: auroille iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

it seems to refer to an Top Israeli officials SENT strong does not inference or new SIGNALS Sunday that Israel refer to communicated feeling wants to withdraw from southern an event more than specific Lebanon, ... event. a group of people did refers to something specific at a an event specific point in 6me. the actors in ques6on (top Israeli officials) refers to performed an ac6on an event during a specified 6me (Sunday). it refers to what the israelis did on sunday, a specific 6me. iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

it is not a par6cular That 1978 resolu6on calls for movement that has or is Israels uncondi6onal does not going on but a request that WITHDRAWAL from the self-‐ refer to the country of Israel declared security zone it an event remove their forces from occupies in south Lebanon, ... the zone they occupy. does not refer to an event the sentence is speaking of a demand for a withdrawal that had not yet occurred. refers to an event Because it is describing a historical issue concerning the resolu6on of 1978 iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

The Dark Side of Crowdsourcing Disagreement • disagreement is beautiful, except when it results from spamming • crowdsourcing has to account for people that want to get paid for not doing any work • spammers generate disagreement for the wrong reasons • most spam detection requires gold standard iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Spam or not? • cut & paste from text • identical to other explanations • much shorter time than the average • low trust value of the worker • shorter than 5-6 words iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Spam or not? + low worker trust Prime Minister Benjamin does not Because being told Netanyahu TOLD his Cabinet on refer to something doesnt Sunday that Israel was willing to ... an event seem like an event. Top Israeli officials sent strong new + low worker trust signals Sunday that Israel wants to refers to Because the WAR is withdraw from southern Lebanon, an event being described as a where a costly WAR of aTri6on costly event. has been claiming soldiers lives. + low worker trust Top Israeli officials sent strong new Because Israel WANTS signals Sunday that Israel WANTS refers to TO WITHDRAW from TO WITHDRAW from southern an event Lebanon. Lebanon, ... Top Israeli officials sent strong new + short time refers to Because WANTS TO signals Sunday that Israel WANTS an event WITHDRAW is an ac6on. TO WITHDRAW ... iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Motivation-Veriﬁcation Method • 2-stage method: • disagreement collection + motivation • spam ﬁltering = motivation judgement • Additionally: • sample the motivation stage to manually extract gold standard for stage 2 iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Extraction of Putative Events • a new way of measuring ground truth input: Manual selection of putative events Gold Questions input: • output of A Phase I: Phase I: a new set of semantic features for learning A. Collect event input: annotations + output of A C. Filtering spam motivations event annotations in event extraction input: input: input: list of events list of events list of events Phase III: Phase IV: Phase II: A. Collect event A. Collect event A. Collect event types modalities + + motivations role ﬁllers + motivations motivations input: input: input: output of A output of A output of A input: input: input: output of A Manual output of A Manual output of A Manual selection of selection of selection of Gold Questions Gold Questions Gold Questions Phase II: Phase IV: Phase III: B. Filtering spam B. Filtering spam B. Filtering spam event types event modalities event role ﬁllers iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Position Artiﬁcially restricting humans d Machines oes not h will learn elp mach from dive ines to le arn. rsity Flickr: elkabong iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Position ts are p art of s ue ne sof even greeme nt & vag human disa vent se mantics The the e Flickr: elkabong iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

ﬁnally ... n ny l ty ra ifu ut t he be a n d is e t en re em isag d Flickr: elkabong iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Acknowledgements Roxane Segers Iina Hellsten Chiel van den Akker Geertje Jacobs Frank de Bakker Marteen Brinkerink Bibiana Armenta Piek Vossen Thomas Johan Chris Welty Michiel Guus Schreiber Ploeger Oomen Hildebrand Riste Gligorov Marieke Lourens van Susan Jacco van Geert-Jan Lotte Belice van Erp der Meij Legêne Ossenbruggen Houben BaltussenMonday, September 10, 12

Questions? @laroyo http://lora-aroyo.orgMonday, September 10, 12

http://agora.cs.vu.nl	958
http://mics.fim.uni-passau.de	165
http://www.scoop.it	31
https://www.dimis.fim.uni-passau.de	22
http://www.linkedin.com	18
https://twitter.com	17
http://localhost	5
http://webcache.googleusercontent.com	1
http://tweetedtimes.com	1
http://translate.googleusercontent.com	1

I-Semantics 2012 Keynote

by Lora Aroyo on Sep 07, 2012

Accessibility

Categories

Upload Details

Usage Rights

10 Embeds 1,219

Statistics