● Subtask S1: Event Questions with Answer=1
○ Which killing incident happened in 2014 in Columbus, OH?
● Subtask S2 Event Questions with Answer=any number
○ How many killing incidents happened in 2016 in Columbus, MS?
● Subtask S3 Participant Questions with Answer=any number
○ How many people were killed in 2016 in Columbus, MS?
Optionally, participants could also provide the text mentions of events in the
3 Domains: gun violence, fire
The data consists of local news
articles and reports on small-world
events and participants.
The data is split into trial and
1. Incident-level evaluation – Did you get the question right?
a. Accuracy (exact answer matching)
2. Document-level evaluation – How many of the gold documents did you
a. Precision, recall and F1-value
3. Mention-level evaluation – Did you extract the correct coreference chains?
a. BLANC, CEAF_E, CEAF_M, MUC, BCUB
Task 5 had four participating systems.
NewsReader and ID-DE built a knowledge graph of incidents, which was then
queried for each question based on its constraints.
NAI-SEA performed clustering on a document level. FEUP applied a supervised
approach to address participants and locations separately.
All systems aided their clustering by extracting temporal expressions, word
senses, participants, locations, semantic roles, ...