We present KOI (Knowledge Of Incidents), a system that given news articles as input, builds a knowledge graph (KOI-KG) of incidental events. KOI-KG can then be used to efficiently answer questions such as "How many killing incidents happened in 2017 that involve Sean?" The required steps in building the KG include: (i) document preprocessing involving word sense disambiguation, named-entity recognition, temporal expression recognition and normalization, and semantic role labeling; (ii) incidental event extraction and coreference resolution via document clustering; and (iii) KG construction and population. Slides made and presented by Paramita.

1. KOI at SemEval-2018 Task 5:
K O
I
Paramita Mirza Fariz Darari Rahmad Mahendra
paramita@mpi-inf.mpg.de. fariz@cs.ui.ac.id rahmad.mahendra@cs.ui.ac.id
1
SemEval-2018
International Workshop on Semantic Evaluation
New Orleans, LA, USA, June 5-6 2018

2. (1) “How many killing incidents happened in June 2016 in San Antonio, Texas?”
(2) “How many people were killed in June 2016 in San Antonio, Texas?”
2
Numerical answer:
(1) 2
(2) 2
Supporting documents:
Knowledge
Graph of
Incidents
News articles

3. KOI
System
Architecture
Evaluation
on
SemEval Task 5
3

4. (1) “How many killing incidents happened in June 2016 in San Antonio, Texas?”
(2) “How many people were killed in June 2016 in San Antonio, Texas?”
4
Document
Preprocessing
Event Extraction
and Coreference
Resolution
KG Construction
& Population
Knowledge
Graph of
Incidents
Question
Parsing
Query
Execution
News articles
Numerical answer:
(1) 2
(2) 2
Supporting documents:

5. Powered by:
5
Document
Preprocessing
Event Extraction
and Coreference
Resolution
KG Construction
& Population
News articles
Word sense disambiguation & entity linking
(Navigli & Ponzetto, 2012)

6. One man, 41-years-old John Doe, was fatally shot early Sunday morning in San Antonio, police said.
Powered by:
Word sense disambiguation & entity linking
(Navigli & Ponzetto, 2012)
6
Document
Preprocessing
Event Extraction
and Coreference
Resolution
KG Construction
& Population
News articles
Named entity recognition
(spacy.io)
HeidelTime
Time expression recognition
& normalization
(Strötgen & Gertz, 2013)
One man, 41-years-old John Doe, was fatally shot early Sunday morning in San Antonio, police said.
June 5, 2018
PERSON 03-06-2018
Semantic role labelling
(Collobert et al., 2011)
shot A1: One man , 41-years-old John Doe
AM-MNR: fatally
AM-TMP: early Sunday Morning
AM-LOC: in San Antonio
said A0: police
A1: One man , 41-years-old John Doe,
was fatally shot early Sunday Morning
in San Antonio

7. Document
Preprocessing
Event Extraction
and Coreference
Resolution
KG Construction
& Population
News articles
7
One man, 41-years-old John Doe, was fatally shot early Sunday morning in San Antonio, police said.One man, 41-years-old John Doe, was fatally shot early Sunday morning in San Antonio, police said.
June 5, 2018
PERSON 03-06-2018
shot A1: One man , 41-years-old John Doe
AM-MNR: fatally
AM-TMP: early Sunday Morning
AM-LOC: in San Antonio
said A0: police
A1: One man , 41-years-old John Doe,
was fatally shot early Sunday Morning
in San Antonio
said A0: police
A1: One man , 41-years-old John Doe,
was fatally shot early Sunday Morning
in San Antonio
Predicate-level event
shot A1: One man , 41-years-old John Doe
AM-MNR: fatally
AM-TMP: early Sunday Morning
AM-LOC: in San Antonio
• Identifying incident-related concepts  via path-
based WordNet similarity (Hirst et al., 1998) > 5.0
• Identifying event participants and their roles (e.g.,
victim)
• Identifying number of victims (except for suspect-
related predicates)
• Identifying and normalizing event time
shot A1: One man , 41-years-old John Doe
AM-MNR: fatally
AM-TMP: early Sunday Morning
AM-LOC: in San Antonio
shot A1: One man , 41-years-old John Doe
AM-MNR: fatally
AM-TMP: early Sunday Morning
AM-LOC: in San Antonio
shot A1: One man , 41-years-old John Doe
AM-MNR: fatally
AM-TMP: early Sunday Morning
AM-LOC: in San Antonio
Predicate-level event, type: killing

8. Document
Preprocessing
Event Extraction
and Coreference
Resolution
KG Construction
& Population
News articles
8
One man, 41-years-old John Doe, was fatally shot early Sunday morning in San Antonio, police said.One man, 41-years-old John Doe, was fatally shot early Sunday morning in San Antonio, police said.
June 5, 2018
PERSON 03-06-2018
shot A1: One man , 41-years-old John Doe
AM-MNR: fatally
AM-TMP: early Sunday Morning
AM-LOC: in San Antonio
said A0: police
A1: One man , 41-years-old John Doe,
was fatally shot early Sunday Morning
in San Antonio
said A0: police
A1: One man , 41-years-old John Doe,
was fatally shot early Sunday Morning
in San Antonio
Sentence-level event
shot A1: One man , 41-years-old John Doe
AM-MNR: fatally
AM-TMP: early Sunday Morning
AM-LOC: in San Antonio
shot A1: One man , 41-years-old John Doe
AM-MNR: fatally
AM-TMP: early Sunday Morning
AM-LOC: in San Antonio
shot A1: One man , 41-years-old John Doe
AM-MNR: fatally
AM-TMP: early Sunday Morning
AM-LOC: in San Antonio
shot A1: One man , 41-years-old John Doe
AM-MNR: fatally
AM-TMP: early Sunday Morning
AM-LOC: in San Antonio
shot A1: One man , 41-years-old John Doe
AM-MNR: fatally
AM-TMP: early Sunday Morning
AM-LOC: in San Antonio
shot A1: One man , 41-years-old John Doe
AM-MNR: fatally
AM-TMP: early Sunday Morning
AM-LOC: in San Antonio
shot A1: One man , 41-years-old John Doe
AM-MNR: fatally
AM-TMP: early Sunday Morning
AM-LOC: in San Antonio
shot A1: One man , 41-years-old John Doe
AM-MNR: fatally
AM-TMP: early Sunday Morning
AM-LOC: in San Antonio
killing
injuring
injuring
fire
Document-level event (incident)
• One incident per incident type per
document
• Determine incident location
• Aggregate event participants per
incident type
• Aggregate number of victims per
incident type
• Determine incident time per
incident type
shot A1: One man , 41-years-old John Doe
AM-MNR: fatally
AM-TMP: early Sunday Morning
AM-LOC: in San Antonio
shot A1: One man , 41-years-old John Doe
AM-MNR: fatally
AM-TMP: early Sunday Morning
AM-LOC: in San Antonio
shot A1: One man , 41-years-old John Doe
AM-MNR: fatally
AM-TMP: early Sunday Morning
AM-LOC: in San Antonio
shot A1: One man , 41-years-old John Doe
AM-MNR: fatally
AM-TMP: early Sunday Morning
AM-LOC: in San Antonio
killing
injuring
fire
Predicate-level event, type: killing

9. Document
Preprocessing
Event Extraction
and Coreference
Resolution
KG Construction
& Population
News articles
9
injuring
fire
Cross-document event coreference resolution
via document clustering
killing
injuring
fire
dct1: June 4, 2018 dct2: June 5, 2018
≈
|dct1-dct2|≤ 3 days
Cosine similarity of TF-IDF-based vectors of:
• BabelNet senses
• spaCy’s PERSON and GPE

10. KOI-KG  https://koi.cs.ui.ac.id/incidents
 Powered by
10
Document
Preprocessing
Event Extraction
and Coreference
Resolution
KG Construction
& Population
News articles
injuring
fire
killing
injuring
fire
dct1: June 4, 2018 dct2: June 5, 2018
killing injuring fire
supported by
John Doe
1 victim
03-06-2018 San Antonio
≈
|dct1-dct2|≤ 3 days

11. (1) “How many killing incidents happened in June 2016 in San Antonio, Texas?”
(2) “How many people were killed in June 2016 in San Antonio, Texas?”
11
Document
Preprocessing
Event Extraction
and Coreference
Resolution
KG Construction
& Population
Knowledge
Graph of
Incidents
Query
Execution
News articles
Numerical answer:
(1) 2
(2) 2
Supporting documents:
SPARQL query
Question
Parsing
SELECT ?event ?document
WHERE {
?event koi:eventType koi:killing .
?event koi:eventDate [
koi:month "06" ;
koi:year "2016" ] .
?event koi:location [
koi:city
<http://dbpedia.org/resource/San_Antonio> ;
koi:state
<http://dbpedia.org/resource/Texas> ] .
?document koi:event ?event .
}

12. KOI
System
Architecture
Evaluation
on
SemEval Task 5
12

13. System
Subtask S1 Subtask S2 Subtask S3
%Ans Doc-F1 %Ans Doc-F1 %Ans Doc-F1
Baseline 16.5 67.3 100.0 26.4 - -
KOI 44.2 83.0 67.5 55.2 66.6 69.6
NAI-SEA 100.0 78.3 100.0 50.5 100.0 63.6
FEUP 100.0 24.7 100.0 30.5 100.0 26.8
NewsReader 51.6 46.2 100.0 36.9 100.0 26.8
13
KOI v2 44.2 83.0 100.0 51.2 100.0 49.1
KOI v3 55.1 85.7 100.0 54.8 100.0 50.9
KOI v2: answer with 0 and empty set of answer documents, if cannot find an answer
KOI v3: bug (fire burning & job firing incident types) is fixed, improved incident time identification

14. KOI v3
Micro-averaged
P R F1
Answered questions 86.6 74.0 79.8
All questions 86.6 41.6 56.2
14
Killing 88.5 43.2 58.1
Injuring 82.8 37.4 51.5
Job firing 100.0 8.7 16.0
Fire burning 96.9 66.2 78.7
Participant 84.8 43.0 57.0
Location 89.1 39.4 54.6
Time 86.0 42.4 56.8

15. System
Subtask S2 Subtask S3
%Ans Accuracy RMSE %Ans Accuracy RMSE
Baseline 18.3 8.5 - -
KOI 67.5 20.4 6.2 66.6 19.3 7.9
FEUP 100.0 26.4 6.1 100.0 30.4 478.7
NewsReader 100.0 21.9 44.0 100.0 21.1 296.5
NAI-SEA 100.0 17.4 20.6 100.0 20.2 13.5
15
KOI v2 100.0 25.6 5.2 100.0 24.8 7.1
KOI v3 100.0 27.4 5.3 100.0 23.0 7.7
KOI v2: answer with 0 and empty set of answer documents, if cannot find an answer
KOI v3: bug (fire burning & job firing incident types) is fixed, improved incident time identification

16. 16
(1) “How many killing incidents happened in June 2016 in San Antonio, Texas?” 2
(2) “How many people were killed in June 2016 in San Antonio, Texas?” 1

17.  KOI  Knowledge Graph of Incidents  to be used for efficiently answering
numerical questions about domain-specific events
 Fully unsupervised approach, utilizing already existing NLP tools
 Simple cross-document event coreference method via document clustering
 One main event/incident per document assumption  does not always hold
 Low performance on identifying the existence of event participants for specific
roles
 “Two boys and a girl were shot while…”
Thank you!
17

18.  When KOI yields perfect (non-empty) sets of answer documents:
 On answering number of incidents (cross-document event coreference)
 On answering number of victims (counting event participants)
 34.3% correct number
18
correct number
overestimate
underestimate
from only counting
from only numeral mentions
combination