OKE Challenge was hosted by European Semantic Web Conference ESWC, 3-7 June 2018, held in Heraklion, Crete, Greece (Aldemar Knossos Royal & Royal Villa). This work was supported by grants from the EU H2020 Framework Programme provided for the project HOBBIT (GA no. 688227).

1. Open Knowledge Extraction Challenge 2018
Kleanthi Georgala1
René Speck1
Axel-C. Ngonga Ngomo2
1
University of Leipzig, Data Science Group, Leipzig, Germany
2
University of Paderborn, Data Science Group, Paderborn, Germany,
5th June 2018
Speck et al. (Paderborn University) OKE 2018 5th June 2018 1 / 10

2. Open Knowledge Extraction (OKE) Challenge 2018
https://project-hobbit.eu/challenges/oke2018-challenge-eswc-2018
Aim:
Pushing further the state-of-the-art of knowledge extraction for the Semantic Web
by re-defining typical tasks from information and knowledge extraction with
specific Semantic Web requirements.
Participants:
Researchers and practitioners from academia as well as industry.
Evaluation:
Evaluation with the HOBBIT platform based on the FAIR data principles.
Speck et al. (Paderborn University) OKE 2018 5th June 2018 2 / 10

3. Knowledge Extraction
Knowledge Extraction for unstructured data
Named Entity Recognition (NER)
Named Entity Disambiguation and Linking (NED)
Relation Extraction (RE)
1 @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
2 @prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
3 @prefix nif: <http://persistence.uni-leipzig.org/nlp2rdf/ontologies/nif-core#> .
4
5 <http://example.com/doc1#char=0,90>
6 a nif:RFC5147String , nif:String , nif:Context ;
7 nif:beginIndex "0"^^xsd:nonNegativeInteger ;
8 nif:endIndex "90"^^xsd:nonNegativeInteger ;
9 nif:isString "Leibniz was born in Leipzig in 1646 and attended the University
of Leipzig from 1661-1666."@en .
Listing 1: Example request document.
Speck et al. (Paderborn University) OKE 2018 5th June 2018 3 / 10

4. Named Entity Recognition and Disambiguation
1 @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
2 @prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
3 @prefix itsrdf: <http://www.w3.org/2005/11/its/rdf#> .
4 @prefix dbr: <http://dbpedia.org/resource/> .
5 @prefix nif: <http://persistence.uni-leipzig.org/nlp2rdf/ontologies/nif-core#> .
6
7 <http://example.com/doc1#char=0,7>
8 a nif:RFC5147String , nif:String ;
9 nif:anchorOf "Leibniz"@en ;
10 nif:beginIndex "0"^^xsd:nonNegativeInteger ;
11 nif:endIndex "7"^^xsd:nonNegativeInteger ;
12 nif:referenceContext <http://example.com/doc1#char=0,90> ;
13 itsrdf:taIdentRef dbr:Gottfried_Wilhelm_Leibniz .
14
15 <http://example.com/doc1#char=20,27>
16 a nif:RFC5147String , nif:String ;
17 nif:anchorOf "Leipzig"@en ;
18 nif:beginIndex "20"^^xsd:nonNegativeInteger ;
19 nif:endIndex "27"^^xsd:nonNegativeInteger ;
20 nif:referenceContext <http://example.com/doc1#char=0,90> ;
21 itsrdf:taIdentRef dbr:Leipzig .
Listing 2: Example NER and NED annotations.
Speck et al. (Paderborn University) OKE 2018 5th June 2018 4 / 10

5. Relation Extraction
1 @prefix dbo: <http://dbpedia.org/ontology/> .
2 @prefix oa: <http://www.w3.org/ns/oa#> .
3 @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
4 @prefix dbr: <http://dbpedia.org/resource/> .
5
6 []
7 a oa:Annotation , rdf:Statement ;
8 rdf:subject dbr:Gottfried_Wilhelm_Leibniz ;
9 rdf:predicate dbo:birthPlace ;
10 rdf:object dbr::Leipzig ;
11 oa:hasTarget [
12 a oa:SpecificResource ;
13 oa:hasSource "http://example.com/doc1#char=0,90"
14 ] .
Listing 3: Example RE annotations.
Speck et al. (Paderborn University) OKE 2018 5th June 2018 5 / 10

6. OKE Challenge 2018 — Tasks
All tasks based on the DBpedia knowledge base.
Task 1 and Task 2
Focused and Broader Named Entity Identification and Linking
Reuse of the former challenge tasks
Task 1 entity types: Person, Place and Organisation
Task 2 entity types: Activity, Agent, ...,Event, ... (12 in total)
Task 3
Relation Extraction
Predicates: almaMater, birthPlace, ceo, ... (31 in total)
Predicates domain/range: Person, Place and Organisation (and subtypes)
Task 4
Knowledge Extraction
Focused Named Entity Identification and Linking and Relation Extraction
Speck et al. (Paderborn University) OKE 2018 5th June 2018 6 / 10

7. OKE Challenge 2018 — KPIs and Datasets
KPIs
Precision (Macro and Micro)
Recall (Macro and Micro)
F1-measure (Macro and Micro)
Average ms per document
Error count
Datasets
Task 1 and 2: reuse the datasets created in former challenge
Task 3 and 4: new datasets manually created
Speck et al. (Paderborn University) OKE 2018 5th June 2018 7 / 10

8. OKE Challenge 2018 — Participants
Task 3 Relation Extraction
Relation Extraction for Knowledge Base Completion: a Supervised Approach
(Héctor Cerezo-Costas and Manuela Martín-Vicente, Vigo University, Spain)
Baseline
Table : KPIs for RelExt and Baseline.
RelExt Baseline
avg ms per doc 799.973 513.42
error count 27 0
macro F1-Score 39.26 8.00
macro Precision 39.02 10.00
macro Recall 45.91 7.18
micro F1-Score 42.02 8.66
micro Precision 40.47 68.75
micro Recall 43.70 4.62
Speck et al. (Paderborn University) OKE 2018 5th June 2018 8 / 10

9. Thank you!
Questions?
Kleanthi Georgala
Dice Research Group
Leipzig, Germany
Speck et al. (Paderborn University) OKE 2018 5th June 2018 9 / 10

10. Acknowledgement
Speck et al. (Paderborn University) OKE 2018 5th June 2018 10 / 10