"Contextualized Scene Knowledge Graphs for XAI Benchmarking" - Takahiro Kawamura, Shusaku Egami, Kyoumoto Matsushita, Takanori Ugai, Ken Fukuda and Kouji Kozaki

1. Contextualized Scene
Knowledge Graphs for XAI
Benchmarking
Takahiro Kawamura1,2, Shusaku Egami2, Kyoumoto Matsushita3,
Takanori Ugai3,2, Ken Fukuda2, Kouji Kozaki4,2
1National Agriculture and Food Research Organization (NARO), Japan
2National Institute of Advanced Industrial Science and Technology (AIST), Japan
3Fujitsu Ltd., Japan
4Osaka Electro-Communication University, Japan
1
IJCKG2022

2. Outline
1. Summary
2. Knowledge Graph Construction and KGRC
3. Guidelines for Knowledge Graph Refinement
4. Application of Guidelines
5. Conclusion
2

3. • Construct scene KG dataset and hold AI competitions to gather methods related to inference and
estimation from wide range of engineers and researchers.
• Guideline for KG refinement based on lessons learned from held competition (2018 – present)
Summary
3
• AI technologies that have explainability (i.e., XAI) or interpretability are attracting attention.
• AI technologies that combine inductive machine learning and deductive knowledge utilization
are expected to become necessary in the future.
Background
• A suitable dataset for the evaluation of XAI tasks should include not only relatively simple
relationships, but also more complex relationships that reflect the real world.
• e.g., spatial, temporal, causal, and contextual relationships
Problem
• Design appropriate indicators and then evaluate, classify, and systematize AI technologies with
explanatory and interpretability, especially those that combine inductive machine learning and deductive
knowledge processing.
Objective
Proposal

4. Knowledge Graph
Construction and KGRC
4

5. Knowledge Graph Reasoning Challenge
Investigation
strategy
Criminal
motive ….
A Contest to develop AI systems which have
abilities for “Reasoning” and “Explanation”
such like Sherlock Holmes.
Sherlock
Holmes
mystery story
Knowledge Graph
(LOD) AI system that estimate criminals with
reasonable explanations using the KG
and other knowledge
The motive is …
Trick is …
The criminal is
XX Because …
2018 – 2021 by the Special Interest Group on Semantic Web
and Ontology (SWO) of the Japanese Society for AI (JSAI)
5

6. Knowledge Graph Construction
• We constructed KGs based on the contents of 8 of Sherlock
Holmes’s short mystery stories.
• The Speckled Band, The Dancing Men, A Case Of Identity, The Devil’s Foot,
The Crooked Man, The Abbey Grange, The Resident Patient, Silver Blaze
• Procedure[kawamura et al. JIST2019]
1. Extract sentences necessary for deduction from mystery stories (in
Japanese) whose copyrights have expired.
2. Rewriting the original text into sentences with clear a subject and object
(i.e., short sentences).
3. Assign semantic roles (e.g., 5W1H) to phrases using natural language
processing tools (Japanese semantic role labeling).
4. Control vocabulary (e.g., predicates, names of characters, and places)
5. Add relationships between scenes (e.g., temporal relationships).
6. Translate the source text into English and convert the entire scenes into a
knowledge graph.
7

7. Knowledge Graph Schema
• Basic policy
• Focus on scenes in a novel and the relationship of those scenes, including
the characters, objects, places, etc., with related scenes
• Only scenes that are judged to be necessary for the deduction are
converted to KGs
• A scene ID (IRI) has subjects, verbs, objects, etc.
• Edges mainly represent five Ws (When, Where, Who, What, and Why).
Scene1 Scene2 Scene3
Scene4
Scene5
Resource
Literal
type subject
source
source
subject subject
hasPredicate
source
hasPredicate
subject
hasPredicate
then
therefore because
then
8

8. Original Sentence (EN|JA)
Absolute Time
Property values are defined as
resources to be referred in the
other scene
Predicate
Subject
Relationship to other Scene ID
Scene Type
- Situation︓ Fact
- Statement︓Remark by A
- Talk︓ Remark by A to B
- Thought︓ Idea of A
Schema (Scene): Example
Unique ID (IRI)
of Each scene
10

9. Submission Categories for KGRC
1. Main Category: development of a system that can accomplish the task of one
or more subject stories
2. Tool Category (From 2019): development of a tool with which a task can be
partially solved
3. Idea Category: development of idea to realize category 1. or 2. (no actual
implementation is necessary)
Numbers of Submissions
11
Main 5
Idea 3
Main 4
Tool 3
Idea 2
Main 2
Tool 2
Idea 3
Main 13
Idea 8
Tool 8
2018 2019 2020 Total
29
Main 2
Tool 3
2021*
*Student only

10. （１）Approach for reasoning/explanations
12
2018 2019 2020 2021 (for student)
Category (1)Aproaches (2)External Knowledge Category (1)Aproaches
(2)External
Knowledge
Category (1)Aproaches
(2)External
Knowledge
Category (1)Aproaches
(2)External
Knowledge
Main.1
knowledge
processing
original rules Main.1
knowledge
processing
＋Machine
Learning
text of novels
original rules
other external
information
Main.1
Machine
Learning
none Main.1
Machine
Learning
ConceptNet,
original ontology
Main.2
knowledge
processing
original ontologies
reasoning rules
Main.2
Machine
Learning
none Main.2
Machine
Learning
ConceptNet Main.2
knowledge
processing
Original ontology,
original rules
Main.3
Machine
Learning
text of other Holmes
novels
Main.3
Machine
Learning
none Tool.１
knowledge
processing
none Tool.1
knowledge
processing
-
Main.4
knowledge
processing
original rules for
Presumption of
culprit
ontology of
motivation
Main.4
knowledge
processing
(＋Machine
Learning)
original ontologies
WordNet
Wikipedia
Tool.２
knowledge
processing
Wikidata Tool.2 Visualization -
Main.5
knowledge
processing
original rules Tool.１
Machine
Learning
none Idea.１
Machine
Learning
WordNet Tool.3
Knowledge
processong
Wikidata, JIWC
Idea.１ Multi agent none Tool.２ NLP none Idea.２
knowledge
processing
Wikidata，ICD-10
Idea.２
knowledge
processing
original knowledge Tool.３
knowledge
processing
NRC Emotion /
Affect Intensity
Lexicon
Idea.３
Machine
Learning
none
Idea.３
knowledge
processing
original ontologies Idea.１
Machine
Learning
Wikipedia
Idea.２
Machine
Learning
none

11. International Knowledge Graph Reasoning
Challenge (IKGRC)
• The 1st International Knowledge Graph Reasoning Challenge (IKGRC2023)
• co-located with 17th IEEE International Conference on Semantic Computing (ICSC)
• The Hills Hotel, Laguna Hills, California, USA (format: Hybrid), February 1-3, 2023
• Important date
• Abstract submission (2 pages, IEEE format) : November 30, 2022
• Accepted papers will be published in the Proceedings of the ICSC.
• Acceptance notification: December 7, 2022
• Submission of application materials: January 15, 2023
• Workshop day: February 2-3, 2023
13
https://ikgrc.org/2023/

12. Guidelines for Knowledge
Graph Refinement
14

13. Guidelines for KG refinement
• We present a guideline consisting of 10 items/steps for refining the KGs
based on lessons learned from our KGs of 8 mystery stories.
1 Short sentences in English are converted to a syntax that is easy
to change to an RDF
Unification of triple structure
2 Adding implicit scenes Addition of implicit
information that is not
explicitly described
3 Add time information
4 Screening of sentences to be treated as scenes
Unification of triple structure
5 Unification of triplication from typical
sentence patterns
6 Division when there is more than one subject or object
7 Giving a type at nesting
8 Mapping verbs to hasPredicate values
Unification of the vocabulary
9 Unification of words such as object and complement
10 Uniform treatment of modifiers 15

14. Guidelines for KG refinement
1. Short sentences in English are converted to a syntax that is
easy to change to an RDF
• Clarify the division between subject and predicate.
• For example, in the sentence “There is no place for Percy Trevelyan,” it
is difficult to understand whether the subject is “Percy Trevelyan” or
“Percy Trevelyanʼs place,” and also whether the predicate is “does not
have place” or “does not exist”.
• In such sentences, the subject is decided to be “Percy Trevelyan” because the
target to which information is to be added is “Percy Trevelyan.”
• Complements omitted objects, complements, places, and so on.
• ex 1) Although there is no location information in “Helen lives with Roylott,” it is
clear from the context that she lives in “Roylott’s house,” so the place should
be added.
• ex 2) In the phrase “Roylott is a father-in-law,” it is not clear from whose
perspective he is a father-in-law, so the additional information should be
provided.
Unification of
triple structure
16

15. Guidelines for KG refinement
4. Screening of sentences to be treated as scenes
• For example, the sentence “The money is 1000 pounds a year” cannot be
understood as a stand-alone scene. Therefore, a triple is added to supplement
the scene such as “Helen and Julia receive their inheritance.”
Unification of
triple structure
17
5. Unification of triplication from typical sentence patterns
• For example, “there is” and “exists” are unified into “exists” to standardize the
symbols used in the inference process.
• Also, information (adjectives) that describe properties are unified with the
value hasProperty;
6. Division when there is more than one subject or object
• For example, the scene “Holmes and Watson got out of the carriage” splits the
subject (value of the subject) into two parts, “Holmes” and “Watson”.
• Also, the scene “Holmes placed a box of matches and a burnt candle near a long,
thin walking stick” splits the object (value of kgc:what) into “a box of matches” and
“a candle”.

16. Guidelines for KG refinement
7. Giving a type at nesting
• In order to express appropriately nested
structures caused by hearsay, each
utterance is decomposed as a scene and
given an appropriate type and source of
information.
• For example, the scene (ID-a)
“Holmes said, “Mr. A said that Mr. B said
something ([ID-y])”([ID-x]).”
is decomposed as follows:
Unification of
triple structure
18
# Holmes said kd:ID-x
kd:ID-a rdf:type kgc:Situation ;
kgc:subject kd:Holmes ;
kgc:hasPredicate kdp:say ;
kgc:what kd:ID-x .
# Mr. A said kd:ID-y
kd:ID-x rdf:type kgc:Statement ;
kgc:InfoSource kd:Holmes ;
kgc:subject kd:A ;
kgc:hasPredicate kdp:say ;
kgc:what kd:ID-y .
# Mr. B said something
kd:ID-y rdf:type kgc:Statement ;
kgc:InfoSource kd:A ;
kgc:subject kd:B ;
kgc:hasPredicate kdp:say ;
kgc:what sometihng .

17. Guidelines for KG refinement
2. Adding implicit scenes
• For example, “the day Helen’s mother died” can be expressed as a single
literal as “death_day_of_helen’s_mother.” However, it cannot be used for
inference because it does not logically express the information that this is the
day that Helen’s mother died.
• Therefore, we introduce a new scene “Helen’s mother died.”
19
3. Add time information
• If there is no description of time in the text, absolute time is given to each
scene to the extent that it does not affect the narrative.
• Qualitative temporal relationships, such as “then,” “before,” “after,” are
added as connections between scenes to clarify the time-series information.
Addition of implicit
information that is not
explicitly described

18. Guidelines for KG refinement
8. Mapping verbs to hasPredicate values
• Verb forms are unified in the active voice
• “Mr. A was shot by Mr. B” is rephrased as “Mr. B shot Mr. A.”
• Verb tenses are unified in the present tense
• The verb (the value of hasPredicate) in the scene “Mr. B shot Mr. A” is “shoot” in the
present tense, not the past tense.
• Emotional expressions are unified into states, not verbs
• The scene “John Straker was excited,” “excited” is not treated as a verb, but is taken as a
state and the value of hasProperty.
• Scenes involving verbs followed by infinitives are broken down
• The scene “John Straker tried to go check the stable”, instead of creating a verb like
tryTo, we break the scene down as follows: “John Straker try [ID-x].” and “John Straker go
to check the table.” ([ID-x])
• Auxiliary verbs and verbs concatenated into one verb
• In the scene “Percy Trevelyan had to prepare the money,” mustPrepare is created as a
verb.
• In addition, that verb is defined that it consists of “must” and “prepare.”
Unification of
the vocabulary
20

19. Guidelines for KG refinement
9. Unification of words such as object and complement
• Assign unique names and IRIs to people and things
• List the people and things that appear first, and assign unique names and IRIs to them.
• Replace collation with named entities and scene IDs
• To distinguish whether it is a concrete person or thing, replace directives, pronouns, and
so forth, with proper nouns
• Unified notation for labels and IRIs
• Establish conventions for the use of camel notation, snake notation, space delimiters,
and so on, to ensure consistency within a KG.
Unification of
the vocabulary
21
10. Uniform treatment of modifiers
• We use are source as it is if it has a qualifier, such as “red carpet,” because a
modifier may be used as a keyword in a story.
• The type is then defined as “carpet” and the property (value of hasProperty) is defined
as “red”.

20. Application of Guidelines
22

21. Verification of the guideline application
Conduct a trial application of the guideline with a third party
• Worker
• Software engineer who has knowledge of RDF and an outline of the KGRC,
but was not involved in the creation of the guidelines
• Task
• Apply the guideline to 8 KGs by one worker
• The data are RDF triples converted to spreadsheet format
23
Approximate
working time:
30 man-days
Work sheet Check sheet

22. Application policy of the guideline
1 Short sentences in English are converted to a syntax
that is easy to change to an RDF
(A) Extracts the points that need to
be modified in advance because
these tasks are high-cost
2 Adding implicit scenes
3 Add time information (B) Can be handled mechanically to
some extent
4 Screening of sentences to be treated as scenes (A)
5 Unification of triplication from typical
sentence patterns (B)
6 Division when there is more than one subject or object
7 Giving a type at nesting (A)
8 Mapping verbs to hasPredicate values
(B)
9 Unification of words such as object and complement
10 Uniform treatment of modifiers Pending due to insufficient
consideration.
24

23. Results
• Can third party apply the guideline?
• The results of the third party's work were generally appropriate, although
some corrections were desirable.
• For the guidelines that were limited to the identification of areas that
should be modified due to the high cost, the guidelines contain items that
should be considered for individual modification policies
• Therefore, it is preferable to consider the detailing of the guidelines.
• Applicability of the guideline to KGs in general
• Next page
25

24. Applicability of the guideline to KGs in general
1 Short sentences in English are converted to a syntax that is easy
to change to an RDF
Refinement methods
common to all KGs
2 Adding implicit scenes
Refinement methods
specific to Scene KGs
3 Add time information
4 Screening of sentences to be treated as scenes
5 Unification of triplication from typical
sentence patterns
6 Division when there is more than one subject or object Refinement methods
common to all KGs
7 Giving a type at nesting Refinement methods
specific to Scene KGs
8 Mapping verbs to hasPredicate values
Refinement methods
common to all KGs
9 Unification of words such as object and complement
10 Uniform treatment of modifiers
26

25. International Knowledge Graph Reasoning
Challenge (IKGRC)
• The 1st International Knowledge Graph Reasoning Challenge (IKGRC2023)
• co-located with 17th IEEE International Conference on Semantic Computing (ICSC)
• The Hills Hotel, Laguna Hills, California, USA (format: Hybrid), February 1-3, 2023
• Important date
• Abstract submission (2 pages, IEEE format) : November 30, 2022
• Accepted papers will be published in the Proceedings of the ICSC.
• Acceptance notification: December 7, 2022
• Submission of application materials: January 15, 2023
• Workshop day: February 2-3, 2023
27
https://ikgrc.org/2023/

26. Conclusion
• Guideline for KG refinement
• We developed a guideline based on lessons learned from Knowledge
Graph Reasoning Challenge in Japan
• We applied the guideline to the eight KGs, and published them on our
GitHub repository: https://github.com/KnowledgeGraphJapan/KGRC-RDF
• We provide the refined KGs for International Knowledge Graph Reasoning
Challenge: https://ikgrc.org/2023/
• Future works
• Consideration of a policy for vocabulary unification
• Consideration of new representation of scenes
28

27. Thank you for your attention!
https://ikgrc.org/2023/
https://github.com/KnowledgeGraphJapan/KGRC-RDF
This work was supported by JSPS KAKENHI Grant Number 19H04168.
http://knowledge-graph.jp/sparql-ikgrc.html