A Family of Reactive-Cognitive Architectures based on Natural Language Processing, as Decision-Making Helpers for IoT, in the Closed- or Open-World Assumption.
WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...
C.F. Longo Ph.D. Thesis Defence
1. A Family of Reactive-Cognitive Architectures based on Natural
Language Processing, as Decision-Making Helpers for IoT, in the
Closed- or Open-World Assumption
Ph.D. Thesis Defense
Candidate: Carmelo Fabio Longo
Advisor: Corrado Santoro
February 17, 2022
5. • High deductive capabilities
• Physical interaction with the environment
• Meta-Reasoning in conceptual spaces for the task of decision-
making
• Session Awareness
• History Awareness
• Consciousness
6.
7. • High deductive capabilities
• Physical interaction with the environment
• Meta-Reasoning in conceptual spaces for the task of decision-
making
• Session Awareness
• History Awareness
• Consciousness
8. [1] P. Thagard. “Critical thinking and informal logic: Neuropsychological perspectives.” In: Informal logic, 51
(2011), pp. 152–170. doi: https://doi.org/10.22329/il.v31i3.3398.
14. [1] Carmelo Fabio Longo, Francesco Longo, Corrado Santoro “CASPAR: towards Decision Making Helpers Agents for IoT, based on Natural Language
and First Order Logic Reasoning”, in Engineering Application of Artificial Intelligence, Elsevier, 2021.
15. • Language
• Complex Planning
• Deductive Reasoning
• Differential equations
• Statistics
• Signal Processing
Meta-Reasoning
CASPAR: Cognitive Architecture System Planned and Reactive
16. • High deductive capabilities
• Physical interaction with the environment
• Meta-Reasoning in conceptual spaces for the task of decision-making
• Session Awareness
• History Awareness
• Consciousness
17.
18. A pipeline of five modules with the task of taking a sound stream in
natural language and translating it in a neo-davidsonian First Order
Logic (FOL) expression inheriting the shape from the event-based
formal representation of Davidson[1].
[1] D. Davidson, “The logical form of action sentences,” in The logic of decision and action, p. 81–95, University of Pittsburg Press,
1967.
Brutus stabbed suddenly Caesar in the agora
∃e stabbed(e, Brutus, Caesar) ⋀ suddenly(e) ⋀ in(e, agora)
e = davidsonian variable
19. The MST Builder, has the purpose to build a novel semantic structure
defined as Macro Semantic Table (MST). It summarizes in a canonical
shape all the semantic features in a sentence, by leveraging a step-by-
step dependencies analysis, in order to derive FOL expressions.
MST(u) = {ACTIONS, VARLIST, PREPS, BINDS, COMPS, CONDS}
ACTIONS = [(label𝑘, e𝑘, x𝑖 , x𝑗),...]
VARLIST = [(x1, label1),...(x𝑛, label𝑛)]
PREPS = [(label𝑗, (e𝑘 | x𝑖), x𝑗),...]
BINDS = [(label𝑖, label𝑗),...]
COMPS = [(label𝑖, label𝑗),...]
CONDS = [e1, e2,...]
where u = utterance in natural language
21. • Managed by the BDI Framework PHIDIAS
• The Speech-To-Text (STT) Front-End: Beliefs generator
• Each Sensor Instance: it asserts specific beliefs related to
sensors, such as:
• Microphones
• Temperature sensors
• Video Capturing
• Etc.
22. Set the cooler at 27 degrees in the bedroom
set:VB(d1, __, x1) ⋀ cooler:NN(x1) ⋀ at:IN(d1, x2) ⋀ 27:CD(x2) ⋀
degree:NNS(x2) ⋀ in:IN(x2, x3) ⋀ bedroom:NN(x3)
INTENT(set, cooler, bedroom, (at 26 degree))
Translation Service
Direct Command Parser
23. Turn off the lights in the living room, when the temperature
is 25 and the time is 12
be:VBZ(d2, x3, x4) ⋀ be:VBZ(d3, x5, x6) ⋀ temperature:NN(x3) ⋀
25:CD(x4) ⋀ time:NN(x5) ⋀ 12:CD(x6) ⟹ turn:VB(d1, __, x1) ⋀ off:RP(d1)
⋀ light:NNS(x1) ⋀ in:IN(d1, x2) ⋀ living:NN(x2) ⋀ room:NN(x2)
COND(337538, be, temperature, 25), COND(337538, be, time, 12)
ROUTINE(337538, turn, light, living room, off)
Translation Service
Routines Command Parser
24. COND(337538, be, temperature, 25), COND(337538, be, time, 12)
ROUTINE(337538, turn, light, living room, off)
Asserted by Sensor Instances:
+SENSOR(be, temperature, 25)
+SENSOR(be, time, 12)
INTENT(turn, light, living room, off)
25. shines:VBZ(e1, x1, __) ⋀ sun:NN(x1) ⋀ strongly:RB(e1) ⟹ is:VBZ(e2, x3 , x4)
⋀ Robert:NN(x3) ⋀ happy:JJ(x4)
Definite clauses Builder
shine:VBZ(sun:NN(x1), __) ⟹ be:VBZ(Robert:NNP(x3), happy:JJ(x4))
When the sun shines strongly, Robert is happy
Translation Service
26. • He likes to eat a bass
• He likes to play the bass
PROBLEM: where “bass” is intended as fish and where as musical
instrument?
27. INSIGHT: in a common sense words embedding, the bag of
words (context) used for the gloss/examples related to a
synset comprising a lemma, should be likely vectorial
closer (among all synsets comprising that lemma) to
another bag of words making effective usage of such a
lemma (in a similar context).
28. Unsupervised [1] naive “bag of words” strategy, taking in account of the higher word2vect similarity (sim)
for each lemma l of the sentence S (depending of the richness of the domain):
[1] Navigli, R., 2009. Word sense disambiguation: A survey. ACM Comput. Surv. 41 (2), 10.
http://dx.doi.org/10.1145/1459352.1459355.
29. • He likes to eat a bass
Like.v.05:VBZ_Feed.v.06:VB(He:PRP(x1), Sea_bass.n.01:NN(x2))
• He likes to play the bass
Like.v.05:VBZ_Play.v.18:VB(He:PRP(x1), Bass.n.07:NN(x2))
Where the glosses are:
Sea_bass.n.01: the lean flesh of a saltwater fish of the family Serranidae
Bass.n.07: the member with the lowest range of a family of musical instruments
30.
31. # turn off the light in the kitchen
+INTENT(X, "light", "kitchen", T) / lemma_in_syn(X, "change_state.v.01") >>
[exec_cmd("change_state.v.01", "light", "kitchen", T)
# turn off the light in the garage
+INTENT(X, "alarm", "garage", T) / (lemma_in_syn(X, "change_state.v.01") &
eval_cls("At_IN(Be_VBP(Person_NN(x1), __), Home_NN(x2))")) >>
[exec("change_state.v.01", "alarm", "garage", T)]
where:
• X = Verb, T = parameters
• eval_cls(C) is an Active belief = True if C can be deducted from Clauses KB, False otherwise
• At_IN(Be_VBP(Person_NN(x1), __), Home_NN(x2)) represents the sentence: A person is
at home
32.
33. • Nono is an hostile nation
• Colonel West is American
• missiles are weapons
• Colonel West sells missiles to Nono
• When an American sells weapons to a hostile nation, that American is a criminal
• Be(Nono(x1), Hostile(Nation(x2)))
• Be(Colonel_West(x1), American(x2))
• Be(Missile(x1), Weapon(x2))
• To(Sell(Colonel_West(x1), Missile(x2)), Nono(x3))
• To(Sell(American(x1), Weapon(x2)), Hostile(Nation(x3)) ⟹ Be(American(x4), Criminal(x5))
Clauses KB
Question: Colonel West is a criminal?
Query: Be(Colonel_West(x1), Criminal(x2)) ? FALSE!
[1] P.N. Stuart J. Russell, Artificial Intelligence: A Modern Approach, Pearson, 2010, Chapter 9.3.
34. Roy_NNP(y) ⟹ man_NN(y) Assignment Rule
Roy_NNP(y) ⟹ man_NN(y)
talk_VBZ(Roy_NNP(x1), woman_NN(x2))
Clauses KB:
talk_VBZ(Roy_NNP(x1), woman_NN(x2)) ⟹ talk_VBZ(man_NN(x1), woman_NN(x2))
Assert implicitly
be:VBZ(e1, x1, x2) ∧ Roy:NNP(x1) ∧ man:NN(x2)
Roy is a man
If the lemma be is disambiguated with be.v.01, which is defined by its gloss as: have the quality of being
[something], we can assert implicitly the following clause:
(Roy talk with a woman)
(Roy talk with a woman ⟹ A man talk with a woman)
35. Having a simple KB as it follows:
𝑃1(𝐺1(x1)) ⋀ 𝑃2(𝐺2(x2)) ⟹ 𝑃3(𝐹3(x3))
𝑃1(𝐹1(x1))
𝑃2(𝐹2(x2))
Clause Conceptual Generalization of KB is:
𝑃1(𝐺1 1(x1)) ⋀ 𝑃2(𝐺2(x2)) ⟹ 𝐹3(x3)
𝐹1(x1)
𝐹2(x2)
𝑃1, 𝑃2, 𝑃3 can be considered as modifiers of respectively 𝐹1, 𝐹2, 𝐹3.
The body (or left hand-side) of the implication is unchanged to hold the quality of the axiom.
36. When the sun shines hard, Barbara drinks slowly a fresh lemonade
Hard(Shine(Sun(x1), __)) ⟹ Drink(Barbara(x3), Lemonade(x4))
Hard(Shine(Sun(x1), __)) ⟹ Slowly(Drink(Barbara(x3), Lemonade(x4)))
Hard(Shine(Sun(x1), __)) ⟹ Drink(Barbara(x3), Fresh(Lemonade(x4)))
Hard(Shine(Sun(x1), __)) ⟹ Slowly(Drink(Barbara(x3), Fresh(Lemonade(x4))))
Generalizations
41. Table: real-time performances (in seconds) in the case of the command: turn off the alarm in the garage, subordinated
by the query: Colonel West is a criminal? (reasoning on the prior KB)
RESULTS: we had just −0.11% loss of performance on Raspberry Pi4 B (4GB) than respect to Intel i5-6600K
CONCLUSION: in a Caspar-based IoT system, the real-time performances will depend mostly on both responsiveness of
the actuators devices and quality of the Internet connection (by using Google API Cloud).
43. 1. Due to KB expansions, large ones can be hard to manage.
2. CASPAR is just able of logic deduction, because it works only in
the presence of unifiable predicates. No feedback for possible close
results.
44. [1] C. Santoro, Carmelo Fabio Longo. “AD-CASPAR: Abductive-Deductive Cognitive Architecture based on Natural Language and First Order Logic Reasoning”. In: 4th
Workshop on Natural Language for Artificial Intelligence (NL4AI2020) co-located with the 19th International Conference of the Italian association for Artificial
Intelligence (AI*IA 2020). 2020.
45. • Affermative/Interrogative questions
• Margot said the truth about her life? → Logical form
• Auxiliary+Affermative Affermative/Interrogative questions
• Has Margot said the truth about her life? → Auxiliary removal → Logical form
✓ First dependency analysis: aux(said, Has)
Logical forms
• Plain
❖ Say_VBD(Margot_NNP(x1), About_IN(Truth_NN(x2), Her_PRP__Life_NN(x3)))
• Disambiguated
❖ State.v.01_VBD(Margot_NNP(x1), About_IN(Truth.n.03_NN(x2), Her_PRP__Animation.n.01_NN(x3)))
46. 1. Dependencies analysys of the question
2. Splitting into chunks as it follow, starting from the delimiters [AUX] and [ROOT] which
are in general are always present in a wh-question:
[PRE AUX][AUX][POST AUX][ROOT][POST ROOT][COMPL ROOT]
3. Ricombination of the chunk by the means of production rules (QA-Shifter) respecting
the grammatical features of the current language, but in assertion-shape.
4. Introduction of additional adverbs (in case of where and when)+Dummy word
5. Simetric version of 3. for copular[1] verbs
Steps to deal with wh-questions (when, where, what, who, how):
[1] a non-transitive verb but identifying the subject with the object in the scope of a verbal phrases (like be)
47. Who could be the president of America?
The chunks are:
[PRE AUX][could][POST AUX][be][the president of America][COMPL ROOT]
with AUX= could, ROOT=be, POST_ROOT=the president of America
Such a wh-question will trigger a QA-Shifter rule to product the following two assertions:
Dummy could be the president of America
The president of America could be Dummy
48. The Translation Service, during parsing, will impose the Part-of-Speech DM to Dummy, whose
parsing is not expected by the Clauses Builder, thus it will discarded. At the end of this process, as
FOL expression of the query we'll have the following literal:
Be_VBZ(Biden_NNP(x1), x2)
Hight Clauses KB:
Be_VBZ(Biden_NNP(x3), Of_IN(President_NN(x4), America_NNP(x5))), ……
Reasoning (Backward-Chaining):
x1: x3, x2: Of_IN(President_NN(x4), America_NNP(x5))
which contains, in correspondence of the variable x2, the logic representation of the snippet:
president of America as possible and correct answer.
49. Low Clauses KB record (stored in a NoSQL db) for a sentence S in natural language:
▪ S: sentence
▪ C: nested definite clause of S
▪ Feat𝒄: vector labels of C
𝐶𝑜𝑛𝑓𝑖𝑑𝑒𝑛𝑐𝑒𝐶 =
| ځ(𝐅𝐞𝐚𝐭𝒒,𝐅𝐞𝐚𝐭𝒄)|
|𝐅𝐞𝐚𝐭𝒒|
Low Clauses KB High Clauses KB
definite clause C
𝐶𝑜𝑛𝑓𝑖𝑑𝑒𝑛𝑐𝑒𝐶 < threshold
50.
51. Realtime cognitive performances (in seconds) considering:
Question: Colonel West is a Criminal? (on the prior KB)
Confidence threshold: 0.6
where:
• West25: colonel West KB (expanded) made of 25 clauses
• West104: West25 + 78 unrelated clauses
• West303: West25 + 278 unrelated clauses
RESULTS
1. 0.416 > 0.378: it is expected such bias to be increased for larger
knowledge bases. Hence, LKB+HKB works better than just HKB.
2. It permits at the same time abduction as pre-stage of deduction,
in order to give back closer results also in presence of non-
successful reasoning.
53. 1. KB expansion depends on clauses assertions order.
2. Such architectures are suitable only in scenarios with the Closed-
World Assumption.
54. CASPAR: meta-reasoning in a conceptual space based on
natural language processing, whose content is made of facts
and axioms in first-order logic with the closed-world
assumption
SW-CASPAR: meta-reasoning in a conceptual space
based on natural language processing, whose content is made
of shared ontologies with the open-world assumption
(Semantic Web)
Transposition
[1] C. F. Longo, C. Santoro, D. F. Santamaria, M. N. Asmundo, and D. Cantone. “Towards Ontological Interoperability of Cognitive IoT Agents, based
on Natural Language Processing”, in Intelligenza Artificiale, IOS Press, 2022. [ACCEPTED]
55.
56. Modelling an ontology reflecting a domain, by means of a description in natural language of the
domain itself, with the aim of human-fashioned reasoning.
BIASES (lacks, noisy, etc)
• Manual
• Cooperative
• Semi-automatic
• Automatic (probably not feasible [1])
[1] A. Browarnik, O. Maimon, Ontology learning from text, in: The First International Conference on Big Data, Small Data, Linked
Data and Open Data, 2015.
57. Biases of Ontology Learning for the aim of human-like fashioned reasoning:
• Nonexistent objects representation
• An information and its complement present in the KB
• Deverbal Nominalization
• “Robert walked down the street” VS “Robert had a walk down the street”
• Deadjectival Nominalization
• “the friends are happy” VS “happy friends”.
• Others[1]
[1] F. Moltmann, Natural language ontology, 2017. URL: https://oxfordre.com/linguistics/view/10.1093/acrefore/9780199384655.001.0001/acrefore-
9780199384655-e-330. doi:10.1093/acrefore/9780199384655.013.330
58. L.O.D.O. (Linguistic Oriented Davidsonian Ontology). Can be
considered a foundational ontology, i.e., a specific type of
ontology designed to model high-level and domain
independent categories about the real world.
59. A set of triples in OWL 2 made by the following classes, properties and instances:
• Verb
• hasId (instance of Id)
• hasSubj (instance of Entity)
• hasObj (instance of Entity)
• hasAdv (instance of Adverb)
• hasPrep (instance of Preposition)
• Id
• Entity
• hasAdj (instance of Adjective)
• hasPrep (instance of Preposition)
• Adverb
• Adjective
• Preposition
• hasObj (instance of Entity)
60. A group of axioms (or part of them) implicitly created by SW-CASPAR, with the aim of
increasing the chances of reasoning. In the presence of the following FOL expression:
Subject:POS(x1) ∧ Cop:POS(e1, x1, x2) ∧ Object:POS(x2)
The Ontology Builder will assert the following SWRL axiom:
Subject(?x) -> Object(?x)
where Cop is copular verb (such as Be, for instance), i.e., an intransitive verb but
identifying its subject with its object; hence, in this case, the class membership of the
verb’s object will be inherited by the subject.
61. Such rules are implicitly asserted together with the Assignment Rules, to let a copular verb’s
subject inherits both adjectives and prepositions properties of the verb’s object.
Formally, considering Subject(?x) -> Object(?x), the corresponding legacy rule will be the
following:
Subject(?x2), Object(?x1), hasAdj(?x1, ?x3), Adjective(?x3) -> hasAdj(?x2, ?x3)
Subject(?x2), Object(?x1), hasPrep(?x1, ?x3), Adjective(?x3) -> hasPrep(?x2, ?x3)
62. • In the presence of an instance of Adjective, such rule asserts a new deadjectivated
instance of the latter as new membership of the adjective related noun. Formally:
Entity(?x1), hasAdj(?x1, ?x2), Adjective(?x2) -> Entity(?x2)
• In the presence of an instance of Verb, such rules assert a new deverbalized
instance of the latter having the same entities as the former, by the means of a
semantic production rules system supported by specific dictionary (in progress of
development).
63. The production rule of the Ontology Builder for such rules assertion takes into account the
following pattern:
Subject(𝑥𝑏𝑜𝑑𝑦) ∧... ⟹ Subject(𝑥𝑠𝑢𝑏𝑗 ) ∧ Object(𝑥𝑜𝑏𝑗) ∧ Cop(𝑒𝑐𝑜𝑝, 𝑥𝑠𝑢𝑏𝑗 , 𝑥𝑜𝑏𝑗)
permitting the formal assertion of the following pattern:
Subject(?𝑥𝑜𝑏𝑗), ... -> Object(?𝑥𝑜𝑏𝑗)
Example: when a dog awake the man, the dog is hungry
Result: in the presence of the verbal phrase: the dog awake the man, an instance related to
dog will acquire implicitly the membership to the class hungry.
64. • Value Giver statement: such a statement contributes to give a value to a data property
hasValue related to a specified individual, which is parsed by the Ontology Builder by
matching the following pattern of beliefs:
GND(FLAT, X, Y), ADJ(FLAT, X, "Equal"), PREP(FLAT, X, "To", S), VALUE(FLAT, S, V)
The property hasValue might be involved in comparison operations in the composition
of a SWRL axiom.
• Comparison Conditional: are parsed from sentences similarly to the Value Giver
Statement, but they will take place within the body of Implicative Copular Rules.
65. Robinson Crusoe is a patient
Robinson Crusoe has diastolic blood pressure equal to 150
When a patient has diastolic blood pressure greater than 140, the patient is hypertensive
66.
67.
68. +INTENT("Rinazina", P) / eval_sem(P, "Hypertensive")) » [say("Nope. Patient is hypertensive")]
+INTENT("Rinazina", P) » [exec_cmd("Rinazina", P), say("execution successful")]
Meta-Reasoning (Active Belief)
Command: Give Rinazina to P (P= patient) SW-CASPAR: +INTENT("Rinazina", P)
72. Together with soundness and time performances, two criteria from the state-of-
the-art have been also used for evalution with good results:
• MGC[1] (Minimal Cognitive Grid)
• SMM [2] (Standard Model of Mind)
CONCLUSION: CASPAR is a powerful startpoint to build complex cognitive
architectures, with an hybrid sub-symbolic/symbolic approach, based on natural
language processing.
[1] A. Lieto. Cognitive Design for Artificial Minds. Routledge, 2021. Chap. 3.
[2] Laird, J. E., Lebiere, C., & Rosenbloom, P. S. (2017). A Standard Model of the Mind: Toward a Common Computational Framework across Artificial
Intelligence, Cognitive Science, Neuroscience, and Robotics. AI Magazine, 38(4), 13-26. https://doi.org/10.1609/aimag.v38i4.2744
74. A Family of Reactive-Cognitive Architectures based on Natural
Language Processing, as Decision-Making Helpers for IoT, in the
Closed- or Open-World Assumption
Ph.D. Thesis Defense
Candidate: Carmelo Fabio Longo
Advisor: Corrado Santoro
February 17, 2022
