Talk esscs2010

Grounding compositional symbols:
no composition without discrimination?

Alberto Greco,
Elena Carrea

University of Genoa, Italy

greco@unige.it
elena.carrea@unige.it

ESSCS 2010 London, 7^ July 2010
lunedì 26 luglio 2010

Overview

 What is Symbol Grounding?
 Symbol Grounding and Compositionality
 Why is Compositionality relevant to
Cognitive Sciences?
 Purposes of the Study
 Experiment
 Provisional conclusions


The Symbol Grounding problem

Searle (1980) argued against the
symbolic model of mind with the
mental experiment of the
Chinese Room.



Can we say
that this
man
understands
Chinese?



Harnad’s Merry-go-round:



Harnad’s Merry-go-round:

This problem is similar to having to learn
Chinese with only a Chinese/Chinese dictionary.



How can symbolic systems
be linked to the world?

Can a purely symbolic model
be grounded only in other
meaningless symbols?

This is the Symbol Grounding problem
(Harnad 1990).



Harnad’s solution:
grounding experience
must be sensorimotor…



Symbols (words) acquire their
meaning only when they are
associated with sensorimotor
grounding representations.


Compositionality

In Harnad's view, grounding has been
conceived as compositional:

from sensorimotor toil
to symbolic theft,

thanks to propositions.


Compositionality

An example:


Compositionality

An example:

“a zebra is like a
horse with stripes”…


Compositionality

What is compositionality?
A definition of compositionality from a conference on this
topic*:

“Compositionality is a key feature of structured
representational systems, be they linguistic, mental or
neuronal.

A system of representations is compositional just in case the
semantic values of complex representations are determined by
the semantic values of their parts.”
* Compositionality, Concepts, and Cognition:
An Interdisciplinary Conference in Cognitive Science,
Düsseldorf, Germany, February 28 to March 3, 2004


Compositionality

 Systematic combination of meaningful
components,
 According to syntactical rules.

 Meaning of complex expressions,
 Productivity (linguistic and conceptual).


Compositionality

Why is it relevant to Cognitive Sciences?

 Formal languages of mathematics, logic, and
computer science are (usually) considered as
compositional.
 Linguistics: are natural languages compositional?
 Philosophy: is whole meaning really coming out from
part meaning? What about context?
 Psychology: is representation compositional? Is
language of thought compositional? (from cognitivism to
ecological, embodiment, re-enacting theories).
 Cognitive and neural modelling (AI, connectionism,
robotics): no easy compositionality in neural networks;
binding problem; Can analog features be represented
compositionally?


Purposes of the Study

Key-question:

What is complex symbolic
composition based on?

…a corresponding
composition of grounding
sensorimotor

representations?



Key-question (in other words):

Are structured grounding
representations needed, to express
meanings that have
a complex structure?



Key-question (in other words):

Are structured grounding
representations needed, to express
meanings that have
a complex structure?

How can we study this empirically?



How could we empirically test whether
grounding representations

have been established?

Starting from scratch…
from nonsense, to meaning…



Our solution:
association task between
nonsense symbols (words)
and target perceptual or motor
stimulus patterns.

nonsense words perceptual or motor
patterns

(arbitrary, not yet
grounded symbols)



nonsense words perceptual or motor
patterns

Our assumption:
successful learning of this association
shows that symbols have been grounded,
and that a corresponding grounding
representation has been established.



The experiment should allow us to
distinguish between two
possibilities:
blue circle
- compositional Grounding
REPresentations: separate GREPs
for each part (when a concept
GREPs
comes from the combination of
the parts );

- holistic GREPs: these combined blue-circle
concepts are best learned by
bringing them back to a single
unifying GREP. GREP



Previous research on compositional grounding
with motor patterns
Greco & Caneva (2005) associated 3 pseudo-words to 3 features
(e.g. GAB DIN FIT or TANEC for “push right hand”).
compositional holistic

Greco & Caneva (2009) associated 2 pseudo-words to 2 features
(a particular nonsense motor pattern & hands up, down, fist)
(e.g. BASPI NOLE or TERPESOVA).
compositional holistic

And in other conditions (e.g. incremental learning) holistic again

…no consistent results



Inconsistency in these results may be
explained by several factors:
• difficulty of learning tasks,
• different learning methods,
• words not balanced across conditions…



We presupposed that:
• a single symbol is best supported by a single
Grounding REPresentation (GREP) when a
pattern tends to be represented holistically
• separate symbols (to be combined) are best
supported by separate GREPs when pattern
features tend to have separate representations



We presupposed that:
• a single symbol is best supported by a single
Grounding REPresentation (GREP) when a
pattern tends to be represented holistically
• separate symbols (to be combined) are best
supported by separate GREPs when pattern
features tend to have separate representations

But we really don’t know much about the nature of
motor representations (do they tend to be
represented analytically or holistically?).
We need stimuli that have a
known representation…

Experiment

From literature…

Handel & Imai (1972) showed different kinds
of properties of visual objects:

Integral Separable


Experiment

From literature…


Integral Separable

When attributes are
fused together, and are
perceived as one.


Experiment

From literature…


Integral Separable

When attributes are When attributes can be
fused together, and are easily perceived
perceived as one. separately.


Experiment

We repeated some classical experiments of
Handel & Imai to select good (integral and
separable) stimuli for our study.


Experiment


Experiment

 It is not an absolute separation;

 integral and separable stimuli,
not features.


Experiment

INTEGRAL STIMULI:
We put the figures in jars to make the task more
engaging.
As integral stimuli, we used triangles with the same
blue which varied according to a matrix with 4
degrees of brightness and saturation.


Experiment

INTEGRAL STIMULI:
We put the figures in jars to make the task more
engaging.
As integral stimuli, we used triangles with the same
blue which varied according to a matrix with 4
degrees of brightness and saturation.

An example:


Experiment

SEPARABLE STIMULI:
We used colored polygons as separable
stimuli.
We put into a matrix 4 shapes and 4 colors
as separable stimuli.


Experiment

SEPARABLE STIMULI:
We used colored polygons as separable
stimuli.
We put into a matrix 4 shapes and 4 colors
as separable stimuli.

An example:


Experiment

In our task meaningless labels were associated with
integral and separable stimuli.

We set also two conditions for the labels:
 holistic
(a single word for the stimulus as a whole);
 compositional
(two words that describe two features of the
stimulus).


Experiment

Four conditions:

compositional holistic label
label (1 word)
(2 words)

integral stimuli
P Q
separable
stimuli R S

Experiment

Conceptual universe:

nole bote sove pofe

baspi terpesova pincelura volsicoda

tispi mutiralda feltorana patrasina

respi rispaguna balartoca luticanza

cuspi sertamina mertogala dortamana


Experiment

• Participants (28 undergraduate students) saw the
instructions on a PC screen.
• The instructions explained that the task was to learn and
remember the names of some jars.

STAGES:
1.familiarization with the stimuli (judging the similarity
between pairs of jars);
2.associative Learning (Ss. saw a jar and its name);
3.test (Ss. should click the correct name after seeing the jar).


Experiment

Results:

After familiarization (similarity judgments), we
found the same results reported in the literature.

(distance best measured by euclidean metric for
integral pairs, by city-block metric for separable
pairs).


Experiment

Expectations


Experiment

Results:


Experiment

Results (percentage of correct answers):


Provisional conclusions

How can we explain these results?
• Integral stimuli: the association with 2 words is
more difficult because of what we called
“competitive grounding”.



• Integral stimuli: the association with 2 words is
more difficult because of what we called
“competitive grounding”.

First the subject thinks that baspi-
BOTE is a certain blue.
In a second moment another blue is
called tispi-BOTE (because of the
same lighting).
This creates a competition between
the 2 representation.



• Separable stimuli: learning with 2 words did not
show any extra-benefit because of the limited
number of words and stimuli to recall.



• Separable stimuli: learning with 2 words did not
show any extra-benefit because of the limited
number of words and stimuli to recall.

Compositionality probably shows its
convenience with a higher number of
words and stimuli.
With only 8 words it’s easier to learn a
single label.



For this reason we are doing a new
experiment with 12 words and stimuli
to learn.

We are interested in studying if
compositionality becomes convenient
(with separable stimuli) beyond a
critical number of stimuli.



Limitations:

Using meaningless words is not “ecological”;

It’s difficult to create words free from
participant’s personal association with common
words;

 It’s difficult to neutralized the prior knowledge
(e.g. about geometric shapes).



Benefits:

 We set a paradigm for testing grounding
representation with a learning task;

 We can test whether 2 labels of the
compositional condition really have 2 different
grounding representations.



In the final step of the experiment we asked
participants to select among 4 jars the one
described by a particular sentence.
In this set 2 jars shared 1 attribute (color) but not
the other (shape).
In other words: 2 jars were BASPI but only 1 was
BASPI-NOLE.
Then we asked the opposite task (two NOLE but
only one BASPI).



By using this setting we could
exclude that the participants
memorized the compositional
sentence as a single word.


Talk esscs2010

Recommended

Recommended

More Related Content

Featured

Featured (20)

Talk esscs2010