Timo Honkela: Self-Organizing Map as a Means for Gaining Perspectives

Timo Honkela in Metalithicum #5: Self-Organizing Map as a Means for Gaining Perspectives, Einsiedeln, 23rd of May, 2014
Einsiedeln
23rd of May
2014
Self-Organizing Map
as a Means for
Gaining Perspectives
Timo Honkela

Timo Honkela
23 May 2014
Self-Organizing Map
as a Means for
Gaining Perspectives
timo.honkela@helsinki.fi
Metalithicum # 5
Computation as literacy: Self Organizing Maps

Part I:
The Self-Organizing Map

Teuvo Kohonen before the SOM
● School time interest in mathematics, physics, chemistry,
psychology, radio technology, etc.
● Studies at Helsinki University of Technology in theoretical
physics, PhD in 1962, Professor 1963-
● First designer of a computer in Finland (REFLAC),
mid-1960s, keen interest on analog computers
● Visiting professor, University of Washington 1968-69
● Research professor (funded by Academy of Finland),
1975-
● Book “Associative Memory: A Systems-Theoretical
Approach”, 1978
Anderson, James A., and Edward Rosenfeld, eds. Talking nets: An oral history of neural networks. MiT Press, 2000.

Kohonen, Teuvo (1982). "Self-Organized
Formation of Topologically Correct Feature
Maps". Biological Cybernetics 43 (1): 59–69.
Kohonen, T. (1981). Self-organized
formation of generalized topological maps
of observations in a physical system.
Report TKK-F-A450, Helsinki University of
Technology, Espoo, Finland.
First SOM publications

Google:

SOMintroduction
(Honkela 1997)

Milos Manic
“Poverty map”
Kaski & Kohonen
“Pockets Full of Memories”
Legrady, Honkela et al.
André Skupin
“Map of Mozart”
Rauber, Lidy &Mayer
“WEBSOM”
Honkela, Kaski,
Kohonen & Lagus

Variants of the SOM
● Input
● Network structure
● Learning rule
– Information-theoretical
– Probabilistic
● Recurrent and recursive versions
● Operator maps for dynamic phenomena
● Output presentation and postprocessing (clustering,
coloring, etc.)
● Etc.

Views into the SOM
● Vector quantization
● Dimensionality reduction (visualization)
● (Clustering)
● Cortical modeling
● Conceptualization (“semantification”)
● Cognitive function modeling
● Antidote against categorical thinking
● ...

Different kinds of input
Somervuo & Kohonen (1999): Self-organizing maps and learning vector quantization
for feature sequences. Neural Processing Letters.

Different kinds of map structures
● Fixed topology (rectangular, hexagonal)
● Fixed unusual topology (e.g. portrait of Mozart)
● Different dimensionalities (1-, 2-, 3-,..., mixed)
● Growing neural gas
● Hierarchical maps
● Etc. etc.

Some other Kohonen algorithms
● Correlation matrix memories (1972)
● Median strings (1985)
● Learning Vector Quantization (1986)
● Dynamically Expanding Context (1986)
● Self-learning musical grammar (1989)
● Adaptive Subspace SOM (1996)
● Symbol string SOM (1998)
● Evolutionary SOM (1999)
● Self-organizing neural projections (2006)
Years are partly approximate

Part II:
Perspectives to
language, cognition
and human knowing

Classical example: Learning meaning from context:
Maps of words in Grimm fairy tales
Honkela, Pulkki & Kohonen 1995
Automated learning of word relations
using self-organizing map on text context data

Map of Finnish Science
Chemistry
Physics and
engineering
Biosciences
Medicine
Culture and
society
A fully automated process from terminology extraction (Likey) to
semantic space construction (SOM) without any manually constructed resources.

You can measure
things that were not
measurable before

A. Measuring meaning

Challenges:
“Language is BIG”
“Human INTERPRETATION is
inherently involved”
Texts as input instead of measurements

Example:
Complexity of
Finnish at the
level of word
forms
Kimmo Koskenniemi (2013):
Johdatus kieliteknologiaan,
sen merkitykseen ja sovelluksiin
(Introduction to language
technology, its significance and
applications)
https://helda.helsinki.fi/bitstream/handle/10138/38503/kt-johd.pdf?sequence=1

> 6000 languages,
many more dialects Billions of people
blogs.state.gov
en.wikipedia.org
A large number of
different cultures
en.wikipedia.org
A vast number of ways to relate
language, concepts and
the world to each other

Simulating processes of language emergence and communication 22
Language as a system
● Considering natural language as a signal and dynamic
system at cognitive and social levels (also in its written
form) rather than a symbolic and logical system
● Importance of embodiment (cf. e.g. Harnad) and
embeddedness (cf. e.g. Edelman)
● Learning and pattern recognition processes are
essential (as opposed to the theories presented e.g. by
Chomsky, Fodor, Pinker); much of the learning is bound
to be unsupervised

Predicate logic is not about meaning
● Formalisms like first-order predicate logic have widely
been used as a basis for theories of meaning; consider
also contemporary efforts such as Semantic Web
● These formalisms provide only limited means for
creating in-depth theories of how language is
understood
● Traditional logic provides means e.g. for modeling
quantification, connectives, analytical truths and
conceptual hierarchies
● However, many semantic phenomena are matters of
degree. Various proposals that apply Bayesian
probability theory or fuzzy sets deal with this.

Traditional AI & logic viewpoint
Agents Language Model of
the world World
= = =

Pattern recognition
● Even these methodological extensions do not
suffice if the pattern recognition processes are
not taken into account
● The world is not straightforwardly experienced
as discrete objects and events but there are
complex underlying cognitive processes
involved

Agents Language
World
Model of
the world
Emergentist viewpoint
(importance of pattern recognition and learning)

General communication system and
measuring information (Shannon & Weaver)
INFORMATION
SOURCE TRANSMITTER RECEIVER DESTINATION
MESSAGE MESSAGE
NOISE
SOURCE
SIGNAL RECEIVED
SIGNAL
H = - Σ pi log piNoisy channel model

Weaver on Shannon
● “Relative to the broad subject of communication, there seem to
be problems at three levels. [...]
– LEVEL A. How accurately can the symbols of communication
be transmitted? (The technical problem)
– LEVEL B. How precisely do the transmitted symbols convey
the desired meaning? (The semantic problem)
– LEVEL C. How effectively does the received meaning affect
conduct in the desired way? (The effectiveness problem)”
● “The semantic problems are concerned with the identity, or
satisfactorily close approximation, in the interpretation of
meaning by the receiver, as compared with the intended
meaning of the sender.” (1949, p. 4)

Distributional hypothesis
● Two words are semantically similar to the
extent that their contextual representations are
similar (Miller & Charles 1991)
● The meaning of words is in their use
(Wittgenstein)

Context is
concretely
relevant
in physics

Meaning is contextual
red wine
red skin
red shirt
Gärdenfors: Conceptual Spaces
Hardin: Color for Philosophers

Meaning is contextual
SNOW -
WHITE?
WHITE

Complex challenge: different
contexts and cultures
“Shall I compare thee
to a summer's day?”
? ?

Modeling distributional similarity:
word space models
● Word space models represent meaning as points
or areas in a high dimensional vector space
– Self-Organizing Semantic Maps (Ritter and Kohonen 1989)
– LSA (Landauer & Dumais 1997)
– HAL (Lund & Burgess 1996)
– Conceptual spaces (Gärdenfors 2000)
– Word ICA (Honkela, Hyvärinen & Väyrynen 2004)
– etc. etc.

Language as dimensionality
reduction?
ICA of word
contexts; nonlinearity
through thresholding
Comparison
with SVD/LSA
Effect of sparseness
and meaningful
emergent components
Data: TOEFL tests
(Väyrynen, Lindqvist, Honkela 2007)

ICA
SVD
precision
active dimensions

Point of view from
cognitive linguistics
● The meaning of linguistic symbols in the mind of the
language users derives from the users' sensory
perceptions, their actions with the world and with each
other.
● For example: the meaning of the word 'walk' involves
– what walking looks like
– what it feels like to walk and after having walked
– how the world looks when walking
(e.g. objects approach at a certain speed, etc.).
– ...

Abstract vs concrete grounding
Ronald Langacker

Motion capture
AnimationImage analysis
Video analysis
Robotics
Machine learning
Language learning
Socio-cognitive modeling
Symbol grounding
Jorma Laaksonen
Tapio Takala
Klaus Förger
Harri Valpola
Oskar Kohonen
Reinforcement
learning
Paul Wagner
Markus Koskela
Xi Chen
Learning relations
Kinect
OptiTrack
Timo Honkela

goo.gl / UZnvH

Förger & Honkela, 2013
WALKING
RUNNINGRUNNING
Consider how different languages
divide the conceptual space
in different ways
(cf. e.g. Melissa Bowerman et al.)

B. Measuring (inter)subjectivity

“Einsiedeln Abbey is a Benedictine
monastery in the town of Einsiedeln
in the Canton of Schwyz,
Switzerland. The abbey is dedicated
to Our Lady of the Hermits, the title
being derived from the
circumstances of its foundation, for
the first inhabitant of the region was
Saint Meinrad, a hermit. It is a
territorial abbey and, therefore, not
part of a diocese, subject to a
bishop. It has been a major resting
point on the Way of St. James for
centuries.” (Wikipedia)
Objective facts?
Other points of view?

Non-linear projections next to Hotel Drei Könige

Meaning is subjective

Meaning is subjective
● Good
● Fair
● Useful
● Scientific
● Democratic
● Sustainable
● etc.
A proper theory of
meaning has to take
this into account

Experiential grounding
of human knowledge
Human understanding of the world and of
the relationship between language use
and perception and action within the
world is based on a long active and
interactive learning process for which the
genotype gives a certain basis but which
is mainly determined by the individual
interaction with the world including other
human beings and the social and cultural
context

Concept Formation and
Communication - General Theory
Timo Honkela, Ville Könönen, Tiina Lindh-Knuutila, and Mari-Sanna Paukkeri. Simulating processes of concept
formation and communication. Journal of Economic Methodology, 15(3):245–259, 2008.
λ : Ci × Cj → R, i ≠ j
A distance between
two points in the
concept spaces of
different agents
S: symbol space,
The vocabulary of an
agent that consists of
discrete symbols
: sξ i S∈ i → C
An individual
mapping function
from symbols to
concepts
φi: Si D→
An individual
mapping from agent
i's vocabulary to the
signal space D and
an inverse mapping φ
1
i from the signal
space to the symbol
space
Ci: Ndimensional
metric concept
space
Observing f1 and after symbol
selection process, agent 1
communicates a symbol s*
to agent 2 as signal d.  When agent
2 observes d, it maps it  to some s2
S∈ 2  by using the function φ 1
1.
Then it maps the symbol to some
point in its concept space by using
ξ2.  If this point is close to its
observation f2 in the sense of λ, the
communication process has
succeeded.

GICA:
Grounded
Intersubjective
Concept
Analysis

Timo Honkela, Juha Raitio, Krista Lagus, Ilari T.
Nieminen, Nina Honkela, and Mika Pantzar.
Subjects on objects in contexts: Using GICA
method to quantify epistemological
subjectivity.
Proceedings of IJCNN 2012, International Joint
Conference on Neural Networks, pp. 2875-
2883, 2012.
Publication:

Case: State of the Union Addresses
● Text mining is used in populating
a Subject-Object-Context tensor
● This took place by calculating the frequencies
on how often a subject uses an object word in
the context of a context word
– Context window of 30 words

Analysis of the word 'health'

This is why
unsupervised learning
is better
in most cases
in comparison
with
supervised learning
Human-made categories cannot
simply be taken
as a ground truth
There are even a large number of
well grounded category systems,
none of which has an objective status
Kuhn
Local … global

Relevance?
● A large proportion of modern human activity in its
different forms (science, industry, society, culture,
etc.) is based on the use of language
● There are at least 6000 languages in the world and
many more dialects
●
Each language has the order of 105
to 1010
different
word forms
● Each word is understood differently by each speaker
of that language at least to some degree

Relevance, cont'd
● The formal basis of in practice all information
systems does not take this basic phenomenon
into account
● The assumption of shared meanings is simply not
adequate
● Socio-cognitive modeling is needed

Language use and theory
formation as social phenomena
data collection
and generalization
theories language
use
regularity,
variation
regularity,
variation
producing/
creating
learning/
observing
producing/
creating
producing/
creating
description and
harmonization

Emergence of a coherent lexicon in
a community of interacting SOM-based agents
(Lindh-Knuutila, Lagus & Honkela, SAB'06)
Related to e.g. Steels and Vogt on language games

Survival and reinforcement
learning in conceptual system evolution
(Honkela & Winter 2003)

Practical consequences
● The traditional notion of uncertainty in decision making does
not cover the uncertainties caused by differences in
conceptual systems of individual agents within a community
● In many transactions, including symbolic/linguistic
communication, the differences in the underlying conceptual
systems play an important role
● Serious efforts have been made to harmonize or to standardize
the classification systems or ontologies used by agents
● Even if standardization is conducted, there can not be any true
guarantee that all participating agents would share the
meaning of all the expressions used in the transactions in
various contexts

Quantifying the effect of
“semantic noise”
● Sintonen, Raitio & Honkela: “Quantifying the
effect of meaning variation in survey analysis”,
forthcoming in ICANN 2014

Part III:
Closing remarks
on digital humanities

Digital humanities
● Research within humanities
with the help of computers
– Digital resources
– Computational models
● Basic motivation
– One can already fly to moon and
build sophisticated factory products
– The most important open questions
in the world are related to humanities
and social sciences

Digital Computational
Humanities
Content
storage and
transfer
Content
analysis

Societal
and
Cultural
Text
Mining

Honkela, Korhonen, Lagus & Saarinen:
Five-dimensional sentiment analysis of
corpora, documents and words,
forthcoming in WSOM 2014

Project ఠ
(ttha,Telugu)
Science Society
Culture

Thank you for your attention!
Danke schön!
Kiitos!
Tack!
Merci!
謝謝！
Σας ευχαριστούμε!
¡Gracias!

Timo Honkela: Self-Organizing Map as a Means for Gaining Perspectives

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (14)

Similar to Timo Honkela: Self-Organizing Map as a Means for Gaining Perspectives

Similar to Timo Honkela: Self-Organizing Map as a Means for Gaining Perspectives (20)

More from Timo Honkela

More from Timo Honkela (20)

Timo Honkela: Self-Organizing Map as a Means for Gaining Perspectives