Human visual system interprets information obtained through eyes to build a model of the surrounding world. This channel is our main source for understanding the world. Walking on a street, reading a book, or watching a movie all rely on our visual system. The relationship between movement, visual perception and language is complex. Movement is a specific focus of this presentation for several reasons. It is a fundamental part of human activities that ground our understanding of the world. Abstract meanings are often constructed as metaphoric extensions of movement schemas. As there is an increasing amount of video and motion tracking data available, formation of semantic models based on movement using computational methods is becoming feasible. In addition to movement, multilinguality and subjectivity of understanding are also addressed.

1. From Patterns of Movement to
Subjectivity of Understanding
Timo Honkela
Aalto University
(former Helsinki University of Technology)
Department of Information and Computer Science
Cognitive Systems research group
Finland
UC Santa Barbara, 4 Dec 2012

2. 0
Background and
(un)Related Topics

3. Natural language database interface
with dependency-based compositional semantics
● H. Jäppinen, T. Honkela, H. Hyötyniemi & A. Lehtola (1988):
A Multilevel Natural Language Processing Model. Nordic
Journal of Linguistics 11:69-87.
What is the turnover of the ten largest stock exchange companies in forestry?
Morphological analysis
Dependency parsing
Logical analysis
Database query formation
Result from the SQL database

4. Classical example: Learning meaning from context:
Maps of words in Grimm fairy tales
Honkela, Pulkki & Kohonen 1995

5. Map of Finnish Science
Chemistry
Medicine
Biosciences
Culture and
Physics and society
engineering

6. WordICA
Timo Honkela, Aapo Hyvärinen, and Jaakko Väyrynen. WordICA - Emergence of
linguistic representations for words by independent component analysis.
Natural Language Engineering, 16(3):277–308, 2010.
Jaakko J. Väyrynen, Lasse Lindqvist, and Timo Honkela. Sparse distributed
representations for words with thresholded independent component
analysis. In Proceedings of IJCNN'07, pages 1031–1036, 2007.

7. Learning taxonomies
Mari-Sanna Paukkeri, Alberto Pérez García-Plaza,
Víctor Fresno, Raquel Martínez Unanue and Timo
Honkela (2012). Learning a taxonomy from a set
of text documents. Applied Soft Computing,
12(3), pp. 1138--1148.

8. Text mining for peer support
User's User modeling
Discussion forum input and analysis of
postings, etc. feedback
(Honkela, Izzatdust, Lagus 2012)
STYLE
TOPIC ANALYSIS SENTIMENT ANALYSIS
ANALYSIS
MULTICRITERIA SELECTION PROCESS
Selected stories
EVALUATION

9. ICA of wellbeing-related terms
in Reddit texts
(Honkela, Izzatdust, Lagus 2012)

10. Analyzing Complexity of Languages
Markus Sadeniemi,
Kimmo Kettunen,
Tiina Lindh-Knuutila,
and Timo Honkela.
Complexity of
European Union
languages: A
comparative approach.
Journal of Quantitative
Linguistics, 15(2):185–
211, 2008.

11. META-NET
Network of Excellence

12. MultilingualWeb

13. Concept Formation and
Communication - General Theory
ξ: si ∈ Si → C
Ci: N­dimensional
An individual
metric concept
mapping function
space
from symbols to
concepts Observing f1 and after symbol
S: symbol space, selection process, agent 1
The vocabulary of an communicates a symbol s*
φi: Si → D
agent that consists of to agent 2 as signal d. When agent
discrete symbols An individual
mapping from agent 2 observes d, it maps it to some s2
λ : Ci × Cj → R, i ≠ j i's vocabulary to the ∈ S2 by using the function φ ­11.
A distance between signal space D and Then it maps the symbol to some
two points in the an inverse mapping φ­ point in its concept space by using
concept spaces of
1
i from the signal
ξ2. If this point is close to its
different agents space to the symbol observation f2 in the sense of λ, the
space communication process has
succeeded.
Timo Honkela, Ville Könönen, Tiina Lindh-Knuutila, and Mari-Sanna Paukkeri. Simulating processes of concept
formation and communication. Journal of Economic Methodology, 15(3):245–259, 2008.

14. 1
Patterns of
Movement

15. Why brains?
● What are the central differences
between plants and animals?
“The original need for a nervous
system was to coordinate movement,
so an organism could go find food,
instead of waiting for the food to
come to it.” http://www.fi.edu/learn/brain/
● An extreme example: A sea squirt transforms
from an “animal” to a “plant”. It absorbs its
own cerebral ganglion that it used to swim
about and find its attachment place.
http://goodheartextremescience.wordpress.com/2010/01/27/meet-the-creature-that-eats-its-own-brain/

16. Human movement

18. David Bailey's thesis (1997):
Verbs related to hand movement

19. Point of view from
cognitive linguistics
● The meaning of linguistic symbols in the mind of the
language users derives from the users' sensory
perceptions, their actions with the world and with each
other.
● For example: the meaning of the word 'walk' involves
● what walking looks like
● what it feels like to walk and after having walked
● how the world looks when walking
(e.g. objects approach at a certain speed, etc.).
● ...

20. Abstract vs concrete grounding
Ronald Langacker

21. Motion capture OptiTrack
Image analysis Animation
Kinect
Video analysis Reinforcement
learning
Klaus Förger
Tapio Takala
Xi Chen
Markus Koskela
Paul Wagner
Jorma Laaksonen
Machine learning
Timo Honkela
Harri Valpola
Robotics Oskar Kohonen
Language learning
Symbol grounding
Socio-cognitive modeling
Learning relations

22. Multimodally Grounded Language Technology
A project funded by Academy of Finland
2011-2014
Timo Honkela as the Principal Investigator
A collaboration between
departments of
* Information and Computer
Science, and
* Media Technology

23. Earlier
today: Turk on Gesture Interaction
Potential uses of gesture
interaction technologies:
● mouse replacement for
user interaction
● video game control
● navigation for
visualization
● sign language
recognition
● automatic transcription
of communication
● medical diagnosis and
rehabilitation
● sculpting
● conducting and playing
music
Matthew Turk, UCSB ● interactive art

24. Labeling movements
From an unpublished manuscript. Experiments by Klaus Förger.

25. Linking between modalities

26. Potential uses of the emerging technologies
● Multimodally grounded natural language interaction
and machine translation
● Animation based on linguistic instruction
● Automated skill instruction
(playing an instrument, learning some sports, etc.)
● Video annotation
● Addressing some of the fundamental issues
in traditional AI, cognitive science and
philosophy

27. 2
Contextuality and
Subjectivity of
Understanding

28. Meaning is contextual
red wine
red skin
red shirt
Gärdenfors: Conceptual Spaces
Hardin: Color for Philosophers

29. Meaning is contextual
WHITE
SNOW -
WHITE?

30. Meaning is contextual
● “Small”, “big” Fuzziness
● “White house”
● “Get”
● “Every” - “Every Swede is tall/blond”
● etc. etc.
Another comment:
Strict compositionality
cannot be assumed

31. Learning meaning from context
● Self-Organizing Semantic Maps
● Latent Semantic Analysis
● Latent Dirichlet Allocation
● WordICA
● etc. etc.

32. Classical example: Learning meaning from context:
Maps of words in Grimm fairy tales
Honkela, Pulkki & Kohonen 1995

33. Meaning is subjective

34. Meaning is subjective
● Good
● Fair
● Useful
A proper theory of
● Scientific meaning has to take
● Democratic this into account
● Sustainable
● etc.

35. Gary B. Fogel
11th of June, 2012
WCCI 2012

36. 2bMeasuring
Subjectivity of
Understanding

37. User-specific
difficulty
assessment
Basic architecture of the method

39. GICA:
Grounded
Intersubjective
Concept
Analysis
Description of the method

40. Publication:
Timo Honkela, Juha Raitio, Krista Lagus, Ilari T.
Nieminen, Nina Honkela, and Mika Pantzar.
Subjects on objects in contexts: Using GICA
method to quantify epistemological
subjectivity.
Proceedings of IJCNN 2012, International Joint
Conference on Neural Networks, pp. 2875-
2883, 2012.

41. Subjectifying: adding subjective
views into object-context matrices
Outcome: Subject-Object-Context (SOC) Tensors

42. Potential sources for subjectification
● Conceptual surveys:
● individual assessment of contextual
appropriateness
● Text mining:
● statistics of word/phrase-context patterns
● Empirical psychology:
● reaction times, etc.
● Brain research

43. Flattening: unfolding 3-way tensor
for traditional 2-way analysis

44. GICA:
Grounded
Intersubjective
Concept
Analysis
Examples of use

45. Data collection
CONTEXTS:
OBJECTS:
Relaxation
Happiness
Fitness
Wellbeing
SUBJECTS:
Event participants

46. MDS: Objects x Subjects
Fitness

47. NeRV: Objects x Subjects
Fitness
NeRV:
J. Venna, J. Peltonen, K. Nybo, H. Aidos, and S. Kaski. Information Retrieval Perspective to Nonlinear
Dimensionality Reduction for Data Visualization. Journal of Machine Learning Research, 11:451-490, 2010.

48. SOM: Objects x Subjects

49. Case 2: State of the Union
Addresses
● In this case, text mining is used for populating
the Subject-Object-Context tensor
● This took place by calculating the frequencies
on how often a subject uses an object word in
the context of a context word
● Context window of 30 words

50. Analysis of the word 'health'

51. Conclusions (1)
● Languages, including formal languages, should
be considered as tools for coordination, storing
and sharing knowledge in a compressed form –
approximate and relative to the point of view
taken
● Constructing a language or symbol system
(such as an ontology) is an investment and
spreading the language into use in a
community is even a larger one
Timo Honkela, Ville Könönen, Tiina Lindh-Knuutila, and Mari-Sanna Paukkeri. Simulating
processes of concept formation and communication. Journal of Economic Methodology,
15(3):245–259, 2008.

52. Conclusions (2)
● Making people aware of the differences in the
conceptual systems among them may have
different applications, e.g.,
● Helping in conflict resolution
● Promoting interdisciplinary communication
● Enhancing participatory processes and
democracy

53. From TEDxAALTO presentation “Measuring Subjectivity of Meaning –
and How it may change our life” with illustrations by Nelli Honkela

54. 1+2
Movement and
Subjectivity

55. goo.gl / UZnvH

56. Survey address:
goo.gl / UZnvH

57. Thank
you!