1. Tadahiro Taniguchi, Associate professor
College of information tech.&sci., Ritsumeikan univ.
Shonan meeting
2013/11/11 ‐ 14
Shonan
village

2. Tadahiro Taniguchi @tanichu
 Associate professor, Emergent system laboratory
 College of information science and technology,
Ritsumeikan University, Japan
 2006, PhD Eng. , Kyoto university
 2008‐ Assistant professor
 2010‐ Associate professor
Machine
learning
Cognitive
robotics
Decentralized
autonomous
system
Human
communication
Symbol Emergence in Robotics

3. Computational understandings of mental development
‐ From behavioral learning to language acquisition ‐
 A human child acquires many
physical skills, concepts and
knowledge including language
through physical and social
interaction with his/her environment.
 How can we become able to
communicate using symbolic system?
 We’d like to obtain computational
under standings of mental
development and symbol emergence.
Constructive approach towards human intelligence
enabling semiotic communication
3

4. Symbol grounding problem
 How can a robot “ground” his/her symbol to the real
physical world? [Harnad ‘90]
 The robot has to give some meaning to a symbol in its
symbolic system designed by a human designer through
sensor‐motor interaction with its environment.
 Arbitrary nature of symbol (in semiotics)
 Arbitrariness of labeling/naming
 Arbitrariness of categorization/segmentation
 SGP implicitly assumes that human “arbitrarily
SGP is
missing
evolved” symbolic system is a “true” symbolic system.
Should a symbolic system for a robot be same as human’s ?

5. Understanding that symbolic system is an emergent
property of human cognitive and social system
 “Worlds of robots”
 “Worlds of animals” (Uexküll 1934)
 Umwelt (self‐centered world)
 Animals can receive information only from
their sensor‐motor system.
 A human has to obtain various behaviors, concepts
and language on the basis of experiences in his/her
umwelt (closed cognitive system).
 Human Symbolic system should be understand
on the basis of human embodiment (sensor‐motor
system).
Concepts have to be formed
on the basis of sensor‐motor information
in a bottom‐up way
Ernst Mach’s famous picture
"Early Scheme for a circular
Feedback Circle by Uxkull

6. Social constraints in semiotic communication
 Concept formation is not “enough” for semiotic
communication.
 We have to share a symbolic system involving
 syntax, semantics and pragmatics
 lexicon and mutual belief
bad interpretation
Shared
symbolic system
bad naming
Sheep the coming,
aren’t they!!!
bad syntax
(true) A thief is coming!!
It’s impossible to help her

7. Social constraints in semiotic communication
 Concept formation is not “enough” for semiotic
communication.
 We have to share a symbolic system involving
 syntax, semantics and lexicon
 pragmatics and mutual belief
situation recognition
constraint
interpretation
constraint
Shared
symbolic system
(;O;)
constraint
A thief is coming!!
speech generation
!
decision making
constraint

8. Before introducing “symbol emergent system”, please remind....
Emergent System
 Emergent property
 Through local interaction between elements of a system,
global (or macro‐level) order or pattern emerges. The
macro‐level order becomes constraints of micro‐level
dynamics, and governs the micro‐level interaction. Such
bidirectional process makes the system have novel
function, morphology and/or behavior.
emergence
micro‐macro loop
constraint

9. Symbol emergent system
Symbolic System
Emergence
Shared lexicon, syntax,
pragmatics, semantics,
and mutual belief
Concept formation through
interaction with environment
Physical interaction
Semiotic / Social interaction
谷口忠大. 「コミュニケーションするロボットは創れるか‐記号創発システムへの構成論的アプローチ」(2010).
Tadahiro Taniguchi “Constructive approach toward symbol emergence system” (in Japanese)

10. Symbol emergence in robotics (SER)
 Constructive approach toward symbol emergent systems
 “Constructive” viewpoint to an intelligent system which
uses symbolic system.
1. Constructive approach
 Understanding human intelligence by constructing robots
obtaining symbolic system
2.
Constructivism
 Understanding human intelligence which construct his/her
subjective world (Umwelt).
Understanding
by developing “models”
(c) Nagai lab.

11. Topics in SER
multimodal
communication
learning
interaction strategy
Prof. Nagai
concept
formation
learning motor skills
language acquisition
and segmentation of
and mental development
time‐series infomation
emergence of
communication

12. Segmentation of sensor‐motor
time‐series information
 Dynamics in sensor‐motor time‐
series information changes
depending on its environmental
situations.
 An autonomous system should
differentiate such situations and
obtain representations.
 How do humans and how can the
robots segment the time‐series
data?
What is the criteria of segmentation?
What is an adequate computational algorithm
for the segmentation?

13. Modular learning architecture
for segmenting sensor‐motor time series
 MOSAIC [Kawato et al.]
 Mixture of experts [Jacobs et al.]
 HAMMER [Yiannis et al.]
 Dual schemata model [Taniguchi et al.]
“Segment” corresponds to
a linear system
local information
a short‐term event
How can we grasp long‐term context?
http://www.cns.atr.jp/cnb/HarunoG/
harunoG.ja.html
T. Taniguchi, T. Sawaragi, "Incremental acquisition of multiple nonlinear forward models based on
differentiation process of schema model“, Neural Networks, Vol.21 (1), pp.13‐27 .(2008)

14. Hierarchical mixture of
RNN experts [Tani and Norfi 1999]
階層的な分節構造の
自己組織的な獲得
Representation of a room and a part of
a room and was encoded in RNN
in a hierarchical manner.
Dynamical System Approach
 Symbols are implicitly obtained.
 It is difficult to understand how the system formed concepts.

15. Double articulation structure in
semiotic data
 Semiotic time‐series data often has double articulation
 Speech signal is a continuous and high‐dimensional time‐series.
 Spoken sentence is considered as a sequence of phoneme
 People don’t give a meaning to a phoneme, but give a meaning to
a word which is a sequence of phoneme.
How
[h a u ]
h au
much
[m ʌ́
m ʌ́
is
tʃ]
[i
tʃ
I
semantic
(meaningful)
this?
z
]
z
[ð
ð
í
s]
í s
meaningless
unsegmented

16. Double Articulation Analyzer
to estimate latent double articulation structure
 Unsupervised learning
 Estimating
NPYLM
[Moachihashi ‘09]



iHMM
[Fox ‘07]
Language model
Emission distribution
Segments and chunks
 Conditions
 Unknown number of
words and letters
 Unknown emission
distributions
 Inference
 Approximate Inference
Procedure of
Double Articulation
Analyzer [Taniguchi ‘11]
Nonparametric Bayesian approach
Tadahiro Taniguchi, Shogo Nagasaka, Double Articulation Analyzer for Unsegmented Human Motion using Pitman‐
Yor Language model and Infinite Hidden Markov Model, 2011 IEEE/SICE SII.(2011)

17. iHMM
[Fox ‘07]
sticky HDP‐HMM [Fox ‘07]
 An infinite HMM is an HMM
which can estimate the number of
hidden state flexibly (potentially
infinite)[Beal ‘02] [Teh ‘06]
 Sticky HDP‐HMM is a generative
model for iHMM with a stickiness
parameter [Fox ‘07].
 Fox et al. developed fast inference
algorithm with weak‐limit
approximation.
 Gaussian emission distribution
γ
β
κ
α
πk
z1
λ
z2
z3
zT
y1
y2
y3
yT
θk
∞
E.B. Fox, E.B. Sudderth, M.I. Jordan, A.S. Willsky, "A Sticky HDP‐HMM with Application to Speaker
Diarization," Annals of Applied Statistics, June 2011. (First appeared as MIT LIDS Technical Report P‐2777, November
2007.)

18. NPYLM
[Moachihashi ‘09]
Unsupervised morphological analysis
Thisisanapple ‐> This is an apple
わたしはたなかです． ‐> わたし は たなか です
 Morphological analysis
 To segment sentences into
words(morpheme).
 This usually requires dictionary
(preexisting knowledge of language
model).
 Unsupervised morphological
analysis
 It does not assume preexisting
dictionary.
 Mochihashi proposed an
unsupervised morphological
analysis method based on Nested
Pitman‐Yor language model
(NPYLM)[Mochihashi ‘09].
http://chasen.org/~daiti‐m/paper/nl190segment‐slides.pdf

19. NPYLM [Mochihashi ‘09]
(Nested Pitman‐Yor Language Model)
 Mochihashi developed NPYLM for unsupervised
morphological analysis.
 NPYLM has word n‐gram model and letter ngram model,
hierarchically. Each adopts hierarchical Pitman‐Yor
language model as a language model (smoothing method).
 Bayesian nonparametric model
 Efficient blocked Gibbs sampler
Daichi Mochihashi, Takeshi Yamada, Naonori Ueda."Bayesian Unsupervised Word Segmentation
with Nested Pitman‐Yor Language Modeling". ACL‐IJCNLP 2009, pp.100‐108, 2009.

20. Application of
Double Articulation Analyzer
Human motion data
•
•
Imitation learning [Taniguchi ‘11]
Motion segmentation [Taniguchi’11]
Driving behavior
DAA
•
•
•
•
•
Extracting driving chunk [Nagasaka ‘12]
Detecting intentional changing points [Takenaka ‘12]
Prediction [Taniguchi ‘12]
Video summarization [Takenaka ‘12]
collaborative work
For topic modeling []
Auditory data*
•
Language acquisition [Araki ‘12]
*Only NPYLM was used.
with DENSO co.
bottle !!
collaborative work
with Nagai lab.

21. Contextual Scene Segmentation of Driving Behavior
based on Double Articulation Analyzer [Takenaka ‘12]
 DAA could extract changing points of driving context
recognized by human
Kazuhito Takenaka, Takashi Bando, Shogo Nagasaka, Tadahiro Taniguchi, Kentarou Hitomi,
Contextual Scene Segmentation of Driving Behavior based on Double Articulation Analyzer, IEEE/RSJ
International Conference on Intelligent Robots and Systems 2012 (IROS 2012), 4847‐4852 .(2012)

22. Semiotic Prediction of Driving Behavior using
Unsupervised Double Articulation Analyzer [Taniguchi ‘12]
Histogram
Averaged number of correctly
predicted hidden states
Tadahiro Taniguchi, Shogo Nagasaka, Kentarou Hitomi, Naiwala P. Chandrasiri, and Takashi Bando
Semiotic Prediction of Driving Behavior using Unsupervised Double Articulation Analyzer
2012 IEEE Intelligent Vehicles Symposium, 849 ‐ 854 .(2012)

23. Drive Video Summarization based on Double Articulation
Structure of Driving Behavior [Takenaka ‘12]
http://www.youtube.com/watch?v=knwiO6dVbnY
Kazuhito Takenaka, Takashi Bando, Shogo Nagasaka, Tadahiro Taniguchi, "Drive
Video Summarization based on Double Articulation Structure of Driving Behavior",
ACM multim media 2012,

24. Online Learning of Concepts and Words Using
Multimodal LDA and Hierarchical Pitman‐Yor
Language Model [Araki ‘12]
 Connecting multimodal categorization and word
segmentation to achieve language acquisition through
daily interaction.
Takaya Araki, Tomoaki Nakamura, Takayuki Nagai, Shogo Nagasaka, Tadahiro Taniguchi, Naoto Iwahashi
Online Learning of Concepts and Words Using Multimodal LDA and Hierarchical Pitman‐Yor Language Model
IEEE/RSJ International Conference on Intelligent Robots and Systems 2012 (IROS 2012), 1623‐1630 .(2012)

25. Conclusion
 Summary
 Defining the symbol emergence system
 Introducing symbol emergence in robotics
 Introducing double articulation analyzer
 Current challenge
 Unsupervised lexicon acquisition using double
articulation analyzer and multi‐modal categorization
 Discussion topic
 What is the important feature of language which we
have to model to obtain computational understanding
of human language.