○Akira Taniguchi, Yoshinobu Hagiwara, Tadahiro Taniguchi, and Tetsunari Inamura, "Online Spatial Concept and Lexical Acquisition with Simultaneous Localization and Mapping", IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS2017), 2017. Video: https://youtu.be/hVKQCdbRQVM

1. Online Spatial Concept and
Lexical Acquisition with
Simultaneous Localization and
Mapping
IROS2017@Vancouver
1
Akira Taniguchi *, Yoshinobu Hagiwara *, Tadahiro Taniguchi *,
Tetsunari Inamura **
* Ritsumeikan University, Japan. (E-mail: a.taniguchi@em.ci.ritsumei.ac.jp)
** National Institute of Informatics / The Graduate University for Advanced
Studies, Japan.

2. Research background
Robots coexisting with humans and operating in various
environments are required to adaptively learn the
spatial concepts (place categories and a lexicon) while
incrementally generating an environmental map.
◦ Spatial concepts are such that their target domain may be
unclear and may differ according to the user and environment.
◦ Therefore, it is difficult to manually design spatial concepts in
advance, and it is desirable for robots to autonomously learn
spatial concepts based on their own experiences.
2
Which area is
the same place?
What scenery
can I see?
What is the name
of this place?

3. Spatial concept
Spatial concept based on multimodal information
◦ Word information (Place names)
◦ Place information (Position distribution)
◦ Image information (Visual features)
Meething room
Laboratory
Elevator hall
3

4. [Taniguchi 16] Taniguchi, A et.al. Spatial Concept Acquisition for a Mobile Robot that Integrates Self-Localization and
Unsupervised Word Discovery from Spoken Sentences, IEEE TCDS, Vol. 8, No. 4, pp. 285–297 (2016)
Previous method : SpCoA [Taniguchi 16]
Nonparametric Bayesian spatial concept acquisition method
The main features
• This model can learn unknown words from continuous speech signals.
• This model can learn an appropriate number of spatial concepts,
depending on the data. (using nonparametric Beysian approach)
• This model can learn many-to-many correspondences between names and
places by relating several places to several names via spatial concepts.
/sofamae//hoNdana//qgeNkaN/
/kiqchiN/
/daidokoro/ /terebimae//gomibakoo/
/tereburunoatari/
4
Learning result in Japanese

5. Previous method : SpCoA [Taniguchi 16]
Nonparametric Bayesian spatial concept acquisition method
◦ Batch learning
◦ The robot learns the spatial concepts after getting sufficient data while
moving the environment.
◦ Environmental map
◦ This method cannot learn spatial concepts from unknown
environments without a map.
◦ Over-segmentation problem
◦ It’s is caused by word segmentation of phoneme-recognition results
including errors.
[Taniguchi 16] Taniguchi, A et.al. Spatial Concept Acquisition for a Mobile Robot that Integrates Self-Localization and
Unsupervised Word Discovery from Spoken Sentences, IEEE TCDS, Vol. 8, No. 4, pp. 285–297 (2016)
This place is the
laboratory.
|dis|pu|rai|su|iz
a|ra|bora|to|ri|
???
5

6. Research purpose
Mobile robots learn spatial concepts, a lexicon, and an environmental
map incrementally from interaction with an environment and human,
even in an unknown environment without prior knowledge.
6

7. The proposed method : SpCoSLAM
This model integrates multimodal place categorization,
lexical acquisition and SLAM as one Bayesian generative model.
7Gray nodes indicate observation variables.
1tx

xt

xt1
tz
tu
tC
ti

l 

00 ,m
00 ,V
1tz
1tu
1tz
1tu

LMtyAM
tS

ltf 
∞
∞
m
lW
k
k

8. The proposed method : SpCoSLAM
This model integrates multimodal place categorization,
lexical acquisition and SLAM as one Bayesian generative model.
8Gray nodes indicate observation variables.
1tx

xt

xt1
tz
tu
tC
ti

l 

00 ,m
00 ,V
1tz
1tu
1tz
1tu

LMtyAM
tS

ltf 
∞
∞
m
lW
k
k
Simultaneous localization and mapping (SLAM)
Position
distribution
(Gaussian
distribution)
Nonparametric Bayesian
multimodal place categorization
Index of place
Image feature
Words
Lexical acquisition (Speech recognition and word segmentation)

9. FastSLAM and SpCoSLAM
Simultaneous Localization And Mapping (SLAM)
◦ FastSLAM has realized an on-line algorithm for efficient
self-localization and mapping using a Rao-Blackwellized
particle filter (RBPF) [Grisetti 05] .
Online learning algorithm of SpCoSLAM
◦ The online learning algorithm can be derived by
introducing sequential update equations for estimating
the parameters of the spatial concepts into the
formulation of FastSLAM based on RBPF.
9
[Grisetti 05] G. Grisetti, C. Stachniss, and W. Burgard, “Improving grid-based SLAM with Rao-Blackwellized
particle filters by adaptive proposals and selective resampling,” in Proceedings of ICRA, 2005.

10. FastSLAM and SpCoSLAM
FastSLAM
SpCoSLAM
Self-
position map
Control
data
Sensor
data
Latent variables
Model parameters
Hyperprameters
Language model
Acoustic model
Speech signal
Image feature
LM is updated. Model parameters
are updated.
10
Rao-Blackwellized
particle filter (RBPF)

11. Online learning algorithm
of SpCoSLAM
2. Calculating the proposal
distribution of FastSLAM 2.0
3. Word segmentation, sampling
latent variables, and calculating
weights
6. Updating a language model
7. Resampling of particles
4. mapping
5. Estimation of parameters of
spatial concepts
1. Speech recognition
11

12. Experiment I : Online learning
We performed experiments for online learning of spatial
concepts in a novel environment.
[1] latticelm: http://www.phontron.com/latticelm/
[2] The robotics data set repository (radish): http://radish.sourceforge.net/
Conditions
12
Middleware Robot Operating System (ROS) indigo
Speech recognition
system
Julius dictation-kit-v4.3.1-linux (GMM-HMM decoding),
Japanese syllable dictionary
Word segmentation
system
latticelm [1]
(WFST-based word segmentation system)
Image feature extractor Caffe (CNN model of Places205-AlexNet)
Dataset Robotics Data Set Repository (Radish) [2]
albert-b-laser-vision by Cyrill Stachniss
• Rosbag file (odometry, depth, image data)
Speech data 50 sentences including 10 types of various phrases
WFST：Weighted Finite-State Transducer

13. Experiment I : Online learning
13
Video: https://youtu.be/hVKQCdbRQVM

14. Experiment I : Online learning
1
2
3 4
5
6
Step 15
1
23
4
5
6
7
8
Step 30
1
23
4
5
6
78
9
10
Step 50
Position distribution: 6 Position distribution: 1 Position distribution: 8
14
Correct: /ikidomari/
(The end of corridor)
Estimated word:
/ikidomaekidayao/
Estimated word:
/kyooyuusehi/
Correct: /kyouyuuseki/
(Sharing desk)
Estimated words:
/upuriNpabeyatarero/
/izaridokourodayo/
Correct1: /puriNtaabeya/
Correct2: /daidokoro/
(Printer room, kitchen)
Words are estimated by

15. Experiment I : Online learning
We compare the performance of four methods as
follows:
(A) SpCoSLAM (The proposed method)
(B) Online SpCoA based on RBPF
(C) Online SpCoA
(D) SpCoA (Batch learning) [Taniguchi 16]
[Taniguchi 16] Taniguchi, A et.al. Spatial Concept Acquisition for a Mobile Robot that Integrates
Self-Localization and Unsupervised Word Discovery from Spoken Sentences, IEEE Transactions on Cognitive
and Developmental Systems, Vol. 8, No. 4, pp. 285–297 (2016)
The number of
particles：30
15
Methods (B), (C), and (D) based on SpCoA did not perform
the update of a language model and did not use image
features.

16. Experiment I : Online learning
16
We compare the performance of SpCoSLAM and SpCoA-based methods.
1tx

xt

xt1
tz
tu
tC
ti

l 

00 ,m
00 ,V
1tz
1tu
1tz
1tu

LMtyAM
tS

ltf 
∞
∞
m
lW
k
k
1tx

xt

xt1
tz
tu
tC
ti

l 

00,m
00 ,V
1tz
1tu
1tz
1tu

tS
∞
∞
m
lW
k
k
SpCoA-based methodSpCoSLAM
SpCoA did not perform the update of a language model and did not use image features.

17. Evaluation I :
The estimated number of spatial concepts
Figures show the number of spatial concepts and the number
of position distributions by online learning.
True data was determined by an user based on teaching data.
SpCoSLAM was closer to the true data than other methods.
17

18. Evaluation II :
Word segmentation in the lexical acquisition
Figure shows the number of segmented words.
SpCoSLAM improved the over-segmentation problem by
updating the language model sequentially.
SpCoSLAM was
closer to the phrase
segmentation.
Over-segmentation
Morpheme: The morphological segmentation (using MeCab)
Phrase: The phrase segmentation (segmenting words only before and after
the name of the place.)
18

19. Evaluation II :
Word segmentation in the lexical acquisition
SpCoSLAM
SpCoA 19
SpCoSLAM
SpCoA
SpCoSLAM
SpCoA
(in English)
(in Japanese)

20. Experiment II :
Place recognition using a speech signal
When the user says “Go to **.”, the estimation of a target
position was calculated as follows:
SpCoSLAM showed the
highest overall evaluation
values of the online methods.
We calculated the place recognition rate (PRR) that the rate
of positions estimated within the correct area in the test data.
SpCoA (0.5)
20

21. Conclusion
 We proposed an online learning method of spatial
concepts and an environmental map by a mobile
robot.
 The proposed method integrated the spatial concept
acquisition into SLAM by an RBPF-based approach.
 In the experiments, we conducted online learning in
a novel environment by the robot without a pre-
existing lexicon and map.
 SpCoSLAM improved the performance of place recognition
using a speech signal in online learning methods.
 SpCoSLAM improved over-segmentation problem in lexical
acquisition by updating the language model sequentially.
21THANK YOU FOR YOUR KIND ATTENTION.