Improving Speech Recognition with Embodied Cognition and Behaviour-based Robotics

Improving Speech Recognition
with Embodied Cognition
and Behaviour-based Robotics
Improving Speech Recognition
with Embodied Cognition
and Behaviour-based Robotics
Jorge Davila-Chacon
University of Hamburg - Knowledge Technology
www.informatik.uni-hamburg.de/WTM/
Spotify ML Meetup – November 3rd
2014

MotivationMotivation
Jorge Davila-Chacon Bio-Inspired SSL for Robot ASR 2
• Why is bio-inspired SSL interesting / useful?

Neurobotic ExperimentsNeurobotic Experiments

Virtual Reality LabVirtual Reality Lab
Bauer, J., Dávila-Chacón, J., Strahl, E., Wermter, S. Smoke and Mirrors — Virtual Realities for Sensor Fusion Experiments in Biomimetic
Robotics. In: Multisensor Fusion and Integration for Intelligent Systems, 2012

Neurobotic ExperimentsNeurobotic Experiments
Jorge Davila-Chacon 5Bio-Inspired SSL for Robot ASR

Bio-Inspired Sound Source LocalisationBio-Inspired Sound Source Localisation
ITD
ILD
ITDs from
Low Frequencies
ITDs from
Low Frequencies
ILDs from
High Frequencies
ILDs from
High Frequencies
Spatial cues allow sound source localisation:
• Interaural Time Difference (ITD)
• Interaural Level Difference (ILD)
Spatial cues allow sound source localisation:
• Interaural Time Difference (ITD)
• Interaural Level Difference (ILD)
Same frequency
component
Same frequency
component

ITDs extracted in Medial Superior Olive
(MSO)
ITDs extracted in Medial Superior Olive
(MSO)
• AVCN - Anterior Ventral
Cochlear Nucleus
• AN - Auditory Nerve
• IC – Inferior Colliculus
Interaural Time Differences
Neuroanatomy
Neuroanatomy

Computational Principle
Computational Principle

ILDs extracted in Lateral Superior Olive
(LSO)
ILDs extracted in Lateral Superior Olive
(LSO)
• MNTB - Medial Nucleus of the Trapezoid Body
• IC – Inferior Colliculus
Interaural Level Differences
Neuroanatomy
Interaural Level Differences
Neuroanatomy

Output of
MSO and LSO
integrated in
IC
Output of
MSO and LSO
integrated in
IC
J. Dávila-Chacón, S. Heinrich, J. Liu, S. Wermter. Biomimetic Binaural Sound Source Localisation with Ego-Noise Cancellation.
International Conference on Artificial Neural Networks, 2012.

 MLP
IC
IC

J. Dávila-Chacón, S. Magg, J. Liu, S. Wermter. Neural and Statistical Processing of Spatial Cues for Sound Source Localisation.
International Joint Conference on Neural Networks, 2013.

Simple IC outputSimple IC output

Complex IC outputComplex IC output

Static SSLStatic SSL
Dynamic
SSL
Dynamic
SSL
 Feed forward
neural network

Robotic Automatic Speech RecognitionRobotic Automatic Speech Recognition
Platforms used for ASR: iCub and
Soundman
Platforms used for ASR: iCub and
Soundman

J. Dávila-Chacón, J. Twiefel, J. Liu, S. Wermter. Improving Humanoid Robot Speech Recognition with Sound Source Localisation.
Binary measure - Static ASRBinary measure - Static ASR

Continuous measure - Static
ASR
Continuous measure - Static
ASR

● Robotics as a “sandbox” for learning ML
● Neuroscience provides clues for computational principles
● Embodiment
• iCub allows computation of spatial cues
• Interaction with environment can reduce noise
● Signal processing with ANN
• Spiking ANN are an effective representation of spatial cues
• Bayesian integration important for dimensionality reduction
• Softmax Neural layer robust to ego-noise and reverberation
ConclusionConclusion

Future WorkFuture Work
● Neural SSL
• Integrate GPU version of MSO and LSO
• Propagation of probabilities through time
• From discrete to continuous
● Integration with vision
• From supervised to unsupervised SSL
• Possible extension to sensorimotor contingencies
• Vision to select between multiple sound sources
• Vision for speech segregation

Thank you for your attention.
jorgedch@gmail.com
LinkedIn: Jorge Davila Chacon
• J. Liu, D. Perez-Gonzalez, A. Rees, H. Erwin, S. Wermter. A biologically inspired spiking neural
network model of the auditory midbrain for sound source localisation. Neurocomputing (2010)
• J. Davila-Chacon, S. Heinrich, J. Liu, and S. Wermter. Biomimetic binaural sound source
localisation with ego-noise cancellation. International Conference on Artificial Neural Networks
(2012)
• J. Bauer, J. Davila-Chacon, E. Strahl, S. Wermter. Smoke and Mirrors — Virtual Realities for
Sensor Fusion Experiments in Biomimetic Robotics. Multisensor Fusion and Integration for
Intelligent Systems (2012)
• J. Davila-Chacon, S. Magg, J. Liu, S. Wermter. Neural and Statistical Processing of Spatial Cues
for Sound Source Localisation. International Joint Conference on Neural Networks (2013)
• J. Dávila-Chacón, J. Twiefel, J. Liu, S. Wermter. Improving Humanoid Robot Speech Recognition
with Sound Source Localisation. International Conference on Artificial Neural Networks (2014)

AppendixAppendix
Best performances with clustering layer

AppendixAppendix
Bayesian IC model

AppendixAppendix
Levenshtein distance

Improving Speech Recognition with Embodied Cognition and Behaviour-based Robotics

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (7)

Similar to Improving Speech Recognition with Embodied Cognition and Behaviour-based Robotics

Similar to Improving Speech Recognition with Embodied Cognition and Behaviour-based Robotics (20)

Improving Speech Recognition with Embodied Cognition and Behaviour-based Robotics