This document provides an overview of automatic speech recognition (ASR) technologies. It discusses the challenges of ASR processing including speaker and language variability as well as noise. It describes pattern-based and statistical approaches to ASR, including hidden Markov models, Gaussian mixture models, and deep neural networks. The document lists some open source and commercial ASR engines and provides steps to build an online ASR system using open source tools like Kaldi, Kaldi gstreamer server, and Dictate.js. It concludes with a live demo of a personal ASR system built with these tools.

1. Kaldi&voice+
Your+personal+speech+recogni4on+
server+using+open+source+code+
Xavier+Anguera+
CTO+&+CSO,+ELSA+Corp.+
xavier@elsanow.io+

2. Outline+
• Intro+
• What+is+speech+recogni4on+
– Applica4ons+
• Approaches+to+ASR+
– PaHern+matching+approaches+
– Sta4s4cal&based+approaches+
• Available+speech+recogni4on+engines+
– “open”+source+
– Online+commercial+systems+
• Building+your+own+online+system+
– Live+demo+

3. Automa4c+Speech+Recogni4on+
• Automa'c)Speech)Recogni'on)(ASR))is+the+
process+of+conver4ng+an+unknown+speech+
waveform+into+the+corresponding+orthographic+
transcrip4on.++
Image:+hHp://blogs.msdn.com/b/devschool/archive/2012/02/06/speech&recogni4on&using&visual&studio&determining&the&bna.aspx+

4. Content2
Personal22
context2
Search+
Summary+
Transcripts+
Meaning+Age+
Gender+
Height+
Spoken+language+
Spoken+dialect+
Spoken+accent+
Literacy+level+
Speaker+ID+
Personality+traits+(OCEAN)+
Speech+likability+
Speech+intelligibility+
Sleepiness/4redness+
Intoxica4on+level+
Emo4on+
State+of+interest+
Image:+Telefonica+I+D+

5. Applica4ons+of+Speech+Recogni4on/Understanding+(ASR/ASU)+
! Dicta4on+
! Telephone&based+Informa4on++
! direc4ons,+air+travel,+banking,+etc+
! Polls,+online+shopping+
! Call+rou4ng+
! Hands&free+
! in+car,+computer,+home(domo4cs),+controlling+tools+
! Second+language+(accent+reduc4on)+
! Audio+archive+searching+
! Help+for+disabled+people+

6. How+do+humans+do+it?+
Ar4cula4on+system+of+one+
person+produces+sound+waves+
which+the+ear+of+another+person+
conveys+to+the+brain+for+
processing+

7. How+can+computers+do+it?+
• Digi4za4on+
• Acous4c+analysis+of+the+
speech+signal+
• Linguis4c+interpreta4on+
Acous4c+waveform+ Acous4c+signal+
Speech+recogni4on+

8. Challenges+in+ASR+processing+
! Speaker+variability+
! Inter&speaker:+Vocal+tract,+gender,+dialects+
! Intra&speaker:+:+stress,+age,+humor,+changes+of+ar4cula4on+due+to+
environment+influence,+…+
! Language+variability+
! From+isolated+words+to+con4nuous+speech+
! Out&of&vocabulary+words+
! Vocabulary+size+and+domain+
! From+just+a+few+words+(e.g.+Isolated+numbers)+to+large+vocabulary+speech+
recogni4on+
! Domain+that+is+being+recognized+(medical,+social,+engineering,+…)+
! Noise+
! Convolu4ve:+recording/transmission+condi4ons,+reverbera4on+
! Addi4ve:+recording+environment,+transmission+SNR+

9. Approaches+to+ASR+
! PaHern&based+approaches+
! Sta4s4cs&based+approaches+

10. PaHern&based+speech+recogni4on+
" Feature measurement: Filter Bank, MFCC, LPC, DFT, ...
" Pattern training: Creation of a reference pattern derived from an averaging technique
" Pattern classification: Compare speech patterns with a local distance measure and a
global time alignment procedure (DTW)
" Decision logic: similarity scores are used to decide which is the best reference
pattern.

11. Template+Matching+Mechanism+

12. TDP:++Speech+Recogni4on+
Alignment+Example+

13. Sta4s4cs&based+approaches+
• Can+be+seen+as+extension+of+template&based+approach,+
using+more+powerful+mathema4cal+and+sta4s4cal+tools+
• Some4mes+seen+as+ an4&linguis4c +approach+
– Fred+Jelinek+(IBM,+1988):+ Every+4me+I+fire+a+linguist+my+
system+improves
• Process:+
1. Collect+a+large2corpus+of+transcribed+speech+recordings+
2. Train+the+computer+to+learn+the+correspondences+
( machine+learning )+
3. At+run+4me,+apply+sta4s4cal+processes+to+search+through+
the+space+of+all+possible+solu4ons,+and+pick+the+
sta4s4cally+most+likely+one+

15. Sta4s4cs&based+approaches+
• Hidden+Markov+Models+(HMM)+
• Gaussian+Mixture+Models+(GMM)+
• Deep+Neural+Networks+(DNN)+

16. Markov+model+
Output2=2sequence2of2states2
Image:+hHp://madhukaudantha.blogspot.pt/2014/05/markov&models&and&hidden&markov&models.html+

17. Hidden+Markov+Models+(HMM)+
Output2=2observa:ons2linked2to2the2states2through2a2predefined2
probability2distribu:on2!2modeled2using2GMM2or2DNN2models2
Image:+hHp://izanami.tl.fukuoka&u.ac.jp/SLPL/HMM/HTKBook/node5.html+

18. 19/34+
HMMs+for+some+words+

19. Gaussian+Mixture+Models+(GMM)+
1D+GMM+
2D+GMM+

20. Dep+neural+networks+
Image:+hHp://www.amax.com/blog/+

21. A2neuron2in2our2brain2
Image:+hHp://www.medicalsciencenavigator.com/how&to&study&for&anatomy&and&physiology/why&sleep&improves&memory+

22. Classical+representa4on+of+a+neuron++
Long+short&term+memory+cells++

23. DNN+evolu4on+
• We+started+to+use+mul4layer+perceptrons+
(MLP’s)+about+25+years+ago+[1]+
– Neural+networks+with+1+or+few+hidden+layers+
• Around+2010+G.+Hinton+and+S.+Bengio+
(separately)+proposed+methods+to+effec4vely+
train+many+hidden+layers+
– Machines+have+become+much+more+powerful+
– Lots+of+audio+data+with+transcrip4ons+areavailable++
[1]+“Merging+Mul4layer+perceptrons+and+Hidden+Markov+Models:+some+experiments+in+con4nuous+
speech+recogni4on”,+Herve+Bourlard+and+Nelson+Morgan,+Technical+report+ICSI,+1989+

24. Image:+hHp://whatsnext.nuance.com/category/in&the&labs/+
Processing+power+evolu4on+

25. Image:+hHp://whatsnext.nuance.com/category/in&the&labs/+
ASR+performance+evolu4on+

26. Speech+recogni4on+engines+
• HTK+(hHp://htk.eng.cam.ac.uk/),+non&
commercial+license+
• Sphinx+(hHp://cmusphinx.sourceforge.net/),+
GPL+
• Julius+(hHp://julius.osdn.jp/en_index.php),+
open+
• Kaldi+(hHp://www.kaldi&asr.org/),+Apache+
license+

27. Online+ASR&STT+services+
• Google+voice+(
hHps://console.developers.google.com/
project)+
• ATT+voice+recogni4on+(
hHp://developer.aH.com/apis/speech)+
• Wit.ai+(hHps://wit.ai/)+

28. Building+an+ASR+with+open+source+tools+
• We+need:+
– Speech+recogni4on+engine+
– Speech+databases+/+models+
– Online+speech+server+
– Frontend+interfaces+

29. Kōnele+app+
Dictate.js+

30. My+toolchain+
• Kaldi+ASR+++++++++++++++++++++
hHp://www.kaldi&asr.org/+
• Kaldi+gstreamer+server+
hHps://github.com/alumae/kaldi&gstreamer&
server+
• Dictate.js++
hHp://kaljurand.github.io/dictate.js/+
• Kōnele+app+
hHps://kaljurand.github.io/K6nele/+

31. Demo+