http://www.meetup.com/es-warsaw/events/220138468/

1. Audio search
Andrzej Dudziec

2. Outline
● Introduction
● Speech recognition
● Phonetic algorithms
● Evaluation
● Results
● Conclusions

3. Introduction

4. Introduction
audio text

5. Speech recognition
● Words consists of letters e.g. ‘ONE’ - ‘O’, ‘N’, ‘E’
● Speech consists of phonemes e.g. /wʌn/ - ‘W’, ‘AH’, ‘N’

6. Speech recognition
phonemes
AM

7. Speech recognition
● one W AH N
● two T UW
● three TH R IY
● four F AO R
● five F AY V
● six S IH K S
● seven S EH V AH N
● eight EY T
● nine N AY N
● ten T EH N
phonemes
words
sentences
AM
dict
LM

8. Speech recognition
phonemes
words
sentences
AM
dict
LM
Issues
● Acoustic level
○ background noise
○ multiple speakers
○ accent, dialect, sex, mood
○ coarticulation
● Dictionary level
○ homonyms (be & bee, I scream & ice cream)

9. Phonetic algorithms
Thompson -> thompson
thompson -> th3mps3n
th3mps3n -> th3mpS3n
th3mpS3n -> Th3mpS3n
Th3mpS3n -> Th3mPS3n
Th3mPS3n -> Th3MPS3n
Th3MPS3n -> Th3MPS3N
Th3MPS3N -> T23MPS3N
T23MPS3N -> TMPSN
TMPSN111111 -> TMPSN1
sixteen sixty
Soundex
Metaphone
Caverphone
Soundex
Metaphone
Caverphone
S235
SKST
SKTN11
S230
SKST
SKTA11
● Soundex
● Metaphone
● Caverphone

10. Phonetic algorithms
Ackermann AzuronSoundex SoundexA265 A265

11. Metaphone code computation algorithm
Remove all repeating neighboring letters except letter C.
The beginning of the word should be transformed using the
following rules:
KN → N
GN → N
PN → N
AE → E
WR → R
Remove B letter at the end, if it is after M letter.
Replace C using the rules below:
With Х: CIA → XIA, SCH → SKH, CH → XH
With S: CI → SI, CE → SE, CY → SY
With K: C → K
Replace D using the following rules:
With J: DGE → JGE, DGY → JGY, DGI → JGY
With T: D → T
Replace GH → H, except it is at the end or before a vowel.
Replace GN → N and GNED → NED, if they are at the end.
Replace G using the following rules
With J: GI → JI, GE → JE, GY → JY
With K: G → K
Remove all H after a vowel but not before a vowel.
Perform following transformations using the rules below:
CK → K
PH → F
Q → K
V → F
Z → S
Replace S with X:
SH → XH
SIO → XIO
SIA → XIA
Replace T using the following rules
With X: TIA → XIA, TIO → XIO
With 0: TH → 0
Remove: TCH → CH
Transform WH → W at the beginning. Remove W if there is no vowel
after it.
If X is at the beginning, then replace X → S, else replace X → KS
Remove all Y which are not before a vowel.
Remove all vowels except vowel at the start of the word.

12. Daitch-Mokotoff Soundex
Letter combination At the
start
After a
vowel
Other
SCHTSCH, SCHTSH, SCHTCH, SHTCH, SHCH,
SHTSH, STCH, STSCH, STRZ, STRS, STSH,
SZCZ, SZCS
2 4 4
SHT, SCHT, SCHD, ST, SZT, SHD, SZD, SD 2 43 43
CSZ, CZS, CS, CZ, DRZ, DRS, DSH, DS, DZH,
DZS, DZ, TRZ, TRS, TRCH, TSH, TTSZ, TTZ,
TZS, TSZ, SZ, TTCH, TCH, TTSCH, ZSCH,
ZHSH, SCH, SH, TTS, TC, TS, TZ, ZH, ZS
4 4 4

13. Phonetic algorithms

14. Evaluation

15. Results
help ≠ helped
hell ≠ heaven

16. Results

17. Results

18. Results
preprocessing
audio
snippets
XMLtext
audio
snippets

19. Results

20. Results

21. Results

22. Conclusions
● good recognition model and audio preprocessing is crucial,
consider speed vs accuracy
● phonetic filtering increases recall but decreases precision
● phonetic filters as improvement, not standalone
● consider fuzzy search

23. Use cases
● audio archive
● looking up broadcast
○ opinion mining
○ collecting information
● voice control
● dictation
○ short notes
○ voice mail -> text messages

24. Discussion?