speech extraction system

1. Introduction to Automatic
Speech Recognition

2. Outline

Define the problem

What is speech?

Feature Selection

Models
 Early methods
 Modern statistical models

Current State of ASR

Future Work

3. The ASR Problem

There is no single ASR problem

The problem depends on many factors
 Microphone: Close-mic, throat-mic, microphone
array, audio-visual
 Sources: band-limited, background noise,
reverberation
 Speaker: speaker dependent, speaker
independent
 Language: open/closed vocabulary, vocabulary
size, read/spontaneous speech
 Output: Transcription, speaker id, keywords

4. Performance Evaluation

Accuracy
 Percentage of tokens correctly recognized

Error Rate
 Inverse of accuracy

Token Type
 Phones
 Words*
 Sentences
 Semantics?

5. What is Speech?

Analog signal produced by humans

You can think about the speech signal being
decomposed into the source and filter

The source is the vocal folds in voiced speech

The filter is the vocal tract and articulators

6. Speech Production

7. Speech Production

8. Speech Production

9. Speech Visualization

10. Speech Visualization

11. Speech Visualization

12. Feature Selection

As in any data-driven task, the data must be
represented in some format

Cepstral features have been found to perform
well

They represent the frequency of the
frequencies

Mel-frequency cepstral coefficients (MFCC)
are the most common variety

13. Where do we stand?

Defined the multiple problems associated with
ASR

Described how speech is produced

Illustrated how speech can be represented in
an ASR system

Now that we have the data, how do we
recognize the speech?

14. Radio Rex

First known attempt at speech recognition

A toy from 1922

Worked by analyzing the signal strength at
500Hz

15. Actual speech recognition
systems

Originally thought to be a relatively simple
task requiring a few years of concerted effort

1969, “Wither speech recognition” is
published

A DARPA project ran from 1971-1976 in
response to the statements in the Pierce
article

We can examine a few general systems

16. Template-Based ASR

Originally only worked for isolated words

Performs best when training and testing
conditions are best

For each word we want to recognize, we
store a template or example based on actual
data

Each test utterance is checked against the
templates to find the best match

Uses the Dynamic Time Warping (DTW)
algorithm

17. Dynamic Time Warping

Create a similarity matrix for the two
utterances

Use dynamic programming to find the lowest
cost path

18. Hearsay-II

One of the systems developed during the
DARPA program

A blackboard-based system utilizing symbolic
problem solvers

Each problem solver was called a knowledge
group

A complex scheduler was used to decide
when each KG should be called

19. Hearsay-II

20. DARPA Results

The Hearsay-II system performed much
better than the two other similar competing
systems

However, only one system met the
performance goals of the project
 The Harpy system was also a CMU built system
 In many ways it was a predecessor to the
modern statistical systems

21. Modern Statistical ASR

22. Modern Statistical ASR

23. Acoustic Model

For each frame of data, we need some way
of describing the likelihood of it belonging to
any of our classes

Two methods are commonly used
 Multilayer perceptron (MLP) gives the likelihood
of a class given the data
 Gaussian Mixture Model (GMM) gives the
likelihood of the data given a class

24. Gaussian Distribution

25. Pronunciation Model

While the pronunciation model can be very
complex, it is typically just a dictionary

The dictionary contains the valid
pronunciations for each word

Examples:
 Cat: k ae t
 Dog: d ao g
 Fox: f aa x s

26. Language Model

Now we need some way of representing the
likelihood of any given word sequence

Many methods exist, but ngrams are the
most common

Ngrams models are trained by simply
counting the occurrences of words in a
training set

27. Ngrams

A unigram is the probability of any word in
isolation

A bigram is the probability of a given word
given the previous word

Higher order ngrams continue in a similar
fashion

A backoff probability is used for any unseen
data

28. How do we put it together?

We now have models to represent the three
parts of our equation

We need a framework to join these models
together

The standard framework used is the Hidden
Markov Model (HMM)

29. Markov Model

A state model using the markov property
 The markov property states that the future
depends only on the present state

Models the likelihood of transitions between
states in a model

Given the model, we can determine the
likelihood of any sequence of states

30. Hidden Markov Model

Similar to a markov model except the states
are hidden

We now have observations tied to the
individual states

We no longer know the exact state sequence
given the data

Allows for the modeling of an underlying
unobservable process

31. HMMs for ASR

First we build an HMM for each phone

Next we combine the phone models based
on the pronunciation model to create word
level models

Finally, the word level models are combined
based on the language model

We now have a giant network with potentially
thousands or even millions of states

32. Decoding

Decoding happens in the same way as the
previous example

For each time frame we need to maintain two
pieces of information
 The likelihood of being at any state
 The previous state for every state

33. State of the Art

What works well
 Constrained vocabulary systems
 Systems adapted to a given speaker
 Systems in anechoic environments without
background noise
 Systems expecting read speech

What doesn't work
 Large unconstrained vocabulary
 Noisy environments
 Conversational speech

34. Future Work

Better representations of audio based on
humans

Better representation of acoustic elements
based on articulatory phonology

Segmental models that do not rely on the
simple frame-based approach

35. Resources

Hidden Markov Model Toolkit (HTK)
 http://htk.eng.cam.ac.uk/

CHIME ( a freely available dataset)
 http://spandh.dcs.shef.ac.uk/projects/chime/PCC
/datasets.html

Machine Learning Lectures
 http://www.stanford.edu/class/cs229/
 http://www.youtube.com/watch?v=UzxYlbK2c7E