Comparative ASR Evaluation - Voxeo - SpeechTEK NY 2010

Comparative ASR evaluation Dan Burnett Director of Speech Technologies, Voxeo SpeechTek New York August 2010

Goals for today • Learn about data selection • Learn all the steps of doing an eval by actually doing them • Leave with code that runs

Outline • Overview of comparative ASR evaluation • How to select an evaluation data set • Why transcription is important and how to do it properly • What and how to test • Analyzing the results

Comparative ASR Evaluation • How could you compare ASR accuracy? • Can you test against any dataset? • What settings should you use? The optimal ones, right?

Today’s approach • Choose representative evaluation data set • Determine human classification of each recording • For each ASR engine • Determine machine classification of each recording at “optimal” setting • Compare to human classification to determine accuracy • Intelligently compare results for the two engines

Evaluation data set • Ideally at least 100 recordings per grammar path for good conﬁdence in results (up to 10000 minimum for large grammars) • Must be representative • Best to take from actual calls (why?) • Do you need all the calls? Consider • Time of day, day of week, holidays • Regional differences • Simplest is to use every nth call

Lab data set • Stored in all-data • In “original” format as recorded • Only post-endpointed data for today • 1607 recordings of answers to yes/no question • Likely to contain yes/no, but not guaranteed

Transcription • Why is it needed? Why not automatic? • Stages • Classiﬁcation • Transcription

Audio classiﬁcation • Motivation: • Applications may distinguish (i.e. possibly behave differently) among the following cases: Case Possible behavior No speech in audio sample Mention that you didn’t hear (nospeech) anything and ask for repeat Speech, but not intelligible Ask for repeat (unintelligible) Intelligible speech, but not in app grammar Encourage in-grammar speech (out-of-grammar speech) Intelligible speech, and within app grammar (in-grammar speech) Respond to what person said

Transcribing speech • Words only, all lower case • No digits • Only punctuation allowed is apostrophe

Lab 1 • Copy yn_files.csv to yn_finaltrans.csv and edit • For each file, append category of nospeech, unintelligible, or speech • Example: all-data/.../utt01.wav,unintelligible • Append transcription if speech • Example: all-data/.../utt01.wav,speech,yes • Transcription instructions in transcription.html • How might you validate transcriptions?

What and how to test • Understanding what to test/measure • Preparing the data • Building a test harness • Running the test

What to test/measure • To measure accuracy, we need • For each data ﬁle • the human categorization and transcription, and • the recognizer’s categorization, recognized string, and conﬁdence score

Preparing the data • Recognizer needs a grammar (typically from your application) • This grammar can be used to classify transcribed speech as In-grammar/Out-of-grammar

Lab 2 • Fix GRXML yes/no grammar in “a” directory called yesno.grxml • Copy yn_ﬁnaltrans.csv to yn_igog.csv • Edit yn_igog.csv and change every “yes” or “no” line to have a category of “in_grammar” (should be 756 yes, 159 no, for total of 915)

Building a test harness • Why build a test harness? What about vendor batch reco tools? • End-to-end vs. recognizer-only testing • Harness should be • generic • customizable to different ASR engines

Lab 3 • Complete the test harness harness.php (see harness_outline.txt) • The harness must use the “a/scripts” scripts • A list of “missing commands” is in harness_components.txt • Please review (examine) these scripts • FYI, ASR engine is a/a.php -- treat as black box

Lab 4 • Now run the test harness: • php harness.php a/scripts <data ﬁle> <rundir> • Output will be in <rundir>/results.csv • Compare your output to “def_results.csv”

Analyzing results • What are the possible outcomes and errors? • How do we evaluate errors?

Possible ASR Engine Classiﬁcations • Silence/nospeech (nospeech) • Reject (rejected) • Recognize (recognized) • What about DTMF?

Possible outcomes ASR nospeech rejected recognized Correct Improperly nospeech Incorrect classiﬁcation rejected Improperly Correct Assume unintelligible treated as silence behavior incorrect True out-of- Improperly Correct Incorrect grammar treated as silence behavior Improperly Improperly Either correct in-grammar treated as silence rejected or incorrect

Possible outcomes Misrecognitions ASR nospeech rejected recognized Correct Improperly nospeech Incorrect classiﬁcation rejected Improperly Correct Assume unintelligible treated as silence behavior incorrect True out-of- Improperly Correct Incorrect grammar treated as silence behavior Improperly Improperly Either correct in-grammar treated as silence rejected or incorrect

Possible outcomes “Misrejections” ASR nospeech rejected recognized Correct Improperly nospeech Incorrect classiﬁcation rejected Improperly Correct Assume unintelligible treated as silence behavior incorrect True out-of- Improperly Correct Incorrect grammar treated as silence behavior Improperly Improperly Either correct in-grammar treated as silence rejected or incorrect

Possible outcomes “Missilences” ASR nospeech rejected recognized Correct Improperly nospeech Incorrect classiﬁcation rejected Improperly Correct Assume unintelligible treated as silence behavior incorrect True out-of- Improperly Correct Incorrect grammar treated as silence behavior Improperly Improperly Either correct in-grammar treated as silence rejected or incorrect

Three types of errors • Missilences -- called silence, but wasn’t • Misrejections -- rejected inappropriately • Misrecognitions -- recognized inappropriately or incorrectly

Three types of errors • Missilences -- called silence, but wasn’t • Misrejections -- rejected inappropriately • Misrecognitions -- recognized inappropriately or incorrectly So how do we evaluate these?

Evaluating errors • Run ASR Engine on data set • Try every rejection threshold value • Plot errors as function of threshold • Find optimal value

Try every rejection threshold value • Ran data ﬁles through test harness with rejection threshold of 0 (i.e., no rejection), but recorded conﬁdence score • Now, for each possible rejection threshold from 0 to 100 • Calculate number of misrecognitions, misrejections, and missilences

Semantic equivalence • We call “yes” in-grammar, but what about “yes yes yes”? • Application only cares about whether it does the right thing, so • Our ﬁnal results need to be semantic results

Lab 5 • Look at synonyms.txt file • Analyze at single threshold and look at the result • php analyze_csv.php <csv file> 50 synonyms.txt • Note the difference between raw and semantic results • Now evaluate at all thresholds and look at the (semantic) results • php analyze_all_thresholds.php <csv file> <synonyms file>

1000 ASR Engine A errors “Misrejections” Misrecognitions 500 “Missilences” 0 Rejection Threshold 100

1000 ASR Engine A errors 500 Minimum Total Error Sum 0 Rejection Threshold 100

Lab 6 • You now have engine B in “b” directory • Change harness and component scripts as necessary to run the same test • You need to know that • The API for engine B is different. Run “php b/b.php” to find out what it is. It takes ABNF grammars instead of XML. • Engine B stores its output in a different file. • Possible outputs from engine B are • <audiofilename>: [NOSPEECH, REJECTION, SPOKETOOSOON, MAXSPEECHTIMEOUT] • <audiofilename>: ERROR processing file

1000 ASR Engine B errors “Misrejections” Misrecognitions 500 “Missilences” 0 Rejection Threshold 100

1000 ASR Engine B errors 500 Minimum Total Error Sum 0 Rejection Threshold 100

Comparing ASR accuracy • Plot and compare • Remember to compare optimal error rates of each (representing tuned accuracy)

1000 Total errors: A vs B 500 ASR Engine A ASR Engine B 0 Rejection Threshold 100

Comparison conclusions • Optimal error rates are very similar on this data set • Engine A is much more sensitive to rejection threshold changes

1000 Natural Numbers Note that optimal thresholds are different! 500 ASR Engine A ASR Engine B 0 Rejection Threshold 100

Today we . . . • Learned all the steps of doing an eval by actually doing them • How to collect data • Transcribing data • Running a test • Analyzing results • Finished with code that runs (and some homework . . .)

Comparative ASR Evaluation - Voxeo - SpeechTEK NY 2010

by Voxeo Corp

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 135

Statistics

Comparative ASR Evaluation - Voxeo - SpeechTEK NY 2010 Presentation Transcript

http://blogs.voxeo.com	134
http://translate.googleusercontent.com	1