SpeechTEK 2009: Optimizing Speech Recognizer Rejection Thresholds

Optimizing speech recognizer rejection thresholds Dan Burnett Director of Speech Technologies, Voxeo August 24, 2009

Why this talk? • Sometimes we forget the basics, which are: • Recognizers are not perfect • They can be optimized in a straightforward manner • The simplest optimization is the rejection threshold

The Goal • End user goal: optimal experience • Our Goal: determine user experience for each possible rejection threshold, then choose optimum threshold • Must compare true classiﬁcation of an audio sample against the ASR engine’s classiﬁcation

True classiﬁcations • Assume human-level recognition • App should still distinguish (i.e. possibly behave differently) among the following cases: Case Possible behavior No speech in audio sample Mention that you didn’t hear (nospeech) anything and ask for repeat Speech, but not intelligible Ask for repeat (unintelligible) Intelligible speech, but not in app grammar Encourage in-grammar speech (out-of-grammar) Intelligible speech, and within app grammar (in-grammar) Respond to what person said

ASR Engine Classiﬁcations • Silence/nospeech (nospeech) • Reject (rejected) • Recognize (recognized)

Crossing these two . . . ASR nospeech rejected recognized Correct Improperly nospeech Incorrect classiﬁcation rejected Improperly Correct Assume unintelligible treated as silence behavior incorrect True out-of- Improperly Correct Incorrect grammar treated as silence behavior Improperly Improperly Either correct in-grammar treated as silence rejected or incorrect

Crossing these two . . . Misrecognitions ASR nospeech rejected recognized Correct Improperly nospeech Incorrect classiﬁcation rejected Improperly Correct Assume unintelligible treated as silence behavior incorrect True out-of- Improperly Correct Incorrect grammar treated as silence behavior Improperly Improperly Either correct in-grammar treated as silence rejected or incorrect

Crossing these two . . . “Misrejections” ASR nospeech rejected recognized Correct Improperly nospeech Incorrect classiﬁcation rejected Improperly Correct Assume unintelligible treated as silence behavior incorrect True out-of- Improperly Correct Incorrect grammar treated as silence behavior Improperly Improperly Either correct in-grammar treated as silence rejected or incorrect

Crossing these two . . . “Missilences” ASR nospeech rejected recognized Correct Improperly nospeech Incorrect classiﬁcation rejected Improperly Correct Assume unintelligible treated as silence behavior incorrect True out-of- Improperly Correct Incorrect grammar treated as silence behavior Improperly Improperly Either correct in-grammar treated as silence rejected or incorrect

Three types of errors • Missilences -- called silence, but wasn’t • Misrejections -- rejected inappropriately • Misrecognitions -- recognized inappropriately or incorrectly

Three types of errors • Missilences -- called silence, but wasn’t • Misrejections -- rejected inappropriately • Misrecognitions -- recognized inappropriately or incorrectly So how do we evaluate these?

Evaluating errors 1. Evaluation data set 2. Try every rejection threshold value 3. Plot errors as function of threshold 4. Select optimal value for your app

1. Evaluation data set(s) • Data selection • Must be representative (“every nth call”) • Ideally at least 100 recordings per grammar path for good conﬁdence in results • Transcription • Goal is to compare against recognition results, so no punctuation, coughs, etc. needed in transcription itself (but good to have in separate comments)

2. Try every rejection threshold value • Run recognizer in batch mode with rejection threshold of 0 (i.e., no rejection) Remember to collect conﬁdence scores! • Then, for each threshold from 0 to 100 • Calculate number of misrecognitions, misrejections, and missilences

3. Plot errors “Misrejections” Misrecognitions Equal Error Rate “Missilences” 0 Rejection Threshold 100

3. Plot errors Minimum Total Error Sum 0 Rejection Threshold 100

4. Select optimal value

4. Select optimal value • Equal-error-rate: not necessarily the optimum

4. Select optimal value • Equal-error-rate: not necessarily the optimum • Minimum of the sum: good starting point, great for comparing across engines (on same data set only!!)

4. Select optimal value • Equal-error-rate: not necessarily the optimum • Minimum of the sum: good starting point, great for comparing across engines (on same data set only!!) • Optimal: depends on your app; some errors may be more critical than others

4. Select optimal value • Equal-error-rate: not necessarily the optimum • Minimum of the sum: good starting point, great for comparing across engines (on same data set only!!) • Optimal: depends on your app; some errors may be more critical than others • Question: if missilences not affected by threshold, why did I include it?

Further optimizations • Move OOG into IG category if semantically correct (“You bet” -> “yes”) • Consider additional threshold for conﬁrmation • Optimize endpointer parameters (affects missilences and/or “too much speech”)

Optimizing speech recognizer rejection thresholds Dan Burnett Director of Speech Technologies, Voxeo August 24, 2009

SpeechTEK 2009: Optimizing Speech Recognizer Rejection Thresholds

0 comments

Notes on slide 1

Favorites, Groups & Events