Using Speech to Interact with In-Car Devices in the Project54 System Andrew Kun University of New Hampshire

Outline Introduction Speech user interface testing Driving simulator studies Conclusion

Introduction

Speech user interface testing

Driving simulator studies

Conclusion

What is the problem?

The system in the car

Speech user interface Command and control interface Microsoft speech recognition (SR) engine Microsoft text-to-speech (TTS) engine Directional microphone Push-to-talk (PTT) button Grammars

Command and control interface

Microsoft speech recognition (SR) engine

Microsoft text-to-speech (TTS) engine

Directional microphone

Push-to-talk (PTT) button

Grammars

Outline Introduction Speech user interface (SUI) testing Driving simulator studies Conclusion

Introduction

Speech user interface (SUI) testing

Driving simulator studies

Conclusion

Testing Officer volunteers: 27 Corpus: just under 50,000 utterances Utterances: while PTT is pressed

Officer volunteers: 27

Corpus: just under 50,000 utterances

Utterances: while PTT is pressed

Speech user interface (SUI) performance Recognized: 85 % Unrecognized: 4 % Misrecognized: 11 %

Recognized: 85 %

Unrecognized: 4 %

Misrecognized: 11 %

Reasons for imperfect recognition SR engine error: 37 % User error: 63 %

SR engine error: 37 %

User error: 63 %

SR engine error example Utterance: 0 3 2 1 8 5 -> Recognized 0 3 2 1 0 5 7 1 -> Recognized OK

Utterance: 0 3 2 1 8 5

-> Recognized 0 3 2 1 0 5 7 1

-> Recognized OK

User errors Utterance not in any grammar: 54 % Utterance in another grammar: 34 % PTT (“Patrol screen” ): 12 %

Utterance not in any grammar: 54 %

Utterance in another grammar: 34 %

PTT (“Patrol screen” ): 12 %

Outline Introduction Speech user interface testing Driving simulator studies Conclusion

Introduction

Speech user interface testing

Driving simulator studies

Conclusion

Driving simulator

Driving simulator studies Multi-threaded dialogues SUI and driving

Multi-threaded dialogues

SUI and driving

Multi-threaded dialogues National Science Foundation grant Goal: interact with multiple real-time devices using speech Manual-visual task! Human-human to human-computer interaction

National Science Foundation grant

Goal: interact with multiple real-time devices using speech

Manual-visual task!

Human-human to human-computer interaction

SUI and driving – police radio

SUI and driving Two experiments (8 subjects) : Baseline + radio Baseline + SUI Record: Lane position Velocity Steering wheel angle

Two experiments (8 subjects) :

Baseline + radio

Baseline + SUI

Record:

Lane position

Velocity

Steering wheel angle

Lane position – radio

Lane position – SUI

Outline Introduction Speech user interface testing Driving simulator studies Conclusion

Introduction

Speech user interface testing

Driving simulator studies

Conclusion

Status February 2007: ≈900 cars on the road in USA ( New Hampshire , Massachusetts, California, Maryland) Industry participation

February 2007: ≈900 cars on the road in USA ( New Hampshire , Massachusetts, California, Maryland)

Industry participation

Research and development directions SUI and driving SUI performance improvements (SR training, …) SUI performance relation to driving task difficulty, recognizer accuracy Intelligent interaction (multi-threaded dialogues, natural language processing, …) Non-speech work (handhelds, telematics, …) Standards

SUI and driving

SUI performance improvements (SR training, …)

SUI performance relation to driving task difficulty, recognizer accuracy

Intelligent interaction (multi-threaded dialogues, natural language processing, …)

Non-speech work (handhelds, telematics, …)

Standards

Acknowledgement US Department of Justice National Science Foundation

US Department of Justice

National Science Foundation

www.project54.unh.edu