The Art and Science of
Grammar Coverage Analysis

Dominique Boucher
Nu Echo Inc.
dominique.boucher@nuecho.com
            ...
The grammar development process
Enter / get initial           Write initial
set of sentences           grammar to cover
 i...
The importance of coverage analysis


  Design and       • Ensure grammars conform to their specification
 development    ...
Grammar coverage challenges

   Provide exhaustive coverage of all sentence patterns…
   … with the smallest possible se...
Two complementary techniques




                       #2
      #1
                    Sentence
   Sentence
             ...
Technique #1
Sentence generation

   Tools usually provided by the ASR engine SDK
    – Operate on the source or compiled...
Technical difficulties

   Some grammars generate an infinite number of
    sentences
    – Exhaustive generation not pos...
Sentence generation best practices

   Avoid generating redundant sentences
    – Powerful sentence generation tools make...
Sentence generation revisited

   Individual rule
    configuration
   More effective strategies
    – Tags coverage
   ...
Sentence generation tool




       Interactive Demo
Technique #2
Exploring sentence patterns

   Interactive expansion of
    grammar rules
   Derive sentence patterns
    ...
Sentence explorer tool




       Interactive Demo
Three typical problems…
… and how to fix them!




       Interactive Demo
Upcoming SlideShare
Loading in...5
×

The Art and Science of Grammar Coverage Analysis

1,239
-1

Published on

This talk, given at SpeechTEK 2009, presents two effective techniques for testing the coverage of speech recognition grammars.

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,239
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
9
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

The Art and Science of Grammar Coverage Analysis

  1. 1. The Art and Science of Grammar Coverage Analysis Dominique Boucher Nu Echo Inc. dominique.boucher@nuecho.com SpeechTEK 2009 New York, USA
  2. 2. The grammar development process Enter / get initial Write initial set of sentences grammar to cover in coverage set utterances Run coverage tests Generate no Problems? yes Fix grammar sentences Objective: Enrich coverage To obtain: set (a) A complete coverage set (b) A grammar that covers the coverage set and produces the correct semantic result
  3. 3. The importance of coverage analysis Design and • Ensure grammars conform to their specification development • Semantic tags testing Maintenance • Grammars evolve over the life of an application • Provides an effective tool for testing that a grammar and optimization isn’t accidentally broken by a grammar change Conversion • Ensure proper conversion between grammar formats projects
  4. 4. Grammar coverage challenges  Provide exhaustive coverage of all sentence patterns…  … with the smallest possible set of sentences. Otherwise: – Analysis of generated sentences will be time-consuming; and – Errors will go undetected
  5. 5. Two complementary techniques #2 #1 Sentence Sentence patterns Generation Exploration
  6. 6. Technique #1 Sentence generation  Tools usually provided by the ASR engine SDK – Operate on the source or compiled grammar  Commonly used generation strategies – Exhaustive generation – Generation of a fixed number of random sentences
  7. 7. Technical difficulties  Some grammars generate an infinite number of sentences – Exhaustive generation not possible  Semantic tags not all tested – Errors may remain undetected until application run time  All interesting cases may not be covered by the generated sentences  Uninteresting patterns are generated over and over
  8. 8. Sentence generation best practices  Avoid generating redundant sentences – Powerful sentence generation tools makes this possible – Too many sentences increases the risk of errors going undetected  Carefully examine generated sentences – If a sentence doesn’t look right, it probably shouldn’t be in there (although, to be sure, look at the parse tree)  Make sure the coverage test is as complete as possible – Should include all semantic tags and all sentence patterns – Full coverage is best (whenever possible)
  9. 9. Sentence generation revisited  Individual rule configuration  More effective strategies – Tags coverage – All grammar paths – Pick from @examples – Use fixed sentence  Generation can be started from any set of sentence patterns
  10. 10. Sentence generation tool Interactive Demo
  11. 11. Technique #2 Exploring sentence patterns  Interactive expansion of grammar rules  Derive sentence patterns – Useful to generate sentences for a specific pattern  Derive complete sentences – Can be debugged, etc.  Ideal to understand the structure of a grammar
  12. 12. Sentence explorer tool Interactive Demo
  13. 13. Three typical problems… … and how to fix them! Interactive Demo
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×