This document summarizes research on a system called Myna that allows users to control the Scratch programming environment using voice commands. Myna has a speech recognizer and dynamic grammars that map voice commands to actions in Scratch. It uses transparent frames to navigate Scratch without requiring precise mouse control. The goals are to make programming accessible for people with disabilities and improve logic skills. Future work includes more testing, portability to other platforms, and configurable screen resolutions.

1. Programming by Voice with Scratch:Teaching the Cat to Obey a Bird Ramaraju Rudraraju, Srinivasa Datla, Avishek Banerjee, Mandar Sudame Univ. of Alabama at Birmingham, Department of Computer and Information Sciences Jeff Gray University of Alabama, Department of Computer Science Gary Edwards United Cerebral Palsy of Greater Birmingham This research sponsored by a Google Research Award

2. Overview of Presentation Scratch Script Video Motivation Myna Features Architecture & Working of Myna Dynamic Grammars

3. Scratch Script Get your popcorn, its Movie Time!!

4. The WIMP Metaphor Motivation Window Menu Pointing Device Icons

5. Motivation The WIMP assumes dexterity The MYNA Solution for Scratch

6. Drag and Drop Navigation Continuous Navigation Navigation using Transparent Frames Features Types of Navigation

7. Features (An extension of Transparent Frames) Drop After Drop In Drop Before 7 Macro Commands

8. Myna Architecture Component Mappings Cloud Garden Model Sprite State Storage MYNA Speech Recognizer Component Hierarchy Grammar View Controller Command Executor Java Robot Scratch UI Transparent Frames

9. Under the Hood of Myna Move Right Controller Model Grammar Grammar Mappings Mappings Command Executor Command Executor Speech Recognizer Speech Recognizer Component Hierarchy Component Hierarchy Java Robot Java Robot

10. Dynamic Grammar Pause Resume File Default Grammar Navigation Grammar Speech Pause Grammar Default Grammar Navigation Grammar File Grammar File Grammar Control Grammar Looks Grammar

11. Limitations Scratch needs to be in maximized mode Not compatible with different screen resolutions Myna is not scroll-enabled Large vocabulary and still growing! Future Work Integration with UCP - Deep human subjects testing with United Cerebral Palsy of Greater Birmingham Portability on other platforms Configurable screen resolutions Looking Ahead

12. Myna is our preliminary attempt towards voice-driven programming applied to Initial Learning Environments (ILEs) to overcome the physical challenges of using the traditional WIMP metaphor Our long-term goal is to assist children with disabilities by giving them a learning tool that enables them to encounter the joys and excitement of programming, while also improving their logic skills Summary

13. Thank you Questions and comments? https://sites.google.com/site/projectmyna2010/ Additional questions? teammyna@googlegroups.com