Dual-Mode Tongue Drive System     Using Speech and Tongue Motion to Improve     Computer Access for People with Disabiliti...
Some Events and Their Consequences                    © 2012 Maysam Ghovanloo   2www.GTBionics.org
Spinal Cord Injuries (SCI)                    © 2012 Maysam Ghovanloo      3www.GTBionics.org
54M         Americans (~20%)                                                     11,000                                   ...
How to Improve their Quality of Life?                    © 2012 Maysam Ghovanloo   5www.GTBionics.org
Why Using Tongue?   •    Along with mouth occupies the amount of        sensory and motor cortex that rivals fingers      ...
Dual-Mode Tongue Drive System (dTDS)     3-axial magnetic sensors + microphones on headset  Wireless data        Process...
Magnetic Tracer Attachment  Temporary attachment:  By tissue adhesives  to test-drive the TDS   Semi-permanent   attachmen...
Command Definition and Training   Current version: Six tongue commands   plus the tongue resting position as neutral      ...
TDS Clinical Trials         Computer access and wheelchair navigation using TDS                    © 2012 Maysam Ghovanloo...
dTDS Wireless Headset                    © 2012 Maysam Ghovanloo   11www.GTBionics.org
dTDS Specifications             Specification                     Value                              Magnetic Sensors     ...
dTDS Performance Evaluation                    © 2012 Maysam Ghovanloo   13www.GTBionics.org
Experimental Design   • 14 able-bodied subjects (age: 21 – 30 years old, 9 males and 5 females)      from the Georgia Tech...
Experimental Methods                    R   L                                   R       R                        R        ...
Experimental Results   Total   completion   time:   Significant   effect of device   on all three   measures   Recognition...
Qualitative Results   Recognition   accuracy:     •   Novice subjects only (7)     •   A higher number represents more pos...
Intraoral Tongue Drive System (iTDS)                    © 2012 Maysam Ghovanloo   Park and Ghovanloo, ISSCC 12   18www.GTB...
iTDS Implementation                                                            S3                     S1                  ...
Conclusions   • Tongue Drive System (TDS) is a wireless, wearable, and      minimally invasive brain-tongue-computer inter...
GT- Bionics Lab Members   2012 Summer picnic                    © 2012 Maysam Ghovanloo   21www.GTBionics.org
Acknowledgements   • Funding provided by:         –   National Science Foundation         –   Christopher and Dana Reeve F...
Upcoming SlideShare
Loading in …5
×

7.1 - Dual-Mode Tongue Drive System: Using Speech and Tongue Motion to Improve Computer Access for People with Disabilities

0 views
1,709 views

Published on

Thursday, October 25, 2012
Technical Session #7

Maysam Ghovanloo, PhD – ON Semiconductor Junior Professor, Georgia Tech; Director, GT Bionics Lab; Associate Editor, IEEE Trans. Circuits & Systems II

0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
0
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
129
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

7.1 - Dual-Mode Tongue Drive System: Using Speech and Tongue Motion to Improve Computer Access for People with Disabilities

  1. 1. Dual-Mode Tongue Drive System Using Speech and Tongue Motion to Improve Computer Access for People with Disabilities Xueliang Huo Hangue Park Maysam Ghovanloo, Ph.D. GT-Bionics Lab School of Electrical and Computer Engineering © 2012 Maysam Ghovanloo 1www.GTBionics.org
  2. 2. Some Events and Their Consequences © 2012 Maysam Ghovanloo 2www.GTBionics.org
  3. 3. Spinal Cord Injuries (SCI) © 2012 Maysam Ghovanloo 3www.GTBionics.org
  4. 4. 54M Americans (~20%) 11,000 cases of severe SCI add every living with disabilities year to a total population of 55% 250,000 of the SCI victims are They need lifelong 16~30 special care Financial, emotional, and productivity years old  cost to the families and the society © 2012 Maysam Ghovanloo 4www.GTBionics.org
  5. 5. How to Improve their Quality of Life? © 2012 Maysam Ghovanloo 5www.GTBionics.org
  6. 6. Why Using Tongue? • Along with mouth occupies the amount of sensory and motor cortex that rivals fingers and hand: sophisticated motor control capability evident in speech and ingestion • Fast movement with many degrees of freedom (DoF). Very flexible • Connected to brain by a cranial nerve: escapes even high level spinal cord injuries • Noninvasive access to tongue is possible. Motor Homunculus • Not afflicted by repetitive motion disorders • Does not fatigue easily. Very low rate of perceived exertion • Cosmetic advantage and privacy. It is all inside the mouth • Not influenced by the position of the rest of the body Hypoglossal • Unlike BCIs does not need concentration. Nerve © 2012 Maysam Ghovanloo Ghovanloo, IEEE EMBC 2007 6www.GTBionics.org
  7. 7. Dual-Mode Tongue Drive System (dTDS) 3-axial magnetic sensors + microphones on headset  Wireless data  Processing in smartphone  Wireless commands  Target devices © 2012 Maysam Ghovanloo US Patents 8044766, 8242880, Other patents pending 7www.GTBionics.org
  8. 8. Magnetic Tracer Attachment Temporary attachment: By tissue adhesives to test-drive the TDS Semi-permanent attachment: By magnetic tongue piercing to use the TDS on a regular basis © 2012 Maysam Ghovanloo 8www.GTBionics.org
  9. 9. Command Definition and Training Current version: Six tongue commands plus the tongue resting position as neutral © 2012 Maysam Ghovanloo 9www.GTBionics.org
  10. 10. TDS Clinical Trials Computer access and wheelchair navigation using TDS © 2012 Maysam Ghovanloo 10www.GTBionics.org
  11. 11. dTDS Wireless Headset © 2012 Maysam Ghovanloo 11www.GTBionics.org
  12. 12. dTDS Specifications Specification Value Magnetic Sensors Type Honeywell HMC1043 AMR sensor Dimensions 3.0 3.0 1.5 mm3 Sensitivity / range 1 mV/V/Gauss / ± 600 T Microphone Type SiSonic SPM0408HE5H Dimensions 4.7 3.8 1.1 mm3 Sensitivity / SNR -22 dB / 59 dB Control Unit Microcontroller TI – CC2510 SoC Wireless frequency / data rate 2.42 GHz / 500 kbps Sampling rate 50 sample/s/sensor Number of sensors /duty 4 / 8% cycle Audio codec / interface TLV320AIC3204 / I2S Audio sampling rate / 8 ksps / 16 bits / μ-Law resolution / compression 3 V / 35 mA (audio on) Operating voltage / current 6 mA (audio off) PCB Dimensions 36 16 mm2 Wireless data transfer to Headset Material Object VeroGray resin a PC running Dragon Total weight 90 g (including battery) Naturally Speaking © 2012 Maysam Ghovanloo 12www.GTBionics.org
  13. 13. dTDS Performance Evaluation © 2012 Maysam Ghovanloo 13www.GTBionics.org
  14. 14. Experimental Design • 14 able-bodied subjects (age: 21 – 30 years old, 9 males and 5 females) from the Georgia Tech graduate and undergraduate student population • 7 subjects had prior experience with TDS, and 7 were naive • 7 subjects were native English speakers and 7 were non-native • A within-subject model with each subject repeating the same tasks using three devices: TDS, Dragon, and dTDS • Two sessions, ~3h each, maximum of one week apart: 1) Instructional session: Subjects learned to use TDS, Dragon, and dTDS 2) Experimental session: Quantitative and qualitative measurements • Two tasks: 1) Text transcription: subjects transcribed two short paragraphs from a hard copy onto a word document to test acoustic input 2) Maze navigation: subjects navigated the mouse cursor through an on- screen maze, clicked on designated areas, and typed words/numbers • Questionnaire: At the end subjects rated their experience with each device © 2012 Maysam Ghovanloo 14www.GTBionics.org
  15. 15. Experimental Methods R L R R R L R L L R L • Subjects were asked to go through the maze as quickly and accurately as possible, issue right/left click on yellow/green boxes, followed by typing • A minimum of 12 cursor movements, 11 clicks (excluding those for typing), and typing 36 characters (on average) in each round of the trial. • Subjects were also asked to perform both tasks with a combination of standard mouse and keyboard to generate a reference point. • Performance Measures: Recognition accuracy, total completion time, typing time, navigation time, typing error, and navigation error (deviation) © 2012 Maysam Ghovanloo 15www.GTBionics.org
  16. 16. Experimental Results Total completion time: Significant effect of device on all three measures Recognition accuracy: Microphone type Native Speakers Non-native Speakers Significant (7)* (7)* effect of Commercial 94.0% ± 1.9% 78.6% ± 5.9% accent dTDS 91.5% ± 2.3% 75.7% ± 5.1% © 2012 Maysam Ghovanloo 16www.GTBionics.org
  17. 17. Qualitative Results Recognition accuracy: • Novice subjects only (7) • A higher number represents more positive perception © 2012 Maysam Ghovanloo 17www.GTBionics.org
  18. 18. Intraoral Tongue Drive System (iTDS) © 2012 Maysam Ghovanloo Park and Ghovanloo, ISSCC 12 18www.GTBionics.org
  19. 19. iTDS Implementation S3 S1 MSP 25mm 430 Front 20mm iTDS chip S2 S4 35mm Magnetoresistive Sensors 49mm[ Top view ] [ Bottom view ] 42mm Charging coil Tx Li-Ion @13.56MHz Antenna rechargeable 432MHz Battery © 2012 Maysam Ghovanloo Park and Ghovanloo, ISSCC 12 19www.GTBionics.org
  20. 20. Conclusions • Tongue Drive System (TDS) is a wireless, wearable, and minimally invasive brain-tongue-computer interface (BTCI) that enables individuals with severe disabilities to access and control with their voluntary tongue motion. • The latest dual-mode TDS (dTDS) prototype appears as a wireless headphone with both tongue motion and speech recognition (SR) capabilities for navigation and typing, respectively. • The subjects’ performance with the dTDS was significantly better than unimodal TDS and SR in a task that involved both navigation and typing (e.g. web surfing). • Subjects preferred dTDS over either TDS or SR in terms of speed, ease of use and overall satisfaction. • We are now in the process of developing a multimodal Tongue Drive System (mTDS) as well as an intraoral Tongue Drive System (iTDS) that is completely inconspicuous. © 2012 Maysam Ghovanloo 20www.GTBionics.org
  21. 21. GT- Bionics Lab Members 2012 Summer picnic © 2012 Maysam Ghovanloo 21www.GTBionics.org
  22. 22. Acknowledgements • Funding provided by: – National Science Foundation – Christopher and Dana Reeve Foundation – National Institutes of Health – Army Research Office (ARO) – ON Semiconductor • Collaborators: – Dr. Michael Jones, Shepherd Center, Atlanta, GA – Dr. Ann Laumann, Northwestern University, Chicago, IL – Dr. Joseph Manns, Emory University, Atlanta, GA – Dr. Elliot Roth, Rehab. Institute of Chicago, Chicago, IL – Dr. Elizabeth Bailey, University of Arizona, Tucson, AZ – Dr. Karim Oweiss, Michigan State University, Lansing, MI – Dr. Kimberly Wilson, Emory Hospital, Atlanta, GA – Dr. Stephen Sprigle, Georgia Tech, Atlanta, GA © 2012 Maysam Ghovanloo 22www.GTBionics.org

×