Eyes-Free User Interaction

1,272 views

Published on

  • Be the first to comment

  • Be the first to like this

Eyes-Free User Interaction

  1. 1. Eyes-Free User Interaction T. V. Raman Charles L. Chen Google Google http://emacspeak.sf.net/raman www.clcworld.net May 21, 2009
  2. 2. Overview Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion Eyes-Free Interaction Google-IO May 2009 – 2 / 39
  3. 3. Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion Android Programming
  4. 4. Android: Engineer’s Dream! Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion s Smart phones — computer in your pocket. s Open platform — liberates innovation. s New I/O — minimize user interaction. Bend technology to your will! Eyes-Free Interaction Google-IO May 2009 – 4 / 39
  5. 5. Innovative User Interfaces Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion Create innovative end-user solutions that: s Fit into the user’s mode of working, s Avoid one-size fits all solutions, s Design context-aware solutions, s Tailor solutions to user’s needs. Eyes-Free Interaction Google-IO May 2009 – 5 / 39
  6. 6. Eyes-Free Interaction Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion Not just for blind users! s Eyes-busy environments. s One-handed use. s Minimize explicit user interaction. Rethink all assumptions about the user. Eyes-Free Interaction Google-IO May 2009 – 6 / 39
  7. 7. Devices That Sense Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion Mobile phones make UI design exciting! s Can capture user intent — keyboard, . . .. s Can see — camera. s Can hear and speak — speech I/O. s Can sense and stimulate touch — touch screen and vibration. s Can sense motion — accelerometer, compass. s Can sense position — GPS, cell towers. . . s .. Our Android has many eyes and ears! Eyes-Free Interaction Google-IO May 2009 – 7 / 39
  8. 8. Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion Touch Screen Interaction, Eyes-Free!
  9. 9. Using A Touch Screen Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion These are atomic tasks: s Identifying the item to touch. s Touching item to activate it. s Eyes-free use requires help with the former. s Activation should produce appropriate feedback. Eyes-Free Interaction Google-IO May 2009 – 9 / 39
  10. 10. On-Screen Touchpads Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion Eyes-Free use of an on-screen touchpad: s Need to see the buttons to know where to touch. s Because: buttons use absolute positioning. s Conclusion: (wrong) — you need to see to use a touch screen! s Solution: Relax constraint of absolute positioning. What if we used relative positioning instead? Eyes-Free Interaction Google-IO May 2009 – 10 / 39
  11. 11. Phone Keypad Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion Eyes-free phone dialer: s Keypad centered where you touch — relative positioning. s Users already know layout of a phone keypad. s Can stroke from the center for a desired digit. Enables eyes-free one-handed dialing! Eyes-Free Interaction Google-IO May 2009 – 11 / 39
  12. 12. Dialer Feedback Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion Enables one-handed, eyes-free use: s Buttons vibrate when you move over them. s Augmented by auditory feedback. s Pressing buttons produces spoken and tactile feedback. s Shake to erase input. Synchronized auditory and haptic feedback is key. Eyes-Free Interaction Google-IO May 2009 – 12 / 39
  13. 13. Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion Text Input Via Touch Screen
  14. 14. Contact Manager Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion No one dials phone numbers any more! s Browsing contacts using touch screen. s Finding contacts. s Managing contacts. Eyes-Free Interaction Google-IO May 2009 – 14 / 39
  15. 15. Entering Text Input Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion Extending stroke dialer for textual input: s Arrange letters in 4 concentric circles. s Strokes access distinct circles. s Choose desired letter by tracing chosen circle. Any letter is at most 3 steps away! Eyes-Free Interaction Google-IO May 2009 – 15 / 39
  16. 16. Keypad Layout Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion Gesture: Pick circle, and trace to desired character. A I Q Letter Gesture A Stroke up to the left ⊙ Y B Stroke A, trace right U M E E Stroke down to the right Eyes-Free Interaction Google-IO May 2009 – 16 / 39
  17. 17. Managing Contacts Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion Contacts live in the cloud. s Contacts managed over the network. s Updated automatically on the device. s No more editing contacts with two fingers! Eyes-Free Interaction Google-IO May 2009 – 17 / 39
  18. 18. Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion Text To Speech On Android
  19. 19. TTS Library Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion s Create speech-enabled apps. s Developer compiles against the TTS library stub. s Text To Speech is an Android service. s Class TTS manages communication with the service. s Enables multiple apps to use TTS without code duplication. s TTS library can be updated without a need to recompile clients. Eyes-Free Interaction Google-IO May 2009 – 19 / 39
  20. 20. TTS Method Summary Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion s speak(String text, int queueMode, String[] params) s stop() s isSpeaking() s Also has methods for synthesizing to a file, specifying the language, etc. Eyes-Free Interaction Google-IO May 2009 – 20 / 39
  21. 21. TTS Object Properties Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion s Can automatically prompt the user to install the TTS Service. s When absent, fail silently (methods become no-ops) See: http://eyes-free.googlecode.com/ Eyes-Free Interaction Google-IO May 2009 – 21 / 39
  22. 22. Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion Gesture Library For Stroke Input
  23. 23. Gesture Overlay Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion s Overlay watches for touch events s Returns identified gesture s Exposes same UI as the stroke dialer s Implemented as a custom transparent View with callbacks Eyes-Free Interaction Google-IO May 2009 – 23 / 39
  24. 24. Gesture Overlay Implementation Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion s Implement Interface GestureListener , s Create a new TouchGestureControlOverlay with it, s Set a FrameLayout as the base View, s Add the View with the content as a child to the FrameLayout, s Add the TouchGestureControlOverlay as a child Can easily enable/disable the TouchGestureControlOverlay Eyes-Free Interaction Google-IO May 2009 – 24 / 39
  25. 25. Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion A Simple Music File Browser
  26. 26. Music File Browser Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion Barebones version: s Uses a simple ListView s Based on a FileBrowser tutorial on Anddev.org s Browse directories and mp3 files on SDCard. s Play music by clicking on the file. s Directory path is the first entry. s Click this to cycle through files in current level. s Tactile feedback when scrolling through list items. Eyes-Free Interaction Google-IO May 2009 – 26 / 39
  27. 27. Adding Spoken Output Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion s Create object TTS in method onCreate. s Result: Runs method TTS.onInit as part of setup. s Can add application-specific code to method TTS.onInit. s Result: Application comes up speaking. Eyes-Free Interaction Google-IO May 2009 – 27 / 39
  28. 28. Music Player Talks Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion private InitListener i = new InitListener() { public void onInit(int version) { tts.speak("Music File Browser started.", 0, null); } }; Eyes-Free Interaction Google-IO May 2009 – 28 / 39
  29. 29. Speaking List Items Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion s Add TTS call to speak list items. s Can use event handler that produces tactile feedback. Eyes-Free Interaction Google-IO May 2009 – 29 / 39
  30. 30. Speaking List Items Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion Augment event handler with a TTS call. public boolean onTrackballEvent(MotionEvent event) { String filename = directoryEntries.get(selectedId); tts.speak(filename.substring(1), 0, null); } Self-voicing functionality added with just 13 additional lines of code! Eyes-Free Interaction Google-IO May 2009 – 30 / 39
  31. 31. Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion Adding Gesture Input To Music Browser
  32. 32. Advantages Of An Overlay Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion s GestureOverlay is a transparent overlay. s Can be layered on an existing View. s Does not interfere with visual appearance. Eyes-Free Interaction Google-IO May 2009 – 32 / 39
  33. 33. Gestures For Music Browser Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion s Add GestureListener to handle user input. s Gesture Tap: play/pause. s Gesture Right: Browse to next track. Add application-specific code in method onGestureFinish. Eyes-Free Interaction Google-IO May 2009 – 33 / 39
  34. 34. Steps In Adding Gesture Input Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion private GestureListener g = new GestureListener() { public void onGestureChange(Gesture arg0) {} public void onGestureFinish(Gesture arg0) { if (arg0 == Gesture.CENTER) { // play/pause music } else if (arg0 == Gesture.RIGHT) { playNext(); } } public void onGestureStart(Gesture arg0) {} }; Eyes-Free Interaction Google-IO May 2009 – 34 / 39
  35. 35. Adding Overlay To Music Player Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion myFrame = new FrameLayout(this); myFrame.addView(myList); myG = new TouchGestureControlOverlay(this, gestureListener); overlayActive = false; setContentView(myFrame); Eyes-Free Interaction Google-IO May 2009 – 35 / 39
  36. 36. Toggling Gesture Support Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion if (overlayActive) { myFrame.removeView(myG); overlayActive = false; tts.speak("Gestures disabled.", 0, null); } else { myFrame.addView(myG); overlayActive = true; tts.speak("Gestures activated.", 0, null); } Gesture support enabled by adding about 40 lines of code. Eyes-Free Interaction Google-IO May 2009 – 36 / 39
  37. 37. Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion Conclusion
  38. 38. Summary Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion s Programmable smart phones are an engineer’s dream! s I/O peripherals open up new UI avenues. s Provide desired information with minimal user interaction. Profound impact on how we work and play! Eyes-Free Interaction Google-IO May 2009 – 38 / 39
  39. 39. Watch Computing Take Off! Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion Eyes-Free Interaction Google-IO May 2009 – 39 / 39

×