1. Eyes-Free User Interaction
Julian Harty Google
http://www.google.co.uk
September 18, 2009

2. Overview
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Android
Dialer
Contacts
TTS
Gestures
MusicBrowser
Overlays
Conclusion
Eyes-Free Interaction Techshare September 2009 – 2 / 39

3. Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Android Programming

4. Android: Engineer’s Dream!
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
s Smart phones — computer in your pocket.
s Open platform — liberates innovation.
s New I/O — minimize user interaction.
Bend technology to your will!
Eyes-Free Interaction Techshare September 2009 – 4 / 39

5. Innovative User Interfaces
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Create innovative end-user solutions that:
s Fit into the user’s mode of working,
s Avoid one-size fits all solutions,
s Design context-aware solutions,
s Tailor solutions to user’s needs.
Eyes-Free Interaction Techshare September 2009 – 5 / 39

6. Eyes-Free Interaction
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Not just for blind users!
s Eyes-busy environments.
s One-handed use.
s Minimize explicit user interaction.
Rethink all assumptions about the user.
Eyes-Free Interaction Techshare September 2009 – 6 / 39

7. Devices That Sense
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Mobile phones make UI design exciting!
s Can capture user intent — keyboard, . . ..
s Can see — camera.
s Can hear and speak — speech I/O.
s Can sense and stimulate touch — touch screen and vibration.
s Can sense motion — accelerometer, compass.
s Can sense position — GPS, cell towers.
.
.
s ..
Our Android has many eyes and ears!
Eyes-Free Interaction Techshare September 2009 – 7 / 39

8. Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Touch Screen Interaction, Eyes-Free!

9. Using A Touch Screen
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
These are atomic tasks:
s Identifying the item to touch.
s Touching item to activate it.
s Eyes-free use requires help with the former.
s Activation should produce appropriate feedback.
Eyes-Free Interaction Techshare September 2009 – 9 / 39

10. On-Screen Touchpads
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Eyes-Free use of an on-screen touchpad:
s Need to see the buttons to know where to touch.
s Because: buttons use absolute positioning.
s Conclusion: (wrong) — you need to see to use a touch screen!
s Solution: Relax constraint of absolute positioning.
What if we used relative positioning instead?
Eyes-Free Interaction Techshare September 2009 – 10 / 39

11. Phone Keypad
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Eyes-free phone dialer:
s Keypad centered where you touch — relative positioning.
s Users already know layout of a phone keypad.
s Can stroke from the center for a desired digit.
Enables eyes-free one-handed dialing!
Eyes-Free Interaction Techshare September 2009 – 11 / 39

12. Dialer Feedback
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Enables one-handed, eyes-free use:
s Buttons vibrate when you move over them.
s Augmented by auditory feedback.
s Pressing buttons produces spoken and tactile feedback.
s Shake to erase input.
Synchronized auditory and haptic feedback is key.
Eyes-Free Interaction Techshare September 2009 – 12 / 39

13. Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Text Input Via Touch Screen

14. Contact Manager
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
No one dials phone numbers any more!
s Browsing contacts using touch screen.
s Finding contacts.
s Managing contacts.
Eyes-Free Interaction Techshare September 2009 – 14 / 39

15. Entering Text Input
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Extending stroke dialer for textual input:
s Arrange letters in 4 concentric circles.
s Strokes access distinct circles.
s Choose desired letter by tracing chosen circle.
Any letter is at most 3 steps away!
Eyes-Free Interaction Techshare September 2009 – 15 / 39

16. Keypad Layout
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Gesture: Pick circle, and trace to desired character.
A I Q Letter Gesture
A Stroke up to the left
⊙ Y
B Stroke A, trace right
U M E E Stroke down to the right
Eyes-Free Interaction Techshare September 2009 – 16 / 39

17. Managing Contacts
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Contacts live in the cloud.
s Contacts managed over the network.
s Updated automatically on the device.
s No more editing contacts with two fingers!
Eyes-Free Interaction Techshare September 2009 – 17 / 39

18. Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Text To Speech On Android

19. TTS Library
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
s Create speech-enabled apps.
s Developer compiles against the TTS library stub.
s Text To Speech is an Android service.
s Class TTS manages communication with the service.
s Enables multiple apps to use TTS without code duplication.
s TTS library can be updated without a need to recompile clients.
Eyes-Free Interaction Techshare September 2009 – 19 / 39

20. TTS Method Summary
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
s speak(String text, int queueMode, String[] params)
s stop()
s isSpeaking()
s Also has methods for synthesizing to a file, specifying the language, etc.
Eyes-Free Interaction Techshare September 2009 – 20 / 39

21. TTS Object Properties
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
s Can automatically prompt the user to install the TTS Service.
s When absent, fail silently (methods become no-ops)
See: http://eyes-free.googlecode.com/
Eyes-Free Interaction Techshare September 2009 – 21 / 39

22. Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Gesture Library For Stroke Input

23. Gesture Overlay
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
s Overlay watches for touch events
s Returns identified gesture
s Exposes same UI as the stroke dialer
s Implemented as a custom transparent View with callbacks
Eyes-Free Interaction Techshare September 2009 – 23 / 39

24. Gesture Overlay Implementation
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
s Implement Interface GestureListener ,
s Create a new TouchGestureControlOverlay with it,
s Set a FrameLayout as the base View,
s Add the View with the content as a child to the FrameLayout,
s Add the TouchGestureControlOverlay as a child
Can easily enable/disable the TouchGestureControlOverlay
Eyes-Free Interaction Techshare September 2009 – 24 / 39

25. Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
A Simple Music File Browser

26. Music File Browser
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Barebones version:
s Uses a simple ListView
s Based on a FileBrowser tutorial on Anddev.org
s Browse directories and mp3 files on SDCard.
s Play music by clicking on the file.
s Directory path is the first entry.
s Click this to cycle through files in current level.
s Tactile feedback when scrolling through list items.
Eyes-Free Interaction Techshare September 2009 – 26 / 39

27. Adding Spoken Output
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
s Create object TTS in method onCreate.
s Result: Runs method TTS.onInit as part of setup.
s Can add application-specific code to method TTS.onInit.
s Result: Application comes up speaking.
Eyes-Free Interaction Techshare September 2009 – 27 / 39

28. Music Player Talks
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
private InitListener i = new InitListener() {
public void onInit(int version) {
tts.speak("Music File Browser started.", 0, null);
}
};
Eyes-Free Interaction Techshare September 2009 – 28 / 39

29. Speaking List Items
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
s Add TTS call to speak list items.
s Can use event handler that produces tactile feedback.
Eyes-Free Interaction Techshare September 2009 – 29 / 39

30. Speaking List Items
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Augment event handler with a TTS call.
public boolean onTrackballEvent(MotionEvent event) {
String filename = directoryEntries.get(selectedId);
tts.speak(filename.substring(1), 0, null);
}
Self-voicing functionality added with just 13 additional lines of code!
Eyes-Free Interaction Techshare September 2009 – 30 / 39

31. Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Adding Gesture Input To Music
Browser

32. Advantages Of An Overlay
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
s GestureOverlay is a transparent overlay.
s Can be layered on an existing View.
s Does not interfere with visual appearance.
Eyes-Free Interaction Techshare September 2009 – 32 / 39

33. Gestures For Music Browser
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
s Add GestureListener to handle user input.
s Gesture Tap: play/pause.
s Gesture Right: Browse to next track.
Add application-specific code in method onGestureFinish.
Eyes-Free Interaction Techshare September 2009 – 33 / 39

34. Steps In Adding Gesture Input
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
private GestureListener g = new GestureListener() {
public void onGestureChange(Gesture arg0) {}
public void onGestureFinish(Gesture arg0) {
if (arg0 == Gesture.CENTER) {
// play/pause music
} else if (arg0 == Gesture.RIGHT) {
playNext();
}
}
public void onGestureStart(Gesture arg0) {}
};
Eyes-Free Interaction Techshare September 2009 – 34 / 39

35. Adding Overlay To Music Player
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
myFrame = new FrameLayout(this);
myFrame.addView(myList);
myG = new TouchGestureControlOverlay(this,
gestureListener);
overlayActive = false;
setContentView(myFrame);
Eyes-Free Interaction Techshare September 2009 – 35 / 39

36. Toggling Gesture Support
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
if (overlayActive) {
myFrame.removeView(myG);
overlayActive = false;
tts.speak("Gestures disabled.", 0, null);
} else {
myFrame.addView(myG);
overlayActive = true;
tts.speak("Gestures activated.", 0, null);
}
Gesture support enabled by adding about 40 lines of code.
Eyes-Free Interaction Techshare September 2009 – 36 / 39

37. Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Conclusion

38. Summary
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
s Programmable smart phones are an engineer’s dream!
s I/O peripherals open up new UI avenues.
s Provide desired information with minimal user interaction.
Profound impact on how we work and play!
Eyes-Free Interaction Techshare September 2009 – 38 / 39

39. Get in touch and get involved
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion
Julian Harty jharty@google.com
http://eyes-free.googlecode.com/
Eyes-Free Interaction Techshare September 2009 – 39 / 39