User Interface Design

1. Dialog Design - Gesture & Pen
Interfaces, Mobile Devices
IAT 334 1
This material has been developed by Georgia Tech HCI faculty, and continues to evolve.
Contributors include Gregory Abowd, Jim Foley, Elizabeth Mynatt, Jeff Pierce, Colin Potts,
Chris Shaw, John Stasko, Bruce Walker, and Melody Moore Jackson. Comments directed to
foley@cc.gatech.edu are encouraged. Permission is granted to use with acknowledgement
for non-profit purposes. Last revision: October 2007.

2. Dialog Styles
1. Command languages
2. WIMP - Window, Icon, Menu, Pointer
3. Direct manipulation
4. Speech/natural language
5. Gesture & pen
IAT 334 2

3. Agenda
PDA overview
Pen input styles
IAT 334 3

4. How to use a PDA
IAT 334 4

5. Personal Digital Asst. (PDA)
IAT 334 5
Palm Treo
Apple Newton (1993)
Apple iPhone
Blackberry
Curve

6. PDAs
Now ubiquitous
Small displays
Often touch and pen interfaces
Small thumb-based keyboards
Recent Improvements
Wi-Fi, GPS, more memory, better CPU, better OS,
BlueTooth
IAT 334 6

7. Is it a PDA? Phone? GPS? Camera? Computer?
Line between devices is blurred today
Apple iPhone – phone, MP3 player, PDA, camera
Palm Treo 700w – phone, Windows computer,
PDA, camera
Nokia N82 – Phone, GPS, 2 cameras, robot brain
IAT 334 7

8. Cally/Callo
IAT 334 8

9. No Shredder…
IAT 334 9

10. Input Options
 Pen / Stylus is dominant form
 Main techniques
 Free-form ink
 Soft keyboard
 Numeric keyboard => text
 Stroke recognition - strokes not in the shape of characters
 Hand printing / writing recognition
 Sometimes have or can connect keyboard
IAT 334 10

11. Free-form Ink
Ink is the data, take as is
Human is responsible for
understanding and
interpretation
Like a sketch pad
IAT 334 11

12. Soft Keyboards
Common on PDAs and mobile devices
IAT 334 12

13. Soft Keyboard
 Presents a small diagram
of keyboard
 You click on buttons/keys
with pen or finger
 QWERTY vs. alphabetical
 Tradeoffs?
 Alternatives?
13
IAT 334
Apple iPhone soft keyboard

14. Numeric Keypad -T9
Tegic Communications developed
You press out letters of your word, it matches
the most likely word, then gives optional
choices
Faster than multiple presses per key
Used in mobile phones
www.tegic.com/t9
IAT 334 14

15. Stroke Recognition - Graffiti
Graffiti - Unistroke alphabet on Palm PDA
What are your
experiences
with Graffiti?
Graffiti demo:
http://www.youtube.com/watch?v=R-hbjG2hzuk
IAT 334 15

16. Stroke Recognition - Cirrin
Developed by Jen Mankoff (GT -> Berkeley CS
Faculty)
Word-level unistroke technique
UIST ‘98 paper
Use stylus to go
from one letter
to the next ->
Nokia N8 does similar
with QWERTY layout
IAT 334 16

17. QuikWriting
• Break the gesture into octant components:
• Start pen in center,
– drag in one of 8 directions
– drag along edge
– drag to center
http://mrl.nyu.edu/~perlin/demos/quikwriting2_1.html
IAT 334 17

18. Hand Printing & Hand Writing
Recognition
 Recognizing letters and numbers and special symbols
 Lots of commercial systems
 English, kanji, etc.
 Not perfect, but people aren’t either!
 People - 96% handprinted single characters
 Computer - >97% is really good
 OCR (Optical Character Recognition)
IAT 334 18

19. Recognition Issues
 Off-line vs. On-line
 Off-line: After all writing is done, speed not an issue, only
quality.
 Work with either a bit map or vector sequence
 On-line: Must respond in real-time - but have richer set of
features - acceleration, velocity, pressure
 Use best-guess pattern matching, including digram,
trigram probabilities and word lists to remove
ambiguity
 1 I l
IAT 334 19

20. More Issues
Boxed vs. Free-Form input
Sometimes encounter boxes on forms
Printed vs. Cursive
Cursive is much more difficult to impossible
Letters vs. Words
Cursive is easier to do in words vs individual letters,
as words create more context
IAT 334 20

21. Pen Gesture Commands
IAT 334 21
-Might mean delete
-Insert
-Paragraph
Define a series of (hopefully) simple drawing gestures
that mean different commands in a system

22. Pen Use Modes
Often, want a mix of free-form drawing and
special commands
How does user switch modes?
Mode icon on screen
Button on pen
Button on device
IAT 334 22

23. Error Correction
Having to correct errors can slow input
tremendously
Strategies
Erase and try again
When uncertain system shows list of best guesses
...
IAT 334 23

24. A Different Application
Signature verification
But not with a mouse :)
IAT 334 24

25. Multi-touch interfaces
Apple iPhone
IAT 334 25
Capacitive touchscreen:
http://www.apple.com/iphone/features/index.html#touch
Gestures:
flick, tap, pinch, un-pinch
http://www.apple.com/iphone/features/index.html#map

26. Pen Videos
• Pick-and-Drop by Rekimoto
http://www.youtube.com/watch?v=rFw9aMubL-Y
• I-Love-Sketch by Seok-Hyung Bae
http://www.youtube.com/watch?v=5Hd2clwURlQ
• Jabberstamp by Hayes Raffle
http://www.youtube.com/watch?v=NIe-XDHcsOE
• ShapeWriter iPhone App
http://www.youtube.com/watch?v=TOg91yfvZpo
• Marginalia : The Hybrid Textbook
http://www.youtube.com/watch?v=zwYdmsdqqTk
IAT 334 26