Gesture Design
for (and with) Mobiles Design For Mobile
April 20, 2009
Lawrence, KS

Who am I?
Barbara Ballard
Principal at Little Springs Design
Curator of Design for Mobile
MBA from the University of Kansas
Author of Designing the Mobile User Experience
Mobilist
2

Setting the scene
Culture
Technology
3

Minority Report
4

Inspired
5

Jeff Han
multi-touch
6

What are gestures?
Kinesthetic
Communicative
Inherent
meaning to
others
Intentional or not
7

Gestures & language
Key part of
language
Part of speech
production
Cognitive
representation
Unique to
speaker
Viewers absorb
meaning
8

Why gestures now?
Everybody is
blind sometimes
9

Multi-touch
10

Efficient interaction
Mouse click: x, y, t
Key click: i, t
Mouse move: x, x’, y, y’, t, t’
Touch screen swipe: x, x’, y, y’, t, t’ and path
Human hand: indefinite, 3D
Gesture: meaning
Richer interaction.
11

Path? So?
Swype
Shapewriter
12

Gestures & brand
Iconographic
gestures
Branding
without product
WWE fans?
West Ham?
Protest?
13

Mobile gesture taxonomy
(Human to environment)
Device to environment
Between devices
At device
With device
On device
15

ASL sign taxonomy
Transparent: guessable
Translucent: explainable
Opaque: inexplicable
16

TAT gesture taxonomy
Iconic Metaphoric Symbolic
Transparent Translucent Opaque
Instantly Advanced users
Easily
understood
remembered Separate from UI
Slow
Depends on UI Difficult to
Context elements remember
dependent
Can be used to Feedback useful
Relies on discover new fn Magic quality
affordances
Gesture is the Gesture is an Gesture is its own
meaning enabler layer
17

Mobile gesture taxonomy
(Human to environment)
Device to environment
Between devices
At device
With device
On device
18

Human to environment
19

Human to environment
20

Human to environment
Canesta
Kiosk systems
Museum
displays
No mobile
phone!
21

Device to environment
Text
22

Device to environment
23

Available technologies
Challenge: device must communicate with
environment
NFC, RFID: rare, forthcoming
Bluetooth: common, somewhat scary to many
Wi-Fi: small number of devices ... iPhone,
Android
What can we do with this, now?
24

Between devices
Information
transfer
Bluetooth,
network, NFC
Security issues
Future
interaction
25

Between devices
Phone locked,
loud when in
purse?
26

At device
Camera input
27

At device
Sonic input
28

With device
Accelerometers
29

With device
Apple patent on
changing UI
when vibrating
30

With device
Pressure
sensors
recognize hand
grip
31

With device
Shake
Tilt
Rotate
Marble game
Koi pond
Driving game
32

On device
Behind screen
33

On device
Eyes-free touch
dialing
5 is where
finger hits
34

On device
35

On device
Flick
Swipe
Pinch
Consider OS
norms
Patents
36

Prototyping & Research
37

Design issues
Human factors
Industrial design
Visual design
38

Technologies
Camera
Accelerometers
Touch screen
Multi-touch
NFC, RFID,
Bluetooth
Pressure sensor
Light sensor
Platform!!
39

Rules for design (Saffer)
Discoverable
Trustworthy
Responsive
Appropriate
Meaningful
Smart
Clever
Playful
Pleasurable
Good
40

Goals for design
Sharable
Replaceable
Memorable
Symmetrical
Responsive
HF principles
41

Responsive
Interaction must feel smooth
to achieve trust
Focus on apps
Visual, auditory, tactile
Immediate
42

Anatomy of gesture (Saffer)
Presence
Duration
Position
Motion
Pressure
Size
Orientation
Other objects
Combination of
touch points
Sequence
(# participants)
43

Detecting context
Sensors detect
Movement
Light
etc.
Change UI
44

How many gestures?
Motivation
Predictability
Affordances
Practice
3 (pinch, swipe,
flick)
1,000 (ASL)
45

Motivation
Ekapad
12-key, 3-finger
chording
46

or Alt P & RELEASE to
R hand L hand
Motivation AltR or activate commands CapsShift
qwerty right side
!R or CtrlR OptionR ctrlR or wwR ShiftR
EkaTetra Cheat Sheet v 5.7 Display.pdf
ekatetra.com Copyright © EkaTetra 2008 All rights reserved
2
x-ray views Start here P&R = Press & Release
P&R boxed chord 1st - then P&R Caps if shown
For a chorded P&R Caps P&R chord
character
1
= +
P&R
with symbol %
2
Every command, prefix, & character (except one finger
1
alphas and numerals) is available whatever the state.
x-ray views are thumb side views of keys & chords.
Right hand
sot
aei fk b m zy qu
hrn gjw pdl vcx
. period Space Return Tab Left hand
Pressing any key completes and/or ends a command action
uq
P&R Alpha
NonBreak
, comma Space Enter esc to return to
sot
aei
hrn
lower case
( )
[ ]
$;:
+=-
@* / { }
‹ ›
GridPad Numerical keypad on full size qwertyboards
« »
%!_
&•\\ \" l' ?
Backspace Forward
Delete
Nlock Delete Caps CapsLock
SOT
321
654 AEI
987 HRN
Windows: In Mac use Navig
12 21
0
P&R Nlock to exit P&R CapsLock to exit
Nlock default layout is N pad p (phone)
HelpInsert Navigation includes Deletes Accent
¸¨
! pU P&R
: \" End # Accent
´
ˆ `~
to move cursor.
H $ pD .+mark
P&R Navig to exit + letter or Space
Macintosh:
Post PostCaps 3postCaps 3post same as Caps
Use 3post for
1 1 symbols with
Caps in Nlock
or anywhere.
2 2
Extended character sets
§ ø¬
å € # …ƒ \\ ß µ æ ¥ œ
¢ | <° ÷
^` ~ ª ç° £†¶
blue symbols vary by font
±> – | ‚ „ ‰¡— ” ’ ¿
-
“‘
+ =
* /
GridPad Enter
white symbols unique to each OS
Macintosh additional character set
¯ !ı
√ Ø™
%& '
Æ$Œ
#flfi ˙˝
˚
Å\" !
˛ *+,
˘©
(Ç\" )‡
⁄®
ˇ
Windows additional character set
! Ø™ ³²¹
¾ ½¼
¤ \" þ ð ¯
Å Æ Œ
#
×
ÞÇ& Ї - ©¦
$®%
soft
Print Scroll keyboard
Windows only
Relic Screen Lock Pause Power
P
P
ShortCut & Keep
Store 99 ShortCuts & 99 Keeps: for 1 digit: Enter # entry Enter; for 2
digit: Enter Enter # # entry Enter. To retrieve: Access # or A A ##.
To clear: Enter # Enter or EE## E.
ShortCut Keep
Enter Access Enter Access Code
*/ +
321 321 321 321
-
654 654 654 654
987 987 987 987
0 all empty 0 0 all empty 0 Config + c
Code+password+KeepAccess unlocks Keeps if passsword protected.
Create 1 to 10 ShortCut sets, each with ten 1-finger chords.
To make it easy to access sets of ShortCuts for an application, organize
them into ranges like 10,11,…19 or 50,51,…59 entering up to ten
ShortCuts in a range. Accessed by 10-ones chord + first digit, each
set gives repeated one finger access to ShortCuts in the range.
Volume F delete/
To Access a set of one finger ShortCuts:
P&R 10-ones, then P&R first digit of two digit location.
10-ones first digit repeated 1-finger chords
3 2 1 1-finger chord 3 2 1 More actions
6 5 4 does ShortCut 6 5 4 until exit with
+
9 8 7 stored in 2nd 9 8 7 Alpha.
digit location
0 0
Mute
Up Down
Eject Clear
Commands Primary: qwerty keyboard left side
! or Ctrl Option ctrl or ww ShiftL
or Alt P & RELEASE to
R hand L hand
AltR or activate commands CapsShift
qwerty right side
!R or CtrlR OptionR ctrlR or wwR ShiftR
L R 2
1
Pressing any key completes and/or ends a command action
GridPad Numerical keypad on full size qwertyboards
Windows: In Mac use Navig
to move cursor.
Macintosh:
LR
-
+ =
* /
GridPad Enter
*/ + -
Volume F delete/
Mute
Up Down
Eject Clear
L R
L R
LR
L R
. period
Alpha exits Gnum Numlock
987
Windows
654
only
321
0 Numlock switches between &
Page
Insert Home End Up Down
. period
Arrow Forward
Delete
Left Right Up Down
Alpha exits Gnum Numlock Configs change settings; Config chord + 1st letter chord
Config OS select GridPad Reset FingerUp
WM 987
M 654
W 321
0 Resets in blue 1st~Last
OS=Config+o+m or w
Code Q Set
987
Create If Alph On,
Alph OnOff
(empty) password with password EkaPad
Q set. Unlock 3 2 1 returns to
Keeps with 6 5 4 AlphaState
Code. after 5
987
Ekapad
Code+password Q Set+pass- minutes of Off is default
Windows
no chords
word+KAccess
+ KeepAccess
Npad (Config+n+letter)
Sleep
phone ten right left
654
Mac dialog box
w 321 987 741 369
e 654 654 852 258
w 987 321 963 147
only
Wake Npd+p 0 Npd+t 0 Npd+r 0 Npd+l 0
Mac: UnPlug EkaPad to wake
Trigger Repeat & Hold-Down Version#
v
321 To hold down Shift, %, Option,
ctrl, Alt, press for Trigger delay
654 e
321
time (0.5 sec-2 secs) and then
987 vr
release. To Repeat, hold down
ver together
Config+t+ chord and release. P&R any key to
Displays the current
1 to 8 sets exit. 9 = off
version of EkaPad
delay time.
software.
Functions
12-key, 3-finger
P&R F once 3 2 1 P&R F twice Function
321
numbers 1
0 Numlock switches between
for one digit 6 5 4 for two digit 654
through 24
&
function function
987 987
work.
0
then P&R the number key(s)
Restart = FF99 Restart
stops &
restarts
computer
Page
Names of State chords are colored: Alpha Navig Nlock
CapsLock ShortCut&KeepEnter GridPad 10-ones
chording
EkaTetra CheatSheet for the USB EkaPad
Find additional Tools at www.ekatetra.com/support
EkaTetra
16580 Maple Circle, Lake Oswego OR 97034 USA
Insert Home End Up Down
503-697-0604 www. ekatetra.com
EkaTetra<Eka_T_USB_CheatSheet
One hand it – with EkaPad!
®
47

Motivation
48

Affordances
Design physical
& visual
Avoid
instructions
49

Metaphors
Direct
manipulation
Symbolic
Gestures as
part of language
50

Thank you!
P.O. Box 44-2302
Lawrence, KS 66044
785 838 3003
littlespringsdesign.com
51