Authoring Sensor-Based
Interactions by Demonstration
with Direct Manipulation and Pattern Recognition
Björn Hartmann, Leith Abdulla
Manas Mittal, Scott R. Klemmer
1

How would you prototype…
…a workout monitoring system?
[Apple, Nike]
2

How would you explore…
…motion-based game controllers?
[Nintendo]
3

How would you build…
…a proximity-aware whiteboard?
[Ju, Lee, Akers]
4

Sensor-based
Interaction Design
5

d.tools
[Hartmann et al., UIST ’06]
6

d.Tools
d.tools
Student Project
7

Designing Sensor-based Interactions
PROTOTYPE APPLICATION LOGIC
SPECIFY RELATIONSHIP
BETWEEN SENSOR DATA
AND APPLICATION LOGIC
PROVIDE SOFTWARE ABSTRACTION
FOR HARDWARE
8

Types of Sensors
Force Sensitive
Resistors
Accelerometers
RFID
IR/Ultrasonic
Rangers
Phototransistors
Discrete Continuous Categorical
9

Current Tools:
Oscilloscopes
[CC image by
Timo Arnall] 10

Current Tools:
LabView
LabView
11

Representation Matters Accelerometer X axis
Accelerometer Y axis
//detect accelerometer peaks
//read data sample
xVal[t++]=readA2DValue(xPin);
//look for changes in derivative
if(((xVal[t]-xVal[t-1]) >= 0
&& (xVal[t-1]-xVal[t-2]) < 0)
(((xVal[t]-xVal[t-1]) < 0
&& (xVal[t-1]-xVal[t-2]) >= 0
//peak detected
//send message
oscSendMessageInt("/x/peak",1);
} else {
//no peak
}
12

Idea: Programming by Demonstration
13

Idea: Programming by Demonstration
14

Tacit Knowledge
15

Crux: Generalization
16

Crux: Generalization
17

Crux: Generalization
18

Exemplar
19

Exemplar
20

Exemplar
21

Exemplar
22

23

Exemplar UI
24

Exemplar UI
25

Exemplar UI
26

Exemplar UI
27

Exemplar UI
28

Exemplar UI
29

Exemplar UI
30

[Flash animations from the talk are not included in this presentation]
31

Dynamic Time Warping
Demonstration Signal
Matching Input Signal
[Sakoe, H. Chiba, S. ‘78]
32

Filters
Offset
Y-Axis Scaling
Smoothing
Rate of Change
33

Extensible Filter Set
34

Understanding the Exemplar Approach
What? Why?
Cognitive Dimensions of Analysis of Exemplar as a
Notation (CDN) Inspection visual authoring environment
First-Use Laboratory Study Threshold and usability
Class Deployment Real-world stress test
Used as Design Tools for Complexity ceiling for
“Building Upon Everyday Play” knowledgeable users
CHI Interactivity Exhibit
35

Lab Study
[sketches by Wendy Ju]
36

Lab Study
26 min 18 min 27 min
47 min 22 min 31 min
[sketches by Wendy Ju]
37

38

Mini Joystick
Bend Sensor
taped to table
Navigation Navigation Test Play
with Accelerometer with XY Joystick
39

Mini Joystick
Bend Sensor
taped to table
Trigger with Test Play Refine trigger Test Play
Bend sensor detection
40

Post-Test Questionnaire
41

Post-Test Questionnaire
42

CHI Interactivity
[Control Freaks by Haiyan Zhang]
43

CHI Interactivity
Compulsory latency of pattern matching
precludes use for fast-paced actions
Threshold Pattern
crossed here matched here
time
[Control Freaks by Haiyan Zhang] 44

Related Work
PBD Tools Musical controller Commercial
for Ubicomp Design tools Dsp software
a Capella HID Toolkit LabView/
[Dey et al., CHI 04] [Steiner, NIME 05] Lego Mindstorms NXT
Crayons MnM
[Fails & Olsen, CHI 03] [Bevilacqua et al., NIME 05]
Monet FlexiGesture
[Li, Landay, UIST 05] [Merrill & Paradiso, CHI 05]
Papier-Mâché
[Klemmer et al., CHI 04]
45

Related Work
A CAPella (Dey et al., CHI 04)
46

Related Work
FlexiGesture (Merrill CHI05)
47

Related Work
LabView
48

Related Work
Lego Mindstorms NXT 49

Future Directions
Export: Generate code +
specifications
Programming sensor networks
by demonstration
d.mix: Programming by
A Sample for Web Service APIs
Integration of direct
manipulation environments
with textual programming
50

Acknowledgments
We thank MediaX/DNP for funding,
Intel for equipment donation,
Wendy Ju for illustrations,
David Merrill & Timo Arnall for photos.
51

Download Exemplar:
http://hci.stanford.edu/exemplar
CHI Interactivity “Building Upon Everyday Play”
Talk tomorrow, 11:30am, Room C2
52