Authoring Sensor-based Interactions by Demonstration with Direct Manipulation and Pattern Recognition Björn Hartmann, Leith Abdulla, Manas Mittal, Scott Klemmer Contributes method and tool for rapidly designing sensor-based interactions by demonstration; emphasizes control of generalization criteria through integrating direct manipulation and pattern recognition; offers theoretical and first-use lab evaluations.

1. Authoring Sensor-Based
Interactions by Demonstration
with Direct Manipulation and Pattern Recognition
Björn Hartmann, Leith Abdulla
Manas Mittal, Scott R. Klemmer
1

2. How would you prototype…
…a workout monitoring system?
[Apple, Nike]
2

3. How would you explore…
…motion-based game controllers?
[Nintendo]
3

4. How would you build…
…a proximity-aware whiteboard?
[Ju, Lee, Akers]
4

5. Sensor-based
Interaction Design
5

6. d.tools
[Hartmann et al., UIST ’06]
6

7. d.Tools
d.tools
Student Project
7

8. Designing Sensor-based Interactions
PROTOTYPE APPLICATION LOGIC
SPECIFY RELATIONSHIP
BETWEEN SENSOR DATA
AND APPLICATION LOGIC
PROVIDE SOFTWARE ABSTRACTION
FOR HARDWARE
8

9. Types of Sensors
Force Sensitive
Resistors
Accelerometers
RFID
IR/Ultrasonic
Rangers
Phototransistors
Discrete Continuous Categorical
9

10. Current Tools:
Oscilloscopes
[CC image by
Timo Arnall] 10

11. Current Tools:
LabView
LabView
11

12. Representation Matters Accelerometer X axis
Accelerometer Y axis
//detect accelerometer peaks
//read data sample
xVal[t++]=readA2DValue(xPin);
//look for changes in derivative
if(((xVal[t]-xVal[t-1]) >= 0
&& (xVal[t-1]-xVal[t-2]) < 0)
(((xVal[t]-xVal[t-1]) < 0
&& (xVal[t-1]-xVal[t-2]) >= 0
//peak detected
//send message
oscSendMessageInt("/x/peak",1);
} else {
//no peak
}
12

13. Idea: Programming by Demonstration
13

14. Idea: Programming by Demonstration
14

15. Tacit Knowledge
15

16. Crux: Generalization
16

17. Crux: Generalization
17

18. Crux: Generalization
18

19. Exemplar
19

20. Exemplar
20

21. Exemplar
21

22. Exemplar
22

23. 23

24. Exemplar UI
24

25. Exemplar UI
25

26. Exemplar UI
26

27. Exemplar UI
27

28. Exemplar UI
28

29. Exemplar UI
29

30. Exemplar UI
30

31. [Flash animations from the talk are not included in this presentation]
31

32. Dynamic Time Warping
Demonstration Signal
Matching Input Signal
[Sakoe, H. Chiba, S. ‘78]
32

33. Filters
Offset
Y-Axis Scaling
Smoothing
Rate of Change
33

34. Extensible Filter Set
34

35. Understanding the Exemplar Approach
What? Why?
Cognitive Dimensions of Analysis of Exemplar as a
Notation (CDN) Inspection visual authoring environment
First-Use Laboratory Study Threshold and usability
Class Deployment Real-world stress test
Used as Design Tools for Complexity ceiling for
“Building Upon Everyday Play” knowledgeable users
CHI Interactivity Exhibit
35

36. Lab Study
[sketches by Wendy Ju]
36

37. Lab Study
26 min 18 min 27 min
47 min 22 min 31 min
[sketches by Wendy Ju]
37

38. 38

39. Mini Joystick
Bend Sensor
taped to table
Navigation Navigation Test Play
with Accelerometer with XY Joystick
39

40. Mini Joystick
Bend Sensor
taped to table
Trigger with Test Play Refine trigger Test Play
Bend sensor detection
40

41. Post-Test Questionnaire
41

42. Post-Test Questionnaire
42

43. CHI Interactivity
[Control Freaks by Haiyan Zhang]
43

44. CHI Interactivity
Compulsory latency of pattern matching
precludes use for fast-paced actions
Threshold Pattern
crossed here matched here
time
[Control Freaks by Haiyan Zhang] 44

45. Related Work
PBD Tools Musical controller Commercial
for Ubicomp Design tools Dsp software
a Capella HID Toolkit LabView/
[Dey et al., CHI 04] [Steiner, NIME 05] Lego Mindstorms NXT
Crayons MnM
[Fails & Olsen, CHI 03] [Bevilacqua et al., NIME 05]
Monet FlexiGesture
[Li, Landay, UIST 05] [Merrill & Paradiso, CHI 05]
Papier-Mâché
[Klemmer et al., CHI 04]
45

46. Related Work
A CAPella (Dey et al., CHI 04)
46

47. Related Work
FlexiGesture (Merrill CHI05)
47

48. Related Work
LabView
48

49. Related Work
Lego Mindstorms NXT 49

50. Future Directions
Export: Generate code +
specifications
Programming sensor networks
by demonstration
d.mix: Programming by
A Sample for Web Service APIs
Integration of direct
manipulation environments
with textual programming
50

51. Acknowledgments
We thank MediaX/DNP for funding,
Intel for equipment donation,
Wendy Ju for illustrations,
David Merrill & Timo Arnall for photos.
51

52. Download Exemplar:
http://hci.stanford.edu/exemplar
CHI Interactivity “Building Upon Everyday Play”
Talk tomorrow, 11:30am, Room C2
52