Exemplar: Designing Sensor-based interactions by demonstration... (a CHI2007 Best Paper)

Slide 1: Authoring Sensor-Based Interactions by Demonstration with Direct Manipulation and Pattern Recognition Björn Hartmann, Leith Abdulla Manas Mittal, Scott R. Klemmer 1

Slide 2: How would you prototype… …a workout monitoring system? [Apple, Nike] 2

Slide 3: How would you explore… …motion-based game controllers? [Nintendo] 3

Slide 4: How would you build… …a proximity-aware whiteboard? [Ju, Lee, Akers] 4

Slide 5: Sensor-based Interaction Design 5

Slide 6: d.tools [Hartmann et al., UIST ’06] 6

Slide 7: d.Tools d.tools Student Project 7

Slide 8: Designing Sensor-based Interactions PROTOTYPE APPLICATION LOGIC SPECIFY RELATIONSHIP BETWEEN SENSOR DATA AND APPLICATION LOGIC PROVIDE SOFTWARE ABSTRACTION FOR HARDWARE 8

Slide 9: Types of Sensors Force Sensitive Resistors Accelerometers RFID IR/Ultrasonic Rangers Phototransistors Discrete Continuous Categorical 9

Slide 10: Current Tools: Oscilloscopes [CC image by Timo Arnall] 10

Slide 11: Current Tools: LabView LabView 11

Slide 12: Representation Matters Accelerometer X axis Accelerometer Y axis //detect accelerometer peaks //read data sample xVal[t++]=readA2DValue(xPin); //look for changes in derivative if(((xVal[t]-xVal[t-1]) >= 0 && (xVal[t-1]-xVal[t-2]) < 0) (((xVal[t]-xVal[t-1]) < 0 && (xVal[t-1]-xVal[t-2]) >= 0 //peak detected //send message oscSendMessageInt("/x/peak",1); } else { //no peak } 12

Slide 13: Idea: Programming by Demonstration 13

Slide 14: Idea: Programming by Demonstration 14

Slide 15: Tacit Knowledge 15

Slide 16: Crux: Generalization 16

Slide 17: Crux: Generalization 17

Slide 18: Crux: Generalization 18

Slide 19: Exemplar 19

Slide 20: Exemplar 20

Slide 21: Exemplar 21

Slide 22: Exemplar 22

Slide 23: 23

Slide 24: Exemplar UI 24

Slide 25: Exemplar UI 25

Slide 26: Exemplar UI 26

Slide 27: Exemplar UI 27

Slide 28: Exemplar UI 28

Slide 29: Exemplar UI 29

Slide 30: Exemplar UI 30

Slide 31: [Flash animations from the talk are not included in this presentation] 31

Slide 32: Dynamic Time Warping Demonstration Signal Matching Input Signal [Sakoe, H. Chiba, S. ‘78] 32

Slide 33: Filters Offset Y-Axis Scaling Smoothing Rate of Change 33

Slide 34: Extensible Filter Set 34

Slide 35: Understanding the Exemplar Approach What? Why? Cognitive Dimensions of Analysis of Exemplar as a Notation (CDN) Inspection visual authoring environment First-Use Laboratory Study Threshold and usability Class Deployment Real-world stress test Used as Design Tools for Complexity ceiling for “Building Upon Everyday Play” knowledgeable users CHI Interactivity Exhibit 35

Slide 36: Lab Study [sketches by Wendy Ju] 36

Slide 37: Lab Study 26 min 18 min 27 min 47 min 22 min 31 min [sketches by Wendy Ju] 37

Slide 38: 38

Slide 39: Mini Joystick Bend Sensor taped to table Navigation Navigation Test Play with Accelerometer with XY Joystick 39

Slide 40: Mini Joystick Bend Sensor taped to table Trigger with Test Play Refine trigger Test Play Bend sensor detection 40

Slide 41: Post-Test Questionnaire 41

Slide 42: Post-Test Questionnaire 42

Slide 43: CHI Interactivity [Control Freaks by Haiyan Zhang] 43

Slide 44: CHI Interactivity Compulsory latency of pattern matching precludes use for fast-paced actions Threshold Pattern crossed here matched here time [Control Freaks by Haiyan Zhang] 44

Slide 45: Related Work PBD Tools Musical controller Commercial for Ubicomp Design tools Dsp software a Capella HID Toolkit LabView/ [Dey et al., CHI 04] [Steiner, NIME 05] Lego Mindstorms NXT Crayons MnM [Fails & Olsen, CHI 03] [Bevilacqua et al., NIME 05] Monet FlexiGesture [Li, Landay, UIST 05] [Merrill & Paradiso, CHI 05] Papier-Mâché [Klemmer et al., CHI 04] 45

Slide 46: Related Work A CAPella (Dey et al., CHI 04) 46

Slide 47: Related Work FlexiGesture (Merrill CHI05) 47

Slide 48: Related Work LabView 48

Slide 49: Related Work Lego Mindstorms NXT 49

Slide 50: Future Directions Export: Generate code + specifications Programming sensor networks by demonstration d.mix: Programming by A Sample for Web Service APIs Integration of direct manipulation environments with textual programming 50

Slide 51: Acknowledgments We thank MediaX/DNP for funding, Intel for equipment donation, Wendy Ju for illustrations, David Merrill & Timo Arnall for photos. 51

Slide 52: Download Exemplar: http://hci.stanford.edu/exemplar CHI Interactivity “Building Upon Everyday Play” Talk tomorrow, 11:30am, Room C2 52