An end-to-end HCI project trajectory: How to learn a model of family logistical routines, and present that information to families to help them feel more in control of their lives.

1. Routine as Resource for the Design of Learning Systems Scott Davidoff Dissertation Defense

2. A pattern of behavior that is followed repeatedly, but is subject to change if conditions change Winter 1964 “ “” 1

3. Koestler 1967 Routines reduce our attention needs 2

4. Zerubavel 1981 Reduce attention needed for tasks 3

5. Wakkary & Maestri 2007 Free attention for bigger challenges 4

6. Wolin + Bennett 1984 5

7. The cyclic nature of routine makes it a natural target for machine learning

8. Machine learning can use this order 7

9. Suchman 1983, Tolmie et al. 2002 Idiosyncrasies are hard to model 8

10. Certain human routines can be modeled, increasing the scope of activity recognition Machine learning Opportunities 9

11. The ability to use learned routines in end-user applications would solve a variety of human problems HCI Opportunities 10

12. Dual-income family logistics 11

13. Sequence of place and transportation (rides) that occur on daily, weekly, and seasonal cycles “ “” 12

14. Darrah et al. 2000 Managing details can be difficult 13

15. Frissen 2000 Routines give a feeling of control 14

16. Beech et al. 2004 Life does not always follow routines 15

17. Breakdowns lead to loss of control 16

18. Perry et al. 2001, Ling + Campbell 2003 A constant need to follow updates 17

19. Gneezy + Rustichini 1998, Darrah 2009 A constant source of anxiety 18

20. We can learn a model of family logistical routines, and present that information to families to help them feel more in control of their lives 19

21. 20

22. 21

23. 2 We can use sensing and modeling to synthesize missing information resources 3 Show how to use the model and evaluate the impact of the information 1 We can use fieldwork to identify missing but needed information resources Fieldwork Modeling Validation 22

24. Create an ordered list of places and rides Attend to the details of a plan as it unfolds Logistics Planning and coordination Coordinate Plan 23

25. Fieldwork 1 24

26. 22 12 Davidoff et al Ubicomp 2006 Months Families 18 06 Davidoff et al Ubicomp 2007 Months Families 05 06 Davidoff, Dey + Zimmerman CHI 2009 Months Families 45 24 Months Families 25

27. Semi-Structured Interviews Davidoff et al. Ubicomp 2006 26

28. Needs Validation scott davidoff, min kyung lee, anind dey + john zimmerman

29. Davidoff et al. Ubicomp 2007 User Enactments 28

30. User Enactments Davidoff et al. Ubicomp 2007 29

31. GPS Phone Calls Email SMS Calendars Davidoff, Dey + Zimmerman CHI 2009 30

32. 528 phone interviews 109 activity Interviews 108 calendar months Davidoff, Dey + Zimmerman CHI 2009 31

33. Less than 20% of days go exactly as planned 1 Routines are not documented People have incomplete knowledge of other people’s routines People make plans that depend on incorrect information 2 3 4 32

34. 33

35. Family E Routine Deviation, Scheduled Deviation, Unscheduled 34

36. Family E All Families Routine Deviation, Scheduled Deviation, Unscheduled 35

37. Less than 20% exactly as planned 36

38. Routineness of Family E Activities 37

39. Routineness of Family E Activities 38

40. Routineness of Family E Activities 39

41. Routines are not documented Fieldwork Finding Davidoff, Dey + Zimmerman CHI 2010 40

42. August November September October Davidoff, Dey + Zimmerman CHI 2010 41

43. People have incomplete knowledge of other people’s routines Fieldwork Finding Davidoff, Dey + Zimmerman CHI 2010 42

44. Davidoff, Dey + Zimmerman CHI 2010 43

45. Davidoff, Dey + Zimmerman CHI 2010 44

46. Davidoff, Dey + Zimmerman CHI 2010 45

47. Davidoff, Dey + Zimmerman CHI 2010 46

48. Davidoff, Dey + Zimmerman CHI 2010 47

49. Davidoff, Dey + Zimmerman CHI 2010 48

50. Davidoff, Dey + Zimmerman CHI 2010 49

51. Davidoff, Dey + Zimmerman CHI 2010 50

52. People make plans that depend on incorrect information Fieldwork Finding Davidoff, Dey + Zimmerman CHI 2010 51

53. View of Plan: Dad 4:00 5:00 6:00 7:00 3:00 2:00 8:00 School Home Track Practice Dad Work Orthodontist Check-up Scouts S16 Dad 52

54. View of Plan: S16 4:00 5:00 6:00 7:00 3:00 2:00 8:00 School Home Paper Route Track Practice Dad Work Orthodontist Scouts S16 Dad 53

55. Who Will do the Paper Route? 4:00 5:00 6:00 7:00 3:00 2:00 8:00 School Home Paper Route Track Practice Dad Work Orthodontist Check-up Scouts S16 Dad 54

56. Information gaps can break down coordination No resources exist to find needed information Davidoff, Dey + Zimmerman CHI 2010 55

57. Modeling 2 56

58. 2 3 1 Ride Detection Driver Prediction Predict Lateness 57

59. In Situ Observation Maintain Current Behaviors Ubiquitous Sensing GPS 58

60. Parent drives kid to an activity Parent drives kidfromactivity Ride Pick-ups and drop-offs Pick-up Drop-off 59

61. Ride Detection

62. Using GPS to Sense a Drop-Off t3 t4 t5 t6 t2 t1 t7 Day Care Work Parent Child 61

63. Using GPS to Sense a Drop-Off t3 t4 t5 t6 t2 t1 t7 Day Care Work Parent Child 62

64. Using GPS to Sense a Drop-Off t3 t4 t5 t6 t2 t1 t7 Day Care Work Parent Child 63

65. Using GPS to Sense a Drop-Off t3 t4 t5 t6 t2 t1 t7 Day Care Work Parent Child 64

66. Using GPS to Sense a Pick-Up t3 t4 t5 t6 t2 t1 t7 Day Care Work Parent Child 65

67. Using GPS to Sense a Pick-Up t3 t4 t5 t6 t2 t1 t7 Day Care Work Parent Child

68. Using GPS to Sense a Pick-Up t3 t4 t5 t6 t2 t1 t7 Day Care Work Parent Child

69. GPS Ride Detection Evaluation 68

70. Smarter power management Wi-Fi, Bluetooth, cell tower ID Apply heuristics to modeling Cultural norms, Individual Behavior Single Location Sensor GPS Sampling Rate 69

71. Driver Prediction

72. Driver Prediction Model Features 71

73. Driver Prediction Performance x Number of Training Weeks of Data 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 72

74. Driver Prediction Performance x Number of Training Weeks of Data 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 73

75. Driver Prediction Performance x Number of Training Weeks of Data 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 74

76. Lateness Prediction

77. Distribution of Parent Late Arrivals Number of Examples TnowTideal Lateness in Minutes N = 83 76

78. 77

79. 78

80. 79

81. 80

82. 81

83. tideal- 30 .6 A’ = 0.659 .5 82

84. tideal A’ = 0.801 83

85. tideal+10 A’ = 0.826 84

86. We can learn a model of family logistical routines, and present that information to families to help them feel more in control of their lives 85

87. Validation 3 86

88. 87

89. Do parents understand the view? Do parents perceive that the information is valuable? Do parents feel more in control having this view? 88

90. 45 24 Davidoff et al Ubicomp 2006 Davidoff et al Ubicomp 2007 Davidoff, Dey + Zimmerman CHI 2009 Davidoff et al. CHI 2010 Months Families 12 Davidoff et al. Ubicomp 2011 (in preparation) Families 57 24 Months Families 89

91. Davidoff et al. Ubicomp 2011 in preparation Experience prototype: Doctor 90

92. 91

93. 92

94. Davidoff et al. Ubicomp 2011 in preparation Experience prototype: Kitchen 93

95. It's nice to have a map of the day. I don't literally write all this stuff down, so sometimes it's just hard to keep in my head. Visual Distills Details “ “” −P2 94

96. Because the calendar [is] all words and numbers, so you have to really think about …how long everything takes… Visual Simplifies Calculation “ −P10 95

97. …With the visual you can see who's doing what at what time. You can make a decision about who's activity to change. Visual Simplifies Calculation “” −P10 96

98. I would be able to know where everyone was gonna be, instead of having to ask around. Visual Clarifies Intentions “ “” −P11 97

99. Venkatesh + Davis 2003 Technology Acceptance Model-3 98

100. …helps me do my job “as a parent.” …would want my family to use it. …more in control of details, more ready for changes Perception of Control Behavioral Intention Perceived Usefulness 99

101. Perceived Usefulness Behavioral Intention Perception of Control Supports Planning Supports Awareness 7 6 5 4 3 2 1 100

102. We can learn a model of family logistical routines, and present that information to families to help them feel more in control of their lives 101

103. Contributions 102

104. Suchman 1983, Tolmie et al. 2002 Certain routines can be modeled 103

105. 2 3 1 Ride Detection Driver Prediction Predict Lateness 104

106. Commodity GPS with real power constraints 105

107. 106

108. Breakdowns lead to loss of control 107

109. Perceived Usefulness Behavioral Intention Perception of Control Supports Planning Supports Awareness 7 6 5 4 3 2 1 108

110. Gneezy + Rustichini 1998, Darrah 2009 A constant source of anxiety 109

111. Empower Parents Davidoff et al. Ubicomp 2007 110

112. Future Work 111

113. Improve algorithms Use algorithms to enable other family coordination systems Add additional sensor inputs Extend to other forms of routine 112

114. 113 Neustaedter + Brush CHI 2006

115. 6pm Orthodontist 114 Neustaedter + Brush CHI 2006

116. 6pm Ortho Paper Route 115 Neustaedter + Brush CHI 2006

117. Baseball 116 Brown et al. Ubicomp 2007

118. Baseball 117 Brown et al. Ubicomp 2007

119. The project on FAMILY CONTROL SMART HOME

120. Thanks for your time, brain power, and support

121. Thanks to our sponsors NSF IIS 1017429 Google Research Award

122. Thanks to our supporters

123. Thanks to my collaborators

124. Thanks to my committee

125. Thanks to my advisors

126. Questions

127. Image Credits http://thesituationist.files.wordpress.com/2008/06/traffic.jpg http://kaleidoscope.cultural-china.com/chinaWH/upload/upfiles/2009-06/22/unique_morning_exercises_a_daily_routine_for_countless_chinese67f17841b1f4bfae458c.jpg http://travellingcam.files.wordpress.com/2006/11/japan-self-defense-force-marching-festival-2006-7.jpg http://thhsmusic.com/Images/2007_Marching_Unit.jpg http://0.tqn.com/d/golosangeles/1/0/-/H/-/-/2007-juggling3.jpg http://popten.net/wp-content/uploads/2009/03/traffic-jam.jpg http://www.lateott.com/images/control_key.jpg http://stuffunemployedpeoplelike.com/wp-content/uploads/2009/05/carpool.jpg http://larchmontgazette.com/commentary/columns/soccermom.jpg http://www.schoolbrief.com/SchoolBrief/images/parent.jpg http://kmcavoy.edublogs.org/files/2010/11/6a00d83451b3c669e200e54f94c4bb8833-500wi-xl20xk.jpg http://www.weshipyourcar.com/LinkPageImages/early-flight.gif http://memory.loc.gov/master/mss/mwright/04/04003/0030d.jpg http://www.mnn.com/sites/default/files/user-60/school%20bus_0.jpg http://arlingtonkids.files.wordpress.com/2011/01/school-bus.jpg http://4.bp.blogspot.com/_K-aTC4UHVR0/S7tDj1UfVSI/AAAAAAAAAB4/QTg9MTxuecY/s1600/confused.jpg http://cdn.sheknows.com/articles/10-Qualities-SM-Art-Img.jpg http://i.dailymail.co.uk/i/pix/2009/10/25/article-0-059F2EC8000005DC-613_468x329.jpg http://static.tvguide.com/MediaBin/Galleries/Editorial/090511/fatguys/fatguys-honeymooners8.jpg http://www.ministryofmanipulation.com/images/8bcube.jpg http://http//vandoverviewpoints.com/wp-content/themes/Yamidoo/yamidoo/scripts/timthumb.php?src=http://vandoverviewpoints.com/wp-content/uploads/2010/04/Family_back.jpg&w=390&h=600&zc=1 http://www.wmich.edu/registrar/assets/images/photos/calendar2.jpg http://www.mediabistro.com/fishbowlDC/files/original/1924_checking_facts.jpg http://www.sbac.edu/~media/images/questionmark.jpg http://img.dailymail.co.uk/i/pix/2008/02_04/textDM2502_468x352.jpg http://farm3.static.flickr.com/2387/2137448527_eb889a39cb.jpg http://www.zoodles.com/blog/wp-content/uploads/2010/08/lost-child4-300x225.jpg 126

128. QUESTION SLIDES 127

129. ROUTINE MODELING RELATED WORK 128

130. FORMULAS 129

131. Driver Distribution 130

132. LWDT 131

133. Bayesian Network 132

134. RIDE SENSING TECHNICAL VERSION 133

135. Davidoff, Dey & Zimmerman CHI 2011 134

136. Two Kinds of Drop-Offs 135

137. Two Kinds of Drop-Offs t3 t2 , t1 , 5:15 5:30 5:45 6:00 5:00 4:45 6:15 Day Care Work P C 136

138. Two Kinds of Drop-Offs t3 t2 , t1 , 5:15 5:30 5:45 6:00 5:00 4:45 6:15 Day Care Work P C 137

139. Two Kinds of Pick-Ups 138

140. Using GPS to Sense a Pick-Up t3 t2 , t1 , 5:15 5:30 5:45 6:00 5:00 4:45 6:15 Day Care Work P C 139

141. Using GPS to Sense a Pick-Up t3 t2 , t1 , 5:15 5:30 5:45 6:00 5:00 4:45 6:15 Day Care Work P C 140

142. Using GPS to Sense a Drop-Off 5:15 5:30 5:45 6:00 5:00 4:45 6:15 Day Care Work 141

143. Using GPS to Sense a Drop-Off 5:15 5:30 5:45 6:00 5:00 4:45 6:15 Day Care Work 142

144. Using GPS to Sense a Drop-Off 5:15 5:30 5:45 6:00 5:00 4:45 6:15 Day Care Work 143

145. Using GPS to Sense a Drop-Off 5:15 5:30 5:45 6:00 5:00 4:45 6:15 Day Care Work P C 144

146. Using GPS to Sense a Drop-Off 5:15 5:30 5:45 6:00 5:00 4:45 6:15 Day Care Work P C 145

147. Using GPS to Sense a Drop-Off 5:15 5:30 5:45 6:00 5:00 4:45 6:15 Day Care Work P C 146

148. Using GPS to Sense a Drop-Off 5:15 5:30 5:45 6:00 5:00 4:45 6:15 Day Care Work P C 147

149. Using GPS to Sense a Drop-Off 5:15 5:30 5:45 6:00 5:00 4:45 6:15 Day Care Work P C 148

150. Using GPS to Sense a Pick-Up 5:15 5:30 5:45 6:00 5:00 4:45 6:15 Day Care Work P C 149

151. Using GPS to Sense a Pick-Up 5:15 5:30 5:45 6:00 5:00 4:45 6:15 Day Care Work P C 150

152. Using GPS to Sense a Pick-Up 5:15 5:30 5:45 6:00 5:00 4:45 6:15 Day Care Work P C 151

153. Using GPS to Sense a Pick-Up 5:15 5:30 5:45 6:00 5:00 4:45 6:15 Day Care Work P C 152

154. Using GPS to Sense a Pick-Up 5:15 5:30 5:45 6:00 5:00 4:45 6:15 Day Care Work P C 153

155. SCHOOL HALF DAY BREAKDOWN 154

156. View of Plan: S16 4:00 5:00 6:00 7:00 3:00 2:00 8:00 School Home Track Practice Dad Work Baseball S16 Dad 155

157. View of Plan: S16 4:00 5:00 6:00 7:00 3:00 2:00 8:00 School Home Track Practice Dad Work Baseball S16 Dad 156

158. Dad Started Baseball League 4:00 5:00 6:00 7:00 3:00 2:00 8:00 School Home Track Practice Dad Work Baseball S16 Dad 157

159. OTHER APPLICATIONS 158

160. Detect conflict with unlabeled routine 4:00 5:00 6:00 7:00 3:00 2:00 8:00 School Home Paper Route Track Practice Dad Work Orthodontist Check-up Scouts S16 Dad 159

161. Dad Started Baseball League 4:00 5:00 6:00 7:00 3:00 2:00 8:00 School Home Track Practice Dad Work Baseball S16 Dad 160

162. ACTIVITY SCATTERPLOT 161

163. RIDE SENSING ERRORS 162

164. Smarter power management GPS Sampling Rate 163

165. Smarter power management Wi-Fi, cell tower, bluetooth Single Location Sensor GPS Sampling Rate 164

166. Simultaneous Departure Errors 4:45 5:15 5:30 5:45 6:00 5:00 6:15 School Home Work Parent Child 165

167. Smarter power management Wi-Fi, cell tower, bluetooth Apply heuristics to modeling Cultural norms, Individual Behavior Single Location Sensor GPS Sampling Rate 166

168. Lost Signal During Daytime 11:00 1:00pm 2:00 3:00 4:00 12:00 5:00 School Home Work Parent Child 167

169. DRIVER PREDICTION ERRORS 168

170. Driver Prediction Performance x Number of Training Weeks of Data 169

171. USER ENACTMENT PICS 170

172. User Enactments Davidoff et al. Ubicomp 2007 171

173. User Enactments Davidoff et al. Ubicomp 2007 172

174. UNUSED PICS 173

175. Koestler 1967 Routines are not documented 174

176. Darrah et al. 2000, Frissen 2000, Beech et al. 2004 People can’t see all the facts 175

177. Davidoff et al. Ubicomp 2006 Managing Rides Causes Anxiety 176

178. Pentland + Reuter 1994 Graceful 177

179. Wolin & Bennett 1984 Path for exercise 178

180. Medved 2004 Confident 179

181. HCI Approach to Machine Learning 180

182. Dual-income families 181

183. Frissen 2000, Fiese et al. 2002, Davidoff et al. 2007 Routines help parents feel in control 182

184. Frissen 2000, Fiese et al. 2002, Davidoff et al. 2007 Just surviving the day 183

185. Darrah et al. 2000, Frissen 2000, Beech et al. 2004 Non-Routine Events Cause Anxiety 184

186. Gneezy + Rustichini 1998, Darrah 2009 A constant source of anxiety 185

187. Gneezy + Rustichini 1998, Darrah 2009 A constant source of anxiety 186

188. How do routines provide support? Davidoff et al. Ubicomp 2007 187

189. LONG-FORM APPROACH 188

190. We can use fieldwork to identify missing but needed information resources Fieldwork 1 189

191. We can use sensing and modeling to synthesize missing information resources Modeling 2 190

192. We can develop an application to deliver synthesized resources and evaluate how it helps coordination Show how to use the model and validate that the application potentially solves the problem by givengfpeople the information they want Build and validation 3 191

193. We can exploit routine as a resource to design learning systems Thesis Statement 192

194. Understanding how routines support daily life generate new design ideas Generative Resource 193

195. Model information about family transportation routines to supplement family knowledge Information Resource 194

196. We can exploit routine as a resource to design learning systems Thesis Statement 195

197. context-aware, adaptive, intelligent, cognitive, expert, decision-support system 196