RoutineMaker: Towards End-User Automation of Daily Routines Using Smartphones


Published on

People use smartphones in daily activities for accessing and storing information in various situations. In this paper, we present a work in progress for detecting and automating some of these activities. To explore the possible patterns we developed an experimental application to detect daily tasks used by smartphones and analyzed it to provide suggestions for “routines”. We conducted a two-week user study with 10 users to evaluate the approach. During the study the application logged the usage patterns, sent information to the server where it was analysed and clustered. The participants could also automate their smartphone tasks using the analysed data. The findings suggest that people would be willing to automatize tasks given that the approach gives flexibility and expressiveness without too much information overload. Future work includes refining the algorithms based on the gathered real-life data and modifying the interaction design to approach the challenges found with the initial study.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

RoutineMaker: Towards End-User Automation of Daily Routines Using Smartphones

  1. 1. RoutineMaker: Towards End-User Automation of Daily Routines Using Smartphones Ville Antila, Jussi Polet, Arttu Lämsä, Jussi Liikka Context-Awareness and Service Interaction VTT Technical Research Centre of Finland Oulu, Finland {ville.antila, jussi.polet, arttu.lamsa, jussi.liikka}@vtt.fiAbstract — People use smartphones in daily activities for the context or situation of the user, maybe even better thanaccessing and storing information in various situations. In this more traditional and quantifiable sensors of context can.paper, we present a work in progress for detecting and In this paper, we study the possibilities of detecting andautomating some of these activities. To explore the possible automating smartphone usage routines. With a routine wepatterns we developed an experimental application to detect mean an association of a location, used application and thedaily tasks used by smartphones and analyzed it to provide time of day. To reveal some of these somewhat hiddensuggestions for “routines”. We conducted a two-week user patterns, we developed an application to detect the day-to-study with 10 users to evaluate the approach. During the study day smartphone use by logging the application usage andthe application logged the usage patterns, sent information to locations and clustering them to identifiable patterns. Wethe server where it was analysed and clustered. Theparticipants could also automate their smartphone tasks using also implemented a functionality to automate these patternsthe analysed data. The findings suggest that people would be using the application. One reason for this functionality waswilling to automatize tasks given that the approach gives to find out whether the users could actually detect someflexibility and expressiveness without too much information routine-like behaviour from their smartphone usage patterns;overload. Future work includes refining the algorithms based which would then help us to evaluate our approachon the gathered real-life data and modifying the interaction to approach the challenges found with the initial study. II. RELATED WORK Keywords - Context-awareness; Routine detection; Sensing; The idea of extracting usage patterns and routines fromSmartphones; Task automation; smartphone usage data is not unique or novel as such. There has been a body of research exploring different quantitative I. INTRODUCTION methods to mine patterns of human activities from large datasets. Eagle and Pentland demonstrate the ability to use Smartphones are becoming ubiquitous and ever more mobile devices to recognise social patterns, identifyimportant for the daily activities of their users. The multitude significant locations, and model organisational rhythms [4].of smartphone applications, dedicated to help in daily tasks, Farrahi and Gatica-Perez suggest that human interactionare used almost everywhere at any time. Smartphones and data, or human proximity, obtained by mobile phonetheir applications, serving as pocket PCs and extending our Bluetooth sensor data, can be integrated with human locationdesktop experience, are becoming so ubiquitous part of our data, obtained by mobile cell tower connections, to mineways to store and access information that some of the tasks meaningful details about human activities from large andwe perform with them have become daily routines. Examples noisy datasets [6]. They also present a framework to classifyof routine-like behaviour can include checking e-mail in the people’s daily routines (defined by day type and by groupmorning, reading the news or listening to music while affiliation type) from the data [7]. Similarly Verkasalocommuting, searching local information, navigating or illustrates the relationships between common locations, suchchecking-in to places to assess and comment our on-the-go as office or home, to the usage patterns of differentexperiences. People also use smartphones to complement applications [10]. In our work we are concentrating on real-other daily activities or routines such as watching TV, time analysis and presentation of routines of an individualreading newspaper and going to the grocery store [8]. user rather than modelling the group behaviour. We are also On the other hand, the latest consumer studies indicate looking into qualitatively evaluating the found patterns bythat the emerging user patterns could be more application- the user (by the act of saving or modifying the routine).specific than they are device-specific [5]. The routine of Chittaranjan et al. investigate the relationship between“checking Facebook in the evening from bed” could be done behavioural characteristics derived from rich smartphoneeither with a smartphone, laptop or a tablet device. The data and self-reported personality traits [1]. The data stemsaction or behaviour is often associated to a specific service or from smartphones of a set of 83 individuals collected over aapplication it is done with more than the device mediating continuous period of 8 months. From the analysis, they showthe experience. Therefore we can hypothesize that the usage that aggregated features obtained from smartphone usageof a specific application can also indicate something about data can be indicators of the Big-Five personality traits.
  2. 2. Additionally, they present an automatic method to infer the automated routine out of it. The saved routine is then sent topersonality type of a user based on cell phone usage with up the server as well for persistent storage and further 75.9% accuracy. This work gives an interesting insightinto how the collected behavioural data can be related to B. Mobile Applicationknown personality traits, and as a concept could be applied The RoutineMaker mobile application visualises thein our research as well in the future. detected routines (see Figure 1) by showing the location In addition to detecting the routines using smartphones, clusters on a map view as well as on a list view. Thethere has been research on how to present it to the user for a MapView shows the location cluster markers, by whichpotential user action. Dearman et al. present an approach to tapping the user can see a preview list of the most usedpresent information to the user based on the location and applications in that cluster. The user can also switch from theknowledge of the task. Examples include location-based task MapView to the ListView, which shows an in-depth list ofnotifications and support for opportunistically suggesting the location clusters. In the ListView, the user can selectplaces for certain activities on-the-go [2, 3]. While these desired applications to be launched at specific times and savestudies have similar goals than our approach, the intended the sequence as a routine. The RoutineMaker mobileusage situations are somewhat different; nevertheless we application checks frequently, if there are any routines to runthink that the application presented in this paper could and if so, it checks whether the current location and time isbenefit from introducing some form of serendipitous or associated with any routines. Should there be a match; theopportunistic presentation of data to the user regarding the specified routine is run automatically.routines. We also acknowledge in our work that the breadth ofanalysis done with the data can also be potentially misused.Shilton discusses the privacy of collecting multi-dimensionalsensor data from mobile phones [9]. As by usingsmartphones it is possible to gather an extensive set ofinformation about people’s locations, habits and routines,even personality traits, it might be that smartphones at theextreme could be the most widespread embeddedsurveillance tools in the history. The trade-off for the user isbetween the perceived benefit and privacy concerns, and wesee that this trade-off should be balanced by the user via heractions using the system (explicitly sharing what is neededand wanted to be shared). III. ROUTINEMAKER APPLICATION In this section we present the developed application fordetecting and automating daily routines with smartphones.The application logs daily smartphone usage data (locations, Figure 1. Cluster overview shown in the MapView andtime and used applications) and tries to detect patterns, such cluster details shown in the ListViewas a sequence of applications used or tasks done on a certain C. Servertime at a certain location. Once the possible routines are The server-side application is responsible for creating thedetected, the application displays them to the user. The user application-location clusters from the logged data receivedcan accept and create a “routine” from the suggested from the client devices. The algorithm is split into two mainpatterns or modify the suggested pattern and then save it. phases: geographical and application clustering. The stepsThe user can also name the routines in similar way than one are illustrated in Figure 2.would do with ordinary applications (e.g. “going to sleep”- First step of the process is the geographical clusteringroutine, “going to movies”-routine, “going to work”- which filters out the most significant locations from the dataroutine). (visited or stayed most often). After the geographicalA. Software Design and Implementation clustering is done, an application table is generated, where The prototype consists of a mobile application, which each column represents a five-minute time slot in a day and acollects usage data and presents the processed usage data to row is generated for each application. Then the clusterthe user and a server-side application, which performs the samples are gone through and the value of the table elementdata processing. The mobile application gathers usage data representing the time-application combination of the sample(location and applications used) from the device. This data is is increased by one. After this, the whole table is normalisedsent to the server and analysed to find location clusters and by dividing it by the maximum element of the table. A usageused applications in those clusters. The mobile client table, containing Boolean values, is generated from thepresents this analysed data to the user. If the user notices application table. The usage table is the same size as thehelpful or useful routines from the data she can create an application table and the elements contain value true, if the application table value in this element is greater than a
  3. 3. threshold value, otherwise the elements contain value false. Table 3 Research questionsThis is followed by applying a smoothing filter to the usage ID Questiontable. This removes false slots that are located in between RQ-1 Is it possible to extract routines or tasks from the historicaltwo true elements in the usage table. usage data? RQ-2 Were the extracted and suggested routines helpful? RQ-3 [Following from the RQ-2] Could they be useful? Geographic clustering Application clustering RQ-4 [Following from the RQ-2] Did they reveal any other possibly interesting or important information? Add samples to clusters Generate table of active applications ordered by time A. Participants Filter out clusters with not Normalize table and get We recruited ten participants from three countries using enough samples application usage times to e-mail lists. There were nine male participants and one usage table female. The participants had to be active smartphone users. Combine clusters close to The participants also had to have suitable mobile phones each other Apply smoothing filter to supported by the application (Android v2.2 or higher). The usage table participants were in the age range of 27 to 33 years with Filter out samples far away average age of 29.7 and were very active smartphone users, from cluster centers Get application launch times as 62% of them used smartphone applications a couple of from usage table times a day and 25% used them a couple of times in an hour. Figure 2. Algorithm structure (repeated for each user) B. Findings In this section, we provide a brief analysis of the gathered The application launch times are then read from the data. The sources for the gathered data are the initialusage table. Always when an element containing the value questionnaire, the logged data from the user study, the posttrue is found proceeded by false; an application launch time questionnaire and open ended questions the users were askedis detected. Table 1 and Table 2 contain an example of the in the end of the study.application and usage tables. The generated application table 1) Perceived usefulness of routine detection and theis shown in the Table 1. The usage table shown in the Table2 is obtained by using a threshold value of 0.7. In this RoutineMaker applicationexample, two launch times are detected; 13:05 for “Music First, we asked how useful the participants ratedplayer” and 13:20 for “E-mail”. detecting their smartphone usage routines. The results show that this was considered as useful (avg. 3.7, sd. 0.8, on a Table 1 Application table scale from 1 to 5). We also asked how useful they perceivedApplication Time the RoutineMaker application as such. The results showed 13:00 13:05 13:10 13:15 13:20 13:25 13:30 that the approach was not perceived as very useful (avg. 2.1,Music player 0 0.8 0.74 0.8 0.4 0 0 sd. 0.9). The reason for this was visible in the comments:Web browser 0 0 0 0 0 0 0 First of all, the application could only automatically launchE-mail 0 0 0 0 0.9 1 0 applications when they were at a certain location at a certainNotebook 0 0 0 0.3 0.1 0.2 0 time. What the users wanted was even more automatic behaviour, such as performing a certain task on its own Table 2 Usage table without any user intervention. With the current design, suchApplication Time 13:00 13:05 13:10 13:15 13:20 13:25 13:30 elaborate tasks were impossible to make with the application.Music player false true true true false false false This lowered the perceived usefulness. Nevertheless, theseWeb browser false false false false false false false comments give good insight into developing the applicationE-mail false false false false true true false further.Notebook false false false false false false false 2) Understanding the important factors of smartphone routine detection IV. USER STUDY To get a better insight into the design space of The RoutineMaker application was evaluated with ten smartphone routine detection, we asked the participants tousers, who used the application for two weeks. During the rate 4 different factors or parameters of the routine detectiontwo weeks, the application logged the routines of the user, on a scale from 1 to 5. These parameters were: quality ofsent this information to the server, where it was analysed and detected patterns, amount of detected patterns, resolution ofclustered. The participants could automate their smartphone detected patterns and the accuracy of detection. Based on thetasks using the analysed data. In the study, we included a results, the most important factor was accuracy (avg. 4.3, sd.start and end questionnaires and a set of open-ended 0.76), while the amount of detected patterns was rated asquestions to probe the participants about their needs and least important (avg. 2.9, sd. 0.9). Resolution and qualityexperiences related to the application concept and the actual were rated as somewhat important factors (avg. 3.6, sd. 0.79usage of the prototype application. The research questions and avg. 3.3, sd. 1.1). The standard deviation was large infor the user study are listed below in Table 3 Research quality ratings; taking a closer look at the results it seemsquestions). that some participants did think that quality was important
  4. 4. whereas some did not rate it as important. It is possible that still offering suggestions based on the detected behaviour,quality as a measurement was not very well understood in we can enable an easy and fast interface for users tothis context, or that it was already incorporated in the other customise and automate their routine-like behaviour withoutratings. limiting only to the specific smartphone tasks. We also asked how well the RoutineMaker application The lessons learned from the application developmentperformed regarding the selected factors (quality, amount, and the user study include that the amount of detectedresolution and accuracy). In general, the application clusters (potential routines) can be quite high, thereforeperformance was in line with the importance of the factors. leaving the selection and creation of routines more to theThe accuracy of routine detections was rated good (avg. 3.5, user. Nevertheless the suggestions should include onlysd. 0.84), as well as the resolution of the detected routines relevant applications, which are detected usually during the(avg. 3.25, sd. 1.17). These were rated as most important same times during the day, in a routine-like manner.factors, so we can conclude that some of the parameters The future work consists in developing the applicationselected for the routine detection algorithms were and algorithms further using the data gathered from thecorresponding to what the participants thought as important study. In some cases the algorithm for detecting the routinesor useful. was “too adaptive” and some clusters were removed if the The amount of detected routines was rated the worst of user had an unordinary day during the week. We are seekingthese factors (avg. 2.25, sd. 1.05). This can be due to a to tweak the threshold for the adaption and include betterrelatively large number of false positive detections of fitness values for the detected, possible routines, andapplications due to the features of the underlying OS weighting them in the algorithm enabling the process to learn(Android), which opportunistically leaves applications which kind of application sequences are perceived as usefulrunning in the background to increase user experience (such routines. We are also looking into doing more user studiesas the response times of applications). This issue can be with larger group of participants learning more about thefixed by filtering out processes which the user is not actively user behaviours and surveying the routines people currentlyusing. Nevertheless, we can also hypothesise that leaving have while using smartphones.some of these applications to be selectable can create certainserendipity in creating the routines and allows users to create REFERENCESroutines that necessarily were not detected as such, but could [1] Chittaranjan, G., Blom, J. & Gatica-Perez, D., Who’s Who with Big-be useful in the future (some of the processes are useful for Five: Analyzing and Classifying Personality Traits with Smartphones.tasks even when they are not active on the UI). Proceedings of the International Symposium of Wearable Computing (ISWC2011), IEEE Computer Society, 2011. V. DISCUSSION AND FUTURE WORK [2] Dearman, D., Sohn, T. & Truong, K. N. Opportunities Exist: Continuous Discovery of Places to Perform Activities. Proceedings of After the user study, we asked the participants whether the 2011 Annual Conference on Human Factors in Computingthe application usage revealed any surprising facts about the Systems, ACM, 2011.people’s smartphone usage. Many participants answered that [3] Dearman, D. & Truong, K. N., Identifying the Activities Supportedthe application did reveal routines, but these were largely by Locations with Community-authored Content, Proceedings of the 12th ACM International Conference on Ubiquitous Computingknown and thus not really surprising as such. In some cases (UbiComp10), ACM, 2010.the participants were surprised of the prevalence of these [4] Eagle, N. & Pentland, A., Reality Mining: Sensing Complex Socialroutines. The detected and “accepted” routines were, for Systems, Personal and Ubiquitous Computing, Vol. 10, No. 4, pp.example, checking e-mail in the morning at home, or 255-268, 2006.checking-in to Foursquare at certain locations at certain [5] Ericsson Inc., From Apps To Everyday Situations - An Ericsson(rather fixed) times. Some routines included using sports Consumer Insight Summary, 2011, Available:tracking when going for a jog or bicycling. Some routines r.pdf.included a set of detected applications, like alarm clock andemail client in a sequence. [6] Farrahi, K. & Gatica-Perez, D., Probabilistic Mining of Socio- Geographic Routines from Mobile Phone Data, Selected Topics in We can conclude based on the study that we found Signal Processing, IEEE Journal, Vol. 4, No. 4, pp. 746-755, 2010.distinct usage patterns which can be potentially automated. [7] Farrahi, K. & Gatica-Perez, D., Daily Routine Classification fromIn addition, the users were interested in creating automated Mobile Phone Data, Machine Learning for Multimodal Interaction,routines based on their smartphone usage behaviour, but in 2008.many cases these routines were more elaborate and complex [8] Google Inc., The Mobile Movement - Understanding Smartphonethan our application could offer. Examples include time and Users, 2011, (last updated on 26th April 2011). Available:location-based triggers with variable sequences, such as study-shows-mobile.html.“change my profile to silent when entering certain location [9] Shilton, K., Four Billion Little Brothers? Privacy, Mobile Phones andbetween certain times”. Going further, we can envision Ubiquitous Data Collection, Communications of the ACM, Vol. 52,location and time dependent notifications like to-dos or No. 11, pp. 48-53, 2009.calendar entries or tasks that could be notified when the [10] Verkasalo, H., Contextual Patterns in Mobile Service Usage.device senses idleness (suggesting upcoming tasks while Personal and Ubiquitous Computing, Vol. 13, No. 5, pp. 331-342,sitting in a bus for example). We argue that by giving the 2009.user control over how these tasks could be performed, while