From Context-Awareness toHuman Behavior PatternsDetection of Daily Routines using SmartphonesVille Antila (Research Scientist, M.Sc.)VTT Technical Research Centre of Finland, Oulu, FinlandPhilips Research, Eindhoven, The Netherlands (Visiting researcher)
Background – Smarcos project• Smarcos creates solutions to allow devices and services to exchange context information, user actions, and semantic data• One important part of the work has been to investigate the practical usage of context information and to develop models that can be dynamic and adaptive as well as applicable to different applications• www.smarcos-project.eu
Outline of the talk• Introduction• From context logging to routine detection • Continuos, low-power “life-logging” • Interpreting the data (what’s the meaning of it) • Using domain knowledge to reason about what we don’t know• Example applications• Discussion and summary• Video?
Introduction - context awareness• Idea that computers can both sense and react to their environment • “any information that can be used to characterize the situation of an entity” [Dey et al., 2001] • Human factors: information about the user, social environment, user’s task • Physical factors: Location (absolute, relative, co-location), infrastructure, physical conditions• Context aware systems should be able to gather context (sense the situation), abstract and understand it and adapt application behavior based on the context• Some classical use cases: • adapt user interfaces • tailor application-relevant data (e.g. ﬁltering information) • increase precision of information retrieval • make the user interaction implicit • discover services & build smart environments
Introduction - context awareness• Smartphone is the epitome of a sensing platform for context-awareness • Personal and mobile (almost always with the user) • Comes with a lot of in-built sensors and communication capabilities • Used everywhere, for multiple tasks (on-the-go)
Introduction - challenges & opportunities• The notion of ‘context’ can not be objectively deﬁned (a prior) by settings, actions and actors• Rather, context is the meaning that the actions and actors acquire at any given time from the subjective perspective [Mancini et al., 2009]• Context information can have multiple purposes (from user’s perspective) • “Declaring one’s position is perhaps as much about deixis (pointing at and referencing features of the environment) as it is about telling someone exactly where you are” [Benford et al., 2004]• When going from individuals to larger groups of people it’s possible to extract patterns from the data [Eagle and Pentland, 2006] • social patterns (e.g. urban heat maps) • identiﬁcation signiﬁcant locations • model organizational rhythms
From Context Logging to Routine Detection Applications • Example applications... Support for decision • Use the gathered knowledge to form decisions about system behavior in making different contexts/situations Domain • Infer new knowledge from information Reasoning knowledge • Transform raw data into information Context Interpreter about the user’s context Data Layer • Capture raw data from device sensors
Data Layer Continuous Life Logging• Context-awareness • Using smartphones as sensors for human activities (e.g. important locations, mobility patterns)• Low-power context logging software • Semantic location detection using cell-id (low power, always available) • Device usage detection (algorithms for mining location relative to smartphone application usage) • Lo-ﬁ physical activity detection (e.g. is the user moving currently) • Scanning Bluetooth snapshots to determine indoor environment (e.g. is the user at his ofﬁce desk)
Context ComponentsFor real-time user behavior detection• Collection of software components for enabling • continuous context logging • development of context-based adaptation for variety of applications• Implementations available for several platforms (some of which are becoming obsolete) • Android, Symbian, Maemo/Meego/Linux, BlackBerry
Context InterpreterInterpreting the data...• Estimation of life patterns such as the semantic location of the user (e.g. “home”, “ofﬁce”)• Detection of device usage in different locations• Detection of physical activity in different situations• Detection of changes in routines
Context CaptureContext-based awareness cues in information sharing• We explored the usage of contextual information cues in informal information sharing• The study focused on practices of ‘abstraction’ when publicly sharing contextual information• Field test for our backend
Smartphones are used almost everywhere...Moreover there is an “app” for almost anything...An opportunity (to use smartphoneapps as sensors for situations...)? Image sources: http://adage.com/article/digital/placing-ads-underestimate-mobile/230853/ http://www.resultrix.com/blog/index.php/tag/tablet-marketing http://www.google.com/googleblogs/pdfs/mobile_understanding_smartphone_users.pdf
Routine MakerEnd-user automation of smartphone routines• An approach to detect day-to-day ‘routines’ by logging the smartphone application usage and locations where they are used• Analysis of logged usage data into identiﬁable patterns (clustering based on location and time of use) • Implemented an experimental smartphone application with a functionality to create automated ‘tasks’ out of the identiﬁed patterns • Conducted a two-week user study to analyze the approach and to gather user feedback
Domain knowledge ReasoningExtending the knowledge...• By modeling domain knowledge we can reason about the consequences of what the derived context information means in the particular application scenario• For example, if know (to a certain degree of accuracy) that the user is cycling, then we can reason that: • available devices are mobile devices (phone & activity monitor) • availability for receiving messages is low
Where would this information be useful?• Determine the devices that surround the user • e.g. at work, the user has access to his personal computer• Time and adapt system feedback based on the situation • e.g. time-shift notiﬁcation to where user is more receptive• Log important events and use those to automate tasks • e.g. migrating task sessions automatically between personal devices
ApplicationsExample application:Context-adaptive Feedback• Goal: increase effectiveness and decrease interruptions• How: adaptive selection of device, modality and timing of feedback• Example of adaptive feedback delivery: • IF the situation is suitable, THEN send the message (as it is) • IF the situation is not suitable AND the message is not urgent, it should be time-shifted • IF the situation is not suitable AND the message is urgent, then the content should be adapted to the situation
ApplicationsExample application:Context-based User Interface Migration• Migration concept • Interface moving from source device to target device • Interaction state persistence • Interaction continuity
ApplicationsExample application:Context-based User Interface Migration• Taking advantage of the known properties of the environment in any given time (e.g. Bluetooth, GPS, device stability and orientation) we can automate tasks such as UI migration triggering
Discussion• Just like the smartphone sensor APIs have matured (GPS, acc, gyro, proximity), the basic context abstractions will also be served as OS- level services in the future (e.g. walking/running/still, home/ofﬁce/ school/on-the-go) • Better optimizations regarding battery, CPU and memory usage etc. (resolves the iOS background processing challenge) • Cross application usage • Fusion with other available data on the device and “in the cloud”• Going further we can also foresee taking inputs from the environment (e.g. WSNs) as well as “negotiating with” other smart devices while trying to reach a better approximation of the situation
Discussion - challenges• Quality/accuracy of detection (have to be started off with simple cases), provenance of quality measures into the application level• User interaction/awareness of application behavior• Designing for applications that adapt to situations... Patterns, guidelines, best practices?• Prototyping context-adaptive applications (from early interactive prototypes to functional prototypes)• Testing • Functional testing, performance/quality testing • User testing (‘in the wild’)
From Context-Awareness to Human Behavior PatternsDetection of Daily Routines Using SmartphonesThank you!Questions? Ville Antila ville.antila@vtt.ﬁ
A particular slide catching your eye?
Clipping is a handy way to collect important slides you want to go back to later.