MuMe Slide M. Wolpers 18 Nov

Context in mobile applicationsHow to achieve context sensitivity in mobile applications. Martin Wolpers© Fraunhofer-Institut für Angewandte Informationstechnik FIT

Agenda Introduction of context Sensors in mobile devices Conclusions based directly on sensor data Aggregating sensor data to derive conclusions Advanced sensor data processing to create higher order conclusions© Fraunhofer-Institut für Angewandte Informationstechnik FIT

Introduction of context Any ideas?© Fraunhofer-Institut für Angewandte Informationstechnik FIT

Context awareness:the essence of adaptabilityContext awareness Resource awareness  Adapt to available resources (connectivity, nearby devices Situation awareness  Adapt to the situation (mode, location, time, event) Intention awareness (?)  Adapt to what the user wants to doContext awareness is found in humans We always adapt our behavior and actions according to the context (i.e. situation) Pervasive computing devices that ubiquitously accompany humans (such as smartphones) must adapt accordingly  Or risk being disruptive and annoying  Taken from lecture slides CSE494/598© Fraunhofer-Institut für Angewandte Informationstechnik FIT

Defining Context • Dictionary definition • “the interrelated conditions in which something exists or occurs” • Definition for pervasive computingapplications. In First International Workshop on Mobile computing SystemsSchilit, B., Adams, N. And Want, T.R. (1994), Context-aware computing • “any parameters that the application needs to perform a task without being explicitly given by the user” One definition [Schilit et-al. 1994]: Another definition [Abowd & Mynatt]:  Computing context: connectivity,  Social context: user identity and communication cost, bandwidth, human partner identities nearby resources (printers,  Functional context: what is being displays, PCs)… done, what needs to be done  User context: user profile, location,  Location context: where it is nearby people, social situation, happening activity, mood …and Applications, pp. 85-90  Temporal context: when it is  Physical context: temperature, happening lighting, noise, traffic conditions …  Motivation context: why it is  Temporal context (time of day, happening (purpose) week, month, year…) GREGORY D. ABOWD and ELIZABETH D. MYNATT (2000). Charting  Context history can also be useful Past, Present, and Future Research in Ubiquitous Computing. ACM Transactions on Computer-Human Interaction, Vol. 7, No. 1, March 2000, Pages 29–58. © Fraunhofer-Institut für Angewandte Informationstechnik FIT

An operational context definition Definition: Context is any information that can be used to characterise the situation of an entity (Dey, 2001). Elements used for the description of context information fall into five categories: individuality, activity, location, time, relations The activity predominantly determines the relevancy of other context information in specific situations. Location and time primarily drive the establishing of relations to other entities enabling the exchange of context information among entities.Based on Zimmermann et.al. 2007, Proceedings of Context 2007© Fraunhofer-Institut für Angewandte Informationstechnik FIT

Elements of context© Fraunhofer-Institut für Angewandte Informationstechnik FIT

Context Information: Individuality captures contextual information strongly related to the entity several types of entities possible:  active and passive  real and virtual  mobile, movable, stationary  human, natural, artificial, group entities© Fraunhofer-Institut für Angewandte Informationstechnik FIT

Context Information: Time covers temporal information related to the entity current time  alternative representations  overlay models time intervals recurring events process-oriented view historical context information  access past contextual information  analyse past contextual information© Fraunhofer-Institut für Angewandte Informationstechnik FIT

Context Information: Location covers spatial information related to the entity physical or virtual absolute or relative quantitative (geometric) and qualitative (symbolic) representations overlay models one entity possesses  one physical quantitative location  several different qualitative locations  several different virtual locations© Fraunhofer-Institut für Angewandte Informationstechnik FIT

Context Information: Activity covers information about activities the entity is involved in described by goals, tasks and actions tasks are goal-oriented activities and small, executable units task models structure task into subtask hierarchies goals potentially change very frequently low-level and high-level goals determines the relevancy of other contextual information© Fraunhofer-Institut für Angewandte Informationstechnik FIT

Context Information: Relations covers information about relations the entity has established to other entities expresses semantic dependencies between two entities spatio-temporal coordinates of two entities are key-driver several relations can be established to the same entity each entity plays a specific role in a relation static and dynamic relations several types of relations:  social relations  functional relations  compositional relations© Fraunhofer-Institut für Angewandte Informationstechnik FIT

Context (cont’d) SensorOther classifications of context: data  Low-level vs High-level Low-level context context relations  Active vs Passive context individual activityPutting it all together location  Gather low-level context time  Process and generate high- Context high-level context level context processing  Separate active from passive active passive context context context  Adjust Context-aware application© Fraunhofer-Institut für Angewandte Informationstechnik FIT

Context-Aware Application Design How to take advantage of this context information? Schilit’s classification of CA applications: 1. Proximate selection: 1. closely related objects & actions are emphasized/made easier to choose 2. Automatic contextual reconfiguration: adding/removing components or changing relationships between components based on context 1. Switch to a different operation mode 2. Enable or disable functionality 3. Context-triggered actions: rules to specify how the system should adapt 3. Contextual information and commands: produce different results according to the context in which they are issued 1. Narrow-down the output to the user using the context 2. Broaden the output to the user using the context© Fraunhofer-Institut für Angewandte Informationstechnik FIT

Problems with processing sensor data From Junehwa Song. Mobile and Sensor OS. MobiSys 2008/TMC 2010/PerCom 2010© Fraunhofer-Institut für Angewandte Informationstechnik FIT

The usual approachRequires costly operations for Continuous data updates from sensors Continuous context processing  Complex feature extraction and context recognition Continuous change detection  Repeated examination of numerous monitoring requests From Junehwa Song. Mobile and Sensor OS. MobiSys 2008/TMC 2010/PerCom 2010© Fraunhofer-Institut für Angewandte Informationstechnik FIT

Introducing feedback loops Early detection of context changes  Remove processing cost for continuous context recognition Utilize the locality of feature data in change detection  Reduce processing cost by evaluating queries in an incremental manner Turn off unnecessary sensors for monitoring results  Reduce energy consumption for wireless data transmission From Junehwa Song. Mobile and Sensor OS. MobiSys 2008/TMC 2010/PerCom 2010© Fraunhofer-Institut für Angewandte Informationstechnik FIT

Sensors in mobile devices  Touch screen  Navigation  Several accelerometers  Browser history  Gyroscope  Social networks  GPS  Calendar  Wifi  Contacts  Microphone  Address resolver  Camera  Music player  Bluetooth  Light  Telephone (Call, SMS)© Fraunhofer-Institut für Angewandte Informationstechnik FIT

Conclusions based directly on sensor dataSensor data generate first level observation data.ExamplesAccelerometer  indication that someone might be movingLocalization + Accelerometer  track of movement activityLocalization + Time  indication that someone might be movingLocalization + Feedback button  someone confirms an activity (e.g. app asks the student to state that he attended a course after attending the course)Time + Lightsensor  indication that someone might be outsideReal world examplesLocation + Accelerometer + Time  Wake up timerLocation + Time + Calendar  Silence mobile phone, e.g. Tasker© Fraunhofer-Institut für Angewandte Informationstechnik FIT

Aggregating sensor data to derive conclusionsCombine sensor data to derive second level observation data.Examples: Location + Contacts + Bluetooth log  Buddies near you; Buddy phone status Location + Calendar + Time + Sound  Identify if in a conversation Location + Accelerometers  Identify if someone is moving indoors and outdoors Time + Location + SMS activity  Identify if someone is waiting for someone elseReal world examples: ContextPhone VibN CenceMe Physical Activity measurement Time tracking: How do figure out if a task is completed.© Fraunhofer-Institut für Angewandte Informationstechnik FIT

The ContextPhone framework http://www.cs.helsinki.fi/group/context/(from 2004/2005: runs on Symbian OS 6 and 7 – Really old -- now part of Google Jaiku http://www.jaiku.com/ )Already then, most of today’s ideas have been addressed,e.g. using bluetooth connections to determine how busy an environment is.OrAccess to status of friends mobile phone: Friends Phone My Phone© Fraunhofer-Institut für Angewandte Informationstechnik FIT

VibN http://sensorlab.cs.dartmouth.edu/vibn/ http://www.youtube.com/watch?v=U37G6uzTu5k Sound with iOS Sound with AndroidUsing the microphone to collect environment information Sound with HTML5 (carefull, some problems)Tagging of places with audio and statistics of people present(To ensure privacy, voices are removed from the recording.)  Points of Interest identified by sound recording and time of stay Uses microphone, localization and accelerometers  Note that accelerometer shut down on iOS if app is in background (not so on Android) Good paper showing implementation at http://sensorlab.cs.dartmouth.edu/pubs/sc i906e-miluzzo.pdf© Fraunhofer-Institut für Angewandte Informationstechnik FIT

CenceMe – sensing and sharing presence http://metrosense.cs.dartmouth.edu/projects.html#cenceme http://cenceme.org/ http://www.youtube.com/watch?v=8rDFbTF47PASensing presence captures a user’s status in terms of his activity (e.g., sitting, walking, meeting friends), disposition (e.g., happy, sad, doing OK), habits (e.g., at the gym, coffee shop today, at work) and surroundings (e.g., noisy, hot, bright, high ozone). iPhone access to calendar Android access to calendarUse of sensors: Accelerometers identify activity of user (sit, run, walk, etc.). Microphone identifies conversation, quite place, loud location, etc. Localization delivers web-based additional info like weather, etc. Access to contacts and calendar provides indications of with whom you are in a conversation.© Fraunhofer-Institut für Angewandte Informationstechnik FIT

Example problem:Physical Activity Measurementusing the iPhoneTask: identify the physical activity in terms of standing, sitting, walking, jogging,moving upstairs and downstairsSensor: Accelerometer in mobile device at different placesProblem: Place where mobile device is on the body is unclearSolution: Best place is the waist. If not possible, use transiton tables from research, e.g. Jennifer R. Kwapisz, Gary M. Weiss, Samuel A. Moore. Activity Recognition using Cell Phone Accelerometers. SensorKDD ’10, July 25, 2010, Washington, DC, USA. Yuichi Fujiki. iPhone as a Physical Activity Measurement Platform. CHI 2010, April 10–15, 2010, Atlanta, Georgia, USA. Accelerometer on the iPhone Accelerometer on Android© Fraunhofer-Institut für Angewandte Informationstechnik FIT

Time tracking – How to...Solution 1: Ask the worker.Solution 2: (Semi-) Automatic detection (one possible solution) Identify starting and ending events/activities of tasks or assignments Ask user to press button when starting a task Ask user to define task in terms of sensor input (change of location, result sent, stop button pressed, participating partners, collaboration events, etc.) Integration with Calendar to ensure pausing at unrelated events Integrate with Telephone and Mic and Calendar to identify F2F collaboration is ongoing Integrate with SMS to detect asynchronous collaboration ... Use facebook timeline upload/store data and to visualize activities© Fraunhofer-Institut für Angewandte Informationstechnik FIT

Advanced sensor data processing to create higher orderconclusions A E D C Emoticon analysis Learning resource context C F E C G Basic learning analytics© Fraunhofer-Institut für Angewandte Informationstechnik FIT

Emoticon Analysis – Goals and IdeaDetecting positive sentiments from computer mediated communication (CMC) between chat partners to qualify the degree of positivity in a relationship Positive emoticons in CMC do convey positivity and respective emotions Take emoticons as a substitute for non-verbal communication. Disregard all verbal information -> ease and speed of processingQuestion: Does positivity as calculated by emoticon extraction correlate with sympathy?© Fraunhofer-Institut für Angewandte Informationstechnik FIT

Emoticon Analysis: Experimental Setup & IndicatorsExtract chats from Skype for test users. Anonymize contacts and user information and store emoticon parameters on central DBCalculated Positivity value:  = Positive Emoticon Quotient  = Global Emoticon Quotient  = Emoticon Mimicry Quotient PEQ relates to positive emoticons per chat session to all chat sessions. GEQ relates to emoticon usages per chat session to all chat sessions. Mimicry rate grabs the amount of mimiced emoticons between chat partners. Scalar weight vector (G) open for modification.© Fraunhofer-Institut für Angewandte Informationstechnik FIT

Emoticon Analysis: Evaluation & ResultsQuestionnaires for participants (N=6) Top ten ranking of skype contacts with pseudonyms to guarantee anonymousity Build pairs of partners to detect differences in relationship interpretationResults Calculated top ten ranking of algorithm includes 50% of the most sympathetic Skype contacts Pairing leads to very interesting results showing emoticon use and mimicry can differ widely in chat communication. Hinting towards personal tendencies and inequalities in relationships© Fraunhofer-Institut für Angewandte Informationstechnik FIT

Paradigmatic Relations Background (corpus linguistic)  Words that occur in similar contexts are commonly semantically related  Example: beer and wine Research question  Do (learning) objects with similar usage contexts have similar content? Approach  Each object holds a usage context profile comprising all its usage contexts  A usage context (UC) consists of a pre- and a post-contexts pre-contexts post-contexts A E D C UC 1 A D C E C F E C G UC 2 C F C G© Fraunhofer-Institut für Angewandte Informationstechnik FIT

Paradigmatic Relations First results using CAM collected in the MACE project: Medium correlation between metadata similarity and object context similarity (0.32), significant due to large sample size (> 65.000.000 object pairs) Manual comparison: 92% of the 100 object pairs with the highest object context similarity are strongly related. The found context similarity was in many cases not entailed in the metadata.© Fraunhofer-Institut für Angewandte Informationstechnik FIT

PPP – Data CollectionEngineering program at Universidad Carlos III de Madrid C programming course from Sep 6 - Dec 16, 2010 (244 students) and Sep 5 – Oct 19, 2011 (342 students)virtual machine with all tools needed, configured by teaching stafflearning management system (.LRN then Moodle) for forums, course material, etc.reminder about data collection at every start of the VM (should be used for course-related work only)existence of a concrete folder functions as a switch (students can move it easily)people in charge can be contacted and request for insight and deletion is possible© Fraunhofer-Institut für Angewandte Informationstechnik FIT

PPP – Analysis and ResultsExtracting key actions to identify user patterns and tendencies throughout the whole course keywords semantically represent the text they are taken from key actions represent the session they are taken fromYear 1: ~120,000 events and 19 event types visualization of key actions showed key action sequences clearly pointing to corrective actions to be deployed analysis of errors also showed problems to discuss in classYear 2: ~125,000 events and 34 event types teachers think key actions to be a very useful form of data distillation  use results for course evaluation teachers liked getting better information from the key actions than from the logs themselves.© Fraunhofer-Institut für Angewandte Informationstechnik FIT

0 50 100 150 200 250 300 350 400 12/9/2010 14/9/2010 16/9/2010 18/9/2010 20/9/2010 22/9/2010 24/9/2010 26/9/2010 28/9/2010 30/9/2010 2/10/2010 4/10/2010 6/10/2010© Fraunhofer-Institut für Angewandte Informationstechnik FIT 8/10/2010 10/10/2010 12/10/2010 14/10/2010 16/10/2010 number of times the error occurred 18/10/2010 20/10/2010 22/10/2010 24/10/2010 26/10/2010 28/10/2010 30/10/2010 1/11/2010 3/11/2010 5/11/2010 PPP – Example Visualizations Year 1 7/11/2010 9/11/2010 11/11/2010 13/11/2010 15/11/2010 17/11/2010 19/11/2010 21/11/2010 number of students getting the error 23/11/2010 25/11/2010 27/11/2010 29/11/2010

A final word...About social media apps Used to communicate context Used to consume context Respect privacy and ensure security of dataDon’t be too overly ambitious: Semi-automatic rule-based volume control is an app that sells for 6 US-$. Don’t try to duplicate it – use it (if possible). Joint To-Do lists including calendar access are already existing, e.g. Family Organizer Follow the principles of architecture design: Copy and improve rather then re-invent.© Fraunhofer-Institut für Angewandte Informationstechnik FIT

MuMe Slide M. Wolpers 18 Nov

by Martin Wolpers

Accessibility

Categories

Upload Details

Usage Rights

Statistics

MuMe Slide M. Wolpers 18 Nov Presentation Transcript