Context-Aware Applications
Workflows, Challenges, and Opportunities

Simon Guest
Distinguished Engineer
Neudesic, LLC
simo...
Workflow for Determining Context

2
1. Determine Sensors
Sensors Critical to Determine Context
•
•
•
•
•

Hard Sensors – GPS, Accelerometer, Gyro, Temperature...
Hard Sensors
GPS
Accel.
Temp
Noise
Step

Soft Sensors
Profile
Schedule
History
Likes/Dis
.
Social
etc.

4
2. Sensor Aggregation
We Don’t
Hard Sensors
GPS

•

Accel.

•

Temp
Noise

•

Step

Need *All* of the Raw Sensor Data to D...
Hard Sensors
GPS
Accel.
Temp
Noise
Step

Soft Sensors
Profile
Schedule
History
Likes/Dis
.
Social
etc.

6
Hard Sensors
GPS

Collections

Accel.

Geo

Temp

Motion

Noise

Environ.

Step

etc.

Soft Sensors
Profile
Schedule

Even...
3. Send to Non-Relational Data Store
Goal is
Hard Sensors

to capture aggregated sensor data quickly and efficiently

Coll...
Hard Sensors
GPS

Collections

Accel.

Geo

Temp

Motion

Noise

Environ.

Step

etc.

Soft Sensors
Profile
Schedule

Even...
Hard Sensors
GPS

Collections

Accel.

Geo

Temp

Motion

Noise

Environ.

Step

etc.

Non-Relational
Store

Soft Sensors
...
4. Determine Perceived Context
Analyze
Hard Sensors

data store to determine one or more perceived contexts

Collections
S...
Hard Sensors
GPS

Collections

Accel.

Geo

Temp

Motion

Noise

Environ.

Step

etc.

Non-Relational
Store

Soft Sensors
...
Query

Hard Sensors
GPS

Collections

Accel.

Geo

Temp

Motion

Noise

Environ.

Step

etc.

Perceived
Context
At Work

N...
5. Triggers on Context Change
Triggers
Hard Sensors

used to monitor change inQuery
context

Perceived
Context

Collection...
Query

Hard Sensors
GPS

Collections

Accel.

Geo

Temp

Motion

Noise

Environ.

Step

etc.

Perceived
Context
At Work

N...
Query

Hard Sensors
GPS

Collections

Accel.

Geo

Temp

Motion

Noise

etc.

Context Change
(Triggers)

Environ.

Step

P...
6. Invoke Functions
Certain
Hard Sensors Triggers
GPS

•

Accel.

Query
Used to Invoke Functions

Perceived
Context

Colle...
Query

Hard Sensors
GPS

Collections

Accel.

Geo

Temp

Motion

Noise

etc.

Context Change
(Triggers)

Environ.

Step

P...
Query

Hard Sensors
GPS

Collections

Accel.

Geo

Perceived
Context

Temp

Motion

Noise

Non-Relational
Store

Started
c...
7. User Interaction and Feedback Loop
Interact
Hard Sensors

Perceived
with user to notify them ofQuery action, and/or pro...
Query

Hard Sensors
GPS

Collections

Accel.

Geo

Perceived
Context

Temp

Motion

Noise

Non-Relational
Store

Started
c...
User
Interaction
Decision Making

Feedback

Query

Hard Sensors
GPS

Collections

Accel.

Geo

Perceived
Context

Temp

Mo...
Types of Context-Aware Applications

23
Types of Context-Aware Applications
Adaptive
•
•
•

Act on behalf of the user
Try to adapt to the user’s context
Often bas...
Challenges When Developing Context-Aware Applications

25
Challenges
Context Mismatch
•
•
•

Sometimes the perceived context is just wrong…
“You really think I want to do this now?...
Opportunities

27
Examples of Opportunities
Retail
•
•

Opportunistic product offers while
shopping
Shopping recommendations based on
curren...
References

29
References
•

•

•

Shilt, Adams, Want (1994): Proceedings of the Workshop on Mobile Computing
Systems and Applications
• ...
Thank You
Simon Guest
Distinguished Engineer
Neudesic, LLC
simon.guest@neudesic.com
@simonguest
31
Upcoming SlideShare
Loading in...5
×

Creating Context-Aware Applications

3,804

Published on

A short presentation on workflow, challenges, and opportunities - and ultimately what it takes to create context-aware application.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,804
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
59
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Creating Context-Aware Applications

  1. 1. Context-Aware Applications Workflows, Challenges, and Opportunities Simon Guest Distinguished Engineer Neudesic, LLC simon.guest@neudesic.com Image Attribution: http://www.flickr.com/photos/48600098314@N01/2612632787 1
  2. 2. Workflow for Determining Context 2
  3. 3. 1. Determine Sensors Sensors Critical to Determine Context • • • • • Hard Sensors – GPS, Accelerometer, Gyro, Temperature, Humidity, etc. Soft Sensors – Calendar, Facebook Friends, Profile, Likes/Dislikes, History, etc. Mobile Devices – Arrays of sensors, with new ones being introduced all the time (e.g. fingerprint democratized with Android) Sensor Co-processors – Sensors moving to co-processors means lower power consumption (which means that sensors can be on all of the time) Bluetooth LE – Greatly extends the the range and types of sensors (e.g. indoor proximity and awareness) 3
  4. 4. Hard Sensors GPS Accel. Temp Noise Step Soft Sensors Profile Schedule History Likes/Dis . Social etc. 4
  5. 5. 2. Sensor Aggregation We Don’t Hard Sensors GPS • Accel. • Temp Noise • Step Need *All* of the Raw Sensor Data to Determine Context Augmented Sensors – Algorithm that augments or abstracts existing sensors (e.g. “Shaking” using the accelerometer) Extended to Motion APIs – Google introducing standing, walking, cycling, driving “sensors” in Android 4.3 Vendor Opportunities – Multiple vendors building SDKs for sensor aggregation – inc. Semusi (Male/Female “sensor” from accelerometer and gyro) Soft Sensors Profile Schedule History Likes/Dis . Social etc. 5
  6. 6. Hard Sensors GPS Accel. Temp Noise Step Soft Sensors Profile Schedule History Likes/Dis . Social etc. 6
  7. 7. Hard Sensors GPS Collections Accel. Geo Temp Motion Noise Environ. Step etc. Soft Sensors Profile Schedule Events History New Like Likes/Dis . Friend Social etc. etc. 7
  8. 8. 3. Send to Non-Relational Data Store Goal is Hard Sensors to capture aggregated sensor data quickly and efficiently Collections • Schema-less– No need to care about schema or structure Accel. Fire and Forget – No need to send confirmation back to client, or handle Geo • Temp missed messages or retries Motion • Scale and Partitions – Data store should be a single entry point that Noise Environ. automatically handles scale and partitioning GPS Step etc. Soft Sensors Profile Schedule Events History New Like Likes/Dis . Friend Social etc. etc. 8
  9. 9. Hard Sensors GPS Collections Accel. Geo Temp Motion Noise Environ. Step etc. Soft Sensors Profile Schedule Events History New Like Likes/Dis . Friend Social etc. etc. 9
  10. 10. Hard Sensors GPS Collections Accel. Geo Temp Motion Noise Environ. Step etc. Non-Relational Store Soft Sensors Profile Schedule Events History New Like Likes/Dis . Friend Social etc. etc. 10
  11. 11. 4. Determine Perceived Context Analyze Hard Sensors data store to determine one or more perceived contexts Collections Simple Matches– Single sensor data can often reveal simple matches (e.g. Accel. movement or in/out geo-fenced location) Geo Non-Relational • Temp Combined Matches – Multiple parallel or serialized sensors to reveal more Motion Store complex matches (e.g. movement with external GPS, combined with loss of Noise signalEnviron.walking activity might indicate parked in underground parking and Step structure). etc. • Learned Matches – Multiple sensor data used to reveal matches through Soft Sensors patterns (e.g. driving detected at same time every day might indicate Profile commuting, which is turn can infer place of work) • Timing – Context often derived from “time blocks” of sensor data – e.g. what is Events Schedule happening now vs. what has happened over the last 30 minutes GPS • History New Like Likes/Dis . Friend Social etc. etc. 11
  12. 12. Hard Sensors GPS Collections Accel. Geo Temp Motion Noise Environ. Step etc. Non-Relational Store Soft Sensors Profile Schedule Events History New Like Likes/Dis . Friend Social etc. etc. 12
  13. 13. Query Hard Sensors GPS Collections Accel. Geo Temp Motion Noise Environ. Step etc. Perceived Context At Work Non-Relational Store Commuting In Traffic Soft Sensors At Home Profile Schedule Events History New Like Likes/Dis . Friend Social etc. etc. Watching TV Asleep Make determination etc. 13
  14. 14. 5. Triggers on Context Change Triggers Hard Sensors used to monitor change inQuery context Perceived Context Collections Change in Context– Move from one perceived context state to another (e.g. At Work Accel. from in Geo office to commuting) Non-Relational • Temp External Data Source– Triggers often rely on external data source (which can Motion Store Commuting be interpreted as another soft sensor). Could include environmental data, move Noise Environ. listings, weather forecasts, etc. Step Machine Learning– Often benefits from some kind of ML to help determine etc. • In Traffic change in context across multiple sensors, especially in recommendations Soft Sensors space GPS • At Home Profile Schedule Events History New Like Likes/Dis . Friend Social etc. etc. Watching TV Asleep Make determination etc. 14
  15. 15. Query Hard Sensors GPS Collections Accel. Geo Temp Motion Noise Environ. Step etc. Perceived Context At Work Non-Relational Store Commuting In Traffic Soft Sensors At Home Profile Schedule Events History New Like Likes/Dis . Friend Social etc. etc. Watching TV Asleep Make determination etc. 15
  16. 16. Query Hard Sensors GPS Collections Accel. Geo Temp Motion Noise etc. Context Change (Triggers) Environ. Step Perceived Context At Work Non-Relational Store Commuting House too cold In Traffic Soft Sensors At Home Profile Schedule Events History Friend Social etc. Timing Loop Flipping channels Better movie New Like Likes/Dis . Started commute etc. Watching TV Asleep Make determination etc. External Data 16
  17. 17. 6. Invoke Functions Certain Hard Sensors Triggers GPS • Accel. Query Used to Invoke Functions Perceived Context Collections Take Action– Functions take action based At Work on triggers. may or Geo not involve some user input. may Temp Motion Noise etc. Commuting House too cold In Traffic Soft Sensors At Home Profile Schedule Events History Friend Social etc. Timing Loop Flipping channels Better movie New Like Likes/Dis . that Started commute Environ. Step Non-Relational Store Context Change Invoke an action (Triggers) etc. Watching TV Asleep Make determination etc. External Data 17
  18. 18. Query Hard Sensors GPS Collections Accel. Geo Temp Motion Noise etc. Context Change (Triggers) Environ. Step Perceived Context At Work Non-Relational Store Commuting House too cold In Traffic Soft Sensors At Home Profile Schedule Events History Friend Social etc. Timing Loop Flipping channels Better movie New Like Likes/Dis . Started commute etc. Watching TV Asleep Make determination etc. External Data 18
  19. 19. Query Hard Sensors GPS Collections Accel. Geo Perceived Context Temp Motion Noise Non-Relational Store Started commute Commuting House too cold Environ. Step At Work Context Change (Triggers) etc. In Traffic Soft Sensors At Home Profile Schedule Events History Friend Social etc. Flipping channels Better movie Turn on heating Adaptive application Make recommendation Proactive application New Like Likes/Dis . Watching TV Timing Loop Functions etc. Asleep Make determination etc. External Data 19
  20. 20. 7. User Interaction and Feedback Loop Interact Hard Sensors Perceived with user to notify them ofQuery action, and/or provide feedback any Context GPS Collections Step etc. Context Change function (Triggers) • User Notification– Notify the user of the result from a At Work Accel. User Interaction/Confirmation – Get confirmation from the user that a function Geo • Functions Started shouldMotion happen, Non-Relationalrecommendation. really or provide a commute Temp Store Commuting • Feedback Loop – Provide option for user to submit feedback based on the Turn on heating House too Noise Environ. cold function (like, dislike, rating, etc.) Adaptive application In Traffic Soft Sensors At Home Profile Schedule Events History Friend Social etc. Flipping channels Better movie Make recommendation Proactive application New Like Likes/Dis . Watching TV Timing Loop etc. Asleep Make determination etc. External Data 20
  21. 21. Query Hard Sensors GPS Collections Accel. Geo Perceived Context Temp Motion Noise Non-Relational Store Started commute Commuting House too cold Environ. Step At Work Context Change (Triggers) etc. In Traffic Soft Sensors At Home Profile Schedule Events History Friend Social etc. Flipping channels Better movie Turn on heating Adaptive application Make recommendation Proactive application New Like Likes/Dis . Watching TV Timing Loop Functions etc. Asleep Make determination etc. External Data 21
  22. 22. User Interaction Decision Making Feedback Query Hard Sensors GPS Collections Accel. Geo Perceived Context Temp Motion Noise Non-Relational Store Started commute Commuting House too cold Environ. Step At Work Context Change (Triggers) etc. In Traffic Soft Sensors At Home Profile Schedule Events History Friend Social etc. Flipping channels Better movie Turn on heating Adaptive application Make recommendation Proactive application New Like Likes/Dis . Watching TV Timing Loop Functions etc. Asleep Make determination etc. External Data 22
  23. 23. Types of Context-Aware Applications 23
  24. 24. Types of Context-Aware Applications Adaptive • • • Act on behalf of the user Try to adapt to the user’s context Often based using single-trigger • • • Turn on heating when leave for home Phone to silent in meetings Offer music library when get on bus Proactive • • • • Involving the user Often tied to a recommendation engine Often requires user-interaction Often uses multiple triggers • Recommend movie to watch when watching TV Display offer based on history when walking through grocery store • • Need to be careful if building adaptive UI, especially if hiding/showing features • UI needs to be unobtrusive, and easy to dismiss 24
  25. 25. Challenges When Developing Context-Aware Applications 25
  26. 26. Challenges Context Mismatch • • • Sometimes the perceived context is just wrong… “You really think I want to do this now?!?!” Multiple contexts help accuracy, and feedback mechanism is critical Being Creepy • • • Sharing everything that you know about the context/trigger can appear creepy “Your Mom’s birthday is tomorrow, and you’ve missed the last three years in a row. Do you want to pull into the 7-Eleven ahead? They have flowers on sale…” Reveal only enough to let the user deduce the same context, and feedback again is critical Being Annoying • • • Interruptions should follow normal human behaviors “I wanted to let you know that I’ve turned the heating on” – “Not now, I’m driving!” Knowing the context of the recipient during the UI loop is also important 26
  27. 27. Opportunities 27
  28. 28. Examples of Opportunities Retail • • Opportunistic product offers while shopping Shopping recommendations based on current context Real Estate • Contextual awareness of prospective buyers searching for properties Gaming • Using context to enhance the gaming experience of patrons Field Employees • Context-aware applications that helps field-based employees become more productive Home Automation • Context-aware applications that interact with other systems in the home Travel Applications • More intelligent travel applications through context 28
  29. 29. References 29
  30. 30. References • • • Shilt, Adams, Want (1994): Proceedings of the Workshop on Mobile Computing Systems and Applications • http://www.interactiondesign.org/references/conferences/proceedings_of_the_workshop_on_mo bile_computing_systems_and_applications.html Swati A. Sonawane: Context-Aware Computing • http://www.slideshare.net/swatibaiger/context-aware-computing14084995 Albrecht Schmidt (2013): Context-Awareness, Context-Aware User Interfaces, and Implicit Interaction • http://www.interaction-design.org/encyclopedia/contextaware_computing.html 30
  31. 31. Thank You Simon Guest Distinguished Engineer Neudesic, LLC simon.guest@neudesic.com @simonguest 31
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×