Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Making Sense
The Road to mobile Awareness
• Jared Sheehan
• Twitter: @jayroo5245
• meetup.com/DCAndroid
• slideshare.net/J...
• What is Contextual
Awareness?
• Use Cases
• Sensor Fusion
• Hard way
• Medium hard way
• Easy way
• Questions
Agenda
“Context is any information that can be used to
characterize the situation of an entity. An entity is
a person, place, or ...
• Mobile sensing of a user’s context
• Sensor based algorithms
• Some Sensor Types on the Android
Platform
• Accelerometer...
Sensor Types in Android
• Detecting when a user:
• Changes the orientation of a
their device
• walking, running or biking
• Driving a vehicle
• Ha...
Sensor fusion is combining of
sensory data or data derived
from disparate sources such
that the resulting information
has ...
If a developer takes
individual sensor output
and combines it with
additional output from
other sensors (or other
hints) t...
Determine the attitude of a
mobile device.
Attitude - orientation of a
device relative to Earth's
horizon
Sensor Fusion – ...
The common way to get the
attitude of an Android
device is to use the
SensorManager.getOrientati
on() method to get the th...
In simple terms, the
accelerometer provides the
gravity vector (the vector
pointing towards the center
of the earth) and t...
Isn’t that enough?
Sensor Fusion – Example
No
Problem is that both sensor
outputs are inaccurate,
especially the output fr...
Gyro drift and noisy orientation
are common issues with this
approach, to solve it, the
gyroscope output is applied
only f...
This is equivalent to low-pass
filtering of the accelerometer
and magnetic field sensor
signals and high-pass filtering
of...
Sensor Fusion – Example
So what exactly does high-pass
and low-pass filtering of the
sensor data mean? The sensors
provide their data at (more or
...
The low-pass filtering of the
noisy
accelerometer/magnetometer
signal (accMagOrientationin
the above figure) are
orientati...
Initialize sensor
containers:
Sensor Fusion – Example
Register you listeners:
Sensor Fusion – Example
Store sensor events:
Sensor Fusion – Example
At some time interval you
process the sensor arrays and
then events can be inferred
from a single or multiple
passes.
Sens...
Example of Rotation Vector
processing:
https://developer.android.com
/reference/android/hardware/
SensorEvent.html#values
...
• https://github.com/Jayroo5245/mak
ingsense
• https://github.com/Jayroo5245
Demo time!
• This is a simple-ish formula to
obtain one feature
• Very large task
• Lots of math, calculations,
sensor state maintena...
• How do you support 100% of
devices?
• Very difficult
• Android Fragmentation
• Not all sensors return values
at the same...
• Process prioritization issues
• OEMs build devices to
their specs, not ours
• Missing sensors on some
devices.
• Android...
Example Platform limitation:
The Android Platform was not
designed to process sensor
data as fast as it is generated.
Usin...
• External Libraries
• Lost - Drop in Replacement for
Google’s Fused Location API
• www.zendrive.com
• www.driversiti.com
...
• External Libraries
• Licensing – IE Cost
• Probably don’t do exactly
what you want
• Inference
Change/Deprecation
• lack...
Battery Issues:
Let Google Do It for you – Awareness API
• Current Local Time
Context #1– Time
• Latitude
• Longitude
Context #2 – Location
• Place, including place Type
Context #3 – Place
• Activity Recognition
• Detected user activity
(walking, running, biking)
Context #4 – Activity
• Nearby beacons (including
namespace, type, and
content)
Context #5 – Beacons
• Are the Headphones plugged?
Context #6 – Headphones
• Current Weather Conditions
Context #7 – Weather
• Apps can combine these
context signals to make
inferences about the
user's current situation,
and use this information
t...
• Easy implementation
• One API
• Signals are processed
for the app
• No need to build
complicated
algorithms
• Optimized ...
• Fence API
• System Notifications
• Snapshot API
• Real time request
Great now what?
• Push Mechanism - React to
specific situations
• Provides notifications
when a specific
combination of actions
occur
• Ex...
• Pull mechanism
• Provides notifications
when a specific
combination of actions
occur
• Exp: Tell me when a user is
bikin...
• Hard way
• Build your own
• Easier Way
• External Lib
• Easiest Way
• Awareness API
Three options
Thank you for coming!
The Road to mobile Awareness
• Jared Sheehan
• Twitter: @jayroo5245
• meetup.com/DCAndroid
• slidesh...
Sources
• https://en.wikipedia.org/wiki/Sensor_fusion
• https://developer.android.com/guide/topics/sensors/sensors_overvie...
Making sense
Making sense
Making sense
Making sense
Upcoming SlideShare
Loading in …5
×

Making sense

302 views

Published on

Mobile devices are becoming more and more powerful. They come with all sorts of wonderful hardware like cpu/gpus, tons of ram and blazing fast download times. Smart phones have become commoditized in a sense. What's the next evolution of mobile? Now these devices are coming with a really solid set of sensors and apis that allow developers to determine a user's context. How does that work? Developers fuse the sensor output to infer context and infer events from the data. This talk will discuss ways to do it, challenges and drawbacks.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Making sense

  1. 1. Making Sense The Road to mobile Awareness • Jared Sheehan • Twitter: @jayroo5245 • meetup.com/DCAndroid • slideshare.net/Jayroo5245
  2. 2. • What is Contextual Awareness? • Use Cases • Sensor Fusion • Hard way • Medium hard way • Easy way • Questions Agenda
  3. 3. “Context is any information that can be used to characterize the situation of an entity. An entity is a person, place, or object that is considered relevant to the interaction between a user and an application, including the user and application themselves. ” • Anind Dey • Director of Human-Computer interaction at Carnegie Mellon University
  4. 4. • Mobile sensing of a user’s context • Sensor based algorithms • Some Sensor Types on the Android Platform • Accelerometer • Gyroscope/Orientation/Rotation Vector • Barometric Pressure • Magnetic Field • Gravity • Relative Humidity • Ambient Room Temperature • Device Temperature Mobile Contextual Awareness
  5. 5. Sensor Types in Android
  6. 6. • Detecting when a user: • Changes the orientation of a their device • walking, running or biking • Driving a vehicle • Handling their device • Driving AND Handling their device • Disclaimer – Don’t do it • Drives by a restaurant or coffee shop when it is open • Driving detection • Google Places • Time Use Cases
  7. 7. Sensor fusion is combining of sensory data or data derived from disparate sources such that the resulting information has less uncertainty than would be possible when these sources were used individually. Sensor Fusion – What is it?
  8. 8. If a developer takes individual sensor output and combines it with additional output from other sensors (or other hints) then you get a better understanding of what is going on with the mobile device. Sensor Fusion – What is it?
  9. 9. Determine the attitude of a mobile device. Attitude - orientation of a device relative to Earth's horizon Sensor Fusion – Example
  10. 10. The common way to get the attitude of an Android device is to use the SensorManager.getOrientati on() method to get the three orientation angles. These two angles are based on the accelerometer and magnetometer output. Sensor Fusion – Example
  11. 11. In simple terms, the accelerometer provides the gravity vector (the vector pointing towards the center of the earth) and the magnetometer works as a compass. The Information from both sensors suffice to calculate the device’s orientation. Sensor Fusion – Example
  12. 12. Isn’t that enough? Sensor Fusion – Example No Problem is that both sensor outputs are inaccurate, especially the output from the magnetic field sensor which includes a lot of noise. How do we fix it?
  13. 13. Gyro drift and noisy orientation are common issues with this approach, to solve it, the gyroscope output is applied only for orientation changes in short time intervals. While the magnetometer/accelerometer data is used as support information over long periods of time. Sensor Fusion – Example
  14. 14. This is equivalent to low-pass filtering of the accelerometer and magnetic field sensor signals and high-pass filtering of the gyroscope signals. The overall sensor fusion and filtering looks like this: Sensor Fusion – Example
  15. 15. Sensor Fusion – Example
  16. 16. So what exactly does high-pass and low-pass filtering of the sensor data mean? The sensors provide their data at (more or less) regular time intervals. Their values can be shown as signals in a graph with the time as the x-axis, similar to an audio signal. Sensor Fusion – Example
  17. 17. The low-pass filtering of the noisy accelerometer/magnetometer signal (accMagOrientationin the above figure) are orientation angles averaged over time within a constant time window. Sensor Fusion – Example
  18. 18. Initialize sensor containers: Sensor Fusion – Example
  19. 19. Register you listeners: Sensor Fusion – Example
  20. 20. Store sensor events: Sensor Fusion – Example
  21. 21. At some time interval you process the sensor arrays and then events can be inferred from a single or multiple passes. Sensor Fusion – Example
  22. 22. Example of Rotation Vector processing: https://developer.android.com /reference/android/hardware/ SensorEvent.html#values Sensor Fusion – Example
  23. 23. • https://github.com/Jayroo5245/mak ingsense • https://github.com/Jayroo5245 Demo time!
  24. 24. • This is a simple-ish formula to obtain one feature • Very large task • Lots of math, calculations, sensor state maintenance • Not something a standard Android developer is used to working with Sensor Fusion – Challenges
  25. 25. • How do you support 100% of devices? • Very difficult • Android Fragmentation • Not all sensors return values at the same frequency Sensor Fusion – Challenges
  26. 26. • Process prioritization issues • OEMs build devices to their specs, not ours • Missing sensors on some devices. • Android/Java platform limitations • Go Native - NDK Sensor Fusion – Challenges
  27. 27. Example Platform limitation: The Android Platform was not designed to process sensor data as fast as it is generated. Using an Executor had the best results but you will not get consistent 16, 32 or 64 hertz. Sensor Fusion – Challenges
  28. 28. • External Libraries • Lost - Drop in Replacement for Google’s Fused Location API • www.zendrive.com • www.driversiti.com • www.pathsense.com • www.locationkit.io Alternatives to the hard way:
  29. 29. • External Libraries • Licensing – IE Cost • Probably don’t do exactly what you want • Inference Change/Deprecation • lack support • Battery Drain Drawbacks to External Libs:
  30. 30. Battery Issues:
  31. 31. Let Google Do It for you – Awareness API
  32. 32. • Current Local Time Context #1– Time
  33. 33. • Latitude • Longitude Context #2 – Location
  34. 34. • Place, including place Type Context #3 – Place
  35. 35. • Activity Recognition • Detected user activity (walking, running, biking) Context #4 – Activity
  36. 36. • Nearby beacons (including namespace, type, and content) Context #5 – Beacons
  37. 37. • Are the Headphones plugged? Context #6 – Headphones
  38. 38. • Current Weather Conditions Context #7 – Weather
  39. 39. • Apps can combine these context signals to make inferences about the user's current situation, and use this information to provide customized experiences. • Exp: Suggest a playlist while jogging in the rain. What is it?
  40. 40. • Easy implementation • One API • Signals are processed for the app • No need to build complicated algorithms • Optimized Battery Awareness Benefits
  41. 41. • Fence API • System Notifications • Snapshot API • Real time request Great now what?
  42. 42. • Push Mechanism - React to specific situations • Provides notifications when a specific combination of actions occur • Exp: Tell me when a user is biking, its lunchtime and near a bike friendly restaurant Fence API
  43. 43. • Pull mechanism • Provides notifications when a specific combination of actions occur • Exp: Tell me when a user is biking, its lunchtime and near a bike friendly restaurant Snapshot API
  44. 44. • Hard way • Build your own • Easier Way • External Lib • Easiest Way • Awareness API Three options
  45. 45. Thank you for coming! The Road to mobile Awareness • Jared Sheehan • Twitter: @jayroo5245 • meetup.com/DCAndroid • slideshare.net/Jayroo5245
  46. 46. Sources • https://en.wikipedia.org/wiki/Sensor_fusion • https://developer.android.com/guide/topics/sensors/sensors_overview.html • http://plaw.info/2012/03/android-sensor-fusion-tutorial/comment-page-1/ • http://www.androidpolice.com/2016/05/19/the-new-awareness-api-will-let- apps-better-understand-your-environment/ • https://developers.google.com/awareness/overview • https://www.interaction-design.org/literature/book/the-encyclopedia-of- human-computer-interaction-2nd-ed/context-aware-computing-context- awareness-context-aware-user-interfaces-and-implicit-interaction

×