Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Rule-based Real-Time Activity Recognition in a Smart Home Environment

This presentation outlines a rule-based approach for both offline and real-time recognition of Activities of Daily Living (ADL), leveraging events produced by a non-intrusive multi-modal sensor infrastructure deployed in a residential environment. Novel aspects of the approach include: the ability to recognise arbitrary scenarios of complex activities
using bottom-up multi-level reasoning, starting from sensor events at the lowest level; an effective heuristics-based method for distinguishing between actual and ghost images in video data; and a highly accurate indoor localisation approach that fuses different sources of location information. The proposed approach is implemented as a rule-based system
using Jess and is evaluated using data collected in a smart home environment. Experimental results show high levels of accuracy and performance,proving the effectiveness of the approach in real world setups.

Related Books

Free with a 30 day trial from Scribd

See all
  • Be the first to comment

  • Be the first to like this

Rule-based Real-Time Activity Recognition in a Smart Home Environment

  1. 1. Rule-based Real-Time Activity Recognition in a Smart Home Environment Przemyslaw Woznowski Grigoris Antoniou 10th International Web Rule Symposium (RuleML) 2016, Stony Brook, New York, USA George Baryannis
  2. 2. Outline 2 Introduction • Activity Recognition and the Internet of Things • The SPHERE Project • Related Work Rule-based ADL Recognition • Offline Version • Online Version • Experimental Evaluation Conclusions & Future Work
  3. 3. Activity Recognition and the Internet of Things • Sensors have become cheaper, small, widely available • Interconnected within an Internet of Things (IoT) setting, benefitting from – Distribution of resources – Support for common naming schemas and ontologies – Common access strategies – Availability of computational resources • Automated Activity Recognition (AR) requires a fusion of multiple sensor-related low-level events • Challenge: to locate and fuse the right pieces of information from an IoT instance (e.g. sensor network) in order to realise AR at the best quality of information possible 3
  4. 4. Approaches for sensor-based AR • Data-driven – exploiting machine learning techniques Noise and uncertainty are handled well Require large, annotated training datasets Data conflicts are not handled well • Knowledge-driven – leveraging logical modelling and reasoning No training data needed Not as robust against noise and uncertainty Require carefully crafted rules 4
  5. 5. Activity Recognition in a multi-modal smart home environment • Focuses on the so-called Activities of Daily Living (ADL), with the purpose of supporting Ambient Assisting Living (AAL) efforts – Long-term monitoring of health-related features – Direct assistance • Main requirements – Increased need for robustness against noise (due to multiple sensors) – Support for complex, uncertain and non-sequential scenarios – Support for user localization within the smart home, with minimal user involvement – Inference of real-time, continuous streams of meaningful and actionable events – Less reliance on training data, since they are difficult to acquire due to them being environment-dependent 5
  6. 6. The SPHERE Project Woznowski et al. (2015) 6 • SPHERE: a Sensor Platform for Healthcare in a Residential Environment – Common platform of non-medical networked sensors – Deployed on a home environment testbed, the SPHERE house – Impact a range of healthcare needs simultaneously
  7. 7. • Chen et al.: equivalence and subsumption reasoning on ontologies modelling both sensors and activities Both offline and real-time modes, incrementally-specific recognition Requires activities to be performed in a predefined, strictly sequential order and fixed time intervals • MetaQ: SPARQL-based reasoning on sensor data represented as RDF graphs Recognition building from atomic gestures to complex activities Works only offline, does not take into account missing activities Related Work (1) 7
  8. 8. • Skarlatidis et al.: hybrid approach, combining event calculus reasoning with Markov Logic Networks High recognition rates, robustness against missing data Only focuses on posture and movement-related activities, as opposed to complex ADL scenarios • Helaoui et al.: hybrid approach, employing a probabilistic DL reasoner Recognition building from atomic gestures to complex activities Requires training data, works only offline, no support for temporal features Related Work (2) 8
  9. 9. Outline 9 Introduction • Activity Recognition • The SPHERE Project • Related Work Rule-based ADL Recognition • Offline Version • Online Version • Experimental Evaluation Conclusions & Future Work
  10. 10. 10 • Rule base – rules defined by examining collected sensor data from scripted experiments • Fact base – derived from sensor data • The system operates in two modes – Offline: precollected sensor data are stored as individual facts • Can provide activity reports for past periods (e.g. hourly or daily) – Real-time: facts represent each deployed sensor node and store its current state/value (as well as its previous one) • Recognises activities as soon as the associated sensor events happen Rule Base Fact Base Inference Engine (JESS) “Expert” knowledge Sensor data Rule-based System Overview
  11. 11. 11 • Environmental Sensors – Door contact, electricity meters, water flow meters, PIR – Ambient light useful only when the effect of sunlight is minimal (i.e. the sun is below the horizon) – Scripted experiment data do not yield patterns from ambient noise, dust, humidity and temperature • Video Sensors – 2D bounding box coordinates – Depth coordinates of 3D bounding box Fact Base
  12. 12. 12 PIR-based Location Camera- based Location Fused Location Second Level Rules Higher Level Rules Door Interaction Electrical Devices Water Flow Water Flow Clean-up Atomic Activity Rules Rule Base rules assume single inhabitant scenarios
  13. 13. 13 • Detect changes in sensor values within their reporting windows – From >0 to 0: OpenDoor / SwitchOff – From 0 to >0: CloseDoor / SwitchOn Doors and Electrical Devices
  14. 14. 14 • Water meters do not have a reporting period, only report instantaneously – Positive flow value: OpenTap – Zero flow value: CloseTap • “Clean up” rules follow to keep only the earliest events for each distinct opening or closing occurrence – If there is no close tap activity between two consecutive open tap activities, remove the latest one Water Flow
  15. 15. 15 • Rules so far recognise atomic activities • Higher-level rules progressively combine recognised activities to infer activities of increasingly higher complexity – SwitchOn(device,t1) and SwitchOff(device,t2)  Use(device, t1, t2) – SwitchOn(tv,t1) and SwitchOff(tv,t2)  WatchingTV(t1, t2) – Use of taps in kitchen or bathroom  WashHands or WashFace – Use of taps in bathroom  BrushTeeth or Bathing/Showering – Use kettle and close tap in kitchen  PreparingDrink – Open fridge and use toaster  PreparingSnack – PreparingDrink or PreparingSnack and use of taps in kitchen  WashDishes Complex Activities
  16. 16. 16 • Basic PIR rule places user in a specific room, from the time PIR is activated till it’s deactivated – Sequences in the same room merged if temporally close or user not in a different room in between • Basic video rule places user in a specific room, for as long as the associated camera reports bounding box coordinates • Detect ghost sequences since they severely compromise validity – Length of less than 30 frames – Stuck in the same coordinates for more than 30 frames – Width and/or height of box consistently and unjustifiably small, in correlation with depth Localisation Rules
  17. 17. 17 • Assign confidence values to PIR and camera location reports – PIR: confidence inversely proportional to the number of PIR sensors simultaneously reporting motion – Video: confidence depends on the probability of being a ghost, based on the detection heuristics • If only a single source reports a location, it is assumed to hold (with the associated confidence value) • If PIR and video report the same location, it is assumed to hold (with confidence values summed) • If PIR and video disagree, the correct location is the one associated with a recognised atomic activity • If both or neither disagreeing reports are supported by an activity, we assume the one with the higher confidence holds – If confidences are equal, we trust PIR Fused Location
  18. 18. 18 • Facts now represent the state of each distinct sensor – Instead of the history of sensor events – To detect state change, previous state is also stored • Changes are necessary only for rules at the lowest level – Second and higher-level rules remain unchanged • Transparent to the way sensor events are generated • Any state change event is linked to a related atomic activity – Holds for DC sensors, electricity and water flow meters – Rules fire only once when sensor values change – no need for “clean up” rules From Offline to Real-time
  19. 19. 19 • Each consecutive activation/deactivation of a PIR sensor corresponds to the user being in the associated room • Subsequent activations extend the user’s stay when – Activation directly follows the last deactivation – The elapsed time between them does not exceed a threshold – No activation has taken place in a different room in the meantime • State-based approach is not applicable for video sensors – Video cameras do not broadcast a single value • Each reported bounding box is stored briefly – Combined to create facts that represent a period of stay in a room – The same heuristics used for ghost detection Online Localisation (1)
  20. 20. 20 • Each time a PIR sensor is activated, the system fuses available information to decide on its validity – If there is no active video sequence and no activity detected, we assume PIR is valid – If the active video sequence with the highest confidence agrees with PIR, we conclude the user is in the room, summing confidence values – If video reports a different room, we assume the user is in the room where the most recently recognised atomic activity was performed Online Localisation (2): Fusion
  21. 21. 21 • Offline and real-time versions implemented in Java, using Jess as a rule engine – Implemented rules designed to accommodate variable reporting periods • Real-time version built as an MQTT client – Sensor messages are broadcast in separate threads • 10 participants executed an ADL script of half-hour duration, twice, in the SPHERE house – Ground-truth data acquired through annotation of video images collected using a head-mounted camera – Subset of performed activities that are recognised: interaction with doors, electrical devices and water taps, preparing a snack/drink, washing hands/dishes, brushing teeth, bathing/showering Implementation and Data Collection
  22. 22. 22 • TP (true positive): activity performed and recognised • FP (false positive): activity not performed but recognised • FN (false negative): activity performed but not recognised • 𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = 𝑇𝑃 𝑇𝑃+𝐹𝑃 %, 𝑟𝑒𝑐𝑎𝑙𝑙 = 𝑇𝑃 𝑇𝑃+𝐹𝑁 % Evaluation Results
  23. 23. Outline 23 Introduction • Activity Recognition • The SPHERE Project • Related Work Rule-based ADL Recognition • Offline Version • Online Version • Experimental Evaluation Conclusions & Future Work
  24. 24. Concluding remarks • Rule-based system capable of operating on both historical and real-time, multi-modal sensor data acquired in a smart home – Bottom-up, multi-level rules to support complex ADL scenarios – Non-deterministic patterns to account for missing activities • Sensor fusion and heuristics to achieve robustness against noise – 95% recall and 88% precision on average for a significant subset of activities – 93% room-level localisation accuracy due to effective ghost detection and location fusion rules 24
  25. 25. 25 • Integrate wearable sensor data – Infer activities unidentifiable with only the other sensors – Improve localisation accuracy or provide an alternative to video cameras when they are not available/allowed • Explore multi-inhabitant scenarios – Use localisation results to pin down activities to the person performing them – For some activities, localisation needs to be more fine-grained than room-level • Explore hybrid approach with Machine Learning research within SPHERE – Incorporate rules as features in ML algorithms – Use rules that act on the results of ML algorithms – Devise ML techniques to learn rules Current and Future Work
  26. 26. 26 Thank you! Questions? g.bargiannis@hud.ac.uk http://www.irc-sphere.ac.uk
  27. 27. 27 • Chen, L., Nugent, C.D., Wang, H.: A knowledge-driven approach to activity recognition in smart homes. IEEE Trans. Knowl. Data Eng. 24(6), 961–974 (2012) • Filippaki, C., Antoniou, G., Tsamardinos, I.: Using constraint optimization for conflict resolution and detail control in activity recognition. In: Keyson, D.V., Maher, M.L., Streitz, N., Cheok, A., Augusto, J.C., Wichert, R., Englebienne, G., Aghajan, H., Krose, B.J.A. (eds.) AmI 2011. LNCS, vol. 7040, pp. 51–60. Springer, Heidelberg (2011) • Helaoui, R., Riboni, D., Stuckenschmidt, H.: A probabilistic ontological framework for the recognition of multilevel human activities. In: Mattern, F., Santini, S., Canny, J.F., Langheinrich, M., Rekimoto, J. (eds.) UbiComp 2013, pp. 345–354. ACM (2013) • Meditskos, G., Dasiopoulou, S., Kompatsiaris, I.: MetaQ: a knowledge-driven framework for context-aware activity recognition combining SPARQL and OWL 2 activity patterns. Pervasive Mob. Comput. 25, 104–124 (2016) • Skarlatidis, A., Paliouras, G., Artikis, A., Vouros, G.A.: Probabilistic event calculus for event recognition. ACM Trans. Comput. Log. 16(2), 11:1–11:37 (2015) • Woznowski, P., Fafoutis, X., Song, T., Hannuna, S., Camplani, M., Tao, L., Paiement, A., Mellios, E., Haghighi, M., Zhu, N., et al.: A multi-modal sensor infrastructure for healthcare in a residential environment. In: 2015 IEEE International Conference on Communication Workshop, pp. 271–277. IEEE (2015) References

    Be the first to comment

    Login to see the comments

This presentation outlines a rule-based approach for both offline and real-time recognition of Activities of Daily Living (ADL), leveraging events produced by a non-intrusive multi-modal sensor infrastructure deployed in a residential environment. Novel aspects of the approach include: the ability to recognise arbitrary scenarios of complex activities using bottom-up multi-level reasoning, starting from sensor events at the lowest level; an effective heuristics-based method for distinguishing between actual and ghost images in video data; and a highly accurate indoor localisation approach that fuses different sources of location information. The proposed approach is implemented as a rule-based system using Jess and is evaluated using data collected in a smart home environment. Experimental results show high levels of accuracy and performance,proving the effectiveness of the approach in real world setups.

Views

Total views

414

On Slideshare

0

From embeds

0

Number of embeds

1

Actions

Downloads

21

Shares

0

Comments

0

Likes

0

×