This document outlines a study that presents a joint probabilistic model to improve activity detection by reasoning about object use, physical actions, and activities together. Sensors collected data from researchers performing 12 activities using 30 objects over 2 weeks. A dynamic Bayesian network model was used to infer the current action and object being used from the sensor data. Results showed the joint model produced better activity detection than models reasoning separately about object use, actions, and activities, and learned action models with less labeling than conventional approaches.

1. Common Sense Based Joint Training of Human Activity Recognizers

2. Outline Introduction Sensors Techniques Evaluation Methodology Results Conclusions

3. Introduction The paper presents a joint probabilistic model of object-use, physical actions and activities that improves activity detection relative to models that reason separately about these quantities, and learns action models with much less labeling overhead than conventional approaches

4. Sensors Two bracelets on the dominant wrist Radio Frequency Identification (RFID) reader, called the iBracelet, for detecting object use The other is a personal sensing reader, called the Mobile Sensing Platform (MSP), for detecting arm movement and ambient conditions eight sensors: a six-degree-of-freedom accelerometer, microphones sampling 8-bit audio at 16kHz, IR/visible light, high-frequency light, barometric pressure, humidity, temperature and compass

6. Techniques Problem Statement A Joint Model The Dynamic Bayesian Network (DBN) A Layered Model

7. Problem Statement Given the above domain information and observed data, we wish to build a classifier over MSP data that: Infers the current action being performed and the object on which it is being performed. Combines with object-use data O (when available) to produce better estimates of current activity A. For efficiency reasons, uses F’ « F features of M.

8. Joint models

9. Action recognition

10. The VirtualBoost Algorithm

11. Evaluation Methodology (cont.) Collected iBracelet and MSP data from two researchers performing 12 activities containing 10 distinct actions of interest using 30 objects in an instrumented apartment. Four to ten executions of each activity were recorded over a period of two weeks. Ground truth was recorded using video, and each sensor data frame was annotated in a post-pass with the actual activity and action (if any) during that frame, and designated “other” if not

12. Evaluation Methodology (cont.)

13. Result(cont.)

14. Result(cont.)

15. Conclusions This paper have demonstrated that it is possible to use data from dense object-use sensors and very simple commonsense models of object use, actions and activities to automatically interpret and learn models for other sensors, a technique we call common sense based joint training.