Activity recognition based on
a multi-sensor hierarchical-
classifier
IWANN 2013, 12-14 June, Tenerife (Spain)
Oresti Baños, Miguel Damas, Héctor Pomares and Ignacio Rojas
Department of Computer Architecture and Computer Technology, CITIC-UGR,
University of Granada, SPAIN
DG-Research Grant #228398

Introduction
• Activity recognition concept
– “Recognize the actions and goals of one or more agents from a series of
observations on the agents' actions and the environmental conditions”
• Applications (among others)
– eHealth (AAL, telerehabilation)
– Sports (performance improvement, injury-free pose)
– Industrial (assembly tasks, avoidance of risk situations)
– Gaming (Kinect, Wii Mote, PlayStationMove)
• Categorization by sensor modality
– Ambient
– On-body
2

Sensing Activity
3
• Ambient sensors

Sensing Activity
• Ambient sensors
Limitations*

3rd Generation
(and beyond…)
2nd Generation1st Generation
Sensing Activity
5
• On-body sensors

Activity Recognition Chain (ARC)
6

Activity Recognition Chain (ARC)
7

Activity Recognition Chain (ARC)
8

Activity Recognition Chain (ARC)
9

Activity Recognition Chain (ARC)
10

Activity Recognition Chain (ARC)
11

Activity Recognition Chain (ARC)
12

Activity Recognition Chain (ARC)
13

Activity Recognition Chain (ARC)
14

Activity Recognition Chain (ARC)
15

Activity Recognition Chain (ARC)
16
SENSOR
FUSION

ARC Fusion: Feature Fusion
17

ARC Fusion: Decision Fusion
18

Multi-Sensor Hierarchical Classifier
19
SM
S2
S1
α11
∑
C12
C1N
C11
∑
C21
C22
C2N
∑
CM1
CM2
CMN
∑
Decision
Class level Source level Fusion
β11
α12
β12
α1N
β1N
α21
β21
α22
β22
α2N
β2N
αM1
βM1
αM2
βM2
αMN
βMN
γ11,…,1N
δ11,…,1N
γ21,…,2N
δ21,…,2N
γM1,…,MN
δM1,…,MN
[-0.14,3.41,4,21,…,6.11]
[-0.84,3.21,4.21,…,6.11]
[-0.81,5.71,4.21,…,6.22]
[-0.14,3.92,4.23,…,7.82]
S1
S2
SM
u1 p1 s11,s12,…,s1k fℝ(s11,s12,…,s1k)
u2 p2 s21,s22,…,s2k fℝ(s21,s22,…,s2k)
uM pM sM1,sM2,…,sMk fℝ(sM1,sM2,…,sMk)

Multi-Sensor Hierarchical Classifier
20
N activities M sensors&Class level Source level Fusion

Multi-Sensor Hierarchical Classifier
21
N activities M sensors&Class level Source level Fusion

Multi-Sensor Hierarchical Classifier
22
N activities M sensors&Class level Source level Fusion

Multi-Sensor Hierarchical Classifier
23
N activities M sensors&Class level Source level Fusion

Experimental setup: dataset
• Fitness benchmark dataset
• Up to 33 activities
• 9 IMUs (XSENS)  ACC, GYR, MAG
• 17 subjects
24
Baños, O., Toth M. A., Damas, M., Pomares, H., Rojas, I., Amft, O.: A benchmark dataset to evaluate sensor displacement in activity recognition.
In: 14th International Conference on Ubiquitous Computing (Ubicomp 2012), Pittsburgh, USA, September 5-8, (2012)

Results
• Segmentation: sliding window (6 seconds)
• Feature extraction: FS1={mean}, FS2={mean,std}, FS3={mean,std,max,min,cr}
• Classification: Decision tree (C4.5) (10-fold cross-validated, 100 repetitions)
2510 activities 20 activities 33 activities
FS1 FS2 FS3 FS1 FS2 FS3 FS1 FS2 FS3
60
65
70
75
80
85
90
95
100
Accuracy(%)
Feature Fusion Weighted Majority Voting Multi-Sensor Hierarchical Classifier
Experimental Parameters

Conclusions
• We propose a multi-sensor hierarchical classifier that allows data
fusion of multiple sensors
– Its assymetric decision weighting (SEinsertions/SPrejections)
leverages the potential of the classifiers either for
classification/rejection or both
– Specially suited for complex scenarios
• Feature Fusion and MSHC are quite in line in terms of performance
however
– Our method outperforms the former when a more informative
feature set is used
– Particularly notable for complex recognition scenarios
• Our model is expected to be particularly suited to deal with sensor
anomalies (work-in-progress)
26

On-going work…
• Our model is expected to be particularly suited to deal with
sensor anomalies (work-in-progress)
27
FEAT-FUSION MSHC
0
20
40
60
80
100
Accuracy(%)
Ideal Self Induced

Thank you for your attention.
Questions?
Oresti Baños Legrán
Dep. Computer Architecture & Computer Technology
Faculty of Computer & Electrical Engineering (ETSIIT)
University of Granada, Granada (SPAIN)
Email: oresti@atc.ugr.es
Phone: +34 958 241 516
Fax: +34 958 248 993
Work supported in part by the HPC-Europa2 project funded by the European Commission - DG Research in the Seventh Framework Programme
under grant agreement no. 228398, the Spanish CICYT Project SAF2010-20558, Junta de Andalucia Project P09-TIC-175476 and the FPU
Spanish grant AP2009-2244.
28