Delivered at the ICCBR 2019 CBR-DL Workshop

1. Human Activity Recognition with
Deep Metric Learners
Kyle Martin and Anjana Wijekoon and Nirmalie Wiratunga

2. Human Activity Recognition (HAR)
• Computational discovery of human activity from sensor
data
• Increasing applications in Health and fitness

3. HAR for Machine Learning
windowing
Feature Representation
Classifier

4. HAR for Machine Learning
windowing
Feature Representation
Classifier

5. Exploiting Similarity for Representations
Personalisation Open-ended HAR
DeploymentTraining

6. Deep Metric Learners (DML)
• Branch of deep learning architectures
• Goal – to learn a feature space optimised for similarity-based return
• Distance or similarity-based objective functions
• Train on multiple cases simultaneously
• The learned representation incorporates knowledge of case relationships
• I.e. Matching – Non-matching, Cluster membership

7. Siamese Neural Networks (SNN)
• Two neural networks which share identical weights and are joined at one or more layers.
• Cases are paired and labelled as either positive or negative
• i.e. Matching pairs and Non-Matching pairs
• The network learns a representation such that matching pair members exist close
together, while non-matching pair members exist at least a margin apart
[Bromley, J. et al. Signature Verification Using a Siamese Time Delay Neural Network 1993]

8. Triplet Network (TN)
• Three neural networks which share identical weights and are joined at one or more layers.
• Cases are put into triplets of [anchor, positive, negative]
• I.e. [Anchor, Matching, Non-matching]
• The goal is to ensure that the distance between the anchor and the matching instance is smaller
than the distance between the anchor and the non-matching instance + margin
[Hoffer, E. et al. Deep Metric Learning Using Triplet Network 2014]

9. Matching Network
• Not specifically a deep metric learner,
but can be adapted
• Relies on comparison of a query case to a
group of cases from a class
• Instead of class, we can use cluster
representatives
• Considers batches of cases as input at
once
• Split into training and support sets
• The goal is to maximise the similarity of
a training cases to the desired
class/cluster within the support set
[Vinyals, O. et al. Matching Networks for One-Shot Learning 2016]
𝛼

10. Evaluation
• Comparative study of the different DML architectures
• Objective was to compare the performance of different DML on HAR
• We compared three DML architectures on three HAR tasks
• SelfBACK, MEx and PAMAP2
• Accuracy with kNN classifier as a proxy for representation goodness
• kNN on DCT representations acted as baseline

11. Datasets
SelfBACK Mex PAMAP2
Accelerometer
Placement
Wrist and Thigh Wrist and Thigh Wrist, Chest and
Ankle
Sampling rate 100Hz 100Hz 100Hz
Accelerometer
Range
+/- 8g +/- 8g +/-16g
Activities 9
Ambulatory and
Sedentary
7
Physiotherapy
18
Activities of Daily
Living
Ambulatory
Sedentary
Users 33 30 8

12. Pre-Processing and Network Parameters
• We concatenate DCT
representations of each
sensor to form the input
features.
• Training - Each model
trained for 10 epochs with
Adam Optimiser (lr=0.01)
• Evaluation – 3-NN with
cosine similarity.
Dense
1200,Relu
Input
(featurelength,1)
Output
(1200)
BatchNorm
1DConv
(3),12,Relu
1DMaxPool
(2)
BatchNorm
Flatten
Dense
1200,Relu
Input
(featurelength,1)
Output
(1200)
BatchNorm
CNN
MLP

13. Results
SelfBACK
kNN
SNN
TN
MN

14. Results
SelfBACK
kNN
SNN
TN
MN
76.70
80.41
81.34
80.60
81.57
88.35
87.01
MLP
CNN
MLP
CNN
MLP
CNN

15. Results
MEx
kNN
SNN
TN
MN

16. Results
MEx
kNN
SNN
TN
MN
68.56
77.27
72.31
79.32
79.27
94.19
95.49
MLP
CNN
MLP
CNN
MLP
CNN

17. Results
PAMAP2
kNN
SNN
TN
MN

18. Results
PAMAP2
kNN
SNN
TN
MN
81.40
82.14
85.88
86.80
87.32
80.72
81.77
MLP
CNN
MLP
CNN
MLP
CNN

19. Conclusions
• We have discussed the capability of DMLs to perform HAR tasks
• We have performed a comparative study to evaluate their
performance on these tasks
• Results are indicative DMLs develop a space well optimised for
similarity
• Particularly when more cases are considered simultaneously
Thank you for listening! Questions?