presentation of Licentiate degree thesis https://www.diva-portal.org/smash/record.jsf?pid=diva2%3A1634036&dswid=3514

1. Towards Reliable, Stable and
Fast Learning for Smart
Home Activity Recognition
Rebeen Ali Hamad Supervisors: Thorsteinn Rögnvaldsson
Eric Jarpe
Mohamed-Rafik Bouguelia
Jens Lundstrom
February-24-2022
Licentiate Thesis

2. Layout
• Introduction
• Motivation
• Challenges of human activity recognition
• Research question
• Addressing activity recognition challenges
• Conclusion and future work

3. Activity Recognition (AR)
Human Activity Recognition (HAR) is a challenging and highly dynamic research
field aiming at recognizing human activities based on sensor observation data.
HAR, as one of the significant applications of intelligent environment and
wearable sensor technologies, can be used to monitor the activity of daily living
(ADL) to support and assist senior people, disabled and cognitive impaired

4. Motivation
• The aging and dependent population have been recognized as a major social
and economic challenge for the coming decades.
• One of the promising solutions to this challenge is ambient assisted living (AAL)
systems. Such systems aim to reduce the costs of healthcare and would enable
elders to live independently in their home.
• One of the most important roles and components of the AAL system is HAR
• HAR could be used to perform recognition of dangerous situations and detects
deviations of behavior to improve elderly-care alert systems

5. • Challenges of human activity recognition
• Labeling sensor readings
• Real-time constraints
• Diversity and frequency of human activity
• Types of activities
• Sensor challenges
• Number of activities and types of activities

6. Research question
• Considering the above challenges of human AR within smart home
environments, this thesis addresses the following research questions
i. How stable are low-dimensional maps of human activities in a smart home?
ii. How AR could be improved at the expense of real-time recognition?
iii. How to handle imbalanced class problems in the context of AR?
These research questions were investigated and relevant contributions were
made in the following sections and papers :

7. Addressing activity recognition challenges
• The long-term goal of the project is to process and share information across
multiple smart homes to reduce the learning time and data collection as well as
increase accuracy for HAR. One solution through machine learning development
is to use transfer learning to enhance systems ability.
• In our work, it is hypothesized that learned manifolds from disparate data sets
could be used for transfer learning. Therefore, it is crucial to investigate the
stability of t-SNE maps in order to properly align manifolds for the purpose of
transfer learning. Therefore, the first contribution of this thesis is investigating
stability of t-SNE maps.

8. Addressing activity recognition challenges
Stability analysis of the t-SNE algorithm for human activity pattern data
• Hypothesize: even if these smart homes are unique their data share a common
latent manifold which resides in a lower-dimensional subspace
• Manifold Alignment : by learning projections from each original space to the
shared manifold, correspondences of observations are recovered and knowledge
from one home can be transferred to another.
• t-SNE algorithm was used to be able to map human activity patterns from smart
home environments to a low-dimension manifold.

9. Stability analysis of the t-SNE algorithm
Despite the non-deterministic setup of the t-SNE algorithm, the visual
interpretation of different runs is easily compared by humans
How to analyze the stability of the t-SNE algorithm output. The proposed
approach utilizes comparisons of several output maps as a whole and partially by
clustered low-dimensional data points.
t-SNE map1 t-SNE map2 (same dataset)

10. Stability analysis of the t-SNE algorithm
Contribution: development of methods and
tools for studying t-SNE output stability on
smart home data used for modeling human
activity patterns
Linearly and non-linearly aligning low-
dimensional manifolds in order to compute
disparity and correct correspondence
observation within the five nearest
neighbors

11. Stability analysis of the t-SNE algorithm
Results of Procrustes analysis

12. Stability analysis of the t-SNE algorithm
LPA was introduced to non-linearly align manifolds by using locally linear mappings.
• It follows a divisive approach to cluster datasets. The algorithm starts by considering
a cluster of all data points and keeps on splitting into two sub-clusters recursively and
terminates if the diversity of a cluster is below a predetermined threshold.
• At each stage, PA is applied to all clusters in the first data set and the corresponding
cluster in the second dataset to compute disparity. If the disparity falls short of the
threshold, the clustering process stops for these clusters at this stage.

13. Stability analysis of the t-SNE algorithm
• Normalized Local Procrustes analysis: we propose an extension of the Local
Procrustes Analysis (LPA) technique to non-linearly align manifolds by using locally linear
mappings.
• The changes of the proposed method NLPA compared to the LPA procedure can be
summarized
• 1) Modification: Firstly the clustering algorithm is modified from k-means to
agglomerative clustering.
• 2) Improvement: Secondly the creating clustering criteria is improved to have two distinct
data points in each cluster and the threshold is minimized to render better alignment.
• 3) Extension: Finally, the NLPA is extended on LPA to normalize the transformed clusters in
order to the combined clusters with NLPA and the whole dataset with PA have a same space

14. Normalized Local Procrustes analysis(NLPA)
Results of PA
Results of NLPA

15. Results of Normalized local Procrustes Analysis

16. Stability analysis of the t-SNE algorithm
• Results of the experiments indicate that t-SNE low-dimensional manifolds are
locally stable which is part of the achievements of this research project.
• t-SNE is preserving the local geometry of the original high-dimensional data
• The long term goal of this research is to achieve automatic knowledge transfer
between related data sets from different smart homes

17. Contribution: Efficient AR in smart home data
using delayed fuzzy temporal windows
• We propose a data-driven approach that aims to delay the recognition process
and includes representations of binary sensor activations that occur before and
after the time where the prediction is made.
• For this, the proposed method uses multiple incremental fuzzy temporal
windows (FTWs) to extract features from both preceding and partial oncoming
sensor activations. To avoid the human configuration of FTWs we have modelled
their shapes with the Fibonacci sequence, which has been defined to model
incremental sequences in a harmonic way under the fields of mathematics,
science, and engineering

18. Efficient activity recognition in smart homes using
delayed fuzzy temporal windows on binary sensors
• Often HAR tasks are designed as sequential temporal learning
(LSTM, 1D ConvNet)
• The proposed method is evaluated with three temporal deep learning models
(CNN and LSTM Network as well as hybrid models combining CNN and LSTM), on
a binary sensor dataset of real daily living activities. The experimental evaluation
shows that the proposed method achieves significantly better results than the
real-time approach

19. Contribution: the second paper
• The proposed temporal models based on the FTWs achieves encouraging
performance particularly in the activities that real-time models have difficulties
recognizing accurately, such as Leaving, Snack, Grooming, and Toileting

20. Efficacy of Imbalanced Data Handling Methods on Deep
Learning for Smart Homes Environments.
• Human activities are highly diverse not only in the form of different sensor
activations but the frequency of activities themselves are inherently imbalanced
and hence accurate AR is challenging from a machine learning perspective.
• The main contribution of this paper is the study of well-known class imbalance
approaches (synthetic minority over-sampling technique, cost-sensitive learning
and ensemble learning) applied to activity recognition data with various
temporal data preprocessing for the deep learning models LSTM and 1D CNN

21. Contribution: Efficacy of Imbalanced Data
Handling Methods on Deep Learning
• The experimental results indicate that handling imbalanced data is more
important than selecting machine learning algorithms and improves
classification performance.

22. Potential questions for future research
• How to transfer knowledge between smart homes with a different layout,
sensor setting, and resident ? The aim of this research question is to exploit
what has been learned in one smart home to improve generalization in
different but related smart homes to reduce the need for labeling data.
• Often HAR tasks are designed as sequential temporal learning, (LSTM,
1D ConvNet)
• Considerable amounts of well-curated human activity data are needed, which
is a notably challenging task, hindered by privacy issues and labelling time and
cost.
• Hence it is a crucial research challenge to design a deep learning model that
can successfully learn to recognize human activities by leveraging only a small
number of annotated samples.

23. Conclusion
This thesis could answer the following questions
• i. How stable are low-dimensional maps of human activities in a smart home?
• ii. How AR could be improved at the expense of real-time recognition?
• iii. How to handle imbalanced class problems in the context of AR?

24. Thank you
• Questions !

25. Multiple incremental fuzzy temporal windows(FTWs)
• Ordonez datasets
These datasets comprise information regarding the ADLs performed by two users on a daily basis in their own
homes. These datasets are composed of two instances of data, each one corresponding to a different user
and summing up to 35 days of fully labelled data. Each instance of the dataset is described by three text files,
namely: description, sensors events (features), activities of daily living (labels). Sensor events were recorded
using a wireless sensor network and data were labelled manually.
• The first dataset is recorded from 28-11-2011 to 12-12-2011 called ‘OrdonezA’ ( 9 activities)
• The second dataset is recorded from 11-11-2012 to 2-12-2012 called OrdonezB ( 10 activities)
• An example of the data

26. Multiple incremental fuzzy temporal windows(FTWs)
• Generating datasets based on the sensor type + sensor location + sensor
place using fuzzy temporal windows.
• An example of all the time intervals of a sensor

27. FTWs
• Febon=[0, 0, 1,1,2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987]
• curent_tme = the_start_time
• L1=curent_tme + 60 * Febon[i]
• U1=curent_tme + 60 *Febon[i+1]
• U2=curent_tme + 60 *Febon[i+2]
• L3=curent_tme + 60 * Febon[i+3]
• curent_tme +60

28. FTWs
• In total we have 15 FTW, so if we have only one sensor time interval
then the dataset will have 15 dimension
in every minute from the started time(2011-11-28 02:27:59') we slide 15 fuzzy temporal winodws (FTWs) on the sensors
time interval. for example if we have one sensor with N time interval, then we slide the first FTW on all the time intervals and
then we select the maximum number to be the first feature of the sensor then we apply the second FTW on all the sensors
again and we select the maximum number to be second feature and we will continue untill we apply 15 FTW and we will
have 15 dimension for every minute
2011-11-28 02:27:59
2011-11-28 02:28:59
2011-11-28 02:29:59