The thesis focuses on enhancing human activity recognition (HAR) through robust deep sequential neural networks, addressing challenges such as accuracy, imbalanced data, and the need for less annotated data. It proposes methods including dilated causal convolution with multi-head self-attention and joint learning for cross-domain activity recognition to improve performance. Future work includes developing hybrid models and better attention mechanisms to recognize complex, multi-user activities.

1. Sequential Learning and Shared
Representation for Sensor-Based
Human Activity Recognition
Rebeen Ali Hamad
Supervisors: Prof. Longzhi Yang Prof. Wai Lok Woo Dr. Bo Wei
PhD Thesis
17- Noveomber-2022

2. Human Activity Recognition (HAR)
• Recognizing human activities based on sensor observation data.
• Intelligent environment and wearable sensor technologies

3. Motivation
HAR has become an active and essential research topic in ubiquitous computing due to its
usability in a variety of applications including healthcare, security.
This allows computing systems to automatically and remotely monitor and analyse
individuals' movements to assist them in their day-to-day tasks .

4. •Challenges of HAR
• Accuracy and Robustness in HAR: Accurately recognising human activities is difficult due
to the diversity and similarity of human activities
• Imbalanced Class Problems: human activities are inherently imbalanced since some
activities require longer time compared to other activities for example snack and
sleeping activities.
• The Need for Less Supervision Data: Acquiring a considerable portion of annotated data
to train a model is time-consuming, erroneous, and could even be impossible for some
scenarios since labelled data requires a domain knowledge expert to manually annotate
sensor recording observations of human activities

5. Objectives
i. Developing robust deep sequential neural
networks to further enhance the
performance of HAR systems compared to
the new state-of-the-art methods
ii. One of the objectives of this thesis is to
handle imbalanced class problems of HAR
systems from sensor data.
iii. Minimizing the need for large annotated
data for HAR systems

6. Entirely dispense the recurrent setting
Addressing imbalanced class problems
Reducing the need for large labeled data
Contributions

7. Dilated causal convolution with multi-head self-attention
To accelerate training time and improve the results of activity
recognition.
Causal convolutions is used to maintain the ordering of sensor
data and prevent information flow from future to past.
Dilated convolutions within the proposed method are used to
maximize the receptive field by orders of magnitude and
aggregate multi-scale contextual information without
considerably increasing computational cost.
Multi-head self-attention is used to effectively expose deep
semantic correlations from action sequences involving human
activities.
Hamad, Rebeen Ali, et al. "Dilated causal convolution with multi-head self attention for sensor human activity recognition." Neural Computing and Applications 33.20 (2021): 13705-13722.

9. ConvNet-based performers attention and supervised contrastive learning
• Supervised contrastive learning within the network is proposed to render expressive
representations
• The focal loss function based on the effective number of samples is proposed to down-weights
well-classified examples and focus on hard-classified examples.
Hamad, Rebeen Ali, et al. "ConvNet-based performers attention and supervised contrastive learning for activity recognition." Applied Intelligence (2022): 1-17.

10. Approximation of the regular attention mechanism AV via random feature maps.
Dashed blocks show the order of computation with corresponding time complexities

11. Resutls

12. Joint Learning of Temporal Models to Handle Imbalanced Data
• Aggregating different exposed
features in the proposed learning
method helps the classifier in
detecting rare activities and avoids
having models biased toward one
class or the other when compared to
an individual learner.
• During updating parameters i.e.,
model weights, both LSTM and 1D
CNN models contribute to adjust the
weights through the shared layer to
correctly map the input data to the
output class activities.
Hamad, Rebeen Ali, et al. "Joint learning of temporal models to handle imbalanced data for human activity recognition." Applied Sciences 10.15 (2020): 5293.

13. Results of the joint learning model

14. Cross-domain activity
recognition using
shared representation
• We develop a multi-domain
learning approach to jointly
learn activity recognition based
on different but related
domains rather than learning
each domain in isolation
Reducing the need and effort for labelled data of the target domain and reducing the training time by rendering a
generic model for related domains compared to fitting a model for each domain separately. Moreover, the proposed
network can also be used to train on small datasets as a target domain by supporting the source domain
Hamad, Rebeen Ali, et al. "Cross-Domain Activity Recognition Using Shared Representation in Sensor Data." IEEE Sensors Journal (2022).

15. t-SNE map for human activity

16. Results of cross domain learning model

17. Self-supervised Learning Based on Datasets with
Imbalanced Classes
self-supervised learning network for Human Activity Recognition (SHAR) that requires only unlabeled data to
formulate a pre-training model and learn a good representation of human activities for a downstream task i.e.
activity recognition.

18. Results of SHAR

19. Results of SHAR

20. Conclusion and Future works
• This thesis proposed robust and more accurate temporal sequential learning
models to enhance the performance of human activity recognition systems
compared with the existing approaches
• The thesis enhanced HAR, addressed imbalanced classed problems and
reduced the need for large labelled data

21. Future works
• Hybrid model of data-level and algorithm-level for human activity recognition.
• Better attention mechanism is required compared with the current
attention mechanism to precisely capture important information, particularly
for fine-grained human activities
• often human activity recognition is
performed for single-user activities at a time, however, in real-life scenarios,
concurrently multiple activities could be conducted by numerous
individual Recognizing multi-user activities and their interactions is still an
open research problem and requires further research
• Self-supervised multi-domain learning mode

23. Thank you!
• Any questions