This document presents a thesis on using recurrent neural networks for skeleton-based human action recognition. The proposed method uses two RNNs - a temporal RNN to model the temporal dynamics of joints over time, and a spatial RNN to model the dependencies between joints spatially. The RNNs are trained on skeleton data extracted from video datasets like NTU RGB+D and Kinetics. Experimental results show the method achieves state-of-the-art accuracy on the NTU datasets and can recognize actions in real-time from new video inputs. Future work involves exploring more advanced temporal modeling and evaluating on larger datasets.

1. Skeleton-based Human Action Recognition
with Recurrent Neural Network
University of Science, VNU-HCM
Faculty of Information Technology
Advanced Program in Computer Science
Võ Trần Thanh Lương
1551020
Vũ Hoàng Quân
1551026
Thesis Advisors:
Dr. Trần Thái Sơn
Ho Chi Minh City
Aug 18th
2019

2. Outlines
●
●
●
●
●
●
●

3. Introduction
Every human action is done to serve a purpose. Machines should be able to learn and understand it.

4. Introduction

5. Introduction
●
●
○
○
○
○
●
●

6. Introduction
●
○
○

7. Motivation
●
○
○
○
○
○
○ …

8. ●
Motivation

9. Contributions
●
●
●
●

10. Related Work
● ●

11. Related Work
●
○
○
Zhuowen Lv 1, Xianglei Xing 1,, Kejun Wang 1, and Donghai Guan 2 , "Class
Energy Image Analysis for Video Sensor-Based Gait Recognition: A Review,"
An example of local representations for human action

12. Related Work
●
○
○
Georgios D. Evangelidis, Gurkirt Singh, Radu
Horaud, "Continuous Gesture Recognition from
Articulated Poses,"
Spatial Temporal interest points. S.F. Wong, T.-K. Kim, and R. Cipolla, "Learning motion
categories using both semantic and structural information”

13. Related Work
●
○
○
■
■
Example of 3D convolution. Karen Simonyan & Andrew Zisserman , "Two-Stream Convolutional Networks for Action Recognition in Videos,"

14. Related Work
●
○
○
Hybrid network for temporal modeling. Lisa Anne Hendricks, Marcus Rohrbach,
Subhashini Venugopalan, Sergio Guadarrama, Kate Saenko, Trevor Darrell,
"Long-term recurrent convolutional networks for visual recognition and description,"
Jun Liu, Gang Wang, Ling-Yu Duan, Kamila Abdiyeva, Alex C. Kot, "Skeleton-Based Human Action
Recognition with Global Context-Aware Attention LSTM Networks,

15. Proposed Method
Temporal RNN
Spatial RNN

16. Temporal RNN
●
●
●

17. Temporal RNN
●
●
●
○
○

20. Spatial RNN
●
●
●
●

21. Spatial RNN

22. Spatial RNN
●
●
●

23. Spatial RNN

24. Spatial RNN

25. 3D Transformation

26. Training Flow
Skeleton
Dataset
Training
Set
Testing
Set
Initialize
RNN
Feature
Extraction
Training
Softmax
Classification
Trained
Model
NTU
Dataset
Kinetics
Dataset
Raw
video
Extract Skeleton
Data
Using OpenPose

27. Predicted Flow
Raw Video
Extract Skeleton
Data
Load Trained
Model
Predicted Value
Using OpenPose

28. Experiments
●
○
○
○

29. NTU RGB+D and NTU RGB+D 120 Dataset
●
●
●
●

31. Skeleton Joints Position (NTU Dataset)

32. Kinetics Dataset
●
●
●

33. Sample frames of Kinetics Dataset

34. Skeleton Joints Position (Coco Model)

35. Proposed Method Result

36. Comparison with the state-of-the-art

37. Accuracy Calculation

38. Problems with Kinetics dataset
NTU RGB+D NTU RGB+D 120 Kinetics
Raw Videos Yes Yes No (can obtain
from given URLs)
3D skeleton data Yes Yes No
Depth maps Yes Yes No

39. Problems with Kinetics dataset
○
○

40. Demo

41. Conclusion
●
●
●
●

42. Future Work
●
●
●
Jun Liu, Gang Wang, Ling-Yu Duan, Kamila Abdiyeva, Alex C. Kot, "Skeleton-Based Human
Action Recognition with Global Context-Aware Attention LSTM Networks,
ASPR Framework. Liu, Jun and Shahroudy, Amir and Perez, Mauricio and Wang, Gang and
Duan, Ling-Yu and Kot, Alex C., "NTU RGB+D 120: A Large-Scale Benchmark for 3D
Human Activity Understanding,"

43. THANK YOU FOR YOUR ATTENTION
University of Science, VNU-HCM
Faculty of Information Technology
Advanced Program in Computer Science
Võ Trần Thanh Lương
1551020
Vũ Hoàng Quân
1551026
Thesis Advisors:
Dr. Trần Thái Sơn
Ho Chi Minh City
Aug 18th
2019