Virtual IMU Data Augmentation by Spring-Joint Model for Motion Exercises Recognition without Using Real Data

1. Virtual IMU Data Augmentation by Spring-Joint Model for Motion Exercises Recognition without Using Real Data 2022 ACM ISWC (note) Chengshuo Xia, Yuta Sugiura Keio University, Japan

2. 2 Background: real IMU dataset for motion recognition Participants Wearing sensor Produce the data Real wearable IMU sensor data collection: time-costly and expensive. Opportunity dataset [1] [1] Daniel Roggen, Alberto Calatroni, Mirco Rossi, Thomas Holleczek, Gerhard Tröster, Paul Lukowicz, Gerald Pirkl, David Bannach, Alois Ferscha, Jakob Doppler, Clemens Holzmann, Marc Kurz, Gerald Holl, Ricardo Chavarriaga, Hesam Sagha, Hamidreza Bayati, and José del R. Millàn. "Collecting complex activity data sets in highly rich networked sensor environments" In Seventh International Conference on Networked Sensing Systems (INSS’10), Kassel, Germany, 2010. Dataset Dataset Machine learning model Application Change Real world A motion recognition system: machine learning (ML).

3. 3 Background: virtual IMU sensor data in machine learning Virtual sensor data generation help to reduce the dataset cost. Dataset Machine learning model Application Change Virtual sensor data Simulation envrionment Virtual IMU sensor data extraction from 3D human motion:

4. 4 Background: current problems in virtual data • Limited 3D motion length. • Obtaining a longer 3D motion length still is time-consuming. • Real data distribution is needed.

5. 5 Proposed Method: spring-joint module-based data augmentation • Spring-joint virtual IMU sensor module. • Used for training the classifier. • Recognize the real IMU data. More virtual sensor data Spring-Joint virtual sensor module 3D motion Training the ML model

6. 6 Related Work: virtual sensor data for human motion recognition Virtual IMU data: IMUTube [1] Virtual Doppler data: Vid2Doppler [2] Virtual distance data [3] [1] Kwon, Hyeokhyen, et al. "IMUTube: Automatic extraction of virtual on-body accelerometry from video for human activity recognition." Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4.3 (2020): 1-29. [2] Ahuja, Karan, et al. "Vid2Doppler: synthesizing Doppler radar data from videos for training privacy-preserving activity recognition." Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 2021. [3] Xia, Chengshuo, Ayane Saito, and Yuta Sugiura. "Using the virtual data-driven measurement to support the prototyping of hand gesture recognition interface with distance sensor." Sensors and Actuators A: Physical 338 (2022): 113463.

7. 7 Related Work: data augmentation for time-series [1] Li, Xi'ang, Jinqi Luo, and Rabih Younes. "ActivityGAN: Generative adversarial networks for data augmentation in sensor-based human activity recognition." Adjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers. 2020. [2] Um, Terry T., et al. "Data augmentation of wearable sensor data for parkinson’s disease monitoring using convolutional neural networks." Proceedings of the 19th ACM international conference on multimodal interaction. 2017. GAN-based [1]: Time-series manipulation [2]: Traditional augmentation: operate on real data Our method: augment the virtual sensor data

8. 8 Overview of Proposed Method More virtual sensor data Spring-joint virtual IMU sensor module Machine learning model 3D human motion Recognize the real motion

9. 9 Method: how to obtain the virtual acceleration data Posistion Velocity Acceleration [1] Young, Alexander D., Martin J. Ling, and Damal K. Arvind. "IMUSim: A simulation environment for inertial sensing algorithm design and evaluation." Proceedings of the 10th ACM/IEEE International Conference on Information Processing in Sensor Networks. IEEE, 2011.

10. 10 Method: spatial and temporal augmentation Spatial Augmentation: Spring-Joint-based Sensor Module Temporal Augmentation: Playback Speed

12. 12 Method: spring-joint module design 1 module  consists of 3 nodes

13. 13 Method: adding multiple modules (The modules follow the movement of Right Upper Leg)

14. 14 Method: the nodes connection and parameters Different stiffness of spring-joint: Different structure of spring-joint: Choose four stiffness 0.2, 0.4, 0.6, 0.8 Two types of connection structure

16. 16 Method: data augmentation by motion speed shift Speed adjustment: access the time scale to shift the motion speed

17. 17 Method: coordinate transformation Virtual acceleration Real acceleration Same sensor coordinate Coordinate transformation Coordinate transformation Fusion & Normalization z y x Uniform coordinate

18. 18 Experiment: validation method Virtual dataset (initial): Real dataset Purpose: validate of effeteness of proposed method Virtual dataset (with augmentation): Real dataset Real dataset Classifier Result Real dataset Train Test Classifier Result Train Test Classifier Result Train Test Baseline 1: Baseline 2: Proposed method: Leave-one-subject-out

19. Experiment: tested exercise motions Test motions: 3 types of aerobic exercises from YouTube online sources, 20 seconds 3D motion conversion: DeepMotion [1] 19 [1] https://www.deepmotion.com/ Reverse Lunge Warm Up High Knee Tap

20. 21 Experiment: information • Real sensor: Xsens Dot [1] • Participants: 7 people Conduct the exercises by 90 seconds • Classifiers: SVM, Random Forest, Decision Tree • Handcrafted features: • Sensor Location: Right upper leg Time & Frequency features 2-dimension features PCA [1] https://www.xsens.com/xsens-dot • Virtual dataset (initial): • 20s * 3 motions • Virtual dataset (with augmentation): • 20s * 8 modules * 3 nodes * 3 speeds * 3 motions • Real dataset: for testing • 90s * 3 motions * 7 participants

21. 22 Experiment: Result PCA-1 PCA-2 Virtual ReverseLunge Real ReverseLunge Real HighKnee Virtual HighKnee Real HighKnee Virtual HighKnee Method (Random Forest) Baseline 1 (initial virutal data) Baseline 2 (using real data) Proposed Method (spring- joint module) Accuracy 45.5 % 78.2 % 85.3 %

22. 23 Discussion and Limitation Further evaluation of different physical simulation parameters Test on more motions Virtual-to-real data features -> domain-invariant features

23. 24 Summary Background Virtual IMU data suffers from limited 3D motion length, which leads to less virtual data samples Related Work Virtual sensor data/Time-series data augmentation Proposed Scheme Using the spring-joint module to simulate different acceleration distribution and augment virtual acceleration data Details Method Spring-joint connection/playback speed adjust/domain adaption Experiment Tested on there aerobic exercises picked from YouTube Result Using proposed augmentation method can get 85% vs. 45% from not augmented Limitation More motions tested/Boundary condition test virtual acceleration 3 virtual acceleration 1 virtual acceleration 2 real acceleration

24. Thank you very much!

Virtual IMU Data Augmentation by Spring-Joint Model for Motion Exercises Recognition without Using Real Data

Virtual IMU Data Augmentation by Spring-Joint Model for Motion Exercises Recognition without Using Real Data

Recommended

More Related Content

Similar to Virtual IMU Data Augmentation by Spring-Joint Model for Motion Exercises Recognition without Using Real Data

Similar to Virtual IMU Data Augmentation by Spring-Joint Model for Motion Exercises Recognition without Using Real Data(20)

More from sugiuralab

More from sugiuralab(20)

Recently uploaded

Recently uploaded(20)

Virtual IMU Data Augmentation by Spring-Joint Model for Motion Exercises Recognition without Using Real Data