This document proposes a method to augment virtual IMU data using a spring-joint model for motion exercise recognition without real data. The method aims to address the limitation of current virtual IMU data which has a limited motion length. It introduces a spring-joint virtual sensor module that can simulate different acceleration distributions and augment the virtual acceleration data spatially and temporally. An experiment tested the method on three aerobic exercises and showed the proposed data augmentation approach improved motion recognition accuracy from 45.5% to 85.3% compared to non-augmented virtual data.

1. Virtual IMU Data Augmentation by Spring-Joint
Model for Motion Exercises Recognition without
Using Real Data
2022 ACM ISWC (note)
Chengshuo Xia, Yuta Sugiura
Keio University, Japan

2. 2
Background: real IMU dataset for motion recognition
Participants
Wearing
sensor
Produce the
data
Real wearable IMU sensor data collection: time-costly and expensive.
Opportunity dataset [1]
[1] Daniel Roggen, Alberto Calatroni, Mirco Rossi, Thomas Holleczek, Gerhard Tröster, Paul Lukowicz, Gerald Pirkl, David Bannach, Alois Ferscha, Jakob Doppler, Clemens Holzmann, Marc Kurz, Gerald Holl, Ricardo Chavarriaga, Hesam Sagha,
Hamidreza Bayati, and José del R. Millàn. "Collecting complex activity data sets in highly rich networked sensor environments" In Seventh International Conference on Networked Sensing Systems (INSS’10), Kassel, Germany, 2010.
Dataset
Dataset
Machine
learning
model
Application
Change
Real world
A motion recognition system: machine learning (ML).

3. 3
Background: virtual IMU sensor data in machine learning
Virtual sensor data generation help to reduce the dataset cost.
Dataset
Machine
learning
model
Application Change
Virtual sensor
data
Simulation envrionment
Virtual IMU sensor data extraction from 3D human motion:

4. 4
Background: current problems in virtual data
• Limited 3D motion length.
• Obtaining a longer 3D motion length still is time-consuming.
• Real data distribution is needed.

5. 5
Proposed Method: spring-joint module-based data augmentation
• Spring-joint virtual IMU sensor module.
• Used for training the classifier.
• Recognize the real IMU data.
More virtual sensor
data
Spring-Joint virtual
sensor module
3D motion
Training the ML
model

6. 6
Related Work: virtual sensor data for human motion recognition
Virtual IMU data: IMUTube [1]
Virtual Doppler data: Vid2Doppler [2]
Virtual distance data [3]
[1] Kwon, Hyeokhyen, et al. "IMUTube: Automatic extraction of virtual on-body accelerometry from video for human activity recognition." Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4.3 (2020): 1-29.
[2] Ahuja, Karan, et al. "Vid2Doppler: synthesizing Doppler radar data from videos for training privacy-preserving activity recognition." Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 2021.
[3] Xia, Chengshuo, Ayane Saito, and Yuta Sugiura. "Using the virtual data-driven measurement to support the prototyping of hand gesture recognition interface with distance sensor." Sensors and Actuators A: Physical 338 (2022): 113463.

7. 7
Related Work: data augmentation for time-series
[1] Li, Xi'ang, Jinqi Luo, and Rabih Younes. "ActivityGAN: Generative adversarial networks for data augmentation in sensor-based human activity recognition." Adjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of
the 2020 ACM International Symposium on Wearable Computers. 2020.
[2] Um, Terry T., et al. "Data augmentation of wearable sensor data for parkinson’s disease monitoring using convolutional neural networks." Proceedings of the 19th ACM international conference on multimodal interaction. 2017.
GAN-based [1]: Time-series manipulation [2]:
Traditional augmentation: operate on real data
Our method: augment the virtual sensor data

8. 8
Overview of Proposed Method
More virtual
sensor data
Spring-joint virtual
IMU sensor
module
Machine learning
model
3D human
motion
Recognize the
real motion

9. 9
Method: how to obtain the virtual acceleration data
Posistion
Velocity
Acceleration
[1] Young, Alexander D., Martin J. Ling, and Damal K. Arvind. "IMUSim: A simulation environment for inertial sensing algorithm design and evaluation." Proceedings of the 10th ACM/IEEE International Conference on Information
Processing in Sensor Networks. IEEE, 2011.

10. 10
Method: spatial and temporal augmentation
Spatial Augmentation:
Spring-Joint-based
Sensor Module
Temporal Augmentation:
Playback Speed

11. 11
Method: spatial and temporal augmentation
Spatial Augmentation:
Spring-Joint-based
Sensor Module
Temporal Augmentation:
Playback Speed

12. 12
Method: spring-joint module design
1 module  consists of 3 nodes

13. 13
Method: adding multiple modules
(The modules follow the movement of Right Upper Leg)

14. 14
Method: the nodes connection and parameters
Different stiffness of spring-joint: Different structure of spring-joint:
Choose four stiffness 0.2, 0.4, 0.6, 0.8 Two types of connection structure

15. 15
Method: spatial and temporal augmentation
Spatial Augmentation:
Spring-Joint-based
Sensor Module
Temporal Augmentation:
Playback Speed

16. 16
Method: data augmentation by motion speed shift
Speed adjustment:
access the time scale to shift the motion speed

17. 17
Method: coordinate transformation
Virtual
acceleration
Real acceleration
Same sensor
coordinate
Coordinate
transformation
Coordinate
transformation
Fusion &
Normalization
z
y
x
Uniform coordinate

18. 18
Experiment: validation method
Virtual
dataset
(initial):
Real
dataset
Purpose: validate of effeteness of proposed method
Virtual dataset
(with
augmentation):
Real
dataset
Real
dataset
Classifier
Result
Real
dataset
Train Test
Classifier
Result
Train Test
Classifier
Result
Train Test
Baseline 1: Baseline 2:
Proposed method:
Leave-one-subject-out

19. Experiment: tested exercise motions
Test motions: 3 types of aerobic exercises from YouTube online sources, 20 seconds
3D motion conversion: DeepMotion [1]
19
[1] https://www.deepmotion.com/
Reverse Lunge Warm Up High Knee Tap

20. 21
Experiment: information
• Real sensor: Xsens Dot [1]
• Participants: 7 people
Conduct the exercises by 90 seconds
• Classifiers: SVM, Random Forest, Decision Tree
• Handcrafted features:
• Sensor Location: Right upper leg
Time & Frequency features
2-dimension features
PCA
[1] https://www.xsens.com/xsens-dot
• Virtual dataset (initial):
• 20s * 3 motions
• Virtual dataset (with augmentation):
• 20s * 8 modules * 3 nodes * 3 speeds * 3 motions
• Real dataset: for testing
• 90s * 3 motions * 7 participants

21. 22
Experiment: Result
PCA-1
PCA-2
Virtual ReverseLunge
Real ReverseLunge
Real HighKnee
Virtual HighKnee
Real HighKnee
Virtual HighKnee
Method
(Random
Forest)
Baseline 1
(initial
virutal
data)
Baseline 2
(using
real data)
Proposed
Method
(spring-
joint
module)
Accuracy 45.5 % 78.2 % 85.3 %

22. 23
Discussion and Limitation
Further evaluation of different physical simulation parameters
Test on more motions
Virtual-to-real data features -> domain-invariant features

23. 24
Summary
Background
Virtual IMU data suffers from limited 3D motion length,
which leads to less virtual data samples
Related Work Virtual sensor data/Time-series data augmentation
Proposed
Scheme
Using the spring-joint module to simulate different
acceleration distribution and augment virtual acceleration
data
Details
Method
Spring-joint connection/playback speed adjust/domain
adaption
Experiment Tested on there aerobic exercises picked from YouTube
Result
Using proposed augmentation method can get 85% vs. 45%
from not augmented
Limitation More motions tested/Boundary condition test
virtual acceleration 3 virtual acceleration 1
virtual acceleration 2
real acceleration

24. Thank you very much!