Volearn is a motion-learning system that integrates virtual avatars and auditory feedback to enhance customized training and real-time performance feedback. It addresses limitations of conventional systems by providing personalized motion learning and effective feedback mechanisms. Evaluation results indicate that Volearn successfully reduces amplitude and speed errors during learning, though it still faces challenges like 3D motion reconstruction and the need for additional feedback elements.

1. VoLearn: A Cross-Modal Operable Motion-Learning
System Combined with Virtual Avatar and Auditory
Feedback
2022 ACM Ubicomp (IMWUT)
Chengshuo Xia1, Xinrui Fang1, Riku Arakawa2, Yuta Sugiura1
1. Keio University, Japan
2. Carnegie Mellon University, United States

2. 2
Background: less feedback and non-customized learning
Conventional motion-learning systems.
The user imitates the subject in the video.
Lack of effective motion understanding
and real-time feedback.

3. 3
Background: non-customized learning
Based on the determined motion intensity. Not allow for personalized and customized programs.

4. 4
Our goal:
• More individuals can have customized motion learning/training process
Effective
feedback
Customized
motion
learning
• Effective real-feedback to improve the behavior

5. 5
Related Work: motion tutorials
[1] FORME
[2] Natsuki Hamanishi and Jun Rekimoto. 2020. PoseAsQuery: Full-Body Interface for Repeated Observation of a Person in a Video with Ambiguous Pose Indexes and Performed Poses. In Proceedings of the
Augmented Humans International Conference. ACM, New York, NY, USA, 1–11.
[3] Christopher Clarke, Doga Cavdir, Patrick Chiu, Laurent Denoue, and Don Kimber. 2020. Reactive Video: Adaptive Video Playback Based on User Motion for Supporting Physical Activity. In Proceedings of the 33rd
Annual ACM Symposium on User Interface Software and Technology. ACM, New York, NY, USA, 196–208.
Exercise mirro [1] PoseAsQuert [2] ReactiveVideo [3]

6. 6
2. Related Work: feedback based on error
[1] Maurício Sousa, João Vieira, Daniel Medeiros, Artur Arsenio, and Joaquim Jorge. 2016. SleeveAR: Augmented reality for rehabilitation using realtime feedback. In Proceedings of the 21st international conference
on intelligent user interfaces. ACM, New York, NY, USA, 175–185.
[2] Richard Tang, Xing-Dong Yang, Scott Bateman, Joaquim Jorge, and Anthony Tang. 2015. Physio@ Home: Exploring visual guidance and feedback techniques for physiotherapy exercises. In Proceedings of the 33rd
Annual ACM Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 4123–4132.
[3] MilkaTrajkova andFrancesco Cafaro. 2018. Takes Tutu to ballet: designing visual and verbal feedback for augmented mirrors. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous
Technologies 2, 1 (2018), 1–30.
Exercise mirro [1] Physio@Home [2]: Tutu[3]:

7. Proposed system
8
VoLearn:
• Motion registration function
• 3D motion editing for cutomized motion learning
• Smartphone-based auditory feedback

8. Compared with related systems
System Feedback Application Daily MoCap Motion
Register
Motion
modify
NursCare Vibration Transfer the patient
Physio@Home Vision Physiotherapy
Ayode et al. Vision Knee rehabilitation
Schonauer et al. Vision/vibration General exercise
YouMove None General exercise
Buttussi et al. Vision Fitness activities
Pose Trainer Vision Posture training
Tutu Vision Ballet dance
SleeveAR Vision Arm rehabilitation
Onebody Vision General exercises
My Tai-Chi Vision Tai Chi Chuan
LightGuide Vision Hand motion
Just Follow me Vision Gneral exercises
MotionMA Vision Gneral exercises
COPD Trainer Audio COPD patients
Nintendo Fit Ring Audio Exergames
Prosed system -
VoLearn
Audio Gneral exercises
9

9. System Design: motion editing interface
11

10. System Design: generation of reference data
12
Motion sequences
Each body part’s
motion intensity
Compare
Maximum body
part
Place the
smartphone

11. System Design: mapping the virtual motion data with
real motion data
13
Wearing position mapping Sensor coordinate mapping Sampling rate mapping

12. System Design: smartphone-based auditory feedback
14

13. 15
Input: motion video and smartphone-based rotation data
Output: virtual sensor (rotation) data
Aim: test the initial error between input motion data and obtained virtual motion data (with no editing)
10 exercise motions selected from YouTube
Evaluation: Task 1 End-to-end test of generated motion
data

14. 16
Motion 1 2 3 4 5
Suggested Sensor
Position
Left upper
arm
Left upper
leg
Left upper
leg
Waist Right upper
leg
RMSE (degree) 14.93 9.62 19.29 5.61 15.52
Amplitude Error
(degree)
0.08 4.07 12.76 6.04 1.35
Motion 6 7 8 9 10
Suggested Sensor
Position
Left upper
leg
Left upper
leg
Left upper
leg
Left upper
leg
Right forearm
RMSE (degree) 3.79 15.48 14.02 16.58 56.15
Amplitude Error
(degree)
2.91 12.27 5.33 8.81 17.89
Average: RMSE --- 17.01; Amplitude Error --- 7.18
Evaluation: Task 1 End-to-end test of generated motion
data

15. Evaluation: Task 2 - User study: information
17
Scenario :
o Scenario(a):
o Scenario(b):
Aim: test effect of designed auditory feedback

16. 18
Participant:
o 20 participants (4 female and 16 male)
o $10
Evaluation: Task 2 - User study: information
Motions:
o Six motions from Task 1
Methodology :
o Aligned Rank Transform (ART) ANOVA
o Mann-Whitney U test
Hypothesis:
amplitude error time error
o H1a: in scenario (a)
o H1b: in scenario (a)
o H2: more effective for
complex motion

17. 19
Evaluation: Task 2 - User study: protocol

18. 20
Evaluation: Task 2 - User study: results

19. 21
Evaluation: Task 2 - User study: results

20. 22
Evaluation: Task 2 - User study: users' comments
“Gives me a goal”
Encourages me and
motivates me
“Remind me”
“Makes it not boring”

21. Evaluation: Task 3 - Evaluation by Professional Users
23
Gym coach:
Design a motion for
the student
Helpful for remote
learning
More limbs can be tracked

22. Evaluation: Task 3 - Evaluation by Professional Users
24
Helps patients
(especially for laying on
the bed)
Motion design good
for different patients
Standard motion
library
Clinical Nurse :

23. Evaluation: Task 3 - Evaluation by Professional Users
25
Relieve burdens
Better for remote
learning
Therapist :

24. Discussion and Limitation
26
High-performance 3D motion reconstruction
Current characteristics
Better for support mode (user should be familiar with motions)
One limb tracking
Further motion edition
More informative feedback with audio
Long learning effect
Limitation :

25. 27
Summary
Background
Current motion-learning system lacks real-time feedback and
customized function
Related Work Motion tutorial systems/feedback design
Proposed
Scheme
VoLearn: a cross-modal motion-learning system combined
virtual avatar and auditory feedback
Details
Method
Motion registration/modification/reference data
generation/auditory feedback
Experiment Three tasks
Result
VoLearn can help to reduce the amplitude and speed error
during learning
Limitation
3D motion reconstruction/more feedback elements/long-
term effect

26. Thank you very much!