Advertisement

VoLearn: A Cross-Modal Operable Motion-Learning System Combined with Virtual Avatar and Auditory Feedback

Oct. 24, 2022
Advertisement

More Related Content

Similar to VoLearn: A Cross-Modal Operable Motion-Learning System Combined with Virtual Avatar and Auditory Feedback(20)

More from sugiuralab(20)

Advertisement

VoLearn: A Cross-Modal Operable Motion-Learning System Combined with Virtual Avatar and Auditory Feedback

  1. VoLearn: A Cross-Modal Operable Motion-Learning System Combined with Virtual Avatar and Auditory Feedback 2022 ACM Ubicomp (IMWUT) Chengshuo Xia1, Xinrui Fang1, Riku Arakawa2, Yuta Sugiura1 1. Keio University, Japan 2. Carnegie Mellon University, United States
  2. 2 Background: less feedback and non-customized learning Conventional motion-learning systems. The user imitates the subject in the video. Lack of effective motion understanding and real-time feedback.
  3. 3 Background: non-customized learning Based on the determined motion intensity. Not allow for personalized and customized programs.
  4. 4 Our goal: • More individuals can have customized motion learning/training process Effective feedback Customized motion learning • Effective real-feedback to improve the behavior
  5. 5 Related Work: motion tutorials [1] FORME [2] Natsuki Hamanishi and Jun Rekimoto. 2020. PoseAsQuery: Full-Body Interface for Repeated Observation of a Person in a Video with Ambiguous Pose Indexes and Performed Poses. In Proceedings of the Augmented Humans International Conference. ACM, New York, NY, USA, 1–11. [3] Christopher Clarke, Doga Cavdir, Patrick Chiu, Laurent Denoue, and Don Kimber. 2020. Reactive Video: Adaptive Video Playback Based on User Motion for Supporting Physical Activity. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. ACM, New York, NY, USA, 196–208. Exercise mirro [1] PoseAsQuert [2] ReactiveVideo [3]
  6. 6 2. Related Work: feedback based on error [1] Maurício Sousa, João Vieira, Daniel Medeiros, Artur Arsenio, and Joaquim Jorge. 2016. SleeveAR: Augmented reality for rehabilitation using realtime feedback. In Proceedings of the 21st international conference on intelligent user interfaces. ACM, New York, NY, USA, 175–185. [2] Richard Tang, Xing-Dong Yang, Scott Bateman, Joaquim Jorge, and Anthony Tang. 2015. Physio@ Home: Exploring visual guidance and feedback techniques for physiotherapy exercises. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 4123–4132. [3] MilkaTrajkova andFrancesco Cafaro. 2018. Takes Tutu to ballet: designing visual and verbal feedback for augmented mirrors. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 1 (2018), 1–30. Exercise mirro [1] Physio@Home [2]: Tutu[3]:
  7. Proposed system 8 VoLearn: • Motion registration function • 3D motion editing for cutomized motion learning • Smartphone-based auditory feedback
  8. Compared with related systems System Feedback Application Daily MoCap Motion Register Motion modify NursCare Vibration Transfer the patient Physio@Home Vision Physiotherapy Ayode et al. Vision Knee rehabilitation Schonauer et al. Vision/vibration General exercise YouMove None General exercise Buttussi et al. Vision Fitness activities Pose Trainer Vision Posture training Tutu Vision Ballet dance SleeveAR Vision Arm rehabilitation Onebody Vision General exercises My Tai-Chi Vision Tai Chi Chuan LightGuide Vision Hand motion Just Follow me Vision Gneral exercises MotionMA Vision Gneral exercises COPD Trainer Audio COPD patients Nintendo Fit Ring Audio Exergames Prosed system - VoLearn Audio Gneral exercises 9
  9. System Design: motion editing interface 11
  10. System Design: generation of reference data 12 Motion sequences Each body part’s motion intensity Compare Maximum body part Place the smartphone
  11. System Design: mapping the virtual motion data with real motion data 13 Wearing position mapping Sensor coordinate mapping Sampling rate mapping
  12. System Design: smartphone-based auditory feedback 14
  13. 15 Input: motion video and smartphone-based rotation data Output: virtual sensor (rotation) data Aim: test the initial error between input motion data and obtained virtual motion data (with no editing) 10 exercise motions selected from YouTube Evaluation: Task 1 End-to-end test of generated motion data
  14. 16 Motion 1 2 3 4 5 Suggested Sensor Position Left upper arm Left upper leg Left upper leg Waist Right upper leg RMSE (degree) 14.93 9.62 19.29 5.61 15.52 Amplitude Error (degree) 0.08 4.07 12.76 6.04 1.35 Motion 6 7 8 9 10 Suggested Sensor Position Left upper leg Left upper leg Left upper leg Left upper leg Right forearm RMSE (degree) 3.79 15.48 14.02 16.58 56.15 Amplitude Error (degree) 2.91 12.27 5.33 8.81 17.89 Average: RMSE --- 17.01; Amplitude Error --- 7.18 Evaluation: Task 1 End-to-end test of generated motion data
  15. Evaluation: Task 2 - User study: information 17 Scenario : o Scenario(a): o Scenario(b): Aim: test effect of designed auditory feedback
  16. 18 Participant: o 20 participants (4 female and 16 male) o $10 Evaluation: Task 2 - User study: information Motions: o Six motions from Task 1 Methodology : o Aligned Rank Transform (ART) ANOVA o Mann-Whitney U test Hypothesis: amplitude error time error o H1a: in scenario (a) o H1b: in scenario (a) o H2: more effective for complex motion
  17. 19 Evaluation: Task 2 - User study: protocol
  18. 20 Evaluation: Task 2 - User study: results
  19. 21 Evaluation: Task 2 - User study: results
  20. 22 Evaluation: Task 2 - User study: users' comments “Gives me a goal” Encourages me and motivates me “Remind me” “Makes it not boring”
  21. Evaluation: Task 3 - Evaluation by Professional Users 23 Gym coach: Design a motion for the student Helpful for remote learning More limbs can be tracked
  22. Evaluation: Task 3 - Evaluation by Professional Users 24 Helps patients (especially for laying on the bed) Motion design good for different patients Standard motion library Clinical Nurse :
  23. Evaluation: Task 3 - Evaluation by Professional Users 25 Relieve burdens Better for remote learning Therapist :
  24. Discussion and Limitation 26 High-performance 3D motion reconstruction Current characteristics Better for support mode (user should be familiar with motions) One limb tracking Further motion edition More informative feedback with audio Long learning effect Limitation :
  25. 27 Summary Background Current motion-learning system lacks real-time feedback and customized function Related Work Motion tutorial systems/feedback design Proposed Scheme VoLearn: a cross-modal motion-learning system combined virtual avatar and auditory feedback Details Method Motion registration/modification/reference data generation/auditory feedback Experiment Three tasks Result VoLearn can help to reduce the amplitude and speed error during learning Limitation 3D motion reconstruction/more feedback elements/long- term effect
  26. Thank you very much!
Advertisement