Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

3D Reconstruction of Hand Postures by Measuring Skin Deformation on Back Hand (ICAT-EGVE 2017)

52 views

Published on

3D Reconstruction of Hand Postures by Measuring Skin Deformation on Back Hand, ICAT-EGVE 2017

Published in: Technology
  • Be the first to comment

  • Be the first to like this

3D Reconstruction of Hand Postures by Measuring Skin Deformation on Back Hand (ICAT-EGVE 2017)

  1. 1. 3D Reconstruction of Hand Postures by Measuring Skin Deformation on Back Hand *Wakaba Kuno, Yuta Sugiura, Nao Asano, Wataru Kawai and Maki Sugimoto
  2. 2. Hand Interaction in Virtual Environment 24th Nov 2017 1 Hand interaction depending on finger posture is important for intuitive interaction in virtual environments. ICAT-EGVE 3D Reconstruction of Hand Postures [1] https://www.leapmotion.com/ (2017/9/21) [2] https://www.youtube.com/watch?v=B9tF7_nK4lI (2017/9/21) [3] https://www.youtube.com/watch?v=4LVVpl9tCNE (2017/9/21)
  3. 3. Camera-based Method 24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 2 Estimate finger postures by processing images capturing hands Merit: Markerless finger tracking Limit: Space for capturing hands / Hand occlusion [4] Jonathan Taylor et al. Siggraph 2016.
  4. 4. Glove-type Method 24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 3 Various sensors (inertial, magnetic, etc) on gloves measure finger postures. Merit: High accuracy without occlusion Limit: Weight and mechanism may inhibit natural finger movements. [5] Tommaso Lisini Baldi et al. IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS. 2017
  5. 5. Gesture Recognition Measuring Wrist / Forearm 24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 4 Myoelectric or pressure sensors measure wrist or forearm. Recognize hand gestures without measuring finger movement directly [6] Myo. https://www.myo.com (2017/9/21) [7] Dementyev A. and Paradiso A. J.. UIST. 161-166. 2014.
  6. 6. Our Previous Research : Behind The Palm 24th Nov 2017 5 A prototype device with thirteen photo-reflective sensors in a straight array Measure skin deformation on back of hand to recognize finger gestures ICAT-EGVE 3D Reconstruction of Hand Postures [8] Yuta Sugiura et al. SICE Annual Conference. 2017
  7. 7. Principle of Measuring Skin Deformation 24th Nov 2017 6 Skin on back of hand deforms when fingers move. Measure skin deformation on back of hand with photo-reflective sensors ICAT-EGVE 3D Reconstruction of Hand Postures
  8. 8. Proposal : Finger Posture Estimation from Back Hand 24th Nov 2017 7ICAT-EGVE 3D Reconstruction of Hand Postures
  9. 9. Principle of Finger Posture Estimation 24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 8 Obtain relationship between skin deformation on back of hand and finger posture by using a multivariate regression model
  10. 10. System Configuration 24th Nov 2017 9 Sensor values are transmitted wirelessly through Microcontroller. Desktop PC learns a model and estimates finger posture. ICAT-EGVE 3D Reconstruction of Hand Postures
  11. 11. Estimation of Finger Posture 24th Nov 2017 10 Use Principal Component Analysis (PCA) to reduce the dimensions of data Use Random Forest Regressor (RFR) model to estimate finger posture ICAT-EGVE 3D Reconstruction of Hand Postures
  12. 12. Finger Posture Representation 24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 11 Some finger joints work in conjunction. -> Measure relative positions of ten parts from hand center five fingertips, one interphalangeal (IP) joint and four proximal IP (PIP) joints Reference
  13. 13. Evaluation1 – Static-state Finger Posture 24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 12 Participants 7 (Male) + 2 (Female) Data set per participant 2500 frames (= 50fps x 10 seconds x 5 trials) Finger posture 5 postures Evaluation 10-fold cross validation Estimation error 3-dimensional Euclidean distance between the finger postures We evaluated the estimation accuracy of finger posture in static state.
  14. 14. Mean Estimation Error (Static-state) 24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 13 Average estimation error for all parts: 3.34 mm. Mean errors for each part: 1.5 – 2.8 mm (middle, ring and little finger) Large variances of the estimation errors
  15. 15. Time Transition of Small Error Sequence (Static-state) 24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 14 Temporal transition of estimation error (small error) Fingers are about 10 - 15 mm thick for adults. -> Our method estimates postures except thumb and index finger sufficiently for interaction in a virtual environment. 10 2.5 7.5 5 10 2.5 7.5 5 Posture in the sequence
  16. 16. Time Transition of Large Error Sequence (Static-state) 24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 15 Temporal transition of estimation error (large error) Large errors in some frames and similar time transitions of errors -> Cause of high variances of estimation errors Posture in the sequence
  17. 17. Discussion (Static-state) 24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 16 Relative positions depend on finger posture and hand rotation. -> Relative position can not be uniquely decided without hand rotation.
  18. 18. Evaluation2 – Dynamic-state Finger Posture 24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 17 Participants 7 (Male) + 2 (Female) Data set per participant 2000 frames (= 50fps x 5 seconds x 8 trials) Finger posture 5 postures Evaluation 4-fold cross validation Estimation error 3-dimensional Euclidean distance between the finger postures We evaluated the estimation accuracy of finger posture in dynamic state.
  19. 19. Mean Estimation Error (Dynamic-state) 24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 18 Average estimation error for all parts: 11.7 mm. Mean errors for each part: 6.5 – 10.9 mm (middle, ring and little finger)
  20. 20. Time Transition of Error Sequence (Dynamic-state) 24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 19 Large estimation errors for thumb and index fingers in some frames. -> Cause of large mean errors for thumb and index finger
  21. 21. Discussion (Dynamic-state) 24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 20 Our device cannot distinguish the skin deformation caused by thumb movement and index finger movement.
  22. 22. Applications 24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 21 Hand manipulation in VR space Without occlusion and inhibiting natural motion Puppet manipulation metaphor
  23. 23. Limitations • Need of User dependent training - Each user has a different distance between the sensors and skin of back of hand. • Affected by re-wearing the device - Different mounting position due to re-attaching the device 24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 22
  24. 24. Future Work • Improvement of device measuring skin deformation - Measure the whole skin deformation on the back of the hand - Measure skin deformation on back of hand and wrist or forearm simultaneously • Improvement of algorithm Estimating finger postures - Data normalization - Other dimension reduction method of data - Other representation of finger postures 24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 23
  25. 25. Conclusion • We proposed a method for estimating finger postures from skin deformation on back of hand. • A regression model provides relationship between skin deformation of back of hand and finger postures. 24th Nov 2017 ICAT-EGVE 3D Reconstruction of Hand Postures 24 Keio University, The University of Tokyo Wakaba Kuno, Yuta Sugiura, Nao Asano, Wataru Kawai and Maki Sugimoto

×