Successfully reported this slideshow.
Your SlideShare is downloading. ×

Video-Based Hand Tracking for Screening Cervical Myelopathy (ISVC2021)

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 21 Ad
Advertisement

More Related Content

Slideshows for you (20)

Similar to Video-Based Hand Tracking for Screening Cervical Myelopathy (ISVC2021) (20)

Advertisement

More from sugiuralab (20)

Recently uploaded (20)

Advertisement

Video-Based Hand Tracking for Screening Cervical Myelopathy (ISVC2021)

  1. 1. Video-Based Hand Tracking for Screening Cervical Myelopathy 16th International Symposium on Visual Computing (ISVC 2021) October 6th, 2021 Ryota Matsui1, Takafumi Koyama 2, Koji Fujita2, Hideo Saito 1, and Yuta Sugiura 1 1 Keio University 2 Tokyo Medical and Dental University
  2. 2. • Spinal cord: a part of central nervous system • CM: pathology of compressed cervical spinal cord • Cause various symptoms • Upper limb disorders • Gait disturbance 2 [1] OrthoInfo: Spine Basics. https://orthoinfo.aaos.org/en/diseases--conditions/spine-basics/. (accessed Sep. 26th, 2021) [2] Cook, C., et al.: Reliability and diagnostic accuracy of clinical special tests for myelopathy in patients seen for cervical dysfunction. The Journal of Orthopaedic and Sports Physical Therapy 39(3), 172–178 (2009). Cervical Myelopathy (CM) Human spine [1] Compressed spinal cord [2]
  3. 3. • A common screening method for CM • Repeat gripping/releasing one’s hand as quickly as possible for 10 sec • Be screened as a CM patient if the repetition count is approx. less than 20 • The simplicity is a big advantage. 3 [3] Ono, K., et al.: Myelopathy hand. New clinical signs of cervical cord damage. The Journal of Bone and Joint Surgery, British Volume 69(2), 215–219 (1987). 10-sec Grip and Release (G&R) Test [3] Image of G&R movement
  4. 4. • Track the G&R movement to screen CM automatically • Non-contact hand tracking with a Leap Motion Controller (LMC) [4] • A random forest classifier • A glove-like sensing device [5] • Fluctuation of sensor values • Cycle of the G&R movement 4 [4] Watanabe, M., et al.: Detection of cervical myelopathy with Leap Motion Sensor by random forests. IEEE LifeTech 2020, 214–216 (2020). [5] Su, X.J., et al.: Clinical application of a new assessment tool for myelopathy hand using virtual reality. Spine 45(24), E1645–E1652 (2020). Expansion of G&R Test Track G&R movement with an LMC [4] Track G&R movement with the glove-like device [5]
  5. 5. • Screen symptoms of various diseases with a camera • Parkinson’s disease (PD) [6] • Changes in the ability to move face • Amyotrophic lateral sclerosis (ALS) [7] • Use a depth camera • Facial movements affected by ALS 5 [6] Gomez, L.F., et al.: Improving Parkinson detection using dynamic features from evoked expressions in video. CVPR 2021, 1562–1570 (2021). [7] Bandini, A., et al.: Automatic detection of amyotrophic lateral sclerosis (ALS) from video-based analysis of facial movements: Speech and non-speech tasks. FG 2018, 150–157 (2018). Video-Based Screening Methods Flow of screening PD [6] Feature points obtained to screen ALS [7]
  6. 6. • An automatic CM screening method with a camera • The main contributions • Achieve high screening performance • Machine learning-based classification without any qualitative indexes 6 Overview of This Work Record G&R test with a camera Image processing for hand tracking Flow of our method
  7. 7. • The front camera of a smartphone is used. • Frame rate of the videos: 30 fps • Flow of the recording • Hold a hand above the smartphone • Start the G&R test • It is recorded as a video until the following two conditions are met: • The repetition count reaches 20. • The recording time exceeds 10 sec. 7 Record G&R Test with Camera Video of recording G&R test
  8. 8. • MediaPipe Hands [8] is used to track the recorded G&R movement. • MediaPipe: an image-processing framework provided by Google • Estimate 21 three-dimensional positions as the feature values • 3 × 21 = 63 dimensions 8 Image Processing for Hand Tracking – Estimation Image processing with MediaPipe [8] [8] Google: MediaPipe. https://mediapipe.dev/. (accessed Sep. 26th, 2021) Estimated 21 positions
  9. 9. • Change the coordinate system • Before: camera coordinate system (𝑥, 𝑦) • After: hand-based (𝑥’, 𝑦’) • Improve the robustness against the relative direction of cameras and hands • Introduce the velocity components • The difference of the positions between the adjacent video frames • 63 + 63 = 126 dimensions 9 Image Processing for Hand Tracking – Adjustment Change of coordinate system
  10. 10. • Divide the time-series data into 20 segments • To align the length of each data • The length of each segment is approx. 2 sec. • The adjacent segments may be overlapped if the data is short. • To obtain segments of the stabilized length 10 Pre-processing – Data Segmentation Segmentation of time-series data
  11. 11. • Conversion to frequency components for each dimension • The G&R movement can be regarded as a periodic movement. • 32 frequency components • 32 components × 126 dimensions = 4,032 elements for each segment 11 Pre-processing – Fast Fourier Transform Flow of fast Fourier transform
  12. 12. • Merge segments of the multiple G&R trials • If we take multiple trials to screen CM • Each of 20 segments are distinguished. • Use these 20 segments as the inputs to the screening classifier 12 Pre-processing – Merging Data Image of merging data
  13. 13. • A support vector machine classifier to classify each segment • CM patient group • Control group • Each of 20 segments is classified independently each other. • The average of the 20 classifications is used as the screening result. 13 Two-Class Classification Image of two-class classification
  14. 14. • Record the G&R tests of 35 participants • Each participant was recorded twice: the right hand and the left hand. • 2 × 35 = 70 videos • All participants were right-handed. • 10-fold cross-validation with these 70 videos • The split was per participant. 14 CM patient group Male CM patient group Female Control group Male Control group Female Number of participants 10 10 8 7 Mean age 66.8 71.6 57.9 64.9 SD of age 7.9 11.1 14.4 12.2 Overview of Experiments Participant information
  15. 15. • Sensitivity • 𝑇𝑃 𝑇𝑃+𝐹𝑁 • Specificity • 𝑇𝑁 𝑇𝑁+𝐹𝑃 • AUC (area under the curve) • ROC curve: relationship between sensitivity and specificity expressed as a graph • AUC: area under the ROC curve 15 Evaluation Indexes Positive / Patient True positive (𝑇𝑃) False positive (𝐹𝑃) Negative / Control False negative (𝐹𝑁) True negative (𝑇𝑁) Positive / Patient Negative / Control Actual label Predicted label Confusion matrix ROC curve and AUC
  16. 16. • Validate the evaluation indexes for the selected combinations of feature values • To select mainly contributed feature values for the classification • Four points of each finger • Fingertip and each joint of five fingers • Fingertip • DIP joint • PIP joint • MCP joint 16 [9] Bullock, I.M., et al.: Assessing assumptions in kinematic hand models: A review. IEEE BioRob 2012, 139–146 (2012). Validation for Each Feature Value – Overview Estimated 21 positions Finger joints [9]
  17. 17. • The tendency to show relatively high AUC • Fingers of the thumb side • Joints of the wrist side • Same sensitivity and specificity for all combinations • Sensitivity: 0.900 • Specificity: 0.867 • Select feature values for high AUC • Thumb and index finger • PIP and MCP joints 17 Validation for Each Feature Value – Results AUC Thumb 0.907 Index finger 0.900 Middle finger 0.877 Ring finger 0.853 Little finger 0.870 AUC Fingertip 0.880 DIP joint 0.867 PIP joint 0.910 MCP joint 0.930 AUC for each finger AUC for fingertip and each joint [9] Bullock, I.M., et al.: Assessing assumptions in kinematic hand models: A review. IEEE BioRob 2012, 139–146 (2012). Finger joints [9]
  18. 18. • Validate under the following three conditions • Use videos of both hands • Use video of right/left hand • Validation with right hands showed relatively high performance. • C.f. all participants were right-handed. 18 Validation for Selected Feature Values Sensitivity, specificity, and AUCs ROC curve for right hand ROC curve for left hand
  19. 19. • Higher than other CM screening methods • Potential to be used in actual medical settings as the new CM screening method 19 [5] Su, X.J., et al.: Clinical application of a new assessment tool for myelopathy hand using virtual reality. Spine 45(24), E1645–E1652 (2020). [10] Machino, M., et al.: : Cut off value in each gender and decade of 10-s grip and release and 10-s step test: A comparative study between 454 patients with cervical spondylotic myelopathy and 818 healthy subjects. Clinical Neurology and Neurosurgery (2019). Comparison with Other Screening Methods Screening Method Sensitivity Specificity AUC The 10-s G&R test [10] 0.703–0.821 0.429–0.740 0.647–0.890 The glove-like device [5] 0.8673 0.86 0.929 Our method 0.900 0.933 0.947 Comparison of screening performance G&R test Glove-like device [5] Our method
  20. 20. Limitations • Focus only on hand disorders, a symptom of CM • Do not take other pathologies/diseases into account Future work • Apply to other pathologies/diseases which cause hand disorders • E.g., carpal tunnel syndrome • Apply to actual medical settings • As the initial screening method used by local doctors and care workers 20 Limitations and Future Work
  21. 21. 21 Summary Background The 10-sec G&R test to screen cervical myelopathy (CM) Related work Track the G&R movement to screen CM automatically Suggestion Use videos of the G&R movement to screen CM Method Classify feature values of the G&R movement Evaluation Cross-validation with videos of 35 participants Results Sensitivity: 0.900 Specificity: 0.933 AUC: 0.947 Limitations Focus only on hand disorders, a symptom of CM Future works Apply our method to the actual medical settings

×