Vision-Based Finger Detection and Its Applications   基於電腦視覺之手指偵測及其應用 Yi-Fan Chuang Advisor: Prof. Yi-Ping Hung Prof. Ming-...
Outline <ul><li>Introduction </li></ul><ul><li>Related Work </li></ul><ul><li>Fingertip Detection and Tracking </li></ul><...
Outline <ul><li>Introduction </li></ul><ul><li>Related Work </li></ul><ul><li>Fingertip Detection and Tracking </li></ul><...
Introduction <ul><li>Interactive system </li></ul><ul><ul><li>Bare-handed operations are more intuitive to manipulate digi...
Outline <ul><li>Introduction </li></ul><ul><li>Related Work </li></ul><ul><li>Fingertip Detection and Tracking </li></ul><...
Related Work <ul><li>Enhance Desk </li></ul><ul><ul><li>Fingertip finding </li></ul></ul><ul><ul><ul><li>Template matching...
Related Work <ul><li>FTIR (Frustrated Total Internal Reflection) </li></ul>J. Y. Han, “Low-cost multi-touch sensing throug...
Related Work <ul><li>Visual Touchpad </li></ul><ul><ul><li>Fingertip detection </li></ul></ul><ul><ul><ul><li>Curvature of...
Related Work <ul><li>PlayAnyWhere </li></ul><ul><ul><li>Touch and Hover </li></ul></ul><ul><ul><ul><li>Use appearance of s...
Related Work <ul><li>Comparison </li></ul>Our Work: Touch/Hover (1 camera) Multi-user with multi-finger Enhance Desk (CHI ...
Outline <ul><li>Introduction </li></ul><ul><li>Related Work </li></ul><ul><li>Fingertip Detection and Tracking </li></ul><...
Overview <ul><li>System setup </li></ul><ul><ul><li>An IR camera with IR illuminators to observe hands </li></ul></ul>J. R...
<ul><li>System setup </li></ul><ul><ul><li>Direct observation </li></ul></ul><ul><ul><ul><li>Clear shapes </li></ul></ul><...
<ul><li>System setup </li></ul><ul><ul><li>Direct observation </li></ul></ul><ul><ul><ul><li>Clear shapes </li></ul></ul><...
Overview <ul><li>Fingertip detection </li></ul><ul><li>Fingertip tracking </li></ul>Detection results Kalman filtering Bac...
Fingertip Detection <ul><li>Background subtraction </li></ul><ul><li>Observation </li></ul><ul><ul><li>Contact area leaves...
Fingertip Detection <ul><li>Extract finger part </li></ul><ul><ul><li>Use a morphological opening operation </li></ul></ul...
Fingertip Detection <ul><li>Difference and binarization </li></ul><ul><li>Principal component analysis </li></ul>Finger di...
Fingertip Detection <ul><li>Template matching (pattern matching) </li></ul><ul><ul><li>Template </li></ul></ul><ul><ul><li...
Detection Results on  i-m- Top (Enhanced image) Separate finger touching  and palm hovering Fingertip detection Difference...
Detection Results on MaC Ball
Fingertip Tracking <ul><li>Kalman filter </li></ul><ul><ul><li>Smooth the path </li></ul></ul><ul><ul><ul><li>Predict the ...
Performance Evaluation <ul><li>Real-time system is possible </li></ul>
Outline <ul><li>Introduction </li></ul><ul><li>Related Work </li></ul><ul><li>Fingertips Detection and Tracking </li></ul>...
Hardware Configuration <ul><li>i-m- Top </li></ul><ul><ul><li>I nteractive  M ulti-resolution Table top </li></ul></ul>Coo...
Hardware Configuration <ul><li>i-m- Top </li></ul><ul><ul><li>I nteractive  M ulti-resolution Table top </li></ul></ul>Coo...
Software Implementation <ul><li>Fingertip detection and tracking </li></ul><ul><li>Palm detection and tracking </li></ul><...
Palm Detection and Tracking <ul><li>Palm detection </li></ul><ul><ul><li>The average position is picked as palm position <...
Association between Fingertips and Palms <ul><li>Allocate fingertips to palms </li></ul><ul><ul><li>Use the angle and the ...
Pen Detection and Priority <ul><li>Pen Detection </li></ul><ul><ul><li>Use a higher threshold </li></ul></ul><ul><li>Prior...
Events Definitions <ul><li>Hover </li></ul><ul><ul><li>Palm position </li></ul></ul><ul><li>Touch </li></ul><ul><ul><li>Pa...
Application I:  Browsing and Editing <ul><li>Operations </li></ul><ul><ul><li>Drag photo/document </li></ul></ul><ul><ul><...
Application II: Video Retrieval System <ul><li>Operations </li></ul><ul><ul><li>Query </li></ul></ul><ul><ul><li>Browse </...
Outline <ul><li>Introduction </li></ul><ul><li>Related Work </li></ul><ul><li>Fingertip Detection and Tracking </li></ul><...
Magic Crystal Ball (MaC Ball) <ul><li>An interactive 3D display </li></ul><ul><ul><li>Slide fingers (like a wizard) on the...
Hardware Configuration Display Module Detection Module IR Camera   IR Illuminator  Pressure Sensors User Transparent Glass...
Software Implementation <ul><li>Fingertip detection and tracking </li></ul><ul><ul><li>Detect fingertips’ positions </li><...
Software Implementation <ul><li>Contact detection – Pressure Sensors </li></ul><ul><ul><li>Observation </li></ul></ul>p1 p...
Events Definitions <ul><li>Gesture switch </li></ul>Pointing Gesture Fingertips Detection Motion Detection Motion Detectio...
Application: Virtual Exhibition <ul><li>Operations </li></ul><ul><ul><li>Rotate </li></ul></ul><ul><ul><li>Select </li></u...
Outline <ul><li>Introduction </li></ul><ul><li>Related Work </li></ul><ul><li>Fingertip Detection and Tracking </li></ul><...
Conclusion & Future Work <ul><li>Multi-finger detection techniques </li></ul><ul><ul><li>Simple and real-time </li></ul></...
Upcoming SlideShare
Loading in …5
×

Vision-based Finger Detection and Its Applications

10,061
-1

Published on

Master Thesis Presentation by Chuang Yi-Fan, Image and Vision Lab, Graduate Institute of Networking and Multimedia, National Taiwan University

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
10,061
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
558
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • Vision-based Finger Detection and Its Applications

    1. 1. Vision-Based Finger Detection and Its Applications 基於電腦視覺之手指偵測及其應用 Yi-Fan Chuang Advisor: Prof. Yi-Ping Hung Prof. Ming-Sui Lee
    2. 2. Outline <ul><li>Introduction </li></ul><ul><li>Related Work </li></ul><ul><li>Fingertip Detection and Tracking </li></ul><ul><li>Applications </li></ul><ul><ul><li>i-m- Top </li></ul></ul><ul><ul><li>Magic Crystal Ball (MaC Ball) </li></ul></ul><ul><li>Conclusion & Future Work </li></ul>
    3. 3. Outline <ul><li>Introduction </li></ul><ul><li>Related Work </li></ul><ul><li>Fingertip Detection and Tracking </li></ul><ul><li>Applications </li></ul><ul><ul><li>i-m- Top </li></ul></ul><ul><ul><li>Magic Crystal Ball (MaC Ball) </li></ul></ul><ul><li>Conclusion & Future Work </li></ul>
    4. 4. Introduction <ul><li>Interactive system </li></ul><ul><ul><li>Bare-handed operations are more intuitive to manipulate digital objects directly </li></ul></ul><ul><ul><li>Interactions </li></ul></ul><ul><ul><ul><li>Users: hover/touch gestures </li></ul></ul></ul><ul><ul><ul><li>Objects: rotation, scaling, selection, special effects…. </li></ul></ul></ul><ul><ul><li>Use computer vision techniques </li></ul></ul><ul><ul><ul><li>Fingertip detection </li></ul></ul></ul><ul><ul><ul><li>Fingertip tracking </li></ul></ul></ul>
    5. 5. Outline <ul><li>Introduction </li></ul><ul><li>Related Work </li></ul><ul><li>Fingertip Detection and Tracking </li></ul><ul><li>Applications </li></ul><ul><ul><li>i-m- Top </li></ul></ul><ul><ul><li>Magic Crystal Ball (MaC Ball) </li></ul></ul><ul><li>Conclusion & Future Work </li></ul>
    6. 6. Related Work <ul><li>Enhance Desk </li></ul><ul><ul><li>Fingertip finding </li></ul></ul><ul><ul><ul><li>Template matching </li></ul></ul></ul><ul><ul><li>Palm finding </li></ul></ul><ul><ul><ul><li>Morphological erosion </li></ul></ul></ul>H. KOIKE and Y. KOBAYASHI, “Integrating paper and digital information on enhanceddesk: a method for realtime finger tracking on an augmented desk system,“ ACM Transation Computer-Human Interaction, vol. 8, no. 4, pp. 307-322, 2001.
    7. 7. Related Work <ul><li>FTIR (Frustrated Total Internal Reflection) </li></ul>J. Y. Han, “Low-cost multi-touch sensing through frustrated total internal reflection,&quot; in Proceedings of the 18th annual ACM symposium on User interface software and technology (UIST '05). New York, NY, USA: ACM Press, 2005, pp. 115-118.
    8. 8. Related Work <ul><li>Visual Touchpad </li></ul><ul><ul><li>Fingertip detection </li></ul></ul><ul><ul><ul><li>Curvature of contour </li></ul></ul></ul><ul><ul><ul><li>Finger Orientation </li></ul></ul></ul><ul><ul><li>Touch detection </li></ul></ul><ul><ul><ul><li>Two warp images </li></ul></ul></ul>S. Malik and J. Laszlo, “Visual touchpad: a two-handed gestural input device,&quot; in Proceedings of the 6th international conference on Multimodal interfaces (ICMI '04). New York, NY, USA: ACM Press, 2004, pp. 289-296.
    9. 9. Related Work <ul><li>PlayAnyWhere </li></ul><ul><ul><li>Touch and Hover </li></ul></ul><ul><ul><ul><li>Use appearance of shadows </li></ul></ul></ul>A. D. Wilson, “Playanywhere: a compact interactive tabletop projection-vision system,&quot; in Proceedings of the 18th annual ACM symposium on User interface software and technology (UIST '05). New York, NY, USA: ACM Press, 2005, pp. 83-92.
    10. 10. Related Work <ul><li>Comparison </li></ul>Our Work: Touch/Hover (1 camera) Multi-user with multi-finger Enhance Desk (CHI 01’) FTIR (UIST 05’) Visual Touchpad (ICMI 04’) PlayAnyWhere (UIST 05’) Method Template matching Connected component analysis Curvature of contour Shadow Detection Touch Touch Touch/Hover (2 cameras) Touch/Hover (1 camera) Disadvantage One user Only touch Only touch One user One finger for one palm
    11. 11. Outline <ul><li>Introduction </li></ul><ul><li>Related Work </li></ul><ul><li>Fingertip Detection and Tracking </li></ul><ul><li>Applications </li></ul><ul><ul><li>i-m- Top </li></ul></ul><ul><ul><li>Magic Crystal Ball (MaC Ball) </li></ul></ul><ul><li>Conclusion & Future Work </li></ul>
    12. 12. Overview <ul><li>System setup </li></ul><ul><ul><li>An IR camera with IR illuminators to observe hands </li></ul></ul>J. Rekimoto and N. Matsushita, “Perceptual surfaces: Towards a human and object sensitive interactive display,&quot; Workshop on Perceptural User Interfaces (PUI'97), 1997. <ul><li>Direct observation </li></ul><ul><ul><li>Clear shapes </li></ul></ul><ul><li>Indirect observation </li></ul><ul><ul><li>Unclear shapes </li></ul></ul>
    13. 13. <ul><li>System setup </li></ul><ul><ul><li>Direct observation </li></ul></ul><ul><ul><ul><li>Clear shapes </li></ul></ul></ul><ul><ul><li>Indirect observation </li></ul></ul><ul><ul><ul><li>Unclear shapes </li></ul></ul></ul>Overview Clear shapes (MaC Ball) Gesture Captured image
    14. 14. <ul><li>System setup </li></ul><ul><ul><li>Direct observation </li></ul></ul><ul><ul><ul><li>Clear shapes </li></ul></ul></ul><ul><ul><li>Indirect observation </li></ul></ul><ul><ul><ul><li>Unclear shapes </li></ul></ul></ul>Overview Unclear shapes ( i-m- Top) Gesture Captured image
    15. 15. Overview <ul><li>Fingertip detection </li></ul><ul><li>Fingertip tracking </li></ul>Detection results Kalman filtering Background subtraction Morphological opening Difference of previous two Principal component analysis Fingertip detection
    16. 16. Fingertip Detection <ul><li>Background subtraction </li></ul><ul><li>Observation </li></ul><ul><ul><li>Contact area leaves strong reflection </li></ul></ul>(Enhanced image) Background subtraction Morphological opening Difference of previous two PCA Fingertip detection
    17. 17. Fingertip Detection <ul><li>Extract finger part </li></ul><ul><ul><li>Use a morphological opening operation </li></ul></ul><ul><ul><ul><li>The size of structuring element is larger than that of a normal finger and smaller than a palm. </li></ul></ul></ul>Original Morphological opening Finger (Enhanced image) Background subtraction Morphological opening Difference of previous two PCA Fingertip detection
    18. 18. Fingertip Detection <ul><li>Difference and binarization </li></ul><ul><li>Principal component analysis </li></ul>Finger direction Possible fingertips’ positions (Enhanced image) Background subtraction Morphological opening Difference of previous two PCA Fingertip detection Background subtraction Morphological opening Difference of previous two PCA Fingertip detection
    19. 19. Fingertip Detection <ul><li>Template matching (pattern matching) </li></ul><ul><ul><li>Template </li></ul></ul><ul><ul><li>Remove false matchings </li></ul></ul><ul><ul><ul><li>Distance check </li></ul></ul></ul><ul><ul><ul><li>Diagonal check </li></ul></ul></ul>Finger patch Background subtraction Morphological opening Difference of previous two PCA Fingertip detection
    20. 20. Detection Results on i-m- Top (Enhanced image) Separate finger touching and palm hovering Fingertip detection Difference of previous two & PCA Morphological opening Background subtraction
    21. 21. Detection Results on MaC Ball
    22. 22. Fingertip Tracking <ul><li>Kalman filter </li></ul><ul><ul><li>Smooth the path </li></ul></ul><ul><ul><ul><li>Predict the new state and its uncertainty </li></ul></ul></ul><ul><ul><ul><li>Correct the tracker with its new measurement </li></ul></ul></ul><ul><ul><ul><li>Assume white noise and uniform velocity </li></ul></ul></ul>Original After Kalman filter
    23. 23. Performance Evaluation <ul><li>Real-time system is possible </li></ul>
    24. 24. Outline <ul><li>Introduction </li></ul><ul><li>Related Work </li></ul><ul><li>Fingertips Detection and Tracking </li></ul><ul><li>Applications </li></ul><ul><ul><li>i-m- Top </li></ul></ul><ul><ul><li>Magic Crystal Ball (MaC Ball) </li></ul></ul><ul><li>Conclusion & Future Work </li></ul>
    25. 25. Hardware Configuration <ul><li>i-m- Top </li></ul><ul><ul><li>I nteractive M ulti-resolution Table top </li></ul></ul>Cooperate with Yi-Wei Chia
    26. 26. Hardware Configuration <ul><li>i-m- Top </li></ul><ul><ul><li>I nteractive M ulti-resolution Table top </li></ul></ul>Cooperate with Yi-Wei Chia
    27. 27. Software Implementation <ul><li>Fingertip detection and tracking </li></ul><ul><li>Palm detection and tracking </li></ul><ul><li>Association between fingertips and palms </li></ul><ul><li>Pen detection </li></ul><ul><li>Priority of pens and palms </li></ul><ul><li>Events definitions </li></ul>
    28. 28. Palm Detection and Tracking <ul><li>Palm detection </li></ul><ul><ul><li>The average position is picked as palm position </li></ul></ul><ul><li>Palm tracking </li></ul><ul><ul><li>Kalman filtering </li></ul></ul>Palm detection Morphological opening (Palm part) Background subtraction (Enhanced Image)
    29. 29. Association between Fingertips and Palms <ul><li>Allocate fingertips to palms </li></ul><ul><ul><li>Use the angle and the distance between fingers and palms </li></ul></ul>(Enhanced Image)
    30. 30. Pen Detection and Priority <ul><li>Pen Detection </li></ul><ul><ul><li>Use a higher threshold </li></ul></ul><ul><li>Priority of pens and palms </li></ul><ul><ul><li>Higher priority for the pen tip </li></ul></ul><ul><ul><ul><li>If there is a palm around the pen tip, this palm and the fingertips associated to the palm will be ignored. </li></ul></ul></ul>
    31. 31. Events Definitions <ul><li>Hover </li></ul><ul><ul><li>Palm position </li></ul></ul><ul><li>Touch </li></ul><ul><ul><li>Palm position </li></ul></ul><ul><ul><li>Finger position and its correspondent palm </li></ul></ul><ul><ul><li>Pen position </li></ul></ul>
    32. 32. Application I: Browsing and Editing <ul><li>Operations </li></ul><ul><ul><li>Drag photo/document </li></ul></ul><ul><ul><li>Zoom in and zoom out </li></ul></ul><ul><ul><li>Stroke </li></ul></ul><ul><ul><li>Move the foveal area </li></ul></ul>
    33. 33. Application II: Video Retrieval System <ul><li>Operations </li></ul><ul><ul><li>Query </li></ul></ul><ul><ul><li>Browse </li></ul></ul><ul><ul><li>Feedback </li></ul></ul>Video Top button Video plane Video wall Scroll bar
    34. 34. Outline <ul><li>Introduction </li></ul><ul><li>Related Work </li></ul><ul><li>Fingertip Detection and Tracking </li></ul><ul><li>Applications </li></ul><ul><ul><li>i-m- Top </li></ul></ul><ul><ul><li>Magic Crystal Ball (MaC Ball ) </li></ul></ul><ul><li>Conclusion & Future Work </li></ul>
    35. 35. Magic Crystal Ball (MaC Ball) <ul><li>An interactive 3D display </li></ul><ul><ul><li>Slide fingers (like a wizard) on the ball surface to control the content </li></ul></ul>
    36. 36. Hardware Configuration Display Module Detection Module IR Camera IR Illuminator Pressure Sensors User Transparent Glass Ball Reflection Mirror Fresnel Lens LCD
    37. 37. Software Implementation <ul><li>Fingertip detection and tracking </li></ul><ul><ul><li>Detect fingertips’ positions </li></ul></ul><ul><ul><li>Pointing gesture </li></ul></ul><ul><li>Motion detection </li></ul><ul><ul><li>Optical flow algorithm </li></ul></ul><ul><ul><li>Waving gesture </li></ul></ul><ul><li>Contact Detection </li></ul><ul><ul><li>Pressure Sensor (Flexi Force) </li></ul></ul><ul><ul><ul><li>Detect whether users touch MaC Ball </li></ul></ul></ul>
    38. 38. Software Implementation <ul><li>Contact detection – Pressure Sensors </li></ul><ul><ul><li>Observation </li></ul></ul>p1 p2 p3 p1 p2 p1 p2
    39. 39. Events Definitions <ul><li>Gesture switch </li></ul>Pointing Gesture Fingertips Detection Motion Detection Motion Detection Fingertips Detection Waving Gesture Pressure sensors Touch / Hover
    40. 40. Application: Virtual Exhibition <ul><li>Operations </li></ul><ul><ul><li>Rotate </li></ul></ul><ul><ul><li>Select </li></ul></ul><ul><ul><li>Magnifier </li></ul></ul><ul><ul><li>Change relics </li></ul></ul>Video
    41. 41. Outline <ul><li>Introduction </li></ul><ul><li>Related Work </li></ul><ul><li>Fingertip Detection and Tracking </li></ul><ul><li>Applications </li></ul><ul><ul><li>i-m- Top </li></ul></ul><ul><ul><li>Magic Crystal Ball (MaC Ball) </li></ul></ul><ul><li>Conclusion & Future Work </li></ul>
    42. 42. Conclusion & Future Work <ul><li>Multi-finger detection techniques </li></ul><ul><ul><li>Simple and real-time </li></ul></ul><ul><ul><li>Cost-effective and flexible </li></ul></ul><ul><li>Applications </li></ul><ul><ul><li>i-m- Top: interactive interface </li></ul></ul><ul><ul><li>MaC Ball: interactive 3D display </li></ul></ul><ul><li>Future work </li></ul><ul><ul><li>Use FTIR to enhance the accuracy of fingertip detection </li></ul></ul><ul><ul><li>More gestures for interactive systems </li></ul></ul>
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×