Mid-air Gesture Recognition by Ultra-Wide Band Radar Echoes
•Download as PPTX, PDF•
0 likes•18 views
Doctoral consortium @ EICS 2022
Recommended
Recommended
More Related Content
Similar to Mid-air Gesture Recognition by Ultra-Wide Band Radar Echoes
Similar to Mid-air Gesture Recognition by Ultra-Wide Band Radar Echoes (20)
Recently uploaded
Recently uploaded (20)
Mid-air Gesture Recognition by Ultra-Wide Band Radar Echoes
- 2. Mid-air Gesture Recognition by Ultra-Wide Band Radar Echoes Arthur Sluÿters, EICS 2022
- 3. Context of the problem 3D mid-air gestures are developing as alternative method of interaction Many sensors can be used, but each type has some limitations
- 4. Vision-based sensors • Privacy concerns • Sensitive to lighting conditions • Occlusion ultraleap.com wikipedia.org intel.fr
- 5. Wearable sensors • More invasive • One per user • Hygiene concerns kinemic.com Medium.com
- 6. In some situations, radars may be preferred to other types of sensors
- 7. Radars sensors: • Less privacy concerns • Not sensitive to lighting conditions • Less intrusive rfbeam.ch ti.com projects-raspberry.com
- 8. However… • Less mature than other solutions • Sensitive to noise (e.g., antennas effects) • Sensitive to clutter • Data interpretation can be complex
- 9. However… • Less mature than other solutions • Sensitive to noise (e.g., antennas effects) • Sensitive to clutter • Data interpretation can be complex
- 10. Internal reflections, transmissions However… • Less mature than other solutions • Sensitive to noise (e.g., antennas effects) • Sensitive to clutter • Data interpretation can be complex
- 11. Radar However… • Less mature than other solutions • Sensitive to noise (e.g., antennas effects) • Sensitive to clutter • Data interpretation can be complex
- 12. Radar Walls However… • Less mature than other solutions • Sensitive to noise (e.g., antennas effects) • Sensitive to clutter • Data interpretation can be complex
- 13. Radar Furniture However… • Less mature than other solutions • Sensitive to noise (e.g., antennas effects) • Sensitive to clutter • Data interpretation can be complex
- 14. However… • Less mature than other solutions • Sensitive to noise (e.g., antennas effects) • Sensitive to clutter • Data interpretation can be complex Signal from multiple receivers Identify hand motion from raw radar data
- 15. Objective: Investigate if radar-based gesture interaction is feasible
- 16. Some hypotheses 1. Walabot device 2. Hand gestures 3. Users can add/remove gestures nearly instantaniously
- 17. Some hypotheses 1. Walabot device 2. Hand gestures 3. Users can add/remove gestures nearly instantaniously
- 18. Some hypotheses 1. Walabot device 2. Hand gestures 3. Users can add/remove gestures nearly instantaneously Template matching recognizers Fast execution time Very short/no training time Few training templates
- 19. Plan of the presentation 1. Research methodology 2. Current status & next steps 3. Conclusion
- 21. 1. Systematic literature review Gestures Radartutorial.eu rfbeam.ch ti.com Radars CNN kNN LSTM Random Forest SVM HMM Algorithms …
- 22. 2. Gesture elicitation studies Explore user-defined gestures… • IoT commands • Multimedia apps • … …in different environments • Dark conditions • Behind a door • …
- 23. 3. Data acquisition Systematic literature review Gesture elicitation studies … Publicly available dataset(s)
- 24. 4. Gesture recognition environment Signal pre-processing pipeline Software environment for gesture recognition Push with the palm
- 26. Current status & next steps
- 27. 1. Systematic literature review
- 28. Overview
- 29. Algorithms • 151 algorithms for gesture recognition • Mostly based on neural networks
- 30. Radars • 123 radar systems • 29 unique models of radars
- 31. Only 7 public datasets Datasets
- 32. Gestures Catalogue and classify all the gestures in the publications
- 33. 2. Gesture elicitation studies
- 34. Gesture elicitation studies • Multiple studies have been conducted • Send commands to IoT devices • Gestures in emergency situations • Interact with an office door • Results not yet analyzed
- 36. First dataset 16 gestures dataset • Walabot + LMC + Custom radar • One user • 5 samples/user
- 37. Extended dataset 20 gestures dataset • Walabot (+ LMC) • Multiple users • At least 10 samples/user
- 38. Other datasets • Different radars: • Walabot US/FCC vs. Walabot EU/CE • Different environments: • Lighting conditions • Clutter • …
- 39. 4. Gesture recognition environment
- 40. Background subtraction 4 Time gating 6 IFFT 5 Antenna effects removal 3 Raw data capture 1 FFT if time- domain radar 2 Filtering 8 Inversion EM forward model Optimization algorithm Error function 7 Signal pre-processing pipeline Pre-processing pipeline with 8 stages
- 41. Part 1: Capture raw radar data Signal pre-processing pipeline Raw data capture 1
- 42. Background subtraction 4 Time gating 6 IFFT 5 Antenna effects removal 3 FFT if time- domain radar 2 Part 2: Filter noise and clutter Signal pre-processing pipeline
- 43. Filtering 8 Inversion EM forward model Optimization algorithm Error function 7 Part 3: Reduce the complexity and size of radar data Signal pre-processing pipeline
- 44. Background subtraction 4 Time gating 6 IFFT 5 Antenna effects removal 3 Raw data capture 1 FFT if time- domain radar 2 Filtering 8 Inversion EM forward model Optimization algorithm Error function 7 Improvements: • Improve performance (real-time) • Reduce variations between different users (hand size, distance) Signal pre-processing pipeline
- 45. Software environment for gesture recognition Framework for engineering gesture-based applications Originally built for the LMC
- 46. Software environment for gesture recognition (challenges) • Integration of the radar pre-processing pipeline • Gesture segmentation from a continuous stream of radar data
- 48. Testing with the 16-gestures dataset
- 49. Next steps • Testing on other datasets • Multiple users • Different sensors • … • Built radar gesture-based applications
- 50. Conclusion
- 51. Expected contributions • SLR • Gesture Elicitation Studies • Datasets of radar gestures • Pipeline for pre-processing radar gestures • Framework for (radar) gesture recognition • Benchmarkings • Radar gesture-based application for IoT or multimedia
- 52. Many challenges remain… • Multi-user interaction • Improve performance across different users • Achieve real-time radar gesture recognition • Jump from the lab to real-life • Interference? • Gesture segmentation • Building real apps
- 53. Thank you for your attention!
Editor's Notes
- Many sensors can be used, but each type has some limitations that may be problematic in some situations
- Perform a Systematic literature review to identify radar systems, algorithms, gesture sets,…
- Conduct gesture elicitation studies to explore user-defined gestures in various environments
- Conduct gesture elicitation studies to explore user-defined gestures in various environments
- Then, using the information from the SLR and gesture elicitation studies, we will acquire some new gesture sets that will be made public
- Then, using the information from the SLR and gesture elicitation studies, we will acquire some new gesture sets that will be made public
- Then, we’ll design an environment for gesture recognition, in two parts: The first part will process radar signal to remove noise and clutter, and extract relevant data from the signal The second part will handle the process of gesture recognition and send the results to applications
- Finally, we will evaluate the system, by testing the performance of gesture recognition and creating some radar-based application.
- first implementation, subject of 1 (soon 2) papers
- Modular architecture for gesture recognition API for associating actions to gestures