Presentation of our paper "FORTE: Few Samples for Recognizing Hand Gestures with a Smartphone-attached Radar" at the EICS 2023 conference in Swansea, UK.

1. FORTE: Few Samples for Recognizing Hand Gestures with a
Smartphone-attached Radar
EICS’23, Swansea (UK), 26 – 31 June 2023

2. Authors
Stefano Chioccarello
University of Padova,
Italy
stefano.chioccarello.1@studenti.unipd.it
Arthur Sluÿters
Université catholique de Louvain,
Belgium
arthur.sluyters@uclouvain.be
Alberto Testolin
University of Padova,
Italy
alberto.testolin@unipd.it
Sébastien Lambot
Université catholique de Louvain,
Belgium
sebastien.lambot@uclouvain.be
Jean Vanderdonckt
Université catholique de Louvain,
Belgium
jean.vanderdonckt@uclouvain.be
2

3. Objective of this paper
Explore the use of CNNs trained with few samples
for hand gesture recognition with radar.
3

4. Objective of this paper
Explore the use of CNNs trained with few samples
for hand gesture recognition with radar.
4

5. Why work with radars for gesture recognition?
5

6. Why work with radars for gesture recognition?
6
Easy integration
(cheap, low power,
and small)

7. Why work with radars for gesture recognition?
7
Low sensitivity
to environmental
conditions
Easy integration
(cheap, low power,
and small)

8. Why work with radars for gesture recognition?
8
See through
surfaces and
materials
Easy integration
(cheap, low power,
and small)
Low sensitivity
to environmental
conditions

9. Why work with radars for gesture recognition?
9
See through
surfaces and
materials
Low sensitivity
to environmental
conditions
Easy integration
(cheap, low power,
and small)
Feeling of
privacy

10. Objective of this paper
Explore the use of CNNs trained with few samples
for hand gesture recognition with radar.
10

11. Why few samples?
11
Faster acquisition Faster (re)training Lightweight
User customization and adaptation

12. Applications of radar-based gesture recognition
13
Mobile gesture
interaction
(smartphone,
smartwatch,…)

13. Applications of radar-based gesture recognition
14
Smart furniture
(desk, door,…)
Mobile gesture
interaction
(smartphone,
smartwatch,…)

14. Applications of radar-based gesture recognition
15
Radar placements in the physical
environment
Şiean et al. (2023). FLEXIBLE GESTURE INPUT WITH
RADARS: Systematic literature review and taxonomy of
radar sensing integration in ambient intelligence
environments. Journal of Ambient Intelligence and
Humanized Computing, 14(6), 7967–7981.
https://doi.org/10.1007/s12652-023-04606-9

15. Plan of this presentation
1. Exploring radar-based gestures
• Sensor
• Dataset
2. Our approach for gesture recognition
• Models
• Results
• Discussion
3. Summary
• Advantages and limitations
• Future works
16

16. Exploring radar-based
gestures
17

17. Design a gesture set with variations in…
19

18. Design a gesture set with variations in scale
20
Finger-level Arm-level Body-level

19. Design a gesture set with variations in distance and angle
21

20. Design a gesture set with variations in radar cross section
22

21. 20 gestures
23

22. 7 finger-level
24

23. 11 arm-level
25

24. 2 body-level
26

25. 20 gestures
27

26. 20 gestures
28

27. 20 gestures
29

28. 20 gestures
30

29. Recording setup
• 22 participants
• 10 repetitions per gesture
per participant
• Walabot Developer device
31

30. Recording setup
• 22 participants
• 10 repetitions per gesture
per participant
• Walabot Developer device
• What is it? Why use this?
32

31. The Walabot Developer
33
thegadgetflow.com
projects-raspberry.com

32. The Walabot Developer
34
thegadgetflow.com
projects-raspberry.com
• FMCW radar, 6.3 – 8 GHz (EU/CE)

33. The Walabot Developer
35
projects-raspberry.com
• FMCW radar, 6.3 – 8 GHz (EU/CE)
• 18 antennas
• 4 emitters
• 15 receivers
• 3 unused

34. The Walabot Developer
36
thegadgetflow.com
projects-raspberry.com

35. Why the Walabot?
43
projects-raspberry.com
• Off-the-shelf, readily available
• Easy retrieval of raw data, facilitating the
transition to other radar sensors

36. Why the Walabot?
44
projects-raspberry.com
• Off-the-shelf, readily available
• Easy retrieval of raw data, facilitating the
transition to other radar sensors
Good device to start investigating radar-based
gestures

37. Our approach for
gesture recognition
45

38. Architecture
46
3. Model
2. Pre-processing 4. Output
1. Antenna effect
2. Background scene
3. Time gating
1. Raw data capture
Push
palm
CNN

39. Models
47
3 CNNs
Model 1 Model 2 Model 3

40. Architecture of model 1
48
Model 1 • 3 convolutional
layers
• 2-layered FCNN
• 20 gestures
• 5 participants
• 2 samples per
gesture per
participant
• No data
augmentation

41. Results and discussion (model 1)
• K-fold cross-validation (k=5)
• 94.96% accuracy
• 95.92% precision
• 96.03% recall
49

42. Results and discussion (model 1)
• K-fold cross-validation (k=5)
• 94.96% accuracy
• 95.92% precision
• 96.03% recall
• Confusion between the « show
1/2/3/4 finger(s) » gestures
50

43. A quick summary
51

44. Summary
« Explore the use of CNNs trained with few samples for hand gesture
recognition with radar. »
• 20 gestures recorded with an off-the-shelf radar
• Comparison of 3 CNNs
• Signal pre-processing
• No data augmentation
• ~95% accuracy
• Fine-grained gestures harder to differentiate
52

45. Few samples vs. larger models
+ Better customization and adaptation
• Faster recording
• Faster training
+ Less space required
- Lower accuracy
- Worse at generalizing to other users
53

46. Few samples vs. template matching
+ Better accuracy
+ Better generalization to other users
- Full re-training if the gesture set is modified
- Training time
54

47. Limitations and future
works
55

48. Cover a broader range of participants
• Body size
• Handedness
• Dexterity
• Gesture execution
56

49. Support data augmentation
57
3. Model
2. Pre-processing 4. Output
1. Raw data capture

50. Support data augmentation
58
4. Model
2. Pre-processing 5. Output
1. Raw data capture

51. Support data augmentation
59
4. Model 5. Output
2. Pre-processing
1. Raw data capture 3. Data augmentation
Original Speed Distance Amplitude

52. Explore real-world applications
The best candidates are applications that…
• …must be usable from the beginning, without preliminary training
• …could benefit from user customization/adaptation in long term use
61

53. Explore real-world applications
62
Cooking app

54. Explore real-world applications
63
Cooking app
In-pocket
multimedia controls

55. Explore real-world applications
64
Cooking app
In-pocket
multimedia controls
TV control

56. Thank you for listening

57. Have a look at our paper
for all the details!

58. Scan the QR code to view
our other projects

59. Any questions or
comments are welcome!

60. Model 1
70

61. Model 2
71

62. Model 3
72