Presentation of a full paper @ IUI 2022. Paper available at https://dl.acm.org/doi/10.1145/3490099.3511107

2. Hand Gesture Recognition for an Off-the-Shelf Radar
by Electromagnetic Modeling and Inversion
Arthur Sluÿters, Sébastien Lambot, and Jean Vanderdonckt

3. The Walabot device
Portable FMCW radar
sensor
• EU/CE: 6.3-8GHz
• US/FCC: 3.3-10GHz

4. The Walabot device
Portable FMCW radar
sensor
• EU/CE: 6.3-8GHz
• US/FCC: 3.3-10GHz

5. The Walabot device
18 antennas

6. The Walabot device
18 antennas
2 profiles
Unused Transmitter

7. The Walabot device
18 antennas
2 profiles
• 40 pairs of antennas,
lower framerate (~20fps)
Unused Transmitter Receiver (profile 1)

8. The Walabot device
18 antennas
2 profiles
• 40 pairs of antennas,
lower framerate (~20fps)
• 12 pairs of antennas,
higher framerate (~40 fps)
Unused Transmitter Receiver (profile 1) Receiver (profile 2)

9. The Walabot device
18 antennas
2 profiles
• 40 pairs of antennas,
lower framerate (~20fps)
• 12 pairs of antennas,
higher framerate (~40 fps)
Unused Transmitter Receiver (profile 1) Receiver (profile 2)

10. Why the Walabot?
Technical point-of-view Scientific point-of-view
• Low price
• Off-the-shelf availability
• Portability
• Wide range of configurations
• C++ API for retrieving raw data
• Subject of research in domains such
as activity recognition and material
identification:
• Avrahami et al., 2018
• Agresti et al., 2019
• Only recently used for hand gesture
recognition in a research paper by
Zhang et al., 2021

11. Why the Walabot?
Technical point-of-view Scientific point-of-view
• Low price
• Off-the-shelf availability
• Portability
• Wide range of configurations
• C++ API for retrieving raw data
• Subject of research in domains such
as activity recognition and material
identification:
• Avrahami et al., 2018
• Agresti et al., 2019
• Only recently used for hand gesture
recognition in a research paper by
Zhang et al., 2021

12. The challenges of radar gesture recognition
1. Sensitivity to noise and clutter
• Antenna effects
• Background reflections

13. The challenges of radar gesture recognition
1. Sensitivity to noise and clutter
• Antenna effects
• Background reflections

14. The challenges of radar gesture recognition
1. Sensitivity to noise and clutter
• Antenna effects
• Background reflections
Internal reflections, transmissions

15. The challenges of radar gesture recognition
1. Sensitivity to noise and clutter
• Antenna effects
• Background reflections
Radar

16. The challenges of radar gesture recognition
1. Sensitivity to noise and clutter
• Antenna effects
• Background reflections
Radar
Walls

17. The challenges of radar gesture recognition
1. Sensitivity to noise and clutter
• Antenna effects
• Background reflections
Radar
Furniture

18. The challenges of radar gesture recognition
2. Complexity and size of radar data vs. e.g., skeleton data

19. The challenges of radar gesture recognition
2. Complexity and size of radar data vs. e.g., skeleton data
Identify hand motion from raw radar data

20. The challenges of radar gesture recognition
2. Complexity and size of radar data vs. e.g., skeleton data
Signal from multiple receivers
Identify hand motion from raw radar data

21. Background
subtraction
4
Time gating
6
IFFT
5
Antenna
effects
removal
3
Raw data
capture
1
FFT if time-
domain
radar
2
Filtering
8
Inversion
EM forward
model
Optimization
algorithm
Error
function
7
A pipeline for radar gesture recognition
8 stages offering a solution to
the two challenges of radar
gesture recognition

22. Background
subtraction
4
Time gating
6
IFFT
5
Antenna
effects
removal
3
FFT if time-
domain
radar
2
A pipeline for radar gesture recognition
Challenge 1:
Sensitivity to noise and clutter

23. Background
subtraction
4
Time gating
6
IFFT
5
Antenna
effects
removal
3
FFT if time-
domain
radar
2
A pipeline for radar gesture recognition
Apply the radar equation (Lambot
et al., 2004) to the raw signal
• Internal reflections, transmissions
• Antenna-target interactions

24. Background
subtraction
4
Time gating
6
IFFT
5
Antenna
effects
removal
3
FFT if time-
domain
radar
2
A pipeline for radar gesture recognition
Subtract the background from the
radar signal to remove all static
reflectors (walls, furniture, etc.)

25. Background
subtraction
4
Time gating
6
IFFT
5
Antenna
effects
removal
3
FFT if time-
domain
radar
2
A pipeline for radar gesture recognition
Truncate the signal and keep only
what is relevant for gesture
recognition

26. Filtering
8
Inversion
EM forward
model
Optimization
algorithm
Error
function
7
Challenge 2:
Complexity and size of radar data
A pipeline for radar gesture recognition

27. Filtering
8
Inversion
EM forward
model
Optimization
algorithm
Error
function
7
Extract 2 parameters from the
radar signal (Lambot et al., 2004)
• Distance
• Relative permittivity
A pipeline for radar gesture recognition

28. Filtering
8
Inversion
EM forward
model
Optimization
algorithm
Error
function
7
Filter errors in the distance and
permittivity data caused by:
• Low SNR ratio
• Multiple objects in the field of view
A pipeline for radar gesture recognition

29. a) Open hand b) Close hand c) Open, then close hand d) Swipe right e) Swipe left
f) Swipe up g) Swipe down h) Push with fist i) Push with palm j) Wave hand k) Draw infinity
m) Extend one finger
l) Barrier gesture n) Extend two fingers o) Extend three fingers p) Extend four fingers
Evaluating the pipeline
• 16 gesture classes
• 5 templates/class/
device
• 3 devices
• Walabot
• Custom horn
antenna
• Leap Motion
Controller

30. Background
subtraction
4
Time gating
6
IFFT
5
Antenna
effects
removal
3
Raw data
capture
1
FFT if time-
domain
radar
2
Filtering
8
Inversion
EM forward
model
Optimization
algorithm
Error
function
7
Evaluating the pipeline
• Jackknife recognizer (Taranta
et al., 2017)
• Data at 4 important stages of
the pipeline
• Up to 4 training templates
• Record recognition rate and
execution time

31. Raw data
capture
1
FFT if time-
domain
radar
2
Evaluating the pipeline
T=4 Walabot Horn
Recognition rate [%] 79.7 94.6
Time [ms] 7.3 3.8
Gesture size [kB] 212.5 111.25

32. Background
subtraction
4
Antenna
effects
removal
3
Raw data
capture
1
FFT if time-
domain
radar
2
Evaluating the pipeline
T=4 Walabot Horn
Recognition rate [%] 84.5 95.4
Time [ms] 5.8 3.3
Gesture size [kB] 170 111.25

33. Background
subtraction
4
Time gating
6
IFFT
5
Antenna
effects
removal
3
Raw data
capture
1
FFT if time-
domain
radar
2
Inversion
EM forward
model
Optimization
algorithm
Error
function
7
Evaluating the pipeline
T=4 Walabot Horn
Recognition rate [%] 62.4 55.9
Time [ms] 0.4 0.2
Gesture size [kB] 5 1.25

34. Background
subtraction
4
Time gating
6
IFFT
5
Antenna
effects
removal
3
Raw data
capture
1
FFT if time-
domain
radar
2
Filtering
8
Inversion
EM forward
model
Optimization
algorithm
Error
function
7
Evaluating the pipeline
T=4 Walabot Horn
Recognition rate [%] 70.7 54.5
Time [ms] 0.5 0.2
Gesture size [kB] 5 1.25

35. Guidelines

36. Guidelines
1. Favor gestures with a highly differentiable surface of exposure

37. Guidelines
2. Favor gestures with motion parallel to the radar beam

38. Guidelines
3. Favor large number of training templates (at least 4 in our testing)

39. Guidelines
4. Carefully select the combination of antenna pairs used for gesture
recognition

40. Thank you for your
attention

41. References
Agresti et al., 2019: G. Agresti and S. Milani. 2019. Material Identification Using RF Sensors and Convolutional Neural
Networks. In Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP ’19). 3662–
3666.
Avrahami et al., 2018: Daniel Avrahami, Mitesh Patel, Yusuke Yamaura, and Sven Kratz. 2018. Below the Surface:
Unobtrusive Activity Recognition for Work Surfaces Using RF-Radar Sensing. In 23rd International Conference on
Intelligent User Interfaces (Tokyo, Japan) (IUI ’18). Association for Computing Machinery, New York, NY, USA, 439–451.
Lambot et al., 2004: Sébastien Lambot, E.C. Slob, Idesbald van den Bosch, Benoit Stockbroeckx, and Marnik
Vanclooster. 2004. Modeling of ground-penetrating Radar for accurate characterization of subsurface electric properties.
IEEE Transactions on Geoscience and Remote Sensing 42, 11 (2004), 2555–2568.
Taranta et al., 2017: Eugene M. Taranta II, Amirreza Samiei, Mehran Maghoumi, Pooya Khaloo, Corey R. Pittman, and
Joseph J. LaViola Jr. 2017. Jackknife: A Reliable Recognizer with Few Samples and Many Modalities. In Proceedings of
the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). ACM, New
York, NY, USA, 5850–5861.
Zang et al., 2021: Bo Zhang, Lei Zhang, Mojun Wu, and Yan Wang. 2021. Dynamic Gesture Recognition Based on RF
Sensor and AE-LSTM Neural Network. In 2021 IEEE International Symposium on Circuits and Systems (ISCAS). 1–5.

42. Objective of this thesis:
Apply it to two devices:
Source: Raspberry Projects
Source: UltraLeap
Vision-based
Data easy to interpret
Low privacy
High sensitivity to ambient conditions
Radar-based
Data difficult to interpret
High privacy
Low sensitivity to ambient conditions

43. 1. Radar pre-processing pipeline
Drivers & Libraries
Radar
Radar
driver
Radar API
Global interaction
technique
Dialog
IF gesture = circle
THEN action1()
IF gesture = swipe
THEN action2()
Functional
adapter
Application
core

44. 1. Radar pre-processing pipeline
Drivers & Libraries
Radar
Radar
driver
Radar API
Radar data pre-
processing pipeline
Global interaction
technique
Dialog
IF gesture = circle
THEN action1()
IF gesture = swipe
THEN action2()
Functional
adapter
Application
core

45. Radar pre-processing
pipeline

46. Why a pre-processing pipeline?
Transform radar data for use with template-matching algorithms
(Jackknife, $P+, µF,…)
1. Clean up data
2. Reduce data to only two physically meaningful features
• Distance
• Relative permittivity

47. Antenna
effects
removal
3
Background
subtraction
4
Time gating
6
IFFT
5
Raw data
capture
1
FFT if time-
domain
radar
2
Filtering
8
Inversion
EM forward
model
Optimization
algorithm
Error
function
7
• 8 stages
• Can be adapted to different
types of radars
• One can choose to only use
part of the pipeline

48. Antenna
effects
removal
3
Background
subtraction
4
Time gating
6
IFFT
5
Raw data
capture
1
FFT if time-
domain
radar
2
Filtering
8
Inversion
EM forward
model
Optimization
algorithm
Error
function
7
Push with palm

49. Antenna
effects
removal
3
Background
subtraction
4
Time gating
6
IFFT
5
Raw data
capture
1
FFT if time-
domain
radar
2
Filtering
8
Inversion
EM forward
model
Optimization
algorithm
Error
function
7
Frame
∝
Distance
Antenna effects

50. Antenna
effects
removal
3
Background
subtraction
4
Time gating
6
IFFT
5
Raw data
capture
1
FFT if time-
domain
radar
2
Filtering
8
Inversion
EM forward
model
Optimization
algorithm
Error
function
7
Apply the radar equation to the
raw signal:
• Remove internal reflections,
transmissions
• Remove antenna-target
interactions
Signal independent of the radar

51. Antenna
effects
removal
3
Background
subtraction
4
Time gating
6
IFFT
5
Raw data
capture
1
FFT if time-
domain
radar
2
Filtering
8
Inversion
EM forward
model
Optimization
algorithm
Error
function
7
Body reflection
Hand reflection

52. Antenna
effects
removal
3
Background
subtraction
4
Time gating
6
IFFT
5
Raw data
capture
1
FFT if time-
domain
radar
2
Filtering
8
Inversion
EM forward
model
Optimization
algorithm
Error
function
7
Remove all static reflectors
• Walls
• Furniture
• …
Signal independent of the
environment

53. Antenna
effects
removal
3
Background
subtraction
4
Time gating
6
IFFT
5
Raw data
capture
1
FFT if time-
domain
radar
2
Filtering
8
Inversion
EM forward
model
Optimization
algorithm
Error
function
7

54. Antenna
effects
removal
3
Background
subtraction
4
Time gating
6
IFFT
5
Raw data
capture
1
FFT if time-
domain
radar
2
Filtering
8
Inversion
EM forward
model
Optimization
algorithm
Error
function
7
Truncate the signal to keep only
what is relevant
• Reduces processing time
• Improves accuracy

55. Antenna
effects
removal
3
Background
subtraction
4
Time gating
6
IFFT
5
Raw data
capture
1
FFT if time-
domain
radar
2
Filtering
8
Inversion
EM forward
model
Optimization
algorithm
Error
function
7

56. Antenna
effects
removal
3
Background
subtraction
4
Time gating
6
IFFT
5
Raw data
capture
1
FFT if time-
domain
radar
2
Filtering
8
Inversion
EM forward
model
Optimization
algorithm
Error
function
7
Extract 2 parameters:
• Distance
• Relative permittivity
Iterative method
• Start with an initial guess
• Converge towards the solution

57. Antenna
effects
removal
3
Background
subtraction
4
Time gating
6
IFFT
5
Raw data
capture
1
FFT if time-
domain
radar
2
Filtering
8
Inversion
EM forward
model
Optimization
algorithm
Error
function
7

58. Antenna
effects
removal
3
Background
subtraction
4
Time gating
6
IFFT
5
Raw data
capture
1
FFT if time-
domain
radar
2
Filtering
8
Inversion
EM forward
model
Optimization
algorithm
Error
function
7
Inversion can return incorrect
values due to:
• Low SNR ratio
• Multiple objects in the field of view
Filtering smoothens the signal to
(hopefully) achieve better
recognition rates

59. Antenna
effects
removal
3
Background
subtraction
4
Time gating
6
IFFT
5
Raw data
capture
1
FFT if time-
domain
radar
2
Filtering
8
Inversion
EM forward
model
Optimization
algorithm
Error
function
7

60. Evaluation
• Jackknife recognizer
• Data from different stages of the pipeline

61. • 3 devices: LMC, horn, and Walabot
• 2 recording sessions
• 1 participant
• 5 templates / gesture / device
Horn antenna
Source: Raspberry Projects
LMC
Walabot

62. Antenna
effects
removal
3
Background
subtraction
4
Time gating
6
IFFT
5
Raw data
capture
1
FFT if time-
domain radar
2
Filtering
8 Inversion
EM forward
model
Optimization
algorithm
Error
function
7
Recognition
(a) (b) (c) (d)
Walabot Horn Walabot Horn Walabot Horn Walabot Horn
Recognition rate [%] 79.7 94.6 84.5 95.4 62.4 55.9 70.7 54.5
Time [ms] 7.3 3.8 5.8 3.3 0.4 0.2 0.5 0.2
Gesture size [kB] 212.5 111.25 170 111.25 5 1.25 5 1.25
(a) (b)
(c)
(d)

63. Limitations of the pipeline
Slow inversion
process
Sensitive to
hand size
Information
loss
Only one-
handed

64. Conclusion

65. What has been done?
1. A pipeline for radar data pre-processing
a. Radar gesture set
b. Performance evaluation
2. A framework for prototyping gesture-based applications with the
LMC
a. Various demo applications
b. Usability testing

66. What’s next?
Radar pipeline
• Real-time execution
• Testing
• Gesture set
• Recognizers
zeroG
• Design
• Implementation
• Modules
• Sensors
• …
• Evaluation

67. Thank you for your
attention

68. Bonus slides

69. LMC datasets
Horn dataset (16 joints) Walabot dataset (16 joints)

70. Horn dataset
Raw frequency data Filtered frequency data

71. Horn dataset
Raw inversion Filtered inversion data

72. Walabot dataset
Raw frequency data (pairs 2, 3, 4, 6, 7) Filtered frequency data (pairs 4, 6, 7, 10)

73. Walabot dataset
Raw inversion data (pairs 2, 3, 6, 7) Filtered inversion data (pairs 2, 3, 6, 7)

74. Walabot dataset
Filtered frequency data (pairs 4, 6, 7, 10) Filtered inversion data (pairs 2, 3, 6, 7)

75. a) Open hand b) Close hand c) Open, then close hand d) Swipe right e) Swipe left
f) Swipe up g) Swipe down h) Push with fist i) Push with palm j) Wave hand k) Draw infinity
m) Extend one finger
l) Barrier gesture n) Extend two fingers o) Extend three fingers p) Extend four fingers