Project GaitNet: Ushering in the ImageNet moment for human Gait kinematics The emergence of the upright human bipedal gait can be traced back 4 to 2.8 million years ago, to the now extinct hominin Australopithecus afarensis. Fine grained analysis of gait using the modern MEMS sensors found on all smartphones  not just reveals a lot about the person’s orthopedic and neuromuscular health status, but also has enough idiosyncratic clues that it can be harnessed as a passive biometric. While there were many siloed attempts made by the machine learning community to model Bipedal Gait sensor data, these were done with small datasets oft collected in restricted  academic environs. In this talk, we will introduce the ImageNet moment for human gait analysis by presenting 'Project GaitNet', the largest ever planet-sized motion sensor based human bipedal gait dataset ever curated. We’ll also present the associated state-of-the-art results in classifying humans harnessing novel deep neural architectures and the related success stories we have enjoyed in transfer-learning  into disparate domains of human kinematics analysis.

1. Project GaitNet
Ushering in the ImageNet moment for human gait kinematics
1
Vinay Uday Prabhu, Ph.D.
Carnegie Mellon University
Chief Scientist , UnifyID AI labs
@vinayprabhu

2. TALK ORGANIZATION
1. INTRODUCTION TO HUMAN BIPEDAL
GAIT
2. IDENTIFYING HUMANS BY GAIT
3. GAITNET 1.0
4. GAIT CLASSIFICATION: CHALLENGES
5. BEYOND GAIT: TRANSFER LEARNING!
2

3. INTRODUCTION
TO HUMAN
BIPEDAL GAIT
3

4. Circa 1976. A team of paleoanthropologists
lead by Mary Leakey make a discovery at
‘Site G’ in Laetoli, Tanzania
4
Source: https://en.wikipedia.org/wiki/Laetoli#Hominid_footprints,
https://discovermagazine.com/2015/oct/19-first-impressions?spJobID=660139281&spMailingID=23650443&spReportId=NjYwMTM5MjgxS0&spUserID=MTE2MDc
3MjkyNjA0S0, https://elifesciences.org/articles/19568

5. It’s an early hominin footprint trail!
27 m (88 ft) long and ~70 footprints
5
Source: https://en.wikipedia.org/wiki/Laetoli#Hominid_footprints,
https://discovermagazine.com/2015/oct/19-first-impressions?spJobID=660139281&spMailingID=23650443&spReportId=NjYwMTM5MjgxS0&spUserID=MTE2MDc
3MjkyNjA0S0, https://elifesciences.org/articles/19568

6. 3 Hominins: G1 : 4.2ft, G2/G3: 5ft walked on
wet volcanic ash ~ 3.5 m years ago!
6
Source: https://en.wikipedia.org/wiki/Laetoli#Hominid_footprints,
https://media.nature.com/m685/nature-assets/srep/2016/160223/srep21916/images/srep21916-f1.jpg,
https://discovermagazine.com/2015/oct/19-first-impressions?spJobID=660139281&spMailingID=23650443&spReportId=NjYwMTM5MjgxS0&spUserID=MTE2MDc
3MjkyNjA0S0, http://humanorigins.si.edu/evidence/behavior/footprints/laetoli-footprint-trails

7. Australopithecus afarensis- bipedalism
7
Source: https://en.wikipedia.org/wiki/Laetoli#Hominid_footprints, https://www.britannica.com/topic/Australopithecus, https://www.youtube.com/watch?v=ygydY3NMDxw

8. Malanga, Gerard, and Joel A. DeLisa. "SECTION ONE." Gait Analysis In The Science Of Rehabilitation 2 (1998): 1.
Source: https://www.youtube.com/watch?v=8NwX6Cl4Uo4, https://www.youtube.com/watch?v=6ObNnCTV6MY
> 20 muscle groups engaged during a Gait cycle

9. The RLA system of
Gait segmentation
Source:https://www.orthobullets.com/foot-and-ankle/7001/gait-cycle
9

10. IDENTIFYING
HUMANS BY GAIT
10

11. Identifying humans by gait in popular culture:
Julius Caesar → Act 1,
Scene 3, Page 6
CASSIUS
'Tis Cinna. I do know
him by his gait.
He is a friend.—Cinna,
where haste you so?
Hamlet → Act 3, Scene 2, Page
2
.. that, neither having th' accent
of Christians nor the gait of
Christian, pagan, nor man, have
so strutted and bellowed that I
have thought some of nature’s
journeymen
The Tempest → Act 4, Scene
1, Page 5
CERES
Highest queen of
state,
Great Juno, comes. I know
her by her gait.
William
Shakespeare
11

12. Mission: Impossible - Rogue Nation (2015)
“Unfortunately, even if you can make it through every other security measure… ..you won't beat the last one.
That's because it's protected by gait analysis.” 12

13. Modalities:
1. Video
2. Pressure sensor
3. Audio
4. WiFi
5. Accelerometery
and motion
sensors
13

14. IMU - Microelectromechanical systems (MEMS) - sensors
14

15. Can we list them? Samsung Phone? Type *#0*#
15

16. Tri-axial MEMS accelerometers
16
Source:
1: https://www.youtube.com/watch?v=eqZgxR6eRjo
2: https://www.youtube.com/watch?time_continue=142&v=KZVgKu6v808
3: https://busy.org/@mike11/accelerometer-in-a-smartphone-how-it-works

17. Tri-axial MEMS gyroscopes
17
Source:
1: https://www.youtube.com/watch?v=eqZgxR6eRjo
2: https://www.youtube.com/watch?time_continue=142&v=KZVgKu6v808
3: https://busy.org/@mike11/accelerometer-in-a-smartphone-how-it-works

18. My bipedal gait cycle measured with IMUs
18

19. A snap-shot of a single Gait cycle
19

20. 20
N X 4 X 100
Gait-tensor
Segment-Interpolate-Tensorize

21. 21
GAITNET
DATASET

22. GaitNet 1.0
22

23. ImageNet versus GaitNet
GaitNet:
1.2M+ Gait cycles
1000+ classes
117 countries
23
ImageNet:
1.2M images
1000 classes

24. The Gait profile of a random sampling of
classes
24

25. GAIT
CLASSIFICATION
CHALLENGES 25

26. 26
Challenge-1: Multi-modal (High intra-class variation)

27. 27
Challenge-1: Multi-modal (High intra-class variation)

28. 28
Challenge-1: Multi-modal (Highly tangled)
PCA UMAP

29. 29
Challenge-2(a): Acoustic injection attacks

30. 30
Challenge-2(b): Adversarial perturbations
In computer vision ..
https://spectrum.ieee.org/the-human-os/robotics/artificial-intelligence/hacking-the-brain-with-adversarial-images

31. Challenge-3: Hybrid architectures- Notoriously hard to train
31

32. Out of the box ideas ...

33. Results: ~70% per walk accuracy on 1534
class problem!
33

34. Disentanglement achieved!
34

35. Transfer
learning
35

36. 36
Transfer learning
Source: https://towardsdatascience.com/a-comprehensive-hands-on-guide-to-transfer-learning-with-real-world-applications-in-deep-learning-212bf3b2f27a

37. Transfer learning: ‘cornerstone of computer vision’
37

38. 38

39. Every major framework has
extensive set of tutorials on TL
39

40. Accelerometric Gait to Human Squats

41. Accelerometric Phone Gait to
Gyroscopic IMU squats
1. Trans-device
2. Trans-sensor
3. Trans-activity

42. How does the data look like?

43. Recipe: Strategically freeze layers + Add new
domain-specific FC layers
Freeze
Trainable

44. Results: Acc2Acc and Acc2Gyr

45. Further
applications
45

46. Small cohort user re-identification
46

47. Device location classification
47

48. Human activity classification
48

49. Does this mean fear of surveillance akin to
Video-based gait recognition?
49

50. 50
Source: http://image-net.org/challenges/talks_2017/imagenet_ilsvrc2017_v1.0.pdf

51. What does being on the cusp of the GaitNet
moment entail?
51
GAITNET
DATASET

52. Differential diagnosis between Parkinson's disease and
essential tremor using the smartphone's accelerometer
● Sergi Barrantes, Antonio J Sánchez Egea, +6 authors Josep Valls-Solé
● Published 2017 in PloS one
● DOI:10.1371/journal.pone.0183843
52

53. Source:
https://www.semanticscholar.org/paper/Implementation-of-a-smartphone-as-a-wireless-for-in-LeMoyne-Mastroianni/bf8ea7c
6ad9315ebd285f623b21e542a32377f01
3:
53

54. Source:
http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0141694
54

55. 55

56. 56

57. Individual models trained on small siloed
datasets —> Transfer-learned models
fine-tuned using model pre-trained on GaitNet
57
GAITNET

58. Reference publications
[1] Stephanie Tietz, Vinay Uday Prabhu, 'GaitID-2-SquatID: Deep transfer learning for
human kinematics', Time Series workshop, ICML-2019, June-2019, Long Beach, California,
USA
[2] Vinay Prabhu, Stephanie Tietz and Anh Ta, 'Classifying humans using Deep time-series
transfer learning : accelerometric gait-cycles to gyroscopic squats , Proceedings, 5th
SIGKDD Workshop on Mining and Learning from Time Series (MiLeTS), KDD-2019,
Anchorage, Aug-2019, Alaska
[3] Vinay Prabhu, John Whaley and Mihail D, 'MODHILL: A framework for debugging gait
in multi-factor authentication systems', Workshop on Human In the Loop Learning
(HILL)-ICLR 2019, May 2019, New Orleans, USA
[4] Vinay Prabhu, John Whaley ,”Vulnerability of deep learning-based gait biometric
recognition to adversarial perturbations", In Proceedings of the CVPR 2017 CV-COPS
workshop, 2017,Honolulu, Hawaii — July 21, 2017.
Code:
https://github.com/vinayprabhu/Gait_FGSM
https://github.com/vinayprabhu/GaitID-2-SquatID

59. https://github.com/vinayprabhu/GaitID-2-SquatID
Vinay Prabhu, Stephanie Tietz
and Anh Ta, 'Classifying humans
using Deep time-series transfer
learning : accelerometric
gait-cycles to gyroscopic squats'
, Proceedings, 5th SIGKDD
Workshop on Mining and
Learning from Time Series
(MiLeTS), KDD-2019, Anchorage,
Aug-2019, Alaska

60. THANK
YOU!
60