This document summarizes research using the Microsoft Kinect v2 depth camera for markerless motion capture of vehicle occupants. The Kinect was able to track joints in real-time and its portability allowed testing in an actively moving vehicle. Software was developed to improve data accuracy, including removing background noise, reducing noise, and predicting joint locations. Various motions were tested with and without a vehicle mockup and compared to a Vicon motion capture system. The results indicate the Kinect system is reliable for motion analysis in vehicles but has limitations when joints are occluded. Further research will explore using multiple Kinect sensors simultaneously and improving joint tracking through motion databases.

1. Marker-less motion capture systems using depth cameras,
such as Microsoft Kinect Sensor, recently became a hot issue
in auto industry for their size, price, performance, and
portability. The marker-less motion and face tracking system
is expected to replace the typical optical motion capture
systems and reduce set-up and motion capture time greatly.
The advantages of size and portability of the depth sensor
will enable testing it in an actively running vehicle. In this
study, the Microsoft Kinect v2 depth camera was used to
gather 2.5-dimensional color and depth (RGB-D) data along
with hypothesized skeletal linkage data in real time.
In order to improve the accuracy and performance of the
obtained data, various techniques were implemented in the
developed software, including background removal, face
recognition, noise reduction, and model-based joint location
prediction. Various motions were tested to validate the
system with and without a vehicle mockup structure while
compared to the golden reference data obtained from Vicon
system.
Authors: Hwan Lee, Nevin Mital, Christopher Atkins; Matthew P. Reed, PhD; and Byoung-Keon D. Park, PhD
CONCLUSION
RESULT
11:24:31:850 11:24:38:515 11:24:45:379 11:24:52:92 11:24:58:756
2.0
2.2
2.4
2.6
2.8
3.0
3.2
3.4
A
11:24:31:850 11:24:38:515 11:24:45:379 11:24:52:92 11:24:58:756
1800
2000
2200
2400
2600
2800
3000
3200
3400
C20
A
0 500 1000 1500 2000 2500
0
100
200
300
400
500
Head x
0 500 1000 1500 2000 2500
200
300
400
500
600
700
Head y
0 500 1000 1500 2000 2500
2000
2100
2200
2300
2400
2500
Head z
0 500 1000 1500 2000 2500
-100
-50
0
50
100
150
200
250
300
350
400
Left Shoulder x
0 500 1000 1500 2000 2500
100
150
200
250
300
350
400
450
500
550
600
Left Shoulder y
0 500 1000 1500 2000 2500
2100
2200
2300
2400
2500
Left Shoulder z
0 500 1000 1500 2000 2500
200
250
300
350
400
450
500
550
600
650
700
Right Shoulder x
0 500 1000 1500 2000 2500
0
50
100
150
200
250
300
350
400
450
500
Right Shoulder y
0 500 1000 1500 2000 2500
2050
2100
2150
2200
2250
2300
2350
2400
2450
2500
Right Shoulder z
ABSTRACT
FUTURE WORK
ACKNOWLEDGEMENT
This research was supported by Ford
Motor Company and the Center for Child
Injury Prevention Studies (CChIPS)
Fragment of original joint depth data (top) and
interpolation processed depth data (bottom)
Comparision of aligned skeleton
data from Vicon and Kinect
SOFTWARE DEVELOPMENT
Kinect sensor tracks 25 joints in real time. Missing joints are marked in red color.
 Marker-less motion capture system with data refinement software
is reliable and can be used for variety of motion analysis and
safety vehicle system including ingress/egress, human machine
interaction (HMI), child backseat safety, doze driving warning,
and crash motion analysis, etc.
 Kinect sensor’s portability benefits variety of testing environment
including testing in active running vehicles which marker-based
motion capture systems cannot achieve.
 Kinect sensor range was adequate for vehicle motion capture.
 Noise reduction, interpolation, and segment fixing technique
enhanced data quality.
 Marker-less motion capture depth sensors do not require any
marker set attached on human body and calibration process
which saves a lot of testing time.
 Single Kinect sensor is limited to track hidden joints occluded by
other body parts or obstacles.
Kinect sensor raw data has non-linear spatial depth-resolution problem. Post data
processing software was developed to reduce noise and interpolate missing points.
To overcome Kinect sensor resolution issue,
joint location refinement software was
developed.
Processed Kinect data (red) vs Vicon data for rising arms motion
 Processed depth data quality depends on the original depth data
obtained from Kinect sensor. Depends on Kinect direction and
subject orientation, joint locations could be reliable/unreliable.
To overcome this issue, multiple Kinect real-time
synchronization development will be considered.
 Using depth data and Vicon data, software will be developed to
improve prediction of poorly tracked motion by matching
motions from motion database pool containing numerous
postures and behaviors of humans.
Monitoring and Modeling Vehicle Occupant Motions