3. Some measures such as electroencephalography (EEG), heart rate,
and skin conductance might provide insight into driver state [8], [9],
but they are uncomfortable, expensive and inappropriate for
commercial applications. Although many systems have been
proposed to analyze eye movements and detect eye position [10]-
[12], few have been extensively tested in on-road driving
environments. The differences between simulator and naturalistic
settings are substantial, and often result in severe reductions in eye
measure reliability [13], [14].
An alternative approach is to exploit sensors that already exist in
current production vehicles and which generate signals that are
already consumed by other vehicle systems. Examples of these
vehicle-based signals include steering wheel angle, vehicle speed and
acceleration, and pedal position. An intermediate option makes use of
cameras or other advanced sensors to measure signals such as lane
deviation and time to lane crossing. Ostlund et al. [15] examined
many vehicle-based measures in relation to visual and cognitive
distraction and recommended several for potential use in driving
performance assessment applications. Greenberg et al. [16] found that
distraction had effects on lane position, following distance and
heading error. Kaber et al. [17] examined the effects of visual and
cognitive distraction on steering smoothness and headway time and
found that drivers increased their headway time when visually
distracted. Liang and Lee have mixed driver-based and vehicle-based
signals to train cognitive distraction algorithms [6], [18].
Defining Driver Distraction
A clear definition of driver distraction is central to the process of
distraction detection and mitigation. Distraction ground truth is
important for training algorithms and requires the interpretation of
distraction to obtain a gold standard set of data. Unfortunately,
considerable uncertainty surrounds the definition of driver distraction.
Studies have proposed a wide variety of definitions, centering on
multiple aspects of the phenomenon. Regan, Lee and Young [19]
attempted to unify these divergent definitions by defining driver
distraction as, “the diversion of attention away from activities critical
for safe driving toward a competing activity.” Implicit in this
definition is the relationship between the attentional demands of the
driving environment and the attention devoted to this environment by
the driver. Distraction represents inadequate attention to the driving
environment relative to the roadway demands, exceeding a driver’s
attentional capacity. The interaction between roadway demand and
task intensity has been considered in a limited fashion [20], [21].
Many previously developed algorithms consider distraction as a state
that is independent of the environment [2], [4], [7], [22]. Thus,
distraction is defined only by a state of mind. The consequences of
distraction may be more severe when roadway demand is greater.
However, just because roadway demand is low does not mean that
distraction is tolerable. NHTSA distraction guidelines are designed to
test driver-vehicle interfaces (DVI) in a low demand environment
[23]. Task interactions that require too much attention in the low-
demand environment constitute a sufficient condition for redesign.
Distraction can be linked to drivers’ glance patterns, and glances
away from the road at inopportune times can increase crash risk. Data
from the 100-Car Naturalistic Driving Study were analyzed during
the period of 2001-2003. These data only include driver behavior
immediately before a safety-critical event that triggered the recording
device in the car, such as a sudden deceleration, swerving, or a crash.
It was observed that 93% of all lead-vehicle crashes occurred when
the driver was inattentive, and four of the top five types of inattention
were linked to glances away from the roadway [24]. Glance times
exceeding more than two seconds away from the road were estimated
to increase near-crash/crash risk by at least twice [25]. Recent
analysis of the SHRP2 naturalistic data found that glances away from
the road longer than two seconds significantly increased the odds of a
crash or near crash [26]. Interestingly, a protective effect of talking on
the phone prior to near crashes was also observed. These results
motivate the consideration of a glance-based metric as ground truth
for visual distraction
Algorithm Design
Similar to the variety of distraction definitions and sensors, many
different algorithms have been employed to detect impairment. These
include traditional machine learning algorithms such as support
vector machines (SVM) [6], [22], Neural Networks [27], graph based
models such as Hidden Markov Models (HMM) [28], temporal graph
based models [29], [30] and deep learning approaches [31]. All of
these methods have been demonstrated with some degree of success
and offer several promising directions for further development.
Ensemble techniques use combinations of algorithms to detect
impairment. Random forests are a very successful example of
ensemble techniques that combines the results of hundreds of simple
decision trees to make a classification [32]. Combining data from
multiple sensors is more effective than relying on a single sensor; and
likewise, combining estimates from multiple algorithms can be more
robust than relying on one.
The deep learning approach is particularly interesting because it
redefines the relationship between feature identification and algorithm
development. Rather than considering these as two separate steps,
deep learning approaches build the feature engineering process into
the model training process [31]. Deep learning goes beyond simply
detecting a difference from baseline behavior, it develops a model
that generates expected behavior and thus can detect impairment by a
failure of drivers to produce the expected behavior [22]. In such a
system, the focus becomes one of predicting drivers’ maneuvers and
using divergence from the predicted outcome as indicators of
impairment [20]. This approach increases computational difficulty
and complexity, but it promises to enhance algorithm accuracy and
robustness [33].
Secondary Tasks
The secondary tasks selected for this study were representative of
ones that drivers perform in the real world, increasing face validity
and promoting well-learned interaction with the task. Secondary task
difficulty has the potential to greatly impact driving performance;
therefore, two levels of task difficulty were used in this study to vary
the amount of distraction.
While Dingus et al. [24] and Klauer et al. [25] showed that tasks of
different difficulties have varying safety impacts, other experimental
research has explored the relationship between task difficulty and
driving performance more directly. Lanseown, Brook-Carter and
Kersloot [34] showed compensatory changes in speed and lateral
Downloaded from SAE International by John Lee, Wednesday, March 16, 2016
4. vehicle control in the presence of secondary tasks as additional
secondary tasks were added. Chisholm, Caird, Lockhard, Fern and
Teteris [35] showed that task difficulty affected performance and
although repeated exposure reduced dual-task costs for simple tasks,
performance under more difficult tasks did not improve with practice.
Blanco, Biever, Gallagher and Dingus [36] manipulated the level of
cognitive engagement by requiring interpretation and planning in
responding to some questions (e.g. “Select the quickest route after
being shown distance and speed limits for three alternative routes”).
They also varied the information density of the displays by using
tables, paragraphs, and graphics. The Human machine interface And
the Safety of Traffic in Europe (HASTE) project tasks also varied in
difficulty: Jamson and Merat [37] demonstrated a direct effect on
driving performance measures by increasing the complexity of the
arrows and auditory continuous memory task.
NADS Driving Impairment Research
The National Advanced Driving Simulator (NADS) has been
involved in driving impairment research for many years and with
many types of impairment. A standard database and set of scenarios
were developed and have been used for all impairment studies
conducted since around 2008. The scenario mimics a nighttime drive
from an urban area to a rural setting via a freeway, and is described in
more detail in the Methodology section.
The first study to make use of the standard scenario was an alcohol-
intoxication study [38]. Several machine learning algorithms were
explored to estimate intoxication from vehicle-based sensors. These
included logistic regression, decision trees and SVMs. Drowsiness
and distraction studies were conducted under the Driver Monitoring
of Inattention and Impairment Using Vehicle Equipment (DrIIVE)
program for NHTSA [39]-[42]. The distraction studies tested several
vision-based distraction-detection algorithms and implemented one
into the simulator environment. Mitigation results compared drivers’
acceptance of real-time and post-drive systems [43]. The drowsiness
study tested drivers during daytime, early night and late night
conditions and tested several types of algorithms including Bayesian
Networks and Random Forests with different notions of drowsiness
ground truth [42]. This project produced a Random Forest algorithm
that was successful at estimating episodic drowsiness at least six
seconds in advance of a drowsy-induced lane departure [44].
This paper fits into the second, final phase of DrIIVE, the overarching
theme of which is to detect and differentiate multiple types of
impairment using vehicle-based sensors. A second-phase DrIIVE
project implemented a real-time drowsiness mitigation system on top
of the Random Forest detection algorithm and tested the effectiveness
of different alert types and modalities on driver performance [45].
The basic framework for the multiple impairment detection was
conceived as a hierarchical combination of Random Forests and
Hidden Markov Models [46].
Methodology
A driving simulator study was conducted to collect a large array of
data from drivers in distracted and undistracted states. This section
describes the methodology used in collecting the data, including the
apparatus, secondary tasks and experimental design.
Apparatus
The National Advanced Driving Simulator (NADS) is located at The
University of Iowa. The main simulator is called the NADS-1. It
consists of a 24-foot dome in which an entire car cab is mounted. All
participants drove the same vehicle-a 1996 Malibu sedan. The motion
system, on which the dome sits, provides 400 square meters of
horizontal and longitudinal travel and ±330 degrees of rotation. The
driver feels acceleration, braking, and steering cues much as if he or
she were actually driving a real vehicle. High frequency road
vibration up to 40 Hz is reproduced from vibration actuators placed
in each wheel well of the cab. A picture of the NADS-1 simulator and
an image from the interior of the dome are shown in Figure 1.
The NADS-1 displays graphics by using sixteen high definition
(1920×1200) LED projectors. These projectors provide a 360 degree
horizontal 40 degree field of view. The visual system also features a
custom-built Image Generator (IG) system that is capable of
generating graphics for 20 channels (16 for the dome and 4 additional
for task-specific displays), and which performs warping and blending
of the image to remove seams between projector images and display
scenery properly on the interior wall of the dome. The NADS
produces a thorough record of vehicle state (e.g., lane position) and
driver inputs (e.g., steering wheel position), sampled at 240 Hz.
The cab is equipped with a Face Lab™ 5.0 eye-tracking system that
is mounted on the dash in front of the driver’s seat above the steering
wheel. The worst-case head-pose accuracy is estimated to be about
5°. In the best case, where the head is motionless and both eyes were
visible, a fixated gaze may be measured with an error of about 2°.
The eye tracker samples at a rate of 60 Hz.
Figure 1. NADS-1 driving simulator (left) with a nighttime driving scene
inside the dome (right).
Participants drove a set of nighttime scenarios that have been
developed as standard environments for NADS driving impairment
research. Each drive was composed of three driving segments. The
drives started with an urban segment composed of a two-lane road
through a city with posted speed limits of 25 to 45 mph with signal-
controlled and uncontrolled intersections. An interstate segment
followed that consisted of a four-lane divided expressway with a speed
limit of 70 mph. After merging onto the interstate segment, drivers
made lane changes to pass several slower-moving trucks. The drives
concluded with a rural segment composed of a two-lane undivided
road with curves and a section of gravel road. These three segments
mimicked a drive home from an urban parking spot to a rural location
via an interstate. Nineteen separate events (e.g. yellow light dilemma,
left turn) combined to provide a representative trip home. Drivers
encountered situations that might be encountered in a real drive. Each
drive was approximately 25 minutes long.
Downloaded from SAE International by John Lee, Wednesday, March 16, 2016
5. Secondary Tasks
Two secondary tasks were designed to create varying levels of
distraction for the driver. Both tasks were presented on the same
display screen. The location of the display screen conformed to
Alliance Guidelines (Driver Focus-Telematics Working Group, 2006)
[47]. The display screen was on the center stack near the heating and
air conditioning controls of the vehicle. The downward viewing angle
was less than 30° and the lateral viewing angle was less than 40°.
However, the location required a head turn by the driver to interact
with the display. Each task was implemented at moderate and high
levels of difficulty.
The purpose of the secondary tasks was to provide clearly marked
driving segments with measureable task engagement. The tasks were
self-paced; that is, the drivers were given the freedom to delay task
engagement and to determine their own task completion pace. This
allowed natural patterns of task chunking and interruptions to be
observed in the data.
Visual Distraction Task
This visual-only task required drivers to read text aloud from a
display. For each task, an auditory prompt, “Read Message,” alerted
the driver that the message was ready to be read. Each message was
roughly the length of a SMS text message (not exceeding 160
characters). Messages contained “interesting facts” and were written
for a Flesch-Kincaid grade level between 6.5 and 8. The participant
read the message aloud. The task contained two levels of difficulty.
The high level of difficulty was achieved by increasing the message
length, removing grammar and line breaks from the text and causing
phrases to run together. An example of the moderate and difficult text
reading tasks is shown in Table 1.
Task engagement began when the participant spoke the first word of
the message and ended with the last word of the message. Voice key
software was used to detect speech from the driver, and a researcher
also marked the beginning and end points of speech in the data
stream. The researcher cleared the message from the screen when the
participant completed reading the message.
Table 1. Difficult message task
Visual-Manual Distraction Task
The visual-manual task required drivers to search through a list of
songs and select a target song from the list. Each target song was
presented to drivers only once and was not repeated. The moderate
difficulty used a list of five songs on a single menu page. The high
level of difficulty was achieved by using a longer list of 15 sings
spanning three menu pages, and possibly requiring one or more page
scrolls to find the target title. An example of the moderate and
difficult list-selection tasks is shown in Table 2
Table 2. Difficult list-selection task
Drivers received an audio prompt, “Find [song title],” instructing
them to begin the task. The beginning of task engagement was
recorded as the driver’s first touch to the display screen to initiate a
scroll. Drivers responded by manually scrolling to the page with the
target song and selecting it, marking the end of task engagement.
Experiment
A 3 (distraction levels) × 3 (order) × 2 (gender) mixed design exposed
participants to three distraction levels in three different orders.
Between-subjects independent variables were gender and order of the
distraction drives. The within-subject independent variable was
distraction level: no distraction, moderate distraction, and high
distraction. The experimental matrix for the study is shown in Table 3
Table 3. Experimental Conditions
Participants
Participants were recruited from the NADS participant database and
through newspaper ads, internet postings, and referrals. An initial
telephone interview determined eligibility for the study. Potential
participants were screened for health history and current health status.
Downloaded from SAE International by John Lee, Wednesday, March 16, 2016
6. Pregnancy, disease, sleep disorders, or evidence of substance abuse
resulted in exclusion from the study. Potential participants taking
prescription medications that cause or prevent drowsiness also were
excluded from the study. If all telephone-screening criteria were met,
individuals were scheduled for a screening visit. If all eligibility
requirements were met during the screening visit, participants were
scheduled for the data collection visit.
A single age group of 21-34 years was selected for this study. This
group represented adult drivers who have the longest expected
number of years remaining as drivers and who are more likely to
multitask while driving.
Procedures
Each participant drove the simulator three times - once in the baseline
distraction condition, once in a moderate distraction condition, and
once in a high distraction condition. For each of the three scenarios,
there were the same number of curves and turns, but their order
varied. For example, the position of the left turn in the urban section
varied so that it was located at a different position for each drive.
Additionally, the order of left and right rural curves varied between
drives. These examples of order variations mitigated the learning
effect experienced on the second and third drives.
Additionally, the order of the drives for the different levels of
distraction were counterbalanced using a Latin Square. Driving
sessions in the simulator alternated between participants to reduce
carryover effects from one distraction level to another as well as to
reduce simulator-induced fatigue.
Algorithm Design
General impairment-detection algorithms and warning systems that
use low-cost vehicle-based sensor suites would be attractive as they
could be adopted quickly and avoid drivers’ privacy objections. The
challenge to such an approach lies in the creation of an algorithm that
is effective at detecting the impairment. Specific challenges in
algorithm design include choosing inputs (or features) that are
sensitive to the impairment, selecting an appropriate ground truth
signal, and choosing from among many machine learning models.
Input Signals
The data from each drive were segmented into consecutive one-
second windows. Raw simulator data were aggregated in each
window by calculating an appropriate statistic on the segment, such
as the mean. If a variable took on only integer values then the mode
was used instead to ensure that an admissible value was obtained.
While further aggregation into coarser segments was done for other
impairment algorithms, the distraction data were left as one-second
segments. This is an appropriate timescale on which to measure and
classify the distraction impairment.
A large set of signals were collected from the simulator data to be
used as inputs to the algorithm. For algorithm development, there
isn’t really a downside to including many inputs, and it is possible to
examine the relative importance of inputs in a trained Random Forest
model. The signals that were used, as well as the statistics used to
aggregate the segments are listed in Table 4. The descriptive statistics
that were used included the following: mean, standard deviation (sd),
maximum (max), mode, and peak-to-peak (p2p).
Table 4. Algorithm input signals
Common driving maneuvers like turning and driving on curves
complicate the use of steering wheel angle as an input to the
algorithm. A trend in the steering signal may be caused by an artifact
from the driving environment, such as the gradual appearance of
steering adjustments (a signal bias) on a curve. These trends can be
characterized by their lower frequency content. On one hand, the
presence of steering trends might confound the training of an
algorithm, causing it to detect biased variations in the trend rather
than informative steering signal content. On the other hand, the
process of removing trends from the signal could potentially strip
vital information`, causing driver state classification to suffer.
Eskandarian et al. [48] subtracted the mean steering angle over the
length of a curve. This approach is not appropriate for real-time
implementation because it uses steering samples that would be
collected in the future. Alternatively, road curvature from a GIS
database, such as from a navigation system, might be used. However,
drivers do not always follow the curvature of the road, especially
when entering or exiting a curve, so GIS data also does not offer a
perfect solution.
Brown’s Double Exponential Smoothing (DES) method [49] can
remove low steering frequencies at the scale of geographic features.
Moreover, its frequency cutoff can be easily adapted to different road
curvatures with a single parameter. A scheme to adapt this parameter
for gentle curves as well as tight turns was developed and tested on
simulator data. The steering signal was first low-pass filtered to
remove noise. This filtered signal, as well as steering wheel angle
detrended using the DES filter, are shown in Figure 2. The steering
signals for four types of curve in the figure illustrate the effect of the
DES filter in removing steering bias, though the short duration and
Downloaded from SAE International by John Lee, Wednesday, March 16, 2016
7. low radius of curvature in the left turn make that case particularly
challenging. De-trended versions of steering and steering rate were
computed and are denoted as str_des and strd_des in Table 4.
Figure 2. Original (blue) and de-trended (green) steering wheel angle for four
roadway types: a left turn (upper left), curved roads in urban setting (upper
right), curved roads on highway (lower left) and curved roads in rural setting
(lower right).
The lanedevmod signal is based on the modified standard deviation of
lateral position (MSDLP) measure evaluated in the AIDE program
[15]. The signal is calculated from a 0.1 Hz high-pass filtered lane
deviation. Table 4 specifies the use of mean and standard deviation of
the one-second segments. The latter corresponds to MSDLP. While
the statistics can be calculated on one-second segments, the high pass
filter should be applied to windows of at least ten seconds.
Some environmental signals are included in the list. The curvature
signal reports the radius of curvature of the roadway in that segment.
The events signal reports the active event number in the segment. The
q measure reports the roadway demand metric that combines several
features of the roadway environment [50]. These signals are included
as proxy measures for a number of environmental inputs that might
realistically be included in a production system.
Driver-based sensor signals may provide the most direct measure of
impairment generally and could be expected to benefit a distraction-
detection algorithm. Visual distraction would especially benefit from
sensors that monitor the drivers gaze direction; however, there are
obstacles to the approach. One problem is that robust eye-tracking is
difficult to achieve in all realistic lighting conditions, and drivers may
not accept this type of monitoring technology. Another is that an
eye-tracking algorithm was used to establish ground truth for distraction
so an eye-based detection algorithm would have an unfair advantage.
A compromise is to consider head pose as a driver-based signal since it
is easier to detect than gaze location, but does not duplicate the
information gleaned from gaze location. Examples of head pose signals
are shown in Figure 3. Notice the clear pattern of horizontal head
rotation that is associated with task engagement. The secondary tasks
in this study were in one fixed location; but in reality, there are infinite
ways for a driver to look away from the road leading to distraction.
Figure 3. Mean vertical and horizontal head pose signals with task
engagement indicated by shaded regions.
Ground Truth
As discussed above, the definition of ground truth for distracted
driving is a challenge. Does task engagement equal distraction? If
not, when does distraction begin and what constitutes the threshold
that separates distraction from alertness? NHTSA’s distraction
guidelines penalize glances longer than two seconds, as well as total
task time greater than 12 seconds [23]. Analysis of crashes and near
crashes from the SHRP 2 dataset reveals the particularly risky nature
of long glances [26].
A slightly modified version of the AttenD [51] algorithm was used to
define ground truth for distraction. AttenD uses gaze location and
requires an eye tracker to collect. It defines a 90 degree horizontal
field of view in which gaze is interpreted to be on the road. The
allowed vertical field of view is 22.5 degrees downward from center.
Glances outside of the front field of view are tested to see whether
they are aimed at any of the mirrors, which are also allowed. Failing
to meet the mirror exception, AttenD accumulates eyes-off-road time
and indicates distraction when a threshold is crossed. When the driver
returns their gaze to the front, the output begins to drop, after a short
delay, at the same rate until it reaches zero. Distraction is indicated
when the output exceeds a nominal threshold of two seconds. Finally,
the output is limited at a maximum value, set to 2.05 in this project.
The higher this limit is set, the longer it takes for the output to fall
below the distraction threshold; and the selected value implies that it
takes at most 100 ms for the algorithm to output ‘undistracted’.
AttenD detects single long glances. Due to its accumulation
mechanism, it also penalizes densely spaced glances separated by
short glances to the front. The published algorithm initializes the
output at two seconds and then drops it towards zero when distracted,
while the modified version starts at zero and raises the output towards
two. Figure 4 shows an example of the AttenD output, whose units
are in seconds. The shaded regions indicate periods of task
engagement, and distraction occurs when the AttenD value exceeds a
threshold of two seconds. Note that it is possible to be engaged in a
task and not distracted, as in the first part of the third task on the right
side Figure 4. It is also possible to be distracted while not engaged in
the tasks, as in the short periods on the left side of Figure 4. Ambient
distractions may be caused by the driver looking around to
acclimatize to the simulation, or for other unexplained reasons.
Downloaded from SAE International by John Lee, Wednesday, March 16, 2016
8. Figure 4. The AttenD distraction metric defines the ground truth of distraction
as values over 2.0, as indicated by the bold horizontal line
Framework
Since data are all important in the machine learning paradigm, it is
important to take proper care to use the available data wisely. A
standard procedure is to use a piece of data to train an algorithm, and
reserve another piece to test the performance of the algorithm fairly.
A good training/testing regime prevents common pitfalls of the
machine learning approach like overfitting to the training data. The
method that was used is represented graphically in Figure 5. 75% of
the data was used to train the model using 10-fold cross validation.
Then the remaining 25% was used to run the model and test its
performance on data it had not seen before.
Given that there were many ways to split the training and test sets, a
conservative approach was used that created distinct sets of
participants. Each participant was assigned either to the training set
or the test set. The set allocation was random; however, an allocation
could be rejected if it did not adequately preserve the ratio between
impaired cases and normal cases. If rejected, the allocation was
randomized again. This avoided the situation in which the training
participants could have had many more (or fewer) cases of
distraction, as a percentage of their total driving time, in relation to
the participants in the test set. A less conservative approach could
have involved splitting participant samples between the training and
test set, but then the training phase might have had unfair knowledge
of a participant’s performance who was also used in the test set.
Figure 5. Training and testing procedure using 10-fold cross validation and a
reserved test set.
The distraction-detection algorithm uses a framework that has
developed over the course of the DrIIVE and alcohol-intoxication
projects, and after several exploratory efforts that considered various
alternatives. MacDonald, et al. reported that a Random Forest model
was able to use 54 second windows of the raw steering wheel angle
signal to classify drowsiness six seconds in advance of drowsy lane
departures. Recent DrIIVE work on drowsiness extended this model
by feeding the output of the Random Forest model(s) into a Hidden
Markov Model to take advantage of the time series estimation
capabilities of the HMM. The algorithm framework adopted for the
distraction-detection algorithm, as well as all the impairment
algorithms developed in DrIIVE used the two stage combination of
Random Forest and Hidden Markov Model.
The first stage in the algorithm framework used the Random Forest. A
Random Forest that classifies time windows of input data as impaired
or not impaired. This ensemble method works by training many
individual decision trees, each with a different sample of data, different
subsets of features, and different branching conditions. Each decision
tree classifies the driver state and these predictions decide the forest’s
classification according to a majority vote. The output of the Random
Forest is the state predicted by the greatest number of decision trees.
Figure 6 shows how predictions from many decision trees are
combined through voting to indicate driver state. However, it is also
useful to count up the number of votes for and against distraction and
use that information rather than the resulting classification.
Figure 6. Random Forest Model
The second component of the algorithm was a Hidden Markov Model
(HMM) that took its inputs from the Random Forest model(s) and
performed time series estimation on the value of the driver state. The
HMM assigns probabilities regarding whether the system remains in
a given state or transitions to a new state at each moment in time.
Moreover, it makes use of state history to estimate the current
driver’s state. This history is particularly important in estimating
driver state assuming that a driver who was recently distracted is
likely to still be distracted. Figure 7 shows how the HMM combines a
series of inputs (or observations) to better predict driver state in a
way that takes into account past states.
Figure 7. Hidden Markov Model
The algorithms were implemented using the R statistics software [52].
The caret package was used to train and test the Random Forests [53].
The mhsmm package was used to build the Hidden Markov Models
[54]. The mhsmm package provides two innovations over the traditional
HMM as described above. First, it allows the creation of Hidden
Semi-Markov Models in which the time step is treated as a variable.
This capability was not used. Second, it allows the specification of
observation data as a distribution rather than a simple table of observed
frequencies. The vote count of the Random Forest over all one-segment
segments provided such a distribution. The package has built-in support
for Poisson and gamma distributions, but the lognormal distribution was
found to provide a better fit. The necessary functions were added to
allow integration of the lognormal distribution into the mshmm
functions for training and evaluating HMMs.
Downloaded from SAE International by John Lee, Wednesday, March 16, 2016
9. Algorithm Evaluation
The algorithms evaluated here consist of a single Random Forest that
generates a number of votes for distraction. These votes are passed
into a Hidden Markov Model which estimates the value of the
distraction state and issues a posterior probability of that state being
distracted. This section reports on the measures used to evaluate the
algorithms and the results obtained.
Performance Measures
The measures of algorithm performance are taken from the theory of
receiver operating characteristics (ROC) and the ROC curve, which
in turn is built upon signal detection theory. Algorithm classifications
either line up with ground truth or they do not. There are four
possible options, as represented in the confusion matrix of Table 5.
Table 5. Confusion Matrix
The sensitivity index, also known as d’, measures the ability of an
algorithm to produce true positive estimates. Denoting the true and
false positives, and true and false negatives in Table 5 as TP, FP, TN
and FN respectively, the sensitivity is expressed as
(1)
Its counterpart is specificity which measures the algorithm’s ability to
estimate true negatives and is given by
(2)
Suppose that an algorithm is parameterized using a single parameter
and let that parameter vary from one of its extremes to the other. Its
sensitivity and specificity will naturally vary with the parameter. The
result is a whole family of parameterized algorithms. An ROC curve
is generated by plotting the sensitivity variable versus one minus the
specificity. The resulting curve should have a concave shape for a
performant algorithm. The entire area under the ROC curve (AUC)
may be calculated, and will have a value of one for a perfect
algorithm and 0.5 for an algorithm that performs no better than
chance. An ROC curve is presented in the results in Figure 9. All
measures in this section other than the AUC are obtained by selecting
one value of the parameter, i.e. selecting an operating point on the
ROC curve.
Other signal detection measures are Accuracy, Kappa and Positive
Predictive Value. Accuracy is calculated as
(3)
and Positive Predictive Value (PPV) is calculated as
(4)
Kappa is a statistic that compares an observed accuracy with an
expected accuracy. For example, accuracy may be observed at 75%,
but that is less impressive if the expected accuracy is actually 80%.
Kappa is a more reliable measure than accuracy when the positive
and negative cases are very unbalanced. The expected accuracy
depends on the expected values for TP and TN, given as
(5)
and
(6)
Then the expected accuracy may be written as
(7)
Observe how the expected accuracy mirrors accuracy in Equation (3).
Finally, the kappa statistic is given by
(8)
Results
A Random Forest was trained using several vehicle-based and
environmental-based signals as inputs from the one-second data
segments. Then the output of the Random Forest was used as an input
to an HMM. Raw output signals from the algorithm for six
participants are shown in Figure 8. The vote ratio and posterior
probability range between 0 and 1, while the AttenD metric has units
of seconds and a maximum value of 2. Shaded engagement periods
have height scaled to 2.
The performance summary of just the Random Forest (RF) stage as
well as the total algorithm (HMM) is given in Table 6. Each set of
parentheses in Table 6, and throughout the section, presents the
estimated value of the measure (center value) as well as the 95%
confidence interval (left and right values). The confidence interval
(CI) for the area under the ROC curve (AUC) was estimated using
the Delong method in the pROC package [55] using the R statistics
software. The confidence intervals for all other measures were
estimated using the Wilson score interval [56], [57]. An alternate
version of the distraction detection algorithm was trained and tested
with driver-based signals for head pose added to the input data. The
performance statistics are summarized in Table 7.
Downloaded from SAE International by John Lee, Wednesday, March 16, 2016
10. Figure 8. Raw algorithm output showing the Random Forest vote ratio as well
as the HMM posterior probability overlaid on a plot of the AttenD ground
truth and task engagement. RF vote ratio is displayed as blue dots. HMM
posterior probability is a red dotted line. AttenD output is a solid black line.
Periods of task engagement are shaded regions.
Table 6. Distraction-detection algorithm statistics
The top five most important measures in the vehicle-based RF were
the following: steer_sd, speed_mean, str_des_sd, strd_des_sd and
lanedev_sd. The top five most important measures in the driver and
vehicle-based RF were the following: headhoriz_sd, headhoriz_mean,
headvert_mean, headvert_sd and headconf_mean. Significantly, the
top five most important measures were all related to head pose.
Table 7. Distraction-detection algorithm statistics with head pose
Random Forest ROC curves were parameterized by the number of
votes for distraction, while the ROC curves for the HMMs were
parameterized by the posterior probability threshold for estimating
distraction. The operating point for the Random Forest was obtained
by setting the vote threshold to 50% of the trees, while that of the
HMM was obtained by setting the posterior probability threshold for
distraction to 0.5. It is important to note that the HMM algorithm uses
vote count distribution from the Random Forest (RF) as an input;
therefore, it was not necessary to set an RF operating point. The ROC
curve from the best-performing model is shown in Figure 9. It shows
the results for the combination of vehicle-based and driver-based
sensors summarized in the HMM column of Table 7.
Figure 9. ROC plot of best-performing model, an RF-HMM algorithm using
both vehicle and driver-based input signals. The 95% CI of AUC is shown as
the blue shaded region.
Summary/Conclusions
A distraction algorithm was successfully developed using the AttenD
distraction metric as ground truth. This choice is appropriate for
visual distraction and matches well with the current wisdom on the
risk of glances away from the road, except that it does not consider
the total engagement time that is part of NHTSA’s distraction
guidelines. As total engagement time increases, a gradual loss of
situational awareness would be expected, however this is more
difficult to quantify. Rather, we have chosen a binary classification of
distraction here instead of one with multiple levels of severity. This
choice for ground truth would not work for cognitive distraction or
mind wandering, which can actually cause an increased concentration
of gaze at the forward roadway. A cognitive distraction algorithm
could be developed using the appropriate ground truth metric.
The addition of a Hidden Markov Model to the algorithm resulted in
modest improvements to its performance. A key benefit of the HMM
is its ability to consider the time series evolution of the impairment,
and it is this property that is responsible for the observed performance
improvement. Additionally, the HMM provides a flexibility to the
framework. It can accept inputs from multiple observation sources,
for example several Random Forest models instead of just one. It also
provides a way to create hierarchical models with multiple layers of
HMMs. Related research in the DrIIVE project used two Random
Forests with an HMM for a drowsiness-detection algorithm, and the
HMM provided a more dramatic increase in performance [58].
Downloaded from SAE International by John Lee, Wednesday, March 16, 2016
11. Why did sensitivity, kappa and PPV not perform as well as the other
measures? This relates directly to the frequency of false positive
cases. PPV is interpreted as the probability that a positive
classification from the algorithm corresponds to a true positive case.
It and specificity are both penalized by having large numbers of false
positive cases. Similarly, false positives raise the expected accuracy
which has the indirect effect of lowering kappa. It may be that much
of this effect can be attributed to the strict adherence to distraction
ground truth. In reality many instances of false positives and false
negatives are edge cases that could go either way.
Observe the raw algorithm outputs in Figure 8 above. False positives
are present in the third subplot at around 140 seconds as well as the
sixth subplot at around 125 seconds. The first false positive occurs
during a task engagement that was not labeled as truly distracted.
This could be a failing of the eye tracker data, or of the choice of
ground truth itself, as the driver could actually be distracted. The
second false positive occurs shortly after a task engagement. It is
reasonable that the driver needs some time after task engagement
completes to regain stable vehicle control. However, the relatively
noisy distribution of Random Forest votes for this driver seems to
dispose the algorithm towards more frequent distraction
classifications. This could be an example for which an individualized
algorithm would take into account the driving style and adjust the
detection setting accordingly.
Incorporating head-tracking hardware and software into a production
vehicle may be feasible in the short term, as it requires less resolution
and accuracy than eye tracking. Not surprisingly, the use of head pose
data improved the performance of the distraction-detection algorithm.
One fixed location was used for both tasks so a concern was that the
algorithm learned a specific pattern of head movement. This
distraction-detection algorithm was also applied to alcohol and
drowsiness datasets that had no specific tasks, but had periods of
miscellaneous distraction. It was observed there that the algorithm
with head pose generalized to those datasets better than the one with
only vehicle-based signals [59]. It may be that the vehicle-based
algorithm trained too narrowly to the specific nature of the visual and
visual-manual tasks described in this paper. For example, tasks that
involve looking and reaching to the right may create a bias in lane
deviation to the right. These types of associations were not explored.
Development of a commercial distraction-detection and mitigation
system should consider a wider variety of visually distracting tasks.
Moreover, an understanding of current roadway demand and its
interaction with task complexity would allow designers to adapt the
urgency and timing of a mitigation system to alert the driver sooner
in more demanding situations. Alternatively, the presence of
distraction and a high-demand environment could be used to alter the
behavior of other advanced driving assistance systems such as
forward collision warning (FCW) system to adjust its timing.
The increasing use of automation in vehicles provides opportunities
and challenges for driver-state monitoring systems. Such systems will
have to make use of driver-based sensor signals while automation is
in control of vehicle handling and speed. On the other hand, vehicle-
based or driver-based systems provide information during manual
control that could recommend the automation should take over
control to maintain safety. When the automation needs to shift control
back to the driver, assessing driver state will be even more important
than it is with conventional vehicles to ensure that the driver is still
engaged in monitoring the road situation. The design of these types of
automation transfers is an active area of research.
The research conducted in the DrIIVE program as well as other
impairment studies at the NADS represents an important step in
establishing a database of impaired driving data based on a common
set of scenarios. These datasets should ultimately facilitate additional
efforts to understand the effects of driver impairment and aid in the
development of methods to assess driver state and enhance safety.
Other driving-safety researchers would benefit from the availability
of carefully controlled simulator data that complements broad
naturalistic datasets lacking experimentally controlled conditions.
References
1. NHTSA, “Distracted Driving 2013,” NHTSA, Washington,
D.C., Research Note DOT HS 812 132, Apr. 2015.
2. Bergasa L. M., Nuevo J., Sotelo M. A., Barea R., and Lopez M.
E., “Real-time system for monitoring driver vigilance,” IEEE
Trans. Intell. Transp. Syst., vol. 7, no. 1, pp. 63-77, Mar. 2006.
3. Jiménez P., Bergasa L. M., Nuevo J., Hernández N., and Daza
I. G., “Gaze Fixation System for the Evaluation of Driver
Distractions Induced by IVIS,” IEEE Trans. Intell. Transp. Syst.,
vol. 13, no. 3, pp. 1167-1178, Sep. 2012.
4. Busso C. and Jain J., “Advances in Multimodal Tracking of
Driver Distraction,” in Digital Signal Processing for In-Vehicle
Systems and Safety, New York, NY: Springer New York, 2012,
pp. 253-270.
5. Lee J. D., Roberts S. C., Hoffman J. D., and Angell L. S.,
“Scrolling and Driving How an MP3 Player and Its Aftermarket
Controller Affect Driving Performance and Visual Behavior,”
Hum. Factors J. Hum. Factors Ergon. Soc., vol. 54, no. 2, pp.
250-263, Apr. 2012.
6. Liang Y., Reyes M. L., and Lee J. D., “Real-Time Detection of
Driver Cognitive Distraction Using Support Vector Machines,”
IEEE Trans. Intell. Transp. Syst., vol. 8, no. 2, pp. 340 -350,
Jun. 2007.
7. Dong Y., Hu Z., Uchimura K., and Murayama N., “Driver
Inattention Monitoring System for Intelligent Vehicles: A
Review,” IEEE Trans. Intell. Transp. Syst., vol. 12, no. 2, pp.
596-614, Jun. 2011.
8. Cai H. and Lin Y., “Modeling of operators’ emotion and task
performance in a virtual driving environment,” Int. J. Hum.-
Comput. Stud., vol. 69, no. 9, pp. 571-586, Aug. 2011.
9. Mehler B., Reimer B., and Coughlin J. F., “Sensitivity of
Physiological Measures for Detecting Systematic Variations in
Cognitive Demand From a Working Memory Task An On-
Road Study Across Three Age Groups,” Hum. Factors J. Hum.
Factors Ergon. Soc., vol. 54, no. 3, pp. 396-412, Jun. 2012.
10. D’Orazio T., Leo M., Guaragnella C., and Distante A., “A visual
approach for driver inattention detection,” Pattern Recognit.,
vol. 40, no. 8, pp. 2341-2355, Aug. 2007.
11. Huang Q., Fan Y., and Lei T., “A novel approach of eye
movement and Expression,” in 2010 2nd International
Conference on Future Computer and Communication (ICFCC),
2010, vol. 3, pp. V3-436-V3-440.
Downloaded from SAE International by John Lee, Wednesday, March 16, 2016
12. 12. Jo J., Lee S. J., Jung H. G., Park K. R., and Kim J., “Vision-
based method for detecting driver drowsiness and distraction
in driver monitoring system,” Opt. Eng., vol. 50, no. 12, pp.
127202-127202-24, 2011.
13. Ahlstrom C., Victor T., Wege C., and Steinmetz E., “Processing
of Eye/Head-Tracking Data in Large-Scale Naturalistic Driving
Data Sets,” IEEE Trans. Intell. Transp. Syst., vol. 13, no. 2, pp.
553-564, Jun. 2012.
14. Fu X., Guan X., Peli E., Liu H., and Luo G., “Automatic
Calibration Method for Driver’s Head Orientation in Natural
Driving Environment,” IEEE Trans. Intell. Transp. Syst., vol.
14, no. 1, pp. 303-312, Mar. 2013.
15. Ostlund J., Peters B., Thorslund B., Engstrom J., Markkula G.,
Keinath A., Horst D., Juch S., Mattes S., and Foehl U., “Driving
Performance Assessment - Methods and Metrics,” Information
Society Technologies, Final Report IST-1-507674-IP, Mar. 2004.
16. Greenberg J., Tijerina L., Curry R., Artz B., Cathey L., Kochhar
D., Kozak K., Blommer M., and Grant P., “Driver Distraction:
Evaluation with Event Detection Paradigm,” Transp. Res. Rec.
J. Transp. Res. Board, vol. 1843, pp. 1-9, Jan. 2003.
17. Kaber D. B., Liang Y., Zhang Y., Rogers M. L., and
Gangakhedkar S., “Driver performance effects of simultaneous
visual and cognitive distraction and adaptation behavior,”
Transp. Res. Part F Traffic Psychol. Behav., vol. 15, no. 5, pp.
491-501, Sep. 2012.
18. Liang Y. and Lee J. D., “A hybrid Bayesian Network approach
to detect driver cognitive distraction,” Transp. Res. Part C
Emerg. Technol., vol. 38, pp. 146-155, Jan. 2014.
19. Regan M. A., Lee J. D., and Young, Kristie, Driver Distraction:
Theory, Effects, and Mitigation. CRC Press, 2008.
20. Sathyanarayana A., Boyraz P., and Hansen J. H. L.,
“Information fusion for robust ‘context and driver aware’ active
vehicle safety systems,” Inf. Fusion, vol. 12, no. 4, pp. 293-303,
Oct. 2011.
21. Aoude G. S., Desaraju V. R., Stephens L. H., and How J. P.,
“Driver Behavior Classification at Intersections and Validation
on Large Naturalistic Data Set,” IEEE Trans. Intell. Transp.
Syst., vol. 13, no. 2, pp. 724-736, Jun. 2012.
22. Ersal T., Fuller H. J. A., Tsimhoni O., Stein J. L., and Fathy
H. K., “Model-Based Analysis and Classification of Driver
Distraction Under Secondary Tasks,” IEEE Trans. Intell.
Transp. Syst., vol. 11, no. 3, pp. 692-701, Sep. 2010.
23. NHTSA, “Visual-Manual NHTSA Driver Distraction Guidelines
for In-Vehicle Electronic Devices,” NHTSA, Washington, D.C.,
NHTSA Guidelines NHTSA-2010-0053, Apr. 2013.
24. Dingus T. A., Klauer S. G., Neale V. L., Petersen A., Lee S. E.,
Sudweeks J. D., Perez M. A., Hankey J., Ramsey D. J., Gupta
S., Bucher C., Doerzaph Z. R., Jermeland J., and Knipling R. R.,
“The 100-Car Naturalistic Driving Study, Phase II -Results of
the 100-Car Field Experiment,” Apr. 2006.
25. Klauer S. G., Dingus T. A., Neale V. L., Sudweeks J. D., and
Ramsey D. J., “The Impact of Driver Inattention on Near-Crash/
Crash Risk: An Analysis Using the 100-Car Naturalistic Driving
Study Data,” USDOT, Washington, DC, Final Report DOT HS
810 594, Apr. 2006.
26. Victor T., Dozza M., Bärgman J., Boda C.-N., Engström
J., Flannagan C., Lee J. D., and Markkula G., “Analysis
of Naturalistic Driving Study Data: Safer Glances, Driver
Inattention, and Crash Risk,” Final Report S2-S08A-RW-1,
2015.
27. Yeo M. V. M., Li X., Shen K., and Wilder-Smith E. P. V., “Can
SVM be used for automatic EEG detection of drowsiness during
car driving?,” Saf. Sci., vol. 47, no. 1, pp. 115-124, Jan. 2009.
28. Wang J., Xu W., and Gong Y., “Real-time driving danger-level
prediction,” Eng. Appl. Artif. Intell., vol. 23, no. 8, pp. 1247-
1254, Dec. 2010.
29. Wollmer M., Blaschke C., Schindl T., Schuller B., Farber B.,
Mayer S., and Trefflich B., “Online Driver Distraction Detection
Using Long Short-Term Memory,” IEEE Trans. Intell. Transp.
Syst., vol. 12, no. 2, pp. 574-582, Jun. 2011.
30. Yang X.-S. and Deb S., “Engineering optimisation by cuckoo
search,” Int. J. Math. Model. Numer. Optim., vol. 1, no. 4, pp.
330-343, Jan. 2010.
31. Veeraraghavan H., Bird N., Atev S., and Papanikolopoulos N.,
“Classifiers for driver activity monitoring,” Transp. Res. Part C
Emerg. Technol., vol. 15, no. 1, pp. 51-67, Feb. 2007.
32. Breiman L., “Random Forests,” Mach. Learn., vol. 45, no. 1,
pp. 5-32, Oct. 2001.
33. Lee B.-G. and Chung W.-Y., “Driver Alertness Monitoring
Using Fusion of Facial Features and Bio-Signals,” IEEE Sens.
J., vol. 12, no. 7, pp. 2416-2422, Jul. 2012.
34. Lansdown T. C., Brook-Carter N., and Kersloot T., “Distraction
from multiple in-vehicle secondary tasks: vehicle performance
and mental workload implications,” Ergonomics, vol. 47, no. 1,
pp. 91-104, Jan. 2004.
35. Chisholm S. L., Caird J. K., Lockhart J., Fern L., and Teteris
E., “Driving Performance while Engaged in MP-3 Player
Interaction: Effects of Practice and Task Difficulty on PRT and
Eye Movements,” presented at the Driving Assessment 2007:
4th International Driving Symposium on Human Factors in
Driver Assessment, Training, and Vehicle Design, 2007.
36. Blanco M., Biever W. J., Gallagher J. P., and Dingus T. A.,
“The impact of secondary task cognitive processing demand
on driving performance,” Accid. Anal. Prev., vol. 38, no. 5, pp.
895-906, Sep. 2006.
37. Jamson A. Hamish and Merat N., “Surrogate in-vehicle
information systems and driver behaviour: Effects of visual and
cognitive load in simulated rural driving,” Transp. Res. Part F
Traffic Psychol. Behav., vol. 8, no. 2, pp. 79-96, Mar. 2005.
38. Lee J. D., Fiorentino D., Reyes M. L., Brown T. L., Ahmad O.,
Fell J., Ward N., and Dufour R., “Assessing the Feasibility of
Vehicle-Based Sensors to Detect Alcohol Impairment,” Aug. 2010.
39. Lee J. D., Moeckli J., Brown T., and Roberts S., “Detection
of Driver Distraction using Vision Based Algorithms,”
in Proceedings of the 23rd Enhanced Safety of Vehicles
Conference, 2013.
40. Lee J. D., Moeckli J., Brown T. L., Roberts S. C., Schwarz C.,
Yekhshatyan L., Nadler E., Liang Y., Victor T., Marshall D., and
Davis C., “Distraction Detection and Mitigation Through Driver
Feedback,” Final Report DOT HS 811 547A, May 2013.
Downloaded from SAE International by John Lee, Wednesday, March 16, 2016
13. 41. Lee J. D., Moeckli J., Brown T. L., Roberts S. C., Schwarz C.,
Yekhshatyan L., Nadler E., Liang Y., Victor T., Marshall D., and
Davis C., “Distraction Detection and Mitigation Through Driver
Feedback: Appendices,” Final Report DOT HS 811 547B, May
2013.
42. Brown T., Lee J., Schwarz C., Fiorentino D., and McDonald A.,
“Assessing the Feasibility of Vehicle-based Sensors to Detect
Drowsy Driving,” NHTSA, Washington, DC, Final Report DOT
HS 811 886, Feb. 2014.
43. Roberts S. C., Ghazizadeh M., and Lee J. D., “Warn me now or
inform me later: Drivers’ acceptance of real-time and post-drive
distraction mitigation systems,” Int. J. Hum.-Comput. Stud., vol.
70, no. 12, pp. 967-979, Dec. 2012.
44. McDonald A. D., Lee J. D., Schwarz C., and Brown T. L.,
“Steering in a Random Forest Ensemble Learning for Detecting
Drowsiness-Related Lane Departures,” Hum. Factors J. Hum.
Factors Ergon. Soc., vol. 56, no. 5, pp. 986-998, Aug. 2014.
45. Schwarz C., Brown T. L., Gaspar J., Marshall D., Lee J.,
Kitazaki S., and Kang J., “Mitigating Drowsiness: Linking
Detection to Mitigation,” in Proceedings of the 24th ESV
Conference, Gothenburg, Sweden, 2015.
46. McDonald A. D., Schwarz C. W., Lee J. D., and Brown T.
L., “Impairment as a hidden state: How Hidden Markov
Models improve drowsiness detection and may differentiate
between distraction, drowsiness, and alcohol impairment,” in
Proceedings of the TRB 93rd Annual Meeting, Washington, DC,
2014.
47. Driver Focus-Telematics Working Group, “Statement of
Principles, Criteria and Verification Procedures on Driver
Interactions with Advanced In-Vehicle Information and
Communication Systems.” Auto Alliance, Jun-2006.
48. Eskandarian A., Sayed R., Delaigue P., Blum J., and Mortazavi
A., “Advanced Driver Fatigue Research,” FMCSA, Washington,
DC, Final Report FMCSA-RRR-07-001, Apr. 2007.
49. LaViola J. J., “Double exponential smoothing: an alternative to
Kalman filter-based predictive tracking,” in Proceedings of the
workshop on Virtual environments 2003, New York, NY, USA,
2003, pp. 199-206.
50. Dingus T. A., Hulse M. C., Antin J. F., and Wierwille W. W.,
“Attentional demand requirements of an automobile moving-
map navigation system,” Transp. Res. Part Gen., vol. 23, no. 4,
pp. 301-315, Jul. 1989.
51. Kircher K. and Ahlstrom C., “Issues related to the driver
distraction detection algorithm AttenD,” Swedish National Road
and Transport Resear ch Institute (VTI), Final Report, 2009.
52. R Development Core Team, {R: A language and environment
for statistical computing}. Vienna, Austria: R Foundation for
Statistical Computing, 2009.
53. Kuhn M., “Building Predictive Models in R Using the caret
Package,” J. Stat. Softw., vol. 28, no. 5, Nov. 2008.
54. O’Connell J. and Højsgaard S., “Hidden Semi Markov Models
for Multiple Observation Sequences: The mhsmm Package for
R,” J. Stat. Softw., vol. 39, no. i04.
55. Robin X., Turck N., Hainard A., Tiberti N., Lisacek F., Sanchez
J.-C., and Müller M., “pROC: an open-source package for R and
S+ to analyze and compare ROC curves,” BMC Bioinformatics,
vol. 12, no. 1, p. 77, Mar. 2011.
56. Wallis S., “Binomial Confidence Intervals and Contingency
Tests: Mathematical Fundamentals and the Evaluation of
Alternative Methods,” J. Quant. Linguist., vol. 20, no. 3, pp.
178-208, 2013.
57. Wilson E., “Probable Inference, the Law of Succession, and
Statistical Inference,” J. Am. Stat. Assoc., vol. 22, no. 158, pp.
209-212, 1927.
58. Brown T., Gaspar J., Schwarz C., Schmitt R., and Marshall D.,
“DrIIVE Track B: Assess Potential Countermeasures for Drowsy
Driving Lane Departures,” National Advanced Driving Simulator,
Iowa City, IA, Technical Report N2015-007, Sep. 2015.
59. Brown T., Schwarz C., Lee J., Gaspar J., Marshall D., and
Ahmad O., “DrIIVE Track A: Develop and Evaluate a System of
Algorithms to Identify Signatures of Alcohol-Impaired, Drowsy
and Distracted Driving,” National Advanced Driving Simulator,
Iowa City, IA, Technical Report N2015-009, Sep. 2015.
Contact Information
The corresponding author may be contacted using the following
information:
Chris Schwarz
National Advanced Driving Simulator
2401 Oakdale Blvd
Iowa City, IA 52242
chris-schwarz@uiowa.edu
319-335-4642
Acknowledgments
This research reported here is part of a program of research sponsord
by the National Highway Traffic Safety Administration (NHTSA)
under the leadership of Julie Kang. The authors would like to
acknowledge the help of Eric Nadler of Volpe and the research staff
at the NADS for their diligent efforts.
Definitions/Abbreviations
AttenD - Gaze-based distraction detection algorithm used for ground
truth in this project.
AUC - Area Under an ROC Curve.
CARET - Classification And REgression Training. An R package for
creating predictive models.
DrIIVE - Driver Monitoring of Inattention and Impairment Using
Vehicle Equipment
DES - Double Exponential Smoothing
DVI - Driver Vehicle Interface
EEG - Electroencephalography
FCW - Forward Collision Warning
HASTE - Human machine interface And the Safety of Traffic in
Europe
HMM - Hidden Markov Model
GIS - Geographic Information System
IG - Image Generator
LED - Light Emitting Diode
Downloaded from SAE International by John Lee, Wednesday, March 16, 2016
14. MHSMM - R package for inference of Hidden Markov and Semi-
Markov models
MSDLP - Modified Standard Deviation of Lane Position
NADS - National Advanced Driving Simulator
NHTSA - National Highway Transportation Safety Administration
P2P - Peak to Peak
PPV - Positive Predictive Value
R - R open source statistical software
RF - Random Forest
ROC - Receiver Operator Characteristic
SD - Standard Deviation
SHRP2 - Strategic Highway Research Program 2
SMS - Short Message Service
SVM - Support Vector Machine
The Engineering Meetings Board has approved this paper for publication. It has successfully completed SAE’s peer review process under the supervision of the session organizer. The process
requires a minimum of three (3) reviews by industry experts.
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or
otherwise, without the prior written permission of SAE International.
Positions and opinions advanced in this paper are those of the author(s) and not necessarily those of SAE International. The author is solely responsible for the content of the paper.
ISSN 0148-7191
http://papers.sae.org/2016-01-0114
Downloaded from SAE International by John Lee, Wednesday, March 16, 2016
![Abstract
Distracted driving remains a serious risk to motorists in the US and
worldwide. Over 3,000 people were killed in 2013 in the US because
of distracted driving; and over 420,000 people were injured. A system
that can accurately detect distracted driving would potentially be able
to alert drivers, bringing their attention back to the primary driving
task and potentially saving lives. This paper documents an effort to
develop an algorithm that can detect visual distraction using
vehicle-based sensor signals such as steering wheel inputs and lane
position. Additionally, the vehicle-based algorithm is compared with
a version that includes driving-based signals in the form of head
tracking data. The algorithms were developed using machine learning
techniques and combine a Random Forest model for instantaneous
detection with a Hidden Markov model for time series predictions.
The AttenD distraction algorithm, based on eye gaze location, was
utilized to generate the ground truth for the algorithm development.
The data collection at the National Advanced Driving Simulator is
summarized, results are presented, and the paper concludes with
discussion on the algorithms. This work falls within a program of
research on Driver Monitoring of Inattention and Impairment Using
Vehicle Equipment (DrIIVE) and is sponsored by NHTSA.
Introduction
Driving impairment poses a serious risk to motorists in the US and
worldwide, and distraction is a significant type of impairment. Over
3,000 people were killed in 2013 in the US because of distracted
driving; and over 420,000 people were injured [1]. Lives could
potentially be saved with an advanced safety system that accurately
detects distracted driving and provides countermeasures. This paper
documents an effort to develop an algorithm that can detect visual
distraction using vehicle-based sensor signals such as steering wheel
inputs and lane position as well as driver-based head pose signals.
The remainder of the introduction reviews the literature on
distraction, algorithms, secondary task difficulty, and previous NADS
impairment research. The methodology section presents the details of
a distraction study that was conducted on the NADS-1 simulator for
the development of the algorithm. Methodology is followed by a
section on the algorithm design and another on algorithm evaluation.
The algorithm evaluation section presents the results of testing and
comparing the algorithms. A summary and conclusions from the
project end the paper.
Sensors and Feature Generation
It is reasonable to argue that driver-based sensor signals, including
physiological signals like gaze or EEG, offer a more direct measure
of impairment generally and of distraction specifically. Substantial
research has been done to define input variables for impairment
detection algorithms based on eye movements, head position, and
even facial expressions. Many algorithms have focused solely on eye
tracking [2], and facial image analysis [3], whereas others combine
multimodal features or include vehicle-based measures [4]-[6]. In a
review of detection systems, Dong et al. [7] suggested that hybrid
measures involving multiple modes perform best. Algorithms based
on features derived from the driver’s face and eyes show great
promise, but the cost, reliability and intrusiveness of camera-based
systems undermines their feasibility and may delay their adoption.
The Detection of Visual Distraction using Vehicle and
Driver-Based Sensors
2016-01-0114
Published 04/05/2016
Chris Schwarz and Timothy Brown
National Advanced Driving Simulator
John Lee
University of Wisconsin
John Gaspar
National Advanced Driving Simulator
Julie Kang
US Dept. of Transportation
CITATION: Schwarz, C., Brown, T., Lee, J., Gaspar, J. et al., "The Detection of Visual Distraction using Vehicle and Driver-Based
Sensors," SAE Technical Paper 2016-01-0114, 2016, doi:10.4271/2016-01-0114.
Copyright © 2016 SAE International
Downloaded from SAE International by John Lee, Wednesday, March 16, 2016](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)