"Pedestrian Density-Based Path Recognition and Risk Prediction for Autonomous Vehicles" that has been published in the "IEEE International Workshop on Robot and Human Communication (ROMAN)".

1. Pedestrian Density Based Path Recognition and Risk Prediction for
Autonomous Vehicles
Kasra Mokhtari1, Ali Ayub2, Vidullan Surendran 3 and Alan R. Wagner4
Abstract— Human drivers continually use social information
to inform their decision making. We believe that incorporating
this information into autonomous vehicle decision making
would improve performance and importantly safety. This paper
investigates how information in the form of pedestrian density
can be used to identify the path being travelled and predict the
number of pedestrians that the vehicle will encounter along that
path in the future. We present experiments which use camera
data captured while driving to evaluate our methods for path
recognition and pedestrian density prediction. Our results show
that we can identify the vehicle’s path using only pedestrian
density at 92.4% accuracy and we can predict the number
of pedestrians the vehicle will encounter with an accuracy of
70.45%. These results demonstrate that pedestrian density can
serve as a source of information both perhaps to augment
localization and for path risk prediction.
I. INTRODUCTION
Driverless vehicles may soon operate autonomously in and
about densely populated cities. These vehicles are currently
designed for point-to-point navigation using GPS with a
variety of obstacle avoidance mechanisms ensuring that the
vehicle will safely arrive at its destination. Although this
type of system design does provide the underpinnings nec-
essary for safe autonomous driving, it unfortunately fails to
utilize available background and experiential information to
enhance safety. For example, rather than relying on obstacle
avoidance to prevent accidents, the vehicle might preemp-
tively avoid roadways with high pedestrian density, high
risk pedestrian behavior, or difficult to recognize pedestrians.
We hypothesize that, ultimately, recognition and avoidance
of risky paths could serve as a first method for enhancing
autonomous vehicle safety.
Pedestrians are the most vulnerable individuals in an
autonomous vehicle’s environment. An autonomous vehicle
should therefore perform in a manner that ensures pedestrian
safety. In order to improve pedestrian safety and decrease
fatalities due to accidents, vehicles could estimate the risk
of collision with a pedestrian along a particular path or at
particular intersections. Although a number of factors play
a role towards generating an accurate estimate of this risk,
certainly the number of pedestrians on a road constitutes an
excellent measure of pedestrian exposure to accidents [1].
1Department of Mechanical Engineering, The Pennsylvania State Uni-
versity, State College, PA 16802, USA kbm5402@psu.edu
2Department of Electrical Engineering, The Pennsylvania State Univer-
sity, State College, PA 16802, USA aja5755@psu.edu
3Department of Aerospace Engineering, The Pennsylvania State Univer-
sity, State College, PA 16802, USA vus133@psu.edu
4Department of Aerospace Engineering, The Pennsylvania State Univer-
sity, State College, PA 16802, USA alan.r.wagner@psu.edu
This paper therefore investigates methods for collecting,
using, and predicting pedestrian density along a path. Specif-
ically, we 1) use patterns of pedestrian density to recognize
specific pathways and 2) generate estimates of pedestrian
density at different times of the day, days of the week, and
along different paths. We hypothesize that pedestrian density
can be used to identify different driving routes. This is not
to suggest that pedestrian density provides a better method
for localization than a global positioning system such as
GPS. Rather our intent is to show that pedestrian density
can be used to identify distinct locations and pathways that
are characterized by high pedestrian density. As such, it
may provide a separate and distinct source of localization
information that allows the vehicle to plan around crowded
areas. Further, we believe that using historical data, one can
estimate the number of pedestrians that would be encoun-
tered on a path and thereby estimate the risk associated
with travelling along it. For example, an autonomous vehicle
tasked with navigating through a college town with high
pedestrian foot traffic might choose to avoid routes near
the football stadium during games or downtown areas on
weekend nights in order to avoid jaywalking pedestrians and
inebriated patrons.
Capturing and incorporating pedestrian density into a
robot’s decision-making process is difficult for a variety of
reasons. First, although classifiers for pedestrian detection
exist (e.g. [2], [3]), methods for characterizing pedestrians
in terms of age, mobility, or behavior have poor accuracy in
realistic scenarios. To the best of our knowledge there are
no behavioral models for pedestrians in ecologically valid
environments. Moreover, pedestrian density is dynamic and
thus difficult to predict. Even though pedestrian density is
highly dependent on external conditions such as social events
and weather, with enough data predictable patterns emerge
that might nevertheless contribute to safer operation of an
autonomous vehicle.
The remainder of this paper is organized as follows:
Section II presents related work, Section III introduces our
pedestrian pattern datasets and then describes two different
classification techniques, Section IV and Section V empiri-
cally demonstrates how pedestrian density might inform an
autonomous vehicle, and finally Section VI offers conclu-
sions and directions for future work.
II. RELATED WORK
Autonomous robot research has long investigated methods
for incorporating social information into robot decision
making. For many applications, these robots need social

2. Fig. 1: Three different paths between the same location in State
College, PA. ”Path A” (left), ”Path B” (middle) and ”Path C” (right).
The car traversed each path for 42 times including 7 days and six
timeslots a day: 8:45, 10:45, 12:45, 14:45, 16:45 and 17:45. Best
viewed in color.
information to navigate around people and through crowds
[4]. Museum tour guide robots, for example, offer the
capability to detect and navigate around people who block
the robot’s path [5], [6]. The Pedestrian Dominance Model
(PDM) [7] is capable of navigating through groups of pedes-
trians by identifying the dominance characteristics of these
groups based on their trajectories. The Human-Aware Motion
Planner (HAMP) [8] considers the safety of the robot’s
movement along with human comfort, attempting to keep
the robot in front of people and visible to them at all times.
The Constraint-Optimizing Method for Person–Acceptable
NavigaTION (COMPANION) [9] framework models human
social conventions, including avoiding people’s personal
space, as well as task-based constraints such as minimizing
distance.
Autonomous vehicles may also use social information in
a variety of different ways. The vehicle could potentially
capture and use information about passengers, the human
drivers of other vehicles or about pedestrians. A number of
patents have focused on driver [10] and passenger monitoring
[11]. Social information has also been used to improve
cooperation within human drivers. For example, a novel
social behavior framework which implements an intention-
integrated Prediction and Cost function Based algorithm
(iPCB) [12] for lane change and entrance ramp scenarios
has been developed. In order to avoid accidents, autonomous
vehicles must be able to recognize some human driver
behaviors such as anticipating other human drivers’ course of
action at stop-sign-controlled intersections [13]. Autonomous
vehicle politeness can encourage human driver cooperation.
To this end, Lee et al., tested the impact of autonomous
vehicle politeness in a vehicle that was operating normally
and, as a second condition, in a vehicle that failed to
detect road signs [14]. The results revealed that the vehicle
could improve human evaluation of the experience by being
polite when operating normally. Sadigh et al. [15] developed
an optimization-based method for behavior planning which
consider the effects that it will have on human drivers.
Studies related to the social aspects of autonomous driving
and pedestrians mainly focus on attention, communication,
and perceived risk [16]. There is an abundance of research
related to camera-based pedestrian detection [17], [18], [19],
Fig. 2: Grid-based representation of ”Path A”, ”Path B” and
”Path C”. For more clear step visualization, the number of
steps for each path is downsized by 10. Best viewed in color.
[3], estimating the time taken to cross a street [20], pedes-
trian tracking [21], [22], [23], pedestrian activity recognition
and classification [24], [25], [26], pedestrian-to-vehicle [27],
[28] and vehicle-to-pedestrian communications [29], [30].
Autonomous vehicles must perform in a manner which not
only ensures pedestrian safety but also comfort. Pedestrian
intention and behavior modeling is studied in [31], [32].
Haoyu et al. [33] develop a Partially Observable Markov
Decision Process (POMDP) that attempts to capture the
underlying uncertainty in pedestrian intention as well as
uncertainties in vehicle control and sensing. This algorithm
is implemented and tested on a golf cart demonstrating
that it can perform in near real time in a complex and
dynamic environment. Rasouli et al. [34] collected a large
dataset of pedestrian trajectories at crosswalks analyzing how
pedestrians and drivers communicate and the factors that
effect their interactions. A game theory model implemented
in [35] analyzes the interaction between pedestrians and
autonomous vehicles where the pedestrians and autonomous
vehicles are to yield at uncontrolled crosswalks. This game
theoretic model suggests that autonomous vehicles will be at
a strategic disadvantage when interacting with human driven
cars.
Intuitively, the risk associated with travel down a particular
path is related to the total number of pedestrians that the
vehicle will encounter. A variety of measures have been
considered as predictors such as pedestrian volume and
exposure at intersections, land use data, pavement conditions,
and sidewalk width [36]. In contrast, the work presented here
uses time series data to estimate pedestrian density for each
route under consideration.
III. METHODOLOGY
This research adopts the pedestrian density along a path
as a feature for a path classification. The subsections below
first describe the datasets used, classification algorithms,
and finally experiments demonstrating the use of pedestrian
density for path recognition and path risk estimation. To

3. the best of our knowledge, no method currently allows an
autonomous vehicle to use pedestrian density to distinguish
between paths and the prediction of future density of pedes-
trians along a path.
A. Datasets
This research required real world data over a series of
repeatedly driven pathways. We therefore created the Pedes-
trian Pattern Dataset [37] from camera data collected while
driving along three different paths, and the Synthetic Pedes-
trian Dataset which is an augmentation of the Pedestrian
Pattern Dataset that increases the numbers of pedestrians
encountered.
1) Pedestrian Pattern Dataset: This dataset was created
by collecting video camera data while repeatedly driving in
State College, PA along the three different paths shown in
Fig. 1. Data collection was carried out over the course of
three weeks at the following times 8:45, 10:45, 12:45, 14:45,
16:45 and 17:45 resulting in 126 distinct video samples.
Each sample contains a full HD video and GPS data for the
entire traversal. A Fast R-CNN based pedestrian detection
algorithm [3] was applied to the captured videos to generate
an estimate of the number of pedestrians per frame in order
to assess pedestrian density along the paths. Fast R-CNN
was selected over other architectures in order to maximize
pedestrian detection accuracy. Since GPS data was updated
every second but the videos were shot at 60 frames-per-
second, the number of pedestrians at each second is the
mean number of pedestrians detected over the 60 frames.
As a result, a vector vi ∈ Rni×3
is generated for each
video, where ni is the length of the i-th video in seconds (s),
and the 3 columns represent the average number of detected
pedestrians, latitude and longitude values of the car at that
particular time. Thus, the dataset is a collection of vectors
DP = {vi : i = 1, 2...126}.
The vectors in the dataset DP vary in length because the
time taken to traverse the paths varied based on external
factors such as weather and traffic conditions. In order to
effectively analyze the data, we first mapped each vector to
a set of vectors of fixed length T such that ˆDP = {ˆvi : ˆv ∈
RT ×3
, i = 1, 2...126}. The value T = 216 was chosen based
on the length of the shortest vector vi ∈ Dp as this eliminated
the need to extrapolate data. Based on the known latitude and
longitude values of the vehicle and the paths travelled, the
domain was discretized to a resolution of (30×30ft2
) where
each path is made up of approximately 216 unit squares as
presented in Fig. 2. The number of detected pedestrians at
each step in a data sample is then computed as the average
of all data points that are contained in that square.
When this normalized dataset ˆDP is clustered using k-
means clustering into 3 groups we found that the number
of detected pedestrians is highly correlated with the path.
Looking at the data as a histogram (Fig. 3 left), it appears
that the data is sparse with 82% of steps having zero or
one detected pedestrian. Moreover, the maximum number of
detected pedestrian at any given step is 6.
2) Synthetic Pedestrian Dataset: The sparsity of pedestri-
ans in the real world dataset prompted us to create a Synthetic
Pedestrian Dataset to enable us to study conditions with
more pedestrians. Once again we generated 126 vectors of
length T = 216 denoted by ˆvi with 42 vectors per path.
To create these vectors a random uniform distribution of
the number of pedestrians was generated. Some steps are
arbitrarily enforced to have zero detected pedestrians (Fig. 3
right). The distribution of data is non-sparse as seen in Figure
3 (right). The maximum number of detected pedestrian at a
single step is 21 vs 6 for the Pedestrian Pattern Dataset.
B. Classification Algorithms
This section introduces two classification algorithms with
some adaptations, Centroid-Based Concept Learning (CBCL)
and LSTMs, that were used to recognize the autonomous
vehicle path from the number of pedestrians detected.
1) Centroid-Based Concept Learning (CBCL): This
method uses clusters of deep CNN features for scene and
object classification [38], [39]. CBCL is applied to the T ×1
dimensional vectors of each path separately. The result of
this process is a collection of centroids for each path, C =
C1, C2, ..., Cm. Each path can have different patterns of the
number of pedestrians at different steps on different days.
Intuitively, each of these different pedestrian patterns in a
path should be represented by separate centroids. By using an
optimal distance threshold D (found using cross-validation),
a set of centroids can be found for each path such that each
centroid represents a different pedestrian pattern in the path.
To classify an unlabeled pedestrian data vector, the Eu-
clidean distance between the vector and all the centroids in
Ci for path i is calculated. The r closest centroids to the
unlabeled vector are selected. The contribution of each of
the r closest centroids to the determination of a path i is a
conditional summation that is defined as:
Pred(i) =
r
j=1
1
distj
[yj = i] (1)
where Pred(i) is the prediction weight of a path i, yj is
the path label of the jth closest centroid and distj is the
Euclidean distance between the jth closest centroid and the
unlabeled pedestrian data vector. The prediction weights for
all the categories are initialized to zero. Then, for the r
closest centroids the prediction weights are updated for the
categories that each of the r centroid pairs belong to. The
prediction weight for each path is further multiplied by the
inverse of the total number of data points in the training set of
the path to manage category imbalance. The test data point
is classified based on the path with the highest prediction
weight.
For a real world application, a vehicle can start from any
particular point along a path and may not have data over the
entire path. For this scenario, we adapt the CBCL algorithm
to take any t×1 dimensional test vector, where t×1 is a small
portion of the complete vector and the starting point of the
smaller t×1 vector does not necessarily have to be the same

4. Fig. 3: Histogram plot of number of pedestrians vs number of steps for the Pedestrian Pattern Dataset (left) and Artificial
Pedestrian Dataset (right). Numbers on top of each bar are the number of steps for the corresponding number of pedestrian(s).
as the complete T ×1 vector. For this smaller input, where the
starting point is unknown, the t×1 unlabeled vector is padded
with zeros to create a T ×1 vector. Since the starting point is
unknown, a total of T −t+1 vectors are generated to cover all
the starting point scenarios. For each centroid, the Euclidean
distance of all these padded vectors is calculated from the
centroid and the minimum of these distances is used. Before
each distance calculation, the respective centroid is masked
at the locations of zeros in the padded vector. This process
is repeated for all the centroids, and using the minimum
distances for each of the centroids, the r closest centroids
are identified. After getting the r closest centroids, the above
mentioned prediction approach is applied to find the path of
the unlabeled t × 1 vector.
2) LSTM-Based Classification: Recurrent neural net-
works are a type of artificial neural networks particularly
suited to temporal or sequential data as their internal state
allows information to persist in contrast to a traditional
feedforward network. A Long Short Term Memory [40] RNN
overcomes limitations such as the vanishing and explod-
ing gradient problems that plague traditional RNNs. Both
datasets considered in this work consist of 126 samples
belonging to one of the 3 categories (paths), each containing
a sequence of length T = 216 steps. Each of the 216
integers making up the sequence represents the number of
pedestrians detected at that particular step. Given a sequential
sub-sequence of arbitrary length extracted from a random
sample, our goal is to classify the path it belongs to. To
deal with the variable length of the sub-sequence which can
range from 1 to 216, we pad the sub-sequence with the
vector [−1] to obtain an input sequence vector of length
216 as this is the maximum possible length. We chose this
particular constant as this number would not be observed
naturally in our dataset. Then we employed a masking layer
which skips the step containing the chosen padding value.
This resultant input vector is fed into an LSTM consisting
of 80 hidden units which was empirically determined to be
sufficient. A fully connected layer is then used to obtain an
output vector of size 3. The network architecture is shown
in Fig. 4. The LSTM uses the hyperbolic tangent activation
function whereas the dense layer uses the softmax activation
to obtain normalized probabilities corresponding to the 3
Fig. 4: Architecture of the LSTM used for path classification
based on sequence of pedestrian density.
prediction categories.
To facilitate training, we preprocess the training data by
normalizing it and then scaling it to the range of 0 to 1.
Random sub-sequences were then sampled from this data to
form the training set. This approach allows the network to
predict a path category regardless of the location from which
the sub-sequence is extracted or its length, as long as it is
sequential. We use the Adam optimizer with a learning rate
of 0.0001 and the categorical cross-entropy loss function:
L(y, ˆy) = −
M
j=0
N
i=0
yij log(ˆyij) (2)
where yij is a vector representing the ground truth and ˆyij
is the vector of class probabilities predicted by the network.
N is the total number of samples used to train the network in
that particular forward-backward pass and M is the number
of categorical classes which in this problem is 3. During the
test phase, we use the mean and standard deviation of the
training set to normalize the test data to maintain consistency
with the input used to train the network, then re-scale it to
fit in the range of 0 to 1.

5. Fig. 5: Classification accuracy of CBCL over the number
of consecutive steps traveled by an autonomous vehicle
along its path for the Pedestrian Pattern Dataset (top), and
Synthetic Pedestrian Dataset (bottom).
IV. USING PEDESTRIAN DENSITY TO RECOGNIZE A PATH
A. Experimental Procedure
The purpose of this experiment was to determine if and
how well the density of pedestrians can be used to iden-
tify the vehicle’s path. We hypothesized that the density
of pedestrians could serve as a signal of the autonomous
vehicle’s path. To test this hypothesis the two datasets
presented in section III-A were used. Notionally we consider
an autonomous vehicle that travels along some part of a
path, collecting information about the number of pedestrians
encountered, starting from an unknown point. In other words,
the vehicle is expected to collect pedestrian information for
t consecutive steps where t ∈ {5, 10, 20, ..., 216}. In the
general case when there are T steps, the dimension of the
vectors in the training set is RT ×1
. However the samples of
the test set have dimension Rt×1
where t is the number of
consecutive data points that the autonomous vehicle collects.
For instance, if the autonomous vehicle is expected to collect
20 consecutive data points on the paths studied in this work,
the dimension of the vectors for the training set and the test
set are R216×1
and R20×1
, respectively.
In order to recognize the correct path, the path recognition
models for the Pedestrian Pattern Dataset and the Synthetic
Pedestrian Dataset are learned separately using the classifi-
cation algorithms introduced in Section III-B. These models
are then evaluated on a test set created from each dataset for
varying values of t. The classification accuracy is calculated
as the percentage of successful classifications over the total
number of test cases. To summarize, path recognition is
applied separately on the Pedestrian Pattern Dataset and the
Synthetic Pedestrian Dataset using CBCL and LSTM-based
classification.
Fig. 6: Classification accuracy for the LSTM-based clas-
sifier vs the number of consecutive steps traveled by an
autonomous vehicle along its path in the Pedestrian Pattern
Dataset (top), and Synthetic Pedestrian Dataset (bottom).
B. Results and Discussion
For this experiment, we discretized the three Paths A,
B and C into T = 216 steps as discussed in section
III-A and the classification accuracy is reported for t ∈
{5, 10, 20, 30, 40, .., 200, 216}. The classification algorithms
were implemented on a computer running Ubuntu 18.04.02
LST equipped with an Intel i7-7700 CPU. The datasets were
divided using an 80:20 split without overlap into a training
and test set allowing us to perform a 5-fold cross validation.
In order to obtain more robust accuracy, the experiment for
each t is run 10 times while the test set is randomly generated
for each run.
The box charts in Fig. 5 and Fig. 6 depict
the classification accuracy of the 10 runs for
t ∈ {5, 10, 20, 30, 40, .., 200, 216} using CBCL and
LSTM-based classification algorithms for both datasets,
respectively. Due to noise in the real world the Pedestrian
Pattern Dataset, the average classification accuracy of
the 10 runs represented by the red curve in the plots is
noticeably higher for the Synthetic Pedestrian Dataset.
In order to more effectively compare the performance of
CBCL and LSTM-based classification for path recognition,
the average classification accuracy for each dataset for each
t is shown separately in Fig. 7.
In the case of the Pedestrian Pattern Dataset, if the
autonomous vehicle collects the pedestrian data only for
pi = 5 steps, the average classification accuracy for CBCL is
44.4% and for LSTM is 46.1%. Collecting more pedestrian
data along a path would improve the average classification
accuracy. For instance, when pi = 70 steps (∼ 1/3 of the
path), the average classification accuracy drastically increases
for CBCL to 68.1% and for LSTM to 70.2%. Eventually,
by collecting the pedestrian information for pi = 100 steps
(∼ 1/2 of the path), accuracy for CBCL and LSTM are
72.8% and 82.1%, respectively.

6. Fig. 7: Comparison of mean classification accuracy of CBCL
and LSTM-based classifier vs the number of consecutive
steps traveled by an autonomous vehicle along its pathway
for the Pedestrian Pattern Dataset (top), and Synthetic Pedes-
trian Dataset (bottom).
These classification methods produce comparatively simi-
lar results for the Synthetic Pedestrian Dataset. In this case,
collecting the pedestrian information only for pi = 10 steps
(∼ 1/10 of the path) generates a classification accuracy of
94.4% and 94.1% for CBCL and LSTM, respectively. While
the LSTM outperforms the CBCL algorithm when little
information is available, as t increases the CBCL method
improves in accuracy resulting in a higher average accuracy
over all the cases as shown in Table I.
TABLE I: Average classification accuracy of CBCL and
LSTM-based classifier for path recognition when t = 216.
CBCL LSTM
Pedestrian Pattern Dataset 92.4 % 89.8%
Synthetic Pedestrian Dataset 100% 97.8%
These results suggest that pedestrian density can be used
to recognize the autonomous vehicle’s path. In other words,
pedestrian density may serve as a landmark for identifying
one’s path. We have shown that this information can be
combined with different types of classification approaches
(CBCL and LSTM-based) for predicting the correct path
with high classification accuracy. As with all data-driven
approaches, the performance of the classifiers depend on
both the quality and the quantity of the training data. Taking
this into consideration, the path recognition approach might
achieve higher classification accuracy when greater and more
diverse number of pedestrians are present on the road as
confirmed by the higher mean classification accuracy on
the Synthetic Pedestrian Dataset compared to that of the
Pedestrian Pattern Dataset.
V. USING PEDESTRIAN DENSITY TO PREDICT PATH RISK
A. Experimental Procedure
The risk to pedestrians associated with an autonomous ve-
hicle’s pathway is related to the total number of pedestrians
present on that path [41]. Therefore, the ability to predict
the number of pedestrians along a given pathway is valuable
for determining the exposure risk and improving safety. In
this section we evaluate a method for predicting the total
number of pedestrians that one might encounter based on
current observations and prior pedestrian density data.
In order to predict the total number of pedestrians that
might be present on the vehicle’s current path, the vehicle
first identifies the path using the method presented in Section
IV. As before, we first discretize the three Paths A, B and
C into T = 216 steps. Here we assume that the autonomous
vehicle starts from the beginning of the path for which data
exists and collects the pedestrian data for t consecutive steps
where t ∈ {30, 60, 90, 120, 150}.
The previous experiment demonstrated that the CBCL
algorithm can be used for path recognition by leveraging the
learned centroids. For this experiment, we first used CBCL
to predict the correct path using the first t steps’ data and
then predicted the the total number of pedestrians which the
autonomous vehicle might encounter for the remaining part
of the predicted path. To accomplish this, first the closest
centroid, from the predicted path’s class to the test data was
calculated. Then the number of pedestrians for the next T −t
steps in the closest centroid of the predicted path are summed
to predict the pedestrian density for the next steps in the test
data.
The accuracy of the this method is evaluated by com-
paring the prediction for the total number of pedestrians
to the ground truth using the Mean Absolute Percentage
Error (MAPE) as a similarity measure. The ground truth
is determined by including the number of pedestrians for
the remaining unknown steps of the associated test path.
For instance, if the autonomous vehicle is assumed to travel
t = 40 steps along one of the test instances, the ground truth
is computed by adding the number of pedestrians for all the
remaining steps to the end point of that test instance.
Prediction Accuracy = 100 −
100
N
ΣN
i=1
| ˆEi − Ei |
Ei
(3)
where ˆEi and Ei are the predicted number of pedestrians
and the ground truth for the total number of pedestrians that
autonomous vehicle will encounter, respectively and N is
the number of the test set instances. In order to evaluate the
performance of this estimation algorithm on a more realistic
scenario, this risk estimation approach was only tested on
the Pedestrian Pattern Dataset using CBCL. The results are
discussed in the following section.
B. Results and Discussion
For this experiment the three Paths A, B and C were
discretized into T = 216 steps and the prediction accuracy is

7. reported for t ∈ {30, 60, 90, 120, 150}. We did not consider
t ≥ 150 because, in a real world scenario, the vehicle would
nearly be at the end location and this information would not
be valuable. The datasets are again divided using an 80:20
split without overlap for the training and test set allowing
us to perform a 5-fold cross validation. The experiment for
each t was run 10 times while the test set was randomly
generated for each run.
The box chart (top) in Fig. 8 presents the prediction
accuracy of the 10 runs for t ∈ {30, 60, 90, 120, 150} using
CBCL on the Pedestrian Pattern Dataset. The red curve
represents the average prediction accuracy of the 10 runs.
Hence, if an autonomous vehicle driving along a path collects
pedestrian data for t = 30 steps (∼ 1/6 of the path),
the average prediction accuracy for CBCL was found to be
37.5%. As expected, the average accuracy increases with the
number of steps. As we travel farther along a path, accuracy
of the total number of pedestrians predicted for the remainder
of the path was to improve. At t = 150 steps (∼ 3/4
of the path) the maximum prediction accuracy of 70.5% is
achieved.
Predicting the total number of pedestrians from prior data
is a challenging problem. Prediction error arises from at least
two different sources: 1) the error arising from incorrect path
recognition and; 2) the error associated with using centroids
to predict the number of pedestrian over the remainder of
the path.
We hypothesized that the prediction accuracy would be
highly dependent on the path recognition accuracy. To test
this hypothesis, we reexamined the data assuming that the
correct path was chosen, thus isolating the error associated
with using centroids to predict the path risk. The results of
this analysis is shown in Fig. 8 (bottom).
Note that path recognition when t ≤ 90 steps are available
has classification accuracy of 51.2% (refer to Fig 7). On
the other hand, if we assume that the path is known, then
the prediction accuracy dramatically improves. For instance,
under this assumption, when t = 30 the prediction accuracy
is 69.1%, an increase of 31.6% comparing to the situation
where the path is unknown (Fig. 8 top). The improved
performance results because knowing the path reduces space
of centroids considered. In other words, instead of comparing
against the centroids of all possible paths in the data, the
algorithm only compares against centroids belonging to one
path which in this case reduces the number of paths by
66% thereby reducing error. These results highlight the
importance of correctly recognizing the path for predicting
the total number of pedestrians. Moreover, the assumption
of an autonomous vehicle knowing the path that it is trav-
eling along is not unreasonable. In fact, our experimental
conditions represent a worst case scenario in which social
information is both being used to localize and to predict
future path risk.
VI. CONCLUSION
This paper presents an initial examination of how an
autonomous vehicle might use pedestrian density to guide
Fig. 8: Prediction accuracy for the total number of pedestri-
ans for Pedestrian Pattern Dataset vs the number of consec-
utive steps traveled along the pathway. The top includes path
prediction error. The bottom graph presents the error when
the path is known.
its route planning. With the goal of assuring pedestrian
safety, autonomous vehicles must be able to predict the risk
associated with a pathway and one of the factors affecting
risk is the total number of pedestrians on the road. We
presented a method for predicting the number of pedestrians
which is related to the risk of traveling down a path. We
demonstrate that two different classifiers can be used to rec-
ognize a path from information regarding pedestrian density.
We then develop and evaluate a system for estimating future
pedestrian density along a path.
Our work does make a number of assumptions. For
instance, our dataset only considers three different paths.
Our results from Table I suggest that it might be possible
to use social information to identify a path. Although GPS
is the predominate method for autonomous vehicle localiza-
tion, the method we present could, perhaps, augment GPS
based localization in densely populated urban areas. More
importantly, patterns of pedestrian density could inform the
vehicle about the path, day and time of travel. Moreover, we
have also shown that, given enough data, it may be possible
to predict the pedestrian density the vehicle will encounter
along a path. This information can then be used to enhance
safety by, for example, influencing the vehicle to take a less
crowded path.
This work offers a variety of avenues for impact and
novel research. Specifically, autonomous taxis might use our
approach to select pathways that avoid crowded intersections
conditioned on the time of day, day of the week, and week of
the month. Moreover, the method might be further enhanced
to include information about pedestrian behavior (e.g. run-
ning, jaywalking, meandering) and the pedestrian themselves
(e.g. disabled, children). It may be possible to incorporate
this information into the vehicle’s decision-making process.
Finally, our approach might allow the vehicle to characterize

8. events in terms of risk. For example, offering a means for the
vehicle to reason about the risk associated with travel near
a stadium during a football game. Overall, we believe that
the social environment can be used as an important source
of information by an autonomous vehicle.
ACKNOWLEDGMENT
This work was supported by Air Force Office of Scientific
Research contract FA9550-17-1-0017.
REFERENCES
[1] W. W. Lam, S. Yao, and B. P. Loo, “Pedestrian exposure measures:
a time-space framework,” Travel behaviour and society, vol. 1, no. 1,
pp. 22–30, 2014.
[2] S. Shao, Z. Zhao, B. Li, T. Xiao, G. Yu, X. Zhang, and J. Sun,
“Crowdhuman: A benchmark for detecting human in a crowd,” arXiv
preprint arXiv:1805.00123, 2018.
[3] J. Li, X. Liang, S. Shen, T. Xu, J. Feng, and S. Yan, “Scale-aware
fast r-cnn for pedestrian detection,” IEEE Transactions on Multimedia,
vol. 20, no. 4, pp. 985–996, 2017.
[4] P. Henry, C. Vollmer, B. Ferris, and D. Fox, “Learning to navigate
through crowded environments,” in 2010 IEEE International Confer-
ence on Robotics and Automation. IEEE, 2010, pp. 981–986.
[5] W. Burgard, A. B. Cremers, D. Fox, D. H¨ahnel, G. Lakemeyer,
D. Schulz, W. Steiner, and S. Thrun, “Experiences with an interactive
museum tour-guide robot,” Artificial intelligence, vol. 114, no. 1-2,
pp. 3–55, 1999.
[6] S. Thrun, M. Bennewitz, W. Burgard, A. B. Cremers, F. Dellaert,
D. Fox, D. Hahnel, C. Rosenberg, N. Roy, J. Schulte, et al., “Minerva:
A second-generation museum tour-guide robot,” in Proceedings 1999
IEEE International Conference on Robotics and Automation (Cat. No.
99CH36288C), vol. 3. IEEE, 1999.
[7] T. Randhavane, A. Bera, E. Kubin, A. Wang, K. Gray, and
D. Manocha, “Pedestrian dominance modeling for socially-aware
robot navigation,” in 2019 International Conference on Robotics and
Automation (ICRA). IEEE, 2019, pp. 5621–5628.
[8] E. A. Sisbot, L. F. Marin-Urias, R. Alami, and T. Simeon, “A human
aware mobile robot motion planner,” IEEE Transactions on Robotics,
vol. 23, no. 5, pp. 874–883, 2007.
[9] R. Kirby, “Social robot navigation,” Ph.D. dissertation, 2010.
[10] S. Davidsson and S. Chen, “Method and arrangement for handover
warning in a vehicle having autonomous driving capabilities,” Jan. 12
2016, uS Patent 9,235,211.
[11] M. K. Rao, M. A. Cuddihy, and J. Le, “Autonomous vehicle with
driver presence and physiological monitoring,” Oct. 28 2014, uS Patent
8,874,301.
[12] J. Wei, “Autonomous vehicle social behavior for highway driving,”
Ph.D. dissertation, 2017.
[13] A. Tavassoli, N. Cymbalist, A. Dunning, and D. Krauss, “Learning
from human naturalistic driving behavior at stop signs for autonomous
vehicles,” SAE Technical Paper, Tech. Rep., 2019.
[14] J.-G. Lee, K. Lee, and S.-H. Ryu, “Vehicle politeness in driving
situations,” Future Internet, vol. 11, no. 2, p. 48, 2019.
[15] D. Sadigh, S. Sastry, S. A. Seshia, and A. D. Dragan, “Planning for
autonomous cars that leverage effects on human actions,” Robotics:
Science and Systems XII, p. 61.
[16] A. Rasouli and J. K. Tsotsos, “Autonomous vehicles that interact with
pedestrians: A survey of theory and practice,” IEEE Transactions on
Intelligent Transportation Systems, vol. 21, no. 3, p. 900–918, 2020.
[17] J. Zhao, H. Xu, H. Liu, J. Wu, Y. Zheng, and D. Wu, “Detection
and tracking of pedestrians and vehicles using roadside lidar sensors,”
Transportation research part C: emerging technologies, vol. 100, pp.
68–87, 2019.
[18] C. Li, D. Song, R. Tong, and M. Tang, “Illumination-aware faster r-
cnn for robust multispectral pedestrian detection,” Pattern Recognition,
vol. 85, pp. 161–171, 2019.
[19] S. K. Choudhury, R. P. Padhy, A. K. Sangaiah, P. K. Sa, K. Muham-
mad, and S. Bakshi, “Scale aware deep pedestrian detection,” Trans-
actions on Emerging Telecommunications Technologies, vol. 30, no. 9,
p. e3522, 2019.
[20] D. O. Pop, A. Rogozan, C. Chatelain, F. Nashashibi, and A. Bensrhair,
“Multi-task deep learning for pedestrian detection, action recognition
and time to cross prediction,” IEEE Access, vol. 7, pp. 149 318–
149 327, 2019.
[21] E. Foxlin, “Pedestrian tracking with shoe-mounted inertial sensors,”
IEEE Computer graphics and applications, no. 6, pp. 38–46, 2005.
[22] R. F. Alonso, E. Z. Casanova, and J. G. Garc´ıa-Bermejo, “Pedestrian
tracking using inertial sensors,” Journal of Physical Agents (JoPha),
vol. 3, no. 1, p. 35–43, 2009.
[23] Q. Liu, Z. He, X. Li, and Y. Zheng, “Ptb-tir: A thermal infrared
pedestrian tracking benchmark,” IEEE Transactions on Multimedia,
vol. 22, no. 3, p. 666–675, 2020.
[24] Z. Sun, X. Mao, W. Tian, and X. Zhang, “Activity classification
and dead reckoning for pedestrian navigation with wearable sensors,”
Measurement science and technology, vol. 20, no. 1, p. 015203, 2008.
[25] G. Panahandeh, N. Mohammadiha, A. Leijon, and P. H¨andel, “Contin-
uous hidden markov model for pedestrian activity classification and
gait analysis,” IEEE Transactions on Instrumentation and Measure-
ment, vol. 62, no. 5, pp. 1073–1083, 2013.
[26] S. Khalifa, M. Hassan, and A. Seneviratne, “Adaptive pedestrian activ-
ity classification for indoor dead reckoning systems,” in International
Conference on Indoor Positioning and Indoor Navigation. IEEE,
2013, pp. 1–7.
[27] M. Nagai, K. Nakaoka, and Y. Doi, “Pedestrian-to-vehicle commu-
nication access method and field test results,” in 2012 International
Symposium on Antennas and Propagation (ISAP). IEEE, 2012, pp.
712–715.
[28] C. Sugimoto, Y. Nakamura, and T. Hashimoto, “Prototype of
pedestrian-to-vehicle communication system for the prevention of
pedestrian accidents using both 3g wireless and wlan communication,”
in 2008 3rd International Symposium on Wireless Pervasive Comput-
ing. IEEE, 2008, pp. 764–767.
[29] R. D. Strickland, M. Yuan, S. Bai, D. W. Weber, and R. Miucic,
“Vehicle to pedestrian communication system and method,” Aug. 23
2016, uS Patent 9,421,909.
[30] P. Sewalkar and J. Seitz, “Vehicle-to-pedestrian communication for
vulnerable road users: Survey, design considerations, and challenges,”
Sensors, vol. 19, no. 2, p. 358, 2019.
[31] M. Bennewitz, W. Burgard, G. Cielniak, and S. Thrun, “Learning mo-
tion patterns of people for compliant robot motion,” The International
Journal of Robotics Research, vol. 24, no. 1, pp. 31–48, 2005.
[32] D. Vasquez, T. Fraichard, and C. Laugier, “Growing hidden markov
models: An incremental tool for learning and predicting human and
vehicle motion,” The International Journal of Robotics Research,
vol. 28, no. 11-12, pp. 1486–1506, 2009.
[33] H. Bai, S. Cai, N. Ye, D. Hsu, and W. S. Lee, “Intention-aware
online pomdp planning for autonomous driving in a crowd,” 2015
IEEE International Conference on Robotics and Automation (ICRA),
p. 454–460, 2015.
[34] A. Rasouli, I. Kotseruba, and J. K. Tsotsos, “Understanding pedestrian
behavior in complex traffic scenes,” IEEE Transactions on Intelligent
Vehicles, vol. 3, no. 1, pp. 61–70, 2017.
[35] A. Millard-Ball, “Pedestrians, autonomous vehicles, and cities,” Jour-
nal of Planning Education and Research, vol. 38, no. 1, pp. 6–12,
2018.
[36] J. Lee, M. Abdel-Aty, and I. Shah, “Evaluation of surrogate measures
for pedestrian trips at intersections and crash modeling,” Accident
Analysis & Prevention, vol. 130, pp. 91–98, 2019.
[37] K. Mokhtari and A. R. Wagner, “The pedestrian patterns dataset,”
AAAI Fall Symposium, 2019.
[38] A. Ayub and A. R. Wagner, “CBCL: Brain inspired model for RGB-D
indoor scene classification,” arXiv:1911.00155, 2019.
[39] ——, “Cognitively-inspired model for incremental learning using a
few examples,” in The IEEE/CVF Conference on Computer Vision
and Pattern Recognition (CVPR) Workshops, June 2020.
[40] S. Hochreiter and J. Schmidhuber, “Long short-term memory,”
Neural Comput., vol. 9, no. 8, pp. 1735–1780, Nov. 1997. [Online].
Available: http://dx.doi.org/10.1162/neco.1997.9.8.1735
[41] J. Matus and P. Risbood, “Method and system for risk modeling in
autonomous vehicles,” July 3 2018, uS Patent App. 10/012,993.