ITSC2015 http://www.itsc2015.org/ The paper presents a fine-grained walking activity recognition toward an inferring pedestrian intention which is an important topic to predict and avoid a pedestrian’s dangerous activity. The fine-grained activity recognition is to distinguish different activities between subtle changes such as walking with different directions. We believe a change of pedestrian’s activity is significant to grab a pedestrian intention. However, the task is challenging since a couple of reasons, namely (i) in-vehicle mounted camera is always moving (ii) a pedestrian area is too small to capture a motion and shape features (iii) change of pedestrian activity (e.g. walking straight into turning) has only small feature difference. To tackle these problems, we apply vision-based approach in order to classify pedestrian activities. The dense trajectories (DT) method is employed for high-level recognition to capture a detailed difference. Moreover, we additionally extract detection-based region-of-interest (ROI) for higher performance in fine-grained activity recognition. Here, we evaluated our proposed approach on “self-collected dataset” and “near-miss driving recorder (DR) dataset” by dividing several activities– crossing, walking straight, turning, standing and riding a bicycle. Our proposal achieved 93.7% on the self-collected NTSEL traffic dataset and 77.9% on the near-miss DR dataset.

1. Fine-grained Walking Activity Recognition
via Driving Recorder Dataset
Hirokatsu KATAOKA, Yoshimitsu AOKI†, Yutaka SATOH
Shoko, OIKAWA‡, Yasuhiro MATSUI‡
National Institute of Advanced Industrial Science and Technology (AIST)
† Keio University
‡ National Traffic Safety and Environment Laboratory (NTSEL)
http://www.hirokatsukataoka.net/

2. Background
• ADAS; Advanced Driver Assistance Systems
– A large amount of technologies have been proposed
– The pedestrian deaths are on the rise
– Detection systems, environment, autonomous driving car
@Pedestrian
and
vehicle
detec0on
@Lane
detec0on
(Environment
understanding)
@Autonomous
driving
in
Google
ADAS technologies are highly required!

3. Pedestrian detection
• Vision-based detection is one of the important techniques
– Pedestrian detection survey [Benenson+, ECCVW2014]
• They implemented and compared 40+ detection approaches
– Deep Learning is applied to detect pedestrians [Sermanet+, CVPR2013]
• Convolutional neural networks (CNN)
• Automatic feature training and classifier
Better
Detection rate has been improving

4. New step toward “pedestrian analysis”
• High-performance pedestrian localization
– Task-assistant CNN (TA-CNN) [Tian+, CVPR2015]
• The framework is consist of CNN feat. & attribute (e.g. background, location)
• Limitations of pedestrian safety systems
– Pedestrian detection at present
– Detection range: width of the vehicle
Going to the next “pedestrian analysis” researches!

5. Motivation
• Fine-grained pedestrian activity recognition in addition to
pedestrian detection
– More detailed activity analysis
– Pedestrian activity intention understanding
Probabilitymapofdanger
1.0 second is crucial time in ADAS
Why fine-grained?
Walking along a sidewalk
Turning
Crossing a roadway

6. Process flow
• Fine-grained walking activity recognition
1. Pedestrian localization
2. Activity analysis
Improved dense trajectories (iDT)
Pedestrian detection
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
Trajectory (in t + L frames)
Feature extraction
(HOG, HOF, MBH, Traj.)
Bag-of-words (BoW)
iDT

7. Detection system
• Per-frame CNN feature and NMS
– Region of interesting (ROI)
– VGGNet feature in the detection problem
– Non-maximum suppression for combining detection windows
・・・~
~・・・
NMS

8. Activity Recognition
• Improved Dense Trajectories (iDT) [Wang+, ICCV2013]
– Pyramidal image sequences and flow tracking
– Feature descriptors on trajectories
– Feature representation with bag-of-words (BoW)
WalkingCrossing Turning

9. Experiments
• Fine-grained walking activity recognition
– Understanding small changes while people walking
• Walking along a side walk & Crossing a road way
• Walking straight & turning
• Walking & riding a bicycle
(a)
crossing
(b)
walking
(c)
turning
(d)
bicycle

10. Datasets and implementations
• NTSEL dataset & Near-miss dataset
• Implementation
– Localization: VGGNet layer-pooling-5
– Feature: IDT (HOG, HOF, MBH, Traj.)
– Classifier: Support vector machine (SVM)
(a)
crossing
(b)
walking
(c)
turning
(d)
bicycle
NTSEL dataset Near-miss DR dataset
http://www.jsae.or.jp/hiyari/0907/

11. Results
• On the NTSEL and Near-miss DR dataset
Descriptor % on NTSEL % on Near-miss
DT (Traj.) 76.5 77.9
DT (HOF) 93.7 75.9
DT (HOG) 85.6 76.4
DT (MBHx) 87.7 59.3
DT (MBHy) 86.7 60.8
– Outstanding performance rate with IDT 93.7% on NTSEL and 77.9% on Near-
miss DR dataset

12. Spatio-temporal analysis
• Using iDT, temporal direction is analyzed
– Fewer frames are better in the space-time
– Sudden motion should be recognized

13. Demonstration
• Fine-grained ped. activity recognition on NTSEL dataset
– Improved Dense Trajectories (93.7%)

14. Conclusion
• Fine-grained walking activity analysis for the new step of
pedestrian intention understanding
– State-of-the-art motion analysis algorithms are implemented
– High-performance localization and recognition on the traffic datasets
– Pedestrian analysis are executed in detail
• More flexible models and intention understanding
– We need more data in learning step
– Transition model or more strong temporal feature should be implemented