1.
Fine-grained Walking Activity Recognition
via Driving Recorder Dataset
Hirokatsu KATAOKA, Yoshimitsu AOKI†, Yutaka SATOH
Shoko, OIKAWA‡, Yasuhiro MATSUI‡
National Institute of Advanced Industrial Science and Technology (AIST)
† Keio University
‡ National Traffic Safety and Environment Laboratory (NTSEL)
http://www.hirokatsukataoka.net/

2.
Background
• ADAS; Advanced Driver Assistance Systems
– A large amount of technologies have been proposed
– The pedestrian deaths are on the rise
– Detection systems, environment, autonomous driving car
@Pedestrian
and
vehicle
detec0on
@Lane
detec0on
(Environment
understanding)
@Autonomous
driving
in
Google
ADAS technologies are highly required!

3.
Pedestrian detection
• Vision-based detection is one of the important techniques
– Pedestrian detection survey [Benenson+, ECCVW2014]
• They implemented and compared 40+ detection approaches
– Deep Learning is applied to detect pedestrians [Sermanet+, CVPR2013]
• Convolutional neural networks (CNN)
• Automatic feature training and classifier
Better
Detection rate has been improving

4.
New step toward “pedestrian analysis”
• High-performance pedestrian localization
– Task-assistant CNN (TA-CNN) [Tian+, CVPR2015]
• The framework is consist of CNN feat. & attribute (e.g. background, location)
• Limitations of pedestrian safety systems
– Pedestrian detection at present
– Detection range: width of the vehicle
Going to the next “pedestrian analysis” researches!

5.
Motivation
• Fine-grained pedestrian activity recognition in addition to
pedestrian detection
– More detailed activity analysis
– Pedestrian activity intention understanding
Probabilitymapofdanger
1.0 second is crucial time in ADAS
Why fine-grained?
Walking along a sidewalk
Turning
Crossing a roadway

6.
Process flow
• Fine-grained walking activity recognition
1. Pedestrian localization
2. Activity analysis
Improved dense trajectories (iDT)
Pedestrian detection
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
Trajectory (in t + L frames)
Feature extraction
(HOG, HOF, MBH, Traj.)
Bag-of-words (BoW)
iDT

7.
Detection system
• Per-frame CNN feature and NMS
– Region of interesting (ROI)
– VGGNet feature in the detection problem
– Non-maximum suppression for combining detection windows
・・・~
~・・・
NMS

8.
Activity Recognition
• Improved Dense Trajectories (iDT) [Wang+, ICCV2013]
– Pyramidal image sequences and flow tracking
– Feature descriptors on trajectories
– Feature representation with bag-of-words (BoW)
WalkingCrossing Turning

9.
Experiments
• Fine-grained walking activity recognition
– Understanding small changes while people walking
• Walking along a side walk & Crossing a road way
• Walking straight & turning
• Walking & riding a bicycle
(a)
crossing
(b)
walking
(c)
turning
(d)
bicycle

10.
Datasets and implementations
• NTSEL dataset & Near-miss dataset
• Implementation
– Localization: VGGNet layer-pooling-5
– Feature: IDT (HOG, HOF, MBH, Traj.)
– Classifier: Support vector machine (SVM)
(a)
crossing
(b)
walking
(c)
turning
(d)
bicycle
NTSEL dataset Near-miss DR dataset
http://www.jsae.or.jp/hiyari/0907/

11.
Results
• On the NTSEL and Near-miss DR dataset
Descriptor % on NTSEL % on Near-miss
DT (Traj.) 76.5 77.9
DT (HOF) 93.7 75.9
DT (HOG) 85.6 76.4
DT (MBHx) 87.7 59.3
DT (MBHy) 86.7 60.8
– Outstanding performance rate with IDT 93.7% on NTSEL and 77.9% on Near-
miss DR dataset

12.
Spatio-temporal analysis
• Using iDT, temporal direction is analyzed
– Fewer frames are better in the space-time
– Sudden motion should be recognized

13.
Demonstration
• Fine-grained ped. activity recognition on NTSEL dataset
– Improved Dense Trajectories (93.7%)

14.
Conclusion
• Fine-grained walking activity analysis for the new step of
pedestrian intention understanding
– State-of-the-art motion analysis algorithms are implemented
– High-performance localization and recognition on the traffic datasets
– Pedestrian analysis are executed in detail
• More flexible models and intention understanding
– We need more data in learning step
– Transition model or more strong temporal feature should be implemented