Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

【ITSC2015】Fine-grained Walking Activity Recognition via Driving Recorder Dataset


Published on


The paper presents a fine-grained walking activity recognition toward an inferring pedestrian intention which is an important topic to predict and avoid a pedestrian’s dangerous activity. The fine-grained activity recognition is to distinguish different activities between subtle changes such as walking with different directions. We believe a change of pedestrian’s activity is significant to grab a pedestrian intention. However, the task is challenging since a couple of reasons, namely (i) in-vehicle mounted camera is always moving (ii) a pedestrian area is too small to capture a motion and shape features (iii) change of pedestrian activity (e.g. walking straight into turning) has only small feature difference. To tackle these problems, we apply vision-based approach in order to classify pedestrian activities. The dense trajectories (DT) method is employed for high-level recognition to capture a detailed difference. Moreover, we additionally extract detection-based region-of-interest (ROI) for higher performance in fine-grained activity recognition. Here, we evaluated our proposed approach on “self-collected dataset” and “near-miss driving recorder (DR) dataset” by dividing several activities– crossing, walking straight, turning, standing and riding a bicycle. Our proposal achieved 93.7% on the self-collected NTSEL traffic dataset and 77.9% on the near-miss DR dataset.

Published in: Science
  • Be the first to comment

【ITSC2015】Fine-grained Walking Activity Recognition via Driving Recorder Dataset

  1. 1. Fine-grained Walking Activity Recognition via Driving Recorder Dataset Hirokatsu KATAOKA, Yoshimitsu AOKI†, Yutaka SATOH Shoko, OIKAWA‡, Yasuhiro MATSUI‡ National Institute of Advanced Industrial Science and Technology (AIST) † Keio University ‡ National Traffic Safety and Environment Laboratory (NTSEL)
  2. 2. Background •  ADAS; Advanced Driver Assistance Systems –  A large amount of technologies have been proposed –  The pedestrian deaths are on the rise –  Detection systems, environment, autonomous driving car @Pedestrian  and  vehicle  detec0on   @Lane  detec0on  (Environment  understanding)   @Autonomous  driving  in  Google   ADAS technologies are highly required!
  3. 3. Pedestrian detection •  Vision-based detection is one of the important techniques –  Pedestrian detection survey [Benenson+, ECCVW2014] •  They implemented and compared 40+ detection approaches –  Deep Learning is applied to detect pedestrians [Sermanet+, CVPR2013] •  Convolutional neural networks (CNN) •  Automatic feature training and classifier Better Detection rate has been improving
  4. 4. New step toward “pedestrian analysis” •  High-performance pedestrian localization –  Task-assistant CNN (TA-CNN) [Tian+, CVPR2015] •  The framework is consist of CNN feat. & attribute (e.g. background, location) •  Limitations of pedestrian safety systems –  Pedestrian detection at present –  Detection range: width of the vehicle Going to the next “pedestrian analysis” researches!
  5. 5. Motivation •  Fine-grained pedestrian activity recognition in addition to pedestrian detection –  More detailed activity analysis –  Pedestrian activity intention understanding Probabilitymapofdanger 1.0 second is crucial time in ADAS Why fine-grained? Walking along a sidewalk Turning Crossing a roadway
  6. 6. Process flow •  Fine-grained walking activity recognition 1.  Pedestrian localization 2.  Activity analysis Improved dense trajectories (iDT) Pedestrian detection x x x x x x x x x x x x x x x x x x Trajectory (in t + L frames) Feature extraction (HOG, HOF, MBH, Traj.) Bag-of-words (BoW) iDT
  7. 7. Detection system •  Per-frame CNN feature and NMS –  Region of interesting (ROI) –  VGGNet feature in the detection problem –  Non-maximum suppression for combining detection windows ・・・~   ~・・・   NMS
  8. 8. Activity Recognition •  Improved Dense Trajectories (iDT) [Wang+, ICCV2013] –  Pyramidal image sequences and flow tracking –  Feature descriptors on trajectories –  Feature representation with bag-of-words (BoW) WalkingCrossing Turning
  9. 9. Experiments •  Fine-grained walking activity recognition –  Understanding small changes while people walking •  Walking along a side walk & Crossing a road way •  Walking straight & turning •  Walking & riding a bicycle (a)  crossing (b)  walking (c)  turning (d)  bicycle
  10. 10. Datasets and implementations •  NTSEL dataset & Near-miss dataset •  Implementation –  Localization: VGGNet layer-pooling-5 –  Feature: IDT (HOG, HOF, MBH, Traj.) –  Classifier: Support vector machine (SVM) (a)  crossing (b)  walking (c)  turning (d)  bicycle NTSEL dataset Near-miss DR dataset
  11. 11. Results •  On the NTSEL and Near-miss DR dataset Descriptor % on NTSEL % on Near-miss DT (Traj.) 76.5 77.9 DT (HOF) 93.7 75.9 DT (HOG) 85.6 76.4 DT (MBHx) 87.7 59.3 DT (MBHy) 86.7 60.8 –  Outstanding performance rate with IDT 93.7% on NTSEL and 77.9% on Near- miss DR dataset
  12. 12. Spatio-temporal analysis •  Using iDT, temporal direction is analyzed –  Fewer frames are better in the space-time –  Sudden motion should be recognized
  13. 13. Demonstration •  Fine-grained ped. activity recognition on NTSEL dataset –  Improved Dense Trajectories (93.7%)
  14. 14. Conclusion •  Fine-grained walking activity analysis for the new step of pedestrian intention understanding –  State-of-the-art motion analysis algorithms are implemented –  High-performance localization and recognition on the traffic datasets –  Pedestrian analysis are executed in detail •  More flexible models and intention understanding –  We need more data in learning step –  Transition model or more strong temporal feature should be implemented