Learning from Dashcam Videos
•
4 likes•1,490 views
This document summarizes two papers presented at a conference on using dashcam videos. The first paper by Chan et al. presents a method for anticipating accidents in dashcam videos using a recurrent neural network with spatial attention. Their method can anticipate accidents about 2 seconds before they occur with 80% recall and 56.14% precision. The second paper by Chang et al. presents a method for extracting driving behavior from dashcam videos by performing global metric reconstruction to match dashcam frames to street view images. Their demo allows matching dashcam frames to similar street view images.
More Related Content
Viewers also liked
Viewers also liked (14)
Similar to Learning from Dashcam Videos
Similar to Learning from Dashcam Videos (20)
More from NVIDIA Taiwan
More from NVIDIA Taiwan (20)
Recently uploaded
Recently uploaded (20)
Learning from Dashcam Videos
- 1. TAIPEI | SEP. 21-22, 2016 Min Sun, Sept. 21, 2016 LEARNING FROM DASHCAM VIDEOS
- 2. 2 AGENDA • Anticipate Accident - Chan et al. ACCV’16 oral • Extracting Driving Behavior - Chang et al. ECCV’16 workshop
- 3. 3 Using Dashcam Videos to Anticipate Accidents 詹富翔 Fu-Hsiang Chan NTHU EE 向 宇 Yu Xiang Stanford CS 陳玉亭 Yu-Ting Chen NTHU EE 孫 民 Min Sun NTHU EE VSLab
- 4. 4 MOTIVATION VSLab Google’s self-driving car is involved in 12 minor accidents mostly caused by other human drivers. Using dashcam videos to anticipate corner cases (e.g., accient). Google self-driving car project monthly report (2015)
- 6. 6 POPULATION AND MOTOR VEHICLES DENSITY Taiwan USA Japan Korea German UK Area (km2 ) 36.2 9,831.5 377.9 99.9 357.1 243.6 Population Density (No./km2 ) 641 32 337 490 229 255 Motorbike Density (No./km2 ) 614 26 232 165 155 140 Vehicles Density (No./km2 ) 195 25 199 147 144 135 資料來源：中華民國環境保護統計年報101年表8-1 VSLab
- 7. 7 MORE COMPLEX ENVIRONMENT Japan Taiwan VS VSLab Japan Taiwan
- 9. 9 ACCIDENT TYPES OF 620 VIDEOS VSLab Bike hits Car 42.6% Car hits Car 19.7% Bike hits Bike 15.6% Others 22%
- 12. 12 Faster-RCNN (Detection) S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS, 2015 VSLab Car Car Person Person Person Motorbike Motorbike Motorbike Car
- 13. 13 Motion VSLab
- 14. 14 VSLab Heng Wang and Cordelia Schmid, “Action recognition with Improved Dense Trajectory (IDT)
- 16. 16 ANTICIPATING ACCIDENTS MODEL VSLab • Recurrent neural network
- 17. 17 • Spatial attention modelｚｘ ANTICIPATING ACCIDENTS MODEL VSLab Time = t RNNRNNRNN Time = t+1 Time = t+2 Weighted sum Weighted sum Weighted sum Attention Attention
- 18. 18 ANTICIPATING ACCIDENTS MODEL VSLab • Exponential loss Time Ashesh Jain, Hema S. Koppula, Bharad Raghavan, Shane Soh, and Ashutosh Saxena, “Car that knows before you do: Anticipating maneuvers via learning temporal driving models,” in ICCV, 2015.
- 19. 19 ANTICIPATING ACCIDENTS MODEL VSLab • Recurrent Neural Network • Spatial Attention Model • Exponential Loss
- 20. 20 EXPERIENCES VSLab Positive examples Negative examples Total Training set 455 829 1284 Testing set 165 301 466 Total 620 1130 1750 • Positive : Negative ≒ 2:3 • Training : Testing ≒ 3:1 Negative example Positive example
- 21. 21 mAP [1] [2] Finetune Faster-RCNN VSLab • Training set: KITTI dataset + 58 additional videos • Testing set: 165 positive examples of testing set 29% 35% 27% 15% 35% 28% [1] M. Everingham et al.“The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results,” 2007. [2] T.-Y. Lin et al. “Microsoft COCO: Com- ´ mon Objects in Context,” in ECCV, 2014. General photos StreetView photos
- 22. 22 ANTICIPATING ACCIDENTS RESULTS VSLab Appearance Motion Recurrent Neural Network Single-frame Classifier (SFC) Frame baseAverage attention Concatenate the frame with the average attentionWeighted-summing frame with attention on objectConcatenating frame with attention on object Frame T SFC VGG or IDT Output Frame T+1 SFC VGG or IDT Output RNN RNN attention Only Attention on object
- 23. 23 ANTICIPATING ACCIDENTS RESULTS VSLab Achieve the best 74.35% mAP Appearance Motion
- 24. 24 ANTICIPATING ACCIDENTS RESULT Our method anticipates accidents about 2 seconds before they occur with 80% recall and 56.14% precision. VSLab 56.14% ≒2
- 29. 29 RELATED WORK VSLab B. Frohlich, M. Enzweiler, and U. Franke, “Will this car change the lane? - turn signal recognition in the frequency domain,” in Intelligent Vehicles Symposium (IV), 2014. A. Doshi, B. Morris, and M. Trivedi, “On-road prediction of driver’s intent with multimodal sensory cues,” IEEE Pervasive Computing, vol. 10, no. 3, pp. 22–34, 2011.
- 30. 30 RELATED WORK VSLab Ashesh Jain, Avi Singh, Hema S Koppula, Shane Soh, and Ashutosh Saxena, “Recurrent neural networks for driver activity anticipation via sensory-fusion architecture,” in ICRA, 2016. Ashesh Jain, Hema S. Koppula, Bharad Raghavan, Shane Soh, and Ashutosh Saxena, “Car that knows before you do: Anticipating maneuvers via learning temporal driving models,” in ICCV, 2015.
- 31. 31 AGENDA • Anticipate Accident - Chan et al. ACCV’16 oral • Extracting Driving Behavior - Chang et al. ECCV’16 workshop
- 32. 32 Extracting Driving Behavior: Global Metric Localization from Dashcam Videos in the Wild 孫 民 Min Sun NTHU EE 陳煥宗 Hwann-Tzong Chen NTHU CS 張劭平 NTHU EE 簡瑞霆 NTHU CS 王福恩 NTHU EE 楊尚達 NTHU EE
- 33. 33
- 34. 34 SfM Global Metric Reconstruction Image-level Matching Patch-level Matching 3D Alignment Top 3 similar images The most similar images Dash frame Street view Output
- 35. 35 DEMO
- 36. 36 AGENDA • Anticipate Accident - Chan et al. ACCV’16 oral • Extracting Driving Behavior - Chang et al. ECCV’16 workshop http://aliensunmin.github.io/ VSLab