Download as PDF, PPTX
Uploaded byUniversitat Politècnica de Catalunya
Temporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks
The document discusses a methodology for temporal activity detection in untrimmed videos using recurrent neural networks and convolutional neural networks. It covers the architecture, training processes, and results of classifying and detecting various activities across a large dataset of nearly 20,000 videos. The authors also highlight challenges, future work in end-to-end training, and attention models for improved performance.
More Related Content
Similar to Temporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks
More from Universitat Politècnica de Catalunya
Temporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks
- 1. Temporal Activity Detectionin Untrimmed Videos with Recurrent Neural Networks Alberto Montes July 15th, 2016 Xavi Giró Amaia Salvador
- 2. Outline 1. Introduction 2. RelatedWork 3. Methodology 4. Results 5. Conclusions and Future Work 2
- 3.
- 4.
- 5.
- 6.
- 7. Problem Definition 7 Videos Activity TemporalLocalization Longboarding
- 8.
- 9.
- 10.
- 11. 11 Large-Scale Activity Recognition Challenge Stats: ●19,994 Videos ● 200 Activities ● 660 hours of video ● 313 hours of activities ● 65.6 million of frames
- 12.
- 13. Outline 1. Introduction 2. RelatedWork 3. Methodology 4. Results 5. Conclusions and Future Work 13
- 14.
- 15.
- 16.
- 17.
- 18. 3D Convolution 18 Tran, D.,Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015, December). Learning spatiotemporal features with 3d convolutional networks. In 2015 IEEE ICCV 2015 (pp. 4489-4497). IEEE.
- 19. 3D Convolution 19 ● 16-framevideo clip as input ● 80 million parameters ● 3x3x3 filter size at all conv layers Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015, December). Learning spatiotemporal features with 3d convolutional networks. In 2015 IEEE ICCV 2015 (pp. 4489-4497). IEEE.
- 20.
- 21.
- 22. Segments Proposals 22 Shou, Z.,Wang, D., & Chang, S. F. Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs CVPR 2016.
- 23.
- 24. RNN for ActivityLocalization 24 Yeung, Serena, Olga Russakovsky, Greg Mori, and Li Fei-Fei. et al. "End-to-end Learning of Action Detection from Frame Glimpses in Videos." CVPR 2016
- 25. Outline 1. Introduction 2. RelatedWork 3. Methodology 4. Results 5. Conclusions and Future Work 25
- 26. Architecture Overview 26 16 frames200 activities + background 16 frames 200 activities + background 16 frames 200 activities + background
- 27. Outline 3. Methodology a. ExtractingC3D Features b. Audio Features c. Network Architecture d. Training Methodology e. Post-Processing 27
- 28. Outline 3. Methodology a. ExtractingC3D Features b. Audio Features c. Network Architecture d. Training Methodology e. Post-Processing 28
- 29. C3D Network 29 Caffe + by featurevector published on:
- 30.
- 31. Outline 3. Methodology a. ExtractingC3D Features b. Audio Features c. Network Architecture d. Training Methodology e. Post-Processing 31
- 32. Audio Features 32 C3D Recurrent NeuralNetwork Input Audio Features: ● MFCC ● Spectral concatvideo features Provided by Ignasi Esquerra
- 33. Outline 3. Methodology a. ExtractingC3D Features b. Audio Features c. Network Architecture d. Training Methodology e. Post-Processing 33
- 34.
- 35.
- 36. Network Architecture 36 LSTM withprevious output feedback
- 37. Outline 3. Methodology a. ExtractingC3D Features b. Audio Features c. Network Architecture d. Training Methodology e. Post-Processing 37
- 38. Training Methodology Categorical CrossEntropy Loss 38
- 39. Training Methodology For unbalanceddata, weighted loss: 39 660 hours of video 313 hours of activities
- 40. Outline 3. Methodology a. ExtractingC3D Features b. Audio Features c. Network Architecture d. Training Methodology e. Post-Processing 40
- 41. Classification Post-Processing 41 Background Activity 1 Activity2 Activity 200 Clip1 Clip2 Clip3 ClipN
- 42. Classification Post-Processing 42 Background Activity 1 Activity2 Activity 200 Clip1 Clip2 Clip3 ClipN Average
- 43. Classification Post-Processing 43 Background Activity 1 Activity2 Activity 200 Clip1 Clip2 Clip3 ClipN Average Max Probability
- 44. Detection Post-Processing 44 Background Activity 1 Activity2 Activity 200 Clip1 Clip2 Clip3 ClipN Applied a mean filter of k samplestime
- 45.
- 46.
- 47. Outline 1. Introduction 2. RelatedWork 3. Methodology 4. Results 5. Conclusions and Future Work 47
- 48. Classification: Audio Features 48 mAP= 0.5755mAP = 0.5938 Music unrelated to the activity is often added to the videos in post-processing, causing a decrease in performance when audio and video features are combined.
- 49. Classification: Depth Analysis 49 mAP= 0.5938 mAP = 0.5492 mAP = 0.5635 Deeper networks present overfitting
- 50. Classification Results PerActivity 50
- 51. Classification Results PerActivity 51 Using the Pommel Horse Sailing Playing Ice Hockey Rock Climbing BMX
- 52. Classification Results PerActivity 52 Drinking Coffee Peeling Potatoes Having an Ice Cream Rock-Paper-Scissors Polishing shoes
- 53.
- 54. Detection 54 mAP = 0.2251mAP = 0.2067 Model with feedback did not improve results
- 55. Training with feedback 55 512-LSTM videofeatures0 0 1 0 0 0 concat When training previous ground truth
- 56. Training with feedback 56 512-LSTM videofeatures0 0.1 0.6 0.2 0.1 0 concat When testing previous prediction
- 57. Comparing Post-Processing 57 Ɣ Grid searchfor optimal parameters
- 58. Detection Results perActivity 58
- 59. Detection Results perActivity 59 Windsurfing Riding Bumper Cars Playing Racquetball Using the Pommel Horse Using Parallel Bars
- 60. Detection Results perActivity 60 Drinking Coffee Putting on Shoes Rock-Paper-Scissors Removing Curlers Smoking a Cigarette
- 61.
- 62. Qualitative Evaluation 62 Ground Truth: Playingwater polo Prediction: 0.765 Playing water polo 0.202 Swimming 0.007 Springboard diving
- 63. Qualitative Evaluation 63 Ground Truth: Hopscotch Prediction: 0.848Running a marathon 0.023 Triple jump 0.022 Javelin throw
- 64.
- 65.
- 66. Challenge Results 66 Classification Task (24participants) Baseline 42.20% 0% 100% 93.23% Winner Average Performance 66.26%58.74% UPC Team * results over test subset Slide Design by Issey Masuda mAP
- 67. Challenge Results 67 Detection Task (6participants) Baseline 9.70% 0% 50% 42.47% Winner Average Performance 29.94%22.36% UPC Team mAP * results over test subset Slide Design by Issey Masuda
- 68. Outline 1. Introduction 2. RelatedWork 3. Methodology 4. Results 5. Conclusions and Future Work 68
- 69.
- 70. Conclusions 70 Video Spatial Net Temporal Net Output Winningentry for ActivityNet Classification task Wang, Limin, et al. "Towards good practices for very deep two-stream convnets." arXiv preprint arXiv:1507.02159 (2015).
- 71.
- 72. Conclusions 72 Best results wereobtained for sport categories, due to the pretraining of C3D with the Sports-1M dataset
- 73. Future Work: E2ETraining 73 Training the whole pipeline end-to-end would reduce the bias towards sport categories
- 74. Future Work: AttentionModels 74 Temporal Attention Filters Neural Network
- 75.
- 76.
- 77. “Thank you foryour attention 77
- 78.
- 79.
- 80.
- 81.
- 82.
- 83.
- 84.
- 85. AP and VideoAppearance Correlation 85
- 86. AP and VideoAppearance Correlation 86
- 87.
- 88. Preparing Data 88 Sequence ofVideo Vector Features Sequence of Activities time
- 89.
- 90.
- 91.
- 92. Gathering Audio Features 92 16-FrameClip 10 ms MFCC Features t 10 ms MFCC Features 10 ms MFCC Features 10 ms MFCC Features 10 ms MFCC Features 10 ms MFCC Features 16-Frame Clip Spectral Features … … …
- 93. Gathering Audio Features 93 16-FrameClip mean MFCC Features t std MFCC Features 16-Frame Clip Spectral Features … … … mean MFCC Features std MFCC Features
- 94. Gathering Audio Features 94 16-FrameClip mean MFCC Features t std MFCC Features 16-Frame Clip Spectral Features … … … mean MFCC Features std MFCC Features Spectral Features
- 95.
- 96.
- 97.
- 98.