Temporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks
Report
Jul. 15, 2016•0 likes•16,316 views
Temporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks
Jul. 15, 2016•0 likes•16,316 views
Download to read offline
Report
Science
https://imatge-upc.github.io/activitynet-2016-cvprw/ This thesis explore different approaches using Convolutional and Recurrent Neural Networks to classify and temporally localize activities on videos, furthermore an implementation to achieve it has been proposed. As the first step, features have been extracted from video frames using an state of the art 3D Convolutional Neural Network. This features are fed in a recurrent neural network that solves the activity classification and temporally location tasks in a simple and flexible way. Different architectures and configurations have been tested in order to achieve the best performance and learning of the video dataset provided. In addition it has been studied different kind of post processing over the trained network's output to achieve a better results on the temporally localization of activities on the videos. The results provided by the neural network developed in this thesis have been submitted to the ActivityNet Challenge 2016 of the CVPR, achieving competitive results using a simple and flexible architecture.
Recommended
Deep Learning for Computer Vision (3/4): Video Analytics @ laSalle 2016Universitat Politècnica de Catalunya
Deep Learning for Computer Vision: Video Analytics (UPC 2016)Universitat Politècnica de Catalunya
Video Analysis (D4L2 2017 UPC Deep Learning for Computer Vision)Universitat Politècnica de Catalunya
Deep Learning for Video: Action Recognition (UPC 2018)Universitat Politècnica de Catalunya
Deep Convnets for Video Processing (Master in Computer Vision Barcelona, 2016)Universitat Politècnica de Catalunya
Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentimen...Universitat Politècnica de Catalunya
More Related Content
What's hot
Deep Learning for Computer Vision (1/4): Image Analytics @ laSalle 2016Universitat Politècnica de Catalunya
Welcome (D1L1 2017 UPC Deep Learning for Computer Vision)Universitat Politècnica de Catalunya
Deep convnets for global recognition (Master in Computer Vision Barcelona 2016)Universitat Politècnica de Catalunya
Language and Vision (D3L5 2017 UPC Deep Learning for Computer Vision)Universitat Politècnica de Catalunya
Image classification on Imagenet (D1L4 2017 UPC Deep Learning for Computer Vi...Universitat Politècnica de Catalunya
Self-supervised Visual Learning 2020 - Xavier Giro-i-Nieto - UPC BarcelonaUniversitat Politècnica de Catalunya
Viewers also liked
Video Analysis with Recurrent Neural Networks (Master Computer Vision Barcelo...Universitat Politècnica de Catalunya
Layer-wise CNN Surgery for Visual Sentiment PredictionUniversitat Politècnica de Catalunya
Temporal Action Localization in Untrimmed Videos via Multi Stage CNNsUniversitat Politècnica de Catalunya
Multi-label Remote Sensing Image Retrieval based on Deep FeaturesUniversitat Politècnica de Catalunya
Human Action Recognition using Lagrangian DescriptorsEsra Açar
Training and Inference for Deep Gaussian ProcessesKeyon Vafa
Similar to Temporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks
Video Classification: Human Action Recognition on HMDB-51 datasetGiorgio Carbone
1-bit semantic segmentationJeonghoonKim30
Transfer Learning Model for Image Segmentation by Integrating U-NetPlusPlus a...YutaSuzuki27
Master Thesis Object Tracking in Video with TensorFlowAndrea Ferri
Perspective Multiscale Detection and Tracking of PersonsMarcos Nieto
Applying your Convolutional Neural NetworksDatabricks
More from Universitat Politècnica de Catalunya
Deep Generative Learning for AllUniversitat Politècnica de Catalunya
The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...Universitat Politècnica de Catalunya
Towards Sign Language Translation & Production | Xavier Giro-i-NietoUniversitat Politècnica de Catalunya
The Transformer - Xavier Giró - UPC Barcelona 2021Universitat Politècnica de Catalunya
Open challenges in sign language translation and productionUniversitat Politècnica de Catalunya
Generation of Synthetic Referring Expressions for Object Segmentation in VideosUniversitat Politècnica de Catalunya
Recently uploaded
Discussion of Jost et al. (2019): Input complexity affects long-term retentio...Pablo Bernabeu
Measuring Trustworthiness in Neuro-Symbolic IntegrationAndrea Omicini
Lecture 1: Introduction to Seiconductor.pdfvaishalideshmukh22
ML algorithms to find associations across biological data.pptxSelvajeyanthi S
Geography of Opportunity - Sansone.pptxnathancone5537
Global QCD analysis and dark photonsSérgio Sacani
Temporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks
- 1. Temporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks Alberto Montes July 15th, 2016 Xavi Giró Amaia Salvador
- 2. Outline 1. Introduction 2. Related Work 3. Methodology 4. Results 5. Conclusions and Future Work 2
- 3. Motivation 3
- 4. Motivation 4
- 5. Problem Definition 5 Videos
- 6. Problem Definition 6 Videos Activity Classification Longboarding
- 7. Problem Definition 7 Videos Activity Temporal Localization Longboarding
- 8. Problem Definition 8 How?
- 9. Problem Definition 9 Neural Network Activity
- 10. Problem Definition 10 Activity CNN RNN+
- 11. 11 Large-Scale Activity Recognition Challenge Stats: ● 19,994 Videos ● 200 Activities ● 660 hours of video ● 313 hours of activities ● 65.6 million of frames
- 12. Dataset 12
- 13. Outline 1. Introduction 2. Related Work 3. Methodology 4. Results 5. Conclusions and Future Work 13
- 14. Literature Approaches 14 Activity CNN RNN+
- 15. Convolutional Neural Network 15 Convolutional Layer
- 16. Recurrent Neural Network 16 c0 c1 c2
- 17. Literature Approaches 17 Activity CNN RNN+
- 18. 3D Convolution 18 Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015, December). Learning spatiotemporal features with 3d convolutional networks. In 2015 IEEE ICCV 2015 (pp. 4489-4497). IEEE.
- 19. 3D Convolution 19 ● 16-frame video clip as input ● 80 million parameters ● 3x3x3 filter size at all conv layers Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015, December). Learning spatiotemporal features with 3d convolutional networks. In 2015 IEEE ICCV 2015 (pp. 4489-4497). IEEE.
- 20. Literature Approaches 20 Activity CNN RNN+
- 21. Literature Approaches 21 Activity CNN RNN+
- 22. Segments Proposals 22 Shou, Z., Wang, D., & Chang, S. F. Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs CVPR 2016.
- 23. Literature Approaches 23 Activity CNN RNN+
- 24. RNN for Activity Localization 24 Yeung, Serena, Olga Russakovsky, Greg Mori, and Li Fei-Fei. et al. "End-to-end Learning of Action Detection from Frame Glimpses in Videos." CVPR 2016
- 25. Outline 1. Introduction 2. Related Work 3. Methodology 4. Results 5. Conclusions and Future Work 25
- 26. Architecture Overview 26 16 frames 200 activities + background 16 frames 200 activities + background 16 frames 200 activities + background
- 27. Outline 3. Methodology a. Extracting C3D Features b. Audio Features c. Network Architecture d. Training Methodology e. Post-Processing 27
- 28. Outline 3. Methodology a. Extracting C3D Features b. Audio Features c. Network Architecture d. Training Methodology e. Post-Processing 28
- 29. C3D Network 29 Caffe + by feature vector published on:
- 30. C3D Network 30 Caffe by feature vector
- 31. Outline 3. Methodology a. Extracting C3D Features b. Audio Features c. Network Architecture d. Training Methodology e. Post-Processing 31
- 32. Audio Features 32 C3D Recurrent Neural Network Input Audio Features: ● MFCC ● Spectral concatvideo features Provided by Ignasi Esquerra
- 33. Outline 3. Methodology a. Extracting C3D Features b. Audio Features c. Network Architecture d. Training Methodology e. Post-Processing 33
- 34. Network Architecture 34
- 35. Network Architecture 35
- 36. Network Architecture 36 LSTM with previous output feedback
- 37. Outline 3. Methodology a. Extracting C3D Features b. Audio Features c. Network Architecture d. Training Methodology e. Post-Processing 37
- 38. Training Methodology Categorical Cross Entropy Loss 38
- 39. Training Methodology For unbalanced data, weighted loss: 39 660 hours of video 313 hours of activities
- 40. Outline 3. Methodology a. Extracting C3D Features b. Audio Features c. Network Architecture d. Training Methodology e. Post-Processing 40
- 41. Classification Post-Processing 41 Background Activity 1 Activity 2 Activity 200 Clip1 Clip2 Clip3 ClipN
- 42. Classification Post-Processing 42 Background Activity 1 Activity 2 Activity 200 Clip1 Clip2 Clip3 ClipN Average
- 43. Classification Post-Processing 43 Background Activity 1 Activity 2 Activity 200 Clip1 Clip2 Clip3 ClipN Average Max Probability
- 44. Detection Post-Processing 44 Background Activity 1 Activity 2 Activity 200 Clip1 Clip2 Clip3 ClipN Applied a mean filter of k samplestime
- 45. Detection Post-Processing 45 Background Activity Clip1 Clip2 Clip3 ClipN Ɣ
- 46. Detection Post-Processing 46 Ɣ
- 47. Outline 1. Introduction 2. Related Work 3. Methodology 4. Results 5. Conclusions and Future Work 47
- 48.