The document summarizes a research paper on multi-modal weakly supervised dense event captioning. The paper aims to show that audio signals alone can achieve good performance on dense event captioning tasks. It uses a dataset of 20k videos with temporally localized sentences. The model uses different audio and video feature representations, which are fused using various strategies. Evaluation results show that fusing MFCC and SoundNet audio features with C3D video features using MUTAN fusion outperforms unimodal baselines. However, the model still has limitations in correctly detecting all event beginnings and generating captions for all events in weakly supervised videos.

1. WATCH, LISTEN AND TELL:
MULTI-MODAL WEAKLY
SUPERVISED DENSE EVENT
CAPTIONING
Seminar Hot Topics in Computer Vision
By Safaa Alnabulsi
1
22.12.20
Safaa Alnabulsi,TU Berlin

2. AGENDA
▪ The goal
▪ The dataset
▪ The baseline model
▪ The model in the paper
▪ The used algorithm
▪ Different feature representations
▪ Different fusion strategies
▪ Different evaluation scehmas
▪ The results
▪ The limitations
2
22.12.20
Safaa Alnabulsi,TU Berlin

3. VIDEO UNDERSTANDING
Safaa Alnabulsi,TU Berlin 22.12.20 3
Applications
Content-based recommednition and
retrieval
Autonomous driving
Surveillance
Software for visually-impaired people
Approaches
Action recognition
Content summarization
Action anticipation
Video question answering
Video captioning

4. GENERAL GOAL
▪ Both Detecting and describing events in a video
4
22.12.20
Safaa Alnabulsi,TU Berlin

5. MAIN GOAL OF THIS PAPER
▪ Prove that audio signals can carry surprising amount of information when it comes to
high-level visual-lingual tasks
▪ show that audio signal alone can achieve impressive performance on the dense
event captioning task
5
22.12.20
Safaa Alnabulsi,TU Berlin

6. THE DATASET
▪ 20k videos
▪ Avg. 3.56 temporally localized
senteces
▪ Avg. 13.48 words per sentences
6
22.12.20
Safaa Alnabulsi,TU Berlin

7. ENTRY KEY IN ACTIVITYNET
CAPTIONS DATASET
7
22.12.20
Safaa Alnabulsi,TU Berlin

8. THE ACTIVITY NET
CHALLANGE
8
22.12.20
Safaa Alnabulsi,TU Berlin

9. THE ACTIVITY NET
CHALLANGE
9
22.12.20
Safaa Alnabulsi,TU Berlin

10. THE BASELINE MODEL
▪ The problem was decomposed into a
pair of dual problems:
▪ event captioning
▪ sentence localization
10
22.12.20
Safaa Alnabulsi,TU Berlin

11. THE BASELINE MODEL
11
22.12.20
Safaa Alnabulsi,TU Berlin

12. THE MODEL IN THIS PAPER
12
22.12.20
Safaa Alnabulsi,TU Berlin

13. THE USED ALGORITHM
▪ Weakly Supervised
13
22.12.20
Safaa Alnabulsi,TU Berlin

14. DIFFERENT FEATURE
REPRESENTATIONS
▪ Audio Feature Processing
▪ MFCC Features
▪ CQT Features
▪ SoundNet Features
▪ Video Feature Processing
▪ 3D-CNN model is used to process the
input video frames into a sequence of
visual features
14
22.12.20
Safaa Alnabulsi,TU Berlin

15. DIFFERENT FUSION
STRATEGIES
15
22.12.20
Safaa Alnabulsi,TU Berlin

16. DIFFERENT EVALUATION
SCEHMAS
▪ TEOR (M)
▪ CIDEr (C)
▪ Rouge-L
▪ Spice (S)
▪ Bleu@N (B@N)
16
22.12.20
Safaa Alnabulsi,TU Berlin

17. THE RESULTS
▪ we find that MUTAN fusion is the most appropriate one for our weakly supervised
multi-modal dense event captioning task.
▪ The multi-modal approach (both MFCC and SoundNet audio with C3D video
features) outperforms state-of-the-art unimodal method.
▪ The multi-modal approaches outperform unimodal ones, both on caption quality
and temporal segment accuracy.
17
22.12.20
Safaa Alnabulsi,TU Berlin

18. EXAMPLE
18
22.12.20
Safaa Alnabulsi,TU Berlin

19. THE LIMITATIONS
▪ Sometimes the multi-modal model can not detect the beginning of an event
correctly.
▪ Most of the time the final model only generates around 2 event captions
which means that the multi-modal approach is still not good enough to detect all the
events in the weakly supervised setting.
19
22.12.20
Safaa Alnabulsi,TU Berlin

20. THANK
YOU FOR
YOUR
ATTENTIO
N
QUESTIO
NS? 20
Safaa Alnabulsi,TU Berlin 22.12.20