This document summarizes an approach for no-audio multimodal speech detection using CNNs and Fisher Vectors. Dense trajectories and Fisher Vectors were used to extract features from video, with 1-second windows labeled as speaking or not. A 1D CNN was used on acceleration data with 3-second windows also labeled at 1 second intervals. Preliminary experiments showed the CNN outperformed logistic regression and was competitive with LSTMs. While the CNN benefited from more context, 1-second windows were too small for accurate video predictions. Opportunities for future work included exploring optimal context sizes and detecting more subtle speech cues.

1. CNNs and Fisher Vectors for No-
Audio Multimodal Speech Detection
Jose Vargas, Hayley Hung
Delft University of Technology
MediaEval 2019
10th Anniversary Workshop
27-29 October 2019
EURECOM, Sophia Antipolis, France

2. Approach
Video
Dense Trajectories + Fisher Vectors
1s window -> label 1s
Why?
- Preliminary experiments using 3s
showed consistently better
performance vs MILES
- Results showed robustness
against different orientations
- Simple
Acceleration
1-D Convolutional Neural Network
3s window -> label 1s
Why?
- Logistic Regression showed strong
performance in this task.
- CNNs can take a larger context
- Preliminary experiments showed
competitive performance vs LSTMs
Chen, Y., Bi, J., & Wang, J. Z. (2006). MILES: Multiple-instance learning via embedded instance selection. IEEE Transactions on
Pattern Analysis and Machine Intelligence. 2

3. Video: dense trajectories
Trajectory extraction:
1. Sample interest points in different scales.
2. Track each point using optical flow in corresponding scale.
3. Describe the trajectory via HOG, HOF, MBH.
Wang, H., Kläser, A., Schmid, C., & Liu, C. L. (2011). Action recognition by dense trajectories. Proceedings of the IEEE
Computer Society Conference on Computer Vision and Pattern Recognition.
3

4. Video: Fisher Vectors
Goal: summarize a set of local descriptors (eg. dense trajectories) of arbitrary
size into a single high-dimensional (feature) vector that describes a window.
GMM SVMBags of dense trajectories Fisher Vectors
i
i+1
i+2
for a GMM, ! can be: mean, variance = 2KD-
dimensional vector
K: dimensionality of GMM (256)
D: dimensionality of input (200)
2KD = 51200 4

5. Acceleration: 1-D CNN
Input: 3 axis of accelerometer for 3s windows.
Architecture:
- 5 conv. layers + 3 FC layers
- filter sizes inspired in AlexNet but along 1D.
5

6. Results & Conclusions
● Simple CNN models outperformed feature extraction (PSD features).
● Windows of 1s are too small for accurate video predictions.
● The CNN model benefits from a larger context.
6

7. Opportunities
Task-specific:
● Exploration of 1-D CNN architectures for time series.
● What is the optimal context size for this task?
● How to detect the more subtle cues?
General:
● Study the ground truth gap: perceived speaking vs. actual speaking.
● We should re-evaluate unsupervised domain adaptation for person specificity.
● How to incorporate a larger context into Fisher vector methods?
● How to interpret what a FV model has learned?
7

8. Questions?
MediaEval 2019
10th Anniversary Workshop
27-29 October 2019
EURECOM, Sophia Antipolis, France