This document summarizes a multimedia video classification approach using multiple modalities. It describes extracting audio features like spectral patterns and visual features like color histograms. Text is derived from automatic speech recognition and metadata. Combining audio, visual, and text features using machine learning classifiers achieves the best performance, with an average F-score of 68% for genre tagging using ASR and metadata. Increasing the number of modalities improves accuracy compared to single modality approaches.

1. ~ Multimodal Video Classification ~
ARF (Austria-Romania-France) team
Bogdan IONESCU*1,3 Ionuț MIRONICĂ1 Klaus SEYERLEHNER2
bionescu@imag.pub.ro imironica@imag.pub.ro music@cp.jku.at
Peter KNEES2 Jan SCHLÜTER4 Markus SCHEDL2
peter.knees@jku.at jan.schlueter@ofai.at markus.schedl@jku.at
Horia CUCU1 Andi BUZO1 Patrick LAMBERT3
horia.cucu@upb.ro andi.buzo@upb.ro patrick.lambert@univ-savoie.fr
*this work was partially supported under European Structural Funds EXCEL POSDRU/89/1.5/S/62557.
1 2 3 4
University Austrian Research
POLITEHNICA Institute for Artificial
of Bucharest Intelligence

2. Presentation outline
• The approach
• Video content description
• Experimental results
• Conclusions and future work
MediaEval - Pisa, Italy, 4-5 October 2012 1/16 2

3. The approach
> challenge: find a way to assign (genre) tags to unknown videos;
> approach: machine learning paradigm;
…
web food autos label data
train
unlabeled data
classifier labeled data
tagged video database
video database
MediaEval - Pisa, Italy, 4-5 October 2012 2/163

4. The approach: classification
> the entire process relies on the concept of “similarity” computed
between content annotations (numeric features),
> this year focus is on:
objective 1: go multimodal (truly)
visual audio text
objective 2: test a broad range of classifiers and descriptor
combinations;
MediaEval - Pisa, Italy, 4-5 October 2012 3/164

5. Video content description - audio
 block-level audio features • Spectral Pattern,
(capture also local temporal information) ~ soundtrack’s timbre;
• delta Spectral Pattern,
e.g. 50% overlapping
~ strength of onsets;
• variance delta Spectral Pattern,
average ~ variation of the onset strength;
median • Logarithmic Fluctuation Pattern,
variance ~ rhythmic aspects;
... • Correlation Pattern,
~ loudness changes;
• Spectral Contrast Pattern,
~ ”toneness”;
• Local Single Gaussian model,
[Klaus Seyerlehner et al., MIREX’11, USA] ~ timbral;
• George Tzanetakis model,
~ timbral;
MediaEval - Pisa, Italy, 4-5 October 2012 4/16
5

6. Video content description - audio
 standard audio features
(audio frame-based)
• Zero-Crossing Rate,
• Linear Predictive Coefficients,
time • Line Spectral Pairs,
• Mel-Frequency Cepstral Coefficients,
global
feature • spectral centroid, flux, rolloff, and
f1 f2 … fn
= kurtosis,
+ mean & + variance of each feature over
var{f2} var{fn} variance a certain window.
[B. Mathieu et al., Yaafe toolbox, ISMIR’10, Netherlands]
MediaEval - Pisa, Italy, 4-5 October 2012 5/16
6

7. Video content description - visual
 MPEG-7 & color/texture descriptors
(visual frame-based)
• Local Binary Pattern,
global • Autocorrelogram,
feature • Color Coherence Vector,
=
mean & • Color Layout Pattern,
dispersion & • Edge Histogram,
skewness &
time
kurtosis & • Classic color histogram,
f1 f2 … fn median &
• Scalable Color Descriptor,
root mean square
• Color moments.
[OpenCV toolbox, http://opencv.willowgarage.com]
MediaEval - Pisa, Italy, 4-5 October 2012 6/16
7

8. Video content description - visual
 feature descriptors
(visual frame-based)
• Histogram of oriented Gradients (HoG)
~ counts occurrences of gradient orientation
feature points (e.g. Harris)
in localized portions of an image (20º per bin)
• Harris corner detector
• Speeded Up Robust Feature (SURF)
image source http://www.ifp.illinois.edu/~yuhuang
[OpenCV toolbox, http://opencv.willowgarage.com]
MediaEval - Pisa, Italy, 4-5 October 2012 7/16
8

9. Video content description - text
 TF-IDF descriptors
(Term Frequency-Inverse Document Frequency)
> text sources: ASR and metadata,
1. remove XML markups,
2. remove terms <5%-percentile of the frequency distribution,
3. select term corpus: retaining for each genre class m terms (e.g. m =
150 for ASR and 20 for metadata) with the highest χ2 values that
occur more frequently than in complement classes,
4. for each document we represent the TF-IDF values.
MediaEval - Pisa, Italy, 4-5 October 2012 8/16
9

10. Experimental results: devset (5,127 seq.)
> classifiers from Weka (Bayes, lazy, functional, trees, etc.),
> cross-validation (train 50% – test 50%),
avg. Fscore (over all genres)
- visual descriptors capabilities 30%±10%,
- using more visual is not more accurate than using few,
- best LBP+CCV+histogram (Fscore=41.2%).
[Weka toolbox, http://www.cs.waikato.ac.nz/ml/weka/]
MediaEval - Pisa, Italy, 4-5 October 2012 9/1610

11. Experimental results: devset (5,127 seq.)
> cross-validation (train 50% – test 50%),
avg. Fscore (over all genres)
- audio still better than visual (improvement ~6%),
- proposed block-based better than standard (by ~10%),
[Weka toolbox, http://www.cs.waikato.ac.nz/ml/weka/]
MediaEval - Pisa, Italy, 4-5 October 2012 10/16
11

12. Experimental results: devset (5,127 seq.)
> cross-validation (train 50% – test 50%),
avg. Fscore (over all genres)
- ASR from LIMSI more representative than LIUM (~3%),
- best performance ASR LIMSI + metadata (Fscore=68%).
[Weka toolbox, http://www.cs.waikato.ac.nz/ml/weka/]
MediaEval - Pisa, Italy, 4-5 October 2012 11/16
12

13. Experimental results: devset (5,127 seq.)
> cross-validation (train 50% – test 50%),
avg. Fscore (over all genres)
- audio-visual close to text (ASR) for the automatic descriptors,
- increasing the number of modalities increases the performance.
[Weka toolbox, http://www.cs.waikato.ac.nz/ml/weka/]
MediaEval - Pisa, Italy, 4-5 October 2012 12/16
13

14. Experimental results: official runs (9,550 seq.)
> train on devset, test on testset (SVM linear),
MediaEval MediaEval
2011 2011
MAP 12% MAP 10.3%
Run1 Run2 Run3 Run4 Run5
LBP+CCV+ TF-IDF on audio block-based + audio TF-IDF on
hist + audio ASR LIMSI LBP + CCV + hist + block-based metadata +
metadata
block-based TF-IDF on ASR ASR LIMSI
LIMSI
MediaEval - Pisa, Italy, 4-5 October 2012 13/16
14

15. Experimental results: official runs (9,550 seq.)
> genre MAP for Run 5: TF-IDF on ASR + metadata,
Run 1: visual + audio
autos gaming religion environment
52% 71% 71% 50%
MediaEval - Pisa, Italy, 4-5 October 2012 14/16
15

16. Conclusions and future work
> classification adapts to the corpus – changing the corpus will
change the performance;
> audio-visual descriptors are inherently limited;
> how far can we go with ad-hoc classification without human
intervention?
> future work:
 more elaborated late-fusion ?
 pursue tests on the entire data set;
 perhaps more elaborated Bag-of-Visual-Words.
Acknowledgement: we would like to thank Prof. Fausto Giunchiglia and
Prof. Nicu Sebe from University of Trento for their support.
MediaEval - Pisa, Italy, 4-5 October 2012 15/16
16

17. thank you !
any questions ?
MediaEval - Pisa, Italy, 4-5 October 2012 16/16
17