The 2018 Recommending Movies
Using Content: Which content is key?
MediaEval Multimedia Evaluation Workshop, 29-31 October 2018, Sophia Antipolis, France
- Task overview -
Yashar Deldjoo (University of Milano Bicocca - Italy)
Mihai Gabriel Constantin (Univ. of Politechnica of
Bucharest-Romania)
Athanasios Dritsas (TU Delft - Netherlands)
Bogdan Ionescu(Univ. of Politechnica of Bucharest - Romania)
Markus Schedl (Johannes Kepler University, Linz - Austria)

Why Video Recommendation ?
2
YouTube:
● YouTube: 1.3 Billion videos
● YouTube: 400 hours of videos
uploaded every minute
= 3-year of watching to see all the
videos uploaded per hour
● Netflix: 7K movies/TV shows
● Instagram: 70M videos/photos per
day
Video overload
Finding novel and relevant video is hard !

Recommender Systems
3

Recommender Systems
4
output = , Rating = ?
input =

5
Signal
Expert-Generated
Content (EGC)
User-Generated
Content (UGC) Signal: audio, visual
EGC: genre, style, mood
UGC: tags, reviews
Different Notions of Content in RecSys and
Multimedia

Different Notions of Content in RecSys and
Multimedia
6
Signal
Expert-Generated
Content (EGC)
User-Generated
Content (UGC)
From inner to outer circle:
Non-semantic → semantic
Objective → subjective
No Cold-start → Cold-start
Credible → Error-prone

The Mission of the MMRecSys Task
7
➔ To bridge the gap in advances between the two communities.
➔ Achieved by integrating audiovisual content features in
content-based RecSys models.
RecSys Multimedia

Goal of the Proposed ME Task
To predict average and standard
deviation of ratings through the
audio visual signals of movies
● Average ratings: represents users’
overall appreciation/disappreciation
● Std. of ratings: represents users’
agreements/disagreements
8
mean: 4.5 1 3 3.5
std: 0.7 0 0 2.1
Measure of
personalization

Dataset: General Information and Statistics
9
MovieLens dataset (ML-20m)
● 20M ratings, 27K movies, 138K users
● 465K tags
Dataset 1: Movie Trailers 14K
● Published at MMSys18 → MMTF14K
Dataset2: Movie Clips 7K clips
● Introduced for MediaEval 2018
Deldjoo, Yashar; Constantin, Mihai Gabriel; Ionescu, Bogdan; Schedl, Markus;
Cremonesi, Paolo; MMTF-14K: A multifaceted movie trailer feature dataset for
recommendation and retrieval, Proceedings of the 9th ACM Multimedia Systems
Conference, 450-455, 2018, ACM

Dataset: provided features
10
Features are similar to MMTF14K dataset
Link to download: ydeldjoo.me >> Datasets
Deldjoo, Yashar; Constantin, Mihai Gabriel; Ionescu, Bogdan; Schedl, Markus;
Cremonesi, Paolo; MMTF-14K: A multifaceted movie trailer feature dataset for
recommendation and retrieval, Proceedings of the 9th ACM Multimedia Systems
Conference, 450-455, 2018, ACM

Dataset: provided descriptors
11
General purpose audio descriptors:
● Block-level features (BLF)
● Describes spectral aspects (3 descriptors), harmonic aspects
(1 descriptor), rhythmic aspects (1 descriptor), and tonal
aspects (1 descriptor) → 6 descriptors
State-of-the-art audio descriptors:
● i-vector
● Describe timbre include different
parameters for GMM and total variability
dimension (tvDim)

Dataset: provided descriptors
12
General purpose visual descriptors:
● Aesthetic visual features (AVF)
● Describes color, texture, object → 26 descriptors
State-of-the-art visual descriptors:
● Deep CNN
● Pretrained network, AlexNet
Metadata:
● Genre → editorial, binary vector of length 18
Tag → user-generated, tf-idf vector, length 13K +

Dataset: basic statistics (MovieClips Dataset)
13
Development set:
● Number of clips= 5562
● Number of movie= 637
● Number of clips/movie = 8.72
Test set:
● Number of clips= 1315
● Number of movie= 159
● Number of clips/movie = 8.27

Dataset: basic statistics (MovieClips Dataset)
14
Development set:
● Number of clips= 5562
● Number of movie= 637
● Number of clips/movie = 8.72
Test set:
● Number of clips= 1315
● Number of movie= 159
● Number of clips/movie = 8.27
For each movie,
there is more than
one associated clip.

Ground truth and Evaluation Metrics
15
Pred 1: Overall ratings
● User ratings are in the set: {0.5, 1, …, 4.5, 5}
● Overall ratings are continuous
Pred 2: Standard deviation of ratings
● Stds are continuous, the distribution is quite narrow
Metric: Root Mean Square Error (RMSE)

Task Participation
16
● 15 teams registered
● 5 teams obtained the data
● 1 organizer submission
○ Problem: we think the dataset was released quite late?
○ Plan to promote the task for the next year
○ Plan to define task in domains where visual and audio content
plays role in human decision making

Thank you
17
Questions?