MMCF: Multimodal Collaborative Filtering
for Automatic Playlist Continuation
Team ‘hello world!’ (2nd place), main track
Hojin Yang*, Yoonki Jeong, Minjin Choi, and Jongwuk Lee
Sungkyunkwan University, Republic of Korea

2
Motivation & Challenges

Automatic Playlist Continuation
➢Dataset: Million Playlist Dataset (MPD)
3
Playlist title
Tracks
in the
playlist
Metadata of tracks
(artist, album)

Challenge Set
4
1 2 3 4 5 6 7 8 9 10
# of tracks 0 1 5 10 5 10 25 100 25 100
Title
available
Yes Yes Yes Yes No No Yes Yes Yes Yes
Track order Seq Seq Seq Seq Seq Seq Seq Seq Shuffled Shuffled
# of playlists 1,000 1,000 1,000 1,000 1,000 1,000 1,000 1,000 1,000 1,000
Few tracks in the
first part
Many tracks in the
first part
Many tracks in the
random position

Challenge Set
5
1 2 3 4 5 6 7 8 9 10
# of tracks 0 1 5 10 5 10 25 100 25 100
Title
available
Yes Yes Yes Yes No No Yes Yes Yes Yes
Track order Seq Seq Seq Seq Seq Seq Seq Seq Shuffled Shuffled
# of playlists 1,000 1,000 1,000 1,000 1,000 1,000 1,000 1,000 1,000 1,000
No tracks in the playlist
How to deal with an edge case?

Challenge Set
6
1 2 3 4 5 6 7 8 9 10
# of tracks 0 1 5 10 5 10 25 100 25 100
Title
available
Yes Yes Yes Yes No No Yes Yes Yes Yes
Track order Seq Seq Seq Seq Seq Seq Seq Seq Shuffled Shuffled
# of playlists 1,000 1,000 1,000 1,000 1,000 1,000 1,000 1,000 1,000 1,000
Scarce information
How to treat playlists with scarce
information?

Challenge Set
7
1 2 3 4 5 6 7 8 9 10
# of tracks 0 1 5 10 5 10 25 100 25 100
Title
available
Yes Yes Yes Yes No No Yes Yes Yes Yes
Track order Seq Seq Seq Seq Seq Seq Seq Seq Shuffled Shuffled
# of playlists 1,000 1,000 1,000 1,000 1,000 1,000 1,000 1,000 1,000 1,000
Various types of Input
How to deal with various types of
input?

8
Proposed Model

Overview of the Proposed Model
➢An ensemble method with two components.
◆ Autoencoder for tracks and metadata for tracks.
◆ CharCNN for playlist titles.
9

Overview of the Proposed Model
➢An ensemble method with two components
◆ Autoencoder for tracks and metadata for tracks
◆ CharCNN for playlist titles
10

Autoencoder
11
encoder decoder
Compressed representation
of the input image
28X28X3 pixels
➢Learn a latent representation by reconstructing the
input itself
28X28X3 pixels

Collaborative Autoencoder
12
1
0
1
1
0
1Hey Jude
Rehab
Yesterday
Dancing Queen
Mamma Mia
Viva la Vida
encoder decoder
➢Learn a latent representation of a given playlist
consisting of a set of tracks
0.9
0.01
0.78
0.9
0. 6
0.8
✔
Top-1
Recommendation
Output

Collaborative Autoencoder
13
1
0
1
1
0
1
1
0
1
0
0
0
0.9
0.01
0.78
0.9
0. 6
0.8Hey Jude
Rehab
Yesterday
Dancing Queen
Mamma Mia
Viva la Vida
dropout
encoder decoder
1
0
1
1
0
1
➢Training with dropout
◆ Some positive input values are corrupted (set to zero).

Collaborative Autoencoder
14
➢Training with dropout
◆ Some positive input values are corrupted (set to zero).
How to utilize the metadata such
as artists and albums?
1
0
1
1
0
1
1
0
1
0
0
0
0.9
0.01
0.78
0.9
0. 6
0.8Hey Jude
Rehab
Yesterday
Dancing Queen
Mamma Mia
Viva la Vida
dropout
encoder decoder
1
0
1
1
0
1

Utilizing Metadata
15
1
0
1
1
0
1
0
1
1
0
0.9
0.01
0.78
0.9
0. 6
0.8
0.2
0.98
0.9
0.6
Hey Jude
Rehab
Yesterday
Dancing Queen
Mamma Mia
Viva la Vida
encoder decoder
➢Simply concatenate an artist vector corresponding
to the track vector.

➢Randomly choose either the playlist or its artists as
input.
16
Training Strategy: Hide-and-Seek

Training Strategy: Hide-and-Seek
➢Randomly choose either the playlist or its artists as
input.
17

Training Strategy: Hide-and-Seek
➢Randomly choose either the playlist or its artists as
input.
18

➢About 0.5~9.0% improvement over the baseline
19
sequential shuffled
Input 1 5 10 25 100 25 100
Track 0.111 0.153 0.183 0.215 0.159 0.306 0.301
Track + Artist 0.121 0.156 0.184 0.216 0.172 0.317 0.303
Gain(%) +9.00 +1.96 +0.55 +0.47 +8.18 +3.60 +0.66
* R-precision
Training Strategy: Hide-and-Seek

➢How to design a dropout strategy?
20
1
0
1
1
0
1
0
1
1
0
1
0
1
0
0
0
0
0
0
0
0.9
0.23
0.78
0.1
0.1
0.8
0.2
0.98
0.9
0.1
Hey Jude
Rehab
Yesterday
Dancing Queen
Mamma Mia
Viva la Vida
dropout
encoder decoder
1
0
1
1
0
1
0
1
1
0
Training Strategy: Dropout Scheme
Hide

Training Strategy: Dropout Scheme
➢Dropout ratios vary in the input size.
21
1 2 3 4 5 6 7 8 9 10
# of tracks 0 1 5 10 5 10 25 100 25 100
Title
available
Yes Yes Yes Yes No No Yes Yes Yes Yes
Track order Seq Seq Seq Seq Seq Seq Seq Seq Shuffled Shuffled
# of playlists 1,000 1,000 1,000 1,000 1,000 1,000 1,000 1,000 1,000 1,000
Dropout a lot
Dropout a little

➢Dropout can be sequential or random.
22
1 2 3 4 5 6 7 8 9 10
# of tracks 0 1 5 10 5 10 25 100 25 100
Title
available
Yes Yes Yes Yes No No Yes Yes Yes Yes
Track order Seq Seq Seq Seq Seq Seq Seq Seq Shuffled Shuffled
# of playlists 1,000 1,000 1,000 1,000 1,000 1,000 1,000 1,000 1,000 1,000
Dropout
the back part
Dropout
randomly
Training Strategy: Dropout Scheme

CharCNN for Playlist Titles
➢An ensemble method with two components
◆ Autoencoder for tracks and metadata for tracks
◆ CharCNN for playlist titles
23

Character-level CNN for NLP
➢Effective for capturing spatial locality of a
sequence of texts
24
I like this
song
very
much
0.1 0.3 0.2 0.6
0.2 0.6 -1.2 -0.2
-2.1 0.2 0.1 0.4
-2.1 0.9 -3.1 1.4
0.1 0.3 -0.2 0.1
0.4 0.1 0.7 0.1
I
like
this
song
very
Filter (3 by k )
2.2
2.3
-1.3
0.9
max
pooling
Conv layer
2.3
Feature
much
k-dimension embedding
convolutional

Character-level CNN for NLP
➢Effective for capturing spatial locality of a
sequence of texts
25
I like this
song
very
much
0.1 0.3 0.2 0.6
0.2 0.6 -1.2 -0.2
-2.1 0.2 0.1 0.4
-2.1 0.9 -3.1 1.4
0.1 0.3 -0.2 0.1
0.4 0.1 0.7 0.1
I
like
this
song
very
Filters (3 by k )
convolutional
2.2
2.3
-1.3
0.9
max
pooling
2.3
Feature
much
k-dimension embedding
Conv layer
2.2
2.3
-1.3
0.9
Conv layers
2.2
2.3
-1.3
0.9
1.2
2.4
-1.1
0.4
max
pooling
2.3
1.2
2.4
Feature vector
convolutional

CharCNN for Playlist Titles
➢The playlist title is represented by a short text, implying
the abstract description of a playlist.
➢Leverage character-level embedding instead of
word-level embedding.
26
Conv layers
Feature vector

Combining Two Models
➢Simplest method: 𝑤𝒊𝒕𝒆𝒎 = 0.5 and 𝑤𝒕𝒊𝒕𝒍𝒆 = 0.5
27

Combining Two Models
➢The accuracy of the AE relies on the number of
tracks in a playlist.
◆ Dynamic: Set weights according to the number of items.
28
Items
Playlist Title
Chill songs
0.7
0.4
0.9
0.1
0.2
0.1
0.2
0.3
0.7
0.1
0.6
0.4
0.7
0.2
0.2
AE
CNN
𝑤_𝑖𝑡𝑒𝑚 = 5
𝑤_𝑡𝑖𝑡𝑙𝑒 = 1

Combining Two Models
➢The weight of two models is combined
dynamically depending on the size of input.
➢The playlist title is primarily useful for playlists with
very few tracks.
29
sequential shuffled
Title leverage 0 1 5 10 25 100 25 100
Item only - 0.121 0.156 0.184 0.216 0.172 0.317 0.303
Title only 0.078 - - - - - - -
Constant(0.5) 0.078 0.131 0.158 0.181 0.211 0.161 0.310 0.292
Dynamic 0.078 0.131 0.158 0.184 0.216 0.172 0.317 0.303
Gain(%) - - - +1.6 +2.3 +6.8 +2.2 +3.7
* R-precision

Q&A
30
Thank you!
Hojin Yang
- undergraduate student, SKKU
- code: github.com/hojinYang/spotify_recSys_challenge_2018
- email: hojin.yang7@gmail.com