We present a way to incorporate human knowledge in our acoustic model of music. We train a deep neural network to classify songs into playlists, and keep the generated embedding space to recommend songs and generate personalized radios. Try it for yourself : http://scarlett.fm

1. Deep learning for music
recommendation and
personalized radio stations
Aloïs GRUSON
niland.io @nilandmusic

2. Can we recommend music with a pure
content-based approach ?
Question

3. Content based music recommendation
?
Embedding space
Audio music signal Processing and
modeling
Close in embedding space ó can be recommended together

4. Evaluation metrics
§ One of our metrics : Precision @50 on a
dataset of scrapped playlists of 8083 tracks
classified in 142 playlists.
§ Perceptive evaluations with real users
showed correlation between this metric and
the users average rating

5. Our results at niland.io
0
2
4
6
8
10
12
14
16
18
2011 2012 2013 2014 2015 2016
precision@50
Classic Approaches Deep Learning
Mirex 2011
Ranked 1st
Submission
+ 66.8%
relative improvement

6. Audio
MFCC
SFM
OC
GMM-SV
GMM-SV
GMM-SV
Spectrogram
Res GMM-SV
0
5
10
15
20
2011
2012
2013
2014
2015
2016
precision@50
MIREX 2011 Ranked 1st Submission
2011 4000

7. 0
5
10
15
20
2011
2012
2013
2014
2015
2016
precision@50
Audio
MFCC
SFM
OC
GMM-SV
GMM-SV
GMM-SV
Spectrogram
Res GMM-SV
Gabor GMM-SV
HoG GMM-SV
Work on more descriptors
2014 9000

8. “Bridge the semantic gap” ?
• We worked to bring the human perception of similarity into
our model
• We train deep neural networks to classify songs into
playlists.
• And we remove the classification layer to get our embedding
space
• Our training set : 115.412 tracks in 3032 playlists

9. 0
5
10
15
20
2011
2012
2013
2014
2015
2016
precision@50
Audio
MFCC
SFM
OC
GMM-SV
GMM-SV
GMM-SV
Spectrogram
Res GMM-SV
Gabor GMM-SV
HoG GMM-SV
Bending the space
2015
DNN
1000
9000

10. 0
5
10
15
20
2011
2012
2013
2014
2015
2016
precision@50
Convolutional Net
2016
Audio Spectrogram
Learning the low-level features
1000

11. An example of CNN structure
• 1D Convolutions
• Global Temporal Pooling Layer : Mean + Max + Variance
• 2 fully connected layers + classification layer
• Residual Connections

12. An example of CNN structure
Global Temporal Pooling Layer : Mean + Max + Variance
• Allows to process variable length tracks
• Generate some temporal invariance

13. An example of CNN structure
Our best system has :
• 1 Frequency Convolution layer
• 15 residual blocks, with 5 convolution layers in each
• A global pooling layer : Mean + Max + Variance
• 2 fully connected layers (2000 + 1000)

14. How to generate personalized radio stations ?
We have this embeddingspace, and we can recommend tracks for
a given track.
How do we create a personalized radio station
for an user ?

15. Let you discover music you like
Understand your various tastes

16. What do you want to listen to right now ?
Fast convergence into the wanted music style

17. Scarlett.fm : our streaming app
http://scarlett.fm
1M tracks
from soundcloud.com
Pure content-based
recommendations

18. Conclusion
• A very effective way to incorporate human knowledge into
an acoustic model
• What’s next ?
Ø Generating music
Ø Using raw audio as an input
Ø More diversity/risk in radio stations