4.6.16 AI&BigData Lab Upcoming events: goo.gl/I2gJ4H Это — рекомендательная система. Если взглянуть на нее со стороны, то она крепко застряла между Collaborative filtering и Content-based filtering. Используются рекомендательные системы уже давно, но рекомендации все еще не идеальны. Обычно проблемы — это выбор технологий или там фреймворка… А у нас — cold-start problem, semantic gap и др.!

1. Igor Kostiuk | 2016
Tags: #music, #recommender_systems, #deep_learning, #neural_networks, #mel_spectrograms
How to train your music recommender system

2. Recommender systems are a family of methods that seek to
predict the rating or preference that a user would give to an item
© Wiki
Is there something similar to something else?
There are two common ways to make recommendations.

3. Collaborative filtering
- cold start problem (requires a large amount of information on a
user in order to make accurate recommendations)
- will not recommend rare or new songs, games, etc. (popular
items will be much easier to recommend than unpopular items)
- bad scalability
+ content-agnostic
Example: Last.fm recommends music based on a comparison of
the listening habits of similar users.

4. http://ru.anime-characters-fight.wikia.com/wiki/%D0%A4%D0%B0%D0%B9%D0%BB:Anime-heroes-wallpaper-hd-cool-7.jpg
Popularity

5. Content-based filtering
- can only make recommendations that are similar to the original
seed
- semantic gap between audio or video, and the various aspects
of music / movie that affect user preferences (genre, mood)
- obvious recommendations ( Doom Doom 4 etc. )→
http://static.giantbomb.com/uploads/original/13/137381/2846580-doom.jpg

6. There is nothing more similar to the tea kettle than the other tea kettle

7. Approaches
1. Automatic generation of social tags
Social tags are user-generated keywords associated with song.
Predicting these social tags directly from MP3 files avoids the ''cold-
start problem''.
Using a set of one vs all classifiers for every tag, we can map audio
features onto social tags collected from the Web.
2. Music genre classification
Attempt to classify songs into a set of genre classes. Clustering – each
cluster represents a specific genre.
Setting label to each cluster by choosing the “majority vote” - which
genre was the most common in that cluster.
https://en.wikipedia.org/wiki/Mel-frequency_cepstrum

8. Deep Learning approach
Predicting listening preferences from audio signals by training a
regression model to predict the latent representations of songs
that were obtained from a collaborative filtering model.
Data
from a collaborative filtering model
Data
raw mp3
Latent factors vector extracting
matrix factorization
Mel-spectrograms extracting
Deep neural network
input output
prediction

9. Advantages
+ Effectiveness in recommending new and unpopular songs
+ Good recommendations despite the semantic gap

10. Development stages
Data retrieval
The Echo Nest Taste Profile Subset
http://labrosa.ee.columbia.edu/millionsong/tasteprofile
b80344d063b5ccb3212f76538f3d9e43d87dca9e SOBSUJE12A6D4F8CF5 2
b80344d063b5ccb3212f76538f3d9e43d87dca9e SOBVFZR12A6D4F8AE3 1
b80344d063b5ccb3212f76538f3d9e43d87dca9e SOBXALG12A8C13C108 1
b80344d063b5ccb3212f76538f3d9e43d87dca9e SOBXHDL12A81C204C0 1
b80344d063b5ccb3212f76538f3d9e43d87dca9e SOBYHAJ12A6701BF1D 1
b80344d063b5ccb3212f76538f3d9e43d87dca9e SOCNMUH12A6D4F6E6D 1
b80344d063b5ccb3212f76538f3d9e43d87dca9e SODACBL12A8C13C273 1
b80344d063b5ccb3212f76538f3d9e43d87dca9e SODDNQT12A6D4F5F7E 5
Taste Profile subset is big. Some numbers:
1,019,318 unique users
384,546 unique MSD songs
48,373,586 user - song - play count triplets

11. Data retrieval
https://www.7digital.com/
We are able to attain 29 second audio clips for over 99% of the
dataset.
Original dataset has no raw audio, only precomputed, badly
documented features.

12. Weighted matrix factorization
https://youtu.be/o8PiWO8C3zs
song_id
user_id
song_id
user_id

13. Weighted matrix factorization
n songs
m users ≈ *
musers
f
f
n songs
R P
Q
R – rating matrix m*n
P – user matrix m*f
Q – song matrix f*n
f – number of features

15. Weighted matrix factorization
Alternating Least Squares
http://mendeley.github.io/mrec/
https://github.com/benanne/wmf
https://github.com/benanne/theano_wmf

16. Weighted matrix factorization
iteration
error

17. http://benanne.github.io/2014/08/05/spotify-cnns.html

18. Mel-spectrograms
A mel-spectrograms is a kind of time-frequency representation.
It is obtained from an audio signal by computing the Fourier
transforms of short, overlapping windows.
Finally, the frequency axis is changed from a linear scale to a mel
scale.
https://en.wikipedia.org/wiki/Mel_scale

19. Mel-spectrograms
series = np.sin(time)
# filename = "The Prodigy - Invaders Must Die.mp3"
# filename = "Lady GaGa - Poker Face.mp3"

22. Mel-spectrograms
Used log-compressed mel-spectrograms with 128 components
and the window size and hop size 1024 and 512 audio frames
respectively.
https://github.com/librosa/librosa
http://librosa.github.io/librosa/generated/librosa.feature.melspectrogram.h
tml#librosa.feature.melspectrogram

23. 1024 1024 * 2
1024 * 4
Mel-spectrograms

24. Convolutional neural network
The deep neural network baseline architecture could be
consisted of two convolutional layers and two fully connected
layers.
http://benanne.github.io/2014/08/05/spotify-cnns.html

25. Convolutional neural network
http://benanne.github.io/2014/08/05/spotify-cnns.html

26. Convolutional neural network
The network can be trained on windows of 3 seconds sampled
randomly from the audio clips.
The last layer of the network is the output layer, which predicts
40 latent factors obtained from the collaborative filtering.

27. http://www.slideshare.net/erikbern/music-recommendations-mlconf-2014
Album cover based models

28. 1) series = (np.sin(time) - np.sin(time / np.pi))
https://www.google.com.ua/#q=y+%3D+sin%28x%29+-+sin%28x+%2F+pi%29
2) Deep content-based music recommendation
http://papers.nips.cc/paper/5004-deep-content-based-music-recommendation.pdf
3) Collaborative Filtering for Implicit Feedback Datasets
http://yifanhu.net/PUB/cf.pdf
4) Alternating Least Squares Method for Collaborative Filtering
http://bugra.github.io/work/notes/2014-04-19/alternating-least-squares-method-for-col
laborative-filtering/
5) Recommending music on Spotify with deep learning
http://benanne.github.io/2014/08/05/spotify-cnns.html
6) *
http://papers.nips.cc/paper/3370-automatic-generation-of-social-tags-for-music-recommendati
on.pdf
http://cs229.stanford.edu/proj2013/FauciCastSchulze-MusicGenreClassification.pdf
http://ismir2011.ismir.net/papers/PS6-10.pdf
http://erikbern.com/2013/12/20/more-insight-into-recommender-algorithms/
http://www.slideshare.net/irecsys/matrix-factorization-in-recommender-systems

29. Let’s stay in touch:
Facebook
https://www.facebook.com/neverdraw
LinkedIn
https://www.linkedin.com/in/awesomengineer
Github
https://github.com/spaceuniverse

30. Thanks
http://cdn.gymnasticstracks.com/wp-content/uploads/2015/09/httyd.jpg