In large musical catalogs such as in streaming companies, manual curation comes at high cost and the amount of data is considerable with tens of thousands of records delivered every week. While not replacing human resources, automatic systems trained directly from audio data help streaming companies describing audio recordings as well as creating relations between them. We will take a look at what is done by Deezer R&D's team in this domain using machine learning techniques.

1. Understanding music with machine learning
From signal processing to representation learning
Jimena Royo-Letelier
Research Scientist
R&D team
WiMLDS Paris
November 23th, 2017

2. 2
Deezer
● 180+ countries
● 40+ millions of track
● 16 millions active users
❖ 50+ curators
❖ Flow: continuous
recommender system
Data playground everywhere, many teams working on ML
> only 22% women in them

3. 3
Automatic catalog structuration
R&D mission
analyze, describe and categorize Deezer catalog

4. 4
An ever evolving, large catalog
● 40+ millions of track
● ~16 millions of artists
● 20k tracks ingested every day
● Audio Content
Music
roughly any commercial music ever recorded
any genre, period, quality, context
Speech
audiobooks / podcasts / radio shows / sport talks
● Usage
playlist
listening sessions
● Textual data
lyrics
biographies
reviews
● Images
album covers
artist photos
of heterogeneous multimodal data
Deezer

5. 5
described primarily by metadata
but with inherent to music indexation problems
● cultural ambiguities
● missing / wrong / ambiguous metadata
with hidden musical relations
● covers / versions / samples / remixes
An ever evolving, large catalog
Deezer

6. 6
R&D mission
analyze, describe and categorize Deezer catalog
(i.e.: a great part of music ever produced)
> automatic extraction of information & relations
> signal processing and machine learning (ML)
natural languages processing
image processing
audio signal processing
music information retrieval (MIR)
representation learning
Automatic catalog structuration

7. 7
Music Information Retrieval

8. 8
Music Information Retrieval
Main Idea
> Study of algorithms and system for extracting musical meaningful parameters from audio signals,
e.g.: beats, chords, structure, mood, etc.
Fourier analysis features engineering machine learning
audio
descriptors
(e.g.: chromagram)
raw audio
decision
system
(e.g.: SVM)
time-frequency
representation
(e.g.: spectrogram)
Principle

9. 9
Music Information Retrieval
Main Idea
> Study of algorithms and system for extracting auditory meaningful parameters from audio signals
Nowadays end-to-end trend
audio
descriptors
(e.g.: chromagram)
raw audio
time-frequency
representation
machine (deep) learning
decision
system

10. 10
Applications @ Deezer
multimodal (audio+lyrics) mood estimation
audio quality discrimination
Towards End-to-end Multimodal Music Mood Detection Based On Audio And Lyrics
R. Delbouys, et al. - Submitted
Music Information Retrieval

11. 11
Representational Learning

12. 12
Representation Learning
Main Idea
> Create a representation space (embedding) by directly learning a parametric map
from input to representation
> Learn a low dimensional space that represents high-level characteristics of audio
content in which proximity may be interpreted as audio similarity.
Principle
map f
(e.g.: convolutional/recurrent neural network)
x = low level feature embedding f(x)
R^d, d small

13. 13
Application @ Deezer
> Artist * Disambiguation
Representation Learning
○ homonym artists (collision)
(same name, different artist)
○ artists unclear naming
(different name, same artist)
Bills Evans, jazz pianist
Bills Evans, jazz saxophonist
Inti-Illimani vs Inti Illimani
Youssou N’Dour vs Youssou Ndour
Cat Stevens vs Yusuf Islam
* not a completely well-posed notion
Metric learning for music artist disambiguation from audio
J. Royo-Letelier, et al. - Submitted

14. 14
Triplet loss mechanism
L(X) =
with triplet X = (x*, x+, x-).
x* x*x+ x-
same artist
different artist from first two
Sample triplets X and leave out
not “hard” triplets: L(X) = 0
hard not-hard
Metric Learning
x = spectrogram f = convolutional neural network f(x)

15. 15
“Do or Die” homonyme artists
● Embedding system learn how to discriminate artists.
● Artists characteristics learned by the system
generalize to data (tracks/albums/artist)
not seen during learning phase.
Qualitative results
Album level generalisation
Artist level generalisation

16. 16
Take Away
❖ Large multimodal music related data (audio/text /images/usage)
❖ Great challenges in automatically understanding music
➢ Big scalability constraints
➢ Big variability constraints
➢ Hidden relations
➢ Possible ambiguities
❖ ML and MIR useful for many applications concerning music indexation and characterisation
❖ ML system are able to learn high-level characteristics of musical audio content
such artist membership
❖ Too bad no more women tackling these challenges !
> https://wimir.wordpress.com/mentoring-program/

17. 17
Thanks !