AcousticBrainz Genre Task
Content-based music genre recognition
from multiple sources
Dmitry Bogdanov, Alastair Porter (Universitat Pompeu Fabra)
Julián Urbano (Delft University of Technology)
Hendrik Schreiber (tagtraum industries incorporated)

Genre recognition in Music Information Retrieval
● A popular task in MIR (Sturm 2014)
● Only small number of broad genres (e.g., rock, jazz, classical, electronic)
● Almost no studies on more specific genres (subgenres)
● Studies don’t consider the subjective nature of genre labels and taxonomies
● Single-class classification problem instead of a multi-class problem
● Genre hierarchy is not exploited
● Small datasets
B. L. Sturm. 2014. The State of the Art Ten Years After a State of the Art: Future Research in Music Information Retrieval. Journal of New Music Research 43, 2 (2014), 147–172.

AcousticBrainz
AcousticBrainz: a community database containing music features extracted from
audio (https://acousticbrainz.org) (Porter et al. 2015)
● Open data computed by open algorithms
● Built on submissions from the community
● Over 10,000,000 analyzed recordings (tracks)
● ~3,000 music features (bags-of-frames)
● Statistical information about spectral shape, rhythm, tonality, loudness, etc.
● Rich music metadata from MusicBrainz (https://musicbrainz.org)
● Lots of data... What can do with it?
A Porter, D Bogdanov, R Kaye, R Tsukanov, and X Serra. 2015. AcousticBrainz: a community platform for gathering music information obtained from audio. In International Society for Music
Information Retrieval (ISMIR’15) Conference. Málaga, Spain, 786–792.

The 2018 AcousticBrainz MediaEval Task
Content-based music genre recognition from multiple ground truth sources
Goal: Predict genre and subgenre of unknown music recordings given
precomputed music features
Task novelty:
● Four different genre annotation sources (and taxonomies)
● Hundreds of specific subgenres
● Multi-label genre classification problem
● A very large dataset (~2 million recordings in total)

Sources of genre information
● Scrape from internet sources
● Discogs (discogs.com) and AllMusic
(allmusic.com)
● Explicit genre and subgenre annotations
at an album level
● predefined taxonomies
● AcousticBrainz song → album → genre

Sources of genre information
● Tagtraum dataset based on beaTunes
○ Consumer application for Windows and Mac by
tagtraum industries incorporated
○ Encourages users to correct metadata
○ Collects anonymized, user-submitted metadata
○ Relationship Song:Genre is 1:n
● Last.fm
○ Folksonomy tags for each song
○ Relative strength (0-100)
● Tag cleaning (normalization and blacklisting)
● Automatic inference of genre-subgenre relations

What does the data look like?
https://www.youtube.com/watch?v=zlaz7aR7B44

Subjectivity in music genre
● Classification tasks typically rely on an agreed answer for ground truth
● What should we do if we can’t find agreement between our ground truth?
● What if different sources use a label, but source has a different definition?
Reggae → Dub Electronic → Ambient Dub
Electronic → World Fusion
World, Dub, Fusion

Sub-tasks
● Task 1: Build a separate system for each ground-truth dataset
● Task 2: Can we benefit from combining different ground truths into one
system?
Task 1 Task 2

Development and testing dataset split
● 4 development, testing and validation datasets (70%-15%-15%-split)
● Album filter
● Each label has at least 40 recordings from 6 release groups in training
dataset (20 from 3 for test dataset)
● Development datasets statistics:

Task History
Baselines
● Random baseline: following the distribution of labels (4 runs)
● Popularity baseline: always predicts the most popular genre (4 runs)
Submissions in 2017
● Approaches by 5 teams (111 runs)
● NNs, SVM + extra trees, Random Forest)
Submissions in 2018: deep learning baselines
● The test set for 2017 is published as a validation set for 2018
● Melbands baseline (8 runs)
● Fusion baseline (8 runs)
A total of 135 runs over both editions

Evaluation metrics
Effectiveness: Precision, Recall and F-score
● Per recording, all labels (genres and subgenres)
● Per recording, only genres
● Per recording, only subgenres
● Per label, all recordings
● Per genre label, all recordings
● Per subgenre label, all recordings

Confusion Matrices - Genres
Random JKU

Confusion Matrices - Subgenres
Random JKU

Results on genres vs subgenres - 2017 test set

Hierarchy Expansion?
We asked for explicit labeling
If a subgenre is predicted, the corresponding genre must be explicitly predicted
There are arguments for and against this
We can also compute evaluation scores after expanding, considering implicit
prediction of genres
Note: per-subgenre results do not change

Results on genres - 2017 test set

Conclusions: The Task is Challenging!
● Subgenre recognition is much more difficult - much space to improve!
● Datasets are heavily unbalanced
● High recall, but poor precision for many systems
● AllMusic dataset is the most difficult
● No significant improvement from exploiting hierarchies more
● No significant overall improvement from combining genre sources yet
Team results
● JKU (2017) is still the best system across all datasets
● New baselines are comparable to the DBIS (2017) that exploits hierarchies
and is significantly better than baselines

Future directions
● AcousticBrainz is an ongoing experiment in collaborative extraction of music
knowledge from audio
● Integrate promising systems to AcousticBrainz
The future of the task
● Low interest in the task, how to improve?
● Task overview paper for ISMIR 2019
● Frame data, updated descriptors on AcousticBrainz
● New datasets?
● New multi-source evaluation approach with hierarchical models

Thank you!