Annotating Music Collections: How Content-Based Similarity Helps to Propagate Labels Mohamed Sordo, Cyril Laurier and Òscar Celma (Music Technology Group, UPF)

Mohamed Sordo, Cyril Laurier and Òscar Celma

(Music Technology Group, UPF)

Outline Introduction Motivation and goals Evaluation Demonstration Conclusions

Introduction

Motivation and goals

Evaluation

Demonstration

Conclusions

introduction Massive Increase in volume of online music Long Tail Economics Make everything available Help me find it

Massive Increase in volume of online music

Long Tail Economics

Make everything available

Help me find it

introduction:: last year's work... Searchsounds (ISMIR, 2006) Exploits MP3-blogs info Natural Lang. Processing Audio similarity (“more like this”)

Searchsounds (ISMIR, 2006)

Exploits MP3-blogs info

Natural Lang. Processing

Audio similarity (“more like this”)

motivation:: annotating music collections Tag sparsity Postal Service - “Such Great Heights” 2.4 million num. plays 100s of tags Mike Shupps's - “All Over Town” 3 num. plays 0 tags Cold start problem for new songs

Tag sparsity

Postal Service - “Such Great Heights”

2.4 million num. plays

100s of tags

Mike Shupps's - “All Over Town”

3 num. plays

0 tags

Cold start problem for new songs

goals Ease the process of annotating large music collections No learning from audio to tags, but... Content-based similarity to propose tags among songs Timbre, rhythm, tonality, etc.

Ease the process of annotating large music collections

No learning from audio to tags, but...

Content-based similarity to propose tags among songs

Timbre, rhythm, tonality, etc.

our proposal Annotated

our proposal “wisdom of (song) crowds” FOR FREE!!! Annotated

“wisdom of (song) crowds”

our proposal evaluation...? “wisdom of crowds” & feedback FOR FREE!!! Annotated

evaluation...? “wisdom of crowds” & feedback

our proposal Relevance feedback from the users Spencer Tunick's “wisdom of crowds” http://www.flickr.com/photos/guacamoleproject/487376758/

Relevance feedback from the users

Spencer Tunick's “wisdom of crowds”

our proposal new song, no tags rock Indie Cute guitar Drums 90s Fast Weird twee Quirky noise pop Cute playful Drums 80s Fast Weird Sweet Quirky Cute rock pop 90s Fun metal rock Edgy guitar thrash 90s Fierce Weird concert Loud ??? ??? ??? ??? ??? ??? thrash loud metal death thrash rock Edgy gothic heavy metal 90s Weird concert Loud

new song, no tags

our proposal propose tags (“wisdom of song crowds”) rock Indie Cute guitar Drums 90s Fast Weird twee Quirky noise pop Cute playful Drums 80s Fast Weird Sweet Quirky Cute rock pop 90s Fun metal rock Edgy guitar thrash 90s Fierce Weird concert Loud thrash loud metal death thrash rock Edgy gothic heavy metal 90s Weird concert Loud thrash rock 90s Loud metal Weird

propose tags (“wisdom of song crowds”)

our proposal users validate/add tags rock Indie Cute guitar Drums 90s Fast Weird twee Quirky noise pop Cute playful Drums 80s Fast Weird Sweet Quirky Cute rock pop 90s Fun metal rock Edgy guitar thrash 90s Fierce Weird concert Loud thrash loud metal death thrash rock Edgy gothic heavy metal 90s Weird concert Loud thrash rock 90s Loud metal Weird

users validate/add tags

experiments Ground Truth 5481 annotated songs from Magnatune (John Buckman) 29 Different labels (rock, instrumental, classical, relaxing, etc.) Avg. 3 tags per song Only CB similarity, no user feedback 2 experiments All tags Only mood tags (smaller collection)

Ground Truth

5481 annotated songs from Magnatune (John Buckman)

29 Different labels (rock, instrumental, classical, relaxing, etc.)

Avg. 3 tags per song

Only CB similarity, no user feedback

2 experiments

All tags

Only mood tags (smaller collection)

experiments:: results Some basic comments... Increasing number of similar songs (10, 20, 30,...) Bad recall, precision Best results: ~10 similars Using album, artist filtering Need to increase the number of similar songs Best results: ~20 similars Metrics Precision / Recall (F-measure, alpha = 2) Spearman Rho (Details on the paper!)

Some basic comments...

Increasing number of similar songs (10, 20, 30,...)

Bad recall, precision

Best results: ~10 similars

Using album, artist filtering

Need to increase the number of similar songs

Best results: ~20 similars

Metrics

Precision / Recall (F-measure, alpha = 2)

Spearman Rho

(Details on the paper!)

demo Searchsounds, with users ~250,000 full songs ~48% songs annotated, avg. 2 tags per song from CDBaby, Magnatune, etc.

Searchsounds, with users

~250,000 full songs

~48% songs annotated, avg. 2 tags per song

from CDBaby, Magnatune, etc.

conclusions Extent annotations in a music collection General experiment 10 similar, recall >0.4 40%+38% = 78% Mood experiment P@1 = 0.5 30%+35% = 65% Avoid cold start problem (new song) Limitations General experiment: Low precision, due to (valid) proposed tags, that were not in the GT Mood experiment: “Mysterious” label

Extent annotations in a music collection

General experiment

10 similar, recall >0.4

40%+38% = 78%

Mood experiment

P@1 = 0.5

30%+35% = 65%

Avoid cold start problem (new song)

Limitations

General experiment: Low precision, due to (valid) proposed tags, that were not in the GT

Mood experiment: “Mysterious” label

questions? Preguntes? ¿Preguntas? Questions ? Fragen ? أسئلة ؟

Preguntes?

¿Preguntas?

Questions ?

Fragen ?

أسئلة ؟

Annotating Music Collections: How Content-Based Similarity Helps to Propagate Labels Mohamed Sordo, Cyril Laurier and Òscar Celma (Music Technology Group, UPF)

Mohamed Sordo, Cyril Laurier and Òscar Celma

(Music Technology Group, UPF)

experiments:: metrics SongId usana31605 Manually annotated (ground truth) Classical, Instrumental, Baroque, Piano Proposed labels (and frequency) Instrumental (0.55), Classical (0.40), Baroque (0.26), Harpsichord (0.24) Precision = TP / (TP + FN) 3 / 4 (added Harpsichord) Recall = TP / (TP + FP) 3 / 4 (Piano is missing) F-measure (alpha = 2 , more weight to Recall) Spearman-Rho

SongId usana31605

Manually annotated (ground truth)

Classical, Instrumental, Baroque, Piano

Proposed labels (and frequency)

Instrumental (0.55), Classical (0.40), Baroque (0.26), Harpsichord (0.24)

Precision = TP / (TP + FN)

3 / 4 (added Harpsichord)

Recall = TP / (TP + FP)

3 / 4 (Piano is missing)

F-measure (alpha = 2 , more weight to Recall)

Spearman-Rho

experiments:: results How does CB performs...?

How does CB performs...?

experiments:: results With album, artist filtering...

With album, artist filtering...

experiments:: results Recall the recall

Recall the recall