Emilia Gómez, Music Similarity: Concepts, Cognition and Computation, Lorentz Center, Leiden, Netherlands, January 2015

1. Music
similarity:
what
for?
emilia.gomez@upf.edu
Building
real
applica.ons
for
real
users

2. PXVLF
FRQWHQW
([DPSOHV
UKWKP
WLPEUH
PHORG
KDUPRQ
ORXGQHVV
VRQJ OULFV
PXVLF
FRQWH[W
XVHU
FRQWH[W
([DPSOHV
VHPDQWLF ODEHOV
SHUIRUPHUµV UHSXWDWLRQ
DOEXP FRYHU DUWZRUN
DUWLVW
V EDFNJURXQG
PXVLF YLGHR FOLSV
([DPSOHV
PRRG
DFWLYLWLHV
VRFLDO FRQWH[W
VSDWLRWHPSRUDO FRQWH[W
SKVLRORJLFDO DVSHFWV
XVHU SURSHUWLHV
PXVLF
SHUFHSWLRQ
([DPSOHV
PXVLF SUHIHUHQFHV
PXVLFDO WUDLQLQJ
PXVLFDO H[SHULHQFH
GHPRJUDSKLFV
RSLQLRQ DERXW SHUIRUPHU
DUWLVW¶V SRSXODULW DPRQJ IULHQGV
Concepts
and
models
of
similarity
• Aim
of
the
day:
modeling
similarity
of
musical
content
– Challenges,
goals
– Formal
models
vs.
informal
expert
knowledge
(Schedl et al. 2011)

3. Outline
• Music
similarity
in
MIR
à
audio
• Challenges
• 2
projects
related
to
music
similarity

4. Music
similarity
in
Music
InformaFon
Retrieval
(Casey et al. 2008)
“Help
people
find
music”:

5. Music
similarity
in
Music
InformaFon
Retrieval
(Casey et al. 2008) (Grosche et al. 2011)
“Help
people
find
music”:
• Specificity

6. Music
similarity
in
Music
InformaFon
Retrieval
(Casey et al. 2008)
“Help
people
find
music”:
• Specificity
• Granularity
/
temporal
scope
! #$
!#$#%' ()*$#%'
%'()* +,'-./01
*+,-.')#
2! !
()*$#%'$/)*$0-)0,12-)
3.-2#2-) 40/.2#1,/#2-)
!
5,-6/,$1''/47
#2' 0/,
!#$#%' .*/,
Keyscape (Sapp 2005) (Martorell Gómez 2011)

7. Music
similarity
in
Music
InformaFon
Retrieval
Different
tasks
and
applicaFons:
(Grosche et al. 2011)

8. Music
similarity
in
Music
InformaFon
Retrieval
Different
tasks
and
applicaFons:
(Grosche et al. 2011)
SIMILARITY

9. Music
similarity
measures
• Task
dependent:
– Content:
audio,
score,
lyrics,
etc.
– Musical
facets:
melody,
rhythm,
tonality,
Fmbre,
instrumentaFon.
– Descriptors.
– Weights.

10. Audio
music
similarity
1. Low-­‐level
spectral
descriptors:
Aucouturier
and Pachet (2004), Pampalk (2006)
– High
specificity
–
global
– “Audio
quality”
(Urbano et al. 2014)
– “Timbre”
à
sound
quality
0 1000 2000 3000 4000 5000 6000 7000
−60
−40
−20
LTSA Flute − C4
Frequency (Hz)
0 1000 2000 3000 4000 5000 6000 7000
−60
−40
−20
LTSA Oboe − C4
Frequency (Hz)
Spectralmagnitude(dB)
(McAdams and Giordano 2008)

11. Audio
music
similarity
1. Low-­‐level
spectral
descriptors:
Aucouturier and
Pachet (2004), Pampalk (2006)
2. Incorporate
mid-­‐level
musical
descriptors:
– Rhythm:
Foote (2002)
– Pitch:
Müller et al. (2006), Serrà et al. (2007)
à
cover
version
iden.fica.on,
audio-­‐score
alignment

12. Cover
version
idenFficaFon
(Gómez et al. 2006)

13. Approaches
in
audio
music
similarity
1. Low-­‐level
spectral
descriptors:
Aucouturier and
Pachet (2004), Pampalk (2006)
2. Incorporate
mid-­‐level
musical
descriptors
3. Combine
those
with
semanFc
descriptors
obtained
by
automaFc
classificaFon
(ex:
genre,
instrument,
mood):
Bogdanov et al. (2013)

14. PersonalizaFon
(Schedl et al. 2012)
1. Let
users
control
weights
– Lot
of
effort
for
a
high
number
of
descriptors
– The
user
should
make
his
preference
explicit
2. Gather
raFngs
of
the
similarity
of
pairs
of
songs
à
robustness
(Urbano et al. 2010)
3. CollecFon
clustering:
ask
users
to
group
songs
in
a
2D
plot
(Stober 2011)

15. EvaluaFon
• Similarity
vs
categorizaFon:
arFst,
genre,
instrument,
covers,
co-­‐occurrence
in
personal
collecFons
and
playlists
(Berenzweig et al. 2003)
• Surveys
(Vignoli and Pauws 2005)

16. But
As…”Similarity
is
an
ill-­‐defined
concept”
we
should
evaluate
each
task
separately!

17. Audio
Music
Similarity
Task
• 7000
30-­‐second
audio
clips
drawn
from
10
genres:
Blues,
Jazz,
Country/Western,
Baroque,
Classical,
Roman.c,
Electronica,
Hip-­‐Hop,
Rock,
HardRock/Metal
• Songs
from
the
same
arFst
filter
out
• EvaluaFon
criteria:
– User
raFngs:
not
similar,
somewhat
similar,
very
similar
– ObjecFve
staFsFcs:
similarity
in
terms
of
genre,
arFst
and
album.
• More
on
talk
by
A.
Flexer.

18. Tasks
related
to
similarity
• Audio
cover
idenFficaFon
• Audio
classificaFon
• Query
by
singing
(humming)
• Query
by
tapping
• Audio
to
score
alignment
• Discovery
of
repeated
themes
/
secFons
• Structural
segmentaFon
• Audio
fingerprinFng
• Symbolic
melodic
similarity
• …

19. Challenges
1. Music
is
mulFmodal,
mulF-­‐faceted
2. Similarity
depends
on
a. the
user/listener,
b. the
repertoire,
and
c. the
task
Use-­‐cases

20. Use
case
1:
• Repertoire:
symphonic
music
• ModaliFes:
audio,
score,
video,
gestures
• Task:
structural
analysis
à
visualizaFon
• PersonalizaFon:
“experts”
–
listeners
exposed
to
it
(me)
–
naïve
listeners
(young
people?)
Beethoven Symphony No. 3 Eroica
http://phenicx.upf.edu/

21. ModaliFes
• Audio:
dynamics,
Fmbre
tempo,
f0
(Grachten et al. 2013) (Bosch
Gómez 2013)
• Score:
key,
pitch-­‐class
sets,
orchestraFon
(Martorell and Gómez 2014)
• Video:
performers,
movement
(Bazzica, Liem and Hanjalic 2014)
• Gestures:
movement
(Sarasúa and Guaus 2014)
• Context:
manual
annotaFons
(Schedl et al. 2014)

22. Strategies
• SynchronizaFon
• Generate
different
layers
of
informaFon
• PersonalizaFon:
– Understand
user
needs:
naïve
listeners,
music
experts,
performers
– Let
them
choose
by
means
of
visualizaFon,
interacFon
à
HCI

23. Use
case
2:
• Repertoire:
flamenco
singing
• ModaliFes:
audio
• Task:
style
and
variant
characterizaFon
• PersonalizaFon:
“experts”
–
listeners
exposed
to
it
(Me)
–
naïve
listeners
(you?)
http://mtg.upf.edu/research/projects/cofla

24. Melodic
similarity
No se puede mostrar la imagen. Puede que su equipo no tenga suficiente memoria para abrir la imagen o que ésta esté dañada. Reinicie el equipo y, a continuación, abra el archivo de nuevo. Si sigue apareciendo la x roja, puede que tenga que borrar la imagen e insertarla de nuevo.
1. Each
style
is
characterized
by
a
common
melodic
skeleton
2. Spontaneous
improvisaFon:
ornamentaFon,
prolongaFon,
rhythmic
and
melodic
modificaFon
Antonio
Mairena
Chano
Lobato

25. Melodic
similarity
-­‐
style
• Ground
truth:
style
annotaFons
• Specific
standard
measures:
– High-­‐level
expert
specific
features
– Fundamental
frequency
(Dynamic
Fme
warping)
– Symbolic-­‐based
descriptors
– Chroma
similarity
(Huson 1998)

26. Melodic
similarity
–
variants
• Ground
truth:
– human
judgements
– flamenco
experts
vs
naïve
listeners
• Strongest
agreement
among
experts
and
different
criteria
à
no
consensus
/
general
soluFon
yet!
– Large
scale
user
studies
(Gómez et al. 2012) (Kroher et al. 2014)

27. Conclusions
• Music
is
mulF-­‐modal,
mulF-­‐faceted,
mulF-­‐
layer
• Similarity
is
not
a
general
concept,
but
depends
on
– the
task
– the
repertoire,
and
– the
listener!
(and
his
context…)