Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Symbolic Melodic Similarity (through Shape Similarity)

919 views

Published on

A brief presentation on techniques to compute the similarity between melodic pieces.

  • Be the first to comment

  • Be the first to like this

Symbolic Melodic Similarity (through Shape Similarity)

  1. 1. Symbolic Melodic Similarity (through Shape Similarity) Julián Urbano Barcelona, November 12th 2013
  2. 2. Spam • @julian_urbano • PhD, Computer Science – (Evaluation in) (Music) Information Retrieval • Postdoctoral researcher – Music Technology Group – Universitat Pompeu Fabra 2
  3. 3. Outline • • • • • • Symbolic Melodic Similarity Representing Melodies Comparing Melodies Melodic Similarity through Shape Similarity Results Demo 3
  4. 4. SYMBOLIC MELODIC SIMILARITY
  5. 5. Symbolic Melodic Similarity • Symbolic – No audio signals, just notes • Melodic – No polyphony – Usually no harmony – Usually a single voice • Similarity – Sound alike – In terms of…good question! – Very ill-defined: “whatever criteria you think makes two melodies sound alike” 5
  6. 6. Typical Use Case • A user has a (large) collection of melodies and a query melody • An SMS system ranks all melodies in the collection according to their estimated melodic similarity to the query melody – Implicitly, similarity is considered non-binary • Two melodies are similar to each other to some degree • Similarity is not a binary yes-no relation – Usually just the top-k most similar 6
  7. 7. Desired: Transposition Invariance • Ignore “irrelevant” differences in pitch 7
  8. 8. Desired: Transposition Invariance • Ignore “irrelevant” differences in pitch – Same note, different octave 7
  9. 9. Desired: Transposition Invariance • Ignore “irrelevant” differences in pitch – Same note, different octave – Same degree, different key 7
  10. 10. Desired: Transposition Invariance • Ignore “irrelevant” differences in pitch – Same note, different octave – Same degree, different key – Same pitch, different key 7
  11. 11. Desired: Time-scale Invariance • Ignore “irrelevant” differences in rhythm 8
  12. 12. Desired: Time-scale Invariance • Ignore “irrelevant” differences in rhythm – Same duration, different time signature 8
  13. 13. Desired: Time-scale Invariance • Ignore “irrelevant” differences in rhythm – Same duration, different time signature – Same duration, different tempo 8
  14. 14. Desired: No Metadata • Try not to rely on metadata, because it is not always available – Obvious: artist, genre, etc. – Less-obvious: key, time signature, tempo, etc. 9
  15. 15. REPRESENTING AND COMPARING MELODIES
  16. 16. (Bad) Representations of Melodies • Transposition Invariance – Pitches: from 0 to 127, in semitones • C4 A4 A#5 = 60 69 82 – Directed interval with tonic, in semitones • Tonic = C4: D4 A4 E3 = +2 +9 -8 – Ignore octaves • C4 D4 = 2 , C4 E4 = 4 , C4 E3 = 4 – Parsons Code • C4 D4 D4 E3 = U R D 11
  17. 17. (Bad) Representations of Melodies • Time-scale Invariance – Actual duration, in milliseconds • 250ms 187ms – Metric duration • Crotchet quaver = 1 1/2 12
  18. 18. (Good) Representation of Melodies • Represent relation between successive notes – Directed pitch interval • C4 D4 E4 E3 = +2 +2 -12 – Duration ratio • 250ms 500ms 190ms 570ms = 2 0.37 3 • Crotchet quaver quaver whole = 0.5 1 4 • Allows us to detect and keep changes locally +3 +2 +3 -3 -2 +2 +7 -2 -3 -4 +2 -5 +3 +2 +3 -3 -2 1 1 1 1 1 6 2/3 1 1 1 1 1/2 1 1 1 1 1 13
  19. 19. Models of Similarity • Given two melodies A and B… – Represent them following these criteria – But how do we compute their similarity? • Several types of models – Melody as text, classic retrieval – Melody as text, editing distance – Melody as graphs or trees, transition similarity – Melody as n-grams, classic retrieval – Melody as set of points, geometric similarities – Melody as orthogonal chains, area between chains – Melody as curves, alignment with geometric similarities 14
  20. 20. Melody as (sequence of) Text • Melodies are just a string of text, much like sentences in a book • Classic retrieval: compare frequencies of symbols – C4 appears 5% of times, D6 does 0.38% of times, etc. – Compare on a pitch-by-pitch basis • Bag-of-words: loses sense of order in music 15
  21. 21. Melody as (sequence of) Text • Melodies are just a string of text, much like sentences in a book • Editing distance: how many transformations are needed to go from melody A to melody B? – C7 E7 A6 C7 – C7 A6 D7 A6 C7 16
  22. 22. Melody as Graphs • Melodies are a graph representing the probability of transitions from pitch to pitch – Compare on a transition-by-transition basis D7 D7 E7 0.1 0.02 E7 0.1 0.1 0.1 0.08 G3 C7 0.15 E4 0.09 0.1 0.09 G3 C7 0.02 E4 0.09 0.01 0.01 • Sort-of sense of order 17
  23. 23. Melody as (sequence of) n-grams • Overlapping sequences of n successive notes – C7 E7 A6 C7 D7 • n=2 : (C7 E7) (E7 A6) (A6 C7) (C7 D7) • n=3 : (C7 E7 A6) (E7 A6 C7) (A6 C7 D7) • n=4 : (C7 E7 A6 C7) (E7 A6 C7 D7) • Classic retrieval: compare frequencies of n-grams • Motifs, though n is fixed • Sort-of sense of order 18
  24. 24. Melody as (set of) Points • Notes are points in the pitch-time plane, spaced according to the pitch and time representation – Compare disposition of points (eg. EMD) 19
  25. 25. Melody as Orthogonal Chain • Represent melodies in a stairway fashion – Compare area between chains, looking for optimum 20
  26. 26. Alignment of Symbols • Independent of representation • Alignment: similar to editing distance, but weight differently insertions, deletions and mismatches – C7 E7 A6 C7 – C7 A6 D7 A6 C7 (mismatch E7-D7 “costs” less than mismatch A6-E7) • Problem: how to decide on weights – Usually just compare numbers (pitch, octave, etc.) – Some apply music theory 21
  27. 27. MELODIC SIMILARITY THROUGH SHAPE SIMILARITY
  28. 28. Melody as a Curve • Distribute notes as points in the pitch-time plane, spaced according to their relative representation • Calculate the curve interpolating those points 23
  29. 29. Intuition Behind • Two melodies are as similar as their curves are • First derivative measures how much change there is in pitch at any given point in time – Nicely models the “change in music” • Area between derivatives measures how much the two melodies differ in their change of pitch 24
  30. 30. Intuition Behind • Naturally meets all transposition and time-scale invariance requirements • Naturally keeps sense of order in music 25
  31. 31. Interpolation • Many (boring) details regarding type of interpolation – Lagrange polynomials – Splines • Type • Degree 26
  32. 32. Interpolation • We chose Uniform B-Splines of various degrees – Changes are kept local – Easy to compute – Easy to differentiate (polynomials) • Degrees 2 and 3 seemed to work better – Computed in a span-by-span basis • Sense of motifs, like n-grams – Easily adaptable to differences in length – Easily incorporate several dimensions of music beyond pitch and time – Easily compute similarity per dimension (partial derivative) – Many other nice properties in terms of extreme behavior 27
  33. 33. Implementation • Represent melodies A and B with relative distances between two successive notes • Compute splines spans – Same as compute n-grams and then smaller splines • Align spline spans – Weights computed from similarity in shape of spans 28
  34. 34. Two Approaches • Based on geometric properties and characteristics of the collection – Reward matches depending on average similarity between spline spans in the collection – Penalize insertions and deletions based on geometric overall change with respect to x-axis – Penalize mismatches based on geometric similarity of spline spans • Again, many (not so boring) details… – Normalization of dimensions and weights 29
  35. 35. Two Approaches • How common spline spans are – Reward matches and penalize insertions and removals based on how common the spline spans are • The more common, the smaller the penalization and the higher the reward – Penalize mismatches based on geometric similarity of spline spans • Roughly compare the direction of the curves • Compute the area between derivatives • Pitch and time together or separately – Weight time differently, perceived as less important 30
  36. 36. Alignment • Local: don’t penalize changes at the beginning and at the end of melodies • Global: penalize changes everywhere • Hybrid: penalize at the beginning, but don’t penalize at the end – Listeners pay attention to the beginning the most 31
  37. 37. DOES ALL OF THIS EVEN WORK? 32
  38. 38. Evaluation of Systems • (Kind of) yearly at MIREX – Collection with >2000 melodies – Select query melodies – Run systems – Humans listen to the retrieved melodies and judge how similar they are to the queries – Systems are scored accordingly • From 0 to 1 (perfect) • And once again, many (not at all boring!) details… 33
  39. 39. 2005 • Our algorithms didn’t participate, but anyway… 34
  40. 40. 2005 • Our algorithms didn’t participate, but anyway… best participant 34
  41. 41. 2010 35
  42. 42. 2011 36
  43. 43. 2012 37
  44. 44. 2013 38
  45. 45. Briefly… • Alignment works better – Hybrid alignment works much better • Geometric representations work better • Incorporating rhythm in the comparison does not improve much, and it’s computationally more expensive – Slightly improves ranking though • Compare curves very roughly, without computing areas between them – Additionally, it is computationally more efficient 39
  46. 46. HANDS-ON DEMO
  47. 47. MelodyShape • A Java library and tool implementing all the algorithms submitted to MIREX https://github.com/julian-urbano/melodyshape 41
  48. 48. MelodyShape • A Java library and tool implementing all the algorithms submitted to MIREX https://github.com/julian-urbano/melodyshape 41
  49. 49. MelodyShape • A Java library and tool implementing all the algorithms submitted to MIREX https://github.com/julian-urbano/melodyshape 41
  50. 50. Two Libraries Needed here ... or here 42
  51. 51. To Run • Just download everything to the same folder – melodyshape-1.1.jar – commons-math3-3.2.jar – commons-cli-1.2.jar • And double click – melodyshape-1.1.jar • Can also run from the command line • All your melodies must be in MIDI format 43
  52. 52. References • Aloupis, G., Fevens, T., Langerman, S., Matsui, T., Mesa, A., Nuñez, Y., Rappaport, D., Toussaint, G.: Algorithms for Computing Geometric Measures of Melodic Similarity. Computer Music Journal 30(3), 67–76 (2006) • Bainbridge, D., Dewsnip, M., Witten, I.H.: Searching Digital Music Libraries. Information Processing and Management 41(1), 41–56 (2005) • de Boor, C.: A Practical guide to Splines. Springer, Heidelberg (2001) • Byrd, D., Crawford, T.: Problems of Music Information Retrieval in the Real World. Information Processing and Management 38(2), 249–272 (2002) • Casey, M.A., Veltkamp, R.C., Goto, M., Leman, M., Rhodes, C., Slaney, M.: Content-Based Music Information Retrieval: Current Directions and Future Challenges. Proceedings of the IEEE 96(4), 668–695 (2008) • Clifford, R., Christodoulakis, M., Crawford, T., Meredith, D., Wiggins, G.: A Fast, Randomised, Maximal Subset Matching Algorithm for Document-Level Music Retrieval.In: International Conference on Music Information Retrieval, pp. 150–155 (2006) • Doraisamy, S., Rüger, S.: Robust Polyphonic Music Retrieval with N-grams. Journal of Intelligent Systems 21(1), 53–70 (2003) • Downie, J.S.: The Scientific Evaluation of Music Information Retrieval Systems: Foundations and Future. Computer Music Journal 28(2), 12–23 (2004) • Downie, J. S., Ehmann, A. F., Bay, M., & Jones, M. C.: The Music Information Retrieval Evaluation eXchange: Some Observations and Insights. In W. R. Zbigniew & A. A. Wieczorkowska (Eds.), Advances in Music Information Retrieval, pp. 93–115 (2010) • Hanna, P., Ferraro, P., Robine, M.: On Optimizing the Editing Algorithms for Evaluating Similarity Between Monophonic Musical Sequences. Journal of New Music Research 36(4), 267–279 (2007) • Hanna, P., Robine, M., Ferraro, P., Allali, J.: Improvements of Alignment Algorithms for Polyphonic Music Retrieval. In: International Symposium on Computer Music Modeling and Retrieval, pp. 244–251 (2008) • Isaacson, E.U.: Music IR for Music Theory. In: The MIR/MDL Evaluation Project White paper Collection, 2nd edn., pp. 23–26 (2002) • Kilian, J., Hoos, H.H.: Voice Separation — A Local Optimisation Approach. In: International Symposium on Music Information Retrieval, pp. 39–46 (2002) • Lin, H.-J., Wu, H.-H.: Efficient Geometric Measure of Music Similarity. Information Processing Letters 109(2), 116–120 (2008) • McAdams, S., Bregman, A.S.: Hearing Musical Streams. In: Roads, C., Strawn, J. (eds.) Foundations of Computer Music, pp. 658–598. The MIT Press, Cambridge (1985) • Mongeau, M., Sankoff, D.: Comparison of Musical Sequences. Computers and the Humanities 24(3), 161–175 (1990) • Rizo, D.: Symbolic Music Comparison with Tree Data Structures. Department of Computer Science, University of Alicante (2010). • Selfridge-Field, E.: Conceptual and Representational Issues in Melodic Comparison. Computing in Musicology 11, 3–64 (1998) • Smith, L.A., McNab, R.J., Witten, I.H.: Sequence-Based Melodic Comparison: A Dynamic Programming Approach. Computing in Musicology 11, 101–117 (1998) • Smith, T.F., Waterman, M.S.: Identification of Common Molecular Subsequences. Journal of Molecular Biology 147(1), 195–197 (1981) • Typke, R., Veltkamp, R.C., Wiering, F.: Searching Notated Polyphonic Music Using Transportation Distances. In: ACM International Conference on Multimedia, pp. 128–135 (2004) • Typke, R., Wiering, F., Veltkamp, R.C.: A Survey of Music Information Retrieval Systems. In: International Conference on Music Information Retrieval, pp. 153–160 (2005) • Uitdenbogerd, A., Zobel, J.: Melodic Matching Techniques for Large Music Databases. In: ACM International Conference on Multimedia, pp. 57–66 (1999) • Ukkonen, E., Lemström, K., Mäkinen, V.: Geometric Algorithms for Transposition Invariant Content-Based Music Retrieval. In: International Conference on Music Information Retrieval, pp. 193– 199 (2003) • Urbano, J.: MIREX 2013 Symbolic Melodic Similarity: A Geometric Model supported with Hybrid Sequence Alignment. Music Information Retrieval Evaluation eXchange (2013) • Urbano, J., Lloréns, J., Morato, J., & Sánchez-Cuadrado, S.: Melodic Similarity through Shape Similarity. In S. Ystad, M. Aramaki, R. Kronland-Martinet, & K. Jensen (Eds.), Exploring Music Contents, pp. 338–355 (2013) • Urbano, J., Schedl, M. and Serra, X.: Evaluation in Music Information Retrieval. Journal of Intelligent Information Systems 41(3), 345-369 (2013) • Ó Maidín, D.: A Geometrical Algorithm for Melodic Difference. Computing in Musicology 11, 65–72 (1998) 44

×