Using Visualizations for Music Discovery

45,696 views
35,510 views

Published on

As the world of online music grows, tools for helping people find new and interesting music in these extremely large collections become increasingly important. In this tutorial we look at one such tool that can be used to help people explore large music collections: information visualization. We survey the state-of-the-art in visualization for music discovery in commercial and research systems. Using numerous examples, we explore different algorithms and techniques that can be used to visualize large and complex music spaces, focusing on the advantages and the disadvantages of the various techniques. We investigate user factors that affect the usefulness of a visualization and we suggest possible areas of exploration for future research.

This is the slide set for the ISMIR 2009 tutorial by Justin Donaldson and Paul Lamere. Note that this presentation originally included a large number of audio and video samples which are not included in this slide deck due to space considerations.

Published in: Technology, Education
3 Comments
74 Likes
Statistics
Notes
  • Yes, the piano can sing! http://bit.ly/1M93Roz https://youtu.be/hPT5zCxIOQ0
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Very interesting from The Echo Nest you can find a real application using those concepts of graph theory (Discovr Music app for Mac/iPhone/iPad). The nice thing is that behind this there's the magic wand of Graphviz, an amazing research project and graph visualization library and SDK, http://www.graphviz.org/. For me, Graphviz is a milestone in data visualization and manipolation.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • great presentation, very well ordered, I wish some visualizations had a description as some times they are not public.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
45,696
On SlideShare
0
From Embeds
0
Number of Embeds
8,979
Actions
Shares
0
Downloads
970
Comments
3
Likes
74
Embeds 0
No embeds

No notes for slide

Using Visualizations for Music Discovery

  1. Using Visualizations for Music Discovery ISMIR 2009 Justin Donaldson Paul Lamere
  2. Using Visualizations for Music Discovery ISMIR 2009 Or ... Justin Donaldson Paul Lamere
  3. 85 ways to visualize your music 2
  4. Speakers Justin Donaldson is a PhD candidate at Indiana University School of Informatics, as well as a regular research intern at Strands, Inc. Justin is interested with the analyses and visualizations of social sources of data, such as those that are generated from playlists, blogs, and bookmarks. Paul Lamere is the Director of Developer Community at The Echo Nest, a research-focused music intelligence startup that provides music information services to developers and partners through a data mining and machine listening platform. Paul is especially interested in hybrid music recommenders and using visualizations to aid music discovery.  Paul also authors 'Music Machinery' a blog focusing on music discovery and recommendation. 3
  5. Outline of the talk • Why visualize • Some history • Core concepts • Survey of Music Visualizations • Tools / Resources • Conclusions 4
  6. Music Discovery Challenge • Millions of songs • We need tools • Recommendation • Playlisting • Visualization 5
  7. Why Visualize? Visualizations can tell a story 6
  8. Why Visualize? Visualizations can engage our pattern-matching brain 7
  9. Why Visualize? property value x mean 9 x var 10 y mean 7.5 y var 3.75 x y cor. 0.816 linear reg. y = 3 + .5x • Anscombe’s “Quartet” • Each distribution has equivalent summary statistics • Means, correlations, variance: Sometimes one number summaries are deceiving. http://en.wikipedia.org/wiki/Anscombe%27s_quartet 8
  10. Semantic Pattern Mappings Colin Ware - Visual Thinking for Design Using Visualizations for Music Discovery - ISMIR 2009 9
  11. Semantic Pattern Mappings Colin Ware - Visual Thinking for Design Using Visualizations for Music Discovery - ISMIR 2009 10
  12. Basic Popout Channels Color Colin Ware - Visual Thinking for Design Using Visualizations for Music Discovery - ISMIR 2009 11
  13. Basic Popout Channels Size Colin Ware - Visual Thinking for Design Using Visualizations for Music Discovery - ISMIR 2009 12
  14. Basic Popout Channels Shape Colin Ware - Visual Thinking for Design Using Visualizations for Music Discovery - ISMIR 2009 13
  15. Basic Popout Channels Motion Colin Ware - Visual Thinking for Design Using Visualizations for Music Discovery - ISMIR 2009 14
  16. Basic Popout Channels Motion Colin Ware - Visual Thinking for Design Using Visualizations for Music Discovery - ISMIR 2009 14
  17. Basic Popout Channels Motion Colin Ware - Visual Thinking for Design Using Visualizations for Music Discovery - ISMIR 2009 14
  18. Basic Popout Channels Spatial Grouping Colin Ware - Visual Thinking for Design Using Visualizations for Music Discovery - ISMIR 2009 15
  19. Overloading the channels Flickr photo by Stéfan Using Visualizations for Music Discovery - ISMIR 2009 16
  20. Visualization of Visualizations Visualization Name Using Visualizations for Music Discovery - ISMIR 2009 17
  21. Types of visualizations Tree Map Graph Time Series Scatter Hybrid Connections Interactive Using Visualizations for Music Discovery - ISMIR 2009 18
  22. Objects to be discovered human machine human machine Artists 23 37 Genre 18 14 Albums 0 6 Users 0 4 Tracks 3 17 Radio 0 2 Samples 3 0 DJs 0 1 Playlists 0 0 Games 0 0 Ordered 0 9 Commu 0 0 Tracks Concerts 0 0 nities Venues 0 0 Blogs 0 0 Taste 2 2 Music 0 0 Trends Labels 1 0 Videos 26 Human rendered 55 Machine rendered visualizations Using Visualizations for Music Discovery - ISMIR 2009 19
  23. Features Used Content Beat, tempo, timbre, harmonic content, lyrics Cultural web and playlist co-occurrence, purchase/play history, social tags, genre, text aura, expert opinion, appearance, popularity Relational member of, collaborated on, supporting musician for, performed as, parent, sibling, married, same label, produced Mood happy, sad, angry, relaxed, autumnal Location city, country, lat/long, Time the hits from the 60s, 70s 80s and today, recent play history Misc IQ, Temperature, Name morphology Using Visualizations for Music Discovery - ISMIR 2009 20
  24. More features A scene from High Fidelity Using Visualizations for Music Discovery - ISMIR 2009 21
  25. More features A scene from High Fidelity Using Visualizations for Music Discovery - ISMIR 2009 21
  26. Human rendered visualizations 22
  27. Human rendered visualizations Some common techniques • The Flow • The Map • The Tree • The Graph 23
  28. Human rendered visualizations Some common techniques • History • Influences • Similarity • Popularity 24
  29. Human rendered visualizations Some common techniques • History • Influences • Similarity • Popularity 24
  30. Human rendered visualizations Some common techniques • History • Influences • Similarity • Popularity 24
  31. Genealogy of Pop and Rock Music Genealogy of Pop and Rock Music - Reebee Garofalo 25
  32. Genealogy of Pop and Rock Music Genealogy of Pop and Rock Music - Reebee Garofalo 25
  33. Genealogy of Pop and Rock Music Genealogy of Pop and Rock Music - Reebee Garofalo 25
  34. Ishkur’s Guide to Electronic Dance Music 26
  35. Ishkur’s Guide to Electronic Dance Music 26
  36. The Guardian | Friday February 3 2006 The Guardian | Friday February 3 2006 9 ver story oing underground Music History as a London Tube Map d we chart the branches and connections of 100 years of music using the London Underground map? Dorian Lynskey explains how a box of coloured crayons and a lot of swearing helped like a deeply implausible to plot the history of 20th music on the London ound map devised by eck in 1933. Artist Simon n transformed the tube nstellation of famous 1992 work The Great Bear, have to make them all link all, a tall order to find a s also a comedian. But for ork every interchange had n the context of musical nlikely prospect. ut with a packet of coloured sheets of A4 taped to- big box of doubt, but the racter of each line quickly certain genre. Pop inter- erything else, so that had to Line; classical music for the upies its own sphere, t perfect for the Docklands y. There were a couple of ut by the end of one after- signed genres to almost all thrashed out most of the ctions. The key stations nt to the most eclectic ecessarily the most impor- les may be more signifi- k but even their most de- st admit that they never . m thus in place, the next s were devoted to writing ibbling them out (sorry, and Lynyrd Skynyrd), ago- ertain omissions, asking ic critic Tom Service for in- with the DLR, and swear- e bit. Amazingly, there mitous blind alleys. It just ake sense. r as possible to be objec- nds I cannot stand are in ome that I love dearly ollowed chronology wher- of the line allowed it. Each epresents a sub-genre: rock to grunge and psychedelia es South-West London; rges, north of Camden, into d New York rap. If I was re- e band name echoed the on name: Highbury & Is- me Sly & the Family Stone. f course, will find flaws. ences are so labyrinthine ple equation will be imper- or example, does pop stop n? How can you draw a de- tween soul and funk? These that have plagued record tors for decades and they’re be solved here. But I think hoices are justifiable given ns of the form. ple will quibble with omis- hame, for example, that e constantly runs in tan- her the District or Metro- thus leaving no room for such as Kylie Minogue and Boys. I should also point eep my head from explod- the remit to western, pre- nglo-American music. e those changes necessi- don Underground’s under- nsitivity to explosive refer- derci, Massive Attack. For they also took exception to er Ol’ Dirty Bastard. not some definitive history an experiment to see if one work can be overlaid on a ifferent one. The elegance Harry Beck’s design – its of bustling intersections, butaries, long, slanting tan- upt dead ends, all sucked turned wine bottle of the eems to spark other con- appeal to the brain’s innate terning and structure. Plus y piece of music journalism coloured crayons should u like it. you think of the map at an.co.uk/arts at umshop.co.uk ks to Chris Townsend at r London and Andrew don Underground. Dorian Lynskey - The Guardian 27
  37. The Guardian | Friday February 3 2006 The Guardian | Friday February 3 2006 9 ver story oing underground Music History as a London Tube Map d we chart the branches and connections of 100 years of music using the London Underground map? Dorian Lynskey explains how a box of coloured crayons and a lot of swearing helped like a deeply implausible to plot the history of 20th music on the London ound map devised by eck in 1933. Artist Simon n transformed the tube nstellation of famous 1992 work The Great Bear, have to make them all link all, a tall order to find a s also a comedian. But for ork every interchange had n the context of musical nlikely prospect. ut with a packet of coloured sheets of A4 taped to- big box of doubt, but the racter of each line quickly certain genre. Pop inter- erything else, so that had to Line; classical music for the upies its own sphere, t perfect for the Docklands y. There were a couple of ut by the end of one after- signed genres to almost all thrashed out most of the ctions. The key stations nt to the most eclectic ecessarily the most impor- les may be more signifi- k but even their most de- st admit that they never . m thus in place, the next s were devoted to writing ibbling them out (sorry, and Lynyrd Skynyrd), ago- ertain omissions, asking ic critic Tom Service for in- with the DLR, and swear- e bit. Amazingly, there mitous blind alleys. It just ake sense. r as possible to be objec- nds I cannot stand are in ome that I love dearly ollowed chronology wher- of the line allowed it. Each epresents a sub-genre: rock to grunge and psychedelia es South-West London; rges, north of Camden, into d New York rap. If I was re- e band name echoed the on name: Highbury & Is- me Sly & the Family Stone. f course, will find flaws. ences are so labyrinthine ple equation will be imper- or example, does pop stop n? How can you draw a de- tween soul and funk? These that have plagued record tors for decades and they’re be solved here. But I think hoices are justifiable given ns of the form. ple will quibble with omis- hame, for example, that e constantly runs in tan- her the District or Metro- thus leaving no room for such as Kylie Minogue and Boys. I should also point eep my head from explod- the remit to western, pre- nglo-American music. e those changes necessi- don Underground’s under- nsitivity to explosive refer- derci, Massive Attack. For they also took exception to er Ol’ Dirty Bastard. not some definitive history an experiment to see if one work can be overlaid on a ifferent one. The elegance Harry Beck’s design – its of bustling intersections, butaries, long, slanting tan- upt dead ends, all sucked turned wine bottle of the eems to spark other con- appeal to the brain’s innate terning and structure. Plus y piece of music journalism coloured crayons should u like it. you think of the map at an.co.uk/arts at umshop.co.uk ks to Chris Townsend at r London and Andrew don Underground. Dorian Lynskey - The Guardian 27
  38. Visualizations as a graph 28
  39. Visualizations as a graph 28
  40. Visualizations as a graph 28
  41. Visualizations as a graph 28
  42. Visualizations as a graph 28
  43. Visualizations as a graph www.flickr.com/photos/angrypirate/364960689/ 29
  44. Visualizations as a graph www.flickr.com/photos/angrypirate/364960689/ 29
  45. Visualizations as a graph www.flickr.com/photos/angrypirate/364960689/ 29
  46. Visualizations as a graph www.flickr.com/photos/angrypirate/364960689/ 29
  47. Visualizations as a graph www.flickr.com/photos/angrypirate/364960689/ 29
  48. Visual Music Guiden Images (cc) Fabian Schneidmadel Fabian Schneidmadel - http://www.firutin.de/blog/ Using Visualizations for Music Discovery - ISMIR 2009 30
  49. Visual Music Guiden Images (cc) Fabian Schneidmadel Fabian Schneidmadel - http://www.firutin.de/blog/ Using Visualizations for Music Discovery - ISMIR 2009 30
  50. Visualizations as a tree 31
  51. Visualizations as a tree 31
  52. Visualizations as a tree 31
  53. Visualizations as a tree 31
  54. Handmade Genre Maps and Trees 32
  55. Handmade Genre Maps and Trees 32
  56. Handmade Genre Maps and Trees 32
  57. Handmade Genre Maps and Trees 32
  58. Handmade Genre Maps and Trees 32
  59. Handmade Genre Maps and Trees 32
  60. Handmade Genre Maps and Trees 32
  61. Handmade Genre Maps and Trees 32
  62. Handmade Genre Maps and Trees 32
  63. Other types of visualizations 33
  64. Other types of visualizations 33
  65. Other types of visualizations 33
  66. Other types of visualizations 33
  67. Other types of visualizations 33
  68. Other types of visualizations 34
  69. Other types of visualizations 34
  70. Artist as a function of temperature 35
  71. ISMIR Tutorial 2005 Stephan Bauman -Senior Researcher - German Research Center for AI Using Visualizations for Music Discovery - ISMIR 2009 36
  72. ISMIR Tutorial 2005 Stephan Bauman -Senior Researcher - German Research Center for AI Using Visualizations for Music Discovery - ISMIR 2009 36
  73. ISMIR Tutorial 2005 Stephan Bauman -Senior Researcher - German Research Center for AI Using Visualizations for Music Discovery - ISMIR 2009 36
  74. ISMIR Tutorial 2005 Stephan Bauman -Senior Researcher - German Research Center for AI Using Visualizations for Music Discovery - ISMIR 2009 36
  75. ISMIR Tutorial 2005 Stephan Bauman -Senior Researcher - German Research Center for AI Using Visualizations for Music Discovery - ISMIR 2009 36
  76. Core Concepts 37
  77. How do computers represent music? • Score - As MIDI data • Signal - As digital wave form • Association - Through associations with tags, people, or other songs 38
  78. (a brief note on) Symbolic score data • Typically provides an explicit compositional structure, and sometimes even explicit performance directions. • “Easy” to analyze/extract meaningful musicological features (time signature, key, etc.) • Scores are typically not present for most music, so other forms of representation are becoming popular. • We won’t cover score based representations of music today. 39
  79. Acoustic Data • Easy (and even occasionally legal) to find digital wave form representations of music • New techniques, methods, and even services for music signal analysis. http://www.echonest.com/ http://www.magnatune.com/ 40
  80. Association Data • Relatively new form of music analysis • New methods/services as well • What connections/ associations/relationships does music share with tags, people, playlists, other music? http://www.last.fm http://sixdegrees.hu/last.fm/ http://www.flickr.com/photos/scwn/153098444/ http://manyeyes.alphaworks.ibm.com/manyeyes/visualizations/matchbox-20-song-lyrics-tag-cloud 41
  81. Representing “Lots of Songs” • Many different types of music- related information. Song Feature1 Feature2 • Problems of sparsity, noise, data “Love me 0.4 0.9 set size. do” • We need a common appropriate “Hound Dog” 3.2 1.4 format of representation and “Paint it method of comparison. Black” 3.4 1.5 • We would like to work with matrices. 42
  82. Distances and Similarity “Love “Hound “Paint it Song me do” Dog” Black” • To form a two/three dimensional “Love me do” n/a 3.2 3.4 visualization, we will need to express a “Hound distance or dissimilarity* between the Dog” 3.2 n/a 1.5 songs in vector form. “Paint it 3.4 1.5 n/a Black” • Distance must be symmetric in order to represent a metric space. B to A song • Distance must be “non-degenerate” (distance from song A to song A == 0) = song A to B 43
  83. “Big Picture” Visualization • A couple of “simple” visualizations help to assess variance, etc. across the relevant dimensions. • Boxplots, matrix plots, image plots, etc. • Selectively normalizing data at this point to have more control, and removing obvious outlier (points/dimensions). 44
  84. Multidimensional Scaling • Athens Barcelona Brussels Calais Cherbourg Reducing the number Athens Barcelona 0 3313 3313 0 2963 1318 3175 1326 3339 1294 of dimensions of the Brussels 2963 1318 0 204 583 Calais 3175 1326 204 0 460 Cherbourg 3339 1294 583 460 0 music vector. Cologne 2762 1498 206 409 785 • Similar to “rotating” the data to find large dimensions of variation/ dissimilarity in the data. http://www.mathpsyc.uni-bonn.de/doc/delbeke/delbeke.htm 45
  85. DEMO 46
  86. Demo: Acoustic Feature Vectors • Working with Magnatagatune dataset • Combination of: • Magnatune creative commons licensed songs (clips) • Tagatune project data • Echo Nest feature data • Freely available http://tagatune.org/Magnatagatune.html 47
  87. Demo: Feature Vector Echonest data includes: • loudness • tempo • segments with: • pitch information • timbre information http://tagatune.org/Magnatagatune.html 48
  88. Demo: Feature Vector Demo motivation • Genre classification has become a prominent trend in the IR community. Explore the data pertaining to such systems. • Current approaches can identify “textural” properties of acoustic sound, but can lack cultural or structural properties of music. West & Lamere note that: “...a common example might be the confusion of a classical lute timbre, with that of an acoustic guitar string that might be found in folk, pop, or rock music...” K. West and P. Lamere, “A model-based approach to constructing music similarity functions,” EURASIP Journal on Advances in Signal Processing 2007 (2007). 49
  89. Demo: Feature Vector Demo motivation • Magnatagatune happens to have a large amount of Classical music, and Classical Lute music to boot. • We’ll try to visualize this part of the magnatagatune data set, along with folk music. 50
  90. Tools 51
  91. Tools: Feature Vector • Download Data • Use R to process, analyze, and visualize data http://cran.r-project.org/ • Free! (Libre!) http://www.cyclismo.org/tutorial/R/ http://cran.r-project.org/doc/manuals/R-intro.html 52
  92. Demo: Feature Vectors 53
  93. Demo: Feature Vector Basic Echo Nest Features • (overall) loudness • tempo • pitch means (12) * • pitch stdev (12) * • timbre means (12) * • timber stdev (12) * * weighted by segment onset loudness 54
  94. Boxplots Boxplots show the five- number summaries of a distribution: • sample maximum • upper quartile • median • lower quartile • sample minimum • (outliers) 55
  95. Boxplots Boxplots show the five- number summaries of a distribution: • sample maximum • upper quartile • median • lower quartile • sample minimum • (outliers) 56
  96. Demo: Feature Vector Raw Feature Boxplots... • Are these all normal distributions? • Are all of these weighted evenly? • Scaling becomes important... why? 57
  97. Demo: Feature Vector Looking at scaled summary statistics These are the ‘expected’ values for any song, so they’re ‘normal’ for a song. We want to enhance relationships between songs that differ from the norm. 58
  98. Demo: Feature Vector Weighted Feature Boxplots... • Standard normalized data. All means are 0, all standard deviations are 1. • We can find ‘interesting’ relationships between songs better this way. Distances can be understood in terms of standard deviations from the mean. • However... 24 pitch and 24 timbre features, vs. just 1 tempo... 59
  99. Demo: Feature Vector Principle Component Analysis (PCA) of timbre and pitch information reveal a high degree of covariance. 60
  100. Demo: Feature Vector Plotting the PCA loadings show the ‘structure’ of how the pitch profiles co- vary across the music. Blue = “Classical Guitar” (almost entirely lute music) Red = “Folk” Green= “Both” (flips and rotations) 61
  101. Demo: Feature Vector Loadings: Performing a Factor analysis Factor1 Factor2 shows evidence of pitch_mean_1 -0.117 pitch_mean_2 0.625 0.576 -0.270 “musicological” factors. pitch_mean_3 -0.585 pitch_mean_4 0.677 0.126 pitch_mean_5 -0.103 -0.487 (alternating positive/negative pitch_mean_6 0.473 loadings for both means and pitch_mean_7 0.332 -0.504 pitch_mean_8 -0.512 0.217 variances, indicating certain pitch_mean_9 0.527 -0.124 “keys” are followed for songs) pitch_mean_10 -0.520 -0.274 pitch_mean_11 0.475 0.452 pitch_mean_12 0.216 -0.359 Should pitch profile distances pitch_var_1 -0.318 0.689 be figured another way? pitch_var_2 pitch_var_3 0.638 -0.377 -0.661 0.100 pitch_var_4 0.695 0.289 pitch_var_5 -0.169 -0.536 pitch_var_6 0.658 pitch_var_7 0.363 -0.550 pitch_var_8 -0.660 0.315 pitch_var_9 0.669 -0.104 pitch_var_10 -0.620 -0.213 pitch_var_11 0.385 0.593 pitch_var_12 0.124 -0.399 62
  102. Demo: Feature Vector Timbre features clearly split the different tags we were concerned with. Blue = “Classical Guitar” Red = “Folk” Green= “Both” 63
  103. Demo: Feature Vector Loadings: Factor1 Factor2 timbre_mean_1 0.987 timbre_mean_2 0.489 timbre_mean_3 0.151 0.728 timbre_mean_4 -0.139 0.349 timbre_mean_5 -0.570 Timbral factors are a lot more timbre_mean_6 straightforward. Certain songs timbre_mean_7 -0.126 timbre_mean_8 0.175 0.248 -0.556 have a lot of timbral variance. timbre_mean_9 0.122 0.589 Other songs simply have timbre_mean_10 0.257 timbre_mean_11 -0.363 -0.461 -0.238 strong overall timbral means. timbre_mean_12 timbre_var_1 0.582 0.614 timbre_var_2 0.681 -0.286 timbre_var_3 0.822 -0.114 timbre_var_4 0.680 -0.387 timbre_var_5 0.779 timbre_var_6 0.241 -0.690 timbre_var_7 0.810 timbre_var_8 0.712 timbre_var_9 0.792 timbre_var_10 0.591 -0.358 timbre_var_11 0.667 -0.140 timbre_var_12 0.736 64
  104. Demo: Association Data 65
  105. Demo: Association Data clip_id no.voice singer duet plucking hard.rock world bongos harpsichord female.singing classical 1 2 0 0 0 0 0 0 0 0 0 0 2 6 0 0 0 0 0 0 0 1 0 0 3 10 0 0 0 1 0 0 0 0 0 0 4 11 0 0 0 0 0 0 0 0 0 0 5 12 0 1 0 0 0 0 1 0 0 1 6 14 0 0 0 0 0 0 0 0 0 0 • Magnatagatune has ‘tag’ data for songs • Tags only exist if two people used the same tag for the same song • 188 total tags 66
  106. Demo: Association Data guitar classical slow techno strings drums electronic rock 4852 4272 3547 2954 2729 2598 2519 2371 fast piano ambient beat violin vocal synth female 2306 2056 1956 1906 1826 1729 1717 1474 indian opera male singing vocals no.vocals harpsichord loud 1395 1296 1279 1211 1184 1158 1093 1086 • Tags are distributed by a power-law • It’s not as meaningful to share a “guitar” tag. A “soprano” tag is far more informative. • We need to form a comparison of tags based on the weighted relevance of the terms. 67
  107. Demo: Association Data • TF/IDF (term frequency/document frequency) • n(i,j) = number of times a term i appears in a ‘document’ j. • n(k,j) = total number of terms in ‘document’ j. • |D| = total number of ‘documents’ • {d:...} = total documents containing i • documents = tags for song http://en.wikipedia.org/wiki/Tf%E2%80%93idf 68
  108. Demo: Association Data • Forming a similarity/dissimilarity between weighted tags commonly involves finding the angle between two vectors. • Cosine dissimilarity is one (of many) ways of expressing a distance between vectors. • A, B = vectors of attributes • ||A||, ||B|| = magnitudes of vectors http://en.wikipedia.org/wiki/Cosine_similarity 69
  109. Demo: Association Data Cosine similarity of tags also (not surprisingly) clearly separates the two tags 70
  110. Demo: Hybrid 71
  111. Demo: Feature Vector Timbre features clearly split the different tags we were concerned with. Blue = “Classical Guitar” Red = “Folk” Green= “Both” 72
  112. Conclusions (demo) • Visualizations can help understand “big picture” relationships among songs, and support that the relationships are meaningful. (left plot) • Visualizations can help understand “mismatches” in modeling/clustering/ classification approaches. (x2) • Visualizations can help detect noise (x1) and outliers. (x3, x4) • Visualizations can help find a way to improve the underlying model.
  113. Opportunity: Hybrid Maps Mixing information from both timbre & term spaces has a great opportunity for forming a ‘better’ representation of genre + = 74
  114. “Reading” Conventional Dimensionality Reductions • Large dissimilarity is preserved over small dissimilarity. • Try to understand the plot in terms of its two biggest orthogonal distances. If those don’t make sense, then the plot is suspect. • Next, look for local structure, clustering, etc. 75
  115. Other Techniques 76
  116. Other Techniques • Locally Linear Embedding : Only consider ‘close’ points in distance matrix. Good at recovering from “kinks”, “folds”, and “spirals” in data (chroma) • Isomap : Establish geodesic “neighborhood” distances using Dijkstra’s algorithm, and solve with mds. • T-SNE : Convert neighbor distances to conditional probabilities. Perform gradient descent on Kullback-Leibler Divergence. Good at recovering clusters at different scales. 77
  117. Other Techniques (observations) • Many advanced techniques sacrifice “large scale” dissimilarities in order to preserve/enhance “local” features. • Sometimes large scale information is useful (to determine if scales are wrong, or if there are significant outliers) • Interpretation of advanced techniques is much more complicated. (What are the loadings, factors, of features etc.?) • It can be harder to use advanced dimensionality reduction techniques to inform classifier design. 78
  118. Building an artist map The Goal - build an artist graph suitable for music exploration Problem: The artist space is very complex Using Visualizations for Music Discovery - ISMIR 2009 79
  119. The Data • Crawled The Echo Nest API for: • Artist names • Artist hotness • Artist familiarity • 15 most similar artists • 70,000 artists • 250,000 artist-artist links • Available at musicviz.googlepages.com 80 Using Visualizations for Music Discovery - ISMIR 2009
  120. Making a good graph • Minimize edge crossings, bends and curves Good Bad • Maximize angular resolution, symmetry Good Bad Cognitive measurements of graph aesthetics, Ware Purchase Colpoys McGill , , , 81
  121. Graphing the Beatles Neighborhood digraph { "The Beatles" -> "Electric Light Orchestra"; "The Beatles" -> "The Rutles"; "The Beatles" -> "The Dave Clark Five"; "The Beatles" -> "The Tremeloes"; "The Beatles" -> "Manfred Mann"; "The Beatles" -> "The Lovin' Spoonful"; "The Beatles" -> "The Rolling Stones"; "The Beatles" -> "Creedence Clearwater Revival"; "The Beatles" -> "Bee Gees"; "The Beatles" -> "the Hollies"; "The Beatles" -> "Badfinger"; "The Beatles" -> "Paul McCartney"; "The Beatles" -> "Julian Lennon"; "The Beatles" -> "George Harrison"; "The Beatles" -> "John Lennon"; } http://www.graphviz.org/ 82 Using Visualizations for Music Discovery - ISMIR 2009
  122. Artist map - level 1 out Already interconnections start to overwhelm Using Visualizations for Music Discovery - ISMIR 2009 83
  123. Artist map - level 1 in/out Using Visualizations for Music Discovery - ISMIR 2009 84
  124. Artist map - level 1 in/out Using Visualizations for Music Discovery - ISMIR 2009 84
  125. Artist map - level 1 in/out Using Visualizations for Music Discovery - ISMIR 2009 84
  126. Artist map - level 1 in/out Using Visualizations for Music Discovery - ISMIR 2009 84
  127. Artist map - level 1 in/out Using Visualizations for Music Discovery - ISMIR 2009 84
  128. Artist map - level 1 in/out Using Visualizations for Music Discovery - ISMIR 2009 84
  129. Artist map - level 1 in/out Using Visualizations for Music Discovery - ISMIR 2009 84
  130. Artist map - level 2 116 artists and 665 edges. We are totally overwhelmed. Using Visualizations for Music Discovery - ISMIR 2009 85
  131. Artist map - level 2 116 artists and 665 edges. We are totally overwhelmed. Using Visualizations for Music Discovery - ISMIR 2009 85
  132. Simplifying the graph Eliminate the edges and use spring force embedding to position the artists Using Visualizations for Music Discovery - ISMIR 2009 86
  133. Simplifying the graph Eliminate the edges and use spring force embedding to position the artists Using Visualizations for Music Discovery - ISMIR 2009 86
  134. Reducing the edges Attach artists to only one most similar artist Using Visualizations for Music Discovery - ISMIR 2009 87
  135. Reducing the edges Attach artists to only one most similar artist Using Visualizations for Music Discovery - ISMIR 2009 87
  136. Reducing the edges Attach artists to only one most similar artist Using Visualizations for Music Discovery - ISMIR 2009 87
  137. Reducing the edges Attach artists to only one most similar artist Using Visualizations for Music Discovery - ISMIR 2009 87
  138. Putting ELO II in its place Attach artists to the most similar artist that has greater familiarity Using Visualizations for Music Discovery - ISMIR 2009 88
  139. Putting ELO II in its place Attach artists to the most similar artist that has greater familiarity Using Visualizations for Music Discovery - ISMIR 2009 88
  140. Putting ELO II in its place Attach artists to the most similar artist that has greater familiarity Using Visualizations for Music Discovery - ISMIR 2009 88
  141. Making the graph connected Build a minimum spanning tree from the graph Using Visualizations for Music Discovery - ISMIR 2009 89
  142. Making the graph connected Build a minimum spanning tree from the graph Using Visualizations for Music Discovery - ISMIR 2009 89
  143. Making the graph connected Build a minimum spanning tree from the graph Using Visualizations for Music Discovery - ISMIR 2009 89
  144. Fixing the swinging blue jeans problem Prefer connections to bands of similar familiarity Using Visualizations for Music Discovery - ISMIR 2009 90
  145. Other near-neighbor graphs Miles Davis Using Visualizations for Music Discovery - ISMIR 2009 91
  146. Other near-neighbor graphs Miles Davis Using Visualizations for Music Discovery - ISMIR 2009 91
  147. 2K artists A browsing network for music discovery Using Visualizations for Music Discovery - ISMIR 2009 92
  148. 2K artists A browsing network for music discovery Using Visualizations for Music Discovery - ISMIR 2009 92
  149. Making the graph interactive The Artist Graph Using Visualizations for Music Discovery - ISMIR 2009 93
  150. Force-Directed Edge Bundling for Graph Visualization Danny Holten, Jarke J. van Wijk Using Visualizations for Music Discovery - ISMIR 2009 94
  151. Survey of Music Visualizations 95
  152. Graphs Using Visualizations for Music Discovery - ISMIR 2009 96
  153. musicmap http://www.dimvision.com/musicmap/ Using Visualizations for Music Discovery - ISMIR 2009 97
  154. Music Plasma MusicPlasma.com Using Visualizations for Music Discovery - ISMIR 2009 98
  155. tuneglue http://audiomap.tuneglue.net/ Using Visualizations for Music Discovery - ISMIR 2009 99
  156. Last.fm artist map http://lastfm.dontdrinkandroot.net/ Using Visualizations for Music Discovery - ISMIR 2009 100
  157. Last.fm artist map http://lastfm.dontdrinkandroot.net/ Using Visualizations for Music Discovery - ISMIR 2009 100
  158. RAMA - Relational Artist Maps Late-Breaking Demo Session http://rama.inescporto.pt/ Using Visualizations for Music Discovery - ISMIR 2009 101
  159. gnod music-map http://www.music-map.com/ Using Visualizations for Music Discovery - ISMIR 2009 102
  160. musicmesh http://www.musicmesh.net/ Using Visualizations for Music Discovery - ISMIR 2009 103
  161. musicovery http://musicovery.com/ 104
  162. SoundBite for SongBird http://www.repeatingbeats.com/thesis.php - Steve Lloyd - QMUL Using Visualizations for Music Discovery - ISMIR 2009 105
  163. SoundBite for SongBird http://www.repeatingbeats.com/thesis.php - Steve Lloyd - QMUL Using Visualizations for Music Discovery - ISMIR 2009 105
  164. Amaznode http://amaznode.fladdict.net/ Using Visualizations for Music Discovery - ISMIR 2009 106
  165. Visualizing and Exploring Personal Libraries http://torrens.files.wordpress.com/2009/06/ismir.pdf Torrens, Patrick Hertzog, Josep-Lluís Arcos Using Visualizations for Music Discovery - ISMIR 2009 107
  166. Visualizing and Exploring Personal Libraries http://torrens.files.wordpress.com/2009/06/ismir.pdf Torrens, Patrick Hertzog, Josep-Lluís Arcos Using Visualizations for Music Discovery - ISMIR 2009 107
  167. Visualizing and Exploring Personal Libraries http://torrens.files.wordpress.com/2009/06/ismir.pdf Torrens, Patrick Hertzog, Josep-Lluís Arcos Using Visualizations for Music Discovery - ISMIR 2009 107
  168. Music Explorer FX http://musicexplorerfx.citytechinc.com/ Using Visualizations for Music Discovery - ISMIR 2009 108
  169. Social Music Discovery Last FM tasteOgraph http://geniol.publishpath.com/last-fm-tasteograph Using Visualizations for Music Discovery - ISMIR 2009 109
  170. Floko-On http://www.flokoon.com/#lastfm Using Visualizations for Music Discovery - ISMIR 2009 110
  171. Project Orpheus http://arcus-associates.com/orpheus/ Using Visualizations for Music Discovery - ISMIR 2009 111
  172. Musician Map: visualizing music collaborations over time http://www.sfu.ca/~jdyim/musicianMap/ Ji-Dong Yim Chris Shaw Lyn Bartram Using Visualizations for Music Discovery - ISMIR 2009 112
  173. lyric connections http://www.identitee.com/visualizer.php music tees and tee shirts on i/denti/tee Using Visualizations for Music Discovery - ISMIR 2009 113
  174. Interacting with linked data about music http://journal.webscience.org/110/ Using Visualizations for Music Discovery - ISMIR 2009 http://journal.webscience.org/110/ 114
  175. Interacting with linked data about music http://journal.webscience.org/110/ Using Visualizations for Music Discovery - ISMIR 2009 http://journal.webscience.org/110/ 115
  176. The World Of Music http://www.stanford.edu/~dgleich/demos/worldofmusic/WorldOfMusic.html David Gleich, Matt Rasmussen, Leonid Zhukov, and Kevin Lang z Using Visualizations for Music Discovery - ISMIR 2009 116
  177. The World Of Music http://www.stanford.edu/~dgleich/demos/worldofmusic/WorldOfMusic.html David Gleich, Matt Rasmussen, Leonid Zhukov, and Kevin Lang z Using Visualizations for Music Discovery - ISMIR 2009 116
  178. Last.fm Artist Map http://sixdegrees.hu/last.fm/index.html Using Visualizations for Music Discovery - ISMIR 2009 117
  179. Last.fm Artist Map http://sixdegrees.hu/last.fm/index.html Using Visualizations for Music Discovery - ISMIR 2009 117
  180. Visualizations of the connections amongst geographically located iTunes libraries http://caleblarsen.com/projects/visualization-of-the-inherent-connections-/#0 118
  181. Visualizations of the connections amongst geographically located iTunes libraries http://caleblarsen.com/projects/visualization-of-the-inherent-connections-/#0 118
  182. Maps Using Visualizations for Music Discovery - ISMIR 2009 119
  183. Maps http://playground.last.fm/iom - Islands of Music - Elias Pampalk Using Visualizations for Music Discovery - ISMIR 2009 120
  184. marGrid http://margrid.sness.net/ Using Visualizations for Music Discovery - ISMIR 2009 121
  185. PlaySOM - Intuitive access to Music Archives http://www.ifs.tuwien.ac.at/mir/playsom.html Vienna University of Technology Using Visualizations for Music Discovery - ISMIR 2009 122
  186. FM4 Soundpark http://fm4.orf.at/soundpark Martin Gasser, Arthur Flexer Using Visualizations for Music Discovery - ISMIR 2009 123
  187. nepTune http://www.cp.jku.at/projects/nepTune/ Using Visualizations for Music Discovery - ISMIR 2009 124
  188. nepTune http://www.cp.jku.at/projects/nepTune/ Using Visualizations for Music Discovery - ISMIR 2009 124
  189. Databionic Music Miner http://musicminer.sourceforge.net/ Using Visualizations for Music Discovery - ISMIR 2009 125
  190. Databionic Music Miner http://musicminer.sourceforge.net/ Using Visualizations for Music Discovery - ISMIR 2009 125
  191. soniXplorer Presented at ISMIR 2009 by Dominik Lübbers Using Visualizations for Music Discovery - ISMIR 2009 126
  192. soniXplorer Presented at ISMIR 2009 by Dominik Lübbers Using Visualizations for Music Discovery - ISMIR 2009 126
  193. soniXplorer Presented at ISMIR 2009 by Dominik Lübbers Using Visualizations for Music Discovery - ISMIR 2009 126
  194. soniXplorer Presented at ISMIR 2009 by Dominik Lübbers Using Visualizations for Music Discovery - ISMIR 2009 126
  195. mHashup http://www.mhashup.com/ Using Visualizations for Music Discovery - ISMIR 2009 127
  196. mHashup http://www.mhashup.com/ Using Visualizations for Music Discovery - ISMIR 2009 127
  197. GeoMuzik ISMIR 2008 - Oscar Celma Using Visualizations for Music Discovery - ISMIR 2009 128
  198. Last on AM/FM http://www.cs.kuleuven.be/~sten/lastonamfm Using Visualizations for Music Discovery - ISMIR 2009 129
  199. Last.fm tube tags http://playground.last.fm/demo/tagstube Using Visualizations for Music Discovery - ISMIR 2009 130
  200. Last.fm tube tags http://playground.last.fm/demo/tagstube Using Visualizations for Music Discovery - ISMIR 2009 130
  201. Scatter plots Using Visualizations for Music Discovery - ISMIR 2009 131
  202. ZMDS Visualization Justin Donaldson Using Visualizations for Music Discovery - ISMIR 2009 132
  203. Visualization Name Using Visualizations for Music Discovery - ISMIR 2009 133
  204. Visualization Name Using Visualizations for Music Discovery - ISMIR 2009 133
  205. Using map for playlist navigation Visual Playlist Generation on the Artist Map - Van Gulick,Vignoli Using Visualizations for Music Discovery - ISMIR 2009 134
  206. Mr. Emo: Music Retrieval in the Emotional Plane http://mpac.ee.ntu.edu.tw/~yihsuan/pub/MM08_MrEmo.pdf Using Visualizations for Music Discovery - ISMIR 2009 135
  207. Mr. Emo: Music Retrieval in the Emotional Plane http://mpac.ee.ntu.edu.tw/~yihsuan/pub/MM08_MrEmo.pdf Using Visualizations for Music Discovery - ISMIR 2009 135
  208. Burst Labs http://www.burstlabs.com/ Using Visualizations for Music Discovery - ISMIR 2009 136
  209. Search Inside the Music http://mediacast.sun.com/share/plamere/sitm_final.pdf - Lamere, Eck Using Visualizations for Music Discovery - ISMIR 2009 137
  210. 3D Track similarity The Mufin Player Using Visualizations for Music Discovery - ISMIR 2009 138
  211. 3D Track similarity The Mufin Player Using Visualizations for Music Discovery - ISMIR 2009 138
  212. Music Box http://thesis.flyingpudding.com/ - Anita Lillie Using Visualizations for Music Discovery - ISMIR 2009 139
  213. Music Box http://thesis.flyingpudding.com/ - Anita Lillie Using Visualizations for Music Discovery - ISMIR 2009 139
  214. Pictures of Songs http://www.flyingpudding.com/projects/viz_music/ Using Visualizations for Music Discovery - ISMIR 2009 140
  215. Visualizing Emotion in Lyrics http://www.cs.kuleuven.be/~jorisk/thrillerMovie.mov Using Visualizations for Music Discovery - ISMIR 2009 141
  216. Sampling Connections Using Visualizations for Music Discovery - ISMIR 2009 142
  217. The History of Sampling http://jessekriss.com/projects/samplinghistory/ Using Visualizations for Music Discovery - ISMIR 2009 143
  218. The History of Sampling http://jessekriss.com/projects/samplinghistory/ Using Visualizations for Music Discovery - ISMIR 2009 143
  219. Mused: Navigating the personal sample library Graham Coleman - http://www.iua.upf.edu/~gcoleman/pubs/mused07.pdf Using Visualizations for Music Discovery - ISMIR 2009 144
  220. Mused: Navigating the personal sample library Graham Coleman - http://www.iua.upf.edu/~gcoleman/pubs/mused07.pdf Using Visualizations for Music Discovery - ISMIR 2009 144
  221. Time Series Using Visualizations for Music Discovery - ISMIR 2009 145
  222. last.fm spiral http://www.diametunim.com/muse/ Using Visualizations for Music Discovery - ISMIR 2009 146
  223. Visualizing Listening History http://www.leebyron.com/what/lastfm/ - Lee Byron Using Visualizations for Music Discovery - ISMIR 2009 147
  224. Visualizing Listening History http://www.leebyron.com/what/lastfm/ - Lee Byron Using Visualizations for Music Discovery - ISMIR 2009 147
  225. Radio Waves http://www.bbc.co.uk/radio/labs/radiowaves/ Using Visualizations for Music Discovery - ISMIR 2009 148
  226. Radio Waves http://www.bbc.co.uk/radio/labs/radiowaves/ Using Visualizations for Music Discovery - ISMIR 2009 148
  227. Pulling strings from a tangle http://bowr.de/blog/?p=49 Dominikus Baur University of Munich Using Visualizations for Music Discovery - ISMIR 2009 149
  228. Pulling strings from a tangle http://bowr.de/blog/?p=49 Dominikus Baur University of Munich Using Visualizations for Music Discovery - ISMIR 2009 149
  229. Trees Genre Map derived from Last.fm tags Lamere Using Visualizations for Music Discovery - ISMIR 2009 150
  230. Trees Visualizing and exploring personal music libraries Torrens, Hertzog, Arcos Using Visualizations for Music Discovery - ISMIR 2009 151
  231. Treemap of a personal music collection Visualizations : Dave’s Music Library Contents Using Visualizations for Music Discovery - ISMIR 2009 152
  232. Interactive Mirrored Radio dial - by thomwatson Using Visualizations for Music Discovery - ISMIR 2009 153
  233. SongExplorer SongExplorer: Exploring Large Musical Databases Using a Tabletop Interface Carles F. Julià - Music Technology Group - Universitat Pompeu Fabra Using Visualizations for Music Discovery - ISMIR 2009 154
  234. SongExplorer SongExplorer: Exploring Large Musical Databases Using a Tabletop Interface Carles F. Julià - Music Technology Group - Universitat Pompeu Fabra Using Visualizations for Music Discovery - ISMIR 2009 154
  235. Fidgt:Visualize http://fidgt.com/visualize Using Visualizations for Music Discovery - ISMIR 2009 155
  236. Fidgt:Visualize http://fidgt.com/visualize Using Visualizations for Music Discovery - ISMIR 2009 155
  237. MusicNodes http://waks.nl/nl/musicalnodes Lisa Dalhuijsen, Lieven van Velthoven -LIACS, Leiden University Using Visualizations for Music Discovery - ISMIR 2009 156
  238. MusicNodes http://waks.nl/nl/musicalnodes Lisa Dalhuijsen, Lieven van Velthoven -LIACS, Leiden University Using Visualizations for Music Discovery - ISMIR 2009 156
  239. Zune MixView social.zune.net Using Visualizations for Music Discovery - ISMIR 2009 157
  240. Zune MixView social.zune.net Using Visualizations for Music Discovery - ISMIR 2009 157
  241. Musicream http://staff.aist.go.jp/m.goto/Musicream/ Goto and Masataka Goto. Using Visualizations for Music Discovery - ISMIR 2009 158
  242. Musicream http://staff.aist.go.jp/m.goto/Musicream/ Goto and Masataka Goto. Using Visualizations for Music Discovery - ISMIR 2009 158
  243. MusicRainbow Elias Pampalk & Masataka Goto, "MusicRainbow: A New User Interface to Discover Artists Using Audio-based Similarity and Web-based Labeling Using Visualizations for Music Discovery - ISMIR 2009 159
  244. MusicSun Elias Pampalk & Masataka Goto, "MusicSun: A New Approach to Artist Recommendation" Using Visualizations for Music Discovery - ISMIR 2009 160
  245. EN Music Explorer Using Visualizations for Music Discovery - ISMIR 2009 161
  246. EN Music Explorer Using Visualizations for Music Discovery - ISMIR 2009 161
  247. Here be Dragons Using Visualizations for Music Discovery - ISMIR 2009 162
  248. Here be Dragons Using Visualizations for Music Discovery - ISMIR 2009 162
  249. Here be Dragons Using Visualizations for Music Discovery - ISMIR 2009 162
  250. Visualization Proponents (The Yale Club) 0 | 0000111111222338 2 | 07 4|5 6|8 8|4 10 | 5 12 | 14 | 16 | 0 F.J. Anscombe John Tukey Leland Wilkinsen Edward Tufte (Anscombe’s Quartet) (stem/leaf plots) (Cluster heat maps) (Sparklines) http://en.wikipedia.org/wiki/F.J._Anscombe http://en.wikipedia.org/wiki/John_Tukey http://en.wikipedia.org/wiki/Leland_Wilkinson http://en.wikipedia.org/wiki/Edward_Tufte 163
  251. Interactive Visualization Proponents Ben Shneiderman Steve Deunes Ben Fry & Casey (Treemaps, NSD (& entire NYT Reas diagrams) Graphics (Processing) Department) 164
  252. Resources Using Visualizations for Music Discovery - ISMIR 2009 165
  253. Resources Using Visualizations for Music Discovery - ISMIR 2009 165
  254. Resources Using Visualizations for Music Discovery - ISMIR 2009 165
  255. Resources Using Visualizations for Music Discovery - ISMIR 2009 165
  256. Resources Using Visualizations for Music Discovery - ISMIR 2009 165
  257. Resources Using Visualizations for Music Discovery - ISMIR 2009 165
  258. Resources Using Visualizations for Music Discovery - ISMIR 2009 165
  259. Resources Using Visualizations for Music Discovery - ISMIR 2009 165
  260. Resources Using Visualizations for Music Discovery - ISMIR 2009 165
  261. Visualizing Music VisualizingMusic.com musicviz.googlepages.com/home Using Visualizations for Music Discovery - ISMIR 2009 166
  262. Other Tools 167
  263. Other Analytic* Tools • Matlab • Dimensionality Reduction Toolbox • (missing good network viz/analysis option) • Extensive Interactive Pub-quality plotting environment • Pajek (standalone Network Visualization/Analysis Platform) • IVC Software Framework (research platform for analysis and viz). • R • Excellent network viz/analysis package “network” • PCA/MDS (base library), ISOMap {vegan} (no LLE, T-SNE yet) • Pub quality plotting environments (default, ggplot2, iplots) http://ict.ewi.tudelft.nl/~lvandermaaten/Matlab_Toolbox_for_Dimensionality_Reduction.html http://cran.r-project.org/web/packages/network/index.html http://cran.r-project.org/web/packages/vegan/index.html http://vlado.fmf.uni-lj.si/pub/networks/pajek/ http://iv.slis.indiana.edu/sw/index.html *for publications 168
  264. Other “Generic” Viz Tools • Prefuse (java/as3) • Processing* • Gephi • TouchGraph • Gnuplot • matplotlib (python) http://www.prefuse.org/ http://processing.org/ http://gephi.org/ http://www.touchgraph.com/navigator.html http://www.gnuplot.info/ http://matplotlib.sourceforge.net/ 169
  265. Conclusions • Music visualizations reflect ways of understanding music. • Music understanding draws from many different types of information (musicological, acoustic, social, bibliographic, etc.) • Music visualizations can help to express this understanding in a way that can help aid navigation and discovery.
  266. Questions? 171

×