Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Knowledge-based Music Recommendation

615 views

Published on

Models, Algorithms, Exploratory Search

Slides of my PhD presentation

Published in: Science
  • Be the first to comment

Knowledge-based Music Recommendation

  1. 1. Knowledge-based Music Recommendation Models, Algorithm and Exploratory Search Michel BUFFA Reviewer Mounia LALMAS Reviewer Gaël RICHARD Examiner Tommaso DI NOIA Examiner Pietro MICHIARDI Examiner Benoit HUET Thesis Director Raphäel TRONCY Thesis Co-Director ThesisCommittee PhD Candidate Pasquale Lisena 11 October 2019
  2. 2. 1. Music in particular Classical Music 2. Knowledge Graphs as part of Semantic Web technologies 3. ML techniques applied to Music KG in particular for recommendation What’s my thesis about
  3. 3. Why Classical Music?
  4. 4. 4 CLASSICALPOPULAR VS
  5. 5. 5 M. Lasar (2011). Digging into Pandora’s Music Genome with musicologist Nolan Gasser. https://arstechnica.com/tech-policy/2011/01/digging-into-pandoras-music-genome-with-musicologist-nolan-gasser/ When it comes to classical music, on the other hand, it's much more about the composition itself, because even though the interpretation can vary in various subtle ways. CLASSICALPOPULAR VS For pop music the experience of the music is really defined by the recording.
  6. 6. 6 CLASSICALPOPULAR VS Track-based Work-based 70 years of history Thousand years from Gregorian chant to a work written last Tuesday Songs Multi-movement works Major, minor Polyphonic, homophonic, monophonic
  7. 7. 7 M. Schedl (2015) Towards Personalizing Classical Music Recommendations. 2015 IEEE International Conference on Data Mining Workshop (ICDMW), Atlantic City, NJ, USA, pp. 1366-1367. https://doi.org/10.1109/ICDMW.2015.8 “Fans of classical music are underrepresented on social media and music streaming platforms.” ● Less data ● Less detailed metadata ● Less involved in research Music recommendation research Classical music recommendation research
  8. 8. 8 Data Metadata Data which describes other data composer composition date genre performer key derivation type 1801
  9. 9. 9 Title, opus, movement Who is the composer? Who is the performer? online music approach Track as “atomic unit”
  10. 10. 10 music archives approach Work as “aggregation unit” ● genre ● date ● author ● title(s) ● … ● publication ● performance ● recordings ● books ● ...
  11. 11. 11 Which model best represents these rich data for final users and music scholars? What strategies to adopt for building a music Knowledge Graph? How to make these data accessible to researchers and developers? How can graph-based algorithms support music recommender systems? What information can be extracted from editorial playlists? Is Graph representation also suitable for music content? Research Questions RQ1 RQ2 RQ3 RQ4 RQ5 RQ6
  12. 12. Roadmap A. Music Model & Vocabularies B. Data Conversion C. Web APIs for KG Building a Music graph Exploiting the Music knowledge PART I PART II 12 A. Embeddings and Similarity B. Playlists and Weights C. Learning MIDI Embeddings
  13. 13. 13 Improve music description to foster music exchange and reuse Travel to the heart of the musical archives in France’s greatest institutions Connect sources, multiply usage, enrich user experience
  14. 14. 14 pic: https://flic.kr/p/29YHAqY Building a Music graph PART I
  15. 15. 15 What is a Knowledge Graph? source: https://hackernoon.com/wtf-is-a-knowledge-graph-a16603a1a25f It is a specific kind of knowledge base which is: ● a directed graph connections between nodes are first-class citizens ● semantic the meaning of the connections are part of the data itself ● smart allows graph-computing techniques and algorithms ● alive easy to extend, access, reuse Semantic Web technologies realize graphs in which nodes and properties linking them are identified by URIs.
  16. 16. 16
  17. 17. 17 pic:https://pxhere.com/it/photo/1523259 I.A Music Model & Vocabularies Building a Music graph Which model to represent this richness? musicologists libraries musical museums conservatories radios concert halls RQ1
  18. 18. 18 - One of the first example of describing music using Semantic Web - Extend FRBR, Timeline Ontology, Event Ontology - Uses vocabularies for Keys, Musical Instrument (by MusicBrainz), Genres (DBpedia) Y. Raimond, S. Abdallah, M. Sandler, and F. Giasson (2007). The Music Ontology. In 15th International Conference on Music Information Retrieval (ISMIR). 417–422 State of the art The Music Ontology Building a Music graph Music Model & Vocabularies I.A
  19. 19. 19 The DOREMUS Model - Relies on Linked Data and Semantic Web principles ○ everything is a URI ○ RDF model - Music specific extension of FRBRoo - Event-based pattern: the knowledge is represented in modules (triangles) which describe events that give birth to work/expression FRBR museum information bibliographic records P. Choffé and F. Leresche (2016). DOREMUS: connecting sources, enriching catalogues and user experience. In 24th IFLA World Library and Information Congress. Building a Music graph Music Model & Vocabularies I.A
  20. 20. 20 F14 Work F22 Expression M2 Opus Statement F28 Expression Creation R3 is realized in E7 Activity 5 1 “Sonate pour violoncelle et piano no 1”@fr “Sonates" , "Sonata in F" Ludwig van Beethoven Ludwig von Beethoven composer compositeur@fr compositore@it R17 created R19createda realizationof U17 has opus statement U12 has genre P102 has title U31 had function of type P14 carried out by P9 consists of P4 has time span1796 Sonata sonata@it , sonate@fr , klaviersonate@de M42 Performed Expression Creation M43 Performed Expression Berlin P4 has time span 1796 P7 took place at F24 Publication Expression F30 Publication Event P4 has time span 1797 P7 took place at Vienna U4 had princeps publication U54 is performed expression of P165 incorporates 1770 1827 P98 born P100 died U11 has key F Major F Dur@de , Fa majeur@fr, Fa maggiore@it , Fa mayor@es M6 Casting M23 Casting Detail U13 has casting 1 U30 quantity U2 foresees mop Piano Pianoforte@it Fortepian@pl M23 Casting Detail 1 U30 quantity U2 foresees mop Cello Violoncello@it Violoncelle@fr F15 Complex Work F19 Publication Work M44 Performed Work U5 had premiere U38 has descriptive expression R10 has member
  21. 21. 21 “Sax”@en “Saxophone”@en “Saxofone”@pt “Sassofono”@it “Saxophone”@fr Alternate labels Alternate languages “English term is preferred globally” Notes “Woodwinds”@en “Legni”@it Hierarchy “Baritone Saxophone”@en Example: http://data.doremus.org/vocabulary/iaml/mop/wsa Controlled Vocabularies Building a Music graph Music Model & Vocabularies I.A
  22. 22. GENRES Diabolo IAML Itema3 Redomi RAMEAU Medium of performance MIMO Itema3 IAML Diabolo RAMEAU Redomi Musical keys Modes Catalogues Derivation types Functions more available at http://data.doremus.org/vocabularies 23 families of vocabularies · 11,000+ concepts · 610 links between terms INTERLINKED INTERLINKED P. Lisena et al. (2018). Controlled Vocabularies for Music Metadata. In 19th International Conference on Music Information Retrieval (ISMIR). Paris, France. Controlled Vocabularies Building a Music graph Music Model & Vocabularies I.A
  23. 23. These and additional competency questions have been collected by experts from our partner institutions and used as requirements and validation for the model. https://github.com/DOREMUS-ANR/knowledge-base/tree/master/ query-examples 23 P. Lisena et al. (2017) Modeling the Complexity of Music Metadata in Semantic Graphs for Exploration and Discovery. In (ISMIR’17) 4th International Workshop on Digital Libraries for Musicology (DLfM’17), Shanghai, China. Building a Music graph Music Model & Vocabularies Which works have been composed by Mozart when he was <10? How many works have been composed and performed for the 1st time in the same city? Which composers had the chance to direct their own work in a performance during the last decade? I.A
  24. 24. 24 Which chamber music works have been composed in the 19th century by Scandinavian composers? Edvard Grieg 1843 - 1907 Work Genre >1800 AND <=1900 CHAMBER MUSIC Composition date ?composed by nationality part of SCANDINAVIA Building a Music graph Music Model & Vocabularies I.A
  25. 25. I.B Data Conversion Building a Music graph
  26. 26. 26 Music archives have very detailed knowledge PROBLEMS ● Multiple formats ○ sometimes complex parsing is required ● No possible interoperability ● Need for discovering overlapping knowledge ● Information codified as free text ○ different practices in codifying the same information (“Op. 27 n. 2” - “Op. 27 no 2”) ○ wrong fields, typos, wrong punctuation ● Not always publicly accessible pic: wikimedia commons Building a Music graph Data Conversion I.B Ryszard Kruk S. andl McDaniel B. (2009). Goals of Semantic Digital Libraries.
  27. 27. Source datasets 27 Works 62 550 | XML Scores 9 154 | XML Concerts 340 609 | XML Discs 9 500 | XML Works 6 846 | UNIMARC Scores 30 319 | UNIMARC Concerts 5 164 | XML Discs 8 602 | XML Works 135 940 | INTERMARC Scores 89 184 | INTERMARC (3 different XML sources) Building a Music graph Data Conversion I.B
  28. 28. 28 001 FRBNF139081882FR 100 $313891295$w.0..b.....$aBeethoven$mLudwig van$d1770-1827 144 $w....b.fre.$aSonates$bPiano$pOp. 27, no 2$tDo dièse mineur 001 FRBNF139081882FR 100 $313891295$w.0..b.....$aBeethoven$mLudwig van$d1770-1827 144 $w....b.fre.$aSonates$bPiano$pOp. 27, no 2$tDo dièse mineur LANG TITLE MOP OPUS KEY MARC FILE Building a Music graph Data Conversion I.B
  29. 29. 29 001 FRBNF139081882FR 100 $313891295$w.0..b.....$aBeethoven$mLudwig van$d1770-1827 144 $w....b.fre.$aSonates$bPiano$pOp. 27, no 2$tDo dièse mineur MARC FILE NUM SUB Building a Music graph Data Conversion I.B
  30. 30. marc2rdf MARC PARSER FREE TEXT INTERPRETER STRING 2 VOCABULARY MARC files vocabularies 1st performance in Moscow, December 29, 1956, by Mstislav Rostropovich on cello and A. Dedukhin on piano “ ” mapping rules Building a Music graph Data Conversion I.B RDF graph What strategies to adopt for building a music Knowledge Graph?RQ2
  31. 31. 31 INTERMARC marc2rdf UNIMARC EUTERPE XML ITEMA3 XML euterpe converter itema3 converter GRAPH BNF GRAPH PHILHARMONIE GRAPH EUTERPE GRAPH ITEMA3 diabolo converter DIABOLO XML GRAPH DIABOLO STRING 2 VOCABULARY Building a Music graph Data Conversion I.B
  32. 32. 32 What is in the Knowledge Graph? 89.872 persons (composers, performers, …) 18.075 corporate bodies (orchestras, chorus, publishers, …) 357.451 musical works 16k components 4k derived works 193.412 concerts and studio recordings 469.131 performed work 3.833 foreseen concerts 31.296 publications 48.006 scores Building a Music graph Data Conversion I.B
  33. 33. 33 pic: https://www.flickr.com/photos/franganillo/2643351571 I.C Web APIs for KG Building a Music graph Pëtr Il'ič Čajkovskij Pyotr Ilyich Tchaikovsky Пётр Ильич Чайковский GALLERY OF COMPOSERS Antonio Vivaldi Ludwig van Beethoven Johann Sebastian Bach Jean Sébastien Bach [FR]
  34. 34. 34 SELECT * WHERE { ?composer a foaf:Person ; foaf:name ?name ; foaf:depiction ?img . } 34 Pëtr Il'ič Čajkovskij Pyotr Ilyich Tchaikovsky Пётр Ильич Чайковский GALLERY OF COMPOSERS Antonio Vivaldi Ludwig van Beethoven Johann Sebastian Bach Jean Sébastien Bach [FR] Building a Music graph Web APIs for KG I.C
  35. 35. 35 -- W3C specification SPARQL result JSON format "bindings": [{ "composer": { "type": "uri", "value": "http://data.doremus.org/artist/0b9d963c-bfd7-337d-b6c3-c874f5e62125" }, "name": { "type": "literal", "value": "Petr Ilʹič Čajkovskij" }, "img": { "type": "uri", "value": "http://.../Pyotr_Ilyich_Tchaikovsky.jpg" } }, { "composer": { "type": "uri", "value": "http://data.doremus.org/artist/0b9d963c-bfd7-337d-b6c3-c874f5e62125" }, "name": { "type": "literal", "value": "Piotr Ilitch Tchaikovski" }, "img": { "type": "uri", "value": "http://.../Pyotr_Ilyich_Tchaikovsky.jpg" } }, { "composer": { "type": "uri", "value": "http://data.doremus.org/artist/b34f92ab-ad86-361b-a8b8-5c3a4db784d0" }, "name": { "type": "literal", "value": "Antonio Vivaldi" }, "img": { "type": "uri", "value": "http://.../Antonio_Vivaldi.jpg" } }, ... SAME DIFFERENT SAME DIFFERENT How to make these data accessible to researchers and developers? RQ3
  36. 36. 36 [{ "id": "http://data.doremus.org/artist/0b9d963c...", "name": [ "Petr Ilʹič Čajkovskij" "Piotr Ilitch Tchaikovski" ], "image": "http://.../Pyotr_Ilyich_Tchaikovsky.jpg" }, { "id": "http://data.doremus.org/artist/b34f92ab...", "name": "Antonio Vivaldi", "image": "http://.../Antonio_Vivaldi.jpg" }] 2 names 1 picture Building a Music graph Web APIs for KG I.C
  37. 37. 37 skip irrelevant metadata reducing and parsing merging “rows” mapping to different structures Building a Music graph Web APIs for KG I.C Booth et al. (2019) Toward Easier RDF. In W3C Workshop on Web Standardization for Graph Data.
  38. 38. 38 SPARQL Transformer /D2KLab/sparql-transformer ● JS and Python library ● A JSON-based syntax ○ template + query ● Integration in grlc.io for web api development { "proto": { "id" : "?composer", "name": "$foaf:name$required", "image": "$foaf:depiction$required" }, "$where": [ "?composer a ecrm:E21_Person" ], "$limit": 100 } Building a Music graph Web APIs for KG I.C Lisena P. et al. (2019). Easy Web API Development with SPARQL Transformer. In ISWC’19.
  39. 39. 39 SPARQL Transformer Building a Music graph Web APIs for KG I.C QUERIES* n. objects (original) n. objects (transformed) 1.Born_in_Berlin 1132 573 2.German_musicians 290 257 3.Musicians_born_in_Berlin 172 109 4.Soccer_players 78 70 5.Games 1020 981 Evaluation #1: Queries’ results * from https://wiki.dbpedia.org/onlineaccess Evaluation #2: User Survey 55 subjects Used in Overhead < 0.1 seconds
  40. 40. 40 pic: https://www.flickr.com/photos/ncculture/2065959239 Exploiting the Music knowledge PART II
  41. 41. 41 discover new music improve their streaming music experience background for their activities FINAL USERS MUSIC EXPERTS playlist producing help for concert programming automatic radio broadcasting How can graph-based algorithms support music recommender systems? RQ4
  42. 42. 42 Antonio Vivaldi Autumn. I Allegro Tomaso Albinoni Symphony n. 3 NEXT SEED TARGET How to find it?
  43. 43. 43 S. Oramas, V. C. Ostuni, T. Di Noia, X. Serra, and E. Di Sciascio. Sound and music recommendation with knowledge graphs. ACM Trans. Intell. Syst. Technol. 8, 2, Article 21 (October 2016), 21 pages. http://dx.doi.org/10.1145/2926718 The vector representation of the item i is computed on his neighborhood of length l. More two items share entities/property at a certain distance, more those items can be considered similar. State of the art Item neighborhood mapping
  44. 44. 44 https://musiclynx.github.io/#/artist/ad79836d-9849-44df-8789-180bbc823f3c/Antonio%2520Vivaldi Alo Allik, Florian Thalmann, and Mark Sandler. 2018. MusicLynx: Exploring Music Through Artist Similarity Graphs. In The Web Conference 2018. Demo Track, pp 167-170. https://doi.org/10.1145/3184558.3186970 ● Access to different knowledge sources ● Maximum Degree Weighted (MDW): links to very large categories (i.e. Living People) are discouraged with respect to more significant ones. State of the art MusicLynx
  45. 45. II.A Embeddings and Similarity pic https://pxhere.com/es/photo/1168914 Exploiting the Music knowledge
  46. 46. 46 Word Embeddings (e.g. word2vec) corpus of document -> vectors that represent the semantic distribution of words in the text Graph Embeddings (e.g. node2vec) set of random walks -> vectors that represent the semantic distribution of entity in the graph Exploiting the Music knowledge Embeddings and Similarity II.A Main idea: nodes that occurs in similar contexts (neighborhood of nodes in a graph) are more similar, and will be closer in the vector space. Aditya Grover and Jure Leskovec. node2vec: Scalable Feature Learning for Networks. In 22nd ACM SIGKDD , 2016. 46
  47. 47. Some problems: ● Our dataset was constantly growing ● The amount of nodes is huge ● Different purposes for recommendation: ○ radio broadcasting ○ concert programming ○ final users 47 computational-wise and time-wise expensive (multiple run of node2vec on huge amount of data) Exploiting the Music knowledge Embeddings and Similarity II.A
  48. 48. 48 Compute embeddings at simple features level period of time musical key medium of performance genre ... Exploiting the Music knowledge Embeddings and Similarity II.A Solution
  49. 49. 49 vocabulary:iaml/mop/wob vocabulary:iaml/mop/wcl Example: MoP The clarinet is more similar to the oboe or to the cello? vocabulary:iaml/mop/svc Exploiting the Music knowledge Embeddings and Similarity II.A
  50. 50. 50 vocabulary:iaml/mop/wob vocabulary:iaml/mop/wcl vocabulary:iaml/mop/w vocabulary:iaml/mop/s vocabulary:iaml/mop/svc Example: MoP Exploiting the Music knowledge Embeddings and Similarity II.A Graph of vocabularies
  51. 51. 51 vocabulary:iaml/mop/wob vocabulary:iaml/mop/wcl vocabulary:iaml/mop/svc Casting Detail Casting Detail Casting Detail Casting 862 times Casting 1213 times Example: MoP Exploiting the Music knowledge Embeddings and Similarity II.A Graph of usage
  52. 52. 52 SPARQL endpoint subgraph (edgelist) selection of interesting properties (i.e. skos:broader) vectors embedding NODE2VEC s 1.34 0.98 0.20 w 1.44 1.21 0.31 svc 0.14 1.31 1.48 wcl -1.2 1.90 0.85 wob -0.83 2.32 1.03 Pasquale Lisena et al. Controlled Vocabularies for Music Metadata. 19th International Society for Music Information Retrieval Conference (ISMIR), Paris, France, September 2018. Exploiting the Music knowledge Embeddings and Similarity II.A
  53. 53. 53 vocabulary:iaml/mop/wob vocabulary:iaml/mop/wcl The clarinet is more similar to the oboe or to the cello? vocabulary:iaml/mop/svc 0.506 0.562 Exploiting the Music knowledge Embeddings and Similarity II.A Example: MoP
  54. 54. 54 VECTOR SPACE OF MoPs ethnic chordophones ethnic flutes percussions brass orchestra woodwinds orchestra strings rare strings Exploiting the Music knowledge Embeddings and Similarity II.A
  55. 55. 55 VECTOR SPACE OF GENRES Exploiting the Music knowledge Embeddings and Similarity II.A
  56. 56. 56 Combine embeddings at complex features level artists works playlists Exploiting the Music knowledge Embeddings and Similarity II.A
  57. 57. Example: Artists Exploiting the Music knowledge Embeddings and Similarity II.A
  58. 58. 58 MOP embeddings: MOPGENRE KEY Artist’s features: BIRTH DATE DEATH DATE CASTING WORKS GENRE WORKS KEYS WORKS PLAYED MOP -0.02 0.01 0.01 0.00 -0.01 -0.02 0.01 0.00 -0.01 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -0.00 0.00 0.00 0.07 -0.03 0.07 -0.02 -0.01 0.19 0.02 0.69 -0.19 -0.14 0.08 0.03 0.03 0.00 0.08 null null null null null -0.06 0.07 0.02 -0.03 0.00 Artist vector BIRTH PLACE DEATH PLACE FUNCTION FUNCTGeoNamesGeoNamesTime DIMENSIONALITY REDUCTION (PCA) Time AVG AVG AVG AVG AVG Exploiting the Music knowledge Embeddings and Similarity II.A Some data are unknown or not applicable null null null null null
  59. 59. 59P. Lisena, R. Troncy (2017). Combining music specific embeddings for computing artist similarity. In 18th International Society for Music Information Retrieval Conference (ISMIR), Late-breaking & demo track. percentage of missing dimensions in artist 2 with respect to artist 1 Exploiting the Music knowledge Embeddings and Similarity II.A Example: Artists
  60. 60. 60 Do all the properties have the same importance? Exploiting the Music knowledge Embeddings and Similarity II.A
  61. 61. II.B Playlists and Weights pic:https://pixabay.com/photo-791076/ Exploiting the Music knowledge Which information is possible to extract from editorial playlists? RQ5 Idea: there are “unaware” rules that experts apply when realising a playlist.
  62. 62. 62 ● Use playlists to give a weight to the influence of each dimension. ● No GOLD STANDARD available, creation of a “silver” one Radio France Playlists (50) Spotify Playlists (65) ITEMA3 Concerts (624) Philharmonie Concerts (186) ● Radio France Web Radio (7 channels) ● Realised by experts ● Classical section of Spotify app ● Realised by Spotify staff ● Real concerts that took place in Paris (studio + concert hall) Exploiting the Music knowledge Playlists and Weights II.B
  63. 63. 63 variance within < variance between HOMOGENEOUS good for recommendation variance within > variance between INHOMOGENEOUS bad for recommendation Exploiting the Music knowledge Playlists and Weights II.B F test statistic = variance between / variance within Weights
  64. 64. evaluation 64 Exploiting the Music knowledge Playlists and Weights II.B ● 7 music experts from partner institutions ● given the seed: ○ put bad items in the trash ○ sort according to preference
  65. 65. evaluation 65 Exploiting the Music knowledge Playlists and Weights II.B The study of variance help us to identify which dimensions should be promoted for better recommendations.
  66. 66. 66 Exploiting the Music knowledge Playlists and Weights II.B Under experimentation in live.philharmoniedeparis.fr
  67. 67. The role of titles: Title2Rec Exploiting the Music knowledge “Relax Driving” Johannes Brahms Symphony n.3 “Beach Party” Luis Fonsi Despacito
  68. 68. 68 Title2Rec: training Exploiting the Music knowledge Playlists and Weights II.B Content (id of tracks) Playlists SEQUENTIAL EMBEDDINGS (word2vec) CLUSTERS of playlists Titles DOCUMENTS fastText MODEL
  69. 69. 69 yy :) christmas litmas guardians christmas christmas holiday christmas christmas the good stuff. xmas himym christmas pop xmas country happy holidays holidays christmas christmas hits 25 just cause stay christmas tis the season 🎄 christmas 🎄 christmas oldbutgold christmas christmas vibes christmas strong christmas winter wonderland christmas time december 15 xmas christmas christmas pop flight christmas deep christmas vibes christmas oldies work in progress christmas christmas playlist christmas music josh 🎄 christmas blah christmas & chill depression secret christmas christmas & chill christmas love :) christmas elite :) christmas special songs christmas christmas christmas jams jessica its lit classy pump up graduation at the moment .... christmas christmas christmas music good old days christmas mix christmas music 80s rock christmas 2015 xmas christmas christmas christmas christmas vibes 2017 songs christmas vibes!! christmas music holidays christmas 2016! christmas christmas club music summer 2015 christmasssss christmassss christmas christmas christmas christmas!! christmas christmas feels christmas christmas(:: christmas playlist great christmas playlist christmas & chill christmas christmas trap blast from the past christmas 2016 classics grad christmas christmas christmas christmas yessss christmas christmas rihanna christmas christmas songs christmas 2016!!!!! good vibes christmas christmas songs christmas christmas christmas favorites christmas christmas 2016 🎄 christmas last christmas christmas all my friends christmas christmas !! chirstmas the weeknd christmas 2015 christmas christmas lyrical party music wake up happy vibes 🎄 christmas calm country winter christmas christmas christmas pop christmas af ❄ christmas feel good :)) christmas christmas af christmas jams moana christmas merry christmas! christmas playlist christmas christmas silly love songs christmas </3 school 🎄 christmas christmas music christmas christmas music 🎄 christmas x-mas christmas bops christmas beachin' dance jamz christmas new wave its christmas christmas 🎄 christmas indie 2 christmas 1980 christmas jams christmas 2015 sunrise christmas christmas playlist christmas jams christmas white ella chirstmas sleep :))))) christmas random christmas dance christmas christmas december; christmas christmas favs christmas old christmas songs ~holidaze~ christmas christmas music xmas christmas holidays december christmas christmas christmas baby wedding music tis the season christmas relax holidays!! 🎅 🏼 christmas christmas christmas december '15 christmas!! christmas new songs christmas christmas Exploiting the Music knowledge Playlists and Weights II.B Title2Rec: training
  70. 70. 70 Title2Rec: predicting Exploiting the Music knowledge Playlists and Weights II.B Given a new title: ● found the most similar titles among the known ones ● propose the most popular tracks among those titles Evaluated on Spotify’s Million Playlists Datasets in the context of the RecSys Challenge 2018 in the challenge: #37 over 112 #13 over 31
  71. 71. II.C Learning MIDI Embeddings Exploiting the Music knowledge Is Graph representation also suitable for music content? RQ5
  72. 72. 72 MIDI2vec Apply graph technologies to MIDI ● Transform MIDI flow in a graph ● Apply node2vec for learning graph embeddings Exploiting the Music knowledge Learning MIDI Embeddings II.C MIDI Group of Notes Pitch Duration Program Time Signature Tempo Velocity + + + + +
  73. 73. 73 Experiment: genre and metadata prediction Dataset 1: SLAC 250 MIDI, balanced on 5/10 genres Accuracies on cross-fold validation: Dataset 2: MuseData 438 MIDI, unbalanced, linked to DOREMUS Accuracies on cross-fold validation: Exploiting the Music knowledge Learning MIDI Embeddings II.C Baseline: McKay et al (2010)
  74. 74. 74 Exploratory Search Engine overture.doremus.org Chatbot chatbot.doremus.org Emotion Detection data.doremus.org/emotion
  75. 75. 75 Which model best represents these rich data for final users and music scholars? DOREMUS model and Vocabularies What strategies to adopt for building a music Knowledge Graph? marc2rdf and other converters result: the DOREMUS Knowledge Graph How to make these data accessible to researchers and developers? SPARQL Transformer reshapes and merges the results for easy use RQ1 RQ2 RQ3 Main contributions
  76. 76. 76 How can graph-based algorithms support music recommender systems? Embedding approach with generation and recombination of partial vectors Which information is possible to extract from editorial playlists? A study of editorial playlists, for weighting a recommender system Title2Rec: recommend music by the title of the playlist Graph representation is suitable also for music content? MIDI2vec: learning MIDI graph embeddings RQ4 RQ5 RQ6 Main contributions
  77. 77. 77 Future Work (1/2) Short Term ● Studies on simplifications of the ontology (schema.org) ● Domain-based NLP for text-field information extraction Long Term ● Strategies for modeling librarian information representing meta-information on a 2nd level (RDF*) Modeling and accessing a KG
  78. 78. 78 Future Work (2/2) Short term ● Split the dataset in historical period more precise training, faster performances ● Title2Rec + similarity-based recommender system application for editors ● Experiment MIDI embeddings on larger dataset Long term ● Gold standard dataset of classical music playlists ● Combining our strategy with more traditional ones (CF) ● MIDI ontology: extend and use in MIDI2vec Knowledge-aware Recommender system
  79. 79. Publications Conference Poster&Demo Journal Tutorial Workshop EKAW'16 ISWC'16 EKAW'16 2016 ISWC'17 X2 ISMIR'17 K-CAP'17 DLfM'17 2017 ISWC'18 X2 ISMIR'18 ISMIR'18 BIBLIOTHEK - Forschung und Praxis ESWC'18 RecSys'18 TheWebConf'18 2018 ISWC'19 2019 PC Member ISWC’18 P&D, SAAM’18, DLfM’18, ISWC’19 P&D, K-CAP’19 as sub-reviewer: KAARS’18, TheWebConf’19 Student Supervision 2 Master Thesis supervisions 10 Semester Projects supervisions Lecturer for WebInt and Aalto BootCamp Talks Des Catalogues au Web des Données - BnF, Paris Classical Music and Knowledge Graphs - Semantic Web course, PoliTo - WAI meeting, VU Amsterdam - Research seminar, Deezer, Paris
  80. 80. 80 References (1/2) ● M. Schedl (2015) Towards Personalizing Classical Music Recommendations. 2015 IEEE International Conference on Data Mining Workshop (ICDMW), Atlantic City, NJ, USA, pp. 1366-1367. ● Y. Raimond, S. Abdallah, M. Sandler, and F. Giasson (2007). The Music Ontology. In 15th International Conference on Music Information Retrieval (ISMIR). 417–422 ● P. Choffé and F. Leresche (2016). DOREMUS: connecting sources, enriching catalogues and user experience. In 24th IFLA World Library and Information Congress. ● P. Lisena et al. (2018). Controlled Vocabularies for Music Metadata. In 19th International Conference on Music Information Retrieval (ISMIR). Paris, France. ● P. Lisena et al. (2017) Modeling the Complexity of Music Metadata in Semantic Graphs for Exploration and Discovery. In 4th International Workshop on Digital Libraries for Musicology (DLfM’17), Shanghai, China. ● Booth et al. (2019) Toward Easier RDF. In W3C Workshop on Web Standardization for Graph Data. ● Lisena P. et al. (2019). Easy Web API Development with SPARQL Transformer. In ISWC’19. ● S. Oramas, V. C. Ostuni, T. Di Noia, X. Serra, and E. Di Sciascio. Sound and music recommendation with knowledge graphs. ACM Trans. Intell. Syst. Technol. 8, 2, Article 21 (October 2016), 21 pages.
  81. 81. 81 References (2/2) ● Alo Allik, Florian Thalmann, and Mark Sandler (2018). MusicLynx: Exploring Music Through Artist Similarity Graphs. In The Web Conference 2018. Demo Track, pp 167-170. ● Aditya Grover and Jure Leskovec. (2016) node2vec: Scalable Feature Learning for Networks. In 22nd ACM SIGKDD. ● McKay, C., Burgoyne, J., Hockman, J., B. L. Smith, J.,Vigliensoni, G., and Fujinaga, I. (2010). Evaluating the Genre Classification Performance of Lyrical Features Relative to Audio, Symbolic and Cultural Features. In ISMIR 2011, Utrecht, The Netherlands ● Meroño-Peñuela, A., Hoekstra, R., Gangemi, A., Bloem, P., de Valk, R., Stringer, B., Janssen, B., de Boer, V.,Allik, A., Schlobach, S., et al. (2017). The MIDI Linked Data Cloud. In ISWC 2017, Vienna, Austria. ● Huang, A. and Wu, R. (2016). Deep Learning for Music. Computing Research Repository (CoRR), https://arxiv.org/abs/1606.04930 . ● Peter Knees and Markus Schedl (2013). A survey of music similarity and recommendation from music context data. ACM Trans. Multimedia Comput. Commun. Appl. 10, 1, Article 2 (December 2013), 21 pages. ● Palumbo, Rizzo, Troncy. (2017) entity2rec: Learning user-item relatedness from knowledge graphs for top-N item recommendation. In RECSYS 2017, Como, Italy.

×