Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
PhD Thesis: Knowledge Extraction and Representation Learning for Music Recommendation and Classification Slide 1

YouTube videos are no longer supported on SlideShare

View original on YouTube

PhD Thesis: Knowledge Extraction and Representation Learning for Music Recommendation and Classification Slide 3
Upcoming SlideShare
What to Upload to SlideShare
What to Upload to SlideShare
Loading in …3
×
1 of 240

PhD Thesis: Knowledge Extraction and Representation Learning for Music Recommendation and Classification

2

Share

Download to read offline

Slides from the PhD defense of Sergio Oramas at Universitat Pompeu Fabra, Barcelona, Spain. 29th November 2017. PhD Thesis: Knowledge Extraction and Representation Learning for Music Recommendation and Classification.

Document: https://doi.org/10.5281/zenodo.1100973
Video: https://youtu.be/NpZhtKNBhZk

PhD Thesis: Knowledge Extraction and Representation Learning for Music Recommendation and Classification

  1. 1. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on Sergio Oramas Mar>n Doctoral Thesis Defense Departament of Informa0on and Communica0on Technologies Thesis Director: Dr. Xavier Serra Wednesday, November 29th, 2017 Thesis Board: Dr. Markus Schedl Dr. Emilia Gómez Dr. Brian Whitman
  2. 2. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on
  3. 3. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on
  4. 4. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on
  5. 5. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on
  6. 6. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on Popularity bias
  7. 7. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on Popularity bias
  8. 8. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on cold-start
  9. 9. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on
  10. 10. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on Cold-start problem New releases Old catalog inges0on
  11. 11. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on Cold-start problem New releases Old catalog inges0on
  12. 12. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on Cold-start problem New releases Old catalog inges0on
  13. 13. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on Exploita0on vs. Explora0on cold-start
  14. 14. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on Exploita0on vs. Explora0on cold-start
  15. 15. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on
  16. 16. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on Collabora0ve Filtering
  17. 17. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on Collabora0ve Filtering
  18. 18. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on Collabora0ve Filtering Content-based
  19. 19. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on Collabora0ve Filtering Content-based Hybrid methods
  20. 20. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on Unstructured text -  Rich informa0on: genre, rela0ons, influences -  Noisy -  Some0mes used for Rec.
  21. 21. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on Audio -  Rich informa0on: genre, 0mber, instruments -  Seman0c gap -  OZen used for Rec.
  22. 22. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on Images -  Rich informa0on: genre, age, style -  Seman0c gap -  Rarely used for Rec.
  23. 23. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on Tags -  Rich and curated informa0on -  Need experts or a crowd -  May be limited -  OZen used for Rec.
  24. 24. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on
  25. 25. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on
  26. 26. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on Music Genres: Categories that share similar musical, regional, or temporal characteris0cs
  27. 27. •  “MIR is a mul0disciplinary field of research concerned with the extrac0on, analysis, and usage of informa0on about any kind of music en0ty (e.g., a song or a music ar0st) on any representa0on level (e.g., audio signal, symbolic MIDI).” (Schedl, 2008) Music Informa0on Retrieval (MIR)
  28. 28. •  “MIR is a mul0disciplinary field of research concerned with the extrac0on, analysis, and usage of informa0on about any kind of music en0ty (e.g., a song or a music ar0st) on any representa>on level (e.g., audio signal, symbolic MIDI).” (Schedl, 2008) Music Informa0on Retrieval (MIR)
  29. 29. Music Informa0on Retrieval (MIR)
  30. 30. Music Informa0on Retrieval (MIR)
  31. 31. Music Informa0on Retrieval (MIR)
  32. 32. Music Informa0on Retrieval (MIR)
  33. 33. Music Informa0on Retrieval (MIR) chords, onsets
  34. 34. Music Informa0on Retrieval (MIR) genre, mood, form
  35. 35. Music Informa0on Retrieval (MIR)
  36. 36. Music Informa0on Retrieval (MIR) word frequencies, co-occurrence, n-grams
  37. 37. Music Informa0on Retrieval (MIR) noun phrases, part-of-speech tags
  38. 38. Music Informa0on Retrieval (MIR) seman0c rela0ons, disambiguated en00es, syntac0c dependencies
  39. 39. Music Informa0on Retrieval (MIR) Most MIR research
  40. 40. Music Informa0on Retrieval (MIR)
  41. 41. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on
  42. 42. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on
  43. 43. Thesis overview
  44. 44. Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  45. 45. Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  46. 46. Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  47. 47. Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  48. 48. Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  49. 49. Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  50. 50. En0ty Linking Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions The task to discover men0ons of en00es in text and link them to a suitable knowledge repository (Moro et al. 2014).
  51. 51. En0ty Linking Elvis Presley covered Guitar Man with Reed Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  52. 52. En0ty Linking Elvis Presley covered Guitar Man with Reed Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  53. 53. En0ty Linking Elvis Presley covered Guitar Man with Reed Ambiguity Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  54. 54. En0ty Linking Elvis Presley covered Guitar Man with Reed Ambiguity Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  55. 55. En0ty Linking Elvis Presley covered Guitar Man with Reed Ambiguity Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  56. 56. En0ty Linking Elvis Presley covered Guitar Man with Reed Ambiguity Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  57. 57. En0ty Linking Elvis Presley covered Guitar Man with Reed Ambiguity Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  58. 58. En0ty Linking Elvis Presley covered Guitar Man with Reed The task to discover men0ons of en00es in text and link them to a suitable knowledge repository (Moro et al. 2014). Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  59. 59. En0ty Linking Elvis Presley covered Guitar Man with Reed hdps://en.wikipedia.org/wiki/Jerry_Reed hdps://en.wikipedia.org/wiki/Elvis_Presley hdps://en.wikipedia.org/wiki/Guitar_Man_(song) Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  60. 60. En0ty Linking •  State-of-the-art systems –  Babelfy •  KB: BabelNet –  TagMe •  KB: Wikipedia –  DBpedia Spotlight •  KB: DBpedia Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  61. 61. En0ty Linking in the Music Domain •  Ambiguity problem –  “This was the third Weezer album that they simply named Weezer.” Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  62. 62. En0ty Linking in the Music Domain •  Ambiguity problem –  “This was the third Weezer album that they simply named Weezer.” –  “Debut is the first interna0onal solo studio album by Björk.” “Led Zeppelin released their debut album 48 years ago.” Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  63. 63. En0ty Linking in the Music Domain •  Ambiguity problem –  “This was the third Weezer album that they simply named Weezer.” –  “Debut is the first interna0onal solo studio album by Björk.” “Led Zeppelin released their debut album 48 years ago.” •  Scant research Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  64. 64. En0ty Linking in the Music Domain •  Ambiguity problem –  “This was the third Weezer album that they simply named Weezer.” –  “Debut is the first interna0onal solo studio album by Björk.” “Led Zeppelin released their debut album 48 years ago.” •  Scant research •  Knowledge Bases are incomplete –  Mainly popular and Western ar0sts Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  65. 65. ELVIS (En0ty Linking Vo0ng and Integra0on System) Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  66. 66. ELMD Dataset •  Linguis>c resource •  13k ar0st biographies with links from last.fm •  4 en0ty types: Ar0st, Album, Track, Record Label •  Disambigua0on to DBpedia with ELVIS •  Manual evalua0on: Precision 0.97 Oramas S., Espinosa-Anke L., Sordo M., Saggion H., & Serra X. (2016). ELMD: An Automa0cally Generated En0ty Linking Gold Standard in the Music Domain. LREC 2016. Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  67. 67. Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  68. 68. The process of iden0fying and annota0ng relevant seman0c rela0ons between en00es in text. Rela0on Extrac0on Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  69. 69. “Gorillaz are a Bri0sh virtual band formed in 1998 by Damon Albarn of Blur, and Jaime Haweled, co-creator of the comic book Tank Girl.” Rela0on Extrac0on Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  70. 70. “Gorillaz are a Bri0sh virtual band formed in 1998 by Damon Albarn of Blur, and Jaime Haweled, co-creator of the comic book Tank Girl.” Rela0on Extrac0on Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  71. 71. “Gorillaz are a Bri0sh virtual band formed in 1998 by Damon Albarn of Blur, and Jaime Haweled, co-creator of the comic book Tank Girl.” Rela0on Extrac0on Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  72. 72. •  Knowledge Base: Repository of knowledge organized in a taxonomic or ontologic structure. –  HandcraZed: WordNet, DBpedia, BabelNet, Freebase –  Fully automa0c: NELL, ReVerb Knowledge Bases Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  73. 73. •  Knowledge Base: Repository of knowledge organized in a taxonomic or ontologic structure. –  HandcraZed: WordNet, DBpedia, BabelNet, Freebase –  Fully automa0c: NELL, ReVerb •  Music Knowledge Bases (or databases) –  HandcraZed: MusicBrainz, Discogs –  Fully automa0c: - Knowledge Bases Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  74. 74. •  Knowledge Base: Repository of knowledge organized in a taxonomic or ontologic structure. –  HandcraZed: WordNet, DBpedia, BabelNet, Freebase –  Fully automa0c: NELL, ReVerb •  Music Knowledge Bases (or databases) –  HandcraZed: MusicBrainz, Discogs –  Fully automa0c: - Knowledge Bases Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  75. 75. Automated Knowledge Base Construc0on Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  76. 76. Automated Knowledge Base Construc0on Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions •  Dependency Parsing: MATE Tools (Bohnet 2010)
  77. 77. Automated Knowledge Base Construc0on Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions •  En>ty Linking: DBpedia Spotlight
  78. 78. Automated Knowledge Base Construc0on Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions •  Group en0ty nodes •  Find shortest path between en00es
  79. 79. Automated Knowledge Base Construc0on Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions •  Shortest path is prone to errors •  Filtering: Regular expressions –  Lexical: Word lemmas –  Syntac>cal: Dependency func0ons –  Morphological: Part-of-speech tags
  80. 80. Automated Knowledge Base Construc0on Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions •  Group different rela0ons with similar meaning •  Simplify the knowledge base was wriden by ar0st was wriden by frontman was wriden by guitarist was wriden by singer was wriden by
  81. 81. Automated Knowledge Base Construc0on Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions •  Confidence measure of extracted rela0ons •  Based on sta0s0cal analysis over all extracted rela0ons •  Select a minimum score to be part of the Knowledge Base
  82. 82. •  Task: Automated crea0on of a Music Knowledge Base (KBSF) •  Text corpora –  Stories about 30k songs gathered from songfacts.com •  Evalua0on –  Quality evalua0on: 2 annotators in random rela0ons –  Coverage evalua0on: Comparison between KBs –  Applica0on evalua0on: Explaining Recommenda0ons Experiment Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions Oramas S., Espinosa-Anke L., Sordo M., Saggion H. & Serra X. (2016). Informa0on Extrac0on for Knowledge Base Construc0on in the Music Domain. Data & Knowledge Engineering, Volume 106, Pages 70-83.
  83. 83. Quality Evalua0on Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  84. 84. Quality Evalua0on Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions Filtering
  85. 85. Quality Evalua0on Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  86. 86. Quality Evalua0on Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions (Fader et al. 2011)
  87. 87. •  Number of rela0ons between en00es present in all KBs Coverage Evalua0on KBSF-th MusicBrainz DBpedia #Rela>ons 3633 1535 1240 Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  88. 88. Norwegian Wood (The Beatles) -> Fourth Time Around (Bob Dylan) Explaining Recommenda0ons Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  89. 89. Norwegian Wood (The Beatles) -> Fourth Time Around (Bob Dylan) Explaining Recommenda0ons Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  90. 90. Norwegian Wood (The Beatles) -> Fourth Time Around (Bob Dylan) Fourth Time Around was wriden in response to Norwegian Wood by The Beatles, since it is similar, both melodically and lyrically. Explaining Recommenda0ons Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  91. 91. •  User Experiment with 35 subjects –  Recommenda0ons provided with different types of explana0ons –  Users rated recommenda0ons between 1 to 5 •  Results –  Explana0ons using original sentences improve ra0ngs by 5% with respect to recommenda0ons without explana0ons –  Higher differences in ra0ngs on musically untrained subjects Explaining Recommenda0ons Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  92. 92. •  Method for the crea0on of Music Knowledge Bases from scratch with high precision and coverage Conclusions Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  93. 93. •  Method for the crea0on of Music Knowledge Bases from scratch with high precision and coverage •  Useful for: –  Crea0ng novel Knowledge Bases –  Popula0on of exis0ng Knowledge Bases –  Explaining music recommenda0ons Conclusions Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  94. 94. •  Method for the crea0on of Music Knowledge Bases from scratch with high precision and coverage •  Useful for: –  Crea0ng novel Knowledge Bases –  Popula0on of exis0ng Knowledge Bases –  Explaining music recommenda0ons •  Novel filtering, clustering, and scoring processes Conclusions Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  95. 95. Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  96. 96. “Gorillaz are a Bri0sh virtual band formed in 1998 by Damon Albarn of Blur, and Jaime Haweled, co-creator of the comic book Tank Girl.” Knowledge Graph Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  97. 97. Seman0c Enrichment via En0ty Linking Elvis Presley covered Guitar Man with Reed Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  98. 98. Seman0c Enrichment via En0ty Linking Elvis Presley covered Guitar Man with Reed hdps://en.wikipedia.org/wiki/Jerry_Reed hdps://en.wikipedia.org/wiki/Elvis_Presley hdps://en.wikipedia.org/wiki/Guitar_Man_(song) Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  99. 99. Seman0c Enrichment via En0ty Linking Elvis Presley covered Guitar Man with Reed hdps://en.wikipedia.org/wiki/Jerry_Reed hdps://en.wikipedia.org/wiki/Elvis_Presley hdps://en.wikipedia.org/wiki/Guitar_Man_(song) Singers from Tennessee American rockabilly musicians 1935 births American country singer-songwriters American country guitarists Musicians from Atlanta 1967 singles Jerry Reed songs RPM Country Tracks number-one singles Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  100. 100. Seman0c Enrichment via En0ty Linking Item Descrip0on Seman0c Enrichment Enriched Graph Enriched Descrip0on En0ty Linking Knowledge Base Seman0c Enrichment via En0ty Linking Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  101. 101. Seman0c Enrichment via En0ty Linking Item Descrip0on Seman0c Enrichment Enriched Graph Enriched Descrip0on En0ty Linking Knowledge Base Seman0c Enrichment via En0ty Linking Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  102. 102. “Gorillaz are a Bri0sh virtual band formed in 1998 by Damon Albarn of Blur, and Jaime Haweled, co-creator of the comic book Tank Girl.” Enriched Graph Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  103. 103. “Gorillaz are a Bri0sh virtual band formed in 1998 by Damon Albarn of Blur, and Jaime Haweled, co-creator of the comic book Tank Girl.” Enriched Graph Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  104. 104. “Gorillaz are a Bri0sh virtual band formed in 1998 by Damon Albarn of Blur, and Jaime Haweled, co-creator of the comic book Tank Girl.” Enriched Graph Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  105. 105. “Gorillaz are a Bri0sh virtual band formed in 1998 by Damon Albarn of Blur, and Jaime Haweled, co-creator of the comic book Tank Girl.” Enriched Graph Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  106. 106. Exploita>on vs. Explora>on •  Exploita>on metrics: Precision@N, Recall@N, MAP •  Explora>on metrics: Aggregated Diversity (ADiv@N) Dis0nct items recommended across all users Evalua0on Metrics Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  107. 107. Ar0st Similarity •  Recommenda0on without personaliza0on •  Similarity between ar0st biographies •  Knowledge Graphs vs. Text-based approach •  Maximal Common Subgraph Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  108. 108. Ar0st Similarity •  Two Experiments (ar0st biographies from Last.fm): –  MIREX: 188 ar0sts, human judgments –  Last.fm API: 2,336 ar0sts, Last.fm similarity MIREX P@5 Last.fm API P@5 Baseline: Text-based (LSA) 0.10 0.09 Rela0on Extrac0on Graph 0.06 0.06 Enriched Graph 0.14 0.16 Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions Oramas S., Sordo M., Espinosa-Anke L., & Serra X. (2015). A Seman0c-based approach for Ar0st Similarity. ISMIR 2015.
  109. 109. •  Seman>c enrichment via En0ty Linking to build an Enriched Graph of every item •  Embed each Enriched Graph into a feature vector •  Recommend using a hybrid feature-combina0on approach –  Train a linear model for each user to predict recommenda0ons Music Recommenda0on Approach Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  110. 110. •  En0ty-based embedding •  Path-based embedding Graph Embedding Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions ϕi = (w1e1 , w2e2 , ..., wtet ) Item Graph Feature vector Weights (wi): •  Distance to the root •  Number of in-links •  Frequency and inverse frequency
  111. 111. Hybrid feature-combina0on approach Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions Train a linear model for each user-i to predict feedback on unseen items Graph features Collabora0ve features
  112. 112. •  Task: Music Recommenda0on •  Datasets –  Last.fm: song tags and stories (Songfacts) •  Last.fm listening habits •  8k items and 5k users –  Freesound.org: sound tags and descrip0ons •  Freesound downloads •  21k items and 20k users •  Support Vector Regression •  Splits: 80% train - 20% test Experiments Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions Oramas S., Ostuni V. C., Di Noia T., Serra, X., & Di Sciascio E. (2016). Music and Sound Recommenda0on with Knowledge Graphs. ACM Transac0ons on Intelligent Systems and Technology, Volume 8, Issue 2, Ar0cle 21.
  113. 113. Knowledge Graph Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  114. 114. Results: Last.fm dataset Approach P@10 ADiv@10 Collab 0.31 0.24 Collab + Tags 0.32 0.34 Collab + Enriched Graph 0.32 0.39 Enriched Graph only 0.11 0.70 Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  115. 115. Results: Freesound dataset Approach P@10 ADiv@10 Collab 0.11 0.18 Collab + Tags 0.12 0.31 Collab + Enriched Graph 0.12 0.39 Enriched Graph only 0.05 0.67 Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  116. 116. Results Approach Exploita>on Explora>on Collab Collab + Tags Collab + Enriched Graph Enriched Graph only Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  117. 117. Results: Last.fm dataset En0ty embedding Path embedding Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  118. 118. Results: Freesound dataset En0ty embedding Path embedding Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  119. 119. Conclusions Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions •  Seman0c Enrichment via En0ty Linking promotes the explora0on of long tail items
  120. 120. Conclusions Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions •  Seman0c Enrichment via En0ty Linking promotes the explora0on of long tail items •  Collabora0ve features are fundamental to obtain good ranking precision
  121. 121. Conclusions Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions •  Seman0c Enrichment via En0ty Linking promotes the explora0on of long tail items •  Collabora0ve features are fundamental to obtain good ranking precision •  The proposed hybrid feature-combina0on approach promotes less popular items with high precision
  122. 122. Conclusions Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions •  Seman0c Enrichment via En0ty Linking promotes the explora0on of long tail items •  Collabora0ve features are fundamental to obtain good ranking precision •  The proposed hybrid feature-combina0on approach promotes less popular items with high precision •  The proposed approach outperforms state-of-the-art hybrid and collabora0ve filtering algorithms
  123. 123. Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  124. 124. Representa0on Learning with Deep Learning Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  125. 125. Representa0on Learning with Deep Learning Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions Internal Representa0on
  126. 126. •  Recommenda0on of songs by novel ar0sts •  Mul0modal: Audio + Text (ar0st biographies) •  Representa0on Learning using Deep Neural Networks •  Hybrid recommenda0on approach using Matrix Factoriza0on Cold-start Music Recommenda0on Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  127. 127. Divide & Conquer The Beatles Let it be The Beatles A day in the life The Beatles Love me do Song features Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  128. 128. Divide & Conquer The Beatles Let it be The Beatles A day in the life The Beatles Love me do The Beatles Song features Ar0st features Track features Let it be A day in the life Love me do Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  129. 129. Divide & Conquer The Beatles Let it be The Beatles A day in the life The Beatles Love me do The Beatles Song features Ar0st features Track features Let it be A day in the life Love me do Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  130. 130. 1.  Aggregate feedback data by ar0st Recommenda0on Approach Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  131. 131. 1.  Aggregate feedback data by ar0st 2.  Obtain latent factors through Matrix Factoriza0on Recommenda0on Approach Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  132. 132. 1.  Aggregate feedback data by ar0st 2.  Obtain latent factors through Matrix Factoriza0on 3.  Learn ar0st representa0ons from text Recommenda0on Approach Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  133. 133. 1.  Aggregate feedback data by ar0st 2.  Obtain latent factors through Matrix Factoriza0on 3.  Learn ar0st representa0ons from text 4.  Learn song representa0ons from audio Recommenda0on Approach Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  134. 134. 1.  Aggregate feedback data by ar0st 2.  Obtain latent factors through Matrix Factoriza0on 3.  Learn ar0st representa0ons from text 4.  Learn song representa0ons from audio 5.  Fusion of mul0modal representa0ons Recommenda0on Approach Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  135. 135. Matrix Factoriza0on (WMF) M = Song Factors Users Songs User Factors Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions Songs Users d d
  136. 136. Matrix Factoriza0on (WMF) M = Song Factors Users Songs User Factors Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions Songs Users d d R = Users Ar0sts ru,a = Σ mu,t Ar0st Factors User Factors Users d Ar0sts d
  137. 137. Learning Approach Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  138. 138. Learning Approach Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  139. 139. Learning Approach Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  140. 140. Learning Approach Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  141. 141. Learning Approach Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  142. 142. Seman0c Enrichment via En0ty Linking Item Descrip0on Seman0c Enrichment Enriched Graph Enriched Descrip0on En0ty Linking Knowledge Base Seman0c Enrichment via En0ty Linking Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  143. 143. Seman0c Enrichment via En0ty Linking Item Descrip0on Seman0c Enrichment Enriched Graph Enriched Descrip0on En0ty Linking Knowledge Base Seman0c Enrichment via En0ty Linking Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  144. 144. Seman0c Enrichment via En0ty Linking Elvis Presley covered Guitar Man with Reed hdps://en.wikipedia.org/wiki/Jerry_Reed hdps://en.wikipedia.org/wiki/Elvis_Presley hdps://en.wikipedia.org/wiki/Guitar_Man_(song) Singers from Tennessee American rockabilly musicians 1935 births American country singer-songwriters American country guitarists Musicians from Atlanta 1967 singles Jerry Reed songs RPM Country Tracks number-one singles Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  145. 145. Text-based Approach: Enriched Descrip0on Singers_from_Tennessee,American_ro c k a b i l l y _ m u s i c i a n s , 1935_births,American_country_singer songwriters,American_country_guitari sts,Musicians_from_Atlanta,20th- century_American_male_actors,Gram my_Award_winners Item Descrip0on Seman0c data + VSM z-idf Singer, songwriter and cer0fied guitar player Jerry Reed found his musical calling as a child. It's interes0ng, albeit a bit disconcer0ng, to hear Reed singing so far outside his earthier country sound, and the folk- and pop- flavored cuts haven't the swagger of his blues. Elvis Presley covered Guitar Man, with Reed reproducing the guitar break from this recording. Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  146. 146. Text-based Learning Approach Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  147. 147. •  Constant-Q audio spectrograms with 96 frequency bins •  4 convolu0onal and max pooling layers •  Time domain filters (Van Den Oord et al. 2013) •  No dense layers Audio-based Learning Approach Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  148. 148. •  l2-norm •  Concatenate representa0ons •  Linear model Mul0modal Fusion Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  149. 149. •  MSD-A –  330k tracks –  24k ar0sts –  1M users Dataset Ar0sts biographies and tags + Audio and user feedback Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  150. 150. •  Ar0st Recommenda0on –  Text-based approaches •  Song Recommenda0on –  Audio, text, and mul0modal approaches •  Splits: 80 train - 10 valida0on – 10 test •  Different ar>sts in each subset •  Evalua0on metric: MAP@500 Experiments Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions Oramas, S., Nieto O., Sordo M., & Serra X. (2017). A Deep Mul0modal Approach for Cold-start Music Recommenda0on. DLRS-RecSys 2017.
  151. 151. Results: Ar0st Recomenda0on Input Approach MAP Text VSM-FF 0.016 Enriched Text VSM-FF 0.020 Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  152. 152. Results: Ar0st Recommenda0on Input Approach MAP Text VSM-FF 0.016 Enriched Text VSM-FF 0.020 Text (Kim 2014) w2v-CNN 0.015 Baseline: Text Random Forest 0.009 Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  153. 153. Results: Ar0st Recomenda0on Input Approach MAP Text VSM-FF 0.016 Enriched Text VSM-FF 0.020 Text (Kim 2014) w2v-CNN 0.015 Baseline: Text Random Forest 0.009 Tags VSM-FF 0.031 Baseline: Tags itemAdributeKnn 0.016 Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  154. 154. Results: Ar0st Recomenda0on Input Approach MAP Text VSM-FF 0.016 Enriched Text VSM-FF 0.020 Text (Kim 2014) w2v-CNN 0.015 Baseline: Text Random Forest 0.009 Tags VSM-FF 0.031 Baseline: Tags itemKnn 0.016 Random - 0.001 Upper-bound - 0.553 Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  155. 155. Results: Song Recommenda0on Input Approach MAP Audio CNN 0.0015 Enriched Text VSM-FF 0.0032 Ar>st Representa>on MLP 0.0034 Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  156. 156. Results: Song Recommenda0on Input Approach MAP Audio CNN 0.0015 Enriched Text VSM-FF 0.0032 Ar0st Representa0on MLP 0.0034 Audio + Enriched Text CNN + VSM-FF 0.0014 Song Repr. + Ar>st Repr. MLP 0.0036 Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  157. 157. Results: Song Recommenda0on Input Approach MAP Audio CNN 0.0015 Enriched Text VSM-FF 0.0032 Ar0st Representa0on MLP 0.0034 Audio + Enriched Text CNN + VSM-FF 0.0014 Song Repr. + Ar>st Repr. MLP 0.0036 Random - 0.0002 Upper-bound - 0.1649 Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  158. 158. •  Aggrega0on of ar0st data led to improved ar0st representa0ons Conclusions Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  159. 159. •  Aggrega0on of ar0st data led to improved ar0st representa0ons •  Seman0c enrichment via En0ty Linking improves performance of text-based recommenda0on Conclusions Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  160. 160. •  Aggrega0on of ar0st data led to improved ar0st representa0ons •  Seman0c enrichment via En0ty Linking improves performance of text-based recommenda0on •  Mul0modal fusion of data representa0ons improves single modali0es in isola0on and fully mul0modal networks Conclusions Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  161. 161. Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  162. 162. Music Genre Classifica0on Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  163. 163. Music Genre Classifica0on is a widely studied problem* Music Genre Classifica0on Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  164. 164. Music Genre Classifica0on is a widely studied problem but only with these characteris0cs: -  Audio-based -  HandcraZed features -  Single-label -  Few broad genres Music Genre Classifica0on Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  165. 165. Mul0modal Learned features Mul0-label Hundreds of genres Music Genre Classifica0on Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions Music Genre Classifica0on is a widely studied problem but only with these characteris0cs: -  Audio-based -  HandcraZed features -  Single-label -  Few broad genres
  166. 166. Mul0modal Learned features Mul0-label Hundreds of genres Music Genre Classifica0on Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions Music Genre Classifica0on is a widely studied problem but only with these characteris0cs: -  Audio-based -  HandcraZed features -  Single-label -  Few broad genres
  167. 167. Mul0modal Learned features Mul0-label Hundreds of genres Music Genre Classifica0on Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions Music Genre Classifica0on is a widely studied problem but only with these characteris0cs: -  Audio-based -  HandcraZed features -  Single-label -  Few broad genres
  168. 168. •  MARD dataset –  1300 albums –  13 genres –  Amazon reviews + Acous0cBrainz audio features •  Features: –  Textual: VSM of uni-grams and bi-grams –  Seman>c: Enriched descrip0on (Wikipedia categories) –  Acous>c: Low-level audio features •  Classifier –  SVM 5-fold cross valida0on Single-label Music Genre Classifica0on Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions Oramas S., Espinosa-Anke L., Lawlor A., Serra X., & Saggion H. (2016). Exploring Music Reviews for Music Genre Classifica0on and Evolu0onary Studies. ISMIR 2016.
  169. 169. Single-label Music Genre Classifica0on Accuracy Baseline: Text-based 62.9 Enriched Descrip>on 69.1 Text-based (Hu 2006) 55.0 Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  170. 170. Single-label Music Genre Classifica0on Accuracy Baseline: Text-based 62.9 Enriched Descrip>on 69.1 Text-based (Hu 2006) 55.0 Audio features 38.7 Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  171. 171. Single-label Music Genre Classifica0on Audio Text Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  172. 172. Single-label Music Genre Classifica0on Audio Text Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  173. 173. Single-label Music Genre Classifica0on Audio Text Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  174. 174. Mul0modal Learned features Mul0-label Hundreds of genres Music Genre Classifica0on Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions Music Genre Classifica0on is a widely studied problem but only with these characteris0cs: -  Audio-based -  HandcraZed features -  Single-label -  Few broad genres
  175. 175. Logis0c output Mul0-label Classifica0on with Deep Learning Output layer: -  1 neuron per label -  Sigmoid ac0va0on -  Cross entropy loss Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions GENRE LABELS
  176. 176. Logis0c output Mul0-label Classifica0on with Deep Learning Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions GENRE LABELS Output layer: -  1 neuron per label -  Sigmoid ac0va0on -  Cross entropy loss Cons: -  Assump0on of mutual independence of labels -  High dimensionality
  177. 177. PMI Factoriza0on (Chollet 2016) Dimensionality reduc0on Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  178. 178. Regression output Mul0-label Classifica0on with Deep Learning Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions LATENT FACTORS Output layer: -  1 neuron per latent dim -  Linear ac0va0on -  Cosine proximity loss
  179. 179. •  MuMu Dataset –  31k albums with •  Cover art images •  150k audio tracks •  450k album reviews –  Mul0-label genre annota0ons of 250 genres Dataset Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions Oramas, S., Nieto O., Barbieri F., & Serra X. (2017). Mul0-label Music Genre Classifica0on from Audio, Text and Images Using Deep Features. ISMIR 2017.
  180. 180. •  Area under the ROC curve for every genre (AUC) •  Aggregated diversity (ADiv@N) of genre predic0ons Different genres in Top-N lists / Total number of genres Evalua0on metrics Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  181. 181. •  Constant-Q spectrograms with 96 frequency bins •  4 convolu0onal layers •  No dense layers Audio-based approach Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  182. 182. •  Filter sizes –  3x3 (Choi 2016) –  4x96 (Van Den Oord 2013) –  4x70 (Pons 2016) •  Number of filters –  HIGH: 256/512/1024/1024 –  LOW: 64/128/128/64 •  Output layer –  GENRE LABELS –  LATENT FACTORS Audio-based approach Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  183. 183. •  Filter sizes –  3x3 (Choi 2016) –  4x96 (Van Den Oord 2013) –  4x70 (Pons 2016) •  Number of filters –  HIGH: 256/512/1024/1024 –  LOW: 64/128/128/64 •  Output layer –  GENRE LABELS –  LATENT FACTORS Audio-based approach Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  184. 184. •  Filter sizes –  3x3 (Choi 2016) –  4x96 (Van Den Oord 2013) –  4x70 (Pons 2016) •  Number of filters –  HIGH: 256/512/1024/1024 –  LOW: 64/128/128/64 •  Output layer –  GENRE LABELS –  LATENT FACTORS Audio-based approach Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  185. 185. •  Filter sizes –  3x3 (Choi 2016) –  4x96 (Van Den Oord 2013) –  4x70 (Pons 2016) •  Number of filters –  HIGH: 256/512/1024/1024 –  LOW: 64/128/128/64 •  Output layer –  GENRE LABELS –  LATENT FACTORS Audio-based approach Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  186. 186. Audio-based classifica0on results Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions Target Input AUC ADiv@1 GENRE LABELS Baseline: Audio features 0.792 0.04 GENRE LABELS CQT + CNN 0.871 0.05
  187. 187. Audio-based classifica0on results Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions Target Input AUC ADiv@1 GENRE LABELS Baseline: Audio features 0.792 0.04 GENRE LABELS CQT + CNN 0.871 0.05 LATENT FACTORS CQT + CNN 0.888 0.35
  188. 188. Qualita0ve Analysis LATENT FACTORS LABELS Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  189. 189. Qualita0ve Analysis LATENT FACTORS LABELS Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  190. 190. Text-based Approach Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  191. 191. Text-based classifica0on results Target Input AUC ADiv@1 GENRE LABELS Text 0.905 0.08 GENRE LABELS Enriched Text 0.916 0.10 Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  192. 192. Text-based classifica0on results Target Input AUC ADiv@1 GENRE LABELS Text 0.905 0.08 GENRE LABELS Enriched Text 0.916 0.10 LATENT FACTORS Enriched Text 0.917 0.42 Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  193. 193. Informa0on Gain Rock Pop Metal Hip Hop Country band song metal hip country rock songs death hop Nashville punk euro band rap her bands trade black rhymes Waylon Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  194. 194. •  Deep Residual Networks (ResNets) •  101 layers •  Pretrained on ImageNet •  Fine-tuning in our task Image-based approach Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  195. 195. Mul0modal classifica0on results Modality AUC Audio 0.888 Text 0.917 Images 0.743 Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  196. 196. Mul0modal classifica0on results Modality AUC Audio 0.888 Text 0.917 Images 0.743 A + T 0.930 A + I 0.900 T + I 0.921 Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  197. 197. Mul0modal classifica0on results Modality AUC Audio 0.888 Text 0.917 Images 0.743 A + T 0.930 A + I 0.900 T + I 0.921 Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  198. 198. Mul0modal classifica0on results Modality AUC Audio 0.888 Text 0.917 Images 0.743 A + T 0.930 A + I 0.900 T + I 0.921 Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  199. 199. Mul0modal classifica0on results Modality AUC Audio 0.888 Text 0.917 Images 0.743 A + T 0.930 A + I 0.900 T + I 0.921 A + T + I 0.936 Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  200. 200. t-SNE of visual features Qualita0ve analysis Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  201. 201. t-SNE of visual features Qualita0ve analysis Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  202. 202. t-SNE of visual features Qualita0ve analysis Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  203. 203. t-SNE of visual features Qualita0ve analysis Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions Features: Color faces Hair Background Clothes Instruments Typographies
  204. 204. Qualita0ve analysis: Aden0on heatmaps RnB Pop RnB Electronic Country Country Pop Folk Jazz Jazz Blues Reggae
  205. 205. •  Representa0on learning beder than handcraZed features Conclusions Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  206. 206. •  Representa0on learning beder than handcraZed features •  Seman0c enrichment improves text classifica0on Conclusions Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  207. 207. •  Representa0on learning beder than handcraZed features •  Seman0c enrichment improves text classifica0on •  Text achieves best single modality results Conclusions Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  208. 208. •  Representa0on learning beder than handcraZed features •  Seman0c enrichment improves text classifica0on •  Text achieves best single modality results •  Audio is nearer Text performance thanks to deep learning Conclusions Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  209. 209. •  Representa0on learning beder than handcraZed features •  Seman0c enrichment improves text classifica0on •  Text achieves best single modality results •  Audio is nearer Text performance thanks to deep learning •  Fusion of learned data representa0ons improves results Conclusions Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  210. 210. •  Representa0on learning beder than handcraZed features •  Seman0c enrichment improves text classifica0on •  Text achieves best single modality results •  Audio is nearer Text performance thanks to deep learning •  Fusion of learned data representa0ons improves results •  Dimensionality reduc0on yields beder accuracy and diversity Conclusions Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  211. 211. Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  212. 212. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  213. 213. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on •  Knowledge from seman0c repositories incorporated via En0ty Linking improves item profiles -> higher diversity (long tail) Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  214. 214. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on •  Knowledge from seman0c repositories incorporated via En0ty Linking improves item profiles -> higher diversity (long tail) •  Learning and combining data representa0ons from mul0modal data -> accurate cold-start recommenda0ons Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  215. 215. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on •  Knowledge from seman0c repositories incorporated via En0ty Linking improves item profiles -> higher diversity (long tail) •  Learning and combining data representa0ons from mul0modal data -> accurate cold-start recommenda0ons Hybrid approaches + Seman0c Enrichment + Representa0on learning beder Long tail Cold-start Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  216. 216. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  217. 217. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on •  Knowledge from seman0c repositories incorporated via En0ty Linking improves item descrip0ons -> higher accuracy Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  218. 218. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on •  Knowledge from seman0c repositories incorporated via En0ty Linking improves item descrip0ons -> higher accuracy •  Learning and combining data representa0ons from mul0modal data -> higher accuracy and diversity Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  219. 219. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on -  Audio-based -  HandcraZed features -  Single-label -  Few broad genres Mul0modal Learned features Mul0-label Hundreds of genres •  Knowledge from seman0c repositories incorporated via En0ty Linking improves item descrip0ons -> higher accuracy •  Learning and combining data representa0ons from mul0modal data -> higher accuracy and diversity Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  220. 220. •  Approach for the automated crea0on of Music KBs Contribu0ons Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  221. 221. •  Approach for the automated crea0on of Music KBs •  Methodology for the seman0c enrichment of text Contribu0ons Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  222. 222. •  Approach for the automated crea0on of Music KBs •  Methodology for the seman0c enrichment of text •  Hybrid and knowledge-based recommenda0on approach that promotes long tail recommenda0ons Contribu0ons Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  223. 223. •  Approach for the automated crea0on of Music KBs •  Methodology for the seman0c enrichment of text •  Hybrid and knowledge-based recommenda0on approach that promotes long tail recommenda0ons •  Mul0modal deep learning approach for cold-start recommenda0ons Contribu0ons Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  224. 224. •  Approach for the automated crea0on of Music KBs •  Methodology for the seman0c enrichment of text •  Hybrid and knowledge-based recommenda0on approach that promotes long tail recommenda0ons •  Mul0modal deep learning approach for cold-start recommenda0ons •  Mul0modal deep learning approach for mul0-label music genre classifica0on Contribu0ons Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  225. 225. •  Development of data-driven methodologies that have the poten0al to help musicologists to discover new hypothesis from text –  Relevance of ar0sts –  Diachronic studies of tendencies and evolu0on of genres Computa0onal Musicology Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  226. 226. •  Development of data-driven methodologies that have the poten0al to help musicologists to discover new hypothesis from text –  Relevance of ar0sts –  Diachronic studies of tendencies and evolu0on of genres •  Crea0on of a flamenco music Knowledge Base Computa0onal Musicology Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  227. 227. •  Crea0on of the “Music meets NLP” research project –  Promo0on of intersec0on of MIR and NLP communi0es –  High number of publica0ons –  Release of several linguis0c resources –  Organiza0on of tutorials and a challenges Outcomes Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  228. 228. Outcomes Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  229. 229. Outcomes hdp://musicbrainz.org/release-group/2536a41d-fde9-35d5-a6c6-cd4d94ffd916 hdp://musicbrainz.org/ar0st/9472e6e4-3e13-430a-900d-6f075720a5c6 Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  230. 230. •  6 Datasets –  ELMD •  Task: Music En0ty Linking •  Content: Annotated documents –  MARD •  Task: Music Genre Classifica0on •  Content: Customer reviews and acous0c features –  SAS •  Task: Ar0st Similarity •  Content: Ar0st biographies and similarity ground truth –  KG-Rec •  Task: Sound and Music Recommenda0on •  Content: User feedback, item descrip0ons and tags Reproducibility Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  231. 231. –  MSD-A •  Task: Music Recommenda0on •  Content: User feedback, ar0st biographies and tags, audio tracks –  MuMu •  Task: Mul0-label Music Genre Classifica0on •  Content: Customer reviews, audio tracks, album cover art •  Sogware –  ELVIS •  En0ty Linking Integra0on System –  TARTARUS •  Mul0modal deep learning framework for recommenda0on and classifica0on Reproducibility Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  232. 232. •  5 Peer-reviewed journals –  Oramas S., Espinosa-Anke L., & Serra X. (Submided). Knowledge Extrac0on for Musicology. Journal of New Musical Research. –  Oramas S., Barbieri F., Nieto O., Serra X. (Submided). Learning and Combining Mul0modal Data Representa0ons for Music Genre Classifica0on. Transac0ons of the Interna0onal Society for Music Informa0on Retrieval. –  Oramas S., Espinosa-Anke L., Sordo M., Saggion H. & Serra X. (2016). Informa0on Extrac0on for Knowledge Base Construc0on in the Music Domain. Data & Knowledge Engineering, Volume 106, Pages 70-83. –  Oramas S., Ostuni V. C., Di Noia T., Serra, X., & Di Sciascio E. (2016). Music and Sound Recommenda0on with Knowledge Graphs. ACM Transac0ons on Intelligent Systems and Technology, Volume 8, Issue 2, Ar0cle 21. –  Oramas S., & Sordo M. (2016). Knowledge is Out There: A New Step in the Evolu0on of Music Digital Libraries. Fontes Ar0s Musicae, Vol 63, no. 4. Publica0ons Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  233. 233. •  14 Peer-reviewed conference papers –  Oramas, S., Nieto O., Barbieri F., & Serra X. (2017). Mul0-label Music Genre Classifica0on from Audio, Text and Images Using Deep Features. ISMIR 2017. –  Oramas, S., Nieto O., Sordo M., & Serra X. (2017). A Deep Mul0modal Approach for Cold-start Music Recommenda0on. DLRS-RecSys 2017. –  Espinosa-Anke, L., Oramas S., Saggion H., & Serra X. (2017). ELMDist: A vector space model with words and MusicBrainz en00es. ESWC 2017. –  Oramas S., Espinosa-Anke L., Lawlor A., Serra X., & Saggion H. (2016). Exploring Music Reviews for Music Genre Classifica0on and Evolu0onary Studies. ISMIR 2016. –  Oramas S., Espinosa-Anke L., Sordo M., Saggion H., & Serra X. (2016). ELMD: An Automa0cally Generated En0ty Linking Gold Standard in the Music Domain. LREC 2016. –  Espinosa-Anke, L., Oramas S., Camacho-Collados J., & Saggion H. (2016). Finding and Expanding Hypernymic Rela0ons in the Music Domain. CCIA 2016. Publica0ons Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  234. 234. –  Oramas S., Sordo M., Espinosa-Anke L., & Serra X. (2015). A Seman0c-based approach for Ar0st Similarity. ISMIR 2015. –  Oramas S., Gómez F., Gómez E., & Mora J. (2015). FlaBase: Towards the crea0on of a Flamenco Music Knowledge Base. ISMIR 2015. –  Ostuni V. C., Oramas S., Di Noia T., Serra, X., & Di Sciascio E. (2015). A Seman0c Hybrid Approach for Sound Recommenda0on. WWW 2015. –  Oramas S., Sordo M., & Espinosa-Anke L. (2015). A Rule-based Approach to Extrac0ng Rela0ons from Music Tidbits. KET-WWW 2015. –  Sordo, M., Oramas S., & Espinosa-Anke L. (2015). Extrac0ng Rela0ons from Unstructured Text Sources for Music Recommenda0on. NLDB 2015. –  Oramas S., Sordo M., & Serra X. (2014). Automa0c Crea0on of Knowledge Graphs from Digital Musical Document Libraries. CIM 2014. –  Oramas S. (2014). Harves0ng and Structuring Social Data in Music Informa0on Retrieval. ESWC 2014. –  Font, F., Oramas, S., Fazekas, G., & Serra, X. (2014). Extending Tagging Ontologies with Domain Specific Knowledge. ISWC 2014. Publica0ons Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  235. 235. •  Other conference presenta>ons –  Oramas, S. (2017). Knowledge Extrac0on and Feature Learning for Music Recommenda0on in the Long Tail. 5th Large Scale Recommenda0on Systems Workshop, co-located with RecSys 2017, Como, Italy. –  Oramas, S. (2017). Discovering Similari0es and Relevance Ranking of Renaissance Composers. The 63rd Annual Mee0ng of the Renaissance Society of America (RSA), Chicago. –  Oramas S. (2015). Informa0on Extrac0on for the Music Domain. The 2nd Interna0onal Workshop on Human History Project: Natural Language Processing and Big Data, CIRMMT, Montreal. –  Oramas, S., & Sordo M. (2015). Knowledge Acquisi0on from Music Digital Libraries. The Interna0onal Associa0on of Music Libraries and Interna0onal Musicological Society Conference (IAML/IMS 2015), New York. Publica0ons Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  236. 236. •  Tutorials and Challenges –  Camacho-Collados J., Delli Bovi C., Espinosa-Anke L., Oramas S., Pasini T., Shwartz V., Santus E., Saggion H., Navigli R. (In press) Task 9: Hypernym Discovery. SemEval 2018. –  Speck R., Röder M., Oramas S., Espinosa-Anke L., & Ngomo A. C. N. (2017). Open Knowledge Extrac0on Challenge 2017. ESWC 2017. –  Oramas S., Espinosa-Anke L., Zhang S., Saggion H., & Serra X. (2016). Natural Language Processing for Music Informa0on Retrieval. ISMIR 2016. •  Prizes –  Best oral presenta0on award at ISMIR 2017 –  Best paper award at SemDeep-ESWC 2017 –  Best poster award at CCIA 2016 –  Maria de Maeztu Research reproducibility award 2016 Publica0ons Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  237. 237. Acknowledgements Knowledge Extrac0on | Recommenda0on | Classifica0on | Conclusions
  238. 238. Thanks!
  239. 239. Knowledge Extrac0on and Representa0on Learning for Music Recommenda0on and Classifica0on Sergio Oramas Mar>n Doctoral Thesis Defense Departament of Informa0on and Communica0on Technologies Thesis Director: Dr. Xavier Serra Wednesday, November 29th, 2017 Thesis Board: Dr. Markus Schedl Dr. Emilia Gómez Dr. Brian Whitman

×