Tutorial on Music Recommendation by Oscar Celma (Gracenote) and Paul Lamere (The Echo Nest). The world of music is changing rapidly. We are now just a few clicks away from being able to listen to nearly any song that has ever been recorded. This easy access to a nearly endless supply of music is changing how we explore, discover, share and experience music. As the world of online music grows, music recommendation and discovery tools become an increasingly important way for music listeners to engage with music. Commercial recommenders such as Last.fm, iTunes Genius and Pandora have enjoyed commercial and critical success. But how well do these systems really work? How good are the recommendations? How far into the "long tail" do these recommenders reach? In this tutorial we look at the current state-of-the-art in music recommendation and discovery. We examine current commercial and research systems, focusing on the advantages and the disadvantages of the various recommendation strategies. We look at some of the challenges in building music recommenders and we explore some of the novel techniques that are being used to improve future music recommendation and discovery systems. Òscar Celma is the Chief Innovation Officer at Barcelona Music and Audio Technologies (BMAT). In 2008, Òscar obtained his Ph.D. in Computer Science and Digital Communication, in the Pompeu Fabra University (Barcelona, Spain). Òscar has a book published by Springer, titled "Music Recommendation and Discovery: The Long Tail, Long Fail and Long Play in the Music Digital Age" (2010). He holds 2 patents (US2003009344 and JP2003323188, 2002) from his work on the Vocaloid system, a singing voice-synthesizer bought by Yamaha in 2004. Follow on Twitter: @ocelma Paul Lamere is the Director of Developer Platform for The Echo Nest, a music intelligence company located in Boston. Paul is interested in using technology to help people explore for new and interesting music. He is active in both the music information retrieval and the recommender systems research communities. Paul authors a popular blog on music technology at MusicMachinery.com. Follow on Twitter: @plamere

1. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Music Recommendation and Discovery
Remastered
Tutorial
@recsys, 2011

2. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
@plamere @ocelma

3. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
INTRO CHORUS VERSE BRIDGE OUTRO

4. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
INTRO CHORUS VERSE BRIDGE OUTRO

5. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Music Recommendation is important
How many songs fit in my pocket?
10 Songs 1,000 Songs 10,000,000 Songs
1979 2001 2011

6. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
What's so special about music?
● Huge item space
● Very low cost per item
● Many item types
● Low consumption time
● Very high per-item reuse
● Highly passionate users
● Highly contextual usage
● Consumed in sequences
● Large personal collections
● Doesn't require our full attention
● Highly Social

7. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Music recommendation is broken ...
If you like Britney Spears
you might like...
...Report on Pre-War Intelligence
Let's look at some of the
issues ....

8. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
What makes a good music recommendation?
● Relevance
● Novelty / Serendipity
● Transparency / Trust
● Reach
● Context

9. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
What makes a good music recommendation?
● Relevance
● Novelty / Serendipity
● Transparency / Trust
● Reach
● Context

10. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Relevance

11. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Relevance – cold start
new or unpopular items
If you like Gregorian Chants you might like Green Day

12. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Cold Start – New User - Enrollment

13. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
New User – Implicit taste data
The Audioscrobbler

14. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Relevance – Metadata Mismatches

15. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Relevance – Metadata Mismatches
Why?

16. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Relevance - The grey sheep problem

17. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Relevance – Cultural Mismatches

18. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
What makes a good music recommendation?
● Relevance
● Novelty / Serendipity
● Transparency / Trust
● Reach
● Context

19. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Novelty and Serendipity

20. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Popularity Bias - The Harry Potter Effect

21. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
...also known as the Coldplay effect

22. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Novelty / Serendipity – the enemy
High stakes competitions focused on relevance can
reduce novelty and serendipity

23. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
What makes a good music recommendation?
● Relevance
● Novelty / Serendipity
● Transparency / Trust
● Reach
● Context

24. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
The Opacity Problem
“If you like NiN you might like Johnny Cash”
Why???

25. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Is this a good recommendation?
If you like Norah Jones ...
You might like Ravi Shankar

26. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Is this a good recommendation?
If you like Norah Jones ...
You might like her father, Ravi Shankar

27. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Brutal Death Metal Quiz
???????
Photo cc by Mithrandir3

28. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Brutal Death Metal Quiz

29. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Hacking the recommender

30. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
What makes a good music recommendation?
● Relevance
● Novelty / Serendipity
● Transparency / Trust
● Reach
● Context

31. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Help! I’m stuck in the head
The limited reach of music recommendation
48% of recommendations
Popularity
0% of
recommendations
52% of recommendations
83 Artists 6,659 Artists 239,798 Artists
Study by Dr. Oscar Celma - MTG UPF
Sales Rank

32. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Music Discovery Challenge
Personal discovery a challenge too
Listener Study
Listeners 5,000
Average Songs
3,500
Per User
Percent of
songs never 65%
listened to

33. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
What makes a good music recommendation?
● Relevance
● Novelty / Serendipity
● Transparency / Trust
● Reach
● Context

34. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Music Recommendation is not just
shopping
● It is not just for shopping, but...
● Discovery
● Exploration
● Play
● Organization
● Playlisting
● Recommendation for groups
● Devices
● Doesn't have to look like a spreadsheet!

35. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Context: Tools for exploration
Ishkur's Guide to Electronic Dance Music
http://techno.org/electronic-music-guide/

36. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
INTRO CHORUS VERSE BRIDGE OUTRO

37. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma

38. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma

39. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Ingestion process
● Input data source
● Own data, Customer, Labels, UGC, ...
● Protocol
● Ingestion format
– TSV, XML, DDEX, XLS!, …
● Method
– FTP, API, ...
● Frequency
– Offline processing: Daily / weekly?
– Data freshness!
● Documentation

40. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Ingestion process
● Post-processing
● Data cleaning: Duplicates, normalization
● Allow customer to use its own Ids!
● Add links to external sources
● Rosetta Stone

41. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Ingestion process
● Considerations
● Allow customer to use its own IDs when using the
rec. system.
● How long does it take to process the whole
collection?
● Incremental updates

42. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma

43. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma

44. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
music recommendation approaches
● Expert-based
● Collaborative filtering
● Social-based
● Content-based
● Hybrid (combination)

45. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
music recommendation approaches
● Expert-based
“X similar to (or influenced by) Y”
Editorial metadata (Genre, Decades, Location, …)
Music Genome
● Collaborative filtering
● Social-based
● Content-based
● Hybrid (combination)

46. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
music recommendation approaches
● Expert-based
● Collaborative filtering
“people who listen to X also listen to Y”
● Social-based
● Content-based
● Hybrid (combination)

47. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
music recommendation approaches
● Expert-based
● Collaborative filtering
“people who listen to X also listen to Y”
● Social-based
● Content-based
● Hybrid (combination)

48. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
music recommendation approaches
● Expert-based
● Collaborative filtering
“people who listen toRawalso listen to Y”
X plays:
● Social-based
● Content-based
● Hybrid (combination)

49. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
music recommendation approaches
● Expert-based
● Collaborative filtering
“people who listen toRawalso listen to Y”
X plays:
● Social-based Normalize to [5..1]
● Content-based
● Hybrid (combination)

50. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
music recommendation approaches
● Expert-based
● Collaborative filtering
“people who listen toRawalso listen to Y”
X plays:
● Social-based Normalize to [5..1]
Probability distribution:
● Content-based 0.32 , 0.24 , 0.18 , 0.1 , 0.07 , 0.04, 0.03 , 0.02
● Hybrid (combination)

51. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
music recommendation approaches
● Expert-based
● Collaborative filtering
“people who listen toRawalso listen to Y”
X plays:
● Social-based Normalize to [5..1]
Probability distribution:
● Content-based 0.32 , 0.24 , 0.18 , 0.1 , 0.07 , 0.04, 0.03 , 0.02
● Hybrid (combination) Binary:
100100000100000011000001

52. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
music recommendation approaches
● Expert-based
● Collaborative filtering
Matrix Factorization. E.g: SVD, NMF, ...
● Social-based
● Content-based
● Hybrid (combination)

53. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
music recommendation approaches
● Expert-based
● Collaborative filtering
● Social-based
●
WebMIR [Schedl, 2008]
Content Reviews Lyrics Blogs Social Tags Bios Playlists
● Content-based
● Hybrid (combination)

54. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
music recommendation approaches
● Expert-based
● Collaborative filtering
● Social-based
● Content-based
“X and Y sound similar”
● Hybrid

55. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
music recommendation approaches
● Expert-based
● Collaborative filtering
● Social-based
● Content-based
Audio features
– Bag-of-frames (MFCC) [Aucouturier, 2004], Rhythm [Gouyon,
2005], Harmony [Gomez, 2006], ...
Similarity
– KL-divergence: GMM [Aucouturier, 2002]
– EMD [Logan, 2001]
– Euclidean: PCA [Cano, 2005]
– Cosine: mean/var (feature vectors)
– Ad-hoc

56. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
music recommendation approaches
● Expert-based
● Collaborative filtering
● Social-based
● Content-based
Audio features
– Bag-of-frames (MFCC) [Aucouturier, 2004], Rhythm [Gouyon,
2005], Harmony [Gomez, 2006], ...
Similarity
– KL-divergence: GMM [Aucouturier, 2002]
– EMD [Logan, 2001]
– Euclidean: PCA [Cano, 2005]
http://xkcd.com/26/
– Cosine: mean/var (feature vectors)
– Ad-hoc

57. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
music recommendation approaches
● Expert-based
● Collaborative filtering
● Social-based
● Content-based
● Hybrid
Weighted (linear combination)
– E.g CF * 0.2 + CT * 0.4 + CB * 0.4
Cascade
– E.g 1st apply CF, then reorder by CT or CB
Switching
...

58. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma

59. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma

60. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Search
● Metadata search
Bruce*
● Using filters: “Popular Irish bands from the 80s”
popularity:[8.0 TO 10.0] AND
iso_country:IE AND decade:1980
● Audio search (and similarity)
● Query by example

61. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Similarity

62. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Similarity
? ? ?
Using Last.fm-360K dataset

63. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Similarity
Using Last.fm-360K dataset

64. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Similarity' (include feedback)
Using Last.fm-360K dataset

65. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Beyond similarity

66. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Recommendation
● “If Paul likes Radiohead he might also like X”
vs.
● “If Oscar likes Radiohead he might also like Y”

67. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Recommendation
● “If Paul likes Radiohead he might also like X”
vs.
● “If Oscar likes Radiohead he might also like Y”
SIMILARITY != RECOMMENDATION

68. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Recommendation
● To whom are we recommending? Phoenix-2 (UK, 2006)

69. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
lamere @ last.fm

70. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
mini-lamere's @ last.fm
●
Clustering (k-means) lamere top-50 artists

71. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
mini-lamere's @ last.fm
●
Clustering (k-means) lamere top-50 artists

72. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
@lamere - Radiohead

73. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
@lamere - Radiohead
Vs. Radiohead similar artists...

74. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
@lamere - Radiohead
Vs. Radiohead similar artists...

75. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
@lamere - Radiohead
Vs. Radiohead similar artists...

76. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
@lamere - Radiohead
Vs. Radiohead similar artists...

77. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Personalization (Itemization?)
● ...but also which Radiohead era?

78. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Analytics
● Big data processing
● capture, storage, search, share, analysis and
visualization
● (local) Trend detection
● Tastemakers
● ...

79. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
INTRO CHORUS VERSE BRIDGE OUTRO

80. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Real-world Music Recommendation

81. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Real-world Music Recommendation

82. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Contextual Web Crawl

83. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Audio Processing

84. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Hybrid Recommendation

85. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
● 100 million registered users
● 37 million active monthly users
● More than 900,000 songs in catalog
● More than 90,000 artists in catalog
● More than 11 billion thumbs
● More than 1.9 billion stations
● 95% of the collection was played in July 2011

86. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Curation and Analysis

87. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Weighting vectors

88. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
For unknown artists

89. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
For popular artists

90. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma

91. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma

92. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Country: UK
Record Labels: Acid Jazz, Sony BMG, Columbia
Genres: Funk, Disco, Acid Jazz, Jazz Fusion, Pop-rock
Years active: 1992 - present
Associated acts: Brand New Heavies, Guru, Julian Perretta

93. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Country: UK
Record Labels: Acid Jazz, Sony BMG, Columbia
Genres: Funk, Disco, Acid Jazz, Jazz Fusion, Pop-rock
Years active: 1992 - present
Associated acts: Brand New Heavies, Guru, Julian Perretta
Mood: upbeat, energetic
Rhythm: 120bpm, no rubato, high percusiveness
Key: Dm
Tags: acid jazz funk dance
Sounds like: Sereia (Tiefschwarz Radio Edit) by Mundo Azul

94. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
“I want some upbeat songs from unknown US bands,
similar to Radiohead“
http://ella.bmat.ws/collections/bmat/artists/radiohead/similar/tracks
?filter=mood:happy
+speed:fast
+iso_country:US
+popularity:[0.0+TO+4.0]

95. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
INTRO CHORUS VERSE BRIDGE OUTRO

96. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Evaluation
"The key utility measure is user happiness. It seems
reasonable to assume that relevance of the results is
the most important factor: blindingly fast, useless
answers do not make a user happy."
– "Introduction to Information Retrieval"
(Manning, Raghavan, and Schutze, 2008)

97. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
RMSE

98. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
RMSE?

99. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
NO RMSE

100. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
NO RMSE
(in music)

101. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Evaluation
● Limitations of current metrics (RMSE, P/R, ROC,
Spearman Rho, Kendall Tau, etc.)
● skewness
– performed on test data that users chose to rate
● do not take into account
– usefulness
– novelty / serendipity
– topology of the (item or user) similarity graph
– ...

102. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Evaluation
If no RMSE then...?
● Predictive Accuracy vs. Perceived Quality
● Does the recommendation help the user? (user
satisfaction)
● Familiarity vs. Novelty
● Does the recommendation help the system?
● $$$
● Catalog exposure

103. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
NEXT SONG?

104. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
NEXT SONG?
?
Mean Reciprocal Rank
+
User feedback

105. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
? ?

106. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma

107. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Novelty & Relevance

108. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Novelty & Relevance

109. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Novelty & Relevance

110. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Novelty & Relevance

111. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Novelty & Relevance
WTF?

112. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Novelty & Relevance
Emitt Rhodes

113. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Novelty & Relevance
WHY as important as WHAT

114. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Novelty & Relevance
WHY as important as WHAT

115. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Novelty & Relevance
WHY as important as WHAT

116. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Novelty & Relevance
WHY as important as WHAT

117. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
WTF

118. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Other evaluation techniques
rd
How can I evaluate a 3 party recommender:
objective measures:
coverage, reach
subject measures:
Focus on precision
Measure irrelevant results: The WTF test

119. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
The WTF Test
Why the Freakomendation?

120. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Evaluation
● Research Datasets
● Million Song Dataset (CB, Social, Lyrics, Tags and
more)
http://labrosa.ee.columbia.edu/millionsong/
● Last.fm (CF)
http://ocelma.net/MusicRecommendationDataset/
– Last.fm 360K users <user, artist, total plays>
– Last.fm 1K users <user, timestamp, artist, song>

121. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma

122. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma

123. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Monitoring
● Do not monitor (or test) only the Algorithm, but the
WHOLE recommender system: KPIs
● Catalog
● % matches against full catalog?
● Ingestion time?
● Availability?
● Data & Algorithms
● Time computing (e.g. Matrix factorization)?
● Matrix size (e.g. ~10M x ~1M) in memory?
– 10M vectors with 300 floats per vector → ~11Gb
●
Time computing vector similarity O(n)?

124. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Monitoring
USAGE
● Search
assert_equal(ID(search('The The')), ID('The The'))
● Similarity
assert(similarity(U2, REM) > 0.8)
assert(similarity(AC/DC, Rebecca Black) < 0.3)
● Recommendation
0) create_profile(@ocelma)
1) assert(similarity(@ocelma, U2) >= 0.8)
2) dislike(@ocelma, track(U2,Lemon))
3) assert(similarity(@ocelma, album(U2,Zooropa)) < 0.8)

125. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Monitoring
● (web) API
● Measure query response
– Jmeter, Apache Benchmark
● Process real logs
– Fake (repeated) queries → fast because using cache?

126. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Monitoring
● (web) API
● Measure query response
– Jmeter, Apache Benchmark
● Process real logs
– Fake (repeated) queries → fast because using cache?

127. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
INTRO CHORUS VERSE BRIDGE OUTRO

128. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Conclusions
● Music Recsys is multidisciplinary
● search and filtering, musicology, data mining,
machine learning, personalization, social networks,
text processing, complex networks, user interaction,
information visualization, and signal processing
(among others!)
● Music Recsys is important
● These technologies will be integral in helping the next
generation of music listeners find that next favorite
song
● Strong industry impact
● Music Recsys is special

129. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Further research
● How well do music recommenders work?
● lack of standardized data sets and objective
evaluation methods
● How to recognize and incorporate context into
recommendations?
● listener’s context (exercising, exploring, working,
driving, relaxing, and so on)

130. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Further research
● How to make recommendations for all music?
● consider all music including new, unknown, and
unpopular content.
● What effect will automatic music recommenders
have on the collective music taste?

131. recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma
Music Recommendation and Discovery
Remastered
Tutorial
@recsys, 2011