1.
Assessing the Impact of Music Recommendation
Diversity on Listeners
Lorenzo Porcaro
Supervisors
Dr. Emilia Gómez and Dr. Carlos Castillo
Committee
Dr. Christine Bauer (Utrecht University)
Dr. Perfecto Herrera (Universitat Pompeu Fabra)
Dr. Mounia Lalmas-Roelleke (Spotify)
----------------------------------------------------------------------
Dr. Dmitry Bogdanov (Universitat Pompeu Fabra)
Dr. Sergio Oramas (Pandora) 16/12/2022
1

2.
Molino, J., & Ayrey, C. (1990). Musical Fact and the Semiology of Music. Music Analysis, 9(2), 105–111; 112–156.
Semiology (“study of signs”) →Discipline that studies the phenomena of signification and communication.
2

3.
** Images from: https://search.creativecommons.org/
3

4.
** Images from: https://search.creativecommons.org/
4

5.
Goldberg, D., Nichols, D., Oki, B. M., & Terry, D. (1992).
Using collaborative filtering to Weave an Information
tapestry. Communications of the ACM, 35(12),
61–70. https://doi.org/10.1145/138859.138867 5

6.
6

7.
Diversity // Differences
7

8.
8
Diversity // Differences

9.
9
Diversity // Differences

10.
Thesis Statement
We provide empirical evidence of the function that diversity
plays in mediating the relationships between music
recommendations and listeners. Connecting the
measurement, perception and impact of diversity we
deepen the understanding of recommender systems' role in
shaping today's music listening experience.
10

11.
11

12.
Outline
Music Recommendation Diversity
➔ Measurements - Chapters 2 and 3 2018/19
➔ Perceptions - Chapter 4 2019/21
➔ Impacts - Chapter 5 2021/22
Final Remarks - Chapter 6
12

13.
Music Recommendation Diversity
Measurements
Porcaro, L., & Gomez, E. (2019). 20 Years of Playlists: a Statistical Analysis on Popularity and Diversity.
Proceedings of the 20th International Symposium on Music Information Retrieval, ISMIR 2019, July, 4–11.
Porcaro, L., Castillo, C., & Gómez, E. (2021). Diversity by Design in Music Recommender Systems.
Transactions of the International Society for Music Information Retrieval, 4(1), 114–126.
13

14.
14
Music Recommender System Diversity
Item Diversity User Diversity
Behavioural
Diversity
Item Features
User
Characteristics
Content Source
Perceived Diversity
Poietic Domain Esthetic Domain
- Audio Signal
- Metadata
- Taxonomies
- ...
- Demographics
- Personality Traits
- Personal Values
- ...
Exposure Exposure

15.
15

16.
16
1998 C
C
C
C
C
C
2010 2011 2012 2013 2015 2018
/
AOTM¹
# tracks: 972K
# playlist: 100K
type: user-generated
catalogue: user
CORN²
# tracks: 15K
# playlist: 75K
type: radio playlist
catalogue: radio
SPOT³
# tracks: 2M
# playlist: 175K
type: user-generated
catalogue: streaming
DEEZ⁴
# tracks: 227K
# playlist: 82K
type: user-generated
catalogue: streaming
¹ McFee, B., & Lanckriet, G. “Hypergraph models of playlist dialects”. Proceedings of the 13th International Society for Music Information Retrieval Conference 343–348. 2012
² S. Chen, J.L. Moore, D. Turnbull, and T. Joachims. “Playlist prediction via metric embedding”, Proc. of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD ’12, 2012
³ M. Pichl, E. Zangerle, and G.Specht. “Towards a Context-Aware Music Recommendation Approach: What is Hidden in the Playlist Name?”, Proc. of the 15th IEEE International Conference on Data Mining Workshop
⁴ Crawled in-house

17.
17
Average popularity of its tracks
Playlist Diversity
Popularity

18.
Popularity Tags
Average popularity of its tracks Average distance between its tracks’ tags
Playlist Diversity

19.
19

20.
20
Popularity Tags
Average frequency of its tracks Average distance between its tracks’ tags
Playlist Diversity

21.
21
Popularity Tags
Average frequency of its tracks Average distance between its tracks’ tags
Shannon / Simpson / Gini index Descriptive / Gini / Qualitative
Playlist Diversity
Playlist Dataset Diversity

22.
22
Radio playlists VS user-generated playlist

23.
23
Radio playlists VS user-generated playlist
Radio more balanced in terms of popularity
and tags.

24.
24
Radio playlists VS user-generated playlist
Radio more balanced in terms of popularity
and tags.
Oldest playlist (pre-streaming) VS newest
playlist (streaming)

25.
Radio playlists VS user-generated playlist
Radio more balanced in terms of popularity
and tags.
Oldest playlist (pre-streaming) VS newest
playlist (streaming)
Oldest lower popularity (especially user
generated), and more balanced in terms of tags.
25

26.
❏ Diversity metrics as tools for comparative analysis.
26

27.
❏ Diversity metrics as tools for comparative analysis.
❏ Playlist creation strategy may reflect technological contexts.
27

28.
❏ Diversity metrics as tools for comparative analysis.
❏ Playlist creation strategy may reflect technological contexts.
❏ There is no average user in the real-world.
28

29.
❏ Diversity metrics as tools for comparative analysis.
❏ Playlist creation strategy may reflect technological contexts.
❏ There is no average user in the real-world.
❏ The concept of diversity may vary among different people.
29

30.
Music Recommendation Diversity
Perceptions
Porcaro, L., Gómez, E., & Castillo, C. (2022). Perceptions of Diversity in Electronic Music: The Impact of
Listener, Artist, and Track Characteristics. Proc. ACM Hum.-Comput. Interact., 6(CSCW1).
Porcaro, L., Gómez, E., & Castillo, C. (2022). Diversity in the Music Listening Experience : Insights from Focus
Group Interviews. Proceedings of the 2022 ACM SIGIR Conference on Human Information Interaction and
Retrieval (CHIIR ’22), 272–276.
30

31.
List A
List B
Listeners
Model

32.
List A
List B
Listeners
Model

33.
List A
List B
?
Listeners
Model

34.
TRACK task
List A List B

35.
ARTIST Task
List A List B List A List B

36.
COMBINED Task
List A List B List A List B List A List B

37.
List A
List B
?
Listeners
Model List A
List B

38.
Track Diversity
Input: hand-crafted MIR features
Input type: numerical
Metric: cosine distance

39.
Track Diversity
Input: hand-crafted MIR features
Input type: numerical
Metric: cosine distance
Artist Diversity
Input: gender, birthplace, skin tone, debut year
Input type: categorical
Metric: Goodall distance

40.
2 2
2
2
2
2
List B
Average: div(List A) > div(List B)
→ all distances are equally important (utilitarian)
avg(di
) = 2
min(di
) = 2
3
4
3
3
4 1
List A
avg(di
) = 3
min(di
) = 1

41.
3
4
3
3
4 1
2 2
2
2
2
2
List A
List B
Average: div(List A) > div(List B)
→ all distances are equally important (utilitarian)
Minimum: div(List B) > div(List A)
→ focus on closest distance (egalitarian)
avg(di
) = 3
min(di
) = 1
avg(di
) = 2
min(di
) = 2

42.
List A
List B
?
Listeners
Model
List A
List B

43.
Participants (N=115)
61% - Aged between 18 and 35
73% - Male
73% - Europe or North America
85% - Bachelor’s degree or higher
60% - Skin tone type I-II (white skin)

44.
Participants (N=115)
61% - Aged between 18 and 35
73% - Male
73% - Europe or North America
85% - Bachelor’s degree or higher
60% - Skin tone type I-II (white skin)
WEIRD (Western, Educated,
Industrialized, Rich, and
Democratic) societies.

45.
Participants (self-declared) Musical Background
❖ Playing, DJ-ing, or producing
❖ Varied listening habits
❖ Mostly electronic music listeners
❖ Within electronic music, listen to different styles

46.
RQ1. To what extent
tracks’ audio-features
and artists’ attributes
can be used to assess
the perceived
diversity?
RQ2. To what extent
domain knowledge
and familiarity
influence participants’
perceptions of
diversity?

47.
Can audio-based features be used to determine diversity?
Sometimes: when ≥ 85% participants agree on the most
diverse list ...
… the model identifies the same list as more diverse.

48.
Can artists’ attributes be used to model a degree of diversity?
Answers of participants aged <35, and WEIRD participants
→ moderate-to-substantial agreement* with Goodall.
* ≥ .55 Light’s Kappa (average pairwise Cohen’s Kappa)

49.
RQ1. To what extent
tracks’ audio-features
and artists’ attributes
can be used to assess
the perceived
diversity?
RQ2. To what extent
domain knowledge
and familiarity
influence participants’
perceptions of
diversity?

50.
RQ2. To what extent domain knowledge and familiarity influence
participants’ perceptions of diversity?
Listeners with
low domain knowledge

51.
RQ2. To what extent domain knowledge and familiarity influence
participants’ perceptions of diversity?
Listeners with
low domain knowledge
Listeners with
high domain knowledge

52.
❏ Diversity-aware music recommendations are good for users with
less domain knowledge, but less useful for more specialized users.
❏ New insights for designing socially-relevant diversity-aware music
recommendations.

53.
❏ Diversity-aware music recommendations are good for users with
less domain knowledge, but less useful for more specialized users.
❏ New insights for designing socially-relevant diversity-aware music
recommendations.

54.
Interviews
❖ 14 participants
❖ 7 semi-structured interview (2-3 participants)
❖ Duration ~ 30’ - 60’
❖ Structure:
1. Music list diversity assessments
2. Experience with music recommender systems

55.
Less knowledge (newcomers)
→ diversity assessment harder
→ rely more generic features.

56.
Less knowledge (newcomers)
→ diversity assessment harder
→ rely more generic features
“People who don’t know much about a
particular genre, probably would agree on
some things that are a bit more generic.”

57.
More knowledge (experts)
→ easier to categorize
→ more receptive to details BUT more bias

58.
More knowledge (experts)
→ easier to categorize
→ more receptive to details BUT more bias
“I can feel like I can make a better decision of
what is diverse [...] but then there is kind of a
bias that comes based on the fact that I like
this music a lot.”

59.
Diversity as a tool for deconstructing stereotyped views of a music culture.

60.
Diversity as a tool for deconstructing stereotyped views of a music culture.
“The electronic music that I went to listen to and
that I liked before the survey was predominantly
white male, which I suppose is still what is
predominant in the industry to some extent.

61.
Diversity as a tool for deconstructing stereotyped views of a music culture.
“I realized while making the questionnaire that I
had a very strict let’s say definition about
electronic music myself.”

62.
Algorithmic recommendation diversity help listeners in discovering new
facets of disliked music genres.

63.
Algorithmic recommendation diversity help listeners in discovering new
facets of disliked music genres.
“I never liked EDM but [...] algorithms presented
to me different tracks, and I found myself
listening to it, and noticing the differences
within this genre. In the end, I started listening
to it more often.”

64.
Music Recommendation Diversity
Impacts
Porcaro, L., Gómez, E., and Castillo, C. (under review). Assessing the Impact of Music Recommendation
Diversity on Listeners: A Longitudinal Study. ACM Transactions of Recommender Systems (TORS).
65

65.
Prescreening
Week 5 Week 6 Week 7 Week 8
Weeks 1-4 Weeks 9-12

66.
Prescreening
First step (Prolific)
❖ Age: 18-42
❖ Nationality: Italy, Spain, Portugal
❖ Gender and sex: No restrictions
❖ Highest education level: No restrictions
❖ Number of Prolific previous submissions: > 20
❖ Prolific approval rate: 90%

67.
Prescreening (cont.)
Second step (PsyToolkit)
❖ Taste variety: No restrictions
❖ Electronic Music listening frequency: Not very often
❖ Average daily listening time: > 1 hour

68.
PRE
Prescreening
EMF
Weeks 1-4
EMF = Electronic Music Feedback Questionnaire
Week 5 Week 6 Week 7 Week 8
Weeks 1-4 Weeks 9-12

69.
Electronic Music Feedback (EMF) Questionnaire
Measuring Openness

70.
Electronic Music Feedback (EMF) Questionnaire
Measuring Openness
→ Guttman Scale

71.
Electronic Music Feedback (EMF) Questionnaire
Measuring Implicit Association
→ Single Category Implicit Association Test (SC-IAT)

72.
PRE
Prescreening
EMF
Weeks 1-4
EMF = Electronic Music Feedback Questionnaire
Week 5 Week 6 Week 7 Week 8
Weeks 1-4 Weeks 9-12

73.
PRE
LS
Week 5 Week 6 Week 7 Week 8
Prescreening
EMF
Weeks 1-4
EMF EMF EMF
COND
EMF
LS
LS
LS
LS
LS
LS
LS
EMF = Electronic Music Feedback Questionnaire
LS = Listening Session
Week 5 Week 6 Week 7 Week 8
Weeks 1-4 Weeks 9-12

74.
Listening Session
Calibration Exposure Feedback
Playlist
Start
End
End
Yes
No

75.
PRE
LS
Week 5 Week 6 Week 7 Week 8
Prescreening
EMF
Weeks 1-4 Weeks 9-12
EMF EMF EMF EMF
COND POST
EMF
LS
LS
LS
LS
LS
LS
LS
EMF = Electronic Music Feedback Questionnaire
LS = Listening Session

76.
Calibration Exposure Feedback
Playlist
Start
End
End
Yes
No

77.
Wikipedia: 20 genres & 320 sub-genres

78.
Wikipedia → Every Noise at Once: 20 genres & 181 subgenres

79.
High Diversity
Low Diversity

80.
PRE
LS
Week 5 Week 6 Week 7 Week 8
Prescreening
EMF
Weeks 1-4 Weeks 9-12
EMF EMF EMF EMF
COND POST
EMF
LS
LS
LS
LS
LS
LS
LS
EMF = Electronic Music Feedback Questionnaire
LS = Listening Session

81.
1. High Diversity group engaged more with the playlists than
Low Diversity group.
2. Low Diversity group liked more the music listened to than
the High Diversity group.
3. High Diversity and Low Diversity had the same level of
familiarity with the music listened to.
86

82.
1. High Diversity group engaged more with the playlists than
Low Diversity group.
2. Low Diversity group liked more the music listened to than
the High Diversity group.
3. High Diversity and Low Diversity had the same level of
familiarity with the music listened to.
87

83.
1. High Diversity group engaged more with the playlists than
Low Diversity group.
2. Low Diversity group liked more the music listened to than
the High Diversity group.
3. High Diversity and Low Diversity had the same level of
familiarity with the music listened to.
88

84.
PRE
LS
Week 5 Week 6 Week 7 Week 8
Prescreening
EMF
Weeks 1-4 Weeks 9-12
EMF EMF EMF EMF
COND POST
EMF
LS
LS
LS
LS
LS
LS
LS
EMF = Electronic Music Feedback Questionnaire
LS = Listening Session

85.
❖ The implicit association with Electronic Music tends
towards less extreme valence.
❖ The openness in listening to Electronic Music increases.
92

86.
❖ The implicit association with Electronic Music tends
towards less extreme valence.
❖ The openness in listening to Electronic Music increases.
* Slight difference between PRE-COND and PRE-POST
** No particular influence by the degree of exposure diversity
93

87.
RQ1. To what extent
listeners’ implicit and
explicit attitudes
towards an unfamiliar
music genre can be
affected by exposure to
music recommendations?
RQ2. What is the
relationship between
music recommendation
diversity and the
impact on listeners’
attitudes?

88.
RQ2. What is the
relationship between
music recommendation
diversity and the
impact on listeners’
attitudes?
RQ1. To what extent
listeners’ implicit and
explicit attitudes
towards an unfamiliar
music genre can be
affected by exposure to
music recommendations?

89.
Impact on Discovery

90.
Impact on Implicit Association

91.
Impact on Openness

92.
Final Remarks
99

93.
Findings
Music recommendation diversity
➔ Measurements
◆ Differences in playlist creation strategies
➔ Perceptions
◆ Agreement between computational metrics and perceived diversity
➔ Impacts
◆ Impacts of music recommendation diversity on listeners’ attitudes
100

94.
L1. WEIRD-centric Data and Study Participants

95.
L2. Electronic Music and its Representation

96.
L3. Standpoint on Diversity

97.
Contributions & Awards
6 papers (2 journal, 2 conference, 2 workshop)
3 EU-funded Project (divinAI, TROMPA, MusicalAI) + 7 papers
Software and datasets public available
Women in RecSys Journal Paper of the Year (RecSys 2022)
SIGIR Student Travel Award (CHIIR 2022)
The Sónar+D Innovation Challenge 2020
104

98.
Dissemination
European Researchers’ Night
Oracle4Girls initiative
MUTEK Digital Art and Music Festival Symposium 2021 & 2022
ORION Open Science Podcast
Interview at TV3 (Catalan Television Broadcaster)
Article featured in the UPF Institutional Website
HUMAINT project, Joint Research Centre, European Commission
105

99.
Assessing the Impact of Music Recommendation
Diversity on Listeners
Lorenzo Porcaro
Supervisors
Dr. Emilia Gómez and Dr. Carlos Castillo
Committee
Dr. Christine Bauer (Utrecht University)
Dr. Perfecto Herrera (Universitat Pompeu Fabra)
Dr. Mounia Lalmas-Roelleke (Spotify)
----------------------------------------------------------------------
Dr. Dmitry Bogdanov (Universitat Pompeu Fabra)
Dr. Sergio Oramas (Pandora) 16/12/2022
106