The document describes using i-vectors for music similarity and artist classification. I-vectors apply factor analysis to song-level Gaussian mixture model (GMM) features to reduce dimensionality while retaining variability between songs. I-vectors have been used successfully in speaker recognition and other audio classification tasks. The document outlines how i-vectors are extracted from songs' acoustic features and can be used as compact song representations for classification.

1. I-VECTORS FOR TIMBRE-BASED
MUSIC SIMILARITY AND MUSIC
ARTIST CLASSIFICATION
Hamid Eghbal-zadeh, Bernhard Lehner
Markus Schedl, Gerhard Widmer
ISMIR 2015

2. Outline
• Introduction
o Timbral features
o Song-level features
o Factor analysis for song-level features
• I-vector Frontend
o Related work
o Overview
o I-vector factor analysis
o Feature spaces
o Total variability
o Extraction procedure
• Experiments
o Setup
o Evaluation
o Results
• Conclusion

3. INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain
Timbral features

4. Introduction – Timbral features
Timbral features (e.g. MFCCs) are very important in
content-based MIR tasks
• Music similarity [Sereylehner2010,Pohle2007]
• Genre classification [Mirex,Sereylehner2010,Pohle2007]
• Artist classification [Ellis2007,Su2013,Kuksa2014]
• etc
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

5. Introduction – Timbral features
Difference in level of features:
• Frame-level
Song 1 features
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

6. Introduction – Timbral features
Difference in level of features:
• Frame-level
• Block-level
Song 1 features
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

7. Introduction – Timbral features
Difference in level of features:
• Frame-level
• Block-level
• Song-level processing
Song 1
Song 2
Song 3
vector 2
vector 1
features
features
features
vector 3
Song 1 features
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

8. Introduction – Timbral features
Difference in level of features:
• Frame-level
• Block-level
• Song-level processing
Song 1
Song 2
Song 3
vector 2
vector 1
features
features
features
vector 3
Song 1 features
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

9. INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain
Song-level features

10. Introduction – Song-level Timbral
features
Difference in level of features:
• Song-level
1. Using single Gaussians [Sereylehner2010,Pohle2007]
processing
Song 1
Song 2
Song 3
vector 2
vector 1
features
features
features
vector 3
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

11. Introduction – Song-level Timbral
features
Difference in level of features:
• Song-level
1. Using single Gaussians [Sereylehner2010,Pohle2007]
2. Using GMMs [Charbuillet2011, Kruspe2014]
a) Train a GMM on a song
processing
Song 1
Song 2
Song 3
vector 2
vector 1
features
features
features
vector 3
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

12. Introduction – Song-level Timbral
features
Difference in level of features:
• Song-level
1. Using single Gaussians [Sereylehner2010,Pohle2007]
2. Using GMMs [Charbuillet2011, Kruspe2014]
a) Train a GMM on a song
b) Adapt a pre-trained GMM on a song
processing
Song 1
Song 2
Song 3
vector 2
vector 1
features
features
features
vector 3
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

13. Introduction – Song-level Timbral
features
Difference in level of features:
• Song-level
1. Using single Gaussians [Sereylehner2010,Pohle2007]
2. Using GMMs [Charbuillet2011, Kruspe2014]
a) Train a GMM on a song
b) Adapt a pre-trained GMM on a song
c) Use a pre-trained GMM as a prior
processing
Song 1
Song 2
Song 3
vector 2
vector 1
features
features
features
vector 3
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

14. Introduction – Song-level Timbral
features
Difference in level of features:
• Song-level
1. Using single Gaussians [Sereylehner2010,Pohle2007]
2. Using GMMs [Charbuillet2011, Kruspe2014]
a) Train a GMM on a song
b) Adapt a pre-trained GMM on a song
c) Use a pre-trained GMM as a prior
processing
Song 1
Song 2
Song 3
vector 2
vector 1
features
features
features
vector 3
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

15. Introduction – Song-level Timbral
features
Difference in level of features:
• Song-level
1. Using single Gaussians [Sereylehner2010,Pohle2007]
2. Using GMMs [Charbuillet2011, Kruspe2014]
a) Train a GMM on a song
b) Adapt a pre-trained GMM on a song
c) Use a pre-trained GMM as a prior
3. Using GMMs+Factor Analysis [Kruspe2014, Eghbal-zadeh2015]
processing
Song 1
Song 2
Song 3
vector 2
vector 1
features
features
features
vector 3
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

16. Introduction – Song-level Timbral
features
Difference in level of features:
• Song-level
1. Using single Gaussians [Sereylehner2010,Pohle2007]
2. Using GMMs [Charbuillet2011, Kruspe2014]
a) Train a GMM on a song
b) Adapt a pre-trained GMM on a song
c) Use a pre-trained GMM as a prior
3. Using GMMs+Factor Analysis [Kruspe2014, Eghbal-zadeh2015]
processing
Song 1
Song 2
Song 3
vector 2
vector 1
features
features
features
vector 3
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

17. INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain
Why Factor Analysis?

18. Introduction – Advantages of GMM+FA
• Having a probabilistic representation for a song:
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

19. • Having a probabilistic representation for a song:
regarding a specific variability
Introduction – Advantages of GMM+FA
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

20. Introduction – Advantages of GMM+FA
• Having a probabilistic representation for a song:
regarding a specific variability
• Lower dimensionality:
With a 1024 components GMM from 20 dim MFCCs, we will have
1024 x 20 dimensions
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

21. Introduction – Advantages of GMM+FA
• Having a probabilistic representation for a song:
regarding a specific variability
• Lower dimensionality:
With a 1024 components GMM from 20 dim MFCCs, we will have
1024 x 20 dimensions
• Better discrimination:
GMM GMM+FA
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain
• Samples are taken form GTZAN dataset, projected using PCA

22. INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain
Related work

23. I-vectors – Related work
• The FA model we use with GMMs is called I-vector FA
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

24. I-vectors – Related work
• The FA model we use with GMMs is called I-vector FA
• Introduced in speaker verification [Dehak2010]
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

25. I-vectors – Related work
• The FA model we use with GMMs is called I-vector FA
• Introduced in speaker verification [Dehak2010]
• Speech Emotion Recognition [Chen2011]
• Speech Language Recognition [Martinez2011]
• Speech Accent Recognition [Bahari2013]
• Speech Audio Scene Detection [Elizalde2013]
• Singing Language Recognition [Kruspe2014]
• Music Artist Recognition [Eghbal-zadeh2015]
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

26. INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain
How it works?

27. I-vectors – Overview
Song 1
Song 2
Song 3
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

28. I-vectors – Overview
Song 1
Song 2
Song 3
features
features
features
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

29. I-vectors – Overview
I-vector extractor
Song 1
Song 2
Song 3
features
features
features
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

30. I-vectors – Overview
i-vector 3
I-vector extractor
Song 1
Song 2
Song 3
i-vector 2
i-vector 1
features
features
features
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

31. INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain
I-vector Factor Analysis

32. I-vectors – Factor Analysis
M = m + T y
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

33. I-vectors – Factor Analysis
Song GMM supervector:
assumed to be normally distributed
with mean vector m and
covariance matrix 𝑻𝑻 𝒕
M = m + T y
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

34. I-vectors – Factor Analysis
Song GMM supervector:
assumed to be normally distributed
with mean vector m and
covariance matrix 𝑻𝑻 𝒕
class- and song-independent supervector
(comes from a Universal Background Model - UBM)
M = m + T y
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

35. I-vectors – Factor Analysis
Song GMM supervector:
assumed to be normally distributed
with mean vector m and
covariance matrix 𝑻𝑻 𝒕
class- and song-independent supervector
(comes from a Universal Background Model - UBM)
low rank matrix, learned
from training
M = m + T y
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

36. I-vectors – Factor Analysis
Song GMM supervector:
assumed to be normally distributed
with mean vector m and
covariance matrix 𝑻𝑻 𝒕
class- and song-independent supervector
(comes from a Universal Background Model - UBM)
low rank matrix, learned
from training
i-vectorM = m + T y
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

37. I-vectors – Factor Analysis
Song GMM supervector:
assumed to be normally distributed
with mean vector m and
covariance matrix 𝑻𝑻 𝒕
class- and song-independent supervector
(comes from a Universal Background Model - UBM)
low rank matrix, learned
from training
i-vectorM = m + T y
• I-vectors are assumed to be normally distributed
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

38. I-vectors – Factor Analysis
Song GMM supervector:
assumed to be normally distributed
with mean vector m and
covariance matrix 𝑻𝑻 𝒕
class- and song-independent supervector
(comes from a Universal Background Model - UBM)
low rank matrix, learned
from training
i-vectorM = m + T y
• I-vectors are assumed to be normally distributed
• Unlike GMM supervectors, i-vectors are from a single-Gaussian space
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

39. INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain
Different feature spaces

40. I-vectors – Feature space
Frame-level feature space
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

41. I-vectors – Feature space
Frame-level feature space GMM feature space
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

42. I-vectors – Feature space
Frame-level feature space GMM feature space I-vector feature space
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

43. I-vectors – Feature space
Frame-level feature space GMM feature space I-vector feature space
[Total Variability Space (TVS)]
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

44. I-vectors – Feature space
Frame-level feature space GMM feature space I-vector feature space
[Total Variability Space (TVS)]
20 dim 1024 x 20 dim 400 dim
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

45. I-vectors – Feature space
Frame-level feature space GMM feature space I-vector feature space
[Total Variability Space (TVS)]
20 dim 1024 x 20 dim 400 dim
[number of total factors]
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

46. INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain
What is total variability?

47. I-vectors – Total variability
In genre classification:
• Genre variability : the variability appears between different
genres.
• Song variability : the variability appears within songs of a genre.
• Total variability : genre + song variability
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

48. I-vectors – Total variability
In genre classification:
• Genre variability : the variability appears between different
genres.
• Song variability : the variability appears within songs of a genre.
• Total variability : genre + song variability
In artist recognition:
• Artist variability : the variability appears between different artists.
• Song variability : the variability appears within songs of an artist.
• Total variability : artist + song variability
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

49. INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain
How to extract i-vectors?
1, 2, 3 …

50. I-vectors – Procedure
step1
i-vectorGMM supervector
Factor
Analysis
Statistics
calculation
Feature
extraction
Audio Frame-level features
(MFCCs)
Frame-level features,
20 dim MFCCs
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

51. I-vectors – Procedure
step1
i-vectorGMM supervector
Factor
Analysis
Statistics
calculation
Feature
extraction
Audio Frame-level features
(MFCCs)
step2
Using a universal
background model (UBM)
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

52. I-vectors – Procedure
step1
i-vectorGMM supervector
Factor
Analysis
Statistics
calculation
Feature
extraction
Audio Frame-level features
(MFCCs)
step2
Using a universal
background model (UBM)
UBM is a GMM
trained on
frame-level
features
extracted from
a set of songs
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

53. I-vectors – Procedure
step1
i-vectorGMM supervector
Factor
Analysis
Statistics
calculation
Feature
extraction
Audio Frame-level features
(MFCCs)
step2
Using a universal
background model (UBM)
𝑁𝑐 = 𝛾𝑡(𝑐)𝑡∈𝑠
𝐹𝑐 = 𝛾𝑡 𝑐𝑡∈𝑠 Y 𝑡
𝛾𝑡:The posterior probability
of component c for
observation t, given UBM
Y 𝑡:Acoustic feature vectors
N and F are referred to as
GMM supervectors.
UBM is a GMM
trained on
frame-level
features
extracted from
a set of songs
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

54. I-vectors – Procedure
step1
i-vectorGMM supervector
Factor
Analysis
Statistics
calculation
Feature
extraction
Audio Frame-level features
(MFCCs)
step2 step3
Assuming GMM
supervector 𝑀
𝑀 ~Normal(𝑚, 𝑇 ∗ 𝑇 𝑡
)
Where 𝑚 is artist and
song independent (from
UBM)
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

55. I-vectors – Procedure
step1
i-vectorGMM supervector
Factor
Analysis
Statistics
calculation
Feature
extraction
Audio Frame-level features
(MFCCs)
step2 step3
Assuming GMM
supervector 𝑀
𝑀 ~Normal(𝑚, 𝑇 ∗ 𝑇 𝑡
)
Where 𝑚 is artist and
song independent (from
UBM)
I-vector 𝑦 is estimated via:
𝑦 = (𝐼 + 𝑇 𝑡 Σ−1 𝑁 𝑇)−1. 𝑇−1Σ−1 𝐹
T: is learned from training data
𝛴: is a diagonal covariance matrix learned
in Factor Analysis procedure
𝐹 and 𝑁: GMM supervector
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

56. I-vectors – Procedure
step1
i-vectorGMM supervector
Factor
Analysis
Statistics
calculation
Feature
extraction
Audio Frame-level features
(MFCCs)
step2 step3
Assuming GMM
supervector 𝑀
𝑀 ~Normal(𝑚, 𝑇 ∗ 𝑇 𝑡
)
Where 𝑚 is artist and
song independent (from
UBM)
I-vector 𝑦 is estimated via:
𝑦 = (𝐼 + 𝑇 𝑡 Σ−1 𝑁 𝑇)−1. 𝑇−1Σ−1 𝐹
T: is learned from training data
𝛴: is a diagonal covariance matrix learned
in Factor Analysis procedure
𝐹 and 𝑁: GMM supervector
UNSUPERVISED
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

57. Tasks:
1.Music artist classification
2.Music similarity
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

58. INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain
Tasks:
1.Music artist classification
2.Music similarity

59. Experiments – Evaluation
In music artist classification:
• Artist20 dataset, 1,413 songs
• 20 artist, mostly rock and pop, 6 albums from each artist
• 6-fold cross validation
• 5 albums for train, 1 album for test
• UBM and T are trained on training set of each fold
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

60. Experiments – Setup
In music artist classification:
• Frame-level features:
a) 20 dim MFCCs
b) Spectrum Derivatives
• 400 total factors
• LDA for dimensionality reduction
• 3 back-ends: PLDA, KNN, DA
{I-vector extraction}
{PLDA,…}{MFCC+SD}
Extract
features
Extract
GMM
supervectors
Front
end
Compensation/
Normalization
{LDA/Length norm}
Frames
Backend
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

61. Experiments – Results
In music artist classification:
Baselines
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

62. Experiments – Results
In music artist classification:
Baselines
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain
MFCCs

63. Experiments – Results
In music artist classification:
Baselines
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain
MFCCs
MFCCs+SD

64. Experiments – Tasks
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain
Tasks:
1.Music artist classification
2.Music similarity

65. Experiments – Evaluation
In music similarity:
• 1517Artists dataset
 used for training T and UBM
• GTZAN dataset
 used for music similarity estimation experiments
• KNN [k=1:20]
 to evaluate similarity matrix with genre labels
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

66. Experiments – Evaluation
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
Blues, Classic, Country, Disco, Hip-Hop, Jazz, Metal, Pop, Reggae, RockGTZAN:
 1,000 songs x 30 sec excerpts
 10 genres
Alternative & Punk, Blues, Children, Classical, Comedy & Spoken Word, Country, Easy
Listening & Vocals, Electronic & Dance, Folk, Hip-Hop, Jazz, Latin, New Age, R&B & Soul,
Reggae, Religious, Rock & Pop, Soundtracks & More, World
 1517Artists:
 3,180 songs x full songs
 19 genres
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

67. Experiments – Setup
In music similarity:
• Frame-level features:
a) 25 dim MFCCs + delta + delta2
b) Spectrum Derivatives
• 400 total factors
• Cosine distance for pairwise distance matrix
• Distance normalization, distance combination
{I-vector extraction}
{Similarity matrix}{MFCC+SD}
Extract
features
Extract
GMM
supervectors
Front
end
Cosine
{Distance}
Frames
Backend
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

68. Experiments – Results
Baselines
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain
In music similarity:

69. Experiments – Results
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
Baselines
MFCC-IVEC
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain
In music similarity:

70. Experiments – Results
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
Baselines
[MFCC+SD] IVEC
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain
In music similarity:

71. Experiments – Results
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
Baselines
[MFCC+ SD] IVEC + RTBOF
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain
In music similarity:

72. Experiments – Results
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
Baselines
[MFCC+ SD] IVEC + CMB
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain
In music similarity:

73. Experiments – Tasks
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain
Tasks:
1.Music artist classification
2.Music similarity
• Extended results

74. Experiments – Extended results
In music similarity:
• 1517Artists
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

75. Experiments – Extended results
In music similarity:
• 1517Artists
• Not in the paper
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

76. Experiments – Extended results
In music similarity:
• 1517Artists
• Not in the paper
• Only MFCCs
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

77. Experiments – Extended results
In music similarity:
• 1517Artists
• Not in the paper
• Only MFCCs
• More results:
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
http://bird.cp.jku.at/ivectors/
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

78. Experiments – Extended results
In music similarity:
• 1517Artists
• Not in the paper
• Only MFCCs
• More results:
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
Baselines
http://bird.cp.jku.at/ivectors/
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

79. Experiments – Extended results
In music similarity:
• 1517Artists
• Not in the paper
• Only MFCCs
• More results:
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
Baselines
BLS
http://bird.cp.jku.at/ivectors/
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

80. Experiments – Extended results
In music similarity:
• 1517Artists
• Not in the paper
• Only MFCCs
• More results:
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
Baselines
RTBOF
http://bird.cp.jku.at/ivectors/
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

81. Experiments – Extended results
In music similarity:
• 1517Artists
• Not in the paper
• Only MFCCs
• More results:
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
Baselines
MFCC-IVEC
http://bird.cp.jku.at/ivectors/
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

82. Experiments – Extended results
In music similarity:
• 1517Artists
• Not in the paper
• Only MFCCs
• More results:
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
Baselines
MFCC-IVEC+RTBOF
http://bird.cp.jku.at/ivectors/
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

83. Experiments – Extended results
In music similarity:
• 1517Artists
• Not in the paper
• Only MFCCs
• More results:
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
Baselines
MFCC-IVEC+BLS
http://bird.cp.jku.at/ivectors/
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

84. INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain
Conclusion

85. Conclusion – Advantages
• Song-level timbral features
• Can capture specific variability
• Low dimensional features
• Can be used for supervised and unsupervised tasks
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

86. Conclusion – Advantages
• Song-level timbral features
• Can capture specific variability
• Low dimensional features
• Can be used for supervised and unsupervised tasks
• Very good timbral features, please use it!
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
• Contact:
hamid.eghbal-zadeh@jku.at
www.cp.jku.at/people/eghbal-zadeh/
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

87. Conclusion – Drawbacks
• Can be used with features can be modeled via GMMs
• Relies on a UBM
• Unreliable for very short segments
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

88. Thank you!
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

89. Acknowledgement
• We would like to acknowledge the tremendous help by Dan Ellis from Columbia
University, who shared the details of his work, which enabled us to reproduce his
experiment results.
• Thank to Pavel Kuksa from University of Pennsylvania for sharing the details of
his work with us.
• Also thank to Jan Schlüter from OFAI for his help with music similarity baselines.
• And at the end, we appreciate helpful suggestions of Rainer Kelz and Filip
Korzeniowski from Johannes Kepler University of Linz to this work.
• This work was supported by the EU-FP7 project no.601166 (PHENICX), and by the
Austrian Science Fund (FWF) under grants TRP307-N23 and Z159.
INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION
I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain