1
The Acoustic Emotion Gaussians
Model for Emotion-based Music
Annotation and Retrieval
Ju-Chiang Wang, Yi-Hsuan Yang,
Hsi...
2
Outline
• Introduction
• Related Work
• The Acoustic Emotion Gaussians (AEG)
Model
• Music Emotion Annotation and Retrie...
3
Introduction
• One of the most exciting but challenging
endeavors in music information retrieval (MIR)
– Develop a compu...
4
Dimensional Emotion:
The Valence-Arousal(Activation) Model
• Emotions are considered as numerical values (instead of
dis...
5
The Valence-Arousal Annotation
• Emotion is subjective, different emotion may be elicited
from a song in the VA space
• ...
6
Related Work:
Regression for Gaussian Parameters
• The Gaussian-parameter approach directly learns five
regression model...
7
The Acoustic Emotion Gaussians Model for
Modeling between VA and Acoustic Feature
• A principled probabilistic/statistic...
8
AEG: Construct Feature Reference Model
Global Set  of frame
vectors randomly
selected from each track
…
A1 N2
NK-1
NK N...
9
Represent a Song into Probabilistic Space
1
2
K-1
K…
Posterior
Probabilities over
the Acoustic GMM
…
A1
A2
AK-1
Acou...
10
Generative Process of VA GMM
• Key idea: Each component VA Gaussian corresponds to
a latent feature class (a specific a...
11
Total Likelihood Function of VA GMM
• To cover the subjectivity, each training clip is annotated
by multiple subjects {...
12
User Prior Model
• Some annotations could be outliers
• The prior weight of each annotation can be described by
the lik...
13
Integrating the Annotation (User) Prior
• Integrating Acoustic GMM Posterior and Annotation Prior
into the Generative P...
14
The Objective Function
• Take log of p(E| ), and according to Jensen’s inequality
we derive the lower bound
where
• Th...
15
The Learning of VA GMM on MER60
Iter=8Iter=4
Iter=32Iter=16
Iter=2
16
Music Emotion Annotation
• Given the acoustic GMM posterior of a test song, predict
the emotion as a single VA Gaussian...
17
Find the Representative Gaussian
• Minimize the cumulative weighted relative entropy
– The representative Gaussian has ...
18
Emotion-Based Music Retrieval
Approach Indexing Matching
Fold-In Acoustic GMM Posterior Cosine Sim (K-dim)
Emotion Pred...
19
The Fold-In Approach
l1
l2
lK-1
lK
…
The Learned VA GMM A VA Point Query
Fold In
The query is Dominated by
the VA Gauss...
20
Evaluation – Dataset
• Two corpora used: MER60 and MTurk
• MER60
– 60 music clips, each is 30-second
– 99 subjects in t...
21
Evaluation – Acoustic Features
• Adopt the bag-of-frames representation
• All the frames of a clip are aggregated into ...
22
Evaluation Metric for Emotion Annotation
• Average KL divergence (AKL)
– Measure the KL divergence from the predicted V...
23
Result for Emotion Annotation
• MER60, leave-one-out train and test
• MTurk, 70%-30% randomly splitting train and test
...
24
Summary for Emotion Annotation
• The performance saturates when K is sufficiently large
• Larger scale corpus prefers l...
25
Result for Music Retrieval
• MTurk: 2,520 clips training, 1,080 clips for retrieval database
• Evaluate the ranking usi...
26
Conclusion and Future Work
• The AEG model provides a principled probabilistic
framework that is technically sound, and...
27
Appendix: PWKL for Emotion Corpus
PWKL
5.095
1.985
• PWKL: the diversity of ground truth among all songs in a
corpus, t...
Upcoming SlideShare
Loading in …5
×

The Acoustic Emotion Gaussians Model for Emotion-based Music Annotation and Retrieval

542 views

Published on

One of the most exciting but challenging endeavors in music research is to develop a computational model that comprehends the affective content of music signals and organizes a music collection according to emotion. In this paper, we propose a novel \emph{acoustic emotion Gaussians} (AEG) model that defines a proper generative process of emotion perception in music. As a generative model, AEG permits easy and straightforward interpretations of the model learning processes. To bridge the acoustic feature space and music emotion space, a set of \emph{latent feature classes}, which are learned from data, is introduced to perform the end-to-end semantic mappings between the two spaces. Based on the space of latent feature classes, the AEG model is applicable to both automatic music emotion annotation and emotion-based music retrieval. To gain insights into the AEG model, we also provide illustrations of the model learning process. A comprehensive performance study is conducted to demonstrate the superior accuracy of AEG over its predecessors, using two emotion annotated music corpora MER60 and MTurk. Our results show that the AEG model outperforms the state-of-the-art methods in automatic music emotion annotation. Moreover, for the first time a quantitative evaluation of emotion-based music retrieval is reported.

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
542
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

The Acoustic Emotion Gaussians Model for Emotion-based Music Annotation and Retrieval

  1. 1. 1 The Acoustic Emotion Gaussians Model for Emotion-based Music Annotation and Retrieval Ju-Chiang Wang, Yi-Hsuan Yang, Hsin-Min Wang, and Skyh-Kang Jeng Academia Sinica, National Taiwan University, Taipei, Taiwan
  2. 2. 2 Outline • Introduction • Related Work • The Acoustic Emotion Gaussians (AEG) Model • Music Emotion Annotation and Retrieval • Evaluation and Result • Conclusion and Future Work
  3. 3. 3 Introduction • One of the most exciting but challenging endeavors in music information retrieval (MIR) – Develop a computational model that comprehends the affective content of music signals • Why is emotion so important to MIR system? – Music is the finest language of emotion – We use music to convey or modulate emotion – Smaller semantic gap, comparing to genre – Each state in our daily life contains emotion, context-dependent music recommendation
  4. 4. 4 Dimensional Emotion: The Valence-Arousal(Activation) Model • Emotions are considered as numerical values (instead of discrete labels) over a number of emotion dimensions • Good visualization, intuitive, a unified model • Easy to capture temporal change of emotion Mufin Player Mr. Emo developed by Yang and Chen
  5. 5. 5 The Valence-Arousal Annotation • Emotion is subjective, different emotion may be elicited from a song in the VA space • Assumption: the VA annotation of a song can be drawn from a Gaussian distribution, as observed above • Subjectivity issue: observed by multiple subjects • Temporal change: summarize the scope of changes
  6. 6. 6 Related Work: Regression for Gaussian Parameters • The Gaussian-parameter approach directly learns five regression models to predict the mean, variance, and covariance of valence and arousal, respectively • Without a joint modeling and estimation for the Gaussian parameters x Regressor 1 Regressor 2 Regressor 3 Regressor 4 Regressor 5 mVal mAro sVal-Aro sAro-Aro sVal-Val
  7. 7. 7 The Acoustic Emotion Gaussians Model for Modeling between VA and Acoustic Feature • A principled probabilistic/statistical approach • Represent the acoustic features of a song by a probabilistic histogram vector • Develop a model to comprehend the relationship between acoustic features and VA space (annotations) Acoustic GMM Posterior Distributions
  8. 8. 8 AEG: Construct Feature Reference Model Global Set  of frame vectors randomly selected from each track … A1 N2 NK-1 NK N3N4 Global GMM for acoustic feature encoding EM Training A Universal Music Database Acoustic GMM Music Tracks & Audio Signal Frame-based Features … … … …
  9. 9. 9 Represent a Song into Probabilistic Space 1 2 K-1 K… Posterior Probabilities over the Acoustic GMM … A1 A2 AK-1 Acoustic GMM AK … Feature Vectors Histogram: Acoustic GMM Posterior prob Each dim corresponds to a specific acoustic pattern, called a latent feature class (or audio word) 1 2 K-1 K…
  10. 10. 10 Generative Process of VA GMM • Key idea: Each component VA Gaussian corresponds to a latent feature class (a specific acoustic pattern) Audio Signal of Each Clip A Mixture of Gaussians in the VA Space … A1 A2 AK-1 Acoustic GMM AK 1 2 K-1 K …
  11. 11. 11 Total Likelihood Function of VA GMM • To cover the subjectivity, each training clip is annotated by multiple subjects {uj}, the corresponding annotation ej • An annotated corpus: assume each annotation eij of clip si can be generated by a weighted VA GMM with {qik}! • Generating the Corpus-level likelihood and maximize it using the EM algorithm 1 1 1 1 ( | ) ( | ) ( | , ) jU KN N i ik ij k k i i j k p p s q = = = = = = å E E e  m S  1 ( | ) ( | , ) K ij i ik ij k k k p s q = = åe e Sm Acoustic GMM posterior Clip-level likelihood: Each annotation contributes equally parameters of each latent VA Gaussian to learn Annotation-level Likelihood
  12. 12. 12 User Prior Model • Some annotations could be outliers • The prior weight of each annotation can be described by the likelihood over the clip-level annotation Gaussian – Larger B indicates lower label consistency (higher uncertainty) – Smaller likelihood implies the annotation could be an outlier ( | , ) ( | , , )jp u s s=e e a B , ( | , ) ( | ) ( | , ) j j s j ju p u s p u s p u s g ¬ = å e e
  13. 13. 13 Integrating the Annotation (User) Prior • Integrating Acoustic GMM Posterior and Annotation Prior into the Generative Process 1 1 1 1 1 1 ( | ) ( | ) ( | ) ( | ) ( | , ) j j UN N i ij i ij i i i j U KN ij ik ij k k i j k p p s p u s p s g q = = = = = = = = = å  å å E E e e   m S Clip-level likelihood: prior weighted sum over annotation-level likelihood Annotation Prior Acoustic GMM posterior
  14. 14. 14 The Objective Function • Take log of p(E| ), and according to Jensen’s inequality we derive the lower bound where • Then, we maximize Lbound with the EM-Algorithm 1 1 1 1 1 1 log ( | ) log ( | , ) log ( | , ) j j UN K ij ik ij k k i j k UN K bound ij ik ij k k i j k p D L g q g q = = = = = = = ³ = å å å åå å E e e   m m S S 1 1 1 jUN ij i j g = = =åå two-layer log sum one-layer log sum parameters to learn
  15. 15. 15 The Learning of VA GMM on MER60 Iter=8Iter=4 Iter=32Iter=16 Iter=2
  16. 16. 16 Music Emotion Annotation • Given the acoustic GMM posterior of a test song, predict the emotion as a single VA Gaussian 1 2 K-1 K … Acoustic GMM Posterior Learned VA GMM Predicted Single Gaussian 1 ˆˆ( | ) ( | , ) K k ij k k k p s q = = åe e m S ^ ^ ^ ^ … { , }* m * S
  17. 17. 17 Find the Representative Gaussian • Minimize the cumulative weighted relative entropy – The representative Gaussian has the minimal cumulative distance from all the component VA Gaussians • The optimal parameters of the Gaussian are ( )KL { , } 1 ˆ( | , ) arg min ( | , ) || ( | , ) K k k k k p D p pq* * = = åe e e S S S S m m m m * 1 ˆ K k k k q = = åm m ( )* * * 1 ˆ ( )( ) K T k k k k k q = = + - -åS S m m m m
  18. 18. 18 Emotion-Based Music Retrieval Approach Indexing Matching Fold-In Acoustic GMM Posterior Cosine Sim (K-dim) Emotion Prediction Predicted VA Gaussian Gaussian Likelihood
  19. 19. 19 The Fold-In Approach l1 l2 lK-1 lK … The Learned VA GMM A VA Point Query Fold In The query is Dominated by the VA Gaussian of A2 Pseudo Song Distribution 1 ˆ ˆarg max log ( | , ) K k k k k pl = = å e l l m S ˆe Using the EM algorithm 1 2 K-1 K … Acoustic GMM Posterior Music Database
  20. 20. 20 Evaluation – Dataset • Two corpora used: MER60 and MTurk • MER60 – 60 music clips, each is 30-second – 99 subjects in total, making each clip annotated by 40 subjects – The VA values are entered by clicking on the emotion space on a computer display • MTurk – 240 clips, each is 15-second – Collected via Amazon's Mechanical Turk – Each subject rated the per-second VA values for 11 randomly- selected clips using a graphical interface – Automatic verification step employed, finalizing each clip with 7 to 23 subjects
  21. 21. 21 Evaluation – Acoustic Features • Adopt the bag-of-frames representation • All the frames of a clip are aggregated into the acoustic GMM posterior and perform the analysis of emotion at the clip-level, instead of frame-level • MER60: extracted by MIRToolbox – Dynamic, spectral, timbre (including 13 MFCCs, 13 delta MFCCs, and 13 delta-delta MFCCs), and tonal – 70-dim all concatenation or 39-dim MFCCs • MTurk: provided by Schmidt et al. – MFCCs, chroma, spectrum descriptors, and spectral contrast – 50-dim all concatenation, 20-dim MFCCs, or 14-dim spectral contrast
  22. 22. 22 Evaluation Metric for Emotion Annotation • Average KL divergence (AKL) – Measure the KL divergence from the predicted VA Gaussian of a test clip to its ground truth VA Gaussian • Average Mean Distance (AMD) – Measure the Euclidean distance between the mean vectors of the predicted and ground truth VA Gaussians ( )1 1 1 P G P G P G G P G 1 tr( ) log ( ) ( ) 2 2 T- - - - + - - -m m m mS S S S S P G P G( ) ( )T - -m m m m
  23. 23. 23 Result for Emotion Annotation • MER60, leave-one-out train and test • MTurk, 70%-30% randomly splitting train and test Smaller Better
  24. 24. 24 Summary for Emotion Annotation • The performance saturates when K is sufficiently large • Larger scale corpus prefers larger K (feature resolution) • Annotation prior is effective for the AKL performance • For MER60, 70-D concat feature performs the best • For MTurk, using MFCCs alone is more effective • MTurk is easier and presents smaller performance scale
  25. 25. 25 Result for Music Retrieval • MTurk: 2,520 clips training, 1,080 clips for retrieval database • Evaluate the ranking using the Normalized Discounted Cumulative Gain (NDCG) with 5, 10, and 20 retrieved clips 2 2 ( )1 NDCG @ (1) log P iP R i P R Z i= ì üï ïï ï= +í ý ï ïï ïî þ å Larger Better
  26. 26. 26 Conclusion and Future Work • The AEG model provides a principled probabilistic framework that is technically sound, and also unifies the emotion-based music annotation and retrieval • AEG can better take into account the subjective nature of emotion perception • Transparency and interpretability of the model learning and semantic mapping processes • The potential for incorporating multi-modal content • Dynamic personalization via model adaptation • Alignment among multi-modal emotion semantics
  27. 27. 27 Appendix: PWKL for Emotion Corpus PWKL 5.095 1.985 • PWKL: the diversity of ground truth among all songs in a corpus, the larger the more diverse • We compute the (pair-wise) KL divergence between the ground truth annotation Gaussians of each pair of clips in a corpus • MTurk is easier, since a safer prediction, the origin, can gain good performance

×