SWISS CENTER FORAFFECTIVE SCIENCESPsychoacoustic cues to emotion inMusic and Speech ProsodyEduardo CoutinhoSwissNex, San F...
Aims of this talk• Basic acoustic features allow people to perceiveemotional meaning in music and the human voice• Both me...
• Changes in tone of voice while speakingcommunicate emotional meaning (butnot only!) independently of verbalunderstanding...
Japanese Psychological Research2004, Volume 46, No. 4, 337–349Special Issue: Cognition and emotion in musicMunksgaardORIGI...
Communication of Emotions in Vocal Expression and Music Performance:Different Channels, Same Code?Patrik N. Juslin and Pet...
Aims• To identify relevant perceptual/acoustic features associatedwith the expression of emotion in both speech and music:...
Empirical experimentsArousalValenceKey issues:1. Continuous measurements framework2. Emotion classification: dimensional mo...
Empirical experimentsArousalValencePhysiological measurements
Stimuli: Music (film music)
Stimuli: Natural speech (German)
Computationalframework• Recurrent neural network• Nonlinear regression model• Training phase: learn fromhuman listeners• E...
(Psycho)acoustic predictorsDynamics Time Pitch Timbre DissonanceLoudnessTempo (BPM)Speech Rate (SPM)Melody ContourProsody ...
Results: Speech* p < 0.05 (exceptValence of sample 6)
Results: Speech0 10 20 30 40 5010.500.51Sample 4ArousalParticipantsModel0 10 20 30 40 5010.500.51Time(s)Valence0 20 40 60 ...
Results: Music* p < 0.05
Results: Music0 20 40 60 80 100 12010.500.51Piece 1ArousalParticipantsModel0 20 40 60 80 100 12010.500.51Time(s)Valence0 2...
Summary• Low-level acoustic parameters are fundamentalcues in the expression of emotion in music andspeech.• Affective inf...
Possible applications• Diagnosis of psychiatric/psychological conditions (speech)• Improvement of emotional communication ...
Thank you!• Collaborators:• Contact: eduardo.coutinho@unige.ch• More information: www.eadward.orgSWISS CENTER FORAFFECTIVE...
Eduardo Coutinho - Psychoacoustic cues to emotion in speech prosody and music
Eduardo Coutinho - Psychoacoustic cues to emotion in speech prosody and music
Eduardo Coutinho - Psychoacoustic cues to emotion in speech prosody and music
Eduardo Coutinho - Psychoacoustic cues to emotion in speech prosody and music
Upcoming SlideShare
Loading in …5
×

Eduardo Coutinho - Psychoacoustic cues to emotion in speech prosody and music

1,436 views

Published on

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,436
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Eduardo Coutinho - Psychoacoustic cues to emotion in speech prosody and music

  1. 1. SWISS CENTER FORAFFECTIVE SCIENCESPsychoacoustic cues to emotion inMusic and Speech ProsodyEduardo CoutinhoSwissNex, San FranciscoMay 2013
  2. 2. Aims of this talk• Basic acoustic features allow people to perceiveemotional meaning in music and the human voice• Both media achieve emotional expression throughparticular configurations of these features• It is possible to make accurate predictions of theemotions expressed by music and voice fromacoustic features alone
  3. 3. • Changes in tone of voice while speakingcommunicate emotional meaning (butnot only!) independently of verbalunderstanding• Consistent associations between patternsof acoustic cues (e.g., speech rate, F0,loudness) and particular emotions• Similar emotions are communicatedintra- and cross-culturally throughsimilar arrangements of acoustic cuesSpeech ProsodyJOURNAL OF CROSS-CULTURAL PSYCHOLOGYScherer et al. / VOCAL EMOTION EXPRESSIONWhereas the perception of emotion from facial expression has been extensively studied cross-culturally, lit-tle is known about judges’ ability to infer emotion from vocal cues. This article reports the results from astudy conducted in nine countries in Europe, the United States, and Asia on vocal emotion portrayals ofanger, sadness, fear, joy, and neutral voice as produced by professional German actors. Data show an overallaccuracy of 66% across all emotions and countries. Although accuracy was substantially better than chance,there were sizable differences ranging from 74% in Germany to 52% in Indonesia. However, patterns of con-fusion were very similar across all countries. These data suggest the existence of similar inference rules fromvocal expression across cultures. Generally, accuracy decreased with increasing language dissimilarityfrom German in spite of the use of language-free speech samples. It is concluded that culture- and lan-guage-specific paralinguistic patterns may influence the decoding process.EMOTION INFERENCES FROM VOCAL EXPRESSIONCORRELATE ACROSS LANGUAGES AND CULTURESKLAUS R. SCHERERUniversity of Geneva, SwitzerlandRAINER BANSEHumboldt University Berlin, GermanyHARALD G. WALLBOTTUniversity of Salzburg, AustriaOne of the key issues of current debate in the psychology of emotion concerns the universal-ity versus cultural relativity of emotional expression. This has important implications for thecentral question of the nature and function of emotion. Although there is a general consensusthat both biological and cultural factors contribute to the emotion process (see Mesquita,Frijda, & Scherer, 1997), the relative contribution of each of the factors, or the respectiveamount of variance explained, remains to be explored. An ideal way to study this issue empiri-cally is to compare outward manifestations of emotional reactions with similar appraisals ofDecoding speech prosody in five languagesWILLIAM FORDE THOMPSON and L-L. BALKWILL
  4. 4. Japanese Psychological Research2004, Volume 46, No. 4, 337–349Special Issue: Cognition and emotion in musicMunksgaardORIGINAL ARTICLERecognition of emotion in musicRecognition of emotion in Japanese, Western, andHindustani music by Japanese listeners1LAURA-LEE BALKWILL2Queen’s University, Kingston, Ontario K7L 3N6, CanadaWILLIAM FORDE THOMPSONUniversity of Toronto, Mississauga, Ontario M5S 1A1, CanadaRIE MATSUNAGADepartment of Psychology, Graduate School of Letters, Hokkaido University,Kita-ku, Sapporo 060-0810, JapanAbstract: Japanese listeners rated the expression of joy, anger and sadness in Japanese,Western, and Hindustani music. Excerpts were also rated for tempo, loudness, and com-plexity. Listeners were sensitive to the intended emotion in music from all three cultures,and judgments of emotion were related to judgments of acoustic cues. High ratings of joywere associated with music judged to be fast in tempo and melodically simple. High ratingsof sadness were associated with music judged to be slow in tempo and melodicallycomplex. High ratings of anger were associated with music judged to be louder and morecomplex. The findings suggest that listeners are sensitive to emotion in familiar and un-familiar music, and this sensitivity is associated with the perception of acoustic cues thattranscend cultural boundaries.Key words: recognition of emotion, music, cross-culture.Music is strongly associated with emotions.Evocative music is used in advertising, television,movies, and the music industry, and the effectsare powerful. Listeners readily interpret theRobitaille, 1992; Terwogt & Van Grinsven, 1991).In some cases, listening to music may giverise to changes in mood and arousal (Husain,Thompson, & Schellenberg, 2002; Thompson,Music• Listeners’ perceive emotional meaning byattending (consciously or unconsciously)to structural aspects of the acoustic signal• Consistent associations between acouticpatterns and particular emotions• At least some emotions arecommunicated universally by means ofsimilar acoustic profilesCurrent Biology 19, 1–4, April 14, 2009 ª2009 Elsevier Ltd All rights reserved DOI 10.1016/j.cub.2009.02.058Universal Recognitionof Three Basic Emotions in MusicThomas Fritz,1,* Sebastian Jentschke,2 Nathalie Gosselin,3Daniela Sammler,1 Isabelle Peretz,3 Robert Turner,1Angela D. Friederici,1 and Stefan Koelsch1,4,*1Max Planck Institute for Human Cognitive and Brain Sciences04103 LeipzigGermany2UCL Institute of Child HealthLondon WC1N 1EHUK3Department of Psychology and BRAMSUniversite´ de Montre´ alMontre´ al H2V 4P3Canada4Department of PsychologyPevensey BuildingUniversity of SussexFalmer BN1 9QHUKSummaryIt has long been debated which aspects of music perceptionare universal and which are developed only after exposure toa specific musical culture [1–5]. Here, we report a crosscul-tural study with participants from a native African population(Mafa) and Western participants, with both groups beingnaive to the music of the other respective culture. Experi-ment 1 investigated the ability to recognize three basicemotions (happy, sad, scared/fearful) expressed in Westernmusic. Results show that the Mafas recognized happy, sad,and scared/fearful Western music excerpts above chance,indicating that the expression of these basic emotions inWestern music can be recognized universally. Experiment2 examined how a spectral manipulation of original, natural-istic music affects the perceived pleasantness of music inWestern as well as in Mafa listeners. The spectral manipula-tion modified, among other factors, the sensory dissonanceof the music. The data show that both groups preferred orig-inal Western music and also original Mafa music over theirspectrally manipulated versions. It is likely that the sensorydissonance produced by the spectral manipulation was atleast partly responsible for this effect, suggesting thatconsonance and permanent sensory dissonance universallyinfluence the perceived pleasantness of music.Results and DiscussionThe expression of emotions is a basic feature of Westernmusic, and the capacity of music to convey emotional expres-sions is often regarded as a prerequisite to its appreciation inWestern cultures. This is not necessarily the case in non-Western music cultures, many of which do not similarlyemphasize emotional expressivity, but rather may appreciatemusic for qualities such as group coordination in rituals. Toour knowledge, there has not yet been a contion into the universals of the recognition of esion in music and music appreciation. Thmusical universals with Western music stimuipants who are completely naive to Westernviduals from non-Western cultures who havWestern music occasionally, and perhapexplicit attention to it (e.g., while listeningwatching a movie), do not qualify as partmusical knowledge is usually acquired impeven shaped through inattentive listening exindividuals investigated in our study belonone of approximately 250 ethnic groups thatulation of Cameroon. They are located in thethe Mandara mountain range, an area culta result of a high regional density of endemmore remote Mafa settlements do not haveand are still inhabited by many individuals wtional lifestyle and have never been exposedThe investigation of the recognition of esions conveyed by the music of other cultuaddressed in three previous studies [1, 7,aimed to investigate cues that transcend cuand the authors made an effort to include lprior exposure to the music presentedlistening to Hindustani music). Although thsignificantly enhanced our understandingexperience may influence music perceptionin these studies were exposed to the masalso inadvertently to emotional cues of the rmusic (for example, by the association of thisTo draw clear conclusions about music univis necessary to address music listeners whculturally isolated from one another. Hera research paradigm to investigate the recoemotion in two groups: Mafa listeners naiveand a group of Western listeners naive to Miment 1 was designed to examine the recbasic emotions as expressed by Western mand scared/fearful), using music pieces thapreviously to investigate the recognition ofbrain-damaged patients [9, 10].Data from experiment 1 showed that alexpressions (happy, sad, and scared/fearfulabove chance level by both Western an(Figure 1A, see also Supplemental Data avstatistical evaluation; note that the Mafa lisbeen exposed to Western music before). Hlisteners showed considerable variability in tand 2 of the 21 Mafa participants perfolevel. The mechanism underlying the univeremotional expressions conveyed by Westernappears to be quite similar for both WesternMafas:ananalysisofrating tendenciesrevealand Westerners relied on temporal cues andjudgment of emotional expressions, althougmore marked in Western listeners (see Supand Discussion for details). For the tempo,*Correspondence: fritz@cbs.mpg.de (T.F.), koelsch@cbs.mpg.de (S.K.)Please cite this article in press as: Fritz et al., Universal Recognition of Three Basic Emotions in Music, Curredoi:10.1016/j.cub.2009.02.058
  5. 5. Communication of Emotions in Vocal Expression and Music Performance:Different Channels, Same Code?Patrik N. Juslin and Petri LaukkaUppsala UniversityMany authors have speculated about a close relationship between vocal expression of emotions andmusical expression of emotions, but evidence bearing on this relationship has unfortunately been lacking.This review of 104 studies of vocal expression and 41 studies of music performance reveals similaritiesbetween the 2 channels concerning (a) the accuracy with which discrete emotions were communicatedto listeners and (b) the emotion-specific patterns of acoustic cues used to communicate each emotion. Thepatterns are generally consistent with K. R. Scherer’s (1986) theoretical predictions. The results canexplain why music is perceived as expressive of emotion, and they are consistent with an evolutionaryperspective on vocal expression of emotions. Discussion focuses on theoretical accounts and directionsfor future research.Music: Breathing of statues.Perhaps: Stillness of pictures. You speech, where speeches end.You time, vertically poised on the courses of vanishing hearts.Feelings for what? Oh, you transformation of feelings into. . . audible landscape!that proposals about a close relationship between vocal expressionand music have a long history (Helmholtz, 1863/1954, p. 371;Rousseau, 1761/1986; Spencer, 1857). In a classic article, “TheOrigin and Function of Music,” Spencer (1857) argued that vocalmusic, and hence instrumental music, is intimately related to vocalexpression of emotions. He ventured to explain the characteristicsPsychological Bulletin Copyright 2003 by the American Psychological Association, Inc.2003, Vol. 129, No. 5, 770–814 0033-2909/03/$12.00 DOI: 10.1037/0033-2909.129.5.770Shared expressive acoustic profiles?
  6. 6. Aims• To identify relevant perceptual/acoustic features associatedwith the expression of emotion in both speech and music:• Focus on low-level features in human audition (e.g., loudness,tempo/speech rate, timbre, pitch): cross-modal and universal• To predict the emotional expressivity of natural speech andmusic from these features• Computational methodology previously developed for music• Evaluate the accuracy of predicted emotional expressivityNeural Network Models of Musical Emotions 1EDUARDO COUTINHO AND ANGELO CANGELOSIUniversity of Plymouth, Plymouth, Devon,United KingdomTHIS ARTICLE PRESENTS A NOVEL METHODOLOGY TOanalyze the dynamics of emotional responses to music.It consists of a computational investigation based onin measurement techniques, have contributed with newinsights for such an old question: how does music affectemotions?Cognitivist and Emotivist ViewsThere are two principle, complementary views regard-THE USE OF SPATIO-TEMPORAL CONNECTIONIST MODELSIN PSYCHOLOGICAL STUDIES OF MUSICAL EMOTIONSc2701_01 8/17/09 4:46 PM Page 1Musical Emotions: Predicting Second-by-Second Subjective Feelings ofEmotion From Low-Level Psychoacoustic Features andPhysiological MeasurementsEduardo CoutinhoUniversity of LiverpoolAngelo CangelosiUniversity of PlymouthWe sustain that the structure of affect elicited by music is largely dependent on dynamic temporalpatterns in low-level music structural parameters. In support of this claim, we have previously providedevidence that spatiotemporal dynamics in psychoacoustic features resonate with two psychologicaldimensions of affect underlying judgments of subjective feelings: arousal and valence. In this article weEmotion © 2011 American Psychological Association2011, Vol. 11, No. 4, 921–937 1528-3542/11/$12.00 DOI: 10.1037/a0024700
  7. 7. Empirical experimentsArousalValenceKey issues:1. Continuous measurements framework2. Emotion classification: dimensional model
  8. 8. Empirical experimentsArousalValencePhysiological measurements
  9. 9. Stimuli: Music (film music)
  10. 10. Stimuli: Natural speech (German)
  11. 11. Computationalframework• Recurrent neural network• Nonlinear regression model• Training phase: learn fromhuman listeners• Evaluation phase: perceiveemotional meaning ashuman listenersO1O0HNH0Hnooooooooo ooo CNC0Cnooo oooINPUT UNITSCONTEXTUNITSHIDDENUNITSOUTPUT UNITSPsychoacoustic encodingMusic SpeechPsychoacoustic featuresArousal Valence
  12. 12. (Psycho)acoustic predictorsDynamics Time Pitch Timbre DissonanceLoudnessTempo (BPM)Speech Rate (SPM)Melody ContourProsody ContourSharpness RoughnessSpectral FluxSpectralCentroidSpectralDissonanceDynamics Time Pitch Timbre DissonanceLoudnessTempo (BPM)Speech Rate (SPM)Melody ContourProsody ContourSharpness RoughnessSpectral FluxSpectralCentroidSpectralDissonanceMusicSpeech
  13. 13. Results: Speech* p < 0.05 (exceptValence of sample 6)
  14. 14. Results: Speech0 10 20 30 40 5010.500.51Sample 4ArousalParticipantsModel0 10 20 30 40 5010.500.51Time(s)Valence0 20 40 60 80 10010.500.51Sample 9ArousalParticipantsModel0 20 40 60 80 10010.500.51Time(s)Valence
  15. 15. Results: Music* p < 0.05
  16. 16. Results: Music0 20 40 60 80 100 12010.500.51Piece 1ArousalParticipantsModel0 20 40 60 80 100 12010.500.51Time(s)Valence0 20 40 60 80 10010.500.51Piece 6ArousalParticipantsModel0 20 40 60 80 10010.500.51Time(s)Valence
  17. 17. Summary• Low-level acoustic parameters are fundamentalcues in the expression of emotion in music andspeech.• Affective information is, at least partially, encodedas dynamic acoustic patterns• A significant part of listeners’ perceived emotionsin music and speech can be predicted from a smallset of basic variables in human audition
  18. 18. Possible applications• Diagnosis of psychiatric/psychological conditions (speech)• Improvement of emotional communication proficiency(e.g, speakers, actors, singers, musicians)• Human-computer and computer mediated human-humaninteraction: automatic recognition of emotion in voice• Music selection for ...• ... Mood induction/regulation• ... Cognitive skills enhancement and patients recovery• ... Music in public spaces• Improvement of hearing aids responses to music
  19. 19. Thank you!• Collaborators:• Contact: eduardo.coutinho@unige.ch• More information: www.eadward.orgSWISS CENTER FORAFFECTIVE SCIENCESProf. Angelo CangelosiProf. Nikki Dibben

×