SlideShare a Scribd company logo
1 of 24
Music Emotion Recognition
A State of the Art Review




Dr Scott Beveridge
Interdisciplinary Perspectives on Music, Emotion and Technology
Glasgow Caledonian University, 25th June 2012


© Fraunhofer IDMT
Outline

 Why emotion?


 Definition of Music Emotion Recognition


 History and motivation for interdisciplinary approach


 Current experiments




© Fraunhofer IDMT
Why emotion?
Emotion and Meaning in Music




© Fraunhofer IDMT
Emotion and Meaning in Music

 Humans treat music with great importance


 Music has a very powerful effect
            One of the primary reasons behind
             humans enjoyment of music


 Henry is an example of the listener
  perspective
            Music is useful for those who
             compose and perform also
                 Method of communication and
                  expression

© Fraunhofer IDMT
Emotion and Meaning in Music

 Emotion is a great way of creating information which facilitates browsing and
  organization of music
            Vast number of audio tracks online


 Why are we doing this?
 MER Applications
            Music organization and browsing – personal and commercial
            Academic – Music Digital Libraries (MDL), archiving
            Health – music therapy, pain management (Knox 2011)




© Fraunhofer IDMT
Emotion and Meaning in Music

 Pin down some aspects of the affective process to make the problem
  computationally tractable
            Expression versus Induction


 Most work in MER focusses on expression
            „What the music is trying to say to you‟
            This is easier to decide on with some types of music than others
                     Film music




© Fraunhofer IDMT
A Definition of Music Emotion
Recognition (MER)
 MER has two steps


           1. Identification, recognition and extraction of musical characteristics
              which express emotions in music


           2. Modelling these characteristics in order to make prediction on
              emotions expressed by „new‟ music




© Fraunhofer IDMT
Emotion Recognition       Happy = Fast Tempo,
Supervised Learning               Major Mode

                      Sad = Slow Tempo,
     Happy                  Minor Mode




         Sad




                      ?
© Fraunhofer IDMT
What can machines learn?
Conceptualization of Emotion

 Assign emotion labels (Classification)
            Previous example
            Exuberant, Anxious, Depressed, Content


 Define a point in 2D space (Numeric Prediction)


 Predict emotions which vary over time (Time-continuous
  prediction)




© Fraunhofer IDMT
A Brief History of MER
Background

 Music Psychology           Engineering
 Cate Hevner        1935

                    1988    Kayatose (Symbolic)

 Patrik Juslin       2001
                    1935
                    2003    Feng (Signal-based)




                    NOW



© Fraunhofer IDMT
A Brief History of MER
Popular Music

 Popular music is becoming…..popular!




© Fraunhofer IDMT
Popular Music
Challenges

 By definition popular music is
            Made commercially
                 Limits the scope of expressed emotions
            Made using ever-changing technologies
                 Over production (dynamic compression)


 This generally leads to homogeneity in the popular music genre


 To overcome these problems psychologists, musicologists, philosophers, and
  engineers must work together



© Fraunhofer IDMT
A Brief History of MER
Why Interdisciplinary?

 Olighara 2003
            “One 39 year old male Chinese” annotator for a corpus of Western
             contemporary popular music
 Wu 2006
            10 second music clips and no mention of music


 Schellenberg12
            Manual tempo calculation
 Yang07
            Expression/Induction distinction



© Fraunhofer IDMT
Current Experiments

 Based on 2 steps in MER
            Features
                 Tone Objects
                 Statistical properties of melody


            Modelling
                 Predict tension gradients for use in syncronisation
                 Includes creation of new features
                     Feature fusion




© Fraunhofer IDMT
Current Experiments
Tone Objects

 Objective: Find novel ways of describing popular music by creating new
             musical features


 Existing features
            Tempo, Mode, Key, Instrument Timbre


 New features
            Examine existing features based on tone objects
                 Musical notes of the main melody




© Fraunhofer IDMT
Current Experiments
Tone Objects

 Extracting tone objects involves
  many signal processing
  techniques
            Source separation
            Computational Auditory
             Scene Analysis (CASA)
                 Identify the main
                  melody


 Results shows that tone objects
  help identify particular types of
  emotion


© Fraunhofer IDMT
Current Projects
Main Melody Statistics

 In linguistics, Zipf’s Law shows that: [CAREFUL! These figures might be
  incorrect]
  Given some corpus of natural language the frequency of any word is inversely
                 proportional to its rank in the frequency table
  Word              # of occurrences                Word                 # of occurrences
  the               69,971                          Unison               69,971
  of                36,411                          Major 3rd            36,411
  and               28,852                          Perfect 5th          28,852




 Studies1 have shown that Zipf law statistics have a relationship with aesthetic
  aspects of music – pleasant, beautiful
 Can Zipf‟s law statistics be applied in emotion classification?
                                          1   http://sger.cs.cofc.edu/
© Fraunhofer IDMT
Current Projects
Tension Prediction

 Objective: Track time-continuous tension gradients in film music
 Applications in syncronization task
            Helps creators of films and adverts find music with specific
             characteristics
 Approach:
            Step 1: Extract time-continuous features from a collection of film
             music
            Step 2: Conduct a study which asks people to rate time-continuous
             tension
            Step 3: Build models with the data which predicts tension gradients
             in new music
 An example of supervised learning!

© Fraunhofer IDMT
Current Projects
Tension Prediction – Feature Extraction

 Step1: Extract time-continuous features




© Fraunhofer IDMT
Current Projects
Tension Prediction – Participant Testing

 Step 2: Asked participants to rate music
  based on perceived tension


 General agreement




© Fraunhofer IDMT
Current Projects
Tension Prediction – Participant Testing

 Features most correlated with tension:


            Timbral Complexity: The rate of change of timbre (How many
             „different sounding‟ instruments are present


            Spectral Dissonance: Perceived roughness


            Pure Tonalness: A measure of how „tone-like‟ a sound is




© Fraunhofer IDMT
Current Projects
Tension Prediction – Demonstration




© Fraunhofer IDMT
The Future of MER

 Automatic MER systems are only the beginning


 For MER systems to be truly effective it is necessary to adopt a user-centred
  approach
            Emotions elicited in music are created through social factors and
             environment
                 Listening with friends
                 Listening on the way to work


 Profile users to create bespoke emotion recommendation systems based on
            Geo-location, time of day, skipping behaviour


© Fraunhofer IDMT
Music Emotion Recognition
A State of the Art Review




                         Thank you !!

                      bevest@idmt.fraunhofer.de
                    http://tinyurl.com/bevestLinkedIn


© Fraunhofer IDMT

More Related Content

Similar to Interdisciplinary Perspectives on Emotion, Music and Technology

The convergence of "hard" and "soft"in music technology, Rolf Inge Godøy, UiO
The convergence of "hard" and "soft"in music technology, Rolf Inge Godøy, UiOThe convergence of "hard" and "soft"in music technology, Rolf Inge Godøy, UiO
The convergence of "hard" and "soft"in music technology, Rolf Inge Godøy, UiOThe Research Council of Norway, IKTPLUSS
 
Computational models of symphonic music
Computational models of symphonic musicComputational models of symphonic music
Computational models of symphonic musicEmilia Gómez
 
Musicand interactivity 1
Musicand interactivity 1Musicand interactivity 1
Musicand interactivity 1david mcandrew
 
Wilkie
WilkieWilkie
Wilkieanesah
 
The Impact Of Technology On The Recording Studios
The Impact Of Technology On The Recording StudiosThe Impact Of Technology On The Recording Studios
The Impact Of Technology On The Recording StudiosKaren Thompson
 
An Introduction To Speech Sciences (Acoustic Analysis Of Speech)
An Introduction To Speech Sciences (Acoustic Analysis Of Speech)An Introduction To Speech Sciences (Acoustic Analysis Of Speech)
An Introduction To Speech Sciences (Acoustic Analysis Of Speech)Jeff Nelson
 
Sonification and Speaker Object
Sonification and Speaker ObjectSonification and Speaker Object
Sonification and Speaker Object柏豪 紀
 
IAFPA 2011- 'No Thank You For the Music'
IAFPA 2011- 'No Thank You For the Music' IAFPA 2011- 'No Thank You For the Music'
IAFPA 2011- 'No Thank You For the Music' owrpresentations
 
Marc-André Rappaz - Metaphors, gestures, and emotions in music
Marc-André Rappaz - Metaphors, gestures, and emotions in musicMarc-André Rappaz - Metaphors, gestures, and emotions in music
Marc-André Rappaz - Metaphors, gestures, and emotions in musicswissnex San Francisco
 
The Birth Of Sound Recording Technology
The Birth Of Sound Recording TechnologyThe Birth Of Sound Recording Technology
The Birth Of Sound Recording TechnologyTracy Richards
 
Human Perception and Recognition of Musical Instruments: A Review
Human Perception and Recognition of Musical Instruments: A ReviewHuman Perception and Recognition of Musical Instruments: A Review
Human Perception and Recognition of Musical Instruments: A ReviewEditor IJCATR
 
Denktank 2010
Denktank 2010Denktank 2010
Denktank 2010ocor203
 
Extraction and Conversion of Vocals
Extraction and Conversion of VocalsExtraction and Conversion of Vocals
Extraction and Conversion of VocalsIRJET Journal
 
IRJET- Music Genre Recognition using Convolution Neural Network
IRJET- Music Genre Recognition using Convolution Neural NetworkIRJET- Music Genre Recognition using Convolution Neural Network
IRJET- Music Genre Recognition using Convolution Neural NetworkIRJET Journal
 
Automatic Music Transcription
Automatic Music TranscriptionAutomatic Music Transcription
Automatic Music TranscriptionKhyati Ganatra
 
Taking A Look At Audio Compression
Taking A Look At Audio CompressionTaking A Look At Audio Compression
Taking A Look At Audio CompressionPatty Buckley
 
Creating an Entertaining and Informative Music Visualization
Creating an Entertaining and Informative Music VisualizationCreating an Entertaining and Informative Music Visualization
Creating an Entertaining and Informative Music Visualizationicchp2012
 
CONTENT BASED AUDIO CLASSIFIER & FEATURE EXTRACTION USING ANN TECNIQUES
CONTENT BASED AUDIO CLASSIFIER & FEATURE EXTRACTION USING ANN TECNIQUESCONTENT BASED AUDIO CLASSIFIER & FEATURE EXTRACTION USING ANN TECNIQUES
CONTENT BASED AUDIO CLASSIFIER & FEATURE EXTRACTION USING ANN TECNIQUESAM Publications
 

Similar to Interdisciplinary Perspectives on Emotion, Music and Technology (20)

The convergence of "hard" and "soft"in music technology, Rolf Inge Godøy, UiO
The convergence of "hard" and "soft"in music technology, Rolf Inge Godøy, UiOThe convergence of "hard" and "soft"in music technology, Rolf Inge Godøy, UiO
The convergence of "hard" and "soft"in music technology, Rolf Inge Godøy, UiO
 
Computational models of symphonic music
Computational models of symphonic musicComputational models of symphonic music
Computational models of symphonic music
 
Musicand interactivity 1
Musicand interactivity 1Musicand interactivity 1
Musicand interactivity 1
 
Ai lab workshop(211206)
Ai lab workshop(211206)Ai lab workshop(211206)
Ai lab workshop(211206)
 
Wilkie
WilkieWilkie
Wilkie
 
The Impact Of Technology On The Recording Studios
The Impact Of Technology On The Recording StudiosThe Impact Of Technology On The Recording Studios
The Impact Of Technology On The Recording Studios
 
An Introduction To Speech Sciences (Acoustic Analysis Of Speech)
An Introduction To Speech Sciences (Acoustic Analysis Of Speech)An Introduction To Speech Sciences (Acoustic Analysis Of Speech)
An Introduction To Speech Sciences (Acoustic Analysis Of Speech)
 
Sonification and Speaker Object
Sonification and Speaker ObjectSonification and Speaker Object
Sonification and Speaker Object
 
IAFPA 2011- 'No Thank You For the Music'
IAFPA 2011- 'No Thank You For the Music' IAFPA 2011- 'No Thank You For the Music'
IAFPA 2011- 'No Thank You For the Music'
 
Marc-André Rappaz - Metaphors, gestures, and emotions in music
Marc-André Rappaz - Metaphors, gestures, and emotions in musicMarc-André Rappaz - Metaphors, gestures, and emotions in music
Marc-André Rappaz - Metaphors, gestures, and emotions in music
 
The Birth Of Sound Recording Technology
The Birth Of Sound Recording TechnologyThe Birth Of Sound Recording Technology
The Birth Of Sound Recording Technology
 
Human Perception and Recognition of Musical Instruments: A Review
Human Perception and Recognition of Musical Instruments: A ReviewHuman Perception and Recognition of Musical Instruments: A Review
Human Perception and Recognition of Musical Instruments: A Review
 
Denktank 2010
Denktank 2010Denktank 2010
Denktank 2010
 
Extraction and Conversion of Vocals
Extraction and Conversion of VocalsExtraction and Conversion of Vocals
Extraction and Conversion of Vocals
 
IRJET- Music Genre Recognition using Convolution Neural Network
IRJET- Music Genre Recognition using Convolution Neural NetworkIRJET- Music Genre Recognition using Convolution Neural Network
IRJET- Music Genre Recognition using Convolution Neural Network
 
Automatic Music Transcription
Automatic Music TranscriptionAutomatic Music Transcription
Automatic Music Transcription
 
Taking A Look At Audio Compression
Taking A Look At Audio CompressionTaking A Look At Audio Compression
Taking A Look At Audio Compression
 
Creating an Entertaining and Informative Music Visualization
Creating an Entertaining and Informative Music VisualizationCreating an Entertaining and Informative Music Visualization
Creating an Entertaining and Informative Music Visualization
 
CONTENT BASED AUDIO CLASSIFIER & FEATURE EXTRACTION USING ANN TECNIQUES
CONTENT BASED AUDIO CLASSIFIER & FEATURE EXTRACTION USING ANN TECNIQUESCONTENT BASED AUDIO CLASSIFIER & FEATURE EXTRACTION USING ANN TECNIQUES
CONTENT BASED AUDIO CLASSIFIER & FEATURE EXTRACTION USING ANN TECNIQUES
 
Animal Voice Morphing System
Animal Voice Morphing SystemAnimal Voice Morphing System
Animal Voice Morphing System
 

Recently uploaded

Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveKeep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveIES VE
 
Trailblazer Community - Flows Workshop (Session 2)
Trailblazer Community - Flows Workshop (Session 2)Trailblazer Community - Flows Workshop (Session 2)
Trailblazer Community - Flows Workshop (Session 2)Muhammad Tiham Siddiqui
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
 
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfQ4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfTejal81
 
Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.IPLOOK Networks
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsDianaGray10
 
Extra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfExtra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfInfopole1
 
My key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIMy key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIVijayananda Mohire
 
AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024Brian Pichman
 
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInOutage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInThousandEyes
 
How to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxHow to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxKaustubhBhavsar6
 
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechWebinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechProduct School
 
.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptxHansamali Gamage
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameKapil Thakar
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applicationsnooralam814309
 
Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...DianaGray10
 
UiPath Studio Web workshop series - Day 1
UiPath Studio Web workshop series  - Day 1UiPath Studio Web workshop series  - Day 1
UiPath Studio Web workshop series - Day 1DianaGray10
 
EMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarEMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarThousandEyes
 
LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0DanBrown980551
 
March Patch Tuesday
March Patch TuesdayMarch Patch Tuesday
March Patch TuesdayIvanti
 

Recently uploaded (20)

Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveKeep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
 
Trailblazer Community - Flows Workshop (Session 2)
Trailblazer Community - Flows Workshop (Session 2)Trailblazer Community - Flows Workshop (Session 2)
Trailblazer Community - Flows Workshop (Session 2)
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
 
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfQ4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
 
Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projects
 
Extra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfExtra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdf
 
My key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIMy key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAI
 
AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024
 
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInOutage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
 
How to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxHow to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptx
 
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechWebinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
 
.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First Frame
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applications
 
Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...
 
UiPath Studio Web workshop series - Day 1
UiPath Studio Web workshop series  - Day 1UiPath Studio Web workshop series  - Day 1
UiPath Studio Web workshop series - Day 1
 
EMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarEMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? Webinar
 
LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0
 
March Patch Tuesday
March Patch TuesdayMarch Patch Tuesday
March Patch Tuesday
 

Interdisciplinary Perspectives on Emotion, Music and Technology

  • 1. Music Emotion Recognition A State of the Art Review Dr Scott Beveridge Interdisciplinary Perspectives on Music, Emotion and Technology Glasgow Caledonian University, 25th June 2012 © Fraunhofer IDMT
  • 2. Outline  Why emotion?  Definition of Music Emotion Recognition  History and motivation for interdisciplinary approach  Current experiments © Fraunhofer IDMT
  • 3. Why emotion? Emotion and Meaning in Music © Fraunhofer IDMT
  • 4. Emotion and Meaning in Music  Humans treat music with great importance  Music has a very powerful effect  One of the primary reasons behind humans enjoyment of music  Henry is an example of the listener perspective  Music is useful for those who compose and perform also  Method of communication and expression © Fraunhofer IDMT
  • 5. Emotion and Meaning in Music  Emotion is a great way of creating information which facilitates browsing and organization of music  Vast number of audio tracks online  Why are we doing this?  MER Applications  Music organization and browsing – personal and commercial  Academic – Music Digital Libraries (MDL), archiving  Health – music therapy, pain management (Knox 2011) © Fraunhofer IDMT
  • 6. Emotion and Meaning in Music  Pin down some aspects of the affective process to make the problem computationally tractable  Expression versus Induction  Most work in MER focusses on expression  „What the music is trying to say to you‟  This is easier to decide on with some types of music than others  Film music © Fraunhofer IDMT
  • 7. A Definition of Music Emotion Recognition (MER)  MER has two steps 1. Identification, recognition and extraction of musical characteristics which express emotions in music 2. Modelling these characteristics in order to make prediction on emotions expressed by „new‟ music © Fraunhofer IDMT
  • 8. Emotion Recognition Happy = Fast Tempo, Supervised Learning Major Mode Sad = Slow Tempo, Happy Minor Mode Sad ? © Fraunhofer IDMT
  • 9. What can machines learn? Conceptualization of Emotion  Assign emotion labels (Classification)  Previous example  Exuberant, Anxious, Depressed, Content  Define a point in 2D space (Numeric Prediction)  Predict emotions which vary over time (Time-continuous prediction) © Fraunhofer IDMT
  • 10. A Brief History of MER Background Music Psychology Engineering Cate Hevner 1935 1988 Kayatose (Symbolic) Patrik Juslin 2001 1935 2003 Feng (Signal-based) NOW © Fraunhofer IDMT
  • 11. A Brief History of MER Popular Music  Popular music is becoming…..popular! © Fraunhofer IDMT
  • 12. Popular Music Challenges  By definition popular music is  Made commercially  Limits the scope of expressed emotions  Made using ever-changing technologies  Over production (dynamic compression)  This generally leads to homogeneity in the popular music genre  To overcome these problems psychologists, musicologists, philosophers, and engineers must work together © Fraunhofer IDMT
  • 13. A Brief History of MER Why Interdisciplinary?  Olighara 2003  “One 39 year old male Chinese” annotator for a corpus of Western contemporary popular music  Wu 2006  10 second music clips and no mention of music  Schellenberg12  Manual tempo calculation  Yang07  Expression/Induction distinction © Fraunhofer IDMT
  • 14. Current Experiments  Based on 2 steps in MER  Features  Tone Objects  Statistical properties of melody  Modelling  Predict tension gradients for use in syncronisation  Includes creation of new features  Feature fusion © Fraunhofer IDMT
  • 15. Current Experiments Tone Objects  Objective: Find novel ways of describing popular music by creating new musical features  Existing features  Tempo, Mode, Key, Instrument Timbre  New features  Examine existing features based on tone objects  Musical notes of the main melody © Fraunhofer IDMT
  • 16. Current Experiments Tone Objects  Extracting tone objects involves many signal processing techniques  Source separation  Computational Auditory Scene Analysis (CASA)  Identify the main melody  Results shows that tone objects help identify particular types of emotion © Fraunhofer IDMT
  • 17. Current Projects Main Melody Statistics  In linguistics, Zipf’s Law shows that: [CAREFUL! These figures might be incorrect] Given some corpus of natural language the frequency of any word is inversely proportional to its rank in the frequency table Word # of occurrences Word # of occurrences the 69,971 Unison 69,971 of 36,411 Major 3rd 36,411 and 28,852 Perfect 5th 28,852  Studies1 have shown that Zipf law statistics have a relationship with aesthetic aspects of music – pleasant, beautiful  Can Zipf‟s law statistics be applied in emotion classification? 1 http://sger.cs.cofc.edu/ © Fraunhofer IDMT
  • 18. Current Projects Tension Prediction  Objective: Track time-continuous tension gradients in film music  Applications in syncronization task  Helps creators of films and adverts find music with specific characteristics  Approach:  Step 1: Extract time-continuous features from a collection of film music  Step 2: Conduct a study which asks people to rate time-continuous tension  Step 3: Build models with the data which predicts tension gradients in new music  An example of supervised learning! © Fraunhofer IDMT
  • 19. Current Projects Tension Prediction – Feature Extraction  Step1: Extract time-continuous features © Fraunhofer IDMT
  • 20. Current Projects Tension Prediction – Participant Testing  Step 2: Asked participants to rate music based on perceived tension  General agreement © Fraunhofer IDMT
  • 21. Current Projects Tension Prediction – Participant Testing  Features most correlated with tension:  Timbral Complexity: The rate of change of timbre (How many „different sounding‟ instruments are present  Spectral Dissonance: Perceived roughness  Pure Tonalness: A measure of how „tone-like‟ a sound is © Fraunhofer IDMT
  • 22. Current Projects Tension Prediction – Demonstration © Fraunhofer IDMT
  • 23. The Future of MER  Automatic MER systems are only the beginning  For MER systems to be truly effective it is necessary to adopt a user-centred approach  Emotions elicited in music are created through social factors and environment  Listening with friends  Listening on the way to work  Profile users to create bespoke emotion recommendation systems based on  Geo-location, time of day, skipping behaviour © Fraunhofer IDMT
  • 24. Music Emotion Recognition A State of the Art Review Thank you !! bevest@idmt.fraunhofer.de http://tinyurl.com/bevestLinkedIn © Fraunhofer IDMT

Editor's Notes

  1. V. Moving video. Most interesting part is the music has some ‘residual effect’ – Henry is animated, lucid …. expressive
  2. V. Moving video. Most interesting part is the music has some ‘residual effect’ – Henry is animated, lucid …. expressive
  3. V. Moving video. Most interesting part is the music has some ‘residual effect’ – Henry is animated, lucid …. Expressive<<David Cameron’s Fav album>>Who cares? What does this mean? and Why is this in the news?Because Music is important to people
  4. V. Moving video. Most interesting part is the music has some ‘residual effect’ – Henry is animated, lucid …. expressive
  5. MER and the tasks mentioned is a very young field. Psychology longer. Going to give a history in a few slides
  6. Recently updated literature review. Reason Mention Survey of 62 conference articles and journal papersMany different disciplines Computer Science and engineering
  7. Recently updated literature review. Reason Mention Survey of 62 conference articles and journal papersMany different disciplines Computer Science and engineering