Psychologically Motivated Techniques for Inger Ekman Department of Media Technology Helsinki University of Technology Emotional Sound in Computer Games

Agenda Emotions – a short introduction Emotion and fiction Real emotions with unreal events? How do we create emotion with sound? Cognitive-evaluative frameworks and sound Pre-cognitive and unconscious emotion Implications for game sound design

Emotions – a short introduction

Emotion and fiction

Real emotions with unreal events?

How do we create emotion with sound?

Cognitive-evaluative frameworks and sound

Pre-cognitive and unconscious emotion

Implications for game sound design

Emotions Temporary affective reactions to events Evaluation of an event of significance in relation to a goal or situation Involves readiness to act and preparation for certain types of actions + sense of urgency Experienced as a distinctive type of mental state + bodily changes , expressions, actions Compare to mood, personality: longer timescale, lack of directionality

Temporary affective reactions to events

Evaluation of an event of significance in relation to a goal or situation

Involves readiness to act and preparation for certain types of actions + sense of urgency

Experienced as a distinctive type of mental state + bodily changes , expressions, actions

Compare to mood, personality: longer timescale, lack of directionality

What causes emotions? EVALUATIVE processes Emotion is evolutionarily motivated, related to survival: Succeeding, positive, beneficial -> Pleasure, ”pursue” Failing, negative, harmful -> Displeasure, ”avoid” More detail: Anticipated outcome, ”I should be able to do this” Perceived cause, ”That guy is standing in my way” Social aspect, cultural norms, ”I’m supposed to stay calm” ... The social aspect requires being able to understand and predict others’ emotions Empathetically relating to how they would feel in a certain situation Automatically mirroring emotions

EVALUATIVE processes

Emotion is evolutionarily motivated, related to survival:

Succeeding, positive, beneficial -> Pleasure, ”pursue”

Failing, negative, harmful -> Displeasure, ”avoid”

More detail:

Anticipated outcome, ”I should be able to do this”

Perceived cause, ”That guy is standing in my way”

Social aspect, cultural norms, ”I’m supposed to stay calm”

...

The social aspect requires being able to understand and predict others’ emotions

Empathetically relating to how they would feel in a certain situation

Automatically mirroring emotions

” Unconscious emotion” and misattribution Emotions require evaluation, just not necessarily conscious evaluation ” Unconscious emotion” or pre-cognitive value judgements – judgements such as like/dislike can be instantaneous Evaluations may change with time as a result of cognitive evaluation and re-evaluation The process of assigning emotion to certain events is prone to error: Evaluation takes place all the time Bodily processes make us feel different all the time How to distinguish which event causes which emotion? When we don’t get it right -> misattribution

Emotions require evaluation, just not necessarily conscious evaluation

” Unconscious emotion” or pre-cognitive value judgements – judgements such as like/dislike can be instantaneous

Evaluations may change with time as a result of cognitive evaluation and re-evaluation

The process of assigning emotion to certain events is prone to error:

Evaluation takes place all the time

Bodily processes make us feel different all the time

How to distinguish which event causes which emotion?

When we don’t get it right -> misattribution

Emotion and Fiction Emotion relates to evaluation. Why would evaluation of fictional events make us feel anything? Fictive game events become meaningful and cause real emotions through relation with the player Empathy with fictive characters if the events seem relevant Emotional investment in the events on screen requires we have adequate understanding to make sense of events. Evaluations related to progress in game Requires understanding basic rules and investing in goal-related behavior. Unconscious emotion ”colors” our perception of temporally congruent events

Emotion relates to evaluation.

Why would evaluation of fictional events make us feel anything?

Fictive game events become meaningful and cause real emotions through relation with the player

Empathy with fictive characters if the events seem relevant

Emotional investment in the events on screen requires we have adequate understanding to make sense of events.

Evaluations related to progress in game

Requires understanding basic rules and investing in goal-related behavior.

Unconscious emotion ”colors” our perception of temporally congruent events

Sound and Emotion Sound can help create a sense of apparent reality which increases chances of investing emotionally in fictive events Not about realistic sound, but facilitating understanding of the story Narrative fit Sound can facilitate gameplay Any sound that supports meaningful play, helps make sense of own actions in relation to game progress Functional fit Sound provides ample material for ’loose’ emotions Pre-cognitive value judgements: low-level psychology (contrast, affective-mimicry) Cognitive processes not consciously attended to (music, periferal processes)

Sound can help create a sense of apparent reality which increases chances of investing emotionally in fictive events

Not about realistic sound, but facilitating understanding of the story

Narrative fit

Sound can facilitate gameplay

Any sound that supports meaningful play, helps make sense of own actions in relation to game progress

Functional fit

Sound provides ample material for ’loose’ emotions

Pre-cognitive value judgements: low-level psychology (contrast, affective-mimicry)

Cognitive processes not consciously attended to (music, periferal processes)

Sound and Emotion Emotions Influencing perception Emotions from story and playing Game: Goal-related evaluation Story: Empathetic evaluation Game goals match Character goals Moods Bodily states Non-attentional evaluations Unconscious emotion Emotional events Jolly mix of everything

Implications for Game Sound Design Two competing mindsets: story-driven and goal-driven -> Two competing sound design strategies Depend on different evaluation processes Partly contradictory A lot of emotion created with ’loose sound’

Two competing mindsets:

story-driven and goal-driven

-> Two competing sound design strategies

Depend on different evaluation processes

Partly contradictory

A lot of emotion created with ’loose sound’

Alternative Roles for Sound Gaming Focus on playing, doing Guide attention to play-related events & object Sounds highlight multiple choices, alternatives Highlight emotional moments in user decision-making Evaluation related to goal-structure, actions within the game Sounds facilitate doing Story Focus on interpretation Focus attention on important story components Sounds support telling a single story Emphasize predetermined emotional arch Evaluation related to empathetic relation with protagonist Sounds explain/hide non-activity

Gaming

Focus on playing, doing

Guide attention to play-related events & object

Sounds highlight multiple choices, alternatives

Highlight emotional moments in user decision-making

Evaluation related to goal-structure, actions within the game

Sounds facilitate doing

Story

Focus on interpretation

Focus attention on important story components

Sounds support telling a single story

Emphasize predetermined emotional arch

Evaluation related to empathetic relation with protagonist

Sounds explain/hide non-activity