Game audio Karen Collins

OUTLINE Defining Games Audio History Functions of Games Audio Approaches to composition diegesis and dynamic audio

Defining Games Audio

History

Functions of Games Audio

Approaches to composition

diegesis and dynamic audio

Who cares about games? Majority of people in western world play games (75% of heads of households (in the United States) play computer or video games)‏ average age for a game player is thirty nearly half of game players are female (55% male, 43% female). “ Women over the age of 18 represent a greater portion of the gameplaying population (28%) than boys from ages 6 to 17 (21%).

Majority of people in western world play games (75% of heads of households (in the United States) play computer or video games)‏

average age for a game player is thirty

nearly half of game players are female (55% male, 43% female).

“ Women over the age of 18 represent a greater portion of the gameplaying population (28%) than boys from ages 6 to 17 (21%).

Who cares about game audio? 83% of Adult Gamers listed sound as one of the most important video game console elements 47% of Game Console owners (18-25 yr) hook up their game console to a home theatre system 48% of Hardcore Gamers said surround sound is a purchase driver for next generation consoles

83% of Adult Gamers listed sound as one of the most important video game console elements

47% of Game Console owners (18-25 yr) hook up their game console to a home theatre system

48% of Hardcore Gamers said surround sound is a purchase driver for next generation consoles

What is game audio? Dialogue/speech sounds Sound effects Music/score Ambience Interface sounds (menu, etc.)‏

Dialogue/speech sounds

Sound effects

Music/score

Ambience

Interface sounds (menu, etc.)‏

What’s wrong?

How games are not movies: The problems of games audio Linearity Vs. Non-Linearity/ unpredictability, mixing Interactivity with player Temporality (length)‏ Budgets Delivery methods/technology

Linearity Vs. Non-Linearity/ unpredictability, mixing

Interactivity with player

Temporality (length)‏

Budgets

Delivery methods/technology

What is Interactive Audio? sound events occur in direct reaction to a player’s movements. The player triggers the cue , and can repeatedly activate it, such as by making a character jump up and down.

sound events occur in direct reaction to a player’s movements. The player triggers the cue , and can repeatedly activate it, such as by making a character jump up and down.

Interactive Audio: Footsteps Enemy music cue

What is adaptive audio? “ adaptive” audio is generally referred to as sound that reacts to transformations in the gameplay environment—such as a change from day to night set by a game’s timer mechanism . Adaptive audio is not directly triggered by a player

“ adaptive” audio is generally referred to as sound that reacts to transformations in the gameplay environment—such as a change from day to night set by a game’s timer mechanism . Adaptive audio is not directly triggered by a player

Adaptive Audio Change in music day to night Wolf, crow

What is Dynamic Audio Dynamic audio is a broad term to refer to both interactive and adaptive audio

Dynamic audio is a broad term to refer to both interactive and adaptive audio

A brief history of game audio (in 4 minutes!)‏

A brief history of game audio

(in 4 minutes!)‏

History

Functions of Games Sound Commercial functions Structural reinforcement Immersion/suspension of disbelief Semiotics functions Narrative functions Emotional functions Aesthetic functions Kinetic functions

Commercial functions

Structural reinforcement

Immersion/suspension of disbelief

Semiotics functions

Narrative functions

Emotional functions

Aesthetic functions

Kinetic functions

1. Commercial functions Sell games Sell artists Sell soundtracks Sell movies Sell other games Branding: familiarization

Sell games

Sell artists

Sell soundtracks

Sell movies

Sell other games

Branding: familiarization

Sales, marketing and branding

2. Structural reinforcement Mask structure and editing Enhance/reinforce structure of game Create structure Situate player in game matrix Indicate change in narrative Enhance continuity

Mask structure and editing

Enhance/reinforce structure of game

Create structure

Situate player in game matrix

Indicate change in narrative

Enhance continuity

Structural reinforcement

3. Immersion/suspension of disbelief Add realism: Emphasize real sounds Represent location/spatialization Create illusion Cover external noise

Add realism:

Emphasize real sounds

Represent location/spatialization

Create illusion

Cover external noise

Immersion/suspension of disbelief

4. Semiotic functions Sonic symbols, leitmotifs: identify objects and decrease learning curve Preparatory function Acousmatic sound: Focus attention, identify goals Convey meaning, emotion Comment Add slant or bias

Sonic symbols, leitmotifs: identify objects and decrease learning curve

Preparatory function

Acousmatic sound: Focus attention, identify goals

Convey meaning, emotion

Comment

Add slant or bias

Semiotic Preparatory Function

A test of the semiotics of space/place in Zelda Ranch (farm)‏ Windmill Fire temple Evil Castle underground Valley of gypsy thieves Flying fairy Dawn Temple of time Ice cavern A B C D E F G H I

Ranch (farm)‏

Windmill

Fire temple

Evil Castle underground

Valley of gypsy thieves

Flying fairy

Dawn

Temple of time

Ice cavern

5. Narrative functions Advancement of plots Locate player in storyline Anchor player in terms of game matrix/level, etc. anticipate/foreshadow Dialogue can be a major event in plot Access character’s thoughts/reveal character details Reveal goals

Advancement of plots

Locate player in storyline

Anchor player in terms of game matrix/level, etc.

anticipate/foreshadow

Dialogue can be a major event in plot

Access character’s thoughts/reveal character details

Reveal goals

Narrative: correct response

6. Emotional functions Emotional and physiological: manipulate player Disturb or enhance gameplay: battle scenes Mood induction Create intimacy (immersive)‏

Emotional and physiological: manipulate player

Disturb or enhance gameplay: battle scenes

Mood induction

Create intimacy (immersive)‏

Emotional

7. Aesthetic functions Improve our appreciation of the game Make the game better/more enjoyable Make it “cool” Indicate genre/style Inter-textual referencing

Improve our appreciation of the game

Make the game better/more enjoyable

Make it “cool”

Indicate genre/style

Inter-textual referencing

8. Kinetic functions Aerobic exercise: fitness Music as motivating factor for movement Edutainment for toddlers, etc.

Aerobic exercise: fitness

Music as motivating factor for movement

Edutainment for toddlers, etc.

Kinetic Functions Dance Dance Revolution, etc.

Dance Dance Revolution, etc.

Problems (and some solutions) of Interactive Audio in Games

Mixing: Problems: -too many sounds -no “post production” mixing -unpredictable timings : results in “muddy”, clash of sounds

Mixing Solutions! Reduce amount of sounds Control frequencies of sounds Assign priorities in code E.g. dialogue = priority “1” Ambience = priority “4” Gunshot = priority “2” Music = priority “3” Footsteps = priority “5”

Reduce amount of sounds

Control frequencies of sounds

Assign priorities in code

E.g. dialogue = priority “1”

Ambience = priority “4”

Gunshot = priority “2”

Music = priority “3”

Footsteps = priority “5”

Problem: technological constraints Trying to write songs in a few kB for constrained technology such as mobile phones

Trying to write songs in a few kB for constrained technology such as mobile phones

Solutions: technological constraints: Playing a single sample at different pitches to produce variation. The "pitch it up, play it down" technique for saving space.

Playing a single sample at different pitches to produce variation.

The "pitch it up, play it down" technique for saving space.

Sampling to save size This game contains only a single "trumpet fall" sound effect modify the playback sample rate so that each time you pick up a piece of cheese, the sound is played at a different pitch randomly vary the pitch during gameplay, so there's no repeating pattern to it The sound tends to mask the transition from one mix to another, helping to create a more seamless audio experience. same kind of pitch-shifting effect applied to car horn beeps: (fig 1)‏

This game contains only a single "trumpet fall" sound effect

modify the playback sample rate so that each time you pick up a piece of cheese, the sound is played at a different pitch

randomly vary the pitch during gameplay, so there's no repeating pattern to it

The sound tends to mask the transition from one mix to another, helping to create a more seamless audio experience.

same kind of pitch-shifting effect applied to car horn beeps: (fig 1)‏

Sampling to save space Another way to save space is using the "pitch it up, play it down" technique. take your original, high-resolution sound effect and transpose it up an octave, halving the length. Then convert it to a low-resolution compressed format, like this: (cat growl, octave up, car start, octave up. Then in the game, play it down an octave: (octave down). Although the game sound might be a little crunchy, you've just cut the size of your file in half without losing too much audio fidelity. Obviously, the higher you pitch the sound, the "crunchier" the playback will become, but the technique can be used for custom instruments as well as sound effects, and is particularly effective on the tiny speakers in mobile devices.

Another way to save space is using the "pitch it up, play it down" technique.

take your original, high-resolution sound effect and transpose it up an octave, halving the length.

Then convert it to a low-resolution compressed format, like this: (cat growl, octave up, car start, octave up. Then in the game, play it down an octave: (octave down).

Although the game sound might be a little crunchy, you've just cut the size of your file in half without losing too much audio fidelity. Obviously, the higher you pitch the sound, the "crunchier" the playback will become, but the technique can be used for custom instruments as well as sound effects, and is particularly effective on the tiny speakers in mobile devices.

Problem: transitions from cue to cue No “post” = unpredictable timings Abrupt jumps are jarring

Problem: transitions from cue to cue

No “post” = unpredictable timings

Abrupt jumps are jarring

(1) Solution: transitions: Cross fade

Problem: Multi-player interactivity Unscripted events! parameter-based music is difficult: Music cannot be tied to specific events or locations Music cannot be tied to specific parameters

Problem: Multi-player interactivity

Unscripted events!

parameter-based music is difficult:

Music cannot be tied to specific events or locations

Music cannot be tied to specific parameters

Solutions: multi-player interactivity Work with multi-players! Asheron’s Call II, Lord of the Rings: players can bring instruments and jam out with each other

Work with multi-players!

Asheron’s Call II, Lord of the Rings: players can bring instruments and jam out with each other

Multi-player music

Problem: listener fatigue Loops are boring. Loops are boring. Loops are boring. Loops are boring. Loops are boring. Loops are boring. Loops are boring. Loops are boring. Loops are boring. Loops are boring.

Loops are boring.

Loops are boring.

Loops are boring.

Loops are boring.

Loops are boring.

Loops are boring.

Loops are boring.

Loops are boring.

Loops are boring.

Loops are boring.

Solutions: Listener fatigue Learn how to use silence! “ boredom switch”: timer for set levels to stop VARIABILITY!

Learn how to use silence!

“ boredom switch”: timer for set levels to stop

VARIABILITY!

Dynamic Composition: Variability variable tempo variable pitch variable volume variable DSP variable melodies (algorithmic generation)‏ Variable harmony (chord arrangements)‏ Variable harmony (keys)‏ variable mixing Variable sequencing variable structure

variable tempo

variable pitch

variable volume

variable DSP

variable melodies (algorithmic generation)‏

Variable harmony (chord arrangements)‏

Variable harmony (keys)‏

variable mixing

Variable sequencing

variable structure

Variable tempo

Variable tempo

Variable pitch

Variable Pitch

Variable volume

Variable DSP Change DSP effects to change music: E.g. reverb to make it softer, dreamier Phasing to create a “dazed” effect Overdrive to make it more aggressive, etc.

Change DSP effects to change music:

E.g. reverb to make it softer, dreamier

Phasing to create a “dazed” effect

Overdrive to make it more aggressive, etc.

Variable melodies Ballblazer (1984): the music plays forever, without repeating itself but without straying too far from the original theme. The bassline doesn't vary at all, only the lead line in the higher register is algorithmically varied.

Ballblazer (1984): the music plays forever, without repeating itself but without straying too far from the original theme. The bassline doesn't vary at all, only the lead line in the higher register is algorithmically varied.

Variable harmony Asheron’s Call 2: changes in mode indicate changes in gameplay: A player standing alone in an area would hear the background score in Lydian, but as creatures fill the area, the music transitioned to Phrygian. If monsters outnumber player, music transition to diminished mode and quickened drum groove

Asheron’s Call 2: changes in mode indicate changes in gameplay:

A player standing alone in an area would hear the background score in Lydian, but as creatures fill the area, the music transitioned to Phrygian. If monsters outnumber player, music transition to diminished mode and quickened drum groove

Variable mixing Adjust mix by Bringing instruments up in mix that were quiet or silent before Drop elements from mix

Adjust mix by

Bringing instruments up in mix that were quiet or silent before

Drop elements from mix

Cheese Racer Muting and unmuting multiple tracks in a MIDI file to produce various mixes. 2 drum loop samples, three tones: bass, melody and chords In the code for the first level, ten different combinations of tracks: percussion + bass, percussion + bass + melody, bass + melody + chords, etc.

Muting and unmuting multiple tracks in a MIDI file to produce various mixes.

2 drum loop samples,

three tones: bass, melody and chords

In the code for the first level, ten different combinations of tracks: percussion + bass, percussion + bass + melody, bass + melody + chords, etc.

(cont..)‏ Notice how the mix changes every time the mouse gets a piece of cheese. (Fig 2 : exit) (Fig 3)‏ Music loop 40 seconds. 10 different combinations: therefore, each level has more than 6 minutes of different music mixes. “Because the mix changes depending on gameplay, the music will never play exactly the same way twice, thereby increasing variation and decreasing "ear fatigue."” In fact, there are two more sets of tracks playing two more styles of music, using the same tempo and percussion tracks as the first level. Entire game contains almost 20 minutes of various music mixes, using only 68K of compressed sample and MIDI data.

Notice how the mix changes every time the mouse gets a piece of cheese. (Fig 2 : exit) (Fig 3)‏

Music loop 40 seconds. 10 different combinations: therefore, each level has more than 6 minutes of different music mixes. “Because the mix changes depending on gameplay, the music will never play exactly the same way twice, thereby increasing variation and decreasing "ear fatigue."”

In fact, there are two more sets of tracks playing two more styles of music, using the same tempo and percussion tracks as the first level.

Entire game contains almost 20 minutes of various music mixes, using only 68K of compressed sample and MIDI data.

Dynamic Music- example Layer of instrumentation added when Mario rides Yoshi in Super Mario World.

Layer of instrumentation added when Mario rides Yoshi in Super Mario World.

Russian Squares Using layers, each cue increment represents a puzzle row being cleared by the player.

Using layers, each cue increment represents a puzzle row being cleared by the player.

Variable sequencing Alter sequences of music Zelda Ocarina: Hyrule field

Alter sequences of music

Zelda Ocarina: Hyrule field

Variable structure Create “transition matrix”: chart out all possible directions for each sequence, and create transitions for marker points/jumps

Create “transition matrix”: chart out all possible directions for each sequence, and create transitions for marker points/jumps

Transition Matrix: No One Lives Forever Music and Arranging by Guy Whitmore and Tobin Buttram This music clip moves through all six of the music states for the Ambush theme, demonstrating transitions called from a transition matrix. Time Music State 0:00 6 (combat 2) 1:22 transition 1:30 2 (sub-ambient) 2:14 transition 2:16 4 (suspense) 3:10 transition 3:12 3 (ambient) 4:08 transition 4:12 5 (combat 1) 5:33 transition 1 (silence)‏

Music and Arranging by Guy Whitmore and Tobin Buttram

This music clip moves through all six of the music states for the Ambush theme, demonstrating transitions called from a transition matrix. Time Music State 0:00 6 (combat 2) 1:22 transition 1:30 2 (sub-ambient) 2:14 transition 2:16 4 (suspense) 3:10 transition 3:12 3 (ambient) 4:08 transition 4:12 5 (combat 1) 5:33 transition 1 (silence)‏

No One Lives Forever EarthOrbit : The Ambush theme starts in music state 5 (combat 1), transitions to music state 2 (ambient), then transitions to music state 6 (combat 2) with motifs.

EarthOrbit :

The Ambush theme starts in music state 5 (combat 1), transitions to music state 2 (ambient), then transitions to music state 6 (combat 2) with motifs.

What else can be done? Dynamic music offers all kinds of interesting possibilities to have music as a driving force in a game!

Dynamic music offers all kinds of interesting possibilities to have music as a driving force in a game!

Super Mario DS New Super Mario Bros DS Enemies jump and fly in time to music

New Super Mario Bros DS

Enemies jump and fly in time to music

Electroplankton

Rez

Vib Ribbon

Levels of dynamic sound non-dynamic, non-diegetic sound Adaptive non-diegetic sound Interactive non-diegetic sound Non-dynamic diegetic sound Adaptive diegetic sound Interactive diegetic sound Kinetic diegetic and non-diegetic (reminder: Non-Diegetic: Outside the character’s space. Diegetic: “real sounds”, inside character’s space)‏

non-dynamic, non-diegetic sound

Adaptive non-diegetic sound

Interactive non-diegetic sound

Non-dynamic diegetic sound

Adaptive diegetic sound

Interactive diegetic sound

Kinetic diegetic and non-diegetic

(reminder: Non-Diegetic: Outside the character’s space. Diegetic: “real sounds”, inside character’s space)‏

Non-dynamic, non-diegetic sound non-dynamic, linear music or sound found most frequently in the cinematic sequences (movie sequences in which the player’s functions are stopped while a short clip to advance the plot is played). In other words, the player has no control over the possibility of interrupting the music (short of resetting/turning the game off).

non-dynamic, linear music or sound found most frequently in the cinematic sequences (movie sequences in which the player’s functions are stopped while a short clip to advance the plot is played). In other words, the player has no control over the possibility of interrupting the music (short of resetting/turning the game off).

Non-dynamic, non-diegetic sound

Adaptive non-diegetic sound Sound events occurring in reaction to gameplay, but which are unaffected by the player’s direct movements The player cannot re-trigger these events.

Sound events occurring in reaction to gameplay, but which are unaffected by the player’s direct movements

The player cannot re-trigger these events.

Adaptive non-diegetic sound

Interactive non-diegetic sound Sound events occurring in the character’s space The player can re-trigger these events.

Sound events occurring in the character’s space

The player can re-trigger these events.

Interactive non-diegetic sound

Non-dynamic diegetic sound In non-dynamic diegetic audio , the sound event occurs in the character’s space, but the character does not directly interact with it.

In non-dynamic diegetic audio , the sound event occurs in the character’s space, but the character does not directly interact with it.

Non-dynamic diegetic sound

Adaptive diegetic sound Diegetic sounds adapt to gameplay environment (e.g. night to day)‏

Diegetic sounds adapt to gameplay environment (e.g. night to day)‏

Adaptive Diegetic Sounds

Interactive diegetic sounds occur in the character’s space and with which the player’s character can interact. The player instigates the audio cue, but does not necessarily affect the sound of the event once the cue is triggered.

occur in the character’s space and with which the player’s character can interact. The player instigates the audio cue, but does not necessarily affect the sound of the event once the cue is triggered.

Interactive Diegetic Sounds

Kinetic interactive Sound Player interacts with the sound by mimicking the sound/image kinetically Guitar Hero Donkey Konga

Player interacts with the sound by mimicking the sound/image kinetically

Guitar Hero

Donkey Konga