Problems and Solutions in Game Audio

Bits, Bytes and Beats: Problems and Solutions in Video Game Audio

Karen Collins

[email_address]

What Goes Into Game Audio?

Games = Gateway to Geekdom?

Video Games = Evil?

Games = Gateway to Innovation

Outline

History of Game Audio Innovations

Three Fundamental Open Problems:

Mixing

Variation

Adaptability

Directions of my research

Summary/Conclusions

2. A Brief History of

Game Audio

Innovations

(What’s so different about

game audio??)

A Brief Historical Outline

2.1 8-Bit (1970-1990)

2.2 16-Bit (1985-1995)

2.3 64-Bit (1995-2000)

2.4 128-Bit (2000-2005)

2.5 Mobile/Handheld

2.6 Today

Computer Space!

(Nutting Associate 1971)

First game to have sound.

2.1 8-BIT

Pong (Atari 1972) (Al Alcorn) 2.1 8-BIT

Space Invaders (Midway 1978)

First use of continuous “background” music.

2.1 8-BIT

Rally X (Namco/Midway 1980)

The birth of the loop as response to memory constraints.

2.1 8-BIT

Making a “beep” in assembly ( Time & memory constraints )

Beep PROC USES AX BX CX IN AL, 61h ; Save state PUSH AX MOV BX, 6818 ; 1193180/175 MOV AL, 6Bh ; Select Channel 2, write LSB/BSB mode 3 OUT 43h, AL MOV AX, BX OUT 24h, AL ; Send the LSB MOV AL, AH OUT 42h, AL ; Send the MSB IN AL, 61h ; Get the 8255 Port Contence OR AL, 3h OUT 61h, AL ; End able speaker and use clock channel 2 for input MOV CX, 03h ; High order wait value MOV DX 0D04h ; Low order wait value MOV AX, 86h ; Wait service INT 15h POP AX ; restore Speaker state OUT 61h, AL RET BEEP ENDP

From Using Assembly Language by Allen L Wyatt 2.1 8-BIT

Technological Constraints

Up ‘N Down (Sega 1983)

2.1 8-BIT

Atari VCS (2600)

Up ‘N Down (Sega 1984)

2.1 8-BIT

Captain Comic (Color Dreams 1988) 2.1 8-BIT

Captain Comic ’s songs:

Borrowing from classical music.

( Skill constraints )

2.1 8-BIT

Working With Constraints: Nintendo NES

Metroid (Nintendo 1987) (Hip Tanaka)

vibrato (pitch modulation), tremolo (volume modulation), slides, portamento, echo effects

2.1 8-BIT

Ballblazer (LucasArts 1984) (Peter Langston)

Algorithmic generation

“ Riffology” method (Optimized randomness) by Peter Langston

32 eight-note melody fragments

Algorithm chooses how fast, how loud, when to omit notes, when to insert rhythmic break

Developed based on “lazy guitarist”

2.8-BIT

Working With Constraints: Commodore 64 Track 1 (e.g. “Level_One”) Instrument 1 (envelope, waveform, effect filters, etc.) Instrument 2 (envelope, waveform, effect filters, etc.) Rob Hubbard’s Module Format 2.1 8-BIT Pattern 1 (sequence of notes) Pattern 2 (sequence of notes) Pattern 3 (sequence of notes)

; track format:

; high address of pattern to execute

; low address of pattern to execute

; max index in pattern to execute (FF=max, can be terminate by instruction)

; number of times to repeat the given pattern

track1:

.byte >pat00, <pat00, $3D, $02

.byte >pat00, <pat00, $51, $00

.byte >pat01, <pat01, $0A, $04

.byte >pat02, <pat02, $16, $08

.byte >pat03, <pat03, $10, $16

.byte >pat03, <pat03, $FF, $00

.byte >pat0b, <pat0b, $FF, $00

.byte >pat0c, <pat0c, $FF, $00

.byte >pat0d, <pat0d, $FF, $00

.byte >pat0e, <pat0e, $FF, $00

.byte >pat0f, <pat0f, $FF, $00

.byte $00, $00, $00, $00

;C900 pat38: .byte $86, $0D, $00 ; set instrument .byte $23, $D5 .byte $9B, $05, $23 ; play note .byte $86, $0E, $00 ; set instrument .byte $9B, $05, $5A ; play note .byte $D6 .byte $9B, $05, $5A ; play note .byte $86, $0D, $00 ; set instrument .byte $9B, $05, $23 .byte $D5 .byte $9B, $05, $23 ; play note .byte $86, $0E, $00 ; set instrument .byte $9B, $05, $5A ; play note .byte $D6 .byte $9B, $05, $5A ; play note .byte $86, $0F, $00 ; set instrument .byte $00, $D0 .byte $9B, $03 ; restore state .byte $00 .byte $00, $00 2.1 8-BIT

Shadow of the Beast 2 (Psygnosis 1989) (David Whittaker) 2.2 16-BIT MOD/Tracker on Amiga

Combining modules (in MIDI) with control statements MIDI and the Creation of iMUSE 2.2 16-BIT Land, Michael Z. and Peter N. McConnell. Method and Apparatus for Dynamically Composing Music and Sound Effect Using a Computer Entertainment System . US Patent No. 5,315,057. 24 May, 1994.

Super Mario World (Nintendo 1991) (Koji Kondo) 2.2 16-BIT Musical layering technique Mario jumps on Yoshi & gets extra layer of music (SNES).

Legend of Zelda: Ocarina of Time (Nintendo 1999) (Koji Kondo) (N64)

Proximity-based algorithms control cross-fades

2.3 64-BIT

The Sims (Maxis 2000)

Player-input/selectable music

2.4 128-BIT

Music driving gameplay elements. New Super Mario Bros (Nintendo DS 2006) (Koji Kondo) 2.5 Mobile

State-of-the-Art Today

7.1 to 8.1 surround sound

Combination of synth with orchestra, choir

At least 512 channels of sound

God of War (Gerard Marino, Sony 2006)

Bioshock (Gary Schymann, 2K 2007)

2.6 TODAY

3. Three Fundamental Open

Problems

Fundamental Problems

3.1 Mixing

3.2 Repetition. Repetition. Repetition (variability!)

3.3 Adaptability

Pathology : turning off sound/music, cognitive dissonance (failure of music to respond)

> reduces immersiveness

3.1 Mixing Who needs mixing? Chicken Shift (Bally 1984)

Problem: Mixing: Unpredictability, Variability 3.1 Mixing

Problem: Mixing: current state of dynamic range … in a popular film … in a popular game Graphics adapted from those supplied by Rob Bridgett of Swordfish Studios . 3.1 Mixing

Solutions: Real-time Weighted Mixing

Weighted permutations

Predict which sounds can recur without making obvious.

Example:

Dialogue, Sound FX A. Sound FX B, player sounds, music, ambience

If dialogue = “run!”, set parameter to 1

If gunshot is coming towards us, set parameter to 2

If no action, fade out music and raise ambience

REQUIREMENT : “intelligent” Engine to predict and set weighting

3.1 Mixing

Solution: Location-Based Run-Time Mixing

Real-time DSP to adjust sound

E.g. bottle drop on hard floor of kitchen or in next carpeted room

Factor in 5.1 surround to adjust real-time panning

REQUIREMENT : audio engine to pass parameters from game and from player back and forth to engine.

3.1 Mixing

3.2 Variability

Problem : Users get bored with hearing same sounds BUT sound designers can’t possibly record enough variations of sounds (time, budget)

Problem : Users need a new experience every time they play the game (promised by LucasArts’ Euphoria technology)

Problem : audio not responding to physics

Solution: Granular Synthesis 3.2 Variability

“ Granular” Synthesis

“ acoustical quanta ” (Dennis Gabor: 1947 "Acoustical Quanta and the Theory of Hearing." Nature 159 (1044):591-594.)

“ sonic quanta ” (Abraham Moles 1968 ”Information Theory and Esthetic Perception”. Urbana: University of Illinois Press.)

“ particle audio ” (Parker and Behm 2007 ”Generating Audio Textures by Example”, Journal of Game Development, 2007)

3.2 Variability

Granular synthesis: Graphic Equivalent 3.2 Variability Input Sample Synthesized Result "Texture Synthesis from Multiple Sources", by Li-Yi Wei. In SIGGRAPH 2003 Applications and Sketches.

Making a Sound Granular 3.2 Variability Parker and Behm 2007 ”Generating Audio Textures by Example”, Journal of Game Development, 2007

Granular Synthesis Examples

Crowd

Tennis

Speech

3.2 Variability Crowd and speech examples borrowed from Leonard Paul at Vancouver Film School

Granular: Remaining Open Questions

What elements in a sound effect can be varied while still maintaining the “ meaning ” of the sound?

How can we create AI systems that are aware of these potential meanings, and make real-time adjustments to sounds in a game?

How to develop an “ audio physics engine ”: e.g. footsteps change based on how much player is carrying, etc.

3.2 Variability

3.3 Adaptability Problem : Games are non-linear, unpredictable and very long! A to B: 16 units A to all: 376 units 30 rooms: 11280 units 10 levels: 100K+ units Transitional Units

Solution: Game Audio Algorithms

By varying existing individual parameters, we can create algorithms to:

Write transitions

Vary compositions

Create new compositions

Allow user- generate d content

3.3 Adaptability

Variable Musical Parameters

Variable tempo

Variable pitch

Variable rhythm/metre

Variable volume/dynamics

Variable DSP/timbres

Variable harmony (chordal arrangements, key or mode)

Variable mixing : from the speaker placement of certain sounds to run-time adjustments of orchestration mix

3.3 Adaptability

8. Variable form ( open form ) random structure

9. Variable form ( branching parameter-based music)

10. Variable melodies : algorithmic generation

… in what follows we will focus on these last three

3.3 Adaptability

#8 Variable (Open) Form

Random structure

Songs are segmented into components whose order can be changed

Used in “hyrule field” of Legend of Zelda: Ocarina of Time : player spends a lot of time, and the same sequence in the same order would get monotonous

3.3 Adaptability

#8 Example: Variable (Open) Form 3.3 Adaptability http://www.home.cs.utwente.nl/~zsofi/mozart/ Variations: 11 14 x 2 2 = 1 518 999 334 332 964

#8 Variable Form: Non-linear Sequencing

Musical control structures (repetitions, jumps, procedure calls) and grammars modelled on existing characteristics

Music is to some extent already hierarchical (notes > phrases > sections > movements> pieces)  how do we teach/learn to composer in this manner?

“ Grammars as Representations for Music” C. Roads; Paul Wieneke, Computer Music Journal , Vol. 3, No. 1. (Mar., 1979), pp. 48-55.

How can we create sequencing software to better prepare composers to write this type of music?

3.3 Adaptability

#9 Parameter Based Music: Parameters

Number/action of non-playing characters

Number/action of playing characters

Actions

Locations (place, time of day, etc.)

Scripted or unscripted events

Player health or enemy health

Difficulty

Timing

Player properties (skills, endurance)

Bonus objects

Movement (speed, direction, rhythm)

“ Camera” angle

The transition matrix approach and the creation of transitional units 3.3 Adaptability

#9 Example: Parameter-Based Music

No One Lives Forever (Guy Whitmore 2000)

Six standard music states are based on number of NPC enemies:

Silence

Super ambient

Ambient

Suspense/sneak

Action/combat 1

Action/combat 2

3.3 Adaptability

#9 Example: No One Lives Forever

Earth Orbit: Ambush theme starts in music state 5 (combat 1), transitions to music state 2 (ambient: in

elevator)

then transitions to music state 6 (combat 2)

3.3 Adaptability

#10. Algorithmic Variations (ongoing research focus)

Problems:

How do we create emotionally effective algorithmic adaptive audio?

What aspects of audio carry meaning ?

How do these work individually and together?

What universals (within the Western world) are there that can be codified?

How generalized/simplified can/do the rules / grammar need to be?

3.3 Adaptability

Semiotics

Sound/music as a symbolic language

What (and how) does music/sound communicate?

How can we study and break these down into a grammar to generate algorithms?

What combinations are effective?

What variations/substitutions can be made with and without changing meanings?

(For more info, see the work of Philip Tagg, Eero Tarasti, Jean-Jacques Nattiez, and Raymond Monelle; especially Phiip Tagg’s “Ten Little Title Tunes”, Mass Media Music Scholars’ Press 2002)

3.3 Adaptability

Semiotics of sound: Why is it important? An example… 3.3 Adaptability

Revised… 3.3 Adaptability

Defining a Sound Semiotics Grammar

Problem : Can we codify a semiotic grammar of sound? How? How do we gather enough data?

One solution : distributed classification, or crowd sourcing

3.3 Adaptability

Distributed Classification Examples 3.3 Adaptability

What Does the User Get?

Contribution to knowledge

Feeling of being part of community

Believe it or not -- fun!

See Luis von Ahn, “Games with a Purpose” IEEE Computer Magazine or

“ Why do tagging systems work?” Conference on Human Factors in Computing Systems CHI 06

3.3 Adaptability

ESP Game 3.3 Adaptability Player 1 Player 2 GUESSING: KID GUESSING: BOY GUESSING: CAR GUESSING: HAT GUESSING: CAR SUCCESS! Consensus on: CAR Input:

Games for Audio Tagging: Interactively Building an Online Database

Three games under development:

Game like ESP game but for audio (PHP and Flash front end with MySQL backend)

Audio-visual game in which users select image to audio

Audio-visual based game where users select appropriate audio content for visual image

3.3 Adaptability

Adapting the Algorithms For MIR

MIR = Music Information Retrieval

Retrieval based on bpm, harmonic content, melodic intervals, timbre, etc.

How can we use MIR techniques to make better game audio?

User-generated playlists + new algorithms =

appropriate and new user-generated audio content

3.3 Adaptability

4. Summary/Conclusions

Unified Architecture Routing, allocation and scheduling (includes system clocks) Input: Game Data Parameters Detection (Beat tracking, phrase matching, pitch matching, harmony and key matching). Prediction (Neural nets, fuzzy logic.) Wave banks Audio Data (MIDI) Algorithmic composition/modelling Samplers, synths, tone generators Intelligent mixing engine AI Audio Engine

Why CS needs Arts (and vice versa)

Thank-you to … Further information: [email_address] www.GamesSound.com (my web site) www.algorithmic.net www.granularsynthesis.com www.audiokinetic.com www.iasig.org