 PRESENTATION FOR Silicon Knights Karen Collins, University of Waterloo www.GamesSound.com There's Nothing Odd About Audio

FOCUS Who cares about audio? Research directions in game audio The future of game audio

Who cares about audio?

Research directions in game audio

The future of game audio

Who Cares about Game Audio? “ Enhance your gaming world with a custom soundtrack on your Xbox 360™” Call of Duty® 2: “Can anything beat storming enemy positions to the sounds of the Bee Gees ‘Stayin’ Alive’?”

“ Enhance your gaming world with a custom soundtrack on your Xbox 360™”

Call of Duty® 2: “Can anything beat storming enemy positions to the sounds of the Bee Gees ‘Stayin’ Alive’?”

Who Cares about Game Audio?

2007 Tarriff 22A ESA: music “is never the main feature of any communication that might occur on a video game publisher’s site.” Who Cares about Game Audio?

ESA: music “is never the main feature of any communication that might occur on a video game publisher’s site.”

83% of Adult Gamers listed sound as one of the most important video game console elements 47% of Game Console owners (18-25) hook up their console to a home theatre system 48% of Hardcore Gamers said surround sound is a purchase driver for next gen consoles. [According to the ESA] Who Cares about Game Audio?

83% of Adult Gamers listed sound as one of the most important video game console elements

47% of Game Console owners (18-25) hook up their console to a home theatre system

48% of Hardcore Gamers said surround sound is a purchase driver for next gen consoles. [According to the ESA]

Who Cares about Game Audio?

What difference does sound make?

What difference does sound make?

Recent research... (Jørgensen) “ ... I thought it’s duller to play without sound. But I didn’t ... think it affected the game so much.” “ it’s like you’re reminded that this actually is a computer game. So when there’s no sound, it’s just like two animated figures standing there, shooting at each other.” “ It becomes like, you become more systematic, they become almost like numbers. Like, that unit fell, that unit fell, and now I lost one and stuff. There’s almost no emotion behind the fact that they’re falling” “ The first thing I notice is that my time of reaction has increased by ten...” “ I didn’t always notice things happening” “ What difference does sound make to games?

Music driving gameplay elements. New Super Mario Bros (Nintendo DS 2006)‏ (Koji Kondo)‏ What difference does sound make?

What difference does sound make? God of War: excellent mixing, good use of music and sound effects Interesting dialogue and vocal samples -won over a dozen "Game of the Year Awards". -named best PlayStation 2 Action game of all time by. Academy of Interactive Arts & Sciences 9th Annual D.I.C.E. (Design, Innovate, Create, Entertain) Awards: Outstanding Achievement: Original Music Composition and Sound Design --coincidence?

God of War: excellent mixing, good use of music and sound effects

Interesting dialogue and vocal samples

-won over a dozen "Game of the Year Awards".

-named best PlayStation 2 Action game of all time by.

Academy of Interactive Arts & Sciences 9th Annual D.I.C.E. (Design, Innovate, Create, Entertain) Awards: Outstanding Achievement: Original Music Composition and Sound Design

--coincidence?

Current Academic Research, and the Future of Game Audio

A very brief intro to semiotics, and how it can be useful for games…. Semiotics of Sound

Semiotics and Game Audio Sound/music as a symbolic language What (and how) does music/sound communicate? Semiotics helps us to understand what messages are being communicated to us. An Example (“stand-alone” music, not associated with visual image)

Sound/music as a symbolic language

What (and how) does music/sound communicate?

Semiotics helps us to understand what messages are being communicated to us.

An Example: User Testing Futuristic Night time Dark Chase Bad guy/monster/insectoid

Futuristic

Night time

Dark

Chase

Bad guy/monster/insectoid

A Very Quick Semiotic Analysis

A VERY QUICK SEMIOTIC ANALYSIS Where have we heard similar sounding motifs before? It’s militant, so we could check military movies It’s about the future, so we could check futuristic movies...

Where have we heard similar sounding motifs before?

It’s militant, so we could check military movies

It’s about the future, so we could check futuristic movies...

So... What is this saying? =

Our Future= ??

What Does This Have to do with Games? Specific combinations of sounds result in specific meanings How can we study and break these meanings down into “rules”/grammar to generate algorithms for generative audio or user- generated playlists? What combinations are effective? What variations/substitutions can be made with and without changing meanings?

Specific combinations of sounds result in specific meanings

How can we study and break these meanings down into “rules”/grammar to generate algorithms for generative audio or user- generated playlists?

What combinations are effective?

What variations/substitutions can be made with and without changing meanings?

Algorithmically Generative Audio Problems: How do we create emotionally effective algorithmic adaptive audio? What aspects of audio carry meaning ? How do these work individually and together? What universals (within the Western world) are there that can be codified? How generalized/simplified can/do the rules / grammar need to be?

Problems:

How do we create emotionally effective algorithmic adaptive audio?

What aspects of audio carry meaning ?

How do these work individually and together?

What universals (within the Western world) are there that can be codified?

How generalized/simplified can/do the rules / grammar need to be?

Parameter Based Music: Number/action of non-playing characters Number/action of playing characters Actions Locations (place, time of day, etc.)‏ Scripted or unscripted events Player health or enemy health Difficulty Timing Player properties (skills, endurance)‏ Bonus objects Movement (speed, direction, rhythm)‏ “ Camera” angle

Number/action of non-playing characters

Number/action of playing characters

Actions

Locations (place, time of day, etc.)‏

Scripted or unscripted events

Player health or enemy health

Difficulty

Timing

Player properties (skills, endurance)‏

Bonus objects

Movement (speed, direction, rhythm)‏

“ Camera” angle

Generative audio in spore

Generative audio: Ballblazer (LucasArts 1985)

GENERATIVE SOUNDTRACKS: SONY’S CINESCORE

Generative/Procedural Aspects in Games .kkrieger: An FPS in 97 kb: procedurally generated in real time.

Input Sample Synthesized Result "Texture Synthesis from Multiple Sources", by Li-Yi Wei. In SIGGRAPH 2003 Applications and Sketches. Generative Game Elements

Making a Sound Granular Parker and Behm 2007 ”Generating Audio Textures by Example”, Journal of Game Development, 2007

Granular Synthesis Examples Crowd Tennis Speech Crowd and speech examples borrowed from Leonard Paul at Vancouver Film School

Crowd

Tennis

Speech

Granular: Remaining Open Questions What elements in a sound effect can be varied while still maintaining the “ meaning ” of the sound? How can we create AI systems that are aware of these potential meanings, and make real-time adjustments to sounds in a game? How to develop an “ audio physics engine ”: e.g. footsteps change based on how much player is carrying, etc.

What elements in a sound effect can be varied while still maintaining the “ meaning ” of the sound?

How can we create AI systems that are aware of these potential meanings, and make real-time adjustments to sounds in a game?

How to develop an “ audio physics engine ”: e.g. footsteps change based on how much player is carrying, etc.

Methods for better algorithms Neural networks on tagged MIDI files tagging games

Neural networks on tagged MIDI files

tagging games

Distributed Classification Examples

What Does the User Get? Contribution to knowledge Feeling of being part of community fun! See Luis von Ahn, “Games with a Purpose” IEEE Computer Magazine or “ Why do tagging systems work?” Conference on Human Factors in Computing Systems CHI 06

Contribution to knowledge

Feeling of being part of community

fun!

See Luis von Ahn, “Games with a Purpose” IEEE Computer Magazine or

“ Why do tagging systems work?” Conference on Human Factors in Computing Systems CHI 06

ESP Game Player 1 Player 2 GUESSING: KID GUESSING: BOY GUESSING: CAR GUESSING: HAT GUESSING: CAR SUCCESS! Consensus on: CAR Input:

Games for Audio Tagging: Interactively Building an Online Database Three games under development: Game like ESP game but for audio Audio-visual game in which users select image to audio Audio-visual based game where users select appropriate audio content for visual image

Three games under development:

Game like ESP game but for audio

Audio-visual game in which users select image to audio

Audio-visual based game where users select appropriate audio content for visual image

Adapting the Algorithms For MIR MIR = Music Information Retrieval Retrieval based on bpm, harmonic content, melodic intervals, timbre, etc. How can we use MIR techniques to make better game audio? User-generated playlists + new algorithms = appropriate and new user-generated audio content

MIR = Music Information Retrieval

Retrieval based on bpm, harmonic content, melodic intervals, timbre, etc.

How can we use MIR techniques to make better game audio?

User-generated playlists + new algorithms =

appropriate and new user-generated audio content

Unified Architecture Routing, allocation and scheduling (includes system clocks) Input: Game Data Parameters Detection (Beat tracking, phrase matching, pitch matching, harmony and key matching). Prediction (Neural nets, fuzzy logic.) Wave banks Audio Data (MIDI) Algorithmic composition/modelling Samplers, synths, tone generators Intelligent mixing engine AI Audio Engine