The document discusses game audio and the importance of sound in games. It notes that 83% of adult gamers and 47% of game console owners aged 18-25 see sound as an important element. Recent research found that without sound, games feel duller and players notice things happening less. The document then discusses using semiotics to understand the meanings communicated through sound and how generative and algorithmic audio could be used to adapt sound based on in-game events and parameters. Finally, it proposes methods for developing algorithms and user-generated audio content through audio tagging games.

1.  PRESENTATION FOR Silicon Knights Karen Collins, University of Waterloo www.GamesSound.com There's Nothing Odd About Audio

2. FOCUS Who cares about audio? Research directions in game audio The future of game audio

3. Who Cares about Game Audio? “ Enhance your gaming world with a custom soundtrack on your Xbox 360™” Call of Duty® 2: “Can anything beat storming enemy positions to the sounds of the Bee Gees ‘Stayin’ Alive’?”

4. Who Cares about Game Audio?

5. 2007 Tarriff 22A ESA: music “is never the main feature of any communication that might occur on a video game publisher’s site.” Who Cares about Game Audio?

6. 83% of Adult Gamers listed sound as one of the most important video game console elements 47% of Game Console owners (18-25) hook up their console to a home theatre system 48% of Hardcore Gamers said surround sound is a purchase driver for next gen consoles. [According to the ESA] Who Cares about Game Audio?

7. Who Cares about Game Audio?

8. What difference does sound make?

9. What difference does sound make?

10. Recent research... (Jørgensen) “ ... I thought it’s duller to play without sound. But I didn’t ... think it affected the game so much.” “ it’s like you’re reminded that this actually is a computer game. So when there’s no sound, it’s just like two animated figures standing there, shooting at each other.” “ It becomes like, you become more systematic, they become almost like numbers. Like, that unit fell, that unit fell, and now I lost one and stuff. There’s almost no emotion behind the fact that they’re falling” “ The first thing I notice is that my time of reaction has increased by ten...” “ I didn’t always notice things happening” “ What difference does sound make to games?

11. Music driving gameplay elements. New Super Mario Bros (Nintendo DS 2006)‏ (Koji Kondo)‏ What difference does sound make?

12. What difference does sound make? God of War: excellent mixing, good use of music and sound effects Interesting dialogue and vocal samples -won over a dozen "Game of the Year Awards". -named best PlayStation 2 Action game of all time by. Academy of Interactive Arts & Sciences 9th Annual D.I.C.E. (Design, Innovate, Create, Entertain) Awards: Outstanding Achievement: Original Music Composition and Sound Design --coincidence?

13. Current Academic Research, and the Future of Game Audio

14. A very brief intro to semiotics, and how it can be useful for games…. Semiotics of Sound

15. Semiotics and Game Audio Sound/music as a symbolic language What (and how) does music/sound communicate? Semiotics helps us to understand what messages are being communicated to us. An Example (“stand-alone” music, not associated with visual image)

16. An Example: User Testing Futuristic Night time Dark Chase Bad guy/monster/insectoid

17. A Very Quick Semiotic Analysis

18. A VERY QUICK SEMIOTIC ANALYSIS Where have we heard similar sounding motifs before? It’s militant, so we could check military movies It’s about the future, so we could check futuristic movies...

19. So... What is this saying? =

20. Our Future= ??

21. What Does This Have to do with Games? Specific combinations of sounds result in specific meanings How can we study and break these meanings down into “rules”/grammar to generate algorithms for generative audio or user- generated playlists? What combinations are effective? What variations/substitutions can be made with and without changing meanings?

22. Algorithmically Generative Audio Problems: How do we create emotionally effective algorithmic adaptive audio? What aspects of audio carry meaning ? How do these work individually and together? What universals (within the Western world) are there that can be codified? How generalized/simplified can/do the rules / grammar need to be?

23. Parameter Based Music: Number/action of non-playing characters Number/action of playing characters Actions Locations (place, time of day, etc.)‏ Scripted or unscripted events Player health or enemy health Difficulty Timing Player properties (skills, endurance)‏ Bonus objects Movement (speed, direction, rhythm)‏ “ Camera” angle

24. Generative audio in spore

25. Generative audio: Ballblazer (LucasArts 1985)

26. GENERATIVE SOUNDTRACKS: SONY’S CINESCORE

27. Generative/Procedural Aspects in Games .kkrieger: An FPS in 97 kb: procedurally generated in real time.

28. Input Sample Synthesized Result "Texture Synthesis from Multiple Sources", by Li-Yi Wei. In SIGGRAPH 2003 Applications and Sketches. Generative Game Elements

29. Making a Sound Granular Parker and Behm 2007 ”Generating Audio Textures by Example”, Journal of Game Development, 2007

30. Granular Synthesis Examples Crowd Tennis Speech Crowd and speech examples borrowed from Leonard Paul at Vancouver Film School

31. Granular: Remaining Open Questions What elements in a sound effect can be varied while still maintaining the “ meaning ” of the sound? How can we create AI systems that are aware of these potential meanings, and make real-time adjustments to sounds in a game? How to develop an “ audio physics engine ”: e.g. footsteps change based on how much player is carrying, etc.

32. Methods for better algorithms Neural networks on tagged MIDI files tagging games

33. Distributed Classification Examples

34. What Does the User Get? Contribution to knowledge Feeling of being part of community fun! See Luis von Ahn, “Games with a Purpose” IEEE Computer Magazine or “ Why do tagging systems work?” Conference on Human Factors in Computing Systems CHI 06

35. ESP Game Player 1 Player 2 GUESSING: KID GUESSING: BOY GUESSING: CAR GUESSING: HAT GUESSING: CAR SUCCESS! Consensus on: CAR Input:

36. Games for Audio Tagging: Interactively Building an Online Database Three games under development: Game like ESP game but for audio Audio-visual game in which users select image to audio Audio-visual based game where users select appropriate audio content for visual image

37. Adapting the Algorithms For MIR MIR = Music Information Retrieval Retrieval based on bpm, harmonic content, melodic intervals, timbre, etc. How can we use MIR techniques to make better game audio? User-generated playlists + new algorithms = appropriate and new user-generated audio content

38. Unified Architecture Routing, allocation and scheduling (includes system clocks) Input: Game Data Parameters Detection (Beat tracking, phrase matching, pitch matching, harmony and key matching). Prediction (Neural nets, fuzzy logic.) Wave banks Audio Data (MIDI) Algorithmic composition/modelling Samplers, synths, tone generators Intelligent mixing engine AI Audio Engine