Problems and Solutions in Game Audio


Published on

Cognos Innovation Lecture 2007--Karen Collins Game Audio overview and discussion of future developments and directions

  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Introduce self, thank Cognos and Carleton.
  • Problems and Solutions in Game Audio

    1. 1. <ul><li>Bits, Bytes and Beats: Problems and Solutions in Video Game Audio </li></ul><ul><li>Karen Collins </li></ul><ul><li>[email_address] </li></ul>
    2. 2. What Goes Into Game Audio?
    3. 3. Games = Gateway to Geekdom?
    4. 4. Video Games = Evil?
    5. 5. Games = Gateway to Innovation
    6. 6. Outline <ul><li>History of Game Audio Innovations </li></ul><ul><li>Three Fundamental Open Problems: </li></ul><ul><ul><ul><li>Mixing </li></ul></ul></ul><ul><ul><ul><li>Variation </li></ul></ul></ul><ul><ul><ul><li>Adaptability </li></ul></ul></ul><ul><ul><ul><ul><li>Directions of my research </li></ul></ul></ul></ul><ul><li>Summary/Conclusions </li></ul>
    7. 7. <ul><li>2. A Brief History of </li></ul><ul><li>Game Audio </li></ul><ul><li>Innovations </li></ul><ul><li>(What’s so different about </li></ul><ul><li>game audio??) </li></ul>
    8. 8. A Brief Historical Outline <ul><li>2.1 8-Bit (1970-1990) </li></ul><ul><li>2.2 16-Bit (1985-1995) </li></ul><ul><li>2.3 64-Bit (1995-2000) </li></ul><ul><li>2.4 128-Bit (2000-2005) </li></ul><ul><li>2.5 Mobile/Handheld </li></ul><ul><li>2.6 Today </li></ul>
    9. 9. <ul><li>Computer Space! </li></ul><ul><li>(Nutting Associate 1971) </li></ul><ul><li>First game to have sound. </li></ul>2.1 8-BIT
    10. 10. Pong (Atari 1972) (Al Alcorn) 2.1 8-BIT
    11. 11. Space Invaders (Midway 1978) <ul><li>First use of continuous “background” music. </li></ul>2.1 8-BIT
    12. 12. Rally X (Namco/Midway 1980) <ul><li>The birth of the loop as response to memory constraints. </li></ul>2.1 8-BIT
    13. 13. Making a “beep” in assembly ( Time & memory constraints ) <ul><li>Beep   PROC USES AX BX CX     IN AL, 61h  ; Save state     PUSH AX           MOV BX, 6818 ; 1193180/175     MOV AL, 6Bh  ; Select Channel 2, write LSB/BSB mode 3     OUT 43h, AL         MOV AX, BX         OUT 24h, AL  ; Send the LSB     MOV AL, AH         OUT 42h, AL  ; Send the MSB     IN AL, 61h     ; Get the 8255 Port Contence     OR AL, 3h             OUT 61h, AL  ; End able speaker and use clock channel 2 for input     MOV CX, 03h ; High order wait value     MOV DX 0D04h ; Low order wait value     MOV AX, 86h ; Wait service     INT 15h                 POP AX ; restore Speaker state     OUT 61h, AL     RET BEEP ENDP </li></ul>From Using Assembly Language by Allen L Wyatt 2.1 8-BIT
    14. 14. Technological Constraints <ul><li>Up ‘N Down (Sega 1983) </li></ul>2.1 8-BIT
    15. 15. Atari VCS (2600) <ul><li>Up ‘N Down (Sega 1984) </li></ul>2.1 8-BIT
    16. 16. Captain Comic (Color Dreams 1988) 2.1 8-BIT
    17. 17. <ul><li>Captain Comic ’s songs: </li></ul><ul><li>Borrowing from classical music. </li></ul><ul><li>( Skill constraints ) </li></ul>2.1 8-BIT
    18. 18. Working With Constraints: Nintendo NES <ul><li>Metroid (Nintendo 1987) (Hip Tanaka) </li></ul><ul><li>vibrato (pitch modulation), tremolo (volume modulation), slides, portamento, echo effects </li></ul>2.1 8-BIT
    19. 19. Ballblazer (LucasArts 1984) (Peter Langston) <ul><li>Algorithmic generation </li></ul><ul><li>“ Riffology” method (Optimized randomness) by Peter Langston </li></ul><ul><li>32 eight-note melody fragments </li></ul><ul><li>Algorithm chooses how fast, how loud, when to omit notes, when to insert rhythmic break </li></ul><ul><li>Developed based on “lazy guitarist” </li></ul>2.8-BIT
    20. 20. Working With Constraints: Commodore 64 Track 1 (e.g. “Level_One”) Instrument 1 (envelope, waveform, effect filters, etc.) Instrument 2 (envelope, waveform, effect filters, etc.) Rob Hubbard’s Module Format 2.1 8-BIT Pattern 1 (sequence of notes) Pattern 2 (sequence of notes) Pattern 3 (sequence of notes)
    21. 21. <ul><li>; track format: </li></ul><ul><li>; high address of pattern to execute </li></ul><ul><li>; low address of pattern to execute </li></ul><ul><li>; max index in pattern to execute (FF=max, can be terminate by instruction) </li></ul><ul><li>; number of times to repeat the given pattern </li></ul><ul><li>track1: </li></ul><ul><li>.byte >pat00, <pat00, $3D, $02 </li></ul><ul><li>.byte >pat00, <pat00, $51, $00 </li></ul><ul><li>.byte >pat01, <pat01, $0A, $04 </li></ul><ul><li>.byte >pat02, <pat02, $16, $08 </li></ul><ul><li>.byte >pat03, <pat03, $10, $16 </li></ul><ul><li>.byte >pat03, <pat03, $FF, $00 </li></ul><ul><li>.byte >pat0b, <pat0b, $FF, $00 </li></ul><ul><li>.byte >pat0c, <pat0c, $FF, $00 </li></ul><ul><li>.byte >pat0d, <pat0d, $FF, $00 </li></ul><ul><li>.byte >pat0e, <pat0e, $FF, $00 </li></ul><ul><li>.byte >pat0f, <pat0f, $FF, $00 </li></ul><ul><li>.byte >pat10, <pat10, $FF, $00 </li></ul><ul><li>.byte >pat11, <pat11, $FF, $00 </li></ul><ul><li>.byte >pat12, <pat12, $FF, $00 </li></ul><ul><li>.byte >pat13, <pat13, $FF, $00 </li></ul><ul><li>.byte >pat14, <pat14, $FF, $00 </li></ul><ul><li>.byte >pat15, <pat15, $FF, $00 </li></ul><ul><li>.byte >pat16, <pat16, $FF, $00 </li></ul><ul><li>.byte >pat17, <pat17, $FF, $00 </li></ul><ul><li>.byte >pat18, <pat18, $FF, $00 </li></ul><ul><li>.byte >pat26, <pat26, $FF, $00 </li></ul><ul><li>.byte >pat37, <pat37, $FF, $00 </li></ul><ul><li>.byte >pat27, <pat27, $FF, $00 </li></ul><ul><li>.byte $00, $00, $00, $00 </li></ul>;C900 pat38: .byte $86, $0D, $00 ; set instrument .byte $23, $D5 .byte $9B, $05, $23 ; play note .byte $86, $0E, $00 ; set instrument .byte $9B, $05, $5A ; play note .byte $D6 .byte $9B, $05, $5A ; play note .byte $86, $0D, $00 ; set instrument .byte $9B, $05, $23 .byte $D5 .byte $9B, $05, $23 ; play note .byte $86, $0E, $00 ; set instrument .byte $9B, $05, $5A ; play note .byte $D6 .byte $9B, $05, $5A ; play note .byte $86, $0F, $00 ; set instrument .byte $00, $D0 .byte $9B, $03 ; restore state .byte $00 .byte $00, $00 2.1 8-BIT
    22. 22. Shadow of the Beast 2 (Psygnosis 1989) (David Whittaker) 2.2 16-BIT MOD/Tracker on Amiga
    23. 23. Combining modules (in MIDI) with control statements MIDI and the Creation of iMUSE 2.2 16-BIT Land, Michael Z. and Peter N. McConnell. Method and Apparatus for Dynamically Composing Music and Sound Effect Using a Computer Entertainment System . US Patent No. 5,315,057. 24 May, 1994.
    24. 24. Super Mario World (Nintendo 1991) (Koji Kondo) 2.2 16-BIT Musical layering technique Mario jumps on Yoshi & gets extra layer of music (SNES).
    25. 25. Legend of Zelda: Ocarina of Time (Nintendo 1999) (Koji Kondo) (N64) <ul><li>Proximity-based algorithms control cross-fades </li></ul>2.3 64-BIT
    26. 26. The Sims (Maxis 2000) <ul><li>Player-input/selectable music </li></ul>2.4 128-BIT
    27. 27. Music driving gameplay elements. New Super Mario Bros (Nintendo DS 2006) (Koji Kondo) 2.5 Mobile
    28. 28. State-of-the-Art Today <ul><li>7.1 to 8.1 surround sound </li></ul><ul><li>Combination of synth with orchestra, choir </li></ul><ul><li>At least 512 channels of sound </li></ul><ul><li>God of War (Gerard Marino, Sony 2006) </li></ul><ul><li>Bioshock (Gary Schymann, 2K 2007) </li></ul>2.6 TODAY
    29. 29. <ul><li>3. Three Fundamental Open </li></ul><ul><li>Problems </li></ul>
    30. 30. Fundamental Problems <ul><li>3.1 Mixing </li></ul><ul><li>3.2 Repetition. Repetition. Repetition (variability!) </li></ul><ul><li>3.3 Adaptability </li></ul><ul><li>Pathology : turning off sound/music, cognitive dissonance (failure of music to respond) </li></ul><ul><li>> reduces immersiveness </li></ul>
    31. 31. 3.1 Mixing Who needs mixing? Chicken Shift (Bally 1984)
    32. 32. Problem: Mixing: Unpredictability, Variability 3.1 Mixing
    33. 33. Problem: Mixing: current state of dynamic range … in a popular film … in a popular game Graphics adapted from those supplied by Rob Bridgett of Swordfish Studios . 3.1 Mixing
    34. 34. Solutions: Real-time Weighted Mixing <ul><li>Weighted permutations </li></ul><ul><ul><li>Predict which sounds can recur without making obvious. </li></ul></ul><ul><li>Example: </li></ul><ul><ul><li>Dialogue, Sound FX A. Sound FX B, player sounds, music, ambience </li></ul></ul><ul><ul><li>If dialogue = “run!”, set parameter to 1 </li></ul></ul><ul><ul><li>If gunshot is coming towards us, set parameter to 2 </li></ul></ul><ul><ul><li>If no action, fade out music and raise ambience </li></ul></ul><ul><ul><li>REQUIREMENT : “intelligent” Engine to predict and set weighting </li></ul></ul>3.1 Mixing
    35. 35. Solution: Location-Based Run-Time Mixing <ul><li>Real-time DSP to adjust sound </li></ul><ul><li>E.g. bottle drop on hard floor of kitchen or in next carpeted room </li></ul><ul><li>Factor in 5.1 surround to adjust real-time panning </li></ul><ul><li>REQUIREMENT : audio engine to pass parameters from game and from player back and forth to engine. </li></ul>3.1 Mixing
    36. 36. 3.2 Variability <ul><li>Problem : Users get bored with hearing same sounds BUT sound designers can’t possibly record enough variations of sounds (time, budget) </li></ul><ul><li>Problem : Users need a new experience every time they play the game (promised by LucasArts’ Euphoria technology) </li></ul><ul><li>Problem : audio not responding to physics </li></ul>
    37. 37. Solution: Granular Synthesis 3.2 Variability
    38. 38. “ Granular” Synthesis <ul><li>“ acoustical quanta ” (Dennis Gabor: 1947 &quot;Acoustical Quanta and the Theory of Hearing.&quot; Nature 159 (1044):591-594.) </li></ul><ul><li>“ sonic quanta ” (Abraham Moles 1968 ”Information Theory and Esthetic Perception”. Urbana: University of Illinois Press.) </li></ul><ul><li>“ particle audio ” (Parker and Behm 2007 ”Generating Audio Textures by Example”, Journal of Game Development, 2007) </li></ul>3.2 Variability
    39. 39. Granular synthesis: Graphic Equivalent 3.2 Variability Input Sample Synthesized Result &quot;Texture Synthesis from Multiple Sources&quot;, by Li-Yi Wei. In SIGGRAPH 2003 Applications and Sketches.
    40. 40. Making a Sound Granular 3.2 Variability Parker and Behm 2007 ”Generating Audio Textures by Example”, Journal of Game Development, 2007
    41. 41. Granular Synthesis Examples <ul><li>Crowd </li></ul><ul><li>Tennis </li></ul><ul><li>Speech </li></ul>3.2 Variability Crowd and speech examples borrowed from Leonard Paul at Vancouver Film School
    42. 42. Granular: Remaining Open Questions <ul><li>What elements in a sound effect can be varied while still maintaining the “ meaning ” of the sound? </li></ul><ul><li>How can we create AI systems that are aware of these potential meanings, and make real-time adjustments to sounds in a game? </li></ul><ul><li>How to develop an “ audio physics engine ”: e.g. footsteps change based on how much player is carrying, etc. </li></ul>3.2 Variability
    43. 43. 3.3 Adaptability Problem : Games are non-linear, unpredictable and very long! A to B: 16 units A to all: 376 units 30 rooms: 11280 units 10 levels: 100K+ units Transitional Units
    44. 44. Solution: Game Audio Algorithms <ul><li>By varying existing individual parameters, we can create algorithms to: </li></ul><ul><ul><li>Write transitions </li></ul></ul><ul><ul><li>Vary compositions </li></ul></ul><ul><ul><li>Create new compositions </li></ul></ul><ul><ul><li>Allow user- generate d content </li></ul></ul>3.3 Adaptability
    45. 45. Variable Musical Parameters <ul><li>Variable tempo </li></ul><ul><li>Variable pitch </li></ul><ul><li>Variable rhythm/metre </li></ul><ul><li>Variable volume/dynamics </li></ul><ul><li>Variable DSP/timbres </li></ul><ul><li>Variable harmony (chordal arrangements, key or mode) </li></ul><ul><li>Variable mixing : from the speaker placement of certain sounds to run-time adjustments of orchestration mix </li></ul>3.3 Adaptability
    46. 46. <ul><li>8. Variable form ( open form ) random structure </li></ul><ul><li>9. Variable form ( branching parameter-based music) </li></ul><ul><li>10. Variable melodies : algorithmic generation </li></ul><ul><li>… in what follows we will focus on these last three </li></ul>3.3 Adaptability
    47. 47. #8 Variable (Open) Form <ul><li>Random structure </li></ul><ul><li>Songs are segmented into components whose order can be changed </li></ul><ul><li>Used in “hyrule field” of Legend of Zelda: Ocarina of Time : player spends a lot of time, and the same sequence in the same order would get monotonous </li></ul>3.3 Adaptability
    48. 48. #8 Example: Variable (Open) Form 3.3 Adaptability Variations: 11 14 x 2 2 = 1 518 999 334 332 964
    49. 49. #8 Variable Form: Non-linear Sequencing <ul><li>Musical control structures (repetitions, jumps, procedure calls) and grammars modelled on existing characteristics </li></ul><ul><li>Music is to some extent already hierarchical (notes > phrases > sections > movements> pieces)  how do we teach/learn to composer in this manner? </li></ul><ul><li>“ Grammars as Representations for Music” C. Roads; Paul Wieneke, Computer Music Journal , Vol. 3, No. 1. (Mar., 1979), pp. 48-55. </li></ul><ul><li>How can we create sequencing software to better prepare composers to write this type of music? </li></ul>3.3 Adaptability
    50. 50. #9 Parameter Based Music: Parameters <ul><li>Number/action of non-playing characters </li></ul><ul><li>Number/action of playing characters </li></ul><ul><li>Actions </li></ul><ul><li>Locations (place, time of day, etc.) </li></ul><ul><li>Scripted or unscripted events </li></ul><ul><li>Player health or enemy health </li></ul><ul><li>Difficulty </li></ul><ul><li>Timing </li></ul><ul><li>Player properties (skills, endurance) </li></ul><ul><li>Bonus objects </li></ul><ul><li>Movement (speed, direction, rhythm) </li></ul><ul><li>“ Camera” angle </li></ul>The transition matrix approach and the creation of transitional units 3.3 Adaptability
    51. 51. #9 Example: Parameter-Based Music <ul><li>No One Lives Forever (Guy Whitmore 2000) </li></ul><ul><li>Six standard music states are based on number of NPC enemies: </li></ul><ul><li>Silence </li></ul><ul><li>Super ambient </li></ul><ul><li>Ambient </li></ul><ul><li>Suspense/sneak </li></ul><ul><li>Action/combat 1 </li></ul><ul><li>Action/combat 2 </li></ul>3.3 Adaptability
    52. 52. #9 Example: No One Lives Forever <ul><li>Earth Orbit: Ambush theme starts in music state 5 (combat 1), transitions to music state 2 (ambient: in </li></ul><ul><li>elevator) </li></ul><ul><li>then transitions to music state 6 (combat 2) </li></ul>3.3 Adaptability
    53. 53. #10. Algorithmic Variations (ongoing research focus) <ul><li>Problems: </li></ul><ul><li>How do we create emotionally effective algorithmic adaptive audio? </li></ul><ul><li>What aspects of audio carry meaning ? </li></ul><ul><ul><li>How do these work individually and together? </li></ul></ul><ul><ul><li>What universals (within the Western world) are there that can be codified? </li></ul></ul><ul><li>How generalized/simplified can/do the rules / grammar need to be? </li></ul>3.3 Adaptability
    54. 54. Semiotics <ul><li>Sound/music as a symbolic language </li></ul><ul><li>What (and how) does music/sound communicate? </li></ul><ul><li>How can we study and break these down into a grammar to generate algorithms? </li></ul><ul><ul><li>What combinations are effective? </li></ul></ul><ul><ul><li>What variations/substitutions can be made with and without changing meanings? </li></ul></ul><ul><li>(For more info, see the work of Philip Tagg, Eero Tarasti, Jean-Jacques Nattiez, and Raymond Monelle; especially Phiip Tagg’s “Ten Little Title Tunes”, Mass Media Music Scholars’ Press 2002) </li></ul>3.3 Adaptability
    55. 55. Semiotics of sound: Why is it important? An example… 3.3 Adaptability
    56. 56. Revised… 3.3 Adaptability
    57. 57. Defining a Sound Semiotics Grammar <ul><li>Problem : Can we codify a semiotic grammar of sound? How? How do we gather enough data? </li></ul><ul><li>One solution : distributed classification, or crowd sourcing </li></ul>3.3 Adaptability
    58. 58. Distributed Classification Examples 3.3 Adaptability
    59. 59. What Does the User Get? <ul><li>Contribution to knowledge </li></ul><ul><li>Feeling of being part of community </li></ul><ul><li>Believe it or not -- fun! </li></ul><ul><li>See Luis von Ahn, “Games with a Purpose” IEEE Computer Magazine or </li></ul><ul><li>“ Why do tagging systems work?” Conference on Human Factors in Computing Systems CHI 06 </li></ul>3.3 Adaptability
    60. 60. ESP Game 3.3 Adaptability Player 1 Player 2 GUESSING: KID GUESSING: BOY GUESSING: CAR GUESSING: HAT GUESSING: CAR SUCCESS! Consensus on: CAR Input:
    61. 61. Games for Audio Tagging: Interactively Building an Online Database <ul><li>Three games under development: </li></ul><ul><ul><li>Game like ESP game but for audio (PHP and Flash front end with MySQL backend) </li></ul></ul><ul><ul><li>Audio-visual game in which users select image to audio </li></ul></ul><ul><ul><li>Audio-visual based game where users select appropriate audio content for visual image </li></ul></ul>3.3 Adaptability
    62. 62. Adapting the Algorithms For MIR <ul><li>MIR = Music Information Retrieval </li></ul><ul><li>Retrieval based on bpm, harmonic content, melodic intervals, timbre, etc. </li></ul><ul><li>How can we use MIR techniques to make better game audio? </li></ul><ul><ul><li>User-generated playlists + new algorithms = </li></ul></ul><ul><ul><li>appropriate and new user-generated audio content </li></ul></ul>3.3 Adaptability
    63. 63. 4. Summary/Conclusions
    64. 64. Unified Architecture Routing, allocation and scheduling (includes system clocks) Input: Game Data Parameters Detection (Beat tracking, phrase matching, pitch matching, harmony and key matching). Prediction (Neural nets, fuzzy logic.) Wave banks Audio Data (MIDI) Algorithmic composition/modelling Samplers, synths, tone generators Intelligent mixing engine AI Audio Engine
    65. 65. Why CS needs Arts (and vice versa)
    66. 66. Thank-you to … Further information: [email_address] (my web site)