Audio and OpenAL for      iPhone GamesKevin AvilaRegistered HEX Offender
Introduction• About Me• Core Audio• iPhone Services• OpenAL• Tips & Tricks
About	 Meemail: eddienull@me.com   twitter: eddienull
About	 Meemail: eddienull@me.com   twitter: eddienull
About	 Meemail: eddienull@me.com   twitter: eddienull
About	 Meemail: eddienull@me.com   twitter: eddienull
About	 Meemail: eddienull@me.com   twitter: eddienull
Core Audio
Why?
Core AudioWhy?   "Easy" and "CoreAudio" cant be used in the   same sentence. CoreAudio is very powerful,     very complex,...
Core AudioWhy?• Problem domain is hard• Performance is hard• Low latency is hard• Reusability is hard
Core AudioWhy?• Doing without would suck• Slowness would suck• Latency would suck• Non-reusability would suck
Theory
How it Works     +1.0f      +10      Intensity      Pressure                  Time     -1.0f      -10
Core AudioIntroduction• Overview of Core Audio• Terminology• Fundamental Concepts    • ASBD    • Properties • Fundamental ...
Terminology
Core AudioTerminology• Sample—a data point for one channel    +1.0f     Intensity                                        T...
Core AudioTerminology• Frame— The number of samples presented at one time    • 1 for mono    • 2 for stereo    • 4 for quad
Core AudioTerminology• Packet—a collection of Frames
The Basics
Core Audio OverviewThe Canonical Format• AudioSampleType    • Used for I/O    • 32-bit float (Mac)    • 16-bit integer (iPh...
Core Audio OverviewAudioStreamBasicDescription - “ASBD”• Contains the minimal information needed to describe audio data• S...
Core Audio OverviewProperties• Key/Value pair used to describe/manipulate API attributes.• Scope and element selectors are...
Core Audio OverviewScopes and Elements• An element is the same as a bus                               input scope         ...
Core Audio OverviewAudioFormat• Provides information about installed codecs• Fills out ASBDs based on Format ID• Provides ...
Core Audio OverviewAudioFormat• ASBDs can be complicated, let the system do the work for you!    asbd.mSampleRate         ...
Core Audio OverviewAudioConverter• Converts   • bit depths   • sample rate   • interleaving & deinterleaving   • channel o...
Core Audio OverviewAudioFile• Parses a file and provides access to the raw data    • Uses properties to query information a...
For Example...
Core Audio OverviewSimple File Reading    // Open the audio file1   ExtAudioFileOpenURL(fileURL, &inputFile);2   // Get th...
Core Audio OverviewSimple File Reading    // Set the desired decode data format4   ExtAudioFileSetProperty(inputFile, kExt...
iPhone Services
iPhone Services Overview• Audio Sessions   • Categories   • Interruptions   • Routes• Hardware Acceleration
Audio SessionFundamental Concepts• Describes an applicationʼs interaction with the audio system• Represents the current st...
Audio SessionSettings & Preferences• Session Settings    • Influences all audio activity       • Except UI sound effects   ...
Audio SessionCategories • Identify a set of audio features for your application    • Mixable with others    • Have input o...
Audio SessionBasic Setup    // Get the session instance1   AVAudioSession *mySession = [AVAudioSession sharedInstance];   ...
Audio SessionInterruptions • System forces session to ʻinactiveʼ • Unable to resume until interrupt task is complete • AVA...
Audio SessionDefining Interruption Delegates-(void) beginInterruption{  if (isPlaying)  {     wasInterrupted = YES;     is...
• The pathway for audio signals      • Where is audio output to?      • Where is audio input from?   • “Last in Wins” rule...
Audio SessionDefining a Property Listener Callbackvoid audioRouteChangeListenerCallback (    void *inUserData, AudioSessio...
Audio SessionHardware Accelerated Codecs• Low CPU cost, low power consumption• Supported HW decoders:   • AAC / HE-AAC   •...
Audio SessionEnabling Hardware Acceleration • Must set “Mix With Others” to false • Overrides not persistent across catego...
OpenAL
OpenAL• Powerful API for 3D audio mixing    • Designed for games    • Cross-platform, used everywhere!• Models audio in 3D...
OpenALFundamental Concepts• Context    • The spatial environment    • Contains the listener• Source    • 3D point emitting...
OpenALArchitecture   Buffer      Buffer      Buffer      Buffer       n    Source       Source       Source            n  ...
OpenALListener• Only 1 per context• Positionable just like Sources• Represents the userʼs experience in the 3D environment...
OpenALPositioning• Applies to Listener• Applies to Sources (Mono-only)• Cartesian coordinate system          ALfloat liste...
OpenALCartesian Coordinates      x:0, y:0, z:+1 = Listener facing the Positive Z                          —             z—...
OpenALCartesian Coordinates      x:0, y:0, z:-1 = Listener facing Negative Z                          —             z—    ...
OpenALBasic Setup    // open an OpenAL Device1   oalDevice = alcOpenDevice(NULL);    // Create a new OpenAL Context (and l...
OpenALCreating Buffers and Sources    // Create an OpenAL buffer to hold our audio data4   alGenBuffers(1, &oalBuffer);   ...
Distance Attenuation
OpenALAttenuation by Distance• Describes the reduction in volume based on the distance    to the listener.•   Set distance...
OpenAL Attenuation by Distance     // Set the distance model to be used     alDistanceModel(AL_INVERSE_DISTANCE_CLAMPED); ...
OpenAL Attenuation by Distance     // Set the Source’s Reference Distance     alSourcef(mySource, AL_REFERENCE_DISTANCE, 2...
OpenAL Attenuation by Distance     // Set the Maximum Distance     alSourcef(mySource, AL_MAX_DISTANCE, 30.0);           r...
OpenAL Attenuation by Distance     // Set the Rolloff Factor     alSourcef(mySource, AL_ROLLOFF_FACTOR, 2.0);           re...
The Doppler Effect
OpenALThe Doppler Effect• No Motion = No Doppler    • Doppler only describes the warping of sound due to motion
OpenALThe Doppler Effect
OpenALThe Doppler Effect• The default value is 0.0 (disabled)    • enabling has small CPU cost• The normal value is 1.0  a...
OpenALThe Doppler Effect• Describes the speed of sound in your universe (per second)  // 1000 units per second  alDopplerV...
OpenALPutting it all together- (void)initOpenAL{  !    ALenum!!      !      error;!  !    device = alcOpenDevice(NULL);  !...
OpenALPutting it all together- (void) initBuffer{  !    ALenum error = AL_NO_ERROR;  !    ALenum format;  !    ALsizei siz...
OpenALPutting it all together- (void) initSource{  !    ALenum error = AL_NO_ERROR;    !       alSourcei(source, AL_BUFFER...
OpenALPutting it all together• Youʼre now ready to go!         alSourcePlay(source);         if((error = alGetError()) != ...
Tips & Tricks
Sample Rates• High-Quality laser *pew pew!* and *beeps* unnecessary   • Example: ʻafconvert -f caff -d LEI16@8000ʼ• The mo...
What Next?
Coming soon eventually
The End
Voice That Matter 2010 - Core Audio
Upcoming SlideShare
Loading in …5
×

Voice That Matter 2010 - Core Audio

490 views
351 views

Published on

Audio for iPhone Games Session

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
490
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Voice That Matter 2010 - Core Audio

  1. 1. Audio and OpenAL for iPhone GamesKevin AvilaRegistered HEX Offender
  2. 2. Introduction• About Me• Core Audio• iPhone Services• OpenAL• Tips & Tricks
  3. 3. About Meemail: eddienull@me.com twitter: eddienull
  4. 4. About Meemail: eddienull@me.com twitter: eddienull
  5. 5. About Meemail: eddienull@me.com twitter: eddienull
  6. 6. About Meemail: eddienull@me.com twitter: eddienull
  7. 7. About Meemail: eddienull@me.com twitter: eddienull
  8. 8. Core Audio
  9. 9. Why?
  10. 10. Core AudioWhy? "Easy" and "CoreAudio" cant be used in the same sentence. CoreAudio is very powerful, very complex, and under-documented. Be prepared for a steep learning curve, APIs with millions of tiny little pieces, and puzzling things out from sample code rather than reading high-level documentation. –Jens Alfke, coreaudio-api list Feb 9, 2009
  11. 11. Core AudioWhy?• Problem domain is hard• Performance is hard• Low latency is hard• Reusability is hard
  12. 12. Core AudioWhy?• Doing without would suck• Slowness would suck• Latency would suck• Non-reusability would suck
  13. 13. Theory
  14. 14. How it Works +1.0f +10 Intensity Pressure Time -1.0f -10
  15. 15. Core AudioIntroduction• Overview of Core Audio• Terminology• Fundamental Concepts • ASBD • Properties • Fundamental API • AudioFormat • AudioConverter • AudioFile
  16. 16. Terminology
  17. 17. Core AudioTerminology• Sample—a data point for one channel +1.0f Intensity Time -1.0f
  18. 18. Core AudioTerminology• Frame— The number of samples presented at one time • 1 for mono • 2 for stereo • 4 for quad
  19. 19. Core AudioTerminology• Packet—a collection of Frames
  20. 20. The Basics
  21. 21. Core Audio OverviewThe Canonical Format• AudioSampleType • Used for I/O • 32-bit float (Mac) • 16-bit integer (iPhone)• AudioUnitSampleType • Used for DSP • 32-bit float (Mac) • 8.24 fixed (iPhone)
  22. 22. Core Audio OverviewAudioStreamBasicDescription - “ASBD”• Contains the minimal information needed to describe audio data• Some formats may not use all of the fields • Unused fields need to be set to zero struct AudioStreamBasicDescription { Float64 mSampleRate; UInt32 mFormatID; UInt32 mFormatFlags; UInt32 mBytesPerPacket; UInt32 mFramesPerPacket; UInt32 mBytesPerFrame; UInt32 mChannelsPerFrame; UInt32 mBitsPerChannel; UInt32 mReserved; };
  23. 23. Core Audio OverviewProperties• Key/Value pair used to describe/manipulate API attributes.• Scope and element selectors are used by some API to further qualify a property• The value of a property can be of whatever type the API needs.• APIs share several common functions • GetProperty, SetProperty, and GetPropertyInfo • AddPropertyListener and RemovePropertyListener
  24. 24. Core Audio OverviewScopes and Elements• An element is the same as a bus input scope element 0 global scope output scope output scope element 0 element 1
  25. 25. Core Audio OverviewAudioFormat• Provides information about installed codecs• Fills out ASBDs based on Format ID• Provides more information about a formatʼs parameters
  26. 26. Core Audio OverviewAudioFormat• ASBDs can be complicated, let the system do the work for you! asbd.mSampleRate = 44100.0; asbd.mFormatID = kAudioFormatMPEG4AAC; asbd.mChannelsPerFrame = 2; UInt32 propSize = sizeof(AudioStreamBasicDescription); AudioFormatGetProperty(kAudioFormatProperty_FormatInfo, 0, NULL, ! ! ! ! ! &propSize, &asbd);
  27. 27. Core Audio OverviewAudioConverter• Converts • bit depths • sample rate • interleaving & deinterleaving • channel ordering • PCM <-> compressed/encoded • Can use all installed codecs
  28. 28. Core Audio OverviewAudioFile• Parses a file and provides access to the raw data • Uses properties to query information about the file• ExtendedAudioFile • High-level access to an audio file • Combines AudioFile + AudioConverter
  29. 29. For Example...
  30. 30. Core Audio OverviewSimple File Reading // Open the audio file1 ExtAudioFileOpenURL(fileURL, &inputFile);2 // Get the file’s audio data format AudioStreamBasicDescription inputFileFormat; UInt32 propSize = sizeof(AudioStreamBasicDescription); ExtAudioFileGetProperty(inputFile, kExtAudioFileProperty_FileDataFormat, &propSize, &inputFileFormat); // configure the output audio format to native canonical format3 ! AudioStreamBasicDescription outputFormat = {0}; outputFormat.mSampleRate = inputFileFormat.mSampleRate; ! outputFormat.mFormatID = kAudioFormatLinearPCM; ! outputFormat.mFormatFlags = kAudioFormatFlagsCanonical; ! outputFormat.mChannelsPerFrame = inputFileFormat.mChannelsPerFrame; ! outputFormat.mBitsPerChannel = 16; UInt32 propSize = sizeof(AudioStreamBasicDescription); AudioFormatGetProperty(kAudioFormatProperty_FormatInfo, 0, NULL, &propSize, &outputFormat);
  31. 31. Core Audio OverviewSimple File Reading // Set the desired decode data format4 ExtAudioFileSetProperty(inputFile, kExtAudioFileProperty_ClientDataFormat, sizeof(outputFormat), &outputFormat);5 // Get the total frame count SInt64 inputFileLengthInFrames; UInt32 propSize = sizeof(SInt64); ExtAudioFileGetProperty(inputFile, kExtAudioFileProperty_FileLengthFrames, &propSize, &inputFileLengthInFrames); // Read all the data into memory6 UInt32 dataSize = (inputFileLengthInFrames * outputFormat.mBytesPerFrame); void *theData = malloc(dataSize); AudioBufferList dataBuffer; ! dataBuffer.mNumberBuffers = 1; ! dataBuffer.mBuffers[0].mDataByteSize = dataSize; ! dataBuffer.mBuffers[0].mNumberChannels = outputFormat.mChannelsPerFrame; ! dataBuffer.mBuffers[0].mData = theData; ExtAudioFileRead(inputFile, (UInt32*)&theFileLengthInFrames, &dataBuffer);
  32. 32. iPhone Services
  33. 33. iPhone Services Overview• Audio Sessions • Categories • Interruptions • Routes• Hardware Acceleration
  34. 34. Audio SessionFundamental Concepts• Describes an applicationʼs interaction with the audio system• Represents the current state of audio on the device • Current Settings • Preferences• State Transitions • Interruptions • Route Changes
  35. 35. Audio SessionSettings & Preferences• Session Settings • Influences all audio activity • Except UI sound effects • Current Session Characteristics
  36. 36. Audio SessionCategories • Identify a set of audio features for your application • Mixable with others • Have input or output • Silence on Screen Lock or Ringer Switch
  37. 37. Audio SessionBasic Setup // Get the session instance1 AVAudioSession *mySession = [AVAudioSession sharedInstance]; // Implement delegates to handle notifications2 mySession.delegate = self; // Establish appropriate category3 [mySession setCategory:AVAudioSessionCategoryAmbient error:nil]; // Activate the session4 [mySession setActive:YES error:nil];
  38. 38. Audio SessionInterruptions • System forces session to ʻinactiveʼ • Unable to resume until interrupt task is complete • AVAudioSession delegates • -(void) beginInterruption • Update UI to reflect non-active audio state • -(void) endInterruption • Resume audio, update UI
  39. 39. Audio SessionDefining Interruption Delegates-(void) beginInterruption{ if (isPlaying) { wasInterrupted = YES; isPlaying = NO; }}-(void) endInterruption{ if (wasInterrupted) { [[AVAudioSession sharedInstance] setActive:YES error:nil]; [self startSound];! ! ! wasInterrupted = NO; }}
  40. 40. • The pathway for audio signals • Where is audio output to? • Where is audio input from? • “Last in Wins” rule • Notification when route changes • Reason why the route changedAudioSessionAddPropertyListener(kAudioSessionProperty_AudioRouteChange, audioRouteChangeListenerCallback, self);
  41. 41. Audio SessionDefining a Property Listener Callbackvoid audioRouteChangeListenerCallback ( void *inUserData, AudioSessionPropertyID inPropertyID UInt32 inPropertyValueSize, const void *inPropertyValue){ MyAudioController *controller = (MyAudioController *)inUserData; if (inPropertyID != kAudioSessionProperty_AudioRouteChange) return; if(controller.isPlaying != NO) {!! NSDictionary *routeChangeDictionary = (NSDictionary*)inPropertyValue;!! SInt32 routeChangeReason = [[routeChangeDictionary objectWithKey: !! ! CFSTR (kAudioSession_AudioRouteChangeKey_Reason)] intValue]; if (routeChangeReason == kAudioSessionRouteChangeReason_OldDeviceUnavailable) {!! ! [controller pause]; } }}
  42. 42. Audio SessionHardware Accelerated Codecs• Low CPU cost, low power consumption• Supported HW decoders: • AAC / HE-AAC • Apple Lossless • MP3 • IMA4 (IMA/ADPCM)• Supported HW encoders: • AAC (3GS)
  43. 43. Audio SessionEnabling Hardware Acceleration • Must set “Mix With Others” to false • Overrides not persistent across category changes// Override our current categories ‘mix with others’ attributeUInt32 value = 1;AudioSessionSetProperty(kAudioSessionProperty_OverrideCategoryMixWithOthers, sizeof(value), &value);
  44. 44. OpenAL
  45. 45. OpenAL• Powerful API for 3D audio mixing • Designed for games • Cross-platform, used everywhere!• Models audio in 3D space, as heard by a single listener• Designed as a compliment to OpenGL • Mimics OpenGL conventions • Uses the same coordinate system • Implemented using Core Audioʼs 3D Mixer AU
  46. 46. OpenALFundamental Concepts• Context • The spatial environment • Contains the listener• Source • 3D point emitting audio • Many attributes to control rendering• Buffer • Container for audio data • alBufferData() - copies data to internal buffers • alBufferDataStatic() - application owns memory
  47. 47. OpenALArchitecture Buffer Buffer Buffer Buffer n Source Source Source n Listener OpenAL Context OpenAL Device iPhone Hardware
  48. 48. OpenALListener• Only 1 per context• Positionable just like Sources• Represents the userʼs experience in the 3D environment• Orientation described by two Vectors: • AT = Direction the Listener is facing • UP = Direction pointing up from the top of the Listenerʼs head // Orient the Listener facing +Z ALfloat listenerOrientation[6] = {!0.0, 0.0, 1.0, // AT !! 0.0, 1.0, 0.0} // UP alListenerfv(AL_ORIENTATION, listenerOrientation);
  49. 49. OpenALPositioning• Applies to Listener• Applies to Sources (Mono-only)• Cartesian coordinate system ALfloat listenerPosition[] = {!0.0, 2.0, 0.0} alListenerfv(AL_POSITION, listenerPosition); ALfloat sourcePosition[] = {!0.0, 25.0, 0.0} alSourcefv(AL_POSITION, sourcePosition);
  50. 50. OpenALCartesian Coordinates x:0, y:0, z:+1 = Listener facing the Positive Z — z— + + x
  51. 51. OpenALCartesian Coordinates x:0, y:0, z:-1 = Listener facing Negative Z — z— + + x
  52. 52. OpenALBasic Setup // open an OpenAL Device1 oalDevice = alcOpenDevice(NULL); // Create a new OpenAL Context (and listener)2 oalContext = alcCreateContext(oalDevice, NULL); // Set our new context to be the current OpenAL Context3 alcMakeContextCurrent(oalContext);
  53. 53. OpenALCreating Buffers and Sources // Create an OpenAL buffer to hold our audio data4 alGenBuffers(1, &oalBuffer); // Fill the OpenAL buffer with data alBufferDataStatic(oalBuffer, AL_FORMAT_MONO16, audioData, audioDataSize, 44100); // Create an OpenAL Source object5 alGenSources(1, &oalSource); // Attach the OpenAL Buffer to the OpenAL Source alSourcei(oalSource, AL_BUFFER, oalBuffer);
  54. 54. Distance Attenuation
  55. 55. OpenALAttenuation by Distance• Describes the reduction in volume based on the distance to the listener.• Set distance model on the context with alDistanceModel() AL_INVERSE_DISTANCE AL_INVERSE_DISTANCE_CLAMPED AL_NONE• Configure Source attributes AL_REFERENCE_DISTANCE AL_MAX_DISTANCE AL_ROLLOFF_FACTOR AL_SOURCE_RELATIVE
  56. 56. OpenAL Attenuation by Distance // Set the distance model to be used alDistanceModel(AL_INVERSE_DISTANCE_CLAMPED); +20dB 0 -20 0 Distance from Listener 20
  57. 57. OpenAL Attenuation by Distance // Set the Source’s Reference Distance alSourcef(mySource, AL_REFERENCE_DISTANCE, 2.0); reference distance +20dB 0 -20 0 Distance from Listener 20
  58. 58. OpenAL Attenuation by Distance // Set the Maximum Distance alSourcef(mySource, AL_MAX_DISTANCE, 30.0); reference maximum distance distance +20dB 0 -20 0 Distance from Listener 50
  59. 59. OpenAL Attenuation by Distance // Set the Rolloff Factor alSourcef(mySource, AL_ROLLOFF_FACTOR, 2.0); reference maximum distance distance +20dB 0 -20 0 Distance from Listener 50
  60. 60. The Doppler Effect
  61. 61. OpenALThe Doppler Effect• No Motion = No Doppler • Doppler only describes the warping of sound due to motion
  62. 62. OpenALThe Doppler Effect
  63. 63. OpenALThe Doppler Effect• The default value is 0.0 (disabled) • enabling has small CPU cost• The normal value is 1.0 alDopplerFactor(1.0);
  64. 64. OpenALThe Doppler Effect• Describes the speed of sound in your universe (per second) // 1000 units per second alDopplerVelocity(1000);
  65. 65. OpenALPutting it all together- (void)initOpenAL{ ! ALenum!! ! error;! ! device = alcOpenDevice(NULL); ! if (device != NULL) ! { ! ! context = alcCreateContext(device, 0); ! ! if (context != NULL) ! ! { ! ! ! alcMakeContextCurrent(context);!! ! ALfloat listenerPosition[] = {! 0.0, 2.0, 0.0} alListenerfv(AL_POSITION, listenerPosition); ! ! ! alGenBuffers(1, &buffer); ! ! ! if((error = alGetError()) != AL_NO_ERROR) ! ! ! ! exit(1);!! ! ! ! ! alGenSources(1, &source); ! ! ! if(alGetError() != AL_NO_ERROR) ! ! ! ! exit(1); ! ! } ! }!![self initBuffer];!![self initSource];}
  66. 66. OpenALPutting it all together- (void) initBuffer{ ! ALenum error = AL_NO_ERROR; ! ALenum format; ! ALsizei size; ! ALsizei freq;! ! data = MyGetOpenALAudioData(fileURL, &size, &freq);! ! alBufferDataStatic(buffer, AL_FORMAT_MONO16, data, size, freq);! ! if((error = alGetError()) != AL_NO_ERROR) ! { ! ! NSLog(@"error attaching audio to buffer: %xn", error); ! }! !}
  67. 67. OpenALPutting it all together- (void) initSource{ ! ALenum error = AL_NO_ERROR; ! alSourcei(source, AL_BUFFER, buffer); ! alSourcei(source, AL_LOOPING, AL_TRUE);! ! float sourcePosAL[] = {sourcePos.x, kDefaultDistance, sourcePos.y}; ! alSourcefv(source, AL_POSITION, sourcePosAL);! ! alSourcef(source, AL_REFERENCE_DISTANCE, 5.0f);!! ! if((error = alGetError()) != AL_NO_ERROR) ! { ! ! NSLog(@"Error attaching buffer to source: %xn", error); ! }!}
  68. 68. OpenALPutting it all together• Youʼre now ready to go! alSourcePlay(source); if((error = alGetError()) != AL_NO_ERROR) { ! NSLog(@"error starting source: %xn", error); }
  69. 69. Tips & Tricks
  70. 70. Sample Rates• High-Quality laser *pew pew!* and *beeps* unnecessary • Example: ʻafconvert -f caff -d LEI16@8000ʼ• The more SRCs the less performance you get
  71. 71. What Next?
  72. 72. Coming soon eventually
  73. 73. The End

×