Your SlideShare is downloading. ×
0
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Voice That Matter 2010 - Core Audio
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Voice That Matter 2010 - Core Audio

249

Published on

Audio for iPhone Games Session

Audio for iPhone Games Session

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
249
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Audio and OpenAL for iPhone GamesKevin AvilaRegistered HEX Offender
  • 2. Introduction• About Me• Core Audio• iPhone Services• OpenAL• Tips & Tricks
  • 3. About Meemail: eddienull@me.com twitter: eddienull
  • 4. About Meemail: eddienull@me.com twitter: eddienull
  • 5. About Meemail: eddienull@me.com twitter: eddienull
  • 6. About Meemail: eddienull@me.com twitter: eddienull
  • 7. About Meemail: eddienull@me.com twitter: eddienull
  • 8. Core Audio
  • 9. Why?
  • 10. Core AudioWhy? "Easy" and "CoreAudio" cant be used in the same sentence. CoreAudio is very powerful, very complex, and under-documented. Be prepared for a steep learning curve, APIs with millions of tiny little pieces, and puzzling things out from sample code rather than reading high-level documentation. –Jens Alfke, coreaudio-api list Feb 9, 2009
  • 11. Core AudioWhy?• Problem domain is hard• Performance is hard• Low latency is hard• Reusability is hard
  • 12. Core AudioWhy?• Doing without would suck• Slowness would suck• Latency would suck• Non-reusability would suck
  • 13. Theory
  • 14. How it Works +1.0f +10 Intensity Pressure Time -1.0f -10
  • 15. Core AudioIntroduction• Overview of Core Audio• Terminology• Fundamental Concepts • ASBD • Properties • Fundamental API • AudioFormat • AudioConverter • AudioFile
  • 16. Terminology
  • 17. Core AudioTerminology• Sample—a data point for one channel +1.0f Intensity Time -1.0f
  • 18. Core AudioTerminology• Frame— The number of samples presented at one time • 1 for mono • 2 for stereo • 4 for quad
  • 19. Core AudioTerminology• Packet—a collection of Frames
  • 20. The Basics
  • 21. Core Audio OverviewThe Canonical Format• AudioSampleType • Used for I/O • 32-bit float (Mac) • 16-bit integer (iPhone)• AudioUnitSampleType • Used for DSP • 32-bit float (Mac) • 8.24 fixed (iPhone)
  • 22. Core Audio OverviewAudioStreamBasicDescription - “ASBD”• Contains the minimal information needed to describe audio data• Some formats may not use all of the fields • Unused fields need to be set to zero struct AudioStreamBasicDescription { Float64 mSampleRate; UInt32 mFormatID; UInt32 mFormatFlags; UInt32 mBytesPerPacket; UInt32 mFramesPerPacket; UInt32 mBytesPerFrame; UInt32 mChannelsPerFrame; UInt32 mBitsPerChannel; UInt32 mReserved; };
  • 23. Core Audio OverviewProperties• Key/Value pair used to describe/manipulate API attributes.• Scope and element selectors are used by some API to further qualify a property• The value of a property can be of whatever type the API needs.• APIs share several common functions • GetProperty, SetProperty, and GetPropertyInfo • AddPropertyListener and RemovePropertyListener
  • 24. Core Audio OverviewScopes and Elements• An element is the same as a bus input scope element 0 global scope output scope output scope element 0 element 1
  • 25. Core Audio OverviewAudioFormat• Provides information about installed codecs• Fills out ASBDs based on Format ID• Provides more information about a formatʼs parameters
  • 26. Core Audio OverviewAudioFormat• ASBDs can be complicated, let the system do the work for you! asbd.mSampleRate = 44100.0; asbd.mFormatID = kAudioFormatMPEG4AAC; asbd.mChannelsPerFrame = 2; UInt32 propSize = sizeof(AudioStreamBasicDescription); AudioFormatGetProperty(kAudioFormatProperty_FormatInfo, 0, NULL, ! ! ! ! ! &propSize, &asbd);
  • 27. Core Audio OverviewAudioConverter• Converts • bit depths • sample rate • interleaving & deinterleaving • channel ordering • PCM <-> compressed/encoded • Can use all installed codecs
  • 28. Core Audio OverviewAudioFile• Parses a file and provides access to the raw data • Uses properties to query information about the file• ExtendedAudioFile • High-level access to an audio file • Combines AudioFile + AudioConverter
  • 29. For Example...
  • 30. Core Audio OverviewSimple File Reading // Open the audio file1 ExtAudioFileOpenURL(fileURL, &inputFile);2 // Get the file’s audio data format AudioStreamBasicDescription inputFileFormat; UInt32 propSize = sizeof(AudioStreamBasicDescription); ExtAudioFileGetProperty(inputFile, kExtAudioFileProperty_FileDataFormat, &propSize, &inputFileFormat); // configure the output audio format to native canonical format3 ! AudioStreamBasicDescription outputFormat = {0}; outputFormat.mSampleRate = inputFileFormat.mSampleRate; ! outputFormat.mFormatID = kAudioFormatLinearPCM; ! outputFormat.mFormatFlags = kAudioFormatFlagsCanonical; ! outputFormat.mChannelsPerFrame = inputFileFormat.mChannelsPerFrame; ! outputFormat.mBitsPerChannel = 16; UInt32 propSize = sizeof(AudioStreamBasicDescription); AudioFormatGetProperty(kAudioFormatProperty_FormatInfo, 0, NULL, &propSize, &outputFormat);
  • 31. Core Audio OverviewSimple File Reading // Set the desired decode data format4 ExtAudioFileSetProperty(inputFile, kExtAudioFileProperty_ClientDataFormat, sizeof(outputFormat), &outputFormat);5 // Get the total frame count SInt64 inputFileLengthInFrames; UInt32 propSize = sizeof(SInt64); ExtAudioFileGetProperty(inputFile, kExtAudioFileProperty_FileLengthFrames, &propSize, &inputFileLengthInFrames); // Read all the data into memory6 UInt32 dataSize = (inputFileLengthInFrames * outputFormat.mBytesPerFrame); void *theData = malloc(dataSize); AudioBufferList dataBuffer; ! dataBuffer.mNumberBuffers = 1; ! dataBuffer.mBuffers[0].mDataByteSize = dataSize; ! dataBuffer.mBuffers[0].mNumberChannels = outputFormat.mChannelsPerFrame; ! dataBuffer.mBuffers[0].mData = theData; ExtAudioFileRead(inputFile, (UInt32*)&theFileLengthInFrames, &dataBuffer);
  • 32. iPhone Services
  • 33. iPhone Services Overview• Audio Sessions • Categories • Interruptions • Routes• Hardware Acceleration
  • 34. Audio SessionFundamental Concepts• Describes an applicationʼs interaction with the audio system• Represents the current state of audio on the device • Current Settings • Preferences• State Transitions • Interruptions • Route Changes
  • 35. Audio SessionSettings & Preferences• Session Settings • Influences all audio activity • Except UI sound effects • Current Session Characteristics
  • 36. Audio SessionCategories • Identify a set of audio features for your application • Mixable with others • Have input or output • Silence on Screen Lock or Ringer Switch
  • 37. Audio SessionBasic Setup // Get the session instance1 AVAudioSession *mySession = [AVAudioSession sharedInstance]; // Implement delegates to handle notifications2 mySession.delegate = self; // Establish appropriate category3 [mySession setCategory:AVAudioSessionCategoryAmbient error:nil]; // Activate the session4 [mySession setActive:YES error:nil];
  • 38. Audio SessionInterruptions • System forces session to ʻinactiveʼ • Unable to resume until interrupt task is complete • AVAudioSession delegates • -(void) beginInterruption • Update UI to reflect non-active audio state • -(void) endInterruption • Resume audio, update UI
  • 39. Audio SessionDefining Interruption Delegates-(void) beginInterruption{ if (isPlaying) { wasInterrupted = YES; isPlaying = NO; }}-(void) endInterruption{ if (wasInterrupted) { [[AVAudioSession sharedInstance] setActive:YES error:nil]; [self startSound];! ! ! wasInterrupted = NO; }}
  • 40. • The pathway for audio signals • Where is audio output to? • Where is audio input from? • “Last in Wins” rule • Notification when route changes • Reason why the route changedAudioSessionAddPropertyListener(kAudioSessionProperty_AudioRouteChange, audioRouteChangeListenerCallback, self);
  • 41. Audio SessionDefining a Property Listener Callbackvoid audioRouteChangeListenerCallback ( void *inUserData, AudioSessionPropertyID inPropertyID UInt32 inPropertyValueSize, const void *inPropertyValue){ MyAudioController *controller = (MyAudioController *)inUserData; if (inPropertyID != kAudioSessionProperty_AudioRouteChange) return; if(controller.isPlaying != NO) {!! NSDictionary *routeChangeDictionary = (NSDictionary*)inPropertyValue;!! SInt32 routeChangeReason = [[routeChangeDictionary objectWithKey: !! ! CFSTR (kAudioSession_AudioRouteChangeKey_Reason)] intValue]; if (routeChangeReason == kAudioSessionRouteChangeReason_OldDeviceUnavailable) {!! ! [controller pause]; } }}
  • 42. Audio SessionHardware Accelerated Codecs• Low CPU cost, low power consumption• Supported HW decoders: • AAC / HE-AAC • Apple Lossless • MP3 • IMA4 (IMA/ADPCM)• Supported HW encoders: • AAC (3GS)
  • 43. Audio SessionEnabling Hardware Acceleration • Must set “Mix With Others” to false • Overrides not persistent across category changes// Override our current categories ‘mix with others’ attributeUInt32 value = 1;AudioSessionSetProperty(kAudioSessionProperty_OverrideCategoryMixWithOthers, sizeof(value), &value);
  • 44. OpenAL
  • 45. OpenAL• Powerful API for 3D audio mixing • Designed for games • Cross-platform, used everywhere!• Models audio in 3D space, as heard by a single listener• Designed as a compliment to OpenGL • Mimics OpenGL conventions • Uses the same coordinate system • Implemented using Core Audioʼs 3D Mixer AU
  • 46. OpenALFundamental Concepts• Context • The spatial environment • Contains the listener• Source • 3D point emitting audio • Many attributes to control rendering• Buffer • Container for audio data • alBufferData() - copies data to internal buffers • alBufferDataStatic() - application owns memory
  • 47. OpenALArchitecture Buffer Buffer Buffer Buffer n Source Source Source n Listener OpenAL Context OpenAL Device iPhone Hardware
  • 48. OpenALListener• Only 1 per context• Positionable just like Sources• Represents the userʼs experience in the 3D environment• Orientation described by two Vectors: • AT = Direction the Listener is facing • UP = Direction pointing up from the top of the Listenerʼs head // Orient the Listener facing +Z ALfloat listenerOrientation[6] = {!0.0, 0.0, 1.0, // AT !! 0.0, 1.0, 0.0} // UP alListenerfv(AL_ORIENTATION, listenerOrientation);
  • 49. OpenALPositioning• Applies to Listener• Applies to Sources (Mono-only)• Cartesian coordinate system ALfloat listenerPosition[] = {!0.0, 2.0, 0.0} alListenerfv(AL_POSITION, listenerPosition); ALfloat sourcePosition[] = {!0.0, 25.0, 0.0} alSourcefv(AL_POSITION, sourcePosition);
  • 50. OpenALCartesian Coordinates x:0, y:0, z:+1 = Listener facing the Positive Z — z— + + x
  • 51. OpenALCartesian Coordinates x:0, y:0, z:-1 = Listener facing Negative Z — z— + + x
  • 52. OpenALBasic Setup // open an OpenAL Device1 oalDevice = alcOpenDevice(NULL); // Create a new OpenAL Context (and listener)2 oalContext = alcCreateContext(oalDevice, NULL); // Set our new context to be the current OpenAL Context3 alcMakeContextCurrent(oalContext);
  • 53. OpenALCreating Buffers and Sources // Create an OpenAL buffer to hold our audio data4 alGenBuffers(1, &oalBuffer); // Fill the OpenAL buffer with data alBufferDataStatic(oalBuffer, AL_FORMAT_MONO16, audioData, audioDataSize, 44100); // Create an OpenAL Source object5 alGenSources(1, &oalSource); // Attach the OpenAL Buffer to the OpenAL Source alSourcei(oalSource, AL_BUFFER, oalBuffer);
  • 54. Distance Attenuation
  • 55. OpenALAttenuation by Distance• Describes the reduction in volume based on the distance to the listener.• Set distance model on the context with alDistanceModel() AL_INVERSE_DISTANCE AL_INVERSE_DISTANCE_CLAMPED AL_NONE• Configure Source attributes AL_REFERENCE_DISTANCE AL_MAX_DISTANCE AL_ROLLOFF_FACTOR AL_SOURCE_RELATIVE
  • 56. OpenAL Attenuation by Distance // Set the distance model to be used alDistanceModel(AL_INVERSE_DISTANCE_CLAMPED); +20dB 0 -20 0 Distance from Listener 20
  • 57. OpenAL Attenuation by Distance // Set the Source’s Reference Distance alSourcef(mySource, AL_REFERENCE_DISTANCE, 2.0); reference distance +20dB 0 -20 0 Distance from Listener 20
  • 58. OpenAL Attenuation by Distance // Set the Maximum Distance alSourcef(mySource, AL_MAX_DISTANCE, 30.0); reference maximum distance distance +20dB 0 -20 0 Distance from Listener 50
  • 59. OpenAL Attenuation by Distance // Set the Rolloff Factor alSourcef(mySource, AL_ROLLOFF_FACTOR, 2.0); reference maximum distance distance +20dB 0 -20 0 Distance from Listener 50
  • 60. The Doppler Effect
  • 61. OpenALThe Doppler Effect• No Motion = No Doppler • Doppler only describes the warping of sound due to motion
  • 62. OpenALThe Doppler Effect
  • 63. OpenALThe Doppler Effect• The default value is 0.0 (disabled) • enabling has small CPU cost• The normal value is 1.0 alDopplerFactor(1.0);
  • 64. OpenALThe Doppler Effect• Describes the speed of sound in your universe (per second) // 1000 units per second alDopplerVelocity(1000);
  • 65. OpenALPutting it all together- (void)initOpenAL{ ! ALenum!! ! error;! ! device = alcOpenDevice(NULL); ! if (device != NULL) ! { ! ! context = alcCreateContext(device, 0); ! ! if (context != NULL) ! ! { ! ! ! alcMakeContextCurrent(context);!! ! ALfloat listenerPosition[] = {! 0.0, 2.0, 0.0} alListenerfv(AL_POSITION, listenerPosition); ! ! ! alGenBuffers(1, &buffer); ! ! ! if((error = alGetError()) != AL_NO_ERROR) ! ! ! ! exit(1);!! ! ! ! ! alGenSources(1, &source); ! ! ! if(alGetError() != AL_NO_ERROR) ! ! ! ! exit(1); ! ! } ! }!![self initBuffer];!![self initSource];}
  • 66. OpenALPutting it all together- (void) initBuffer{ ! ALenum error = AL_NO_ERROR; ! ALenum format; ! ALsizei size; ! ALsizei freq;! ! data = MyGetOpenALAudioData(fileURL, &size, &freq);! ! alBufferDataStatic(buffer, AL_FORMAT_MONO16, data, size, freq);! ! if((error = alGetError()) != AL_NO_ERROR) ! { ! ! NSLog(@"error attaching audio to buffer: %xn", error); ! }! !}
  • 67. OpenALPutting it all together- (void) initSource{ ! ALenum error = AL_NO_ERROR; ! alSourcei(source, AL_BUFFER, buffer); ! alSourcei(source, AL_LOOPING, AL_TRUE);! ! float sourcePosAL[] = {sourcePos.x, kDefaultDistance, sourcePos.y}; ! alSourcefv(source, AL_POSITION, sourcePosAL);! ! alSourcef(source, AL_REFERENCE_DISTANCE, 5.0f);!! ! if((error = alGetError()) != AL_NO_ERROR) ! { ! ! NSLog(@"Error attaching buffer to source: %xn", error); ! }!}
  • 68. OpenALPutting it all together• Youʼre now ready to go! alSourcePlay(source); if((error = alGetError()) != AL_NO_ERROR) { ! NSLog(@"error starting source: %xn", error); }
  • 69. Tips & Tricks
  • 70. Sample Rates• High-Quality laser *pew pew!* and *beeps* unnecessary • Example: ʻafconvert -f caff -d LEI16@8000ʼ• The more SRCs the less performance you get
  • 71. What Next?
  • 72. Coming soon eventually
  • 73. The End

×