Audio and OpenAL Guide for iPhone Games

Audio and OpenAL for
iPhone Games

Kevin Avila
Registered HEX Offender

Introduction
• About Me
• Core Audio
• iPhone Services
• OpenAL
• Tips & Tricks

About Me

email: eddienull@me.com
twitter: eddienull

Core Audio
Why?

"Easy" and "CoreAudio" can't be used in the
same sentence. CoreAudio is very powerful,
very complex, and under-documented. Be
prepared for a steep learning curve, APIs with
millions of tiny little pieces, and puzzling things
out from sample code rather than reading
high-level documentation.

–Jens Alfke, coreaudio-api list
Feb 9, 2009

Core Audio
Why?
• Problem domain is hard

• Performance is hard

• Low latency is hard

• Reusability is hard

Core Audio
Why?
• Doing without would suck

• Slowness would suck

• Latency would suck

• Non-reusability would suck

How it Works

+1.0f
+10
Intensity
Pressure

Time

-1.0f
-10

Core Audio
Introduction
• Overview of Core Audio
• Terminology
• Fundamental Concepts
• ASBD
• Properties

• Fundamental API
• AudioFormat
• AudioConverter
• AudioFile

Core Audio
Terminology

• Sample—a data point for one channel

+1.0f
Intensity

Time

-1.0f

Core Audio
Terminology

• Frame— The number of samples presented at one time
• 1 for mono
• 2 for stereo
• 4 for quad

Core Audio
Terminology

• Packet—a collection of Frames

Core Audio Overview
The Canonical Format
• AudioSampleType
• Used for I/O
• 32-bit float (Mac)
• 16-bit integer (iPhone)

• AudioUnitSampleType
• Used for DSP
• 32-bit float (Mac)
• 8.24 fixed (iPhone)

Core Audio Overview
AudioStreamBasicDescription - “ASBD”
• Contains the minimal information needed to describe
audio data

• Some formats may not use all of the ﬁelds
• Unused ﬁelds need to be set to zero
struct AudioStreamBasicDescription {
Float64 mSampleRate;
UInt32 mFormatID;
UInt32 mFormatFlags;
UInt32 mBytesPerPacket;
UInt32 mFramesPerPacket;
UInt32 mBytesPerFrame;
UInt32 mChannelsPerFrame;
UInt32 mBitsPerChannel;
UInt32 mReserved;
};

Core Audio Overview
Properties
• Key/Value pair used to describe/manipulate API attributes.

• Scope and element selectors are used by some API to further
qualify a property

• The value of a property can be of whatever type the API needs.

• APIs share several common functions
• GetProperty, SetProperty, and GetPropertyInfo
• AddPropertyListener and RemovePropertyListener

Core Audio Overview
Scopes and Elements
• An element is the same as a bus

input scope
element 0

global scope

output scope output scope
element 0 element 1

Core Audio Overview
AudioFormat
• Provides information about installed codecs

• Fills out ASBDs based on Format ID

• Provides more information about a formatʼs parameters

Core Audio Overview
AudioFormat
• ASBDs can be complicated, let the system do the work for
you!

asbd.mSampleRate = 44100.0;
asbd.mFormatID = kAudioFormatMPEG4AAC;
asbd.mChannelsPerFrame = 2;

UInt32 propSize = sizeof(AudioStreamBasicDescription);
AudioFormatGetProperty(kAudioFormatProperty_FormatInfo,
0, NULL,
!
! ! ! ! &propSize, &asbd);

Core Audio Overview
AudioConverter
• Converts
• bit depths
• sample rate
• interleaving & deinterleaving
• channel ordering
• PCM <-> compressed/encoded

• Can use all installed codecs

Core Audio Overview
AudioFile
• Parses a file and provides access to the raw data
• Uses properties to query information about the file

• ExtendedAudioFile
• High-level access to an audio file
• Combines AudioFile + AudioConverter

Core Audio Overview
Simple File Reading

// Open the audio file
1 ExtAudioFileOpenURL(fileURL, &inputFile);

2 // Get the file’s audio data format
AudioStreamBasicDescription inputFileFormat;
ExtAudioFileGetProperty(inputFile, kExtAudioFileProperty_FileDataFormat,
&propSize, &inputFileFormat);

// configure the output audio format to native canonical format
3 !
AudioStreamBasicDescription outputFormat = {0};
outputFormat.mSampleRate = inputFileFormat.mSampleRate;
!
outputFormat.mFormatID = kAudioFormatLinearPCM;
!
outputFormat.mFormatFlags = kAudioFormatFlagsCanonical;
!
outputFormat.mChannelsPerFrame = inputFileFormat.mChannelsPerFrame;
!
outputFormat.mBitsPerChannel = 16;

AudioFormatGetProperty(kAudioFormatProperty_FormatInfo,
0, NULL, &propSize, &outputFormat);

Core Audio Overview
Simple File Reading
// Set the desired decode data format
4 ExtAudioFileSetProperty(inputFile, kExtAudioFileProperty_ClientDataFormat,
sizeof(outputFormat), &outputFormat);

5 // Get the total frame count
SInt64 inputFileLengthInFrames;
UInt32 propSize = sizeof(SInt64);
ExtAudioFileGetProperty(inputFile, kExtAudioFileProperty_FileLengthFrames,
&propSize, &inputFileLengthInFrames);

// Read all the data into memory
6 UInt32 dataSize = (inputFileLengthInFrames * outputFormat.mBytesPerFrame);
void *theData = malloc(dataSize);
AudioBufferList dataBuffer;
!
dataBuffer.mNumberBuffers = 1;
!
dataBuffer.mBuffers[0].mDataByteSize = dataSize;
!
dataBuffer.mBuffers[0].mNumberChannels = outputFormat.mChannelsPerFrame;
!
dataBuffer.mBuffers[0].mData = theData;

ExtAudioFileRead(inputFile, (UInt32*)&theFileLengthInFrames, &dataBuffer);

iPhone Services Overview

• Audio Sessions
• Categories
• Interruptions
• Routes

• Hardware Acceleration

Audio Session
Fundamental Concepts
• Describes an applicationʼs interaction with the audio system

• Represents the current state of audio on the device
• Current Settings
• Preferences

• State Transitions
• Interruptions
• Route Changes

Audio Session
Settings & Preferences
• Session Settings
• Inﬂuences all audio activity
• Except UI sound effects
• Current Session Characteristics

Audio Session
Categories
• Identify a set of audio features for your application
• Mixable with others
• Have input or output
• Silence on Screen Lock or Ringer Switch

Audio Session
Basic Setup

// Get the session instance
1 AVAudioSession *mySession = [AVAudioSession sharedInstance];

// Implement delegates to handle notifications
2 mySession.delegate = self;

// Establish appropriate category
3 [mySession setCategory:AVAudioSessionCategoryAmbient error:nil];

// Activate the session
4 [mySession setActive:YES error:nil];

Audio Session
Interruptions
• System forces session to ʻinactiveʼ
• Unable to resume until interrupt task is complete

• AVAudioSession delegates
• -(void) beginInterruption
• Update UI to reﬂect non-active audio state

• -(void) endInterruption
• Resume audio, update UI

Audio Session
Defining Interruption Delegates

-(void) beginInterruption
{
if (isPlaying)
{
wasInterrupted = YES;
isPlaying = NO;
}
}

-(void) endInterruption
{
if (wasInterrupted)
{
[[AVAudioSession sharedInstance] setActive:YES error:nil];
[self startSound];! ! !
wasInterrupted = NO;
}
}

• The pathway for audio signals
• Where is audio output to?
• Where is audio input from?

• “Last in Wins” rule

• Notification when route changes
• Reason why the route changed

AudioSessionAddPropertyListener(kAudioSessionProperty_AudioRouteChange,
audioRouteChangeListenerCallback, self);

Audio Session
Defining a Property Listener Callback

void audioRouteChangeListenerCallback (
void *inUserData, AudioSessionPropertyID inPropertyID
UInt32 inPropertyValueSize, const void *inPropertyValue)
{
MyAudioController *controller = (MyAudioController *)inUserData;

if (inPropertyID != kAudioSessionProperty_AudioRouteChange) return;

if(controller.isPlaying != NO)
{
!
! NSDictionary *routeChangeDictionary = (NSDictionary*)inPropertyValue;
!
! SInt32 routeChangeReason = [[routeChangeDictionary objectWithKey:
!! ! CFSTR (kAudioSession_AudioRouteChangeKey_Reason)] intValue];

if (routeChangeReason == kAudioSessionRouteChangeReason_OldDeviceUnavailable)
{
!
! ! [controller pause];
}
}
}

Audio Session
Hardware Accelerated Codecs
• Low CPU cost, low power consumption

• Supported HW decoders:
• AAC / HE-AAC
• Apple Lossless
• MP3
• IMA4 (IMA/ADPCM)

• Supported HW encoders:
• AAC (3GS)

Audio Session
Enabling Hardware Acceleration
• Must set “Mix With Others” to false

• Overrides not persistent across category changes

// Override our current categories ‘mix with others’ attribute
UInt32 value = 1;
AudioSessionSetProperty(kAudioSessionProperty_OverrideCategoryMixWithOthers,
sizeof(value), &value);

OpenAL

• Powerful API for 3D audio mixing
• Designed for games
• Cross-platform, used everywhere!

• Models audio in 3D space, as heard by a single listener

• Designed as a compliment to OpenGL
• Mimics OpenGL conventions
• Uses the same coordinate system

• Implemented using Core Audioʼs 3D Mixer AU

OpenAL
Fundamental Concepts
• Context
• The spatial environment
• Contains the listener

• Source
• 3D point emitting audio
• Many attributes to control rendering

• Buffer
• Container for audio data
• alBufferData() - copies data to internal buffers
• alBufferDataStatic() - application owns memory

OpenAL
Architecture

Buffer Buffer Buffer Buffer n

Source Source Source n

Listener
OpenAL
Context

OpenAL Device

iPhone Hardware

OpenAL
Listener
• Only 1 per context
• Positionable just like Sources
• Represents the userʼs experience in the 3D environment
• Orientation described by two Vectors:
• AT = Direction the Listener is facing
• UP = Direction pointing up from the top of the Listenerʼs head

// Orient the Listener facing +Z
ALfloat listenerOrientation[6] = {!0.0, 0.0, 1.0, // AT
!! 0.0, 1.0, 0.0} // UP
alListenerfv(AL_ORIENTATION, listenerOrientation);

OpenAL
Positioning
• Applies to Listener
• Applies to Sources (Mono-only)
• Cartesian coordinate system

ALfloat listenerPosition[] = {!0.0, 2.0, 0.0}
alListenerfv(AL_POSITION, listenerPosition);

ALfloat sourcePosition[] = {!0.0, 25.0, 0.0}
alSourcefv(AL_POSITION, sourcePosition);

OpenAL
Cartesian Coordinates

x:0, y:0, z:+1 = Listener facing the Positive Z

—

z— +

+
x

OpenAL
Cartesian Coordinates

x:0, y:0, z:-1 = Listener facing Negative Z

—

z— +

+
x

OpenAL
Basic Setup

// open an OpenAL Device
1 oalDevice = alcOpenDevice(NULL);

// Create a new OpenAL Context (and listener)
2 oalContext = alcCreateContext(oalDevice, NULL);

// Set our new context to be the current OpenAL Context
3 alcMakeContextCurrent(oalContext);

OpenAL
Creating Buffers and Sources

// Create an OpenAL buffer to hold our audio data
4 alGenBuffers(1, &oalBuffer);

// Fill the OpenAL buffer with data
alBufferDataStatic(oalBuffer, AL_FORMAT_MONO16,
audioData, audioDataSize, 44100);

// Create an OpenAL Source object
5 alGenSources(1, &oalSource);

// Attach the OpenAL Buffer to the OpenAL Source
alSourcei(oalSource, AL_BUFFER, oalBuffer);

OpenAL
Attenuation by Distance
• Describes the reduction in volume based on the distance
to the listener.

• Set distance model on the context with alDistanceModel()
AL_INVERSE_DISTANCE
AL_INVERSE_DISTANCE_CLAMPED
AL_NONE

• Conﬁgure Source attributes
AL_REFERENCE_DISTANCE
AL_MAX_DISTANCE
AL_ROLLOFF_FACTOR
AL_SOURCE_RELATIVE

OpenAL

// Set the distance model to be used
alDistanceModel(AL_INVERSE_DISTANCE_CLAMPED);

+20

dB 0

-20

0 Distance from Listener 20

OpenAL

// Set the Source’s Reference Distance
alSourcef(mySource, AL_REFERENCE_DISTANCE, 2.0);

reference
distance

+20

dB 0

-20


OpenAL

// Set the Maximum Distance
alSourcef(mySource, AL_MAX_DISTANCE, 30.0);

reference maximum
distance distance

+20

dB 0

-20


OpenAL

// Set the Rolloff Factor
alSourcef(mySource, AL_ROLLOFF_FACTOR, 2.0);

reference maximum
distance distance

+20

dB 0

-20


OpenAL
The Doppler Effect
• No Motion = No Doppler
• Doppler only describes the warping of sound due to motion

OpenAL
The Doppler Effect
• The default value is 0.0 (disabled)
• enabling has small CPU cost
• The normal value is 1.0

alDopplerFactor(1.0);

OpenAL
The Doppler Effect
• Describes the speed of sound in your universe (per second)

// 1000 units per second
alDopplerVelocity(1000);

OpenAL
Putting it all together
- (void)initOpenAL
{
! ALenum!! ! error;
!
! device = alcOpenDevice(NULL);
! if (device != NULL)
! {
! ! context = alcCreateContext(device, 0);
! ! if (context != NULL)
! ! {
! ! ! alcMakeContextCurrent(context);
!
! !
ALfloat listenerPosition[] = {! 0.0, 2.0, 0.0}
alListenerfv(AL_POSITION, listenerPosition);

! ! ! alGenBuffers(1, &buffer);
! ! ! if((error = alGetError()) != AL_NO_ERROR)
! ! ! ! exit(1);
!
! !
! ! ! alGenSources(1, &source);
! ! ! if(alGetError() != AL_NO_ERROR)
! ! ! ! exit(1);
! ! }
! }
!
!
[self initBuffer];!
!
[self initSource];
}

OpenAL
- (void) initBuffer
{
! ALenum error = AL_NO_ERROR;
! ALenum format;
! ALsizei size;
! ALsizei freq;
!
! data = MyGetOpenALAudioData(fileURL, &size, &freq);
!
! alBufferDataStatic(buffer, AL_FORMAT_MONO16, data, size, freq);
!
! if((error = alGetError()) != AL_NO_ERROR)
! {
! ! NSLog(@"error attaching audio to buffer: %xn", error);
! }! !
}

OpenAL
- (void) initSource
{
! ALenum error = AL_NO_ERROR;

! alSourcei(source, AL_BUFFER, buffer);

! alSourcei(source, AL_LOOPING, AL_TRUE);
!
! float sourcePosAL[] = {sourcePos.x, kDefaultDistance, sourcePos.y};
! alSourcefv(source, AL_POSITION, sourcePosAL);
!
! alSourcef(source, AL_REFERENCE_DISTANCE, 5.0f);
!
!
! if((error = alGetError()) != AL_NO_ERROR)
! {
! ! NSLog(@"Error attaching buffer to source: %xn", error);
! }!
}

OpenAL
• Youʼre now ready to go!

alSourcePlay(source);
if((error = alGetError()) != AL_NO_ERROR)
{
! NSLog(@"error starting source: %xn", error);
}

Sample Rates
• High-Quality laser *pew pew!* and *beeps* unnecessary
• Example: ʻafconvert -f caff -d LEI16@8000ʼ

• The more SRCs the less performance you get

Audio and OpenAL Guide for iPhone Games

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (12)

Similar to Audio and OpenAL Guide for iPhone Games

Similar to Audio and OpenAL Guide for iPhone Games (20)

Audio and OpenAL Guide for iPhone Games