Core Audio: Don't Be Afraid to Play it LOUD! [360iDev, San Jose 2010]

Core Audio: Don’t Be Afraid To Play It LOUD! Chris Adamson 360iDev San Jose 2010 @invalidname

Previously… Robert Strojan — “iPhone Audio Lab” Monday, 12:30

"Easy" and "CoreAudio" can't be used in the same sentence. CoreAudio is very powerful, very complex, and under-documented. Be prepared for a steep learning curve, APIs with millions of tiny little pieces, and puzzling things out from sample code rather than reading high-level documentation. –Jens Alfke, coreaudio-api list Feb 9, 2009

• Problem domain is hard • Performance is hard • Low latency is hard • Reusability is hard

• Doing without would suck • Slowness would suck • Latency would suck • Non-reusability would suck

Last Exit Before Toll • Alternatives to Core Audio: • Media Player: plays audio from iPod library • AV Framework: Obj-C API for ﬁle playback and recording

Are they gone?

What They Missed • Mixing • Effects • Streaming from/to network • Lots of C

Core Audio Programming

CA Style • Procedural C • Work with buffers of samples • Most functions return an OSStatus • You must check this, every time • Heavily callback-oriented • Heavily property-oriented

Example: Audio Session • iPhone-only API • Inspect and set audio hardware and software properties • Declare interaction with other audio processes on the device (e.g., iPod)

#import-ant! #import <AudioToolbox/AudioToolbox.h>

Get Thee Some Docs!

Initialize audio session OSStatus setupAudioSessionErr= ! AudioSessionInitialize ( ! ! ! NULL, // default run loop ! ! ! NULL, // default run loop mode ! ! ! MyInterruptionHandler, // interruption callback ! ! ! self); // client callback data ! NSAssert (setupAudioSessionErr == noErr, @"Couldn't initialize audio session");

Initialize audio session C function call OSStatus setupAudioSessionErr= ! AudioSessionInitialize ( ! ! ! NULL, // default run loop ! ! ! NULL, // default run loop mode ! ! ! MyInterruptionHandler, // interruption callback ! ! ! self); // client callback data ! NSAssert (setupAudioSessionErr == noErr, @"Couldn't initialize audio session");

Initialize audio session OSStatus setupAudioSessionErr= ! AudioSessionInitialize ( C function pointer ! ! ! NULL, // default run loop ! ! ! NULL, // default run loop mode ! ! ! MyInterruptionHandler, // interruption callback ! ! ! self); // client callback data ! NSAssert (setupAudioSessionErr == noErr, @"Couldn't initialize audio session");

Initialize audio session OSStatus setupAudioSessionErr= ! AudioSessionInitialize ( ! ! ! NULL, // default run loop ! ! ! NULL, // default run loop mode ! ! ! MyInterruptionHandler, // interruption callback ! ! ! self); // client callback data ! Obj-C pointer NSAssert (setupAudioSessionErr == noErr, @"Couldn't initialize audio session");

Initialize audio session Return value OSStatus setupAudioSessionErr= ! AudioSessionInitialize ( ! ! ! NULL, // default run loop ! ! ! NULL, // default run loop mode ! ! ! MyInterruptionHandler, // interruption callback ! ! ! self); // client callback data ! NSAssert (setupAudioSessionErr == noErr, @"Couldn't initialize audio session");

Set category property UInt32 sessionCategory = kAudioSessionCategory_PlayAndRecord; setupAudioSessionErr = AudioSessionSetProperty ( kAudioSessionProperty_AudioCategory, sizeof (sessionCategory), &sessionCategory); ! NSAssert (setupAudioSessionErr == noErr, @"Couldn't set audio session property");

Set category property UInt32 sessionCategory = kAudioSessionCategory_PlayAndRecord; Property name setupAudioSessionErr = (constant) AudioSessionSetProperty ( kAudioSessionProperty_AudioCategory, sizeof (sessionCategory), &sessionCategory); ! NSAssert (setupAudioSessionErr == noErr, @"Couldn't set audio session property");

Set category property UInt32 sessionCategory = kAudioSessionCategory_PlayAndRecord; setupAudioSessionErr = AudioSessionSetProperty ( Size of property kAudioSessionProperty_AudioCategory, sizeof (sessionCategory), &sessionCategory); ! NSAssert (setupAudioSessionErr == noErr, @"Couldn't set audio session property");

Set category property UInt32 sessionCategory = kAudioSessionCategory_PlayAndRecord; setupAudioSessionErr = AudioSessionSetProperty ( kAudioSessionProperty_AudioCategory, sizeof (sessionCategory), &sessionCategory); ! Pointer to value NSAssert (setupAudioSessionErr == noErr, @"Couldn't set audio session property");

Is audio input available? UInt32 ui32PropertySize = sizeof (UInt32); UInt32 inputAvailable; setupAudioSessionErr = ! AudioSessionGetProperty( kAudioSessionProperty_AudioInputAvailable, &ui32PropertySize, &inputAvailable); NSAssert (setupAudioSessionErr == noErr, @"Couldn't get current audio input available prop");

Is audio input available? Pointable size UInt32 ui32PropertySize = sizeof (UInt32); UInt32 inputAvailable; setupAudioSessionErr = ! AudioSessionGetProperty( kAudioSessionProperty_AudioInputAvailable, &ui32PropertySize, &inputAvailable); NSAssert (setupAudioSessionErr == noErr, @"Couldn't get current audio input available prop");

Is audio input available? UInt32 ui32PropertySize = sizeof (UInt32); UInt32 inputAvailable; Pointable variable setupAudioSessionErr = ! AudioSessionGetProperty( kAudioSessionProperty_AudioInputAvailable, &ui32PropertySize, &inputAvailable); NSAssert (setupAudioSessionErr == noErr, @"Couldn't get current audio input available prop");

Is audio input available? UInt32 ui32PropertySize = sizeof (UInt32); UInt32 inputAvailable; setupAudioSessionErr = ! AudioSessionGetProperty( kAudioSessionProperty_AudioInputAvailable, &ui32PropertySize, &inputAvailable); Address of size and variable NSAssert (setupAudioSessionErr == noErr, @"Couldn't get current audio input available prop");

Callback on a property setupAudioSessionErr = AudioSessionAddPropertyListener ( ! ! ! kAudioSessionProperty_AudioInputAvailable, ! ! ! MyInputAvailableListener, ! ! ! self); NSAssert (setupAudioSessionErr == noErr, @"Couldn't setup audio input prop listener");

Callback on a property setupAudioSessionErr = AudioSessionAddPropertyListener ( ! ! ! kAudioSessionProperty_AudioInputAvailable, ! ! ! MyInputAvailableListener, ! ! ! self); Property to listen to NSAssert (setupAudioSessionErr == noErr, @"Couldn't setup audio input prop listener");

Callback on a property setupAudioSessionErr = AudioSessionAddPropertyListener ( ! ! ! kAudioSessionProperty_AudioInputAvailable, ! ! ! MyInputAvailableListener, ! ! ! self); C function pointer NSAssert (setupAudioSessionErr == noErr, @"Couldn't setup audio input prop listener");

Callback on a property setupAudioSessionErr = AudioSessionAddPropertyListener ( ! ! ! kAudioSessionProperty_AudioInputAvailable, ! ! ! MyInputAvailableListener, ! ! ! self); “Client data” pointer NSAssert (setupAudioSessionErr == noErr, @"Couldn't setup audio input prop listener");

CA style recap • Lots of getting and setting properties • Always check the return OSStatus • Asychronous callbacks to C functions • Look up function templates in docs • “Context” or “user info” pointer can be an Obj-C object

What’s In Core Audio?

Engines Utilities

Engines Utilities Audio Units

Engines Utilities Open AL Audio Units

Engines Utilities Audio Queue Open AL Audio Units

Engines Utilities AV Fndtn Audio Queue Open AL Audio Units

Engines Utilities AV Fndtn Audio Queue Open AL Audio Units Audio File

Engines Utilities AV Fndtn Audio Queue Open AL Audio Conversion Audio Units Audio File

Engines Utilities AV Fndtn Ext Audio File Audio Queue Open AL Audio Conversion Audio Units Audio File

Engines Utilities Audio File Stream AV Fndtn Ext Audio File Audio Queue Open AL Audio Conversion Audio Units Audio File

Engines Utilities Audio Session Audio File Stream AV Fndtn Ext Audio File Audio Queue Open AL Audio Conversion Audio Units Audio File

iPhone Audio Engines • Media Player • AV Foundation • Audio Queue Services • OpenAL • Audio Unit Services

Media Player • Objective-C class MPMusicPlayerController — Plays music from iPod library • play, pause, stop. Properties for currentPlaybackTime, volume, etc. • No access to decoded samples or ﬁles

AV Framework • Objective-C classes: AVAudioPlayer, AVAudioRecorder, AVAudioSession • Play from / record to ﬁles • Handles compressed formats • play, record, pause, stop. volume, currentTime properties • No access to decoded samples

Audio Queue • C functions for playback, recording • Callback-driven: you ﬁll or process audio buffers periodically • Plays compressed formats • Latency is negotiable

OpenAL • C API intended for gaming, designed to resemble OpenGL • Attach audio to “sources” in 3D space • Listener hears sound from source relative to their own position • Uncompressed (PCM) only • Can stream to a source (push model)

Audio Units • Lowest publicly-accessed part of Core Audio • C API for audio processing • Uncompressed (PCM) only, no ﬂoats • Extremely low latency • OpenAL and Queue implemented atop units

So What’s An Audio Unit?

Elements / Elements / Buses Buses

Input Output Scope Scope

Global Scope

AU “pull” model • Each unit in the chain pulls audio data from some source • Another unit • A callback to your code • Last unit is typically an I/O unit

I/O Unit (output)

Effect I/O Unit Unit (output)

I/O Unit Effect I/O Unit (input) Unit (output)

I/O Unit Effect I/O Unit (input) Unit (output) Caveat: Not entirely accurate, for reasons to be explained soon…

Units on iPhone OS • I/O Units • “Remote” I/O: Abstraction around harware in/out • Voice Processing I/O: Same as RemoteIO, but with echo supression for VOIP • Generic I/O: No hardware access; use for software audio rendering

Units on iPhone OS • Mixer Units • Multichannel Mixer: Mixes multiple input streams into one output stream • 3D Mixer: Mixes inputs with more control over panning, resampling. Used by OpenAL

Units on iPhone OS • Effect units • iPod Equalizer: Provides same features as iPod playback equalizer • Converter units • Converter: Converts between ﬂavors of PCM, but not compressed formats

Missing Units • Unit types absent from iPhone OS • Generators • Music devices (MIDI) and effects • Many units found on Mac absent on iPhone • Filters, reverb, compressor, varispeed, other effects

RemoteIO • Your app’s most direct access to audio input/output hardware • Used at the end of unit chains that play audio • Used at the beginning of chains that capture audio

RemoteIO Buses • Bus 0 is for H/W output • Headphones, speakers, etc. • Bus 1 is for H/W input • Headset mic, phone mic, etc.

Bus 1 Bus 1 Remote I/O Bus 0 Bus 0

But! • There can only be one RemoteIO unit!

Example 1: Play Through

Play Through steps 1. Enable recording via Audio Session 2. Get the Remote I/O unit 3. Set stream format 4. Connect input to output 5. Let ’er rip

1. Set Up Audio Session for Recording • Initialize the audio session • Set a capture-enabling category • Get/set other useful session properties

Initialize Audio Session OSStatus setupAudioSessionErr= AudioSessionInitialize ( ! ! ! NULL, // default run loop ! ! ! NULL, // default run loop mode ! ! ! NULL, // interruption callback ! ! ! NULL); // client callback data NSAssert (setupAudioSessionErr == noErr, @"Couldn't initialize audio session");

Enable Recording UInt32 sessionCategory = kAudioSessionCategory_PlayAndRecord; ! setupAudioSessionErr = AudioSessionSetProperty ! (kAudioSessionProperty_AudioCategory, ! sizeof (sessionCategory), ! &sessionCategory); ! NSAssert (setupAudioSessionErr == noErr, @"Couldn't set audio session property");

Get HW sample rate UInt32 f64PropertySize = sizeof (Float64); setupAudioSessionErr = AudioSessionGetProperty (kAudioSessionProperty_CurrentHardwareSampleRate, ! &f64PropertySize, ! &hardwareSampleRate); ! NSAssert (setupAudioSessionErr == noErr, @"Couldn't get current hardware sample rate"); NSLog (@"current hardware sample rate = %f", hardwareSampleRate);

2. Get the Remote I/O Unit • Describe the unit you want • Search until you ﬁnd it • Enable its input and output properties

Describing a Unit AudioComponentDescription audioCompDesc; audioCompDesc.componentType = kAudioUnitType_Output; audioCompDesc.componentSubType = kAudioUnitSubType_RemoteIO; audioCompDesc.componentManufacturer = kAudioUnitManufacturer_Apple; audioCompDesc.componentFlags = 0; audioCompDesc.componentFlagsMask = 0;

Discovering a Unit AudioUnit remoteIOUnit = NULL; AudioComponent rioComponent = AudioComponentFindNext (NULL, &audioCompDesc); OSErr setupErr = AudioComponentInstanceNew (rioComponent, &remoteIOUnit); NSAssert (setupErr == noErr, @"Couldn't get RIO unit instance"); • If there can be more than one match, iterate over AudioComponentFindNext, passing in last match you found

Enable output property UInt32 oneFlag = 1; AudioUnitElement bus0 = 0; setupErr = ! AudioUnitSetProperty (remoteIOUnit, ! ! ! ! ! ! kAudioOutputUnitProperty_EnableIO, ! ! ! ! ! ! kAudioUnitScope_Output, ! ! ! ! ! ! bus0, ! ! ! ! ! ! &oneFlag, ! ! ! ! ! ! sizeof(oneFlag)); NSAssert (setupErr == noErr, @"Couldn't enable RIO output");

Enable input property AudioUnitElement bus1 = 1; setupErr = AudioUnitSetProperty(remoteIOUnit, ! ! ! kAudioOutputUnitProperty_EnableIO, ! ! ! ! ! kAudioUnitScope_Input, ! ! ! ! ! bus1, ! ! ! ! &oneFlag, ! ! ! ! ! sizeof(oneFlag)); NSAssert (setupErr == noErr, @"couldn't enable RIO input");

3. Set stream properties • Deﬁne an AudioStreamBasicDescription • Set it as the kAudioUnitProperty_StreamFormat • You will screw this up at least once

The ASBD • Description of the common traits of an entire audio stream • Bitrate, channel count, format, format- specific flags, etc. • Not all fields apply to all formats • On iPhone, Audio Units always work with LPCM

Create an ASBD AudioStreamBasicDescription myASBD; memset (&myASBD, 0, sizeof (myASBD)); myASBD.mSampleRate = hardwareSampleRate; myASBD.mFormatID = kAudioFormatLinearPCM; myASBD.mFormatFlags = kAudioFormatFlagsCanonical; myASBD.mBytesPerPacket = 4; myASBD.mFramesPerPacket = 1; myASBD.mBytesPerFrame = myASBD.mBytesPerPacket * myASBD.mFramesPerPacket; myASBD.mChannelsPerFrame = 2; myASBD.mBitsPerChannel = 16;

Set RemoteIO’s stream properties • Declares format you want to read from input bus, and write to output bus • Set the kAudioUnitProperty_StreamFormat property • On RemoteIO’s output scope for bus 1 • On RemoteIO’s input scope for bus 0

Set bus 1 / output- scope stream format setupErr = ! AudioUnitSetProperty (remoteIOUnit, ! ! ! ! ! ! kAudioUnitProperty_StreamFormat, ! ! ! ! ! ! kAudioUnitScope_Output, ! ! ! ! ! ! bus1, ! ! ! ! ! ! &myASBD, ! ! ! ! ! ! sizeof (myASBD)); NSAssert (setupErr == noErr, @"Couldn't set ASBD for RIO on output scope / bus 1");

Set bus 0 / input-scope stream format setupErr = ! AudioUnitSetProperty (remoteIOUnit, ! ! ! ! ! ! kAudioUnitProperty_StreamFormat, ! ! ! ! ! ! kAudioUnitScope_Input, ! ! ! ! ! ! bus0, ! ! ! ! ! ! &myASBD, ! ! ! ! ! ! sizeof (myASBD)); NSAssert (setupErr == noErr, @"Couldn't set ASBD for RIO on input scope / bus 0");

4. Connect Input to Output • Create a structure describing the connection • Set it as a property on the destination audio unit

Declare connection AudioUnitConnection connection; connection.sourceAudioUnit = remoteIOUnit; connection.sourceOutputNumber = bus1; connection.destInputNumber = bus0;

Set connection property setupErr = ! AudioUnitSetProperty(remoteIOUnit, ! ! ! ! ! ! kAudioUnitProperty_MakeConnection, ! ! ! ! ! ! kAudioUnitScope_Input, ! ! ! ! ! ! bus0, ! ! ! ! ! ! &connection, ! ! ! ! ! ! sizeof (connection)); NSAssert (setupErr == noErr, @"Couldn't set RIO connection");

5. Let ’er Rip setupErr =! AudioUnitInitialize(remoteIOUnit); NSAssert (setupErr == noErr, @"Couldn't initialize RIO unit"); // in handleStartTapped: OSStatus startErr = noErr; startErr = AudioOutputUnitStart (remoteIOUnit); NSAssert (startErr == noErr, @"Couldn't start RIO unit");

Recap • We created the Remote I/O unit • Enabled input and output • Created an ASBD to describe stream format and set it on bus 1 output and bus 0 input • Connected bus 1 output to bus 0 input • Initialized and started the unit

So frakkin’ what?

The Grind • Setting up the Remote IO unit and its stream properties is something you’ll do all the time • Now let’s try something different

We can do more than just connect… Bus 1 Bus 1 Remote I/O Bus 0 Bus 0

Example 2: Play through with render callbacks

Pulling audio • A unit’s pull can be done one of several ways: • By being connected directly to another audio unit • By registering a “render callback” to provide samples • This is the key to all things cool…

Setting a render callback • Instead of connecting the audio units kAudioUnitProperty_MakeConnection… • Set the property kAudioUnitProperty_SetRenderCallback • Provide a function pointer to your own function, and a “user info” object • Any samples you provide get played

Render Callback steps 1. Enable recording via Audio Session 2. Get the Remote I/O unit 3. Set stream format 4. Set up callback function 5. Let ’er rip You’ve already done all these

Creating a render callback • Deﬁne a struct containing whatever data your callback function will need • Set up an AUCallbackStruct • Set it as the kAudioUnitProperty_SetRenderCallback on bus 0 • Implement the callback function

Create your user data / context typedef struct { ! AudioUnit rioUnit; } EffectState; //... EffectState effectState;

AUCallbackStruct AURenderCallbackStruct callbackStruct; callbackStruct.inputProc = MyAURenderCallback; callbackStruct.inputProcRefCon = effectState;

Set callback property setupErr = ! AudioUnitSetProperty(remoteIOUnit, ! ! kAudioUnitProperty_SetRenderCallback, ! ! kAudioUnitScope_Global, ! ! bus0, ! ! &callbackStruct, ! ! sizeof (callbackStruct)); NSAssert (setupErr == noErr, @"Couldn't set RIO render callback on bus 0");

Callback function template typedef OSStatus (*AURenderCallback) ( void *inRefCon, AudioUnitRenderActionFlags *ioActionFlags, const AudioTimeStamp *inTimeStamp, UInt32 inBusNumber, UInt32 inNumberFrames, AudioBufferList *ioData );

Create callback function OSStatus MyAURenderCallback ( ! ! ! void *! ! ! ! ! ! ! inRefCon, ! ! ! AudioUnitRenderActionFlags *!ioActionFlags, ! ! ! const AudioTimeStamp *! ! ! inTimeStamp, ! ! ! UInt32! ! ! ! ! ! ! inBusNumber, ! ! ! UInt32! ! ! ! ! ! ! inNumberFrames, ! ! ! AudioBufferList *! ! ! ! ioData) { ! EffectState *effectState = (EffectState*) inRefCon; AudioUnit rioUnit = effectState->rioUnit;

Do something OSStatus renderErr = noErr; UInt32 bus1 = 1; renderErr = AudioUnitRender(rioUnit, ! ! ! ! ! ! ! ! ioActionFlags, ! ! ! ! ! ! ! ! inTimeStamp, ! ! ! ! ! ! ! ! bus1, ! ! ! ! ! ! ! ! inNumberFrames, ! ! ! ! ! ! ! ! ioData); NSAssert (renderErr == noErr, @”Couldn’t render”); • AudioUnitRender() gets available samples from a unit (either by copying from its buffer or calling an upstream unit)

We are still not impressed!

Almost there • You now have access to raw samples, in your render callback, by way of the ioData pointer • Anything you care to do with those samples can now be played out to hardware

Example 3: Play through with gain effect

iPhone Audio Effects • Generally need to be performed in render callbacks • On Mac OS X, you can create custom units to encapsulate effects • This was supposed to be in iPhone OS 3, but doesn’t actually work • Try AUPlugin.h See how far you get.

Trivial effect: gain

Gain effect • Get gain value of 0.0 to 1.0 from UISlider • Multiply each sample by this value

Render considerations • Render callbacks are on a real-time thread, with a hard deadline to return • If you miss the deadline you get silence • Render code must be highly performant

Bad Ideas for Callbacks • File or network I/O • Blocking threads • Heavy use of Obj-C messaging • malloc() • Basically anything that’s potentially slow, of indeterminate duration, and/or blocks

Good ideas for Callbacks • Pass a struct, rather than an Obj-C object, as the user info object for your callback • Let other threads do work for you and leave their work somewhere that the callback can just read it • Note possible race conditions

Slider value in struct typedef struct { ! AudioUnit rioUnit; ! float slider1Value; } EffectState; // set up callback state object effectState.rioUnit = remoteIOUnit; effectState.slider1Value = [slider1 value]; // set callback method AURenderCallbackStruct callbackStruct; callbackStruct.inputProc = MyAURenderCallback; callbackStruct.inputProcRefCon = &effectState; -(IBAction) handleSlider1ValueChanged { ! effectState.slider1Value = [slider1 value]; }

Read Slider Value in Callback EffectState *effectState = (EffectState*) inRefCon; AudioUnit rioUnit = effectState->rioUnit; float gain = effectState->slider1Value; // Call AudioUnitRender() as before to copy samples // into ioData

Apply Effect to Samples // walk the samples AudioSampleType sample = 0; for (int bufCount=0; bufCount<ioData->mNumberBuffers; bufCount++) { ! AudioBuffer buf = ioData->mBuffers[bufCount]; ! ! int currentFrame = 0; ! ! while ( currentFrame < inNumberFrames ) { ! ! ! // copy sample to buffer, across all channels ! ! ! for (int currentChannel=0; currentChannel<buf.mNumberChannels; currentChannel++) { ! ! ! ! memcpy(&sample, ! ! ! ! ! buf.mData + (currentFrame * 4) + (currentChannel*2), ! ! ! ! ! sizeof(AudioSampleType)); ! ! ! ! sample = (float) sample * gain; ! ! ! ! memcpy(buf.mData + (currentFrame * 4) + (currentChannel*2), ! ! ! ! ! &sample, ! ! ! ! ! sizeof(AudioSampleType)); ! ! ! }! ! ! currentFrame++; ! } }

Example 4: More Interesting Effects

Ring Modulator R(t) = C(t) x M(t)

Ring Modulation • Multiplication of two signals • One is usually a sine wave • Originally implemented as a ring-shaped circuit

Modulate! Modulate! • Ring modulator best known as the “Dalek” voice effect on Doctor Who (circa 1963) • Also used in early electronic music

Building a ring modulator • Need to model the sine wave • Multiply samples • Write modiﬁed sample back to ioData

Sine wave state typedef struct { ! AudioUnit rioUnit; ! float slider1Value; ! AudioStreamBasicDescription asbd; ! float sineFrequency; ! float sinePhase; } EffectState; // in setup method... // 23 Hz according to “Doctor Who” fan page // http://homepage.powerup.com.au/~spratleo/ Tech/Dalek_Voice_Primer.html effectState.sineFrequency = 23; effectState.sinePhase = 0; effectState.asbd = myASBD;

Sine wave value // AudioUnitRender() from RIO bus 1 here... // memcpy() from buffer here... float theta = effectState->sinePhase * M_PI * 2; sample = (sin(theta) * sample); // memcpy() to buffer here... effectState->sinePhase += 1.0 / (asbd.mSampleRate / sineFrequency); if (effectState->sinePhase > 1.0) { ! effectState->sinePhase -= 1.0; } /* to play pure sine wave sample = (sin (theta) * 0x7FFF); */

More Effects • http://musicdsp.org/, et. al.

Example 5: Mixing with Effects

Bus 1 Mixer Bus 0 Bus 0

Render Callback Bus 1 Remote Mixer Bus 0 Bus 0 I/O Bus 0 Remote Bus 1 I/O

Render Callback Bus 1 Remote Mixer Bus 0 Bus 0 I/O Bus 0 Audio Remote Unit Render Bus 1 Callback I/O Render()

Create a mixer unit AudioComponentDescription mixerDesc; mixerDesc.componentManufacturer = kAudioUnitManufacturer_Apple; mixerDesc.componentFlags = 0; mixerDesc.componentFlagsMask = 0; mixerDesc.componentType = kAudioUnitType_Mixer; mixerDesc.componentSubType = kAudioUnitSubType_MultiChannelMixer; AudioComponent mixerComponent = AudioComponentFindNext(NULL, &mixerDesc); setupErr = AudioComponentInstanceNew (mixerComponent, &mixerUnit); NSAssert (setupErr == noErr, @"Couldn't get mixer unit instance");

Conﬁg mixer unit • Set the stream property on its inputs • Set the render callback or connection properties on its inputs • Bus 0 is RobotVoiceRenderCallback • What shall we mix with?

Playing audio ﬁles in units • Can’t do I/O in render callback • If compressed, we’d also need to convert to PCM • Have a separate thread read/convert, then place PCM data in a ring buffer • See CARingBuffer (C++) in /Developer/ Extras/PublicUtility

Cheat #1: Convert AAC ﬁle to PCM afconvert --data LEI16 Girlfriend.m4a Girlfriend.caf

Cheat #2: Read ﬁle into RAM • Obviously not recommended

Set up struct for ﬁle- playing callback typedef struct { ! void* audioData; ! AudioSampleType *samplePtr; } MusicPlaybackState; //... MusicPlaybackState musicPlaybackState;

Open audio ﬁle NSURL *songURL = [NSURL fileURLWithPath: ! ! [[NSBundle mainBundle] pathForResource: @"Girlfriend" ! ! ! ! ofType: @"caf"]]; AudioFileID songFile; setupErr = AudioFileOpenURL((CFURLRef) songURL, ! ! ! kAudioFileReadPermission, ! ! ! 0, ! ! ! &songFile); NSAssert (setupErr == noErr, @"Couldn't open audio file");

Get size of audio data and malloc() UInt64 audioDataByteCount; UInt32 audioDataByteCountSize = sizeof (audioDataByteCount); setupErr = AudioFileGetProperty(songFile, ! ! ! ! ! kAudioFilePropertyAudioDataByteCount, ! ! ! ! ! &audioDataByteCountSize, ! ! ! ! ! &audioDataByteCount); NSAssert (setupErr == noErr, @"Couldn't get size property"); musicPlaybackState.audioData = malloc (audioDataByteCount); musicPlaybackState.samplePtr = musicPlaybackState.audioData;

Read ﬁle into RAM UInt32 bytesRead = audioDataByteCount; setupErr = AudioFileReadBytes(songFile, ! ! ! ! ! false, ! ! ! ! ! 0, ! ! ! ! ! &bytesRead, ! ! ! ! ! musicPlaybackState.audioData); NSAssert (setupErr == noErr, @"Couldn't read audio data");

Get ASBD from ﬁle AudioStreamBasicDescription fileASBD; setupErr = AudioFileGetProperty(songFile, ! ! ! ! ! kAudioFilePropertyDataFormat, ! ! ! ! ! &asbdSize, ! ! ! ! ! &fileASBD); NSAssert (setupErr == noErr, @"Couldn't get file asbd");

Provide samples to mixer Render Callback Bus 1 Mixer

Set up render callback AURenderCallbackStruct musicPlayerCallbackStruct; musicPlayerCallbackStruct.inputProc = MusicPlayerCallback; musicPlayerCallbackStruct.inputProcRefCon = &musicPlaybackState; ! setupErr = ! AudioUnitSetProperty(mixerUnit, ! ! ! ! ! kAudioUnitProperty_SetRenderCallback, ! ! ! ! ! kAudioUnitScope_Global, ! ! ! ! ! 1, ! ! ! ! ! &musicPlayerCallbackStruct, ! ! ! ! ! sizeof (musicPlayerCallbackStruct)); NSAssert (setupErr == noErr, @"Couldn't set mixer render callback on bus 1");

Create MusicPlayerCallback() MusicPlaybackState *musicPlaybackState = (MusicPlaybackState*) inRefCon; ! ! // walk the samples AudioSampleType sample = 0; for (int bufCount=0; bufCount<ioData->mNumberBuffers; bufCount++) { ! AudioBuffer buf = ioData->mBuffers[bufCount]; ! // AudioSampleType* bufferedSample = (AudioSampleType*) &buf.mData; ! int currentFrame = 0; ! while ( currentFrame < inNumberFrames ) { ! ! // copy sample to buffer, across all channels ! ! for (int currentChannel=0; currentChannel<buf.mNumberChannels; currentChannel++) { ! ! ! sample = *musicPlaybackState->samplePtr++; ! ! ! memcpy(buf.mData + (currentFrame * 4) + (currentChannel*2), ! ! ! ! &sample, ! ! ! ! sizeof(AudioSampleType)); ! ! }! ! ! currentFrame++; ! } } return noErr;

Connect mixer to RIO Remote Mixer Bus 0 Bus 0 I/O

Connect mixer to RIO AudioUnitConnection connection; connection.sourceAudioUnit = mixerUnit; connection.sourceOutputNumber = bus0; connection.destInputNumber = bus0; ! setupErr = ! AudioUnitSetProperty(remoteIOUnit, kAudioUnitProperty_MakeConnection, kAudioUnitScope_Input, bus0, &connection, sizeof (connection)); NSAssert (setupErr == noErr, @"Couldn't set mixer-to-RIO connection");

Bonus! • Multichannel mixer unit has volume parameter on each of its inputs

Adjusting mixer input volume -(IBAction) handleMicSliderValueChanged { ! OSStatus propSetErr = noErr; ! AudioUnitParameterValue sliderVal = [micSlider value]; ! propSetErr = AudioUnitSetParameter(mixerUnit, ! ! ! ! ! ! kMultiChannelMixerParam_Volume, ! ! ! ! ! ! kAudioUnitScope_Input, ! ! ! ! ! ! 0, ! ! ! ! ! ! sliderVal, ! ! ! ! ! ! 0); NSAssert (propSetErr == noErr, @"Couldn't set mixer volume on bus 0"); }

Takeaways • Core Audio is hard • Core Audio lets you do things that are freaking awesome

Since you’ll need help • coreaudio-api at lists.apple.com • Apple developers post here every day • stackoverﬂow.com • devforums.apple.com • Look for Michael Tyson’s post on Remote I/O unit

Coming soon eventually

Ask me? • [Time code]; blog • http://www.subfurther.com/blog • @invalidname • invalidname [at] gmail [dot] com

http://www.slideshare.net	106
http://kaustubhvibhute.blogspot.com	31
http://www.kaustubhvibhute.blogspot.com	4
http://webcache.googleusercontent.com	1
http://www.techgig.com	1

Core Audio: Don't Be Afraid to Play it LOUD! [360iDev, San Jose 2010]

by Chris Adamson on Apr 26, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

5 Embeds 143

Statistics

Core Audio: Don’t Be Afraid to Play it LOUD! [360iDev, San Jose 2010] — Presentation Transcript