• Save
Core Audio: Don't Be Afraid to Play it LOUD! [360iDev, San Jose 2010]
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Core Audio: Don't Be Afraid to Play it LOUD! [360iDev, San Jose 2010]

on

  • 21,646 views

Some of the best apps on the iPhone so far are about audio: capturing it, processing it, playing it, and even synthesizing it. But how are some of these apps even possible? In this session, we'll deep ...

Some of the best apps on the iPhone so far are about audio: capturing it, processing it, playing it, and even synthesizing it. But how are some of these apps even possible? In this session, we'll deep into the darkest depths of Core Audio, the iPhone's enormously powerful and often challenging audio API. You'll learn the tricks of the Core Audio that aren't obvious from the docs, and the essential techniques you'll need to shake your users' headphones.

Statistics

Views

Total Views
21,646
Views on SlideShare
21,484
Embed Views
162

Actions

Likes
19
Downloads
0
Comments
1

8 Embeds 162

http://www.slideshare.net 106
http://kaustubhvibhute.blogspot.com 31
https://twitter.com 15
http://www.kaustubhvibhute.blogspot.com 4
http://www.pursuitofthewow.blogspot.com 3
http://webcache.googleusercontent.com 1
http://www.techgig.com 1
http://pursuitofthewow.blogspot.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Core Audio: Don't Be Afraid to Play it LOUD! [360iDev, San Jose 2010] Presentation Transcript

  • 1. Core Audio: Don’t Be Afraid To Play It LOUD! Chris Adamson 360iDev San Jose 2010 @invalidname
  • 2. Previously… Robert Strojan — “iPhone Audio Lab” Monday, 12:30
  • 3. "Easy" and "CoreAudio" can't be used in the same sentence. CoreAudio is very powerful, very complex, and under-documented. Be prepared for a steep learning curve, APIs with millions of tiny little pieces, and puzzling things out from sample code rather than reading high-level documentation. –Jens Alfke, coreaudio-api list Feb 9, 2009
  • 4. Why?
  • 5. • Problem domain is hard • Performance is hard • Low latency is hard • Reusability is hard
  • 6. • Doing without would suck • Slowness would suck • Latency would suck • Non-reusability would suck
  • 7. Last Exit Before Toll • Alternatives to Core Audio: • Media Player: plays audio from iPod library • AV Framework: Obj-C API for file playback and recording
  • 8. Are they gone?
  • 9. What They Missed • Mixing • Effects • Streaming from/to network • Lots of C
  • 10. Core Audio Programming
  • 11. CA Style • Procedural C • Work with buffers of samples • Most functions return an OSStatus • You must check this, every time • Heavily callback-oriented • Heavily property-oriented
  • 12. Example: Audio Session • iPhone-only API • Inspect and set audio hardware and software properties • Declare interaction with other audio processes on the device (e.g., iPod)
  • 13. #import-ant! #import <AudioToolbox/AudioToolbox.h>
  • 14. Get Thee Some Docs!
  • 15. Initialize audio session OSStatus setupAudioSessionErr= ! AudioSessionInitialize ( ! ! ! NULL, // default run loop ! ! ! NULL, // default run loop mode ! ! ! MyInterruptionHandler, // interruption callback ! ! ! self); // client callback data ! NSAssert (setupAudioSessionErr == noErr, @"Couldn't initialize audio session");
  • 16. Initialize audio session C function call OSStatus setupAudioSessionErr= ! AudioSessionInitialize ( ! ! ! NULL, // default run loop ! ! ! NULL, // default run loop mode ! ! ! MyInterruptionHandler, // interruption callback ! ! ! self); // client callback data ! NSAssert (setupAudioSessionErr == noErr, @"Couldn't initialize audio session");
  • 17. Initialize audio session OSStatus setupAudioSessionErr= ! AudioSessionInitialize ( C function pointer ! ! ! NULL, // default run loop ! ! ! NULL, // default run loop mode ! ! ! MyInterruptionHandler, // interruption callback ! ! ! self); // client callback data ! NSAssert (setupAudioSessionErr == noErr, @"Couldn't initialize audio session");
  • 18. Initialize audio session OSStatus setupAudioSessionErr= ! AudioSessionInitialize ( ! ! ! NULL, // default run loop ! ! ! NULL, // default run loop mode ! ! ! MyInterruptionHandler, // interruption callback ! ! ! self); // client callback data ! Obj-C pointer NSAssert (setupAudioSessionErr == noErr, @"Couldn't initialize audio session");
  • 19. Initialize audio session Return value OSStatus setupAudioSessionErr= ! AudioSessionInitialize ( ! ! ! NULL, // default run loop ! ! ! NULL, // default run loop mode ! ! ! MyInterruptionHandler, // interruption callback ! ! ! self); // client callback data ! NSAssert (setupAudioSessionErr == noErr, @"Couldn't initialize audio session");
  • 20. Set category property UInt32 sessionCategory = kAudioSessionCategory_PlayAndRecord; setupAudioSessionErr = AudioSessionSetProperty ( kAudioSessionProperty_AudioCategory, sizeof (sessionCategory), &sessionCategory); ! NSAssert (setupAudioSessionErr == noErr, @"Couldn't set audio session property");
  • 21. Set category property UInt32 sessionCategory = kAudioSessionCategory_PlayAndRecord; Property name setupAudioSessionErr = (constant) AudioSessionSetProperty ( kAudioSessionProperty_AudioCategory, sizeof (sessionCategory), &sessionCategory); ! NSAssert (setupAudioSessionErr == noErr, @"Couldn't set audio session property");
  • 22. Set category property UInt32 sessionCategory = kAudioSessionCategory_PlayAndRecord; setupAudioSessionErr = AudioSessionSetProperty ( Size of property kAudioSessionProperty_AudioCategory, sizeof (sessionCategory), &sessionCategory); ! NSAssert (setupAudioSessionErr == noErr, @"Couldn't set audio session property");
  • 23. Set category property UInt32 sessionCategory = kAudioSessionCategory_PlayAndRecord; setupAudioSessionErr = AudioSessionSetProperty ( kAudioSessionProperty_AudioCategory, sizeof (sessionCategory), &sessionCategory); ! Pointer to value NSAssert (setupAudioSessionErr == noErr, @"Couldn't set audio session property");
  • 24. Is audio input available? UInt32 ui32PropertySize = sizeof (UInt32); UInt32 inputAvailable; setupAudioSessionErr = ! AudioSessionGetProperty( kAudioSessionProperty_AudioInputAvailable, &ui32PropertySize, &inputAvailable); NSAssert (setupAudioSessionErr == noErr, @"Couldn't get current audio input available prop");
  • 25. Is audio input available? Pointable size UInt32 ui32PropertySize = sizeof (UInt32); UInt32 inputAvailable; setupAudioSessionErr = ! AudioSessionGetProperty( kAudioSessionProperty_AudioInputAvailable, &ui32PropertySize, &inputAvailable); NSAssert (setupAudioSessionErr == noErr, @"Couldn't get current audio input available prop");
  • 26. Is audio input available? UInt32 ui32PropertySize = sizeof (UInt32); UInt32 inputAvailable; Pointable variable setupAudioSessionErr = ! AudioSessionGetProperty( kAudioSessionProperty_AudioInputAvailable, &ui32PropertySize, &inputAvailable); NSAssert (setupAudioSessionErr == noErr, @"Couldn't get current audio input available prop");
  • 27. Is audio input available? UInt32 ui32PropertySize = sizeof (UInt32); UInt32 inputAvailable; setupAudioSessionErr = ! AudioSessionGetProperty( kAudioSessionProperty_AudioInputAvailable, &ui32PropertySize, &inputAvailable); Address of size and variable NSAssert (setupAudioSessionErr == noErr, @"Couldn't get current audio input available prop");
  • 28. Callback on a property setupAudioSessionErr = AudioSessionAddPropertyListener ( ! ! ! kAudioSessionProperty_AudioInputAvailable, ! ! ! MyInputAvailableListener, ! ! ! self); NSAssert (setupAudioSessionErr == noErr, @"Couldn't setup audio input prop listener");
  • 29. Callback on a property setupAudioSessionErr = AudioSessionAddPropertyListener ( ! ! ! kAudioSessionProperty_AudioInputAvailable, ! ! ! MyInputAvailableListener, ! ! ! self); Property to listen to NSAssert (setupAudioSessionErr == noErr, @"Couldn't setup audio input prop listener");
  • 30. Callback on a property setupAudioSessionErr = AudioSessionAddPropertyListener ( ! ! ! kAudioSessionProperty_AudioInputAvailable, ! ! ! MyInputAvailableListener, ! ! ! self); C function pointer NSAssert (setupAudioSessionErr == noErr, @"Couldn't setup audio input prop listener");
  • 31. Callback on a property setupAudioSessionErr = AudioSessionAddPropertyListener ( ! ! ! kAudioSessionProperty_AudioInputAvailable, ! ! ! MyInputAvailableListener, ! ! ! self); “Client data” pointer NSAssert (setupAudioSessionErr == noErr, @"Couldn't setup audio input prop listener");
  • 32. CA style recap • Lots of getting and setting properties • Always check the return OSStatus • Asychronous callbacks to C functions • Look up function templates in docs • “Context” or “user info” pointer can be an Obj-C object
  • 33. What’s In Core Audio?
  • 34. Engines Utilities
  • 35. Engines Utilities Audio Units
  • 36. Engines Utilities Open AL Audio Units
  • 37. Engines Utilities Audio Queue Open AL Audio Units
  • 38. Engines Utilities AV Fndtn Audio Queue Open AL Audio Units
  • 39. Engines Utilities AV Fndtn Audio Queue Open AL Audio Units Audio File
  • 40. Engines Utilities AV Fndtn Audio Queue Open AL Audio Conversion Audio Units Audio File
  • 41. Engines Utilities AV Fndtn Ext Audio File Audio Queue Open AL Audio Conversion Audio Units Audio File
  • 42. Engines Utilities Audio File Stream AV Fndtn Ext Audio File Audio Queue Open AL Audio Conversion Audio Units Audio File
  • 43. Engines Utilities Audio Session Audio File Stream AV Fndtn Ext Audio File Audio Queue Open AL Audio Conversion Audio Units Audio File
  • 44. iPhone Audio Engines • Media Player • AV Foundation • Audio Queue Services • OpenAL • Audio Unit Services
  • 45. Media Player • Objective-C class MPMusicPlayerController — Plays music from iPod library • play, pause, stop. Properties for currentPlaybackTime, volume, etc. • No access to decoded samples or files
  • 46. AV Framework • Objective-C classes: AVAudioPlayer, AVAudioRecorder, AVAudioSession • Play from / record to files • Handles compressed formats • play, record, pause, stop. volume, currentTime properties • No access to decoded samples
  • 47. Audio Queue • C functions for playback, recording • Callback-driven: you fill or process audio buffers periodically • Plays compressed formats • Latency is negotiable
  • 48. OpenAL • C API intended for gaming, designed to resemble OpenGL • Attach audio to “sources” in 3D space • Listener hears sound from source relative to their own position • Uncompressed (PCM) only • Can stream to a source (push model)
  • 49. Audio Units • Lowest publicly-accessed part of Core Audio • C API for audio processing • Uncompressed (PCM) only, no floats • Extremely low latency • OpenAL and Queue implemented atop units
  • 50. So What’s An Audio Unit?
  • 51. Elements / Elements / Buses Buses
  • 52. Input Output Scope Scope
  • 53. Global Scope
  • 54. AU “pull” model • Each unit in the chain pulls audio data from some source • Another unit • A callback to your code • Last unit is typically an I/O unit
  • 55. I/O Unit (output)
  • 56. Effect I/O Unit Unit (output)
  • 57. I/O Unit Effect I/O Unit (input) Unit (output)
  • 58. I/O Unit Effect I/O Unit (input) Unit (output) Caveat: Not entirely accurate, for reasons to be explained soon…
  • 59. Units on iPhone OS • I/O Units • “Remote” I/O: Abstraction around harware in/out • Voice Processing I/O: Same as RemoteIO, but with echo supression for VOIP • Generic I/O: No hardware access; use for software audio rendering
  • 60. Units on iPhone OS • Mixer Units • Multichannel Mixer: Mixes multiple input streams into one output stream • 3D Mixer: Mixes inputs with more control over panning, resampling. Used by OpenAL
  • 61. Units on iPhone OS • Effect units • iPod Equalizer: Provides same features as iPod playback equalizer • Converter units • Converter: Converts between flavors of PCM, but not compressed formats
  • 62. Missing Units • Unit types absent from iPhone OS • Generators • Music devices (MIDI) and effects • Many units found on Mac absent on iPhone • Filters, reverb, compressor, varispeed, other effects
  • 63. RemoteIO • Your app’s most direct access to audio input/output hardware • Used at the end of unit chains that play audio • Used at the beginning of chains that capture audio
  • 64. RemoteIO Buses • Bus 0 is for H/W output • Headphones, speakers, etc. • Bus 1 is for H/W input • Headset mic, phone mic, etc.
  • 65. Bus 1 Bus 1 Remote I/O Bus 0 Bus 0
  • 66. Bus 1 Bus 1 Remote I/O Bus 0 Bus 0
  • 67. Bus 1 Bus 1 Remote I/O Bus 0 Bus 0
  • 68. Bus 1 Bus 1 Remote I/O Bus 0 Bus 0
  • 69. Bus 1 Bus 1 Remote I/O Bus 0 Bus 0
  • 70. But! • There can only be one RemoteIO unit!
  • 71. Bus 1 Bus 1 Remote I/O Bus 0 Bus 0
  • 72. Bus 1 Bus 1 Remote I/O Bus 0 Bus 0
  • 73. Example 1: Play Through
  • 74. Play Through steps 1. Enable recording via Audio Session 2. Get the Remote I/O unit 3. Set stream format 4. Connect input to output 5. Let ’er rip
  • 75. 1. Set Up Audio Session for Recording • Initialize the audio session • Set a capture-enabling category • Get/set other useful session properties
  • 76. Initialize Audio Session OSStatus setupAudioSessionErr= AudioSessionInitialize ( ! ! ! NULL, // default run loop ! ! ! NULL, // default run loop mode ! ! ! NULL, // interruption callback ! ! ! NULL); // client callback data NSAssert (setupAudioSessionErr == noErr, @"Couldn't initialize audio session");
  • 77. Enable Recording UInt32 sessionCategory = kAudioSessionCategory_PlayAndRecord; ! setupAudioSessionErr = AudioSessionSetProperty ! (kAudioSessionProperty_AudioCategory, ! sizeof (sessionCategory), ! &sessionCategory); ! NSAssert (setupAudioSessionErr == noErr, @"Couldn't set audio session property");
  • 78. Get HW sample rate UInt32 f64PropertySize = sizeof (Float64); setupAudioSessionErr = AudioSessionGetProperty (kAudioSessionProperty_CurrentHardwareSampleRate, ! &f64PropertySize, ! &hardwareSampleRate); ! NSAssert (setupAudioSessionErr == noErr, @"Couldn't get current hardware sample rate"); NSLog (@"current hardware sample rate = %f", hardwareSampleRate);
  • 79. 2. Get the Remote I/O Unit • Describe the unit you want • Search until you find it • Enable its input and output properties
  • 80. Describing a Unit AudioComponentDescription audioCompDesc; audioCompDesc.componentType = kAudioUnitType_Output; audioCompDesc.componentSubType = kAudioUnitSubType_RemoteIO; audioCompDesc.componentManufacturer = kAudioUnitManufacturer_Apple; audioCompDesc.componentFlags = 0; audioCompDesc.componentFlagsMask = 0;
  • 81. Discovering a Unit AudioUnit remoteIOUnit = NULL; AudioComponent rioComponent = AudioComponentFindNext (NULL, &audioCompDesc); OSErr setupErr = AudioComponentInstanceNew (rioComponent, &remoteIOUnit); NSAssert (setupErr == noErr, @"Couldn't get RIO unit instance"); • If there can be more than one match, iterate over AudioComponentFindNext, passing in last match you found
  • 82. Enable output property UInt32 oneFlag = 1; AudioUnitElement bus0 = 0; setupErr = ! AudioUnitSetProperty (remoteIOUnit, ! ! ! ! ! ! kAudioOutputUnitProperty_EnableIO, ! ! ! ! ! ! kAudioUnitScope_Output, ! ! ! ! ! ! bus0, ! ! ! ! ! ! &oneFlag, ! ! ! ! ! ! sizeof(oneFlag)); NSAssert (setupErr == noErr, @"Couldn't enable RIO output");
  • 83. Enable input property AudioUnitElement bus1 = 1; setupErr = AudioUnitSetProperty(remoteIOUnit, ! ! ! kAudioOutputUnitProperty_EnableIO, ! ! ! ! ! kAudioUnitScope_Input, ! ! ! ! ! bus1, ! ! ! ! &oneFlag, ! ! ! ! ! sizeof(oneFlag)); NSAssert (setupErr == noErr, @"couldn't enable RIO input");
  • 84. 3. Set stream properties • Define an AudioStreamBasicDescription • Set it as the kAudioUnitProperty_StreamFormat • You will screw this up at least once
  • 85. The ASBD • Description of the common traits of an entire audio stream • Bitrate, channel count, format, format- specific flags, etc. • Not all fields apply to all formats • On iPhone, Audio Units always work with LPCM
  • 86. Create an ASBD AudioStreamBasicDescription myASBD; memset (&myASBD, 0, sizeof (myASBD)); myASBD.mSampleRate = hardwareSampleRate; myASBD.mFormatID = kAudioFormatLinearPCM; myASBD.mFormatFlags = kAudioFormatFlagsCanonical; myASBD.mBytesPerPacket = 4; myASBD.mFramesPerPacket = 1; myASBD.mBytesPerFrame = myASBD.mBytesPerPacket * myASBD.mFramesPerPacket; myASBD.mChannelsPerFrame = 2; myASBD.mBitsPerChannel = 16;
  • 87. Set RemoteIO’s stream properties • Declares format you want to read from input bus, and write to output bus • Set the kAudioUnitProperty_StreamFormat property • On RemoteIO’s output scope for bus 1 • On RemoteIO’s input scope for bus 0
  • 88. Bus 1 Bus 1 Remote I/O Bus 0 Bus 0
  • 89. Set bus 1 / output- scope stream format setupErr = ! AudioUnitSetProperty (remoteIOUnit, ! ! ! ! ! ! kAudioUnitProperty_StreamFormat, ! ! ! ! ! ! kAudioUnitScope_Output, ! ! ! ! ! ! bus1, ! ! ! ! ! ! &myASBD, ! ! ! ! ! ! sizeof (myASBD)); NSAssert (setupErr == noErr, @"Couldn't set ASBD for RIO on output scope / bus 1");
  • 90. Set bus 0 / input-scope stream format setupErr = ! AudioUnitSetProperty (remoteIOUnit, ! ! ! ! ! ! kAudioUnitProperty_StreamFormat, ! ! ! ! ! ! kAudioUnitScope_Input, ! ! ! ! ! ! bus0, ! ! ! ! ! ! &myASBD, ! ! ! ! ! ! sizeof (myASBD)); NSAssert (setupErr == noErr, @"Couldn't set ASBD for RIO on input scope / bus 0");
  • 91. 4. Connect Input to Output • Create a structure describing the connection • Set it as a property on the destination audio unit
  • 92. Declare connection AudioUnitConnection connection; connection.sourceAudioUnit = remoteIOUnit; connection.sourceOutputNumber = bus1; connection.destInputNumber = bus0;
  • 93. Set connection property setupErr = ! AudioUnitSetProperty(remoteIOUnit, ! ! ! ! ! ! kAudioUnitProperty_MakeConnection, ! ! ! ! ! ! kAudioUnitScope_Input, ! ! ! ! ! ! bus0, ! ! ! ! ! ! &connection, ! ! ! ! ! ! sizeof (connection)); NSAssert (setupErr == noErr, @"Couldn't set RIO connection");
  • 94. 5. Let ’er Rip setupErr =! AudioUnitInitialize(remoteIOUnit); NSAssert (setupErr == noErr, @"Couldn't initialize RIO unit"); // in handleStartTapped: OSStatus startErr = noErr; startErr = AudioOutputUnitStart (remoteIOUnit); NSAssert (startErr == noErr, @"Couldn't start RIO unit");
  • 95. Demo
  • 96. Recap • We created the Remote I/O unit • Enabled input and output • Created an ASBD to describe stream format and set it on bus 1 output and bus 0 input • Connected bus 1 output to bus 0 input • Initialized and started the unit
  • 97. So frakkin’ what?
  • 98. The Grind • Setting up the Remote IO unit and its stream properties is something you’ll do all the time • Now let’s try something different
  • 99. We can do more than just connect… Bus 1 Bus 1 Remote I/O Bus 0 Bus 0
  • 100. Example 2: Play through with render callbacks
  • 101. Pulling audio • A unit’s pull can be done one of several ways: • By being connected directly to another audio unit • By registering a “render callback” to provide samples • This is the key to all things cool…
  • 102. Setting a render callback • Instead of connecting the audio units kAudioUnitProperty_MakeConnection… • Set the property kAudioUnitProperty_SetRenderCallback • Provide a function pointer to your own function, and a “user info” object • Any samples you provide get played
  • 103. Render Callback steps 1. Enable recording via Audio Session 2. Get the Remote I/O unit 3. Set stream format 4. Set up callback function 5. Let ’er rip You’ve already done all these
  • 104. Creating a render callback • Define a struct containing whatever data your callback function will need • Set up an AUCallbackStruct • Set it as the kAudioUnitProperty_SetRenderCallback on bus 0 • Implement the callback function
  • 105. Create your user data / context typedef struct { ! AudioUnit rioUnit; } EffectState; //... EffectState effectState;
  • 106. AUCallbackStruct AURenderCallbackStruct callbackStruct; callbackStruct.inputProc = MyAURenderCallback; callbackStruct.inputProcRefCon = effectState;
  • 107. Set callback property setupErr = ! AudioUnitSetProperty(remoteIOUnit, ! ! kAudioUnitProperty_SetRenderCallback, ! ! kAudioUnitScope_Global, ! ! bus0, ! ! &callbackStruct, ! ! sizeof (callbackStruct)); NSAssert (setupErr == noErr, @"Couldn't set RIO render callback on bus 0");
  • 108. Callback function template typedef OSStatus (*AURenderCallback) ( void *inRefCon, AudioUnitRenderActionFlags *ioActionFlags, const AudioTimeStamp *inTimeStamp, UInt32 inBusNumber, UInt32 inNumberFrames, AudioBufferList *ioData );
  • 109. Create callback function OSStatus MyAURenderCallback ( ! ! ! void *! ! ! ! ! ! ! inRefCon, ! ! ! AudioUnitRenderActionFlags *!ioActionFlags, ! ! ! const AudioTimeStamp *! ! ! inTimeStamp, ! ! ! UInt32! ! ! ! ! ! ! inBusNumber, ! ! ! UInt32! ! ! ! ! ! ! inNumberFrames, ! ! ! AudioBufferList *! ! ! ! ioData) { ! EffectState *effectState = (EffectState*) inRefCon; AudioUnit rioUnit = effectState->rioUnit;
  • 110. Do something OSStatus renderErr = noErr; UInt32 bus1 = 1; renderErr = AudioUnitRender(rioUnit, ! ! ! ! ! ! ! ! ioActionFlags, ! ! ! ! ! ! ! ! inTimeStamp, ! ! ! ! ! ! ! ! bus1, ! ! ! ! ! ! ! ! inNumberFrames, ! ! ! ! ! ! ! ! ioData); NSAssert (renderErr == noErr, @”Couldn’t render”); • AudioUnitRender() gets available samples from a unit (either by copying from its buffer or calling an upstream unit)
  • 111. Demo
  • 112. We are still not impressed!
  • 113. Almost there • You now have access to raw samples, in your render callback, by way of the ioData pointer • Anything you care to do with those samples can now be played out to hardware
  • 114. Example 3: Play through with gain effect
  • 115. iPhone Audio Effects • Generally need to be performed in render callbacks • On Mac OS X, you can create custom units to encapsulate effects • This was supposed to be in iPhone OS 3, but doesn’t actually work • Try AUPlugin.h See how far you get.
  • 116. Trivial effect: gain
  • 117. Gain effect • Get gain value of 0.0 to 1.0 from UISlider • Multiply each sample by this value
  • 118. Render considerations • Render callbacks are on a real-time thread, with a hard deadline to return • If you miss the deadline you get silence • Render code must be highly performant
  • 119. Bad Ideas for Callbacks • File or network I/O • Blocking threads • Heavy use of Obj-C messaging • malloc() • Basically anything that’s potentially slow, of indeterminate duration, and/or blocks
  • 120. Good ideas for Callbacks • Pass a struct, rather than an Obj-C object, as the user info object for your callback • Let other threads do work for you and leave their work somewhere that the callback can just read it • Note possible race conditions
  • 121. Slider value in struct typedef struct { ! AudioUnit rioUnit; ! float slider1Value; } EffectState; // set up callback state object effectState.rioUnit = remoteIOUnit; effectState.slider1Value = [slider1 value]; // set callback method AURenderCallbackStruct callbackStruct; callbackStruct.inputProc = MyAURenderCallback; callbackStruct.inputProcRefCon = &effectState; -(IBAction) handleSlider1ValueChanged { ! effectState.slider1Value = [slider1 value]; }
  • 122. Read Slider Value in Callback EffectState *effectState = (EffectState*) inRefCon; AudioUnit rioUnit = effectState->rioUnit; float gain = effectState->slider1Value; // Call AudioUnitRender() as before to copy samples // into ioData
  • 123. Apply Effect to Samples // walk the samples AudioSampleType sample = 0; for (int bufCount=0; bufCount<ioData->mNumberBuffers; bufCount++) { ! AudioBuffer buf = ioData->mBuffers[bufCount]; ! ! int currentFrame = 0; ! ! while ( currentFrame < inNumberFrames ) { ! ! ! // copy sample to buffer, across all channels ! ! ! for (int currentChannel=0; currentChannel<buf.mNumberChannels; currentChannel++) { ! ! ! ! memcpy(&sample, ! ! ! ! ! buf.mData + (currentFrame * 4) + (currentChannel*2), ! ! ! ! ! sizeof(AudioSampleType)); ! ! ! ! sample = (float) sample * gain; ! ! ! ! memcpy(buf.mData + (currentFrame * 4) + (currentChannel*2), ! ! ! ! ! &sample, ! ! ! ! ! sizeof(AudioSampleType)); ! ! ! }! ! ! currentFrame++; ! } }
  • 124. Demo
  • 125. Example 4: More Interesting Effects
  • 126. Ring Modulator R(t) = C(t) x M(t)
  • 127. Ring Modulation • Multiplication of two signals • One is usually a sine wave • Originally implemented as a ring-shaped circuit
  • 128. Modulate! Modulate! • Ring modulator best known as the “Dalek” voice effect on Doctor Who (circa 1963) • Also used in early electronic music
  • 129. Building a ring modulator • Need to model the sine wave • Multiply samples • Write modified sample back to ioData
  • 130. Sine wave state typedef struct { ! AudioUnit rioUnit; ! float slider1Value; ! AudioStreamBasicDescription asbd; ! float sineFrequency; ! float sinePhase; } EffectState; // in setup method... // 23 Hz according to “Doctor Who” fan page // http://homepage.powerup.com.au/~spratleo/ Tech/Dalek_Voice_Primer.html effectState.sineFrequency = 23; effectState.sinePhase = 0; effectState.asbd = myASBD;
  • 131. Sine wave value // AudioUnitRender() from RIO bus 1 here... // memcpy() from buffer here... float theta = effectState->sinePhase * M_PI * 2; sample = (sin(theta) * sample); // memcpy() to buffer here... effectState->sinePhase += 1.0 / (asbd.mSampleRate / sineFrequency); if (effectState->sinePhase > 1.0) { ! effectState->sinePhase -= 1.0; } /* to play pure sine wave sample = (sin (theta) * 0x7FFF); */
  • 132. Demo
  • 133. More Effects • http://musicdsp.org/, et. al.
  • 134. Example 5: Mixing with Effects
  • 135. Bus 1 Mixer Bus 0 Bus 0
  • 136. Render Callback Bus 1 Remote Mixer Bus 0 Bus 0 I/O Bus 0 Remote Bus 1 I/O
  • 137. Render Callback Bus 1 Remote Mixer Bus 0 Bus 0 I/O Bus 0 Audio Remote Unit Render Bus 1 Callback I/O Render()
  • 138. Create a mixer unit AudioComponentDescription mixerDesc; mixerDesc.componentManufacturer = kAudioUnitManufacturer_Apple; mixerDesc.componentFlags = 0; mixerDesc.componentFlagsMask = 0; mixerDesc.componentType = kAudioUnitType_Mixer; mixerDesc.componentSubType = kAudioUnitSubType_MultiChannelMixer; AudioComponent mixerComponent = AudioComponentFindNext(NULL, &mixerDesc); setupErr = AudioComponentInstanceNew (mixerComponent, &mixerUnit); NSAssert (setupErr == noErr, @"Couldn't get mixer unit instance");
  • 139. Config mixer unit • Set the stream property on its inputs • Set the render callback or connection properties on its inputs • Bus 0 is RobotVoiceRenderCallback • What shall we mix with?
  • 140. Playing audio files in units • Can’t do I/O in render callback • If compressed, we’d also need to convert to PCM • Have a separate thread read/convert, then place PCM data in a ring buffer • See CARingBuffer (C++) in /Developer/ Extras/PublicUtility
  • 141. Cheat #1: Convert AAC file to PCM afconvert --data LEI16 Girlfriend.m4a Girlfriend.caf
  • 142. Cheat #2: Read file into RAM • Obviously not recommended
  • 143. Set up struct for file- playing callback typedef struct { ! void* audioData; ! AudioSampleType *samplePtr; } MusicPlaybackState; //... MusicPlaybackState musicPlaybackState;
  • 144. Open audio file NSURL *songURL = [NSURL fileURLWithPath: ! ! [[NSBundle mainBundle] pathForResource: @"Girlfriend" ! ! ! ! ofType: @"caf"]]; AudioFileID songFile; setupErr = AudioFileOpenURL((CFURLRef) songURL, ! ! ! kAudioFileReadPermission, ! ! ! 0, ! ! ! &songFile); NSAssert (setupErr == noErr, @"Couldn't open audio file");
  • 145. Get size of audio data and malloc() UInt64 audioDataByteCount; UInt32 audioDataByteCountSize = sizeof (audioDataByteCount); setupErr = AudioFileGetProperty(songFile, ! ! ! ! ! kAudioFilePropertyAudioDataByteCount, ! ! ! ! ! &audioDataByteCountSize, ! ! ! ! ! &audioDataByteCount); NSAssert (setupErr == noErr, @"Couldn't get size property"); musicPlaybackState.audioData = malloc (audioDataByteCount); musicPlaybackState.samplePtr = musicPlaybackState.audioData;
  • 146. Read file into RAM UInt32 bytesRead = audioDataByteCount; setupErr = AudioFileReadBytes(songFile, ! ! ! ! ! false, ! ! ! ! ! 0, ! ! ! ! ! &bytesRead, ! ! ! ! ! musicPlaybackState.audioData); NSAssert (setupErr == noErr, @"Couldn't read audio data");
  • 147. Get ASBD from file AudioStreamBasicDescription fileASBD; setupErr = AudioFileGetProperty(songFile, ! ! ! ! ! kAudioFilePropertyDataFormat, ! ! ! ! ! &asbdSize, ! ! ! ! ! &fileASBD); NSAssert (setupErr == noErr, @"Couldn't get file asbd");
  • 148. Provide samples to mixer Render Callback Bus 1 Mixer
  • 149. Set up render callback AURenderCallbackStruct musicPlayerCallbackStruct; musicPlayerCallbackStruct.inputProc = MusicPlayerCallback; musicPlayerCallbackStruct.inputProcRefCon = &musicPlaybackState; ! setupErr = ! AudioUnitSetProperty(mixerUnit, ! ! ! ! ! kAudioUnitProperty_SetRenderCallback, ! ! ! ! ! kAudioUnitScope_Global, ! ! ! ! ! 1, ! ! ! ! ! &musicPlayerCallbackStruct, ! ! ! ! ! sizeof (musicPlayerCallbackStruct)); NSAssert (setupErr == noErr, @"Couldn't set mixer render callback on bus 1");
  • 150. Create MusicPlayerCallback() MusicPlaybackState *musicPlaybackState = (MusicPlaybackState*) inRefCon; ! ! // walk the samples AudioSampleType sample = 0; for (int bufCount=0; bufCount<ioData->mNumberBuffers; bufCount++) { ! AudioBuffer buf = ioData->mBuffers[bufCount]; ! // AudioSampleType* bufferedSample = (AudioSampleType*) &buf.mData; ! int currentFrame = 0; ! while ( currentFrame < inNumberFrames ) { ! ! // copy sample to buffer, across all channels ! ! for (int currentChannel=0; currentChannel<buf.mNumberChannels; currentChannel++) { ! ! ! sample = *musicPlaybackState->samplePtr++; ! ! ! memcpy(buf.mData + (currentFrame * 4) + (currentChannel*2), ! ! ! ! &sample, ! ! ! ! sizeof(AudioSampleType)); ! ! }! ! ! currentFrame++; ! } } return noErr;
  • 151. Connect mixer to RIO Remote Mixer Bus 0 Bus 0 I/O
  • 152. Connect mixer to RIO AudioUnitConnection connection; connection.sourceAudioUnit = mixerUnit; connection.sourceOutputNumber = bus0; connection.destInputNumber = bus0; ! setupErr = ! AudioUnitSetProperty(remoteIOUnit, kAudioUnitProperty_MakeConnection, kAudioUnitScope_Input, bus0, &connection, sizeof (connection)); NSAssert (setupErr == noErr, @"Couldn't set mixer-to-RIO connection");
  • 153. Bonus! • Multichannel mixer unit has volume parameter on each of its inputs
  • 154. Adjusting mixer input volume -(IBAction) handleMicSliderValueChanged { ! OSStatus propSetErr = noErr; ! AudioUnitParameterValue sliderVal = [micSlider value]; ! propSetErr = AudioUnitSetParameter(mixerUnit, ! ! ! ! ! ! kMultiChannelMixerParam_Volume, ! ! ! ! ! ! kAudioUnitScope_Input, ! ! ! ! ! ! 0, ! ! ! ! ! ! sliderVal, ! ! ! ! ! ! 0); NSAssert (propSetErr == noErr, @"Couldn't set mixer volume on bus 0"); }
  • 155. Demo
  • 156. Takeaways • Core Audio is hard • Core Audio lets you do things that are freaking awesome
  • 157. Since you’ll need help • coreaudio-api at lists.apple.com • Apple developers post here every day • stackoverflow.com • devforums.apple.com • Look for Michael Tyson’s post on Remote I/O unit
  • 158. Coming soon eventually
  • 159. Ask me? • [Time code]; blog • http://www.subfurther.com/blog • @invalidname • invalidname [at] gmail [dot] com