Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Core Audio in iOS 6                          Chris Adamson • @invalidname                                 CocoaConf DC    ...
Plug!Monday, March 25, 13
The Reviews Are In!Monday, March 25, 13
The Reviews Are In!Monday, March 25, 13
The Reviews Are In!Monday, March 25, 13
The Reviews Are In!Monday, March 25, 13
Legitimate copies!                       •   Amazon (paper or Kindle)                       •   Barnes & Noble (paper or N...
What You’ll Learn                       •   What Core Audio does and doesn’t do                       •   When to use and ...
Monday, March 25, 13
Simple things should be simple,                       complex things should be possible.                                  ...
AV Foundation,      Media Player                         Simple things should be simple,                       complex thi...
AV Foundation,      Media Player                         Simple things should be simple,                       complex thi...
Core Audio                       •   Low-level C framework for processing                           audio                 ...
Chris’ CA Taxonomy                       •   Engines: process streams of audio                           •   Capture, play...
Helpers: Audio File                       •   Read from / write to multiple audio file                           types (.ai...
Helpers: Audio File                                 Stream                       •   Read audio from non-random-access    ...
Helpers: Converters                       •   Convert buffers of audio to and from                           different enc...
Helpers: ExtAudioFile                       •   Combine file I/O and format conversion                       •   Read a com...
Helpers: Audio Session                       •   iOS-only API to negotiate use of audio                           resource...
Engines: Audio Units                       •   Low-latency (~10ms) processing of                           capture/play-ou...
Engines: Audio Queue                       •   Convenience API for recording or play-out,                           built ...
Engines: Open AL                       •   API for 3D spatialized audio, implemented                           atop audio ...
Engines and Helpers                       •   Audio Units   •   Audio File                       •   Audio Queue   •   Aud...
Audio UnitsMonday, March 25, 13
Audio Unit                         AUSomethingMonday, March 25, 13
Types of Audio Units                       •   Output (which also do input)                       •   Generator           ...
Pull Model                        AUSomethingMonday, March 25, 13
Pull Model                        AUSomething                                      AudioUnitRender()Monday, March 25, 13
Pull Model                       AUSomethingElse   AUSomethingMonday, March 25, 13
Buses (aka, Elements)                          AUSomethingElse                                            AUSomething     ...
AUGraph                       AUSomethingElse                                         AUSomething                       AU...
Render Callbacks                OSStatus converterInputRenderCallback (void *inRefCon,                                    ...
AURemoteIO                       •   Output unit used for play-out, capture                       •   A Core Audio thread ...
Create AURemoteIO      CheckError(NewAUGraph(&_auGraph),      ! !     "couldnt create au graph");      !      CheckError(A...
Getting an AudioUnit                           from AUNode                !      CheckError(AUGraphNodeInfo(self.auGraph, ...
AURemoteIO Buses                            AURemoteIOMonday, March 25, 13
AURemoteIO Buses                            AURemoteIO                                              bus 0                 ...
AURemoteIO Buses                            AURemoteIO        bus 0                                 bus 0      from app   ...
AURemoteIO Buses     bus 1from input H/W                            AURemoteIO        bus 0                               ...
AURemoteIO Buses     bus 1                                   bus 1from input H/W                              to app      ...
EnableIO         !    UInt32 oneFlag = 1;         !    UInt32 busZero = 0;         !    CheckError(AudioUnitSetProperty(se...
Pass Through                            bus 1                       from input H/W                                        ...
Connect In to Out            !    UInt32 busZero = 0;            !    UInt32 busOne = 1;            !    CheckError(AUGrap...
Pass-Through with Effect                                         AUEffect                            bus 1                ...
Demo: Delay Effect                             New in iOS 6!Monday, March 25, 13
Creating the AUDelay      ! componentDesc.componentType = kAudioUnitType_Effect;      ! componentDesc.componentSubType = k...
The problem with effect                           units                       •   Audio Units available since iPhone OS 2....
Setting formats       !    AudioStreamBasicDescription effectDataFormat;       !    UInt32 propSize = sizeof (effectDataFo...
AUNewTimePitch                       •   New in iOS 6!                       •   Allows you to change pitch independent of...
AUTimePitch   !   AudioComponentDescription effectcd = {0};   !   effectcd.componentType = kAudioUnitType_FormatConverter;...
AudioUnitParameters.h                   // Parameters for AUNewTimePitch                   enum {                   ! ! //...
AUNewTimePitch                                 parameters                       •   Rate: kNewTimePitchParam_Rate takes a ...
Pitch shifting                       •   Pitch can vary, time does not                       •   Suitable for real-time so...
Demo: Pitch Shift                            New in iOS 6!Monday, March 25, 13
Rate shifting                       •   Rate can vary, pitch does not                           •   Think of 1.5x and 2x s...
Demo: Rate Shift                            New in iOS 6!Monday, March 25, 13
AUSplitter                                          AUSomethingElse                       AUSplitter                      ...
AUMatrixMixer                       AUSomethingElse                                                            AUSomething...
Audio Queues                       (and the APIs that help them)Monday, March 25, 13
AudioQueue                       •   Easier than AURemoteIO - provide data                           when you want to, les...
AudioQueue                          2   1   0Monday, March 25, 13
Common AQ scenarios                       •   File player - Read from file and “prime”                           queue buff...
Web Radio                       •   Project from CocoaConfs Columbus,                           Portland, and Raleigh     ...
Parsing web radioMonday, March 25, 13
Parsing web radio   NSURLConnection delivers   NSData buffers, containing audio   and framing info. We pass it to         ...
Parsing web radio   NSURLConnection delivers   NSData buffers, containing audio   and framing info. We pass it to         ...
Parsing web radio   NSURLConnection delivers   NSData buffers, containing audio   and framing info. We pass it to         ...
A complex thing!                       •   What if we want to see that data after it’s                           been deco...
AudioQueue Tap!                       http://www.last.fm/music/Spinal+TapMonday, March 25, 13
AudioQueueProcessingTap                       •   Set as a property on the Audio Queue                       •   Calls bac...
Creating an AQ Tap   !    !      // create the tap   !    !      UInt32 maxFrames = 0;   !    !      AudioStreamBasicDescr...
AQ Tap Proc       void tapProc (void *                            inClientData,       ! ! !     AudioQueueProcessingTapRef...
So what should we do                          with the audio?Monday, March 25, 13
So what should we do                          with the audio?                         Let’s apply our pitch-shift effectMo...
Shouldn’t this work?                              AUEffectMonday, March 25, 13
Shouldn’t this work?                              AUEffect                                         AudioUnitRender()Monday...
AudioUnitRender()                       •   Last argument is an AudioBufferList, whose                           AudioBuff...
Psych!                       •   AQ tap provides data as signed ints                       •   Effect units only work with...
invalidname’s convert-                          and-effect recipe                        OSStatus converterInputRenderCall...
How it works                       •   AUGraph: AUConverter → AUEffect →                           AUConverter → AUGeneric...
The trick!                       •   Copy mData pointer to a state variable and                           NULL it in ioDat...
Yes, really                          This is the rest of tapProc()                ! // copy off the ioData so the graph ca...
Yes, really           This is the render callback that supplies data to the int→float converter    OSStatus converterInputR...
Demo: AQ Tap +                       AUNewTimePitch                            New in iOS 6!Monday, March 25, 13
Monday, March 25, 13
Other new stuffMonday, March 25, 13
Multi-Route                       •   Ordinarily, one input or output is active:                           earpiece, speak...
Utility classes moved                                   again                       •   C++ utilities, including the CARin...
Takeaways                       •   Core Audio fundamentals never change                       •   New stuff is added as p...
Q&A                       •   Slides will be posted to slideshare.net/                           invalidname              ...
Upcoming SlideShare
Loading in …5
×

Core Audio in iOS 6 (CocoaConf DC, March 2013)

1,277 views

Published on

Published in: Technology
  • Be the first to comment

Core Audio in iOS 6 (CocoaConf DC, March 2013)

  1. 1. Core Audio in iOS 6 Chris Adamson • @invalidname CocoaConf DC March 23, 2013 Sides and code available on my blog: http://www.subfurther.com/blogMonday, March 25, 13
  2. 2. Plug!Monday, March 25, 13
  3. 3. The Reviews Are In!Monday, March 25, 13
  4. 4. The Reviews Are In!Monday, March 25, 13
  5. 5. The Reviews Are In!Monday, March 25, 13
  6. 6. The Reviews Are In!Monday, March 25, 13
  7. 7. Legitimate copies! • Amazon (paper or Kindle) • Barnes & Noble (paper or Nook) • Apple (iBooks) • Direct from InformIT (paper, eBook [.epub + .mobi + .pdf], or Bundle) • 35% off with code COREAUDIO3174Monday, March 25, 13
  8. 8. What You’ll Learn • What Core Audio does and doesn’t do • When to use and not use it • What’s new in Core Audio for iOS 6Monday, March 25, 13
  9. 9. Monday, March 25, 13
  10. 10. Simple things should be simple, complex things should be possible. –Alan KayMonday, March 25, 13
  11. 11. AV Foundation, Media Player Simple things should be simple, complex things should be possible. –Alan KayMonday, March 25, 13
  12. 12. AV Foundation, Media Player Simple things should be simple, complex things should be possible. –Alan Kay Core AudioMonday, March 25, 13
  13. 13. Core Audio • Low-level C framework for processing audio • Capture, play-out, real-time or off-line processing • The “complex things should be possible” part of audio on OS X and iOSMonday, March 25, 13
  14. 14. Chris’ CA Taxonomy • Engines: process streams of audio • Capture, play-out, mixing, effects processing • Helpers: deal with formats, encodings, etc. • File I/O, stream I/O, format conversion, iOS “session” managementMonday, March 25, 13
  15. 15. Helpers: Audio File • Read from / write to multiple audio file types (.aiff, .wav, .caf, .m4a, .mp3) in a content-agnostic way • Get metadata (data format, duration, iTunes/ID3 info)Monday, March 25, 13
  16. 16. Helpers: Audio File Stream • Read audio from non-random-access source like a network stream • Discover encoding and encapsulation on the fly, then deliver audio packets to client applicationMonday, March 25, 13
  17. 17. Helpers: Converters • Convert buffers of audio to and from different encodings • One side must be in an uncompressed format (i.e., Linear PCM)Monday, March 25, 13
  18. 18. Helpers: ExtAudioFile • Combine file I/O and format conversion • Read a compressed file into PCM buffers • Write PCM buffers into a compressed fileMonday, March 25, 13
  19. 19. Helpers: Audio Session • iOS-only API to negotiate use of audio resources with the rest of the system • Deetermine whether your app mixes with other apps’ audio, honors ring/silent switch, can play in background, etc. • Gets notified of audio interruptions • See also AVAudioSessionMonday, March 25, 13
  20. 20. Engines: Audio Units • Low-latency (~10ms) processing of capture/play-out audio data • Effects, mixing, etc. • Connect units manually or via an AUGraph • Much more on this topic momentarily…Monday, March 25, 13
  21. 21. Engines: Audio Queue • Convenience API for recording or play-out, built atop audio units • Rather than processing on-demand and on Core Audio’s thread, your callback provides or receives buffers of audio (at whatever size is convenient to you) • Higher latency, naturally • Supports compressed formats (MP3, AAC)Monday, March 25, 13
  22. 22. Engines: Open AL • API for 3D spatialized audio, implemented atop audio units • Set a source’s properties (x/y/z coordinates, orientation, audio buffer, etc.), OpenAL renders what it sounds like to the listener from that locationMonday, March 25, 13
  23. 23. Engines and Helpers • Audio Units • Audio File • Audio Queue • Audio File Stream • Open AL • Audio Converter • ExtAudioFile • Audio SessionMonday, March 25, 13
  24. 24. Audio UnitsMonday, March 25, 13
  25. 25. Audio Unit AUSomethingMonday, March 25, 13
  26. 26. Types of Audio Units • Output (which also do input) • Generator • Converter • Effect • Mixer • MusicMonday, March 25, 13
  27. 27. Pull Model AUSomethingMonday, March 25, 13
  28. 28. Pull Model AUSomething AudioUnitRender()Monday, March 25, 13
  29. 29. Pull Model AUSomethingElse AUSomethingMonday, March 25, 13
  30. 30. Buses (aka, Elements) AUSomethingElse AUSomething AUSomethingElseMonday, March 25, 13
  31. 31. AUGraph AUSomethingElse AUSomething AUSomethingElseMonday, March 25, 13
  32. 32. Render Callbacks OSStatus converterInputRenderCallback (void *inRefCon, AudioUnitRenderActionFlags *ioActionFlags, const AudioTimeStamp *inTimeStamp, UInt32 inBusNumber, UInt32 inNumberFrames, AudioBufferList * ioData) { CCFWebRadioPlayer *player = (__bridge CCFWebRadioPlayer*) inRefCon; // read from buffer ioData->mBuffers[0].mData = player.preRenderData; return noErr; } AUSomething AUSomethingElseMonday, March 25, 13
  33. 33. AURemoteIO • Output unit used for play-out, capture • A Core Audio thread repeatedly and automatically calls AudioUnitRender() • Must set EnableIO property to explicitly enable capture and/or play-out • Capture requires setting appropriate AudioSession categoryMonday, March 25, 13
  34. 34. Create AURemoteIO CheckError(NewAUGraph(&_auGraph), ! ! "couldnt create au graph"); ! CheckError(AUGraphOpen(_auGraph), ! ! "couldnt open au graph"); ! AudioComponentDescription componentDesc; componentDesc.componentType = kAudioUnitType_Output; componentDesc.componentSubType = kAudioUnitSubType_RemoteIO; componentDesc.componentManufacturer = kAudioUnitManufacturer_Apple; ! AUNode remoteIONode; CheckError(AUGraphAddNode(_auGraph, ! ! ! ! ! ! &componentDesc, ! ! ! ! ! ! &remoteIONode), ! ! "couldnt add remote io node");Monday, March 25, 13
  35. 35. Getting an AudioUnit from AUNode ! CheckError(AUGraphNodeInfo(self.auGraph, ! ! ! ! ! ! ! remoteIONode, ! ! ! ! ! ! ! NULL, ! ! ! ! ! ! ! &_remoteIOUnit), ! ! ! "couldnt get remote io unit from node");Monday, March 25, 13
  36. 36. AURemoteIO Buses AURemoteIOMonday, March 25, 13
  37. 37. AURemoteIO Buses AURemoteIO bus 0 to output H/WMonday, March 25, 13
  38. 38. AURemoteIO Buses AURemoteIO bus 0 bus 0 from app to output H/WMonday, March 25, 13
  39. 39. AURemoteIO Buses bus 1from input H/W AURemoteIO bus 0 bus 0 from app to output H/WMonday, March 25, 13
  40. 40. AURemoteIO Buses bus 1 bus 1from input H/W to app AURemoteIO bus 0 bus 0 from app to output H/WMonday, March 25, 13
  41. 41. EnableIO ! UInt32 oneFlag = 1; ! UInt32 busZero = 0; ! CheckError(AudioUnitSetProperty(self.remoteIOUnit, ! ! ! ! ! ! ! ! ! kAudioOutputUnitProperty_EnableIO, ! ! ! ! ! ! ! ! ! kAudioUnitScope_Output, ! ! ! ! ! ! ! ! ! busZero, ! ! ! ! ! ! ! ! ! &oneFlag, ! ! ! ! ! ! ! ! ! sizeof(oneFlag)), ! ! ! "couldnt enable remote io output"); ! UInt32 busOne = 1; ! CheckError(AudioUnitSetProperty(self.remoteIOUnit, ! ! ! ! ! ! ! ! ! kAudioOutputUnitProperty_EnableIO, ! ! ! ! ! ! ! ! ! kAudioUnitScope_Input, ! ! ! ! ! ! ! ! ! busOne, ! ! ! ! ! ! ! ! ! &oneFlag, ! ! ! ! ! ! ! ! ! sizeof(oneFlag)), ! ! ! "couldnt enable remote io input");Monday, March 25, 13
  42. 42. Pass Through bus 1 from input H/W AURemoteIO bus 0 to output H/WMonday, March 25, 13
  43. 43. Connect In to Out ! UInt32 busZero = 0; ! UInt32 busOne = 1; ! CheckError(AUGraphConnectNodeInput(self.auGraph, ! ! ! ! ! ! ! ! ! remoteIONode, ! ! ! ! ! ! ! ! ! busOne, ! ! ! ! ! ! ! ! ! remoteIONode, ! ! ! ! ! ! ! ! ! busZero), ! ! ! "couldnt connect remote io bus 1 to 0");Monday, March 25, 13
  44. 44. Pass-Through with Effect AUEffect bus 1 from input H/W AURemoteIO bus 0 to output H/WMonday, March 25, 13
  45. 45. Demo: Delay Effect New in iOS 6!Monday, March 25, 13
  46. 46. Creating the AUDelay ! componentDesc.componentType = kAudioUnitType_Effect; ! componentDesc.componentSubType = kAudioUnitSubType_Delay; ! componentDesc.componentManufacturer = kAudioUnitManufacturer_Apple; ! ! AUNode effectNode; ! CheckError(AUGraphAddNode(self.auGraph, ! ! ! ! ! ! ! &componentDesc, ! ! ! ! ! ! ! &effectNode), ! ! ! "couldnt create effect node"); ! AudioUnit effectUnit; ! CheckError(AUGraphNodeInfo(self.auGraph, ! ! ! ! ! ! ! effectNode, ! ! ! ! ! ! ! NULL, ! ! ! ! ! ! ! &effectUnit), ! ! ! "couldnt get effect unit from node");Monday, March 25, 13
  47. 47. The problem with effect units • Audio Units available since iPhone OS 2.0 prefer int formats • Effect units arrived with iOS 5 (arm7 era) and only work with float format • Have to set the AUEffect unit’s format on AURemoteIOMonday, March 25, 13
  48. 48. Setting formats ! AudioStreamBasicDescription effectDataFormat; ! UInt32 propSize = sizeof (effectDataFormat); ! CheckError(AudioUnitGetProperty(effectUnit, ! ! ! ! ! ! ! ! ! kAudioUnitProperty_StreamFormat, ! ! ! ! ! ! ! ! ! kAudioUnitScope_Output, ! ! ! ! ! ! ! ! ! busZero, ! ! ! ! ! ! ! ! ! &effectDataFormat, ! ! ! ! ! ! ! ! ! &propSize), ! ! ! "couldnt read effect format"); ! CheckError(AudioUnitSetProperty(self.remoteIOUnit, ! ! ! ! ! ! ! ! ! kAudioUnitProperty_StreamFormat, ! ! ! ! ! ! ! ! ! kAudioUnitScope_Output, ! ! ! ! ! ! ! ! ! busOne, ! ! ! ! ! ! ! ! ! &effectDataFormat, ! ! ! ! ! ! ! ! ! propSize), ! ! ! "couldnt set bus one output format"); Then repeat AudioUnitSetProperty() for input scope / bus 0Monday, March 25, 13
  49. 49. AUNewTimePitch • New in iOS 6! • Allows you to change pitch independent of time, or time independent of pitch • How do you use it?Monday, March 25, 13
  50. 50. AUTimePitch ! AudioComponentDescription effectcd = {0}; ! effectcd.componentType = kAudioUnitType_FormatConverter; ! effectcd.componentSubType = kAudioUnitSubType_NewTimePitch; ! effectcd.componentManufacturer = kAudioUnitManufacturer_Apple; ! ! AUNode effectNode; ! CheckError(AUGraphAddNode(self.auGraph, ! ! ! ! ! ! ! &effectcd, ! ! ! ! ! ! ! &effectNode), ! ! ! "couldnt get effect node [time/pitch]"); Notice the type is AUFormatConverter, not AUEffectMonday, March 25, 13
  51. 51. AudioUnitParameters.h // Parameters for AUNewTimePitch enum { ! ! // Global, rate, 1/32 -> 32.0, 1.0 ! kNewTimePitchParam_Rate! ! ! ! ! ! = ! 0, ! ! // Global, Cents, -2400 -> 2400, 1.0 ! kNewTimePitchParam_Pitch! ! ! ! ! ! = 1, ! ! // Global, generic, 3.0 -> 32.0, 8.0 ! kNewTimePitchParam_Overlap! ! ! ! ! ! = 4, ! ! // Global, Boolean, 0->1, 1 ! kNewTimePitchParam_EnablePeakLocking! ! ! = 6 }; This is the entire documentation for the AUNewTimePitch parametersMonday, March 25, 13
  52. 52. AUNewTimePitch parameters • Rate: kNewTimePitchParam_Rate takes a Float32 rate from 1/32 speed to 32x speed. • Use powers of 2: 1/32, 1/16, …, 2, 4, 8… • Pitch: kNewTimePitchParam_Pitch takes a Float32 representing cents, meaning 1/100 of a musical semitoneMonday, March 25, 13
  53. 53. Pitch shifting • Pitch can vary, time does not • Suitable for real-time sources, such as audio captureMonday, March 25, 13
  54. 54. Demo: Pitch Shift New in iOS 6!Monday, March 25, 13
  55. 55. Rate shifting • Rate can vary, pitch does not • Think of 1.5x and 2x speed modes in Podcasts app • Not suitable for real-time sources, as data will be consumed faster. Files work well. • Sources must be able to map time systems with kAudioUnitProperty_InputSamplesInOutputMonday, March 25, 13
  56. 56. Demo: Rate Shift New in iOS 6!Monday, March 25, 13
  57. 57. AUSplitter AUSomethingElse AUSplitter AUSomethingElse New in iOS 6!Monday, March 25, 13
  58. 58. AUMatrixMixer AUSomethingElse AUSomethingElse AUSomethingElse AUMatrixMixer AUSomethingElse AUSomethingElse New in iOS 6!Monday, March 25, 13
  59. 59. Audio Queues (and the APIs that help them)Monday, March 25, 13
  60. 60. AudioQueue • Easier than AURemoteIO - provide data when you want to, less time pressure, can accept or provide compressed formats (MP3, AAC) • Recording queue - receive buffers of captured audio in a callback • Play-out queue - enqueue buffers of audio to play, optionally refill in a callbackMonday, March 25, 13
  61. 61. AudioQueue 2 1 0Monday, March 25, 13
  62. 62. Common AQ scenarios • File player - Read from file and “prime” queue buffers, start queue, when called back with used buffer, refill from next part of file • Synthesis - Maintain state in your own code, write raw samples into buffers during callbacksMonday, March 25, 13
  63. 63. Web Radio • Project from CocoaConfs Columbus, Portland, and Raleigh • Use Audio File Stream Services to pick out audio data from a network stream • Enqueue these packets as new AQ buffers • Dispose used buffers in callbackMonday, March 25, 13
  64. 64. Parsing web radioMonday, March 25, 13
  65. 65. Parsing web radio NSURLConnection delivers NSData buffers, containing audio and framing info. We pass it to NSData NSData Audio File Services. Packets Packets Packets Packets PacketsMonday, March 25, 13
  66. 66. Parsing web radio NSURLConnection delivers NSData buffers, containing audio and framing info. We pass it to NSData NSData Audio File Services. Packets Packets Packets Packets Packets Packets Packets Audio File Services calls us back with parsed packets of audio data. Packets Packets PacketsMonday, March 25, 13
  67. 67. Parsing web radio NSURLConnection delivers NSData buffers, containing audio and framing info. We pass it to NSData NSData Audio File Services. Packets Packets Packets Packets Packets Packets Packets Audio File Services calls us back with parsed packets of audio data. Packets Packets Packets We create an AudioQueueBuffer Packets Packets with those packets and enqueue it Packets 2 Packets 1 0 for play-out. Packets PacketsMonday, March 25, 13
  68. 68. A complex thing! • What if we want to see that data after it’s been decoded to PCM and is about to be played? • e.g., spectrum analysis, effects, visualizers • AudioQueue design is “fire-and-forget”Monday, March 25, 13
  69. 69. AudioQueue Tap! http://www.last.fm/music/Spinal+TapMonday, March 25, 13
  70. 70. AudioQueueProcessingTap • Set as a property on the Audio Queue • Calls back to your function with decoded (PCM) audio data • Three types: pre- or post- effects (that the AQ performs), or siphon. First two can modify the data. • Only documentation is in AudioQueue.hMonday, March 25, 13
  71. 71. Creating an AQ Tap ! ! // create the tap ! ! UInt32 maxFrames = 0; ! ! AudioStreamBasicDescription tapFormat = {0}; ! ! AudioQueueProcessingTapRef tapRef; ! ! CheckError(AudioQueueProcessingTapNew(audioQueue, ! ! ! ! ! ! ! ! ! ! ! tapProc, ! ! ! ! ! ! ! ! ! ! ! (__bridge void *)(player), ! ! ! ! ! ! ! ! ! ! ! kAudioQueueProcessingTap_PreEffects, ! ! ! ! ! ! ! ! ! ! ! &maxFrames, ! ! ! ! ! ! ! ! ! ! ! &tapFormat, ! ! ! ! ! ! ! ! ! ! ! &tapRef), ! ! ! ! "couldnt create AQ tap"); Notice that you receive maxFrames and tapFormat. These do not appear to be settable.Monday, March 25, 13
  72. 72. AQ Tap Proc void tapProc (void * inClientData, ! ! ! AudioQueueProcessingTapRef inAQTap, ! ! ! UInt32 inNumberFrames, ! ! ! AudioTimeStamp * ioTimeStamp, ! ! ! UInt32 * ioFlags, ! ! ! UInt32 * outNumberFrames, ! ! ! AudioBufferList * ioData) { ! CCFWebRadioPlayer *player = (__bridge CCFWebRadioPlayer*) inClientData; ! UInt32 getSourceFlags = 0; ! UInt32 getSourceFrames = 0; ! AudioQueueProcessingTapGetSourceAudio(inAQTap, ! ! ! ! ! ! ! ! ! ! inNumberFrames, ! ! ! ! ! ! ! ! ! ! ioTimeStamp, ! ! ! ! ! ! ! ! ! ! &getSourceFlags, ! ! ! ! ! ! ! ! ! ! &getSourceFrames, ! ! ! ! ! ! ! ! ! ! ioData); // then do something with ioData // ...Monday, March 25, 13
  73. 73. So what should we do with the audio?Monday, March 25, 13
  74. 74. So what should we do with the audio? Let’s apply our pitch-shift effectMonday, March 25, 13
  75. 75. Shouldn’t this work? AUEffectMonday, March 25, 13
  76. 76. Shouldn’t this work? AUEffect AudioUnitRender()Monday, March 25, 13
  77. 77. AudioUnitRender() • Last argument is an AudioBufferList, whose AudioBuffer members have mData pointers • If mData != NULL, audio unit does its thing with those samples • If mData == NULL, audio data pulls from whatever it’s connected to • So we just call with AudioBufferList ioData we got from tap callback, right?Monday, March 25, 13
  78. 78. Psych! • AQ tap provides data as signed ints • Effect units only work with floating point • We need to do an on-the-spot format conversionMonday, March 25, 13
  79. 79. invalidname’s convert- and-effect recipe OSStatus converterInputRenderCallback (void *inRefCon, AudioUnitRenderActionFlags *ioActionFlags, const AudioTimeStamp *inTimeStamp, UInt32 inBusNumber, UInt32 inNumberFrames, AudioBufferList * ioData) { CCFWebRadioPlayer *player = (__bridge CCFWebRadioPlayer*) inRefCon; // read from buffer ioData->mBuffers[0].mData = player.preRenderData; return noErr; } AUConverter AUEffect AUConverter AUGenericOutput Note: red arrows are float format, yellow arrows are intMonday, March 25, 13
  80. 80. How it works • AUGraph: AUConverter → AUEffect → AUConverter → AUGenericOutput • Top AUConverter is connected to a render callback functionMonday, March 25, 13
  81. 81. The trick! • Copy mData pointer to a state variable and NULL it in ioData • Call AudioQueueRender() on output unit. The NULL makes it pull from the graph. • Top of the graph pulls on render callback, which gives it back the mData we copied off.Monday, March 25, 13
  82. 82. Yes, really This is the rest of tapProc() ! // copy off the ioData so the graph can read from it // in render callback ! player.preRenderData = ioData->mBuffers[0].mData; ! ioData->mBuffers[0].mData = NULL; ! ! OSStatus renderErr = noErr; ! AudioUnitRenderActionFlags actionFlags = 0; ! renderErr = AudioUnitRender(player.genericOutputUnit, ! ! ! ! ! ! ! ! &actionFlags, ! ! ! ! ! ! ! ! player.renderTimeStamp, ! ! ! ! ! ! ! ! 0, ! ! ! ! ! ! ! ! inNumberFrames, ! ! ! ! ! ! ! ! ioData); ! NSLog (@"AudioUnitRender, renderErr = %ld",renderErr); }Monday, March 25, 13
  83. 83. Yes, really This is the render callback that supplies data to the int→float converter OSStatus converterInputRenderCallback (void *inRefCon, ! ! ! ! ! ! ! ! ! AudioUnitRenderActionFlags *ioActionFlags, ! ! ! ! ! ! ! ! ! const AudioTimeStamp *inTimeStamp, ! ! ! ! ! ! ! ! ! UInt32 inBusNumber, ! ! ! ! ! ! ! ! ! UInt32 inNumberFrames, ! ! ! ! ! ! ! ! ! AudioBufferList * ioData) { ! CCFWebRadioPlayer *player = (__bridge CCFWebRadioPlayer*) inRefCon; ! ! // read from buffer ! ioData->mBuffers[0].mData = player.preRenderData; ! return noErr; }Monday, March 25, 13
  84. 84. Demo: AQ Tap + AUNewTimePitch New in iOS 6!Monday, March 25, 13
  85. 85. Monday, March 25, 13
  86. 86. Other new stuffMonday, March 25, 13
  87. 87. Multi-Route • Ordinarily, one input or output is active: earpiece, speaker, headphones, dock- connected device • “Last in wins” • With AV Session “multi-route” category, you can use several at once • WWDC 2012 session 505Monday, March 25, 13
  88. 88. Utility classes moved again • C++ utilities, including the CARingBuffer • < Xcode 4.3, installed into /Developer • Xcode 4.3-4.4, optional download from developer.apple.com • ≧ Xcode 4.5, sample code project “Core Audio Utility Classes”Monday, March 25, 13
  89. 89. Takeaways • Core Audio fundamentals never change • New stuff is added as properties, typedefs, enums, etc. • Watch the SDK API diffs document to find the new stuff • Hope you like header files and experimentationMonday, March 25, 13
  90. 90. Q&A • Slides will be posted to slideshare.net/ invalidname • Code will be linked from there and my blog • Watch CocoaConf glassboard, @invalidname on Twitter/ADN, or [Time code]; blog for announcement • Thanks!Monday, March 25, 13

×