• Like
  • Save
Core Audio in iOS 6 (CocoaConf DC, March 2013)
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Core Audio in iOS 6 (CocoaConf DC, March 2013)

  • 705 views
Published

 

Published in Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
705
On SlideShare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
0
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Core Audio in iOS 6 Chris Adamson • @invalidname CocoaConf DC March 23, 2013 Sides and code available on my blog: http://www.subfurther.com/blogMonday, March 25, 13
  • 2. Plug!Monday, March 25, 13
  • 3. The Reviews Are In!Monday, March 25, 13
  • 4. The Reviews Are In!Monday, March 25, 13
  • 5. The Reviews Are In!Monday, March 25, 13
  • 6. The Reviews Are In!Monday, March 25, 13
  • 7. Legitimate copies! • Amazon (paper or Kindle) • Barnes & Noble (paper or Nook) • Apple (iBooks) • Direct from InformIT (paper, eBook [.epub + .mobi + .pdf], or Bundle) • 35% off with code COREAUDIO3174Monday, March 25, 13
  • 8. What You’ll Learn • What Core Audio does and doesn’t do • When to use and not use it • What’s new in Core Audio for iOS 6Monday, March 25, 13
  • 9. Monday, March 25, 13
  • 10. Simple things should be simple, complex things should be possible. –Alan KayMonday, March 25, 13
  • 11. AV Foundation, Media Player Simple things should be simple, complex things should be possible. –Alan KayMonday, March 25, 13
  • 12. AV Foundation, Media Player Simple things should be simple, complex things should be possible. –Alan Kay Core AudioMonday, March 25, 13
  • 13. Core Audio • Low-level C framework for processing audio • Capture, play-out, real-time or off-line processing • The “complex things should be possible” part of audio on OS X and iOSMonday, March 25, 13
  • 14. Chris’ CA Taxonomy • Engines: process streams of audio • Capture, play-out, mixing, effects processing • Helpers: deal with formats, encodings, etc. • File I/O, stream I/O, format conversion, iOS “session” managementMonday, March 25, 13
  • 15. Helpers: Audio File • Read from / write to multiple audio file types (.aiff, .wav, .caf, .m4a, .mp3) in a content-agnostic way • Get metadata (data format, duration, iTunes/ID3 info)Monday, March 25, 13
  • 16. Helpers: Audio File Stream • Read audio from non-random-access source like a network stream • Discover encoding and encapsulation on the fly, then deliver audio packets to client applicationMonday, March 25, 13
  • 17. Helpers: Converters • Convert buffers of audio to and from different encodings • One side must be in an uncompressed format (i.e., Linear PCM)Monday, March 25, 13
  • 18. Helpers: ExtAudioFile • Combine file I/O and format conversion • Read a compressed file into PCM buffers • Write PCM buffers into a compressed fileMonday, March 25, 13
  • 19. Helpers: Audio Session • iOS-only API to negotiate use of audio resources with the rest of the system • Deetermine whether your app mixes with other apps’ audio, honors ring/silent switch, can play in background, etc. • Gets notified of audio interruptions • See also AVAudioSessionMonday, March 25, 13
  • 20. Engines: Audio Units • Low-latency (~10ms) processing of capture/play-out audio data • Effects, mixing, etc. • Connect units manually or via an AUGraph • Much more on this topic momentarily…Monday, March 25, 13
  • 21. Engines: Audio Queue • Convenience API for recording or play-out, built atop audio units • Rather than processing on-demand and on Core Audio’s thread, your callback provides or receives buffers of audio (at whatever size is convenient to you) • Higher latency, naturally • Supports compressed formats (MP3, AAC)Monday, March 25, 13
  • 22. Engines: Open AL • API for 3D spatialized audio, implemented atop audio units • Set a source’s properties (x/y/z coordinates, orientation, audio buffer, etc.), OpenAL renders what it sounds like to the listener from that locationMonday, March 25, 13
  • 23. Engines and Helpers • Audio Units • Audio File • Audio Queue • Audio File Stream • Open AL • Audio Converter • ExtAudioFile • Audio SessionMonday, March 25, 13
  • 24. Audio UnitsMonday, March 25, 13
  • 25. Audio Unit AUSomethingMonday, March 25, 13
  • 26. Types of Audio Units • Output (which also do input) • Generator • Converter • Effect • Mixer • MusicMonday, March 25, 13
  • 27. Pull Model AUSomethingMonday, March 25, 13
  • 28. Pull Model AUSomething AudioUnitRender()Monday, March 25, 13
  • 29. Pull Model AUSomethingElse AUSomethingMonday, March 25, 13
  • 30. Buses (aka, Elements) AUSomethingElse AUSomething AUSomethingElseMonday, March 25, 13
  • 31. AUGraph AUSomethingElse AUSomething AUSomethingElseMonday, March 25, 13
  • 32. Render Callbacks OSStatus converterInputRenderCallback (void *inRefCon, AudioUnitRenderActionFlags *ioActionFlags, const AudioTimeStamp *inTimeStamp, UInt32 inBusNumber, UInt32 inNumberFrames, AudioBufferList * ioData) { CCFWebRadioPlayer *player = (__bridge CCFWebRadioPlayer*) inRefCon; // read from buffer ioData->mBuffers[0].mData = player.preRenderData; return noErr; } AUSomething AUSomethingElseMonday, March 25, 13
  • 33. AURemoteIO • Output unit used for play-out, capture • A Core Audio thread repeatedly and automatically calls AudioUnitRender() • Must set EnableIO property to explicitly enable capture and/or play-out • Capture requires setting appropriate AudioSession categoryMonday, March 25, 13
  • 34. Create AURemoteIO CheckError(NewAUGraph(&_auGraph), ! ! "couldnt create au graph"); ! CheckError(AUGraphOpen(_auGraph), ! ! "couldnt open au graph"); ! AudioComponentDescription componentDesc; componentDesc.componentType = kAudioUnitType_Output; componentDesc.componentSubType = kAudioUnitSubType_RemoteIO; componentDesc.componentManufacturer = kAudioUnitManufacturer_Apple; ! AUNode remoteIONode; CheckError(AUGraphAddNode(_auGraph, ! ! ! ! ! ! &componentDesc, ! ! ! ! ! ! &remoteIONode), ! ! "couldnt add remote io node");Monday, March 25, 13
  • 35. Getting an AudioUnit from AUNode ! CheckError(AUGraphNodeInfo(self.auGraph, ! ! ! ! ! ! ! remoteIONode, ! ! ! ! ! ! ! NULL, ! ! ! ! ! ! ! &_remoteIOUnit), ! ! ! "couldnt get remote io unit from node");Monday, March 25, 13
  • 36. AURemoteIO Buses AURemoteIOMonday, March 25, 13
  • 37. AURemoteIO Buses AURemoteIO bus 0 to output H/WMonday, March 25, 13
  • 38. AURemoteIO Buses AURemoteIO bus 0 bus 0 from app to output H/WMonday, March 25, 13
  • 39. AURemoteIO Buses bus 1from input H/W AURemoteIO bus 0 bus 0 from app to output H/WMonday, March 25, 13
  • 40. AURemoteIO Buses bus 1 bus 1from input H/W to app AURemoteIO bus 0 bus 0 from app to output H/WMonday, March 25, 13
  • 41. EnableIO ! UInt32 oneFlag = 1; ! UInt32 busZero = 0; ! CheckError(AudioUnitSetProperty(self.remoteIOUnit, ! ! ! ! ! ! ! ! ! kAudioOutputUnitProperty_EnableIO, ! ! ! ! ! ! ! ! ! kAudioUnitScope_Output, ! ! ! ! ! ! ! ! ! busZero, ! ! ! ! ! ! ! ! ! &oneFlag, ! ! ! ! ! ! ! ! ! sizeof(oneFlag)), ! ! ! "couldnt enable remote io output"); ! UInt32 busOne = 1; ! CheckError(AudioUnitSetProperty(self.remoteIOUnit, ! ! ! ! ! ! ! ! ! kAudioOutputUnitProperty_EnableIO, ! ! ! ! ! ! ! ! ! kAudioUnitScope_Input, ! ! ! ! ! ! ! ! ! busOne, ! ! ! ! ! ! ! ! ! &oneFlag, ! ! ! ! ! ! ! ! ! sizeof(oneFlag)), ! ! ! "couldnt enable remote io input");Monday, March 25, 13
  • 42. Pass Through bus 1 from input H/W AURemoteIO bus 0 to output H/WMonday, March 25, 13
  • 43. Connect In to Out ! UInt32 busZero = 0; ! UInt32 busOne = 1; ! CheckError(AUGraphConnectNodeInput(self.auGraph, ! ! ! ! ! ! ! ! ! remoteIONode, ! ! ! ! ! ! ! ! ! busOne, ! ! ! ! ! ! ! ! ! remoteIONode, ! ! ! ! ! ! ! ! ! busZero), ! ! ! "couldnt connect remote io bus 1 to 0");Monday, March 25, 13
  • 44. Pass-Through with Effect AUEffect bus 1 from input H/W AURemoteIO bus 0 to output H/WMonday, March 25, 13
  • 45. Demo: Delay Effect New in iOS 6!Monday, March 25, 13
  • 46. Creating the AUDelay ! componentDesc.componentType = kAudioUnitType_Effect; ! componentDesc.componentSubType = kAudioUnitSubType_Delay; ! componentDesc.componentManufacturer = kAudioUnitManufacturer_Apple; ! ! AUNode effectNode; ! CheckError(AUGraphAddNode(self.auGraph, ! ! ! ! ! ! ! &componentDesc, ! ! ! ! ! ! ! &effectNode), ! ! ! "couldnt create effect node"); ! AudioUnit effectUnit; ! CheckError(AUGraphNodeInfo(self.auGraph, ! ! ! ! ! ! ! effectNode, ! ! ! ! ! ! ! NULL, ! ! ! ! ! ! ! &effectUnit), ! ! ! "couldnt get effect unit from node");Monday, March 25, 13
  • 47. The problem with effect units • Audio Units available since iPhone OS 2.0 prefer int formats • Effect units arrived with iOS 5 (arm7 era) and only work with float format • Have to set the AUEffect unit’s format on AURemoteIOMonday, March 25, 13
  • 48. Setting formats ! AudioStreamBasicDescription effectDataFormat; ! UInt32 propSize = sizeof (effectDataFormat); ! CheckError(AudioUnitGetProperty(effectUnit, ! ! ! ! ! ! ! ! ! kAudioUnitProperty_StreamFormat, ! ! ! ! ! ! ! ! ! kAudioUnitScope_Output, ! ! ! ! ! ! ! ! ! busZero, ! ! ! ! ! ! ! ! ! &effectDataFormat, ! ! ! ! ! ! ! ! ! &propSize), ! ! ! "couldnt read effect format"); ! CheckError(AudioUnitSetProperty(self.remoteIOUnit, ! ! ! ! ! ! ! ! ! kAudioUnitProperty_StreamFormat, ! ! ! ! ! ! ! ! ! kAudioUnitScope_Output, ! ! ! ! ! ! ! ! ! busOne, ! ! ! ! ! ! ! ! ! &effectDataFormat, ! ! ! ! ! ! ! ! ! propSize), ! ! ! "couldnt set bus one output format"); Then repeat AudioUnitSetProperty() for input scope / bus 0Monday, March 25, 13
  • 49. AUNewTimePitch • New in iOS 6! • Allows you to change pitch independent of time, or time independent of pitch • How do you use it?Monday, March 25, 13
  • 50. AUTimePitch ! AudioComponentDescription effectcd = {0}; ! effectcd.componentType = kAudioUnitType_FormatConverter; ! effectcd.componentSubType = kAudioUnitSubType_NewTimePitch; ! effectcd.componentManufacturer = kAudioUnitManufacturer_Apple; ! ! AUNode effectNode; ! CheckError(AUGraphAddNode(self.auGraph, ! ! ! ! ! ! ! &effectcd, ! ! ! ! ! ! ! &effectNode), ! ! ! "couldnt get effect node [time/pitch]"); Notice the type is AUFormatConverter, not AUEffectMonday, March 25, 13
  • 51. AudioUnitParameters.h // Parameters for AUNewTimePitch enum { ! ! // Global, rate, 1/32 -> 32.0, 1.0 ! kNewTimePitchParam_Rate! ! ! ! ! ! = ! 0, ! ! // Global, Cents, -2400 -> 2400, 1.0 ! kNewTimePitchParam_Pitch! ! ! ! ! ! = 1, ! ! // Global, generic, 3.0 -> 32.0, 8.0 ! kNewTimePitchParam_Overlap! ! ! ! ! ! = 4, ! ! // Global, Boolean, 0->1, 1 ! kNewTimePitchParam_EnablePeakLocking! ! ! = 6 }; This is the entire documentation for the AUNewTimePitch parametersMonday, March 25, 13
  • 52. AUNewTimePitch parameters • Rate: kNewTimePitchParam_Rate takes a Float32 rate from 1/32 speed to 32x speed. • Use powers of 2: 1/32, 1/16, …, 2, 4, 8… • Pitch: kNewTimePitchParam_Pitch takes a Float32 representing cents, meaning 1/100 of a musical semitoneMonday, March 25, 13
  • 53. Pitch shifting • Pitch can vary, time does not • Suitable for real-time sources, such as audio captureMonday, March 25, 13
  • 54. Demo: Pitch Shift New in iOS 6!Monday, March 25, 13
  • 55. Rate shifting • Rate can vary, pitch does not • Think of 1.5x and 2x speed modes in Podcasts app • Not suitable for real-time sources, as data will be consumed faster. Files work well. • Sources must be able to map time systems with kAudioUnitProperty_InputSamplesInOutputMonday, March 25, 13
  • 56. Demo: Rate Shift New in iOS 6!Monday, March 25, 13
  • 57. AUSplitter AUSomethingElse AUSplitter AUSomethingElse New in iOS 6!Monday, March 25, 13
  • 58. AUMatrixMixer AUSomethingElse AUSomethingElse AUSomethingElse AUMatrixMixer AUSomethingElse AUSomethingElse New in iOS 6!Monday, March 25, 13
  • 59. Audio Queues (and the APIs that help them)Monday, March 25, 13
  • 60. AudioQueue • Easier than AURemoteIO - provide data when you want to, less time pressure, can accept or provide compressed formats (MP3, AAC) • Recording queue - receive buffers of captured audio in a callback • Play-out queue - enqueue buffers of audio to play, optionally refill in a callbackMonday, March 25, 13
  • 61. AudioQueue 2 1 0Monday, March 25, 13
  • 62. Common AQ scenarios • File player - Read from file and “prime” queue buffers, start queue, when called back with used buffer, refill from next part of file • Synthesis - Maintain state in your own code, write raw samples into buffers during callbacksMonday, March 25, 13
  • 63. Web Radio • Project from CocoaConfs Columbus, Portland, and Raleigh • Use Audio File Stream Services to pick out audio data from a network stream • Enqueue these packets as new AQ buffers • Dispose used buffers in callbackMonday, March 25, 13
  • 64. Parsing web radioMonday, March 25, 13
  • 65. Parsing web radio NSURLConnection delivers NSData buffers, containing audio and framing info. We pass it to NSData NSData Audio File Services. Packets Packets Packets Packets PacketsMonday, March 25, 13
  • 66. Parsing web radio NSURLConnection delivers NSData buffers, containing audio and framing info. We pass it to NSData NSData Audio File Services. Packets Packets Packets Packets Packets Packets Packets Audio File Services calls us back with parsed packets of audio data. Packets Packets PacketsMonday, March 25, 13
  • 67. Parsing web radio NSURLConnection delivers NSData buffers, containing audio and framing info. We pass it to NSData NSData Audio File Services. Packets Packets Packets Packets Packets Packets Packets Audio File Services calls us back with parsed packets of audio data. Packets Packets Packets We create an AudioQueueBuffer Packets Packets with those packets and enqueue it Packets 2 Packets 1 0 for play-out. Packets PacketsMonday, March 25, 13
  • 68. A complex thing! • What if we want to see that data after it’s been decoded to PCM and is about to be played? • e.g., spectrum analysis, effects, visualizers • AudioQueue design is “fire-and-forget”Monday, March 25, 13
  • 69. AudioQueue Tap! http://www.last.fm/music/Spinal+TapMonday, March 25, 13
  • 70. AudioQueueProcessingTap • Set as a property on the Audio Queue • Calls back to your function with decoded (PCM) audio data • Three types: pre- or post- effects (that the AQ performs), or siphon. First two can modify the data. • Only documentation is in AudioQueue.hMonday, March 25, 13
  • 71. Creating an AQ Tap ! ! // create the tap ! ! UInt32 maxFrames = 0; ! ! AudioStreamBasicDescription tapFormat = {0}; ! ! AudioQueueProcessingTapRef tapRef; ! ! CheckError(AudioQueueProcessingTapNew(audioQueue, ! ! ! ! ! ! ! ! ! ! ! tapProc, ! ! ! ! ! ! ! ! ! ! ! (__bridge void *)(player), ! ! ! ! ! ! ! ! ! ! ! kAudioQueueProcessingTap_PreEffects, ! ! ! ! ! ! ! ! ! ! ! &maxFrames, ! ! ! ! ! ! ! ! ! ! ! &tapFormat, ! ! ! ! ! ! ! ! ! ! ! &tapRef), ! ! ! ! "couldnt create AQ tap"); Notice that you receive maxFrames and tapFormat. These do not appear to be settable.Monday, March 25, 13
  • 72. AQ Tap Proc void tapProc (void * inClientData, ! ! ! AudioQueueProcessingTapRef inAQTap, ! ! ! UInt32 inNumberFrames, ! ! ! AudioTimeStamp * ioTimeStamp, ! ! ! UInt32 * ioFlags, ! ! ! UInt32 * outNumberFrames, ! ! ! AudioBufferList * ioData) { ! CCFWebRadioPlayer *player = (__bridge CCFWebRadioPlayer*) inClientData; ! UInt32 getSourceFlags = 0; ! UInt32 getSourceFrames = 0; ! AudioQueueProcessingTapGetSourceAudio(inAQTap, ! ! ! ! ! ! ! ! ! ! inNumberFrames, ! ! ! ! ! ! ! ! ! ! ioTimeStamp, ! ! ! ! ! ! ! ! ! ! &getSourceFlags, ! ! ! ! ! ! ! ! ! ! &getSourceFrames, ! ! ! ! ! ! ! ! ! ! ioData); // then do something with ioData // ...Monday, March 25, 13
  • 73. So what should we do with the audio?Monday, March 25, 13
  • 74. So what should we do with the audio? Let’s apply our pitch-shift effectMonday, March 25, 13
  • 75. Shouldn’t this work? AUEffectMonday, March 25, 13
  • 76. Shouldn’t this work? AUEffect AudioUnitRender()Monday, March 25, 13
  • 77. AudioUnitRender() • Last argument is an AudioBufferList, whose AudioBuffer members have mData pointers • If mData != NULL, audio unit does its thing with those samples • If mData == NULL, audio data pulls from whatever it’s connected to • So we just call with AudioBufferList ioData we got from tap callback, right?Monday, March 25, 13
  • 78. Psych! • AQ tap provides data as signed ints • Effect units only work with floating point • We need to do an on-the-spot format conversionMonday, March 25, 13
  • 79. invalidname’s convert- and-effect recipe OSStatus converterInputRenderCallback (void *inRefCon, AudioUnitRenderActionFlags *ioActionFlags, const AudioTimeStamp *inTimeStamp, UInt32 inBusNumber, UInt32 inNumberFrames, AudioBufferList * ioData) { CCFWebRadioPlayer *player = (__bridge CCFWebRadioPlayer*) inRefCon; // read from buffer ioData->mBuffers[0].mData = player.preRenderData; return noErr; } AUConverter AUEffect AUConverter AUGenericOutput Note: red arrows are float format, yellow arrows are intMonday, March 25, 13
  • 80. How it works • AUGraph: AUConverter → AUEffect → AUConverter → AUGenericOutput • Top AUConverter is connected to a render callback functionMonday, March 25, 13
  • 81. The trick! • Copy mData pointer to a state variable and NULL it in ioData • Call AudioQueueRender() on output unit. The NULL makes it pull from the graph. • Top of the graph pulls on render callback, which gives it back the mData we copied off.Monday, March 25, 13
  • 82. Yes, really This is the rest of tapProc() ! // copy off the ioData so the graph can read from it // in render callback ! player.preRenderData = ioData->mBuffers[0].mData; ! ioData->mBuffers[0].mData = NULL; ! ! OSStatus renderErr = noErr; ! AudioUnitRenderActionFlags actionFlags = 0; ! renderErr = AudioUnitRender(player.genericOutputUnit, ! ! ! ! ! ! ! ! &actionFlags, ! ! ! ! ! ! ! ! player.renderTimeStamp, ! ! ! ! ! ! ! ! 0, ! ! ! ! ! ! ! ! inNumberFrames, ! ! ! ! ! ! ! ! ioData); ! NSLog (@"AudioUnitRender, renderErr = %ld",renderErr); }Monday, March 25, 13
  • 83. Yes, really This is the render callback that supplies data to the int→float converter OSStatus converterInputRenderCallback (void *inRefCon, ! ! ! ! ! ! ! ! ! AudioUnitRenderActionFlags *ioActionFlags, ! ! ! ! ! ! ! ! ! const AudioTimeStamp *inTimeStamp, ! ! ! ! ! ! ! ! ! UInt32 inBusNumber, ! ! ! ! ! ! ! ! ! UInt32 inNumberFrames, ! ! ! ! ! ! ! ! ! AudioBufferList * ioData) { ! CCFWebRadioPlayer *player = (__bridge CCFWebRadioPlayer*) inRefCon; ! ! // read from buffer ! ioData->mBuffers[0].mData = player.preRenderData; ! return noErr; }Monday, March 25, 13
  • 84. Demo: AQ Tap + AUNewTimePitch New in iOS 6!Monday, March 25, 13
  • 85. Monday, March 25, 13
  • 86. Other new stuffMonday, March 25, 13
  • 87. Multi-Route • Ordinarily, one input or output is active: earpiece, speaker, headphones, dock- connected device • “Last in wins” • With AV Session “multi-route” category, you can use several at once • WWDC 2012 session 505Monday, March 25, 13
  • 88. Utility classes moved again • C++ utilities, including the CARingBuffer • < Xcode 4.3, installed into /Developer • Xcode 4.3-4.4, optional download from developer.apple.com • ≧ Xcode 4.5, sample code project “Core Audio Utility Classes”Monday, March 25, 13
  • 89. Takeaways • Core Audio fundamentals never change • New stuff is added as properties, typedefs, enums, etc. • Watch the SDK API diffs document to find the new stuff • Hope you like header files and experimentationMonday, March 25, 13
  • 90. Q&A • Slides will be posted to slideshare.net/ invalidname • Code will be linked from there and my blog • Watch CocoaConf glassboard, @invalidname on Twitter/ADN, or [Time code]; blog for announcement • Thanks!Monday, March 25, 13