Core Audio Cranks It Up                                  Chris Adamson • @invalidname                           Voices Tha...
What Well Cover               • Core Audio: what it is and why its there               • Audio Units and graphs           ...
Core Audio               • Low-level, low-latency, professional audio                 processing API, ported from OS X to ...
Whats in the box               • #import AudioToolbox, possibly also AudioUnit,                 CoreMIDI               • H...
Audio Units               • Process audio in some way: generate, apply                 affects, mix, hardware I/O, etc.   ...
Audio Unit types               • I/O: microphone input / headphone output               • Music/generators: create sound  ...
Tuesday, November 15, 11
Pull               • Audio flows through units via a "pull" metaphor               • Typically the I/O unit, but can be an ...
Why pull?               • Caller, often an I/O thread, is responsible for                 knowing when it needs new buffer...
I/O units               • Abstraction over audio hardware               • AURemoteIO for local use, echo-cancelling       ...
Tuesday, November 15, 11
Tuesday, November 15, 11
Demo         VTM AU Input  http://dl.dropbox.com/u/12216224/conferences/vtm11/VTMAUGraphDemo.zipTuesday, November 15, 11
Adding IO Unit Node      NewAUGraph(&_auGraph);      AudioComponentDescription compDesc = {0};      compDesc.componentType...
Connect nodes and                              start graph          AUGraphConnectNodeInput(_auGraph              ioNode, ...
The problem with                              AUGraphs               • Prior to iOS 5, AUGraphs were boring:              ...
iOS 5 cranks it up               • Generators: AUFilePlayer and AUSampler               • Effects: Reverb, distortion box,...
AUFilePlayer               • Generator unit that produces sound from a file               • Prior to this, playing a file in...
AUFilePlayer properties               • kAudioUnitProperty_ScheduledFileIDs - indicates                 which files to play...
Getting some beats                           Live Edgy Drum Kit 42.aiffTuesday, November 15, 11
Mixing               • AUMultichannelMixer - Mixes multiple input                 buses into one output busTuesday, Novemb...
Demo         VTM AU Mix  http://dl.dropbox.com/u/12216224/conferences/vtm11/VTMAUGraphDemo.zipTuesday, November 15, 11
More fun with mixers               • Set volume for an input or output bus with                 kMultiChannelMixerParam_Vo...
Also new: MIDI               • Core MIDI added in iOS 4.2               • Provides I/O with MIDI hardware, or MIDI-       ...
MIDI packetsTuesday, November 15, 11
MIDI Messages               • Channel Voice Messages — Note On, Note                 Off, After-touch, Pitch wheel        ...
MIDI hardware on iOS               • MIDI adapters must be "MIDI class                 compliant" (i.e., no drivers)      ...
MIDI set-up    MIDIClientCreate(CFSTR("VTM iOS Demo"), MyMIDINotifyProc,    ! ! callbackContext, &client);    MIDIInputPor...
So what do we do in                            MyMIDIReadProc?               • Prior to iOS 5, there was no API that actua...
AUSampler               • Music unit that starts with a source clip and                 pitch-shifts to create different t...
http://developer.apple.com/library/mac/#technotes/tn2283/_index.htmlTuesday, November 15, 11
Orchestral String Section 02-cropped.aifTuesday, November 15, 11
Orchestral String Section 02-cropped.aifTuesday, November 15, 11
/Developer/Applications/Audio/AU LabTuesday, November 15, 11
Tuesday, November 15, 11
Creating an .aupresetTuesday, November 15, 11
Tuesday, November 15, 11
Tuesday, November 15, 11
Tuesday, November 15, 11
Tuesday, November 15, 11
Loading .aupreset (1/3)   NSString *filePath = [[NSBundle mainBundle]   ! pathForResource: AU_SAMPLER_PRESET_FILE   ! ofTy...
Loading .aupreset (2/3)      // convert this into a property list      CFPropertyListFormat presetPlistFormat = {0};      ...
Loading .aupreset (3/3)          // set this plist as the          // kAudioUnitProperty_ClassInfo on _auSampler          ...
Now what?               • We have configured Core MIDI to call                 MyMIDIReadProc on incoming MIDI events      ...
Tuesday, November 15, 11
Tuesday, November 15, 11
MusicDevice.h               • Small API to deliver MIDI events to instrument units               • Not in Xcode documentat...
Delivering MIDI event    MIDIPacket *packet = (MIDIPacket *)pktlist->packet;!    for (int i=0; i < pktlist->numPackets; i+...
Demo         VTM AU Mix (w/MIDI)  http://dl.dropbox.com/u/12216224/conferences/vtm11/VTMAUGraphDemo.zipTuesday, November 1...
Effects               • iOS 5 first version to offer multiple audio effects               • Many effects use floating-point ...
Two sample effects               • Distortion effect - audio "fuzzing" effect, takes                 no parameters        ...
Tuesday, November 15, 11
Configuring                           AULowPassFilter           Float32 lowPassCutoffFrequency = 800.0;           AudioUnit...
Demo         VTM AU Effects  http://dl.dropbox.com/u/12216224/conferences/vtm11/VTMAUGraphDemo.zipTuesday, November 15, 11
Bypassing units                           UInt32 bypassed =                           ! self.effectEnabledSwitch.on ? 0 : ...
Recap               • Audio Units are greatly enhanced in iOS 5               • Real-time mixing and effects on mic input,...
Resources               • http://lists.apple.com/mailman/listinfo2/coreaudio-                 api               • https://...
Q&A                   http://my.safaribooksonline.com/book/audio/                                  9780321636973Tuesday, N...
Upcoming SlideShare
Loading in …5
×

Core Audio Cranks It Up

8,455 views

Published on

Core Audio, the only media framework available since day one of the public iPhone SDK, offers extremely low latency and powerful access to the device's audio processing system... assuming you can handle what's renowned as one of the hardest APIs on the platform. In iOS 5, Core Audio gets even better, with great new features that had previous been burdensome, if not impossible, to develop on your own. Once the iOS 5 NDA drops, the shiny new bits will be available to all, and this talk will be one of your first chances to learn how they work. Attendees will learn the basics of Core Audio -- the engine APIs that process sound (Audio Queue, Audio Units, and OpenAL) and the helper APIs that get samples into and out of them -- and then look where iOS 5 fills in some of the holes that have existed up to now.

Published in: Technology, Business
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
8,455
On SlideShare
0
From Embeds
0
Number of Embeds
357
Actions
Shares
0
Downloads
0
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide

Core Audio Cranks It Up

  1. 1. Core Audio Cranks It Up Chris Adamson • @invalidname Voices That Matter: iOS Developer Conference Nov. 13, 2011 • Boston, MATuesday, November 15, 11
  2. 2. What Well Cover • Core Audio: what it is and why its there • Audio Units and graphs • Fun with I/O • Special effects • Music with MIDITuesday, November 15, 11
  3. 3. Core Audio • Low-level, low-latency, professional audio processing API, ported from OS X to iPhone OS. • Legendarily difficult: see Mike Ashs "Why Core Audio Is Hard" (Oct. 2006) • Consists of three audio engines and various helpersTuesday, November 15, 11
  4. 4. Whats in the box • #import AudioToolbox, possibly also AudioUnit, CoreMIDI • Helper APIs — File I/O, format conversion, stream parsing, MIDI events, sessions • Audio Engine APIs — Audio Units, Audio Queues, OpenAL • Queues and OpenAL implemented atop unitsTuesday, November 15, 11
  5. 5. Audio Units • Process audio in some way: generate, apply affects, mix, hardware I/O, etc. • Arbitrary number of inputs and outputs (usually one each) • Rare case: process a buffer of samples in place • Common case: connect units in a "graph"Tuesday, November 15, 11
  6. 6. Audio Unit types • I/O: microphone input / headphone output • Music/generators: create sound • Mixers/effects: process soundTuesday, November 15, 11
  7. 7. Tuesday, November 15, 11
  8. 8. Pull • Audio flows through units via a "pull" metaphor • Typically the I/O unit, but can be an app- created unit • Pull begins with a call to AudioUnitRender() • Last unit pulls audio from upstream unit(s), or from "render callbacks" to application codeTuesday, November 15, 11
  9. 9. Why pull? • Caller, often an I/O thread, is responsible for knowing when it needs new buffers • Push may seem more sensible, but ends up being burdensome • OpenAL model: poll to see if sources need new buffers and, if so, push them inTuesday, November 15, 11
  10. 10. I/O units • Abstraction over audio hardware • AURemoteIO for local use, echo-cancelling AUVoiceProcessingIO for VOIP • Can only have one per graph • Special meaning for buses: 0 is output to headphones/speakers, 1 is input from microphoneTuesday, November 15, 11
  11. 11. Tuesday, November 15, 11
  12. 12. Tuesday, November 15, 11
  13. 13. Demo VTM AU Input http://dl.dropbox.com/u/12216224/conferences/vtm11/VTMAUGraphDemo.zipTuesday, November 15, 11
  14. 14. Adding IO Unit Node NewAUGraph(&_auGraph); AudioComponentDescription compDesc = {0}; compDesc.componentType = kAudioUnitType_Output; compDesc.componentSubType = ! kAudioUnitSubType_RemoteIO; compDesc.componentManufacturer = ! kAudioUnitManufacturer_Apple; // adds a node with above description to the graph AUNode ioNode; AUGraphAddNode(self.auGraph, &compDesc, &ioNode); A subsequent call to AUGraphNodeInfo() gets an AudioUnit from the AUNodeTuesday, November 15, 11
  15. 15. Connect nodes and start graph AUGraphConnectNodeInput(_auGraph ioNode, 1, mixerNode, 0); AUGraphInitialize(_auGraph) AUGraphStart(_auGraph);!Tuesday, November 15, 11
  16. 16. The problem with AUGraphs • Prior to iOS 5, AUGraphs were boring: • No generator units: only input was AURemoteIO (or synthesized audio in a render callback) • Only one effect unit (AUiPodEQ)Tuesday, November 15, 11
  17. 17. iOS 5 cranks it up • Generators: AUFilePlayer and AUSampler • Effects: Reverb, distortion box, high/low pass filters, high/low shelf filters, dynamics processor, N-band EQTuesday, November 15, 11
  18. 18. AUFilePlayer • Generator unit that produces sound from a file • Prior to this, playing a file in an AUGraph was unduly burdensome • Render callbacks cannot perform blocking calls like file I/O, so youd read from file in one thread, put samples in a ring buffer, and have callback pull from buffer • We have an OSX example of this in book. Ouch.Tuesday, November 15, 11
  19. 19. AUFilePlayer properties • kAudioUnitProperty_ScheduledFileIDs - indicates which files to play • kAudioUnitProperty_ScheduledFileRegion - struct describing what portion of file to play • kAudioUnitProperty_ScheduledFilePrime - optionally preload audio frames for performance • kAudioUnitProperty_ScheduleStartTimeStamp - when to start playing (-1 for next audio render cycle)Tuesday, November 15, 11
  20. 20. Getting some beats Live Edgy Drum Kit 42.aiffTuesday, November 15, 11
  21. 21. Mixing • AUMultichannelMixer - Mixes multiple input buses into one output busTuesday, November 15, 11
  22. 22. Demo VTM AU Mix http://dl.dropbox.com/u/12216224/conferences/vtm11/VTMAUGraphDemo.zipTuesday, November 15, 11
  23. 23. More fun with mixers • Set volume for an input or output bus with kMultiChannelMixerParam_Volume • This is a parameter, not a property • Also read-only pre-/post- peak/average levels for creating level meters (must be enabled first)Tuesday, November 15, 11
  24. 24. Also new: MIDI • Core MIDI added in iOS 4.2 • Provides I/O with MIDI hardware, or MIDI- over-WiFi • Hardware access is via Camera Connection Kit and MIDI-to-USB adapterTuesday, November 15, 11
  25. 25. MIDI packetsTuesday, November 15, 11
  26. 26. MIDI Messages • Channel Voice Messages — Note On, Note Off, After-touch, Pitch wheel • High nybble of status is command, low nybble is channel number • Channel Mode, System Messages • Various extensions to the spec over the years http://www.midi.org/techspecs/midimessages.phpTuesday, November 15, 11
  27. 27. MIDI hardware on iOS • MIDI adapters must be "MIDI class compliant" (i.e., no drivers) • Dock connector / CCK provides minuscule power, not enough to power some MIDI adapters • Powered adapters generally work • Otherwise check http://www.iosmidi.com/ devicesTuesday, November 15, 11
  28. 28. MIDI set-up MIDIClientCreate(CFSTR("VTM iOS Demo"), MyMIDINotifyProc, ! ! callbackContext, &client); MIDIInputPortCreate(client, CFSTR("Input port"), ! ! MyMIDIReadProc, callbackContext, &inPort); unsigned long sourceCount = MIDIGetNumberOfSources(); for (int i = 0; i < sourceCount; ++i) { ! MIDIEndpointRef src = MIDIGetSource(i); ! CFStringRef endpointName = NULL; ! MIDIObjectGetStringProperty(src, kMIDIPropertyName, ! ! &endpointName); ! char endpointNameC[255]; ! CFStringGetCString(endpointName, endpointNameC, ! ! 255, kCFStringEncodingUTF8); ! printf(" source %d: %sn", i, endpointNameC); ! MIDIPortConnectSource(inPort, src, NULL); }Tuesday, November 15, 11
  29. 29. So what do we do in MyMIDIReadProc? • Prior to iOS 5, there was no API that actually did anything with MIDI events • On OSX, Core Audio defines music units, which play sounds in response to MIDI events • iOS 5 and Lion add the AUSamplerTuesday, November 15, 11
  30. 30. AUSampler • Music unit that starts with a source clip and pitch-shifts to create different tones • Can use DLS or SoundFont 2 files as source • Can also create your own sample files with AULabTuesday, November 15, 11
  31. 31. http://developer.apple.com/library/mac/#technotes/tn2283/_index.htmlTuesday, November 15, 11
  32. 32. Orchestral String Section 02-cropped.aifTuesday, November 15, 11
  33. 33. Orchestral String Section 02-cropped.aifTuesday, November 15, 11
  34. 34. /Developer/Applications/Audio/AU LabTuesday, November 15, 11
  35. 35. Tuesday, November 15, 11
  36. 36. Creating an .aupresetTuesday, November 15, 11
  37. 37. Tuesday, November 15, 11
  38. 38. Tuesday, November 15, 11
  39. 39. Tuesday, November 15, 11
  40. 40. Tuesday, November 15, 11
  41. 41. Loading .aupreset (1/3) NSString *filePath = [[NSBundle mainBundle] ! pathForResource: AU_SAMPLER_PRESET_FILE ! ofType:@"aupreset"]; CFURLRef presetURL = ! (__bridge CFURLRef) [NSURL fileURLWithPath:filePath]; // load preset file into a CFDataRef CFDataRef presetData = NULL; SInt32 errorCode = noErr; Boolean gotPresetData = ! CFURLCreateDataAndPropertiesFromResource( ! ! kCFAllocatorSystemDefault, presetURL, ! ! &presetData, NULL, NULL, &errorCode);Tuesday, November 15, 11
  42. 42. Loading .aupreset (2/3) // convert this into a property list CFPropertyListFormat presetPlistFormat = {0}; CFErrorRef presetPlistError = NULL; CFPropertyListRef presetPlist = CFPropertyListCreateWithData( ! kCFAllocatorSystemDefault, presetData, ! kCFPropertyListImmutable,&presetPlistFormat, ! &presetPlistError);Tuesday, November 15, 11
  43. 43. Loading .aupreset (3/3) // set this plist as the // kAudioUnitProperty_ClassInfo on _auSampler if (presetPlist) { ! ! AudioUnitSetProperty( ! ! ! ! self.auSampler, ! ! ! ! kAudioUnitProperty_ClassInfo, ! ! ! ! kAudioUnitScope_Global, ! ! ! ! 0, ! ! ! ! &presetPlist, ! ! ! ! sizeof(presetPlist); }Tuesday, November 15, 11
  44. 44. Now what? • We have configured Core MIDI to call MyMIDIReadProc on incoming MIDI events • We have configured the AUSampler audio unit with a sampled sound • We need to deliver the MIDI events to the AUSampler unitTuesday, November 15, 11
  45. 45. Tuesday, November 15, 11
  46. 46. Tuesday, November 15, 11
  47. 47. MusicDevice.h • Small API to deliver MIDI events to instrument units • Not in Xcode documentation. Check out the header file • Only 4 functions • MusicDeviceMIDIEvent() sends status, data1, data2 to a MusicDeviceComponent (i.e., an instrument Audio Unit)Tuesday, November 15, 11
  48. 48. Delivering MIDI event MIDIPacket *packet = (MIDIPacket *)pktlist->packet;! for (int i=0; i < pktlist->numPackets; i++) { ! Byte midiStatus = packet->data[0]; ! Byte midiCommand = midiStatus >> 4; ! // is it a note-on or note-off ! if ((midiCommand == 0x09) || (midiCommand == 0x08)) { ! ! Byte note = packet->data[1] & 0x7F; ! ! Byte velocity = packet->data[2] & 0x7F; ! ! // send to augraph ! ! MusicDeviceMIDIEvent (myVC.auSampler, midiStatus, note, velocity, 0); ! } ! packet = MIDIPacketNext(packet); }Tuesday, November 15, 11
  49. 49. Demo VTM AU Mix (w/MIDI) http://dl.dropbox.com/u/12216224/conferences/vtm11/VTMAUGraphDemo.zipTuesday, November 15, 11
  50. 50. Effects • iOS 5 first version to offer multiple audio effects • Many effects use floating-point samples, rather than ints or fixed-point; requires adjusting the stream formats of units connected to them (or in-line AUConverters) • Adjust with parameters (see AudioUnitParameters.h)Tuesday, November 15, 11
  51. 51. Two sample effects • Distortion effect - audio "fuzzing" effect, takes no parameters • Low pass filter - Cuts off frequencies above a given frequencyTuesday, November 15, 11
  52. 52. Tuesday, November 15, 11
  53. 53. Configuring AULowPassFilter Float32 lowPassCutoffFrequency = 800.0; AudioUnitSetParameter(_auLowPassFilter, ! ! ! ! ! kLowPassParam_CutoffFrequency, ! ! ! ! ! kAudioUnitScope_Global, ! ! ! ! ! 0, ! ! ! ! ! lowPassCutoffFrequency, ! ! ! ! ! 0);Tuesday, November 15, 11
  54. 54. Demo VTM AU Effects http://dl.dropbox.com/u/12216224/conferences/vtm11/VTMAUGraphDemo.zipTuesday, November 15, 11
  55. 55. Bypassing units UInt32 bypassed = ! self.effectEnabledSwitch.on ? 0 : 1; AudioUnitSetProperty (self.audioUnit, ! ! kAudioUnitProperty_BypassEffect, ! ! kAudioUnitScope_Global, ! ! 0, ! ! &bypassed, ! ! sizeof(bypassed));Tuesday, November 15, 11
  56. 56. Recap • Audio Units are greatly enhanced in iOS 5 • Real-time mixing and effects on mic input, pitch-shifted samples, and file playback • Best choice for serious audio productionTuesday, November 15, 11
  57. 57. Resources • http://lists.apple.com/mailman/listinfo2/coreaudio- api • https://devforums.apple.com/community/ios/ graphics/audio • Blogs: • Mine: http://www.subfurther.com/blog • Michael Tyson: http://atastypixel.com/blog/Tuesday, November 15, 11
  58. 58. Q&A http://my.safaribooksonline.com/book/audio/ 9780321636973Tuesday, November 15, 11

×