Core Audio Cranks It Up

Core Audio Cranks It Up
Chris Adamson • @invalidname
Voices That Matter: iOS Developer Conference
Nov. 13, 2011 • Boston, MA

Tuesday, November 15, 11

What We'll Cover
• Core Audio: what it is and why it's there

• Audio Units and graphs

• Fun with I/O

• Special effects

• Music with MIDI


Core Audio
• Low-level, low-latency, professional audio
processing API, ported from OS X to iPhone
OS.

• Legendarily difﬁcult: see Mike Ash's "Why
Core Audio Is Hard" (Oct. 2006)

• Consists of three audio engines and various
helpers


What's in the box
• #import AudioToolbox, possibly also AudioUnit,
CoreMIDI

• Helper APIs — File I/O, format conversion,
stream parsing, MIDI events, sessions

• Audio Engine APIs — Audio Units, Audio
Queues, OpenAL

• Queues and OpenAL implemented atop units


Audio Units
• Process audio in some way: generate, apply
affects, mix, hardware I/O, etc.

• Arbitrary number of inputs and outputs (usually
one each)

• Rare case: process a buffer of samples in
place

• Common case: connect units in a "graph"


Audio Unit types

• I/O: microphone input / headphone output

• Music/generators: create sound

• Mixers/effects: process sound


Pull
• Audio ﬂows through units via a "pull" metaphor

• Typically the I/O unit, but can be an app-
created unit

• Pull begins with a call to AudioUnitRender()

• Last unit pulls audio from upstream unit(s), or
from "render callbacks" to application code


Why pull?
• Caller, often an I/O thread, is responsible for
knowing when it needs new buffers

• Push may seem more sensible, but ends up
being burdensome

• OpenAL model: poll to see if sources need
new buffers and, if so, push them in


I/O units
• Abstraction over audio hardware

• AURemoteIO for local use, echo-cancelling
AUVoiceProcessingIO for VOIP

• Can only have one per graph

• Special meaning for buses: 0 is output to
headphones/speakers, 1 is input from
microphone


Demo
VTM AU Input

http://dl.dropbox.com/u/12216224/conferences/vtm11/VTMAUGraphDemo.zip


Adding IO Unit Node
NewAUGraph(&_auGraph);

AudioComponentDescription compDesc = {0};
compDesc.componentType = kAudioUnitType_Output;
compDesc.componentSubType =
! kAudioUnitSubType_RemoteIO;
compDesc.componentManufacturer =
! kAudioUnitManufacturer_Apple;
// adds a node with above description to the graph
AUNode ioNode;
AUGraphAddNode(self.auGraph, &compDesc, &ioNode);

A subsequent call to AUGraphNodeInfo() gets an AudioUnit from
the AUNode

Connect nodes and
start graph

AUGraphConnectNodeInput(_auGraph
ioNode, 1, mixerNode, 0);

AUGraphInitialize(_auGraph)

AUGraphStart(_auGraph);!


The problem with
AUGraphs
• Prior to iOS 5, AUGraphs were boring:

• No generator units: only input was
AURemoteIO (or synthesized audio in a
render callback)

• Only one effect unit (AUiPodEQ)


iOS 5 cranks it up

• Generators: AUFilePlayer and AUSampler

• Effects: Reverb, distortion box, high/low pass
ﬁlters, high/low shelf ﬁlters, dynamics
processor, N-band EQ


AUFilePlayer
• Generator unit that produces sound from a file

• Prior to this, playing a file in an AUGraph was
unduly burdensome

• Render callbacks cannot perform blocking calls
like file I/O, so you'd read from file in one thread,
put samples in a ring buffer, and have callback
pull from buffer

• We have an OSX example of this in book. Ouch.


AUFilePlayer properties
• kAudioUnitProperty_ScheduledFileIDs - indicates
which ﬁles to play

• kAudioUnitProperty_ScheduledFileRegion - struct
describing what portion of ﬁle to play

• kAudioUnitProperty_ScheduledFilePrime - optionally
preload audio frames for performance

• kAudioUnitProperty_ScheduleStartTimeStamp -
when to start playing (-1 for next audio render cycle)


Getting some beats
Live Edgy Drum Kit 42.aiff


Mixing
• AUMultichannelMixer - Mixes multiple input
buses into one output bus


Demo
VTM AU Mix



More fun with mixers

• Set volume for an input or output bus with
kMultiChannelMixerParam_Volume

• This is a parameter, not a property

• Also read-only pre-/post- peak/average levels
for creating level meters (must be enabled ﬁrst)


Also new: MIDI

• Core MIDI added in iOS 4.2

• Provides I/O with MIDI hardware, or MIDI-
over-WiFi

• Hardware access is via Camera Connection
Kit and MIDI-to-USB adapter


MIDI packets


MIDI Messages
• Channel Voice Messages — Note On, Note
Off, After-touch, Pitch wheel

• High nybble of status is command, low nybble
is channel number

• Channel Mode, System Messages

• Various extensions to the spec over the years

http://www.midi.org/techspecs/midimessages.php

MIDI hardware on iOS
• MIDI adapters must be "MIDI class
compliant" (i.e., no drivers)

• Dock connector / CCK provides minuscule
power, not enough to power some MIDI adapters

• Powered adapters generally work

• Otherwise check http://www.iosmidi.com/
devices


MIDI set-up
MIDIClientCreate(CFSTR("VTM iOS Demo"), MyMIDINotifyProc,
! ! callbackContext, &client);
MIDIInputPortCreate(client, CFSTR("Input port"),
! ! MyMIDIReadProc, callbackContext, &inPort);
unsigned long sourceCount = MIDIGetNumberOfSources();
for (int i = 0; i < sourceCount; ++i) {
! MIDIEndpointRef src = MIDIGetSource(i);
! CFStringRef endpointName = NULL;
! MIDIObjectGetStringProperty(src, kMIDIPropertyName,
! ! &endpointName);
! char endpointNameC[255];
! CFStringGetCString(endpointName, endpointNameC,
! ! 255, kCFStringEncodingUTF8);
! printf(" source %d: %sn", i, endpointNameC);
! MIDIPortConnectSource(inPort, src, NULL);
}

So what do we do in
MyMIDIReadProc?
• Prior to iOS 5, there was no API that actually
did anything with MIDI events

• On OSX, Core Audio deﬁnes music units,
which play sounds in response to MIDI events

• iOS 5 and Lion add the AUSampler


AUSampler

• Music unit that starts with a source clip and
pitch-shifts to create different tones

• Can use DLS or SoundFont 2 ﬁles as source

• Can also create your own sample ﬁles with
AULab


http://developer.apple.com/library/mac/#technotes/tn2283/_index.html


Orchestral String Section 02-cropped.aif


/Developer/Applications/Audio/AU Lab


Creating an .aupreset


Loading .aupreset (1/3)
NSString *filePath = [[NSBundle mainBundle]
! pathForResource: AU_SAMPLER_PRESET_FILE
! ofType:@"aupreset"];
CFURLRef presetURL =
! (__bridge CFURLRef) [NSURL fileURLWithPath:filePath];

// load preset file into a CFDataRef
CFDataRef presetData = NULL;
SInt32 errorCode = noErr;
Boolean gotPresetData =
! CFURLCreateDataAndPropertiesFromResource(
! ! kCFAllocatorSystemDefault, presetURL,
! ! &presetData, NULL, NULL, &errorCode);



// convert this into a property list
CFPropertyListFormat presetPlistFormat = {0};
CFErrorRef presetPlistError = NULL;
CFPropertyListRef presetPlist =
CFPropertyListCreateWithData(
! kCFAllocatorSystemDefault, presetData,
! kCFPropertyListImmutable,&presetPlistFormat,
! &presetPlistError);


// set this plist as the
// kAudioUnitProperty_ClassInfo on _auSampler
if (presetPlist) {
! ! AudioUnitSetProperty(
! ! ! ! self.auSampler,
! ! ! ! kAudioUnitProperty_ClassInfo,
! ! ! ! kAudioUnitScope_Global,
! ! ! ! 0,
! ! ! ! &presetPlist,
! ! ! ! sizeof(presetPlist);
}


Now what?
• We have conﬁgured Core MIDI to call
MyMIDIReadProc on incoming MIDI events

• We have conﬁgured the AUSampler audio
unit with a sampled sound

• We need to deliver the MIDI events to the
AUSampler unit


MusicDevice.h
• Small API to deliver MIDI events to instrument units

• Not in Xcode documentation. Check out the
header ﬁle

• Only 4 functions

• MusicDeviceMIDIEvent() sends status, data1,
data2 to a MusicDeviceComponent (i.e., an
instrument Audio Unit)


Delivering MIDI event
MIDIPacket *packet = (MIDIPacket *)pktlist->packet;!
for (int i=0; i < pktlist->numPackets; i++) {
! Byte midiStatus = packet->data[0];
! Byte midiCommand = midiStatus >> 4;
! // is it a note-on or note-off
! if ((midiCommand == 0x09) || (midiCommand == 0x08)) {
! ! Byte note = packet->data[1] & 0x7F;
! ! Byte velocity = packet->data[2] & 0x7F;
! ! // send to augraph
! ! MusicDeviceMIDIEvent (myVC.auSampler, midiStatus,
note, velocity, 0);
! }
! packet = MIDIPacketNext(packet);
}


Demo
VTM AU Mix (w/MIDI)



Effects
• iOS 5 first version to offer multiple audio effects

• Many effects use floating-point samples, rather
than ints or fixed-point; requires adjusting the
stream formats of units connected to them (or
in-line AUConverters)

• Adjust with parameters (see
AudioUnitParameters.h)


Two sample effects

• Distortion effect - audio "fuzzing" effect, takes
no parameters

• Low pass ﬁlter - Cuts off frequencies above a
given frequency


Conﬁguring
AULowPassFilter

Float32 lowPassCutoffFrequency = 800.0;
AudioUnitSetParameter(_auLowPassFilter,
! ! ! ! ! kLowPassParam_CutoffFrequency,
! ! ! ! ! kAudioUnitScope_Global,
! ! ! ! ! 0,
! ! ! ! ! lowPassCutoffFrequency,
! ! ! ! ! 0);


Demo
VTM AU Effects



Bypassing units

UInt32 bypassed =
! self.effectEnabledSwitch.on ? 0 : 1;
AudioUnitSetProperty (self.audioUnit,
! ! kAudioUnitProperty_BypassEffect,
! ! kAudioUnitScope_Global,
! ! 0,
! ! &bypassed,
! ! sizeof(bypassed));


Recap

• Audio Units are greatly enhanced in iOS 5

• Real-time mixing and effects on mic input,
pitch-shifted samples, and ﬁle playback

• Best choice for serious audio production


Resources
• http://lists.apple.com/mailman/listinfo2/coreaudio-
api

• https://devforums.apple.com/community/ios/
graphics/audio

• Blogs:

• Mine: http://www.subfurther.com/blog

• Michael Tyson: http://atastypixel.com/blog/


Q&A

http://my.safaribooksonline.com/book/audio/
9780321636973

Core Audio Cranks It Up

More Related Content

What's hot

Similar to Core Audio Cranks It Up

More from Chris Adamson

Recently uploaded

Core Audio Cranks It Up