Core Audio: Don’t Be
Afraid To Play It LOUD!
         Chris Adamson
      360iDev San Jose 2010
          @invalidname
Previously…




Robert Strojan — “iPhone Audio Lab”
           Monday, 12:30
"Easy" and "CoreAudio" can't be used in the
same sentence. CoreAudio is very powerful,
  very complex, and under-documente...
Why?
• Problem domain is hard
• Performance is hard
• Low latency is hard
• Reusability is hard
• Doing without would suck
• Slowness would suck
• Latency would suck
• Non-reusability would suck
Last Exit Before Toll

• Alternatives to Core Audio:
 • Media Player: plays audio from iPod
    library
 • AV Framework: O...
Are they gone?
What They Missed

• Mixing
• Effects
• Streaming from/to network
• Lots of C
Core Audio
Programming
CA Style
• Procedural C
• Work with buffers of samples
• Most functions return an OSStatus
 • You must check this, every t...
Example: Audio Session

• iPhone-only API
 • Inspect and set audio hardware and
    software properties
 • Declare interac...
#import-ant!
#import <AudioToolbox/AudioToolbox.h>
Get Thee Some Docs!
Initialize audio session

OSStatus setupAudioSessionErr=
! AudioSessionInitialize (
! !   ! NULL, // default run loop
! ! ...
Initialize audio session
                                       C function call
OSStatus setupAudioSessionErr=
! AudioSess...
Initialize audio session

OSStatus setupAudioSessionErr=
! AudioSessionInitialize (            C function pointer
! !   ! ...
Initialize audio session

OSStatus setupAudioSessionErr=
! AudioSessionInitialize (
! !   ! NULL, // default run loop
! ! ...
Initialize audio session
                                      Return value

OSStatus setupAudioSessionErr=
! AudioSession...
Set category property

UInt32 sessionCategory =
  kAudioSessionCategory_PlayAndRecord;

setupAudioSessionErr =
  AudioSess...
Set category property

UInt32 sessionCategory =
  kAudioSessionCategory_PlayAndRecord;     Property name
setupAudioSession...
Set category property

UInt32 sessionCategory =
  kAudioSessionCategory_PlayAndRecord;

setupAudioSessionErr =
  AudioSess...
Set category property

UInt32 sessionCategory =
  kAudioSessionCategory_PlayAndRecord;

setupAudioSessionErr =
  AudioSess...
Is audio input available?
UInt32 ui32PropertySize = sizeof (UInt32);
UInt32 inputAvailable;

setupAudioSessionErr =
! Audi...
Is audio input available?
                                         Pointable size
UInt32 ui32PropertySize = sizeof (UInt32...
Is audio input available?
UInt32 ui32PropertySize = sizeof (UInt32);
UInt32 inputAvailable;
                              ...
Is audio input available?
UInt32 ui32PropertySize = sizeof (UInt32);
UInt32 inputAvailable;

setupAudioSessionErr =
! Audi...
Callback on a property

setupAudioSessionErr = AudioSessionAddPropertyListener (
! !   ! kAudioSessionProperty_AudioInputA...
Callback on a property

setupAudioSessionErr = AudioSessionAddPropertyListener (
! !   ! kAudioSessionProperty_AudioInputA...
Callback on a property

setupAudioSessionErr = AudioSessionAddPropertyListener (
! !   ! kAudioSessionProperty_AudioInputA...
Callback on a property

setupAudioSessionErr = AudioSessionAddPropertyListener (
! !   ! kAudioSessionProperty_AudioInputA...
CA style recap
• Lots of getting and setting properties
• Always check the return OSStatus
• Asychronous callbacks to C fu...
What’s In Core Audio?
Engines   Utilities
Engines       Utilities




Audio Units
Engines             Utilities




          Open AL
Audio Units
Engines          Utilities




Audio Queue Open AL
    Audio Units
Engines          Utilities




 AV Fndtn
Audio Queue Open AL
    Audio Units
Engines           Utilities




 AV Fndtn
Audio Queue Open AL
    Audio Units       Audio File
Engines              Utilities




 AV Fndtn
Audio Queue Open AL   Audio Conversion
    Audio Units          Audio File
Engines              Utilities




 AV Fndtn              Ext Audio File
Audio Queue Open AL   Audio Conversion
    Audio ...
Engines              Utilities




                      Audio File Stream
 AV Fndtn              Ext Audio File
Audio Que...
Engines              Utilities




                       Audio Session
                      Audio File Stream
 AV Fndtn ...
iPhone Audio Engines

• Media Player
• AV Foundation
• Audio Queue Services
• OpenAL
• Audio Unit Services
Media Player

• Objective-C class MPMusicPlayerController
  — Plays music from iPod library
• play, pause, stop. Propertie...
AV Framework
• Objective-C classes: AVAudioPlayer,
  AVAudioRecorder, AVAudioSession
• Play from / record to files
• Handle...
Audio Queue

• C functions for playback, recording
• Callback-driven: you fill or process audio
  buffers periodically
• Pl...
OpenAL
• C API intended for gaming, designed to
  resemble OpenGL
• Attach audio to “sources” in 3D space
 • Listener hear...
Audio Units
• Lowest publicly-accessed part of Core
  Audio
• C API for audio processing
• Uncompressed (PCM) only, no floa...
So What’s An Audio
      Unit?
Elements /   Elements /
    Buses    Buses
Input   Output
Scope    Scope
Global
Scope
AU “pull” model

• Each unit in the chain pulls audio data from
  some source
  • Another unit
  • A callback to your code...
I/O Unit
(output)
Effect   I/O Unit
Unit     (output)
I/O Unit   Effect   I/O Unit
 (input)   Unit     (output)
I/O Unit        Effect       I/O Unit
     (input)        Unit         (output)




Caveat: Not entirely accurate, for rea...
Units on iPhone OS
• I/O Units
 • “Remote” I/O: Abstraction around
    harware in/out
 • Voice Processing I/O: Same as Rem...
Units on iPhone OS

• Mixer Units
 • Multichannel Mixer: Mixes multiple input
    streams into one output stream
 • 3D Mix...
Units on iPhone OS
• Effect units
 • iPod Equalizer: Provides same features as
    iPod playback equalizer
• Converter uni...
Missing Units
• Unit types absent from iPhone OS
 • Generators
 • Music devices (MIDI) and effects
• Many units found on M...
RemoteIO

• Your app’s most direct access to audio
  input/output hardware
• Used at the end of unit chains that play
  au...
RemoteIO Buses

• Bus 0 is for H/W output
 • Headphones, speakers, etc.
• Bus 1 is for H/W input
 • Headset mic, phone mic...
Bus 1            Bus 1
        Remote
          I/O
Bus 0            Bus 0
Bus 1            Bus 1
        Remote
          I/O
Bus 0            Bus 0
Bus 1            Bus 1
        Remote
          I/O
Bus 0            Bus 0
Bus 1            Bus 1
        Remote
          I/O
Bus 0            Bus 0
Bus 1            Bus 1
        Remote
          I/O
Bus 0            Bus 0
But!


• There can only be one RemoteIO unit!
Bus 1            Bus 1
        Remote
          I/O
Bus 0            Bus 0
Bus 1            Bus 1
        Remote
          I/O
Bus 0            Bus 0
Example 1: Play
   Through
Play Through steps
1. Enable recording via Audio Session
2. Get the Remote I/O unit
3. Set stream format
4. Connect input ...
1. Set Up Audio Session
     for Recording

• Initialize the audio session
• Set a capture-enabling category
• Get/set oth...
Initialize Audio Session
OSStatus setupAudioSessionErr=
AudioSessionInitialize (
! !   ! NULL, // default run loop
! !   !...
Enable Recording
UInt32 sessionCategory =
  kAudioSessionCategory_PlayAndRecord;
!
setupAudioSessionErr = AudioSessionSetP...
Get HW sample rate
UInt32 f64PropertySize = sizeof (Float64);

setupAudioSessionErr = AudioSessionGetProperty
  (kAudioSes...
2. Get the Remote I/O
         Unit

• Describe the unit you want
• Search until you find it
• Enable its input and output ...
Describing a Unit

AudioComponentDescription audioCompDesc;
audioCompDesc.componentType = kAudioUnitType_Output;
audioComp...
Discovering a Unit
 AudioUnit remoteIOUnit = NULL;
 AudioComponent rioComponent = AudioComponentFindNext
     (NULL, &audi...
Enable output property
UInt32 oneFlag = 1;
AudioUnitElement bus0 = 0;

setupErr =
! AudioUnitSetProperty (remoteIOUnit,
! ...
Enable input property
AudioUnitElement bus1 = 1;

setupErr = AudioUnitSetProperty(remoteIOUnit,
         ! ! ! kAudioOutpu...
3. Set stream
         properties

• Define an AudioStreamBasicDescription
• Set it as the
  kAudioUnitProperty_StreamForma...
The ASBD
• Description of the common traits of an
  entire audio stream
 • Bitrate, channel count, format, format-
    spe...
Create an ASBD
AudioStreamBasicDescription myASBD;
memset (&myASBD, 0, sizeof (myASBD));

myASBD.mSampleRate = hardwareSam...
Set RemoteIO’s stream
      properties
• Declares format you want to read from
  input bus, and write to output bus
• Set ...
Bus 1            Bus 1
        Remote
          I/O
Bus 0            Bus 0
Set bus 1 / output-
   scope stream format
setupErr =
! AudioUnitSetProperty (remoteIOUnit,
! !   ! ! ! !        kAudioUni...
Set bus 0 / input-scope
     stream format
setupErr =
! AudioUnitSetProperty (remoteIOUnit,
! !   ! ! ! !        kAudioUni...
4. Connect Input to
        Output

• Create a structure describing the
  connection
• Set it as a property on the destina...
Declare connection

AudioUnitConnection connection;

connection.sourceAudioUnit = remoteIOUnit;
connection.sourceOutputNum...
Set connection
              property
setupErr =
! AudioUnitSetProperty(remoteIOUnit,
! !   ! ! ! !       kAudioUnitProper...
5. Let ’er Rip

setupErr =! AudioUnitInitialize(remoteIOUnit);
NSAssert (setupErr == noErr,
  @"Couldn't initialize RIO un...
Demo
Recap
• We created the Remote I/O unit
• Enabled input and output
• Created an ASBD to describe stream
  format and set it...
So frakkin’ what?
The Grind

• Setting up the Remote IO unit and its
  stream properties is something you’ll do all
  the time
• Now let’s t...
We can do more than
  just connect…

  Bus 1            Bus 1
          Remote
            I/O
  Bus 0            Bus 0
Example 2: Play through
 with render callbacks
Pulling audio
• A unit’s pull can be done one of several
  ways:
  • By being connected directly to another
    audio unit...
Setting a render
          callback
• Instead of connecting the audio units
  kAudioUnitProperty_MakeConnection…
• Set the...
Render Callback steps
1. Enable recording via Audio Session
2. Get the Remote I/O unit
3. Set stream format
4. Set up call...
Creating a render
        callback
• Define a struct containing whatever data
  your callback function will need
• Set up a...
Create your user data /
       context

    typedef struct {
    ! AudioUnit rioUnit;
    } EffectState;

    //...

    E...
AUCallbackStruct


AURenderCallbackStruct callbackStruct;
callbackStruct.inputProc = MyAURenderCallback;
callbackStruct.in...
Set callback property

setupErr =
! AudioUnitSetProperty(remoteIOUnit,
! !    kAudioUnitProperty_SetRenderCallback,
! !   ...
Callback function
         template

typedef OSStatus (*AURenderCallback) (
   void                        *inRefCon,
   A...
Create callback
            function
OSStatus MyAURenderCallback (
! !   ! void *! ! ! ! ! ! ! inRefCon,
! !   ! AudioUnit...
Do something
OSStatus renderErr = noErr;

UInt32 bus1 = 1;
renderErr = AudioUnitRender(rioUnit,
! !   ! ! ! ! ! ! ioAction...
Demo
We are still not
 impressed!
Almost there

• You now have access to raw samples, in
  your render callback, by way of the ioData
  pointer
• Anything y...
Example 3: Play through
   with gain effect
iPhone Audio Effects
• Generally need to be performed in render
  callbacks
• On Mac OS X, you can create custom units
  t...
Trivial effect: gain
Gain effect


• Get gain value of 0.0 to 1.0 from UISlider
• Multiply each sample by this value
Render considerations

• Render callbacks are on a real-time thread,
  with a hard deadline to return
  • If you miss the ...
Bad Ideas for Callbacks
• File or network I/O
• Blocking threads
• Heavy use of Obj-C messaging
• malloc()
• Basically any...
Good ideas for
         Callbacks
• Pass a struct, rather than an Obj-C object,
  as the user info object for your callbac...
Slider value in struct
typedef struct {
! AudioUnit rioUnit;
! float slider1Value;
} EffectState;



// set up callback st...
Read Slider Value in
        Callback
EffectState *effectState = (EffectState*) inRefCon;
AudioUnit rioUnit = effectState-...
Apply Effect to Samples
// walk the samples
AudioSampleType sample = 0;
for (int bufCount=0; bufCount<ioData->mNumberBuffe...
Demo
Example 4: More
Interesting Effects
Ring Modulator




   R(t) = C(t) x M(t)
Ring Modulation

• Multiplication of two signals
 • One is usually a sine wave
• Originally implemented as a ring-shaped
 ...
Modulate! Modulate!
• Ring modulator best
  known as the
  “Dalek” voice effect
  on Doctor Who (circa
  1963)
• Also used...
Building a ring
        modulator

• Need to model the sine wave
• Multiply samples
• Write modified sample back to ioData
Sine wave state
typedef struct {
! AudioUnit rioUnit;
! float slider1Value;
! AudioStreamBasicDescription asbd;
! float si...
Sine wave value
// AudioUnitRender() from RIO bus 1 here...
// memcpy() from buffer here...

float theta = effectState->si...
Demo
More Effects




• http://musicdsp.org/, et. al.
Example 5: Mixing with
       Effects
Bus 1

        Mixer   Bus 0

Bus 0
Render
        Callback
                   Bus 1

                                           Remote
                      ...
Render
                            Callback
                                        Bus 1

                               ...
Create a mixer unit
AudioComponentDescription mixerDesc;
mixerDesc.componentManufacturer =
  kAudioUnitManufacturer_Apple;...
Config mixer unit

• Set the stream property on its inputs
• Set the render callback or connection
  properties on its inpu...
Playing audio files in
          units
• Can’t do I/O in render callback
 • If compressed, we’d also need to convert
    to...
Cheat #1: Convert
    AAC file to PCM


afconvert --data LEI16 Girlfriend.m4a Girlfriend.caf
Cheat #2: Read file into
        RAM


• Obviously not recommended
Set up struct for file-
   playing callback
  typedef struct {
  ! void* audioData;
  ! AudioSampleType *samplePtr;
  } Mus...
Open audio file

NSURL *songURL = [NSURL fileURLWithPath:
! !   [[NSBundle mainBundle] pathForResource: @"Girlfriend"
! !  ...
Get size of audio data
         and malloc()
UInt64 audioDataByteCount;
UInt32 audioDataByteCountSize = sizeof (audioDataB...
Read file into RAM

UInt32 bytesRead = audioDataByteCount;
setupErr = AudioFileReadBytes(songFile,
! !   ! ! !       false,...
Get ASBD from file

AudioStreamBasicDescription fileASBD;
setupErr = AudioFileGetProperty(songFile,
! !   ! ! ! kAudioFileP...
Provide samples to
      mixer

  Render
  Callback
             Bus 1


                     Mixer
Set up render callback
AURenderCallbackStruct musicPlayerCallbackStruct;
musicPlayerCallbackStruct.inputProc = MusicPlayer...
Create
     MusicPlayerCallback()
MusicPlaybackState *musicPlaybackState = (MusicPlaybackState*) inRefCon;
! !
// walk the...
Connect mixer to RIO


                    Remote
    Mixer   Bus 0            Bus 0
                      I/O
Connect mixer to RIO
AudioUnitConnection connection;
connection.sourceAudioUnit = mixerUnit;
connection.sourceOutputNumber...
Bonus!




• Multichannel mixer unit has volume
  parameter on each of its inputs
Adjusting mixer input
            volume
-(IBAction) handleMicSliderValueChanged {
! OSStatus propSetErr = noErr;
! AudioU...
Demo
Takeaways


• Core Audio is hard
• Core Audio lets you do things that are
  freaking awesome
Since you’ll need help
• coreaudio-api at lists.apple.com
 • Apple developers post here every day
• stackoverflow.com
• dev...
Coming soon eventually
Ask me?

• [Time code]; blog
 • http://www.subfurther.com/blog
• @invalidname
• invalidname [at] gmail [dot] com
Core Audio: Don't Be Afraid to Play it LOUD! [360iDev, San Jose 2010]
Core Audio: Don't Be Afraid to Play it LOUD! [360iDev, San Jose 2010]
Core Audio: Don't Be Afraid to Play it LOUD! [360iDev, San Jose 2010]
Core Audio: Don't Be Afraid to Play it LOUD! [360iDev, San Jose 2010]
Core Audio: Don't Be Afraid to Play it LOUD! [360iDev, San Jose 2010]
Core Audio: Don't Be Afraid to Play it LOUD! [360iDev, San Jose 2010]
Core Audio: Don't Be Afraid to Play it LOUD! [360iDev, San Jose 2010]
Core Audio: Don't Be Afraid to Play it LOUD! [360iDev, San Jose 2010]
Core Audio: Don't Be Afraid to Play it LOUD! [360iDev, San Jose 2010]
Core Audio: Don't Be Afraid to Play it LOUD! [360iDev, San Jose 2010]
Core Audio: Don't Be Afraid to Play it LOUD! [360iDev, San Jose 2010]
Core Audio: Don't Be Afraid to Play it LOUD! [360iDev, San Jose 2010]
Core Audio: Don't Be Afraid to Play it LOUD! [360iDev, San Jose 2010]
Upcoming SlideShare
Loading in …5
×

Core Audio: Don't Be Afraid to Play it LOUD! [360iDev, San Jose 2010]

20,326 views
20,188 views

Published on

Some of the best apps on the iPhone so far are about audio: capturing it, processing it, playing it, and even synthesizing it. But how are some of these apps even possible? In this session, we'll deep into the darkest depths of Core Audio, the iPhone's enormously powerful and often challenging audio API. You'll learn the tricks of the Core Audio that aren't obvious from the docs, and the essential techniques you'll need to shake your users' headphones.

Published in: Technology
2 Comments
21 Likes
Statistics
Notes
No Downloads
Views
Total views
20,326
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
0
Comments
2
Likes
21
Embeds 0
No embeds

No notes for slide

Core Audio: Don't Be Afraid to Play it LOUD! [360iDev, San Jose 2010]

  1. 1. Core Audio: Don’t Be Afraid To Play It LOUD! Chris Adamson 360iDev San Jose 2010 @invalidname
  2. 2. Previously… Robert Strojan — “iPhone Audio Lab” Monday, 12:30
  3. 3. "Easy" and "CoreAudio" can't be used in the same sentence. CoreAudio is very powerful, very complex, and under-documented. Be prepared for a steep learning curve, APIs with millions of tiny little pieces, and puzzling things out from sample code rather than reading high-level documentation. –Jens Alfke, coreaudio-api list Feb 9, 2009
  4. 4. Why?
  5. 5. • Problem domain is hard • Performance is hard • Low latency is hard • Reusability is hard
  6. 6. • Doing without would suck • Slowness would suck • Latency would suck • Non-reusability would suck
  7. 7. Last Exit Before Toll • Alternatives to Core Audio: • Media Player: plays audio from iPod library • AV Framework: Obj-C API for file playback and recording
  8. 8. Are they gone?
  9. 9. What They Missed • Mixing • Effects • Streaming from/to network • Lots of C
  10. 10. Core Audio Programming
  11. 11. CA Style • Procedural C • Work with buffers of samples • Most functions return an OSStatus • You must check this, every time • Heavily callback-oriented • Heavily property-oriented
  12. 12. Example: Audio Session • iPhone-only API • Inspect and set audio hardware and software properties • Declare interaction with other audio processes on the device (e.g., iPod)
  13. 13. #import-ant! #import <AudioToolbox/AudioToolbox.h>
  14. 14. Get Thee Some Docs!
  15. 15. Initialize audio session OSStatus setupAudioSessionErr= ! AudioSessionInitialize ( ! ! ! NULL, // default run loop ! ! ! NULL, // default run loop mode ! ! ! MyInterruptionHandler, // interruption callback ! ! ! self); // client callback data ! NSAssert (setupAudioSessionErr == noErr, @"Couldn't initialize audio session");
  16. 16. Initialize audio session C function call OSStatus setupAudioSessionErr= ! AudioSessionInitialize ( ! ! ! NULL, // default run loop ! ! ! NULL, // default run loop mode ! ! ! MyInterruptionHandler, // interruption callback ! ! ! self); // client callback data ! NSAssert (setupAudioSessionErr == noErr, @"Couldn't initialize audio session");
  17. 17. Initialize audio session OSStatus setupAudioSessionErr= ! AudioSessionInitialize ( C function pointer ! ! ! NULL, // default run loop ! ! ! NULL, // default run loop mode ! ! ! MyInterruptionHandler, // interruption callback ! ! ! self); // client callback data ! NSAssert (setupAudioSessionErr == noErr, @"Couldn't initialize audio session");
  18. 18. Initialize audio session OSStatus setupAudioSessionErr= ! AudioSessionInitialize ( ! ! ! NULL, // default run loop ! ! ! NULL, // default run loop mode ! ! ! MyInterruptionHandler, // interruption callback ! ! ! self); // client callback data ! Obj-C pointer NSAssert (setupAudioSessionErr == noErr, @"Couldn't initialize audio session");
  19. 19. Initialize audio session Return value OSStatus setupAudioSessionErr= ! AudioSessionInitialize ( ! ! ! NULL, // default run loop ! ! ! NULL, // default run loop mode ! ! ! MyInterruptionHandler, // interruption callback ! ! ! self); // client callback data ! NSAssert (setupAudioSessionErr == noErr, @"Couldn't initialize audio session");
  20. 20. Set category property UInt32 sessionCategory = kAudioSessionCategory_PlayAndRecord; setupAudioSessionErr = AudioSessionSetProperty ( kAudioSessionProperty_AudioCategory, sizeof (sessionCategory), &sessionCategory); ! NSAssert (setupAudioSessionErr == noErr, @"Couldn't set audio session property");
  21. 21. Set category property UInt32 sessionCategory = kAudioSessionCategory_PlayAndRecord; Property name setupAudioSessionErr = (constant) AudioSessionSetProperty ( kAudioSessionProperty_AudioCategory, sizeof (sessionCategory), &sessionCategory); ! NSAssert (setupAudioSessionErr == noErr, @"Couldn't set audio session property");
  22. 22. Set category property UInt32 sessionCategory = kAudioSessionCategory_PlayAndRecord; setupAudioSessionErr = AudioSessionSetProperty ( Size of property kAudioSessionProperty_AudioCategory, sizeof (sessionCategory), &sessionCategory); ! NSAssert (setupAudioSessionErr == noErr, @"Couldn't set audio session property");
  23. 23. Set category property UInt32 sessionCategory = kAudioSessionCategory_PlayAndRecord; setupAudioSessionErr = AudioSessionSetProperty ( kAudioSessionProperty_AudioCategory, sizeof (sessionCategory), &sessionCategory); ! Pointer to value NSAssert (setupAudioSessionErr == noErr, @"Couldn't set audio session property");
  24. 24. Is audio input available? UInt32 ui32PropertySize = sizeof (UInt32); UInt32 inputAvailable; setupAudioSessionErr = ! AudioSessionGetProperty( kAudioSessionProperty_AudioInputAvailable, &ui32PropertySize, &inputAvailable); NSAssert (setupAudioSessionErr == noErr, @"Couldn't get current audio input available prop");
  25. 25. Is audio input available? Pointable size UInt32 ui32PropertySize = sizeof (UInt32); UInt32 inputAvailable; setupAudioSessionErr = ! AudioSessionGetProperty( kAudioSessionProperty_AudioInputAvailable, &ui32PropertySize, &inputAvailable); NSAssert (setupAudioSessionErr == noErr, @"Couldn't get current audio input available prop");
  26. 26. Is audio input available? UInt32 ui32PropertySize = sizeof (UInt32); UInt32 inputAvailable; Pointable variable setupAudioSessionErr = ! AudioSessionGetProperty( kAudioSessionProperty_AudioInputAvailable, &ui32PropertySize, &inputAvailable); NSAssert (setupAudioSessionErr == noErr, @"Couldn't get current audio input available prop");
  27. 27. Is audio input available? UInt32 ui32PropertySize = sizeof (UInt32); UInt32 inputAvailable; setupAudioSessionErr = ! AudioSessionGetProperty( kAudioSessionProperty_AudioInputAvailable, &ui32PropertySize, &inputAvailable); Address of size and variable NSAssert (setupAudioSessionErr == noErr, @"Couldn't get current audio input available prop");
  28. 28. Callback on a property setupAudioSessionErr = AudioSessionAddPropertyListener ( ! ! ! kAudioSessionProperty_AudioInputAvailable, ! ! ! MyInputAvailableListener, ! ! ! self); NSAssert (setupAudioSessionErr == noErr, @"Couldn't setup audio input prop listener");
  29. 29. Callback on a property setupAudioSessionErr = AudioSessionAddPropertyListener ( ! ! ! kAudioSessionProperty_AudioInputAvailable, ! ! ! MyInputAvailableListener, ! ! ! self); Property to listen to NSAssert (setupAudioSessionErr == noErr, @"Couldn't setup audio input prop listener");
  30. 30. Callback on a property setupAudioSessionErr = AudioSessionAddPropertyListener ( ! ! ! kAudioSessionProperty_AudioInputAvailable, ! ! ! MyInputAvailableListener, ! ! ! self); C function pointer NSAssert (setupAudioSessionErr == noErr, @"Couldn't setup audio input prop listener");
  31. 31. Callback on a property setupAudioSessionErr = AudioSessionAddPropertyListener ( ! ! ! kAudioSessionProperty_AudioInputAvailable, ! ! ! MyInputAvailableListener, ! ! ! self); “Client data” pointer NSAssert (setupAudioSessionErr == noErr, @"Couldn't setup audio input prop listener");
  32. 32. CA style recap • Lots of getting and setting properties • Always check the return OSStatus • Asychronous callbacks to C functions • Look up function templates in docs • “Context” or “user info” pointer can be an Obj-C object
  33. 33. What’s In Core Audio?
  34. 34. Engines Utilities
  35. 35. Engines Utilities Audio Units
  36. 36. Engines Utilities Open AL Audio Units
  37. 37. Engines Utilities Audio Queue Open AL Audio Units
  38. 38. Engines Utilities AV Fndtn Audio Queue Open AL Audio Units
  39. 39. Engines Utilities AV Fndtn Audio Queue Open AL Audio Units Audio File
  40. 40. Engines Utilities AV Fndtn Audio Queue Open AL Audio Conversion Audio Units Audio File
  41. 41. Engines Utilities AV Fndtn Ext Audio File Audio Queue Open AL Audio Conversion Audio Units Audio File
  42. 42. Engines Utilities Audio File Stream AV Fndtn Ext Audio File Audio Queue Open AL Audio Conversion Audio Units Audio File
  43. 43. Engines Utilities Audio Session Audio File Stream AV Fndtn Ext Audio File Audio Queue Open AL Audio Conversion Audio Units Audio File
  44. 44. iPhone Audio Engines • Media Player • AV Foundation • Audio Queue Services • OpenAL • Audio Unit Services
  45. 45. Media Player • Objective-C class MPMusicPlayerController — Plays music from iPod library • play, pause, stop. Properties for currentPlaybackTime, volume, etc. • No access to decoded samples or files
  46. 46. AV Framework • Objective-C classes: AVAudioPlayer, AVAudioRecorder, AVAudioSession • Play from / record to files • Handles compressed formats • play, record, pause, stop. volume, currentTime properties • No access to decoded samples
  47. 47. Audio Queue • C functions for playback, recording • Callback-driven: you fill or process audio buffers periodically • Plays compressed formats • Latency is negotiable
  48. 48. OpenAL • C API intended for gaming, designed to resemble OpenGL • Attach audio to “sources” in 3D space • Listener hears sound from source relative to their own position • Uncompressed (PCM) only • Can stream to a source (push model)
  49. 49. Audio Units • Lowest publicly-accessed part of Core Audio • C API for audio processing • Uncompressed (PCM) only, no floats • Extremely low latency • OpenAL and Queue implemented atop units
  50. 50. So What’s An Audio Unit?
  51. 51. Elements / Elements / Buses Buses
  52. 52. Input Output Scope Scope
  53. 53. Global Scope
  54. 54. AU “pull” model • Each unit in the chain pulls audio data from some source • Another unit • A callback to your code • Last unit is typically an I/O unit
  55. 55. I/O Unit (output)
  56. 56. Effect I/O Unit Unit (output)
  57. 57. I/O Unit Effect I/O Unit (input) Unit (output)
  58. 58. I/O Unit Effect I/O Unit (input) Unit (output) Caveat: Not entirely accurate, for reasons to be explained soon…
  59. 59. Units on iPhone OS • I/O Units • “Remote” I/O: Abstraction around harware in/out • Voice Processing I/O: Same as RemoteIO, but with echo supression for VOIP • Generic I/O: No hardware access; use for software audio rendering
  60. 60. Units on iPhone OS • Mixer Units • Multichannel Mixer: Mixes multiple input streams into one output stream • 3D Mixer: Mixes inputs with more control over panning, resampling. Used by OpenAL
  61. 61. Units on iPhone OS • Effect units • iPod Equalizer: Provides same features as iPod playback equalizer • Converter units • Converter: Converts between flavors of PCM, but not compressed formats
  62. 62. Missing Units • Unit types absent from iPhone OS • Generators • Music devices (MIDI) and effects • Many units found on Mac absent on iPhone • Filters, reverb, compressor, varispeed, other effects
  63. 63. RemoteIO • Your app’s most direct access to audio input/output hardware • Used at the end of unit chains that play audio • Used at the beginning of chains that capture audio
  64. 64. RemoteIO Buses • Bus 0 is for H/W output • Headphones, speakers, etc. • Bus 1 is for H/W input • Headset mic, phone mic, etc.
  65. 65. Bus 1 Bus 1 Remote I/O Bus 0 Bus 0
  66. 66. Bus 1 Bus 1 Remote I/O Bus 0 Bus 0
  67. 67. Bus 1 Bus 1 Remote I/O Bus 0 Bus 0
  68. 68. Bus 1 Bus 1 Remote I/O Bus 0 Bus 0
  69. 69. Bus 1 Bus 1 Remote I/O Bus 0 Bus 0
  70. 70. But! • There can only be one RemoteIO unit!
  71. 71. Bus 1 Bus 1 Remote I/O Bus 0 Bus 0
  72. 72. Bus 1 Bus 1 Remote I/O Bus 0 Bus 0
  73. 73. Example 1: Play Through
  74. 74. Play Through steps 1. Enable recording via Audio Session 2. Get the Remote I/O unit 3. Set stream format 4. Connect input to output 5. Let ’er rip
  75. 75. 1. Set Up Audio Session for Recording • Initialize the audio session • Set a capture-enabling category • Get/set other useful session properties
  76. 76. Initialize Audio Session OSStatus setupAudioSessionErr= AudioSessionInitialize ( ! ! ! NULL, // default run loop ! ! ! NULL, // default run loop mode ! ! ! NULL, // interruption callback ! ! ! NULL); // client callback data NSAssert (setupAudioSessionErr == noErr, @"Couldn't initialize audio session");
  77. 77. Enable Recording UInt32 sessionCategory = kAudioSessionCategory_PlayAndRecord; ! setupAudioSessionErr = AudioSessionSetProperty ! (kAudioSessionProperty_AudioCategory, ! sizeof (sessionCategory), ! &sessionCategory); ! NSAssert (setupAudioSessionErr == noErr, @"Couldn't set audio session property");
  78. 78. Get HW sample rate UInt32 f64PropertySize = sizeof (Float64); setupAudioSessionErr = AudioSessionGetProperty (kAudioSessionProperty_CurrentHardwareSampleRate, ! &f64PropertySize, ! &hardwareSampleRate); ! NSAssert (setupAudioSessionErr == noErr, @"Couldn't get current hardware sample rate"); NSLog (@"current hardware sample rate = %f", hardwareSampleRate);
  79. 79. 2. Get the Remote I/O Unit • Describe the unit you want • Search until you find it • Enable its input and output properties
  80. 80. Describing a Unit AudioComponentDescription audioCompDesc; audioCompDesc.componentType = kAudioUnitType_Output; audioCompDesc.componentSubType = kAudioUnitSubType_RemoteIO; audioCompDesc.componentManufacturer = kAudioUnitManufacturer_Apple; audioCompDesc.componentFlags = 0; audioCompDesc.componentFlagsMask = 0;
  81. 81. Discovering a Unit AudioUnit remoteIOUnit = NULL; AudioComponent rioComponent = AudioComponentFindNext (NULL, &audioCompDesc); OSErr setupErr = AudioComponentInstanceNew (rioComponent, &remoteIOUnit); NSAssert (setupErr == noErr, @"Couldn't get RIO unit instance"); • If there can be more than one match, iterate over AudioComponentFindNext, passing in last match you found
  82. 82. Enable output property UInt32 oneFlag = 1; AudioUnitElement bus0 = 0; setupErr = ! AudioUnitSetProperty (remoteIOUnit, ! ! ! ! ! ! kAudioOutputUnitProperty_EnableIO, ! ! ! ! ! ! kAudioUnitScope_Output, ! ! ! ! ! ! bus0, ! ! ! ! ! ! &oneFlag, ! ! ! ! ! ! sizeof(oneFlag)); NSAssert (setupErr == noErr, @"Couldn't enable RIO output");
  83. 83. Enable input property AudioUnitElement bus1 = 1; setupErr = AudioUnitSetProperty(remoteIOUnit, ! ! ! kAudioOutputUnitProperty_EnableIO, ! ! ! ! ! kAudioUnitScope_Input, ! ! ! ! ! bus1, ! ! ! ! &oneFlag, ! ! ! ! ! sizeof(oneFlag)); NSAssert (setupErr == noErr, @"couldn't enable RIO input");
  84. 84. 3. Set stream properties • Define an AudioStreamBasicDescription • Set it as the kAudioUnitProperty_StreamFormat • You will screw this up at least once
  85. 85. The ASBD • Description of the common traits of an entire audio stream • Bitrate, channel count, format, format- specific flags, etc. • Not all fields apply to all formats • On iPhone, Audio Units always work with LPCM
  86. 86. Create an ASBD AudioStreamBasicDescription myASBD; memset (&myASBD, 0, sizeof (myASBD)); myASBD.mSampleRate = hardwareSampleRate; myASBD.mFormatID = kAudioFormatLinearPCM; myASBD.mFormatFlags = kAudioFormatFlagsCanonical; myASBD.mBytesPerPacket = 4; myASBD.mFramesPerPacket = 1; myASBD.mBytesPerFrame = myASBD.mBytesPerPacket * myASBD.mFramesPerPacket; myASBD.mChannelsPerFrame = 2; myASBD.mBitsPerChannel = 16;
  87. 87. Set RemoteIO’s stream properties • Declares format you want to read from input bus, and write to output bus • Set the kAudioUnitProperty_StreamFormat property • On RemoteIO’s output scope for bus 1 • On RemoteIO’s input scope for bus 0
  88. 88. Bus 1 Bus 1 Remote I/O Bus 0 Bus 0
  89. 89. Set bus 1 / output- scope stream format setupErr = ! AudioUnitSetProperty (remoteIOUnit, ! ! ! ! ! ! kAudioUnitProperty_StreamFormat, ! ! ! ! ! ! kAudioUnitScope_Output, ! ! ! ! ! ! bus1, ! ! ! ! ! ! &myASBD, ! ! ! ! ! ! sizeof (myASBD)); NSAssert (setupErr == noErr, @"Couldn't set ASBD for RIO on output scope / bus 1");
  90. 90. Set bus 0 / input-scope stream format setupErr = ! AudioUnitSetProperty (remoteIOUnit, ! ! ! ! ! ! kAudioUnitProperty_StreamFormat, ! ! ! ! ! ! kAudioUnitScope_Input, ! ! ! ! ! ! bus0, ! ! ! ! ! ! &myASBD, ! ! ! ! ! ! sizeof (myASBD)); NSAssert (setupErr == noErr, @"Couldn't set ASBD for RIO on input scope / bus 0");
  91. 91. 4. Connect Input to Output • Create a structure describing the connection • Set it as a property on the destination audio unit
  92. 92. Declare connection AudioUnitConnection connection; connection.sourceAudioUnit = remoteIOUnit; connection.sourceOutputNumber = bus1; connection.destInputNumber = bus0;
  93. 93. Set connection property setupErr = ! AudioUnitSetProperty(remoteIOUnit, ! ! ! ! ! ! kAudioUnitProperty_MakeConnection, ! ! ! ! ! ! kAudioUnitScope_Input, ! ! ! ! ! ! bus0, ! ! ! ! ! ! &connection, ! ! ! ! ! ! sizeof (connection)); NSAssert (setupErr == noErr, @"Couldn't set RIO connection");
  94. 94. 5. Let ’er Rip setupErr =! AudioUnitInitialize(remoteIOUnit); NSAssert (setupErr == noErr, @"Couldn't initialize RIO unit"); // in handleStartTapped: OSStatus startErr = noErr; startErr = AudioOutputUnitStart (remoteIOUnit); NSAssert (startErr == noErr, @"Couldn't start RIO unit");
  95. 95. Demo
  96. 96. Recap • We created the Remote I/O unit • Enabled input and output • Created an ASBD to describe stream format and set it on bus 1 output and bus 0 input • Connected bus 1 output to bus 0 input • Initialized and started the unit
  97. 97. So frakkin’ what?
  98. 98. The Grind • Setting up the Remote IO unit and its stream properties is something you’ll do all the time • Now let’s try something different
  99. 99. We can do more than just connect… Bus 1 Bus 1 Remote I/O Bus 0 Bus 0
  100. 100. Example 2: Play through with render callbacks
  101. 101. Pulling audio • A unit’s pull can be done one of several ways: • By being connected directly to another audio unit • By registering a “render callback” to provide samples • This is the key to all things cool…
  102. 102. Setting a render callback • Instead of connecting the audio units kAudioUnitProperty_MakeConnection… • Set the property kAudioUnitProperty_SetRenderCallback • Provide a function pointer to your own function, and a “user info” object • Any samples you provide get played
  103. 103. Render Callback steps 1. Enable recording via Audio Session 2. Get the Remote I/O unit 3. Set stream format 4. Set up callback function 5. Let ’er rip You’ve already done all these
  104. 104. Creating a render callback • Define a struct containing whatever data your callback function will need • Set up an AUCallbackStruct • Set it as the kAudioUnitProperty_SetRenderCallback on bus 0 • Implement the callback function
  105. 105. Create your user data / context typedef struct { ! AudioUnit rioUnit; } EffectState; //... EffectState effectState;
  106. 106. AUCallbackStruct AURenderCallbackStruct callbackStruct; callbackStruct.inputProc = MyAURenderCallback; callbackStruct.inputProcRefCon = effectState;
  107. 107. Set callback property setupErr = ! AudioUnitSetProperty(remoteIOUnit, ! ! kAudioUnitProperty_SetRenderCallback, ! ! kAudioUnitScope_Global, ! ! bus0, ! ! &callbackStruct, ! ! sizeof (callbackStruct)); NSAssert (setupErr == noErr, @"Couldn't set RIO render callback on bus 0");
  108. 108. Callback function template typedef OSStatus (*AURenderCallback) ( void *inRefCon, AudioUnitRenderActionFlags *ioActionFlags, const AudioTimeStamp *inTimeStamp, UInt32 inBusNumber, UInt32 inNumberFrames, AudioBufferList *ioData );
  109. 109. Create callback function OSStatus MyAURenderCallback ( ! ! ! void *! ! ! ! ! ! ! inRefCon, ! ! ! AudioUnitRenderActionFlags *!ioActionFlags, ! ! ! const AudioTimeStamp *! ! ! inTimeStamp, ! ! ! UInt32! ! ! ! ! ! ! inBusNumber, ! ! ! UInt32! ! ! ! ! ! ! inNumberFrames, ! ! ! AudioBufferList *! ! ! ! ioData) { ! EffectState *effectState = (EffectState*) inRefCon; AudioUnit rioUnit = effectState->rioUnit;
  110. 110. Do something OSStatus renderErr = noErr; UInt32 bus1 = 1; renderErr = AudioUnitRender(rioUnit, ! ! ! ! ! ! ! ! ioActionFlags, ! ! ! ! ! ! ! ! inTimeStamp, ! ! ! ! ! ! ! ! bus1, ! ! ! ! ! ! ! ! inNumberFrames, ! ! ! ! ! ! ! ! ioData); NSAssert (renderErr == noErr, @”Couldn’t render”); • AudioUnitRender() gets available samples from a unit (either by copying from its buffer or calling an upstream unit)
  111. 111. Demo
  112. 112. We are still not impressed!
  113. 113. Almost there • You now have access to raw samples, in your render callback, by way of the ioData pointer • Anything you care to do with those samples can now be played out to hardware
  114. 114. Example 3: Play through with gain effect
  115. 115. iPhone Audio Effects • Generally need to be performed in render callbacks • On Mac OS X, you can create custom units to encapsulate effects • This was supposed to be in iPhone OS 3, but doesn’t actually work • Try AUPlugin.h See how far you get.
  116. 116. Trivial effect: gain
  117. 117. Gain effect • Get gain value of 0.0 to 1.0 from UISlider • Multiply each sample by this value
  118. 118. Render considerations • Render callbacks are on a real-time thread, with a hard deadline to return • If you miss the deadline you get silence • Render code must be highly performant
  119. 119. Bad Ideas for Callbacks • File or network I/O • Blocking threads • Heavy use of Obj-C messaging • malloc() • Basically anything that’s potentially slow, of indeterminate duration, and/or blocks
  120. 120. Good ideas for Callbacks • Pass a struct, rather than an Obj-C object, as the user info object for your callback • Let other threads do work for you and leave their work somewhere that the callback can just read it • Note possible race conditions
  121. 121. Slider value in struct typedef struct { ! AudioUnit rioUnit; ! float slider1Value; } EffectState; // set up callback state object effectState.rioUnit = remoteIOUnit; effectState.slider1Value = [slider1 value]; // set callback method AURenderCallbackStruct callbackStruct; callbackStruct.inputProc = MyAURenderCallback; callbackStruct.inputProcRefCon = &effectState; -(IBAction) handleSlider1ValueChanged { ! effectState.slider1Value = [slider1 value]; }
  122. 122. Read Slider Value in Callback EffectState *effectState = (EffectState*) inRefCon; AudioUnit rioUnit = effectState->rioUnit; float gain = effectState->slider1Value; // Call AudioUnitRender() as before to copy samples // into ioData
  123. 123. Apply Effect to Samples // walk the samples AudioSampleType sample = 0; for (int bufCount=0; bufCount<ioData->mNumberBuffers; bufCount++) { ! AudioBuffer buf = ioData->mBuffers[bufCount]; ! ! int currentFrame = 0; ! ! while ( currentFrame < inNumberFrames ) { ! ! ! // copy sample to buffer, across all channels ! ! ! for (int currentChannel=0; currentChannel<buf.mNumberChannels; currentChannel++) { ! ! ! ! memcpy(&sample, ! ! ! ! ! buf.mData + (currentFrame * 4) + (currentChannel*2), ! ! ! ! ! sizeof(AudioSampleType)); ! ! ! ! sample = (float) sample * gain; ! ! ! ! memcpy(buf.mData + (currentFrame * 4) + (currentChannel*2), ! ! ! ! ! &sample, ! ! ! ! ! sizeof(AudioSampleType)); ! ! ! }! ! ! currentFrame++; ! } }
  124. 124. Demo
  125. 125. Example 4: More Interesting Effects
  126. 126. Ring Modulator R(t) = C(t) x M(t)
  127. 127. Ring Modulation • Multiplication of two signals • One is usually a sine wave • Originally implemented as a ring-shaped circuit
  128. 128. Modulate! Modulate! • Ring modulator best known as the “Dalek” voice effect on Doctor Who (circa 1963) • Also used in early electronic music
  129. 129. Building a ring modulator • Need to model the sine wave • Multiply samples • Write modified sample back to ioData
  130. 130. Sine wave state typedef struct { ! AudioUnit rioUnit; ! float slider1Value; ! AudioStreamBasicDescription asbd; ! float sineFrequency; ! float sinePhase; } EffectState; // in setup method... // 23 Hz according to “Doctor Who” fan page // http://homepage.powerup.com.au/~spratleo/ Tech/Dalek_Voice_Primer.html effectState.sineFrequency = 23; effectState.sinePhase = 0; effectState.asbd = myASBD;
  131. 131. Sine wave value // AudioUnitRender() from RIO bus 1 here... // memcpy() from buffer here... float theta = effectState->sinePhase * M_PI * 2; sample = (sin(theta) * sample); // memcpy() to buffer here... effectState->sinePhase += 1.0 / (asbd.mSampleRate / sineFrequency); if (effectState->sinePhase > 1.0) { ! effectState->sinePhase -= 1.0; } /* to play pure sine wave sample = (sin (theta) * 0x7FFF); */
  132. 132. Demo
  133. 133. More Effects • http://musicdsp.org/, et. al.
  134. 134. Example 5: Mixing with Effects
  135. 135. Bus 1 Mixer Bus 0 Bus 0
  136. 136. Render Callback Bus 1 Remote Mixer Bus 0 Bus 0 I/O Bus 0 Remote Bus 1 I/O
  137. 137. Render Callback Bus 1 Remote Mixer Bus 0 Bus 0 I/O Bus 0 Audio Remote Unit Render Bus 1 Callback I/O Render()
  138. 138. Create a mixer unit AudioComponentDescription mixerDesc; mixerDesc.componentManufacturer = kAudioUnitManufacturer_Apple; mixerDesc.componentFlags = 0; mixerDesc.componentFlagsMask = 0; mixerDesc.componentType = kAudioUnitType_Mixer; mixerDesc.componentSubType = kAudioUnitSubType_MultiChannelMixer; AudioComponent mixerComponent = AudioComponentFindNext(NULL, &mixerDesc); setupErr = AudioComponentInstanceNew (mixerComponent, &mixerUnit); NSAssert (setupErr == noErr, @"Couldn't get mixer unit instance");
  139. 139. Config mixer unit • Set the stream property on its inputs • Set the render callback or connection properties on its inputs • Bus 0 is RobotVoiceRenderCallback • What shall we mix with?
  140. 140. Playing audio files in units • Can’t do I/O in render callback • If compressed, we’d also need to convert to PCM • Have a separate thread read/convert, then place PCM data in a ring buffer • See CARingBuffer (C++) in /Developer/ Extras/PublicUtility
  141. 141. Cheat #1: Convert AAC file to PCM afconvert --data LEI16 Girlfriend.m4a Girlfriend.caf
  142. 142. Cheat #2: Read file into RAM • Obviously not recommended
  143. 143. Set up struct for file- playing callback typedef struct { ! void* audioData; ! AudioSampleType *samplePtr; } MusicPlaybackState; //... MusicPlaybackState musicPlaybackState;
  144. 144. Open audio file NSURL *songURL = [NSURL fileURLWithPath: ! ! [[NSBundle mainBundle] pathForResource: @"Girlfriend" ! ! ! ! ofType: @"caf"]]; AudioFileID songFile; setupErr = AudioFileOpenURL((CFURLRef) songURL, ! ! ! kAudioFileReadPermission, ! ! ! 0, ! ! ! &songFile); NSAssert (setupErr == noErr, @"Couldn't open audio file");
  145. 145. Get size of audio data and malloc() UInt64 audioDataByteCount; UInt32 audioDataByteCountSize = sizeof (audioDataByteCount); setupErr = AudioFileGetProperty(songFile, ! ! ! ! ! kAudioFilePropertyAudioDataByteCount, ! ! ! ! ! &audioDataByteCountSize, ! ! ! ! ! &audioDataByteCount); NSAssert (setupErr == noErr, @"Couldn't get size property"); musicPlaybackState.audioData = malloc (audioDataByteCount); musicPlaybackState.samplePtr = musicPlaybackState.audioData;
  146. 146. Read file into RAM UInt32 bytesRead = audioDataByteCount; setupErr = AudioFileReadBytes(songFile, ! ! ! ! ! false, ! ! ! ! ! 0, ! ! ! ! ! &bytesRead, ! ! ! ! ! musicPlaybackState.audioData); NSAssert (setupErr == noErr, @"Couldn't read audio data");
  147. 147. Get ASBD from file AudioStreamBasicDescription fileASBD; setupErr = AudioFileGetProperty(songFile, ! ! ! ! ! kAudioFilePropertyDataFormat, ! ! ! ! ! &asbdSize, ! ! ! ! ! &fileASBD); NSAssert (setupErr == noErr, @"Couldn't get file asbd");
  148. 148. Provide samples to mixer Render Callback Bus 1 Mixer
  149. 149. Set up render callback AURenderCallbackStruct musicPlayerCallbackStruct; musicPlayerCallbackStruct.inputProc = MusicPlayerCallback; musicPlayerCallbackStruct.inputProcRefCon = &musicPlaybackState; ! setupErr = ! AudioUnitSetProperty(mixerUnit, ! ! ! ! ! kAudioUnitProperty_SetRenderCallback, ! ! ! ! ! kAudioUnitScope_Global, ! ! ! ! ! 1, ! ! ! ! ! &musicPlayerCallbackStruct, ! ! ! ! ! sizeof (musicPlayerCallbackStruct)); NSAssert (setupErr == noErr, @"Couldn't set mixer render callback on bus 1");
  150. 150. Create MusicPlayerCallback() MusicPlaybackState *musicPlaybackState = (MusicPlaybackState*) inRefCon; ! ! // walk the samples AudioSampleType sample = 0; for (int bufCount=0; bufCount<ioData->mNumberBuffers; bufCount++) { ! AudioBuffer buf = ioData->mBuffers[bufCount]; ! // AudioSampleType* bufferedSample = (AudioSampleType*) &buf.mData; ! int currentFrame = 0; ! while ( currentFrame < inNumberFrames ) { ! ! // copy sample to buffer, across all channels ! ! for (int currentChannel=0; currentChannel<buf.mNumberChannels; currentChannel++) { ! ! ! sample = *musicPlaybackState->samplePtr++; ! ! ! memcpy(buf.mData + (currentFrame * 4) + (currentChannel*2), ! ! ! ! &sample, ! ! ! ! sizeof(AudioSampleType)); ! ! }! ! ! currentFrame++; ! } } return noErr;
  151. 151. Connect mixer to RIO Remote Mixer Bus 0 Bus 0 I/O
  152. 152. Connect mixer to RIO AudioUnitConnection connection; connection.sourceAudioUnit = mixerUnit; connection.sourceOutputNumber = bus0; connection.destInputNumber = bus0; ! setupErr = ! AudioUnitSetProperty(remoteIOUnit, kAudioUnitProperty_MakeConnection, kAudioUnitScope_Input, bus0, &connection, sizeof (connection)); NSAssert (setupErr == noErr, @"Couldn't set mixer-to-RIO connection");
  153. 153. Bonus! • Multichannel mixer unit has volume parameter on each of its inputs
  154. 154. Adjusting mixer input volume -(IBAction) handleMicSliderValueChanged { ! OSStatus propSetErr = noErr; ! AudioUnitParameterValue sliderVal = [micSlider value]; ! propSetErr = AudioUnitSetParameter(mixerUnit, ! ! ! ! ! ! kMultiChannelMixerParam_Volume, ! ! ! ! ! ! kAudioUnitScope_Input, ! ! ! ! ! ! 0, ! ! ! ! ! ! sliderVal, ! ! ! ! ! ! 0); NSAssert (propSetErr == noErr, @"Couldn't set mixer volume on bus 0"); }
  155. 155. Demo
  156. 156. Takeaways • Core Audio is hard • Core Audio lets you do things that are freaking awesome
  157. 157. Since you’ll need help • coreaudio-api at lists.apple.com • Apple developers post here every day • stackoverflow.com • devforums.apple.com • Look for Michael Tyson’s post on Remote I/O unit
  158. 158. Coming soon eventually
  159. 159. Ask me? • [Time code]; blog • http://www.subfurther.com/blog • @invalidname • invalidname [at] gmail [dot] com

×