Get On The Audiobus (CocoaConf Columbus, September 2013)

1,217 views
1,129 views

Published on

Audiobus is an iOS app that allows other apps to work together as an audio-processing toolchain: play your MIDI keyboard into one app, run it through filters in other apps, and mix it in a third. All in real-time, foreground or background. That such a thing is possible on the locked down iOS platform is remarkable enough, but what's even more remarkable is that hundreds of audio apps have added Audiobus support in the few months since its debut, including Apple's own GarageBand. In this session, we'll take a look at the Audiobus SDK and see how to create inputs, outputs, and filters that can be managed by the Audiobus app to process audio in collaboration with other apps on the device.

1 Comment
3 Likes
Statistics
Notes
No Downloads
Views
Total views
1,217
On SlideShare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
0
Comments
1
Likes
3
Embeds 0
No embeds

No notes for slide

Get On The Audiobus (CocoaConf Columbus, September 2013)

  1. 1. Get on the Audiobus Chris Adamson • @invalidname CocoaConf Columbus • September 2013 Slides & code will be posted to the CocoaConf Glassboard, and announced on my Twitter & app.net Sunday, September 29, 13
  2. 2. Sunday, September 29, 13
  3. 3. Roadmap • How audio works on iOS • What Audiobus is and how it connects apps • Adopting Audiobus in your audio app Sunday, September 29, 13
  4. 4. Audio on iOS • Each app is responsible for its own audio • No access to audio to/from other apps • Apps use the Audio Session API to interact with the system • See also AVAudioSession in AV Foundation Sunday, September 29, 13
  5. 5. Audio Session • Allows inspection of hardware properties (sampling rate, hardware latencies) and negotiation of access to system audio resources • Audio “category” declares what your app does with audio • This affects things like whether you mix with other apps’ audio, honor ring/silent, can play in background, etc. Sunday, September 29, 13
  6. 6. Audio Categories • Ambient • Solo Ambient • Playback • Record • Play and Record • Audio Processing • Multi-Route Sunday, September 29, 13
  7. 7. Audio Engines • How your app interacts with audio hardware, i.e., captures or produces sound • OpenAL (play-out only) • Audio Queue • Audio Units Sunday, September 29, 13
  8. 8. AURemoteIO AURemoteIO Bus 0: audio out Bus 1: audio in Sunday, September 29, 13
  9. 9. Mixing between apps AURemoteIO AURemoteIO Sunday, September 29, 13
  10. 10. Mixing between apps AURemoteIO AURemoteIO ! Sunday, September 29, 13
  11. 11. Every App is an Island • Only awareness of other apps’ audio is value of kAudioSessionProperty_OtherAudioIsPlaying property • No access to what other apps are playing audio, how loud it is, etc. Sunday, September 29, 13
  12. 12. Which means… • You can’t record audio from one app in another app • Production apps can’t specialize; have to provide everything (instruments, filters/ effects, recording) that you’d ever need Sunday, September 29, 13
  13. 13. Enter Audiobus Sunday, September 29, 13
  14. 14. Audiobus • Standalone app that coordinates inter-app audio • Currently on 50% off sale ($4.99) • Only works with apps that adopt the Audiobus API • 300 and counting! Sunday, September 29, 13
  15. 15. Demo Sunday, September 29, 13
  16. 16. What Audiobus Is • Audiobus is an app for users to coordinate audio across supported apps • User decides which apps are the inputs, effects, and outputs Sunday, September 29, 13
  17. 17. What Audiobus Isn’t • Audiobus is not a general-purpose system- level audio capture (like Audio Hijack on Mac) • Audiobus cannot get audio from or send audio to an arbitrary app • Apps must adopt the Audiobus SDK and register with the Audiobus website Sunday, September 29, 13
  18. 18. How the heck does it even work? Considering that inter-app communication is nearly impossible on iOS… Sunday, September 29, 13
  19. 19. Secret Sauce! • Audiobus began with MIDI “System Exclusive” (SysEx) messages, defined as being arbitrary blobs of data unique to a given MIDI device • Originally meant for synths to exchange waveforms, patches or other software/firmware upgrades, etc. • MIDI messages available to all interested apps via Core MIDI • Later switched to Mach Ports, which Core MIDI is implemented atop Sunday, September 29, 13
  20. 20. Audiobus Concepts • Apps take on roles based on their relationship to Audiobus • Inputs produce audio • Outputs receive audio • Filters receive from inputs and send to outputs • Points of connection are called ports Sunday, September 29, 13
  21. 21. http://developer.audiob.us/doc/index.html Sunday, September 29, 13
  22. 22. Basic Audiobus Integration • Decide if you’re an input, output, or filter • Decide if you can work with the Remote IO unit or Audiobus’ port API • Adopt the Audiobus SDK to connect to Audiobus at runtime • Register at audiob.us Sunday, September 29, 13
  23. 23. An Audiobus Integration Case Study Sunday, September 29, 13
  24. 24. Audiobus Web Radio • Web Radio app developed as in-class exercise for all-day Core Audio class • Coming to CocoaConfs Boston & Atlanta • Uses Audio File Stream to receive packets of MP3 and play them with Audio Queue 012 Packets Packets Packets Packets Packets Packets Sunday, September 29, 13
  25. 25. LPCM or GTFO • Audiobus ports and AURemoteIO only work with uncompressed LPCM audio • Web radio app is dealing in MP3 or AAC • Conversion to LPCM happens inside the Audio Queue Sunday, September 29, 13
  26. 26. Offline Queues 012 AURemoteIO Packets Packets Packets Packets Packets Packets AudioQueueOfflineRender() Sunday, September 29, 13
  27. 27. Libraries • Download the Audiobus SDK from developer.audiob.us • Add libAudiobus.a and the Audiobus headers to your project • Add Accelerate,AudioToolbox, QuartzCore, CoreGraphics, and Security frameworks to your project Sunday, September 29, 13
  28. 28. Sunday, September 29, 13
  29. 29. Enable background audio • Add “audio” to the app’s “Required Background Modes” if it’s not already present • All Audiobus-enabled apps must participate in backgrounding, since they must be able to be running when Audiobus is in foreground Sunday, September 29, 13
  30. 30. Sunday, September 29, 13
  31. 31. Create a launch URL scheme • You must have a URL scheme for your app that ends in “.audiobus” for Audiobus to be able to launch you • Add this to the target’s “URL Types” Sunday, September 29, 13
  32. 32. Sunday, September 29, 13
  33. 33. Get Audiobus API key • For apps registered with iTunes Connect, submit your App Store URL or ID • For unpublished apps or tinkering, register for a temporary ID, good for 14 days • This requires dropping the Info.plist from your app bundle (not from project!) Sunday, September 29, 13
  34. 34. Sunday, September 29, 13
  35. 35. Audiobus API Keys • Audiobus app gets a master list of known keys from a server every 30 minutes • For temporary IDs, click the link from the developer page on the device that you’re using Audiobus on to register your App ID • e.g., audiobus-registry:// developer.audiob.us/tempreg?t=0ff37 • The dev page can mail you the link Sunday, September 29, 13
  36. 36. Sunday, September 29, 13
  37. 37. Set Audiobus-compatible behaviors • Audio Session category must be playback or play-and-record • Must also set the mix-with-others property on the audio session • Often do both these things in the AppDelegate Sunday, September 29, 13
  38. 38. UInt32 audioCategory = kAudioSessionCategory_MediaPlayback; AudioSessionSetProperty(kAudioSessionProperty_AudioCategory, !! ! ! ! ! ! ! sizeof(audioCategory), !! ! ! ! ! ! ! &audioCategory); UInt32 allowMixing = YES; AudioSessionSetProperty(kAudioSessionProperty_OverrideCategory MixWithOthers, ! ! ! ! ! ! ! ! ! sizeof (allowMixing), ! ! ! ! ! ! ! ! ! &allowMixing); Note:Audio Session is deprecated in iOS 7 Modern apps can use the AV Foundation equivalents Sunday, September 29, 13
  39. 39. SetupYour App’s Audio System • Your app’s audio infrastructure needs to be up and running before you connect to Audiobus • If you’re going to send audio via the ABAudioBusAudioUnitWrapper, you’ll need to initialize your AURemoteIO • For the web radio app, I send an NSNotification once the player class starts playing Sunday, September 29, 13
  40. 40. Instantiate ABAudiobusController • ABAudiobusController is your app’s connection to Audiobus; probably gets held as a strong property somewhere • init method takes the launch URL and your API key • Obviously, these must match what you registered on audiob.us Sunday, September 29, 13
  41. 41. self.audiobusController = [[ABAudiobusController alloc] ! ! ! ! ! ! initWithAppLaunchURL: [NSURL URLWithString:@"audiobuswebradio.audiobus://"] ! ! ! ! ! ! apiKey:AUDIOBUS_API_KEY]; Sunday, September 29, 13
  42. 42. Create ABOutputPort (and Audio Unit wrapper) • ABOutputPort sends audio to Audiobus (via ABOutputPortSendAudio() function) • If you use a RemoteIO unit for your output, the ABAudiobusAudioUnitWrapper will make these calls for you Sunday, September 29, 13
  43. 43. CCFViewController *vc = (CCFViewController*) self.window.rootViewController; ABOutputPort *output = [self.audiobusController ! ! ! ! ! ! addOutputPortNamed:@"Audio Output" ! ! ! ! ! ! title:NSLocalizedString(@"Main App Output", @"")]; self.audiobusAudioUnitWrapper = [[ABAudiobusAudioUnitWrapper alloc] ! ! ! ! ! ! initWithAudiobusController:self.audiobusController ! ! ! ! ! ! ! ! audioUnit:vc.player.remoteIOUnit ! ! ! ! ! ! ! ! output:output ! ! ! ! ! ! ! ! input:nil]; Sunday, September 29, 13
  44. 44. Demo Sunday, September 29, 13
  45. 45. Demo Sunday, September 29, 13
  46. 46. Ports • Apps that don’t use the Audio Unit Wrapper use ports directly instead • ABOutputPortSendAudio() for senders (inputs and filters) • Block-based callback or poll with ABInputPortReceive() for receivers (filters and outputs) Sunday, September 29, 13
  47. 47. Filters & Outputs • If you produce audio output based on input, you need to tell the ABInputPort, so that the signal isn’t doubled in Audiobus. Sunday, September 29, 13
  48. 48. As for iOS 7… Sunday, September 29, 13
  49. 49. Sunday, September 29, 13
  50. 50. Sherlocked? Embrace & Extend! From: michael@audiob.us Date: June 19, 2013 Subject: Important Information Regarding Audiobus, iOS 7 and Inter-App Audio iOS 7 introduces many new features, including Apple’s own Inter-App Audio framework which we’re planning to incorporate into Audiobus so you don’t have to. For details and further discussion, we highly recommend checking out our thread on the Apple developer forums: https://devforums.apple.com/thread/191197 Sunday, September 29, 13
  51. 51. Closing Thoughts • Audiobus is approachable for developers already working at the Audio Unit level • Adding Audiobus will get it seen by users who’ve proven willing to pay for good apps (we love music app users!) • Future-proofed for iOS 7 Sunday, September 29, 13
  52. 52. Q&A Slides & code will be posted to the CocoaConf Glassboard, and announced on my Twitter & app.net (@invalidname) Sunday, September 29, 13

×