Your SlideShare is downloading. ×
0
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Core Audio Intro (Detroit Mobile City 2013)
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Core Audio Intro (Detroit Mobile City 2013)

1,109

Published on

Overview of when (and when not) to use Core Audio on iOS. Includes an overview of the various APIs and a deeper look at Audio Queues and Audio Units.

Overview of when (and when not) to use Core Audio on iOS. Includes an overview of the various APIs and a deeper look at Audio Queues and Audio Units.

Published in: Technology
1 Comment
2 Likes
Statistics
Notes
No Downloads
Views
Total Views
1,109
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
1
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Core Audio Intro Chris Adamson • @invalidname Detroit Mobile City — Feb. 2, 2013 Slides will be available at: http://www.slideshare.net/invalidnameFriday, February 1, 13
  • 2. Agenda • iOS Media Technologies & Frameworks • How digital audio works • The Core Audio APIs • Neat demos!Friday, February 1, 13
  • 3. iOS Media http://www.apple.com/ipod-touch/Friday, February 1, 13
  • 4. Media Frameworks • AV Foundation • Core Audio • Core Media • Core MIDI • Core Video • OpenAL • Media Player Special iOS features supported by the media frameworks: • AirPlay •HTTP Live StreamingFriday, February 1, 13
  • 5. AV Foundation http://www.apple.com/apps/imovie/Friday, February 1, 13
  • 6. AV Foundation • Audio/Video framework for: • capture • editing • export • playbackFriday, February 1, 13
  • 7. AV Foundation APIs • Capture APIs work with a “capture session” object that delivers data to outputs (files, callbacks to your app) • Playback uses “assets”: the items to be played • Editing/export combines assets into “compositions”Friday, February 1, 13
  • 8. Media PlayerFriday, February 1, 13
  • 9. Media Player • MPMoviePlayerController — simple video player • MPMediaQuery, MPMediaItem — access to iPod library • MPVolumeView — system volume control, plus AirPlay button (if available)Friday, February 1, 13
  • 10. Wait, I thought this was a Core Audio session?Friday, February 1, 13
  • 11. Friday, February 1, 13
  • 12. Friday, February 1, 13
  • 13. When not to use Core Audio • Anything video: AV Foundation • iPod library access: Media Player • Playing a file/URL: AVAudioPlayer • Recording to a file: AVAudioRecorderFriday, February 1, 13
  • 14. So when do I use Core Audio? • Well, how much do you care about the audio data?Friday, February 1, 13
  • 15. Digital Audio 101Friday, February 1, 13
  • 16. Friday, February 1, 13
  • 17. Friday, February 1, 13
  • 18. 1.2 0.8 0.4 -2.4 -2 -1.6 -1.2 -0.8 -0.4 0 0.4 0.8 1.2 1.6 2 2.4 -0.4 -0.8 -1.2Friday, February 1, 13
  • 19. 0.6 0.5 0.4 0.3 0.2 0.1 -0.5 -0.25 0 0.25 0.5Friday, February 1, 13
  • 20. 0.6 0.5 0.4 0.3 0.2 0.1 -0.5 -0.25 0 0.25 0.5Friday, February 1, 13
  • 21. 0.6 0.5 0.4 0.3 0.2 0.1 -0.5 -0.25 0 0.25 0.5Friday, February 1, 13
  • 22. 0.6 0.5 0.4 0.3 0.2 0.1 -0.5 -0.25 0 0.25 0.5Friday, February 1, 13
  • 23. 0.6 0.5 0.4 0.3 0.2 0.1 -0.5 -0.25 0 0.25 0.5Friday, February 1, 13
  • 24. 0.6 0.5 0.4 0.3 0.2 0.1 -0.5 -0.25 0 0.25 0.5Friday, February 1, 13
  • 25. 0.6 0.5 0.4 0.3 0.2 0.1 -0.5 -0.25 0 0.25 0.5Friday, February 1, 13
  • 26. 0.6 0.5 0.4 0.3 0.2 0.1 -0.5 -0.25 0 0.25 0.5Friday, February 1, 13
  • 27. Samples! 00010d0 00 00 00 00 00 00 00 00 00 00 00 00 00 04 00 0e 00010e0 00 25 00 4a 00 77 00 af 00 e9 01 31 01 68 01 9e 00010f0 01 96 01 78 01 36 00 df 00 51 ff c9 ff 19 fe 69 0001100 fd ad fc fa fc 23 fb 59 fa 8b f9 d6 f8 f0 f8 77 0001110 f7 e0 f7 ad f7 36 f7 1e f6 f5 f7 0b f7 65 f7 8c 0001120 f7 e3 f8 18 f8 80 f8 e8 f9 54 f9 c7 fa 48 fa d0 0001130 fb 74 fb fe fc 7e fd 0e fd a4 fe 33 fe cb ff 5b 0001140 ff e5 00 61 00 c2 01 1c 01 6b 01 b9 01 e2 02 0b 0001150 02 09 02 3c 02 2b 02 2a 02 00 01 c9 01 84 01 56 0001160 01 10 00 d5 00 84 00 3e ff fa ff c1 ff 8a ff 60 0001170 ff 28 ff 00 fe d2 fe c7 fe be fe d8 fe e2 fe ed 0001180 ff 0f ff 41 ff 8e ff eb 00 2f 00 99 01 0a 01 99 0001190 02 1c 02 bd 03 3f 03 ab 04 37 04 b3 05 9a 06 09 00011a0 06 be 07 00 07 b0 08 31 08 b1 09 09 09 68 09 a4 00011b0 09 e6 0a 3a 0a 35 0a 3d 0a 43 0a 25 0a 07 09 d6Friday, February 1, 13
  • 28. Digital Audio!Friday, February 1, 13
  • 29. Friday, February 1, 13
  • 30. LPCM • “Linear Pulse Code Modulation” • Samples represent signal strength • Values are linear: double the value equals double the signal • But what values?Friday, February 1, 13
  • 31. Limits of human hearingFriday, February 1, 13
  • 32. Typical PCM • Human hearing: 12 - 20,000 Hz • Higher if you are a dog or Superman • Nyquist-Shannon Sampling theorem: • “If a function x(t) contains no frequencies higher than B hertz, it is completely determined by giving its ordinates at a series of points spaced 1/(2B) seconds apart.” • Sampling at 44,100 Hz reproduces frequencies up to 22,050 HzFriday, February 1, 13
  • 33. PCM files • CD audio: 16 bit stereo, 44.1 KHz • 74 minutes = 650 MBFriday, February 1, 13
  • 34. Entropy EncodingFriday, February 1, 13
  • 35. Entropy Encoding • Ordered, predictable signals convey no information • You only want to encode the part of the signal that is not predictable (disorder, hence entropy) • Find common patterns in signal and represent them in a more compact formFriday, February 1, 13
  • 36. http://en.wikipedia.org/wiki/Information_entropyFriday, February 1, 13
  • 37. Encoded Formats • Can be lossy or lossless • AAC, MP3, Apple Lossless (ALAC), etc. • There is no “best” codec • Encoding helps with storage and transmission • Must be converted to LPCM to playFriday, February 1, 13
  • 38. Core Audio ProgrammingFriday, February 1, 13
  • 39. “Easy” and “CoreAudio” cant be used in the same sentence. CoreAudio is very powerful, very complex, and under- documented. Be prepared for a steep learning curve, APIs with millions of tiny little pieces, and puzzling things out from sample code rather than reading high- level documentation. —Jens Alfke (creator of iChat & Stickies) coreaudio-api mailing list Feb. 9, 2009Friday, February 1, 13
  • 40. When to use Core Audio • You need to process media in real-time • You need to mix, measure, or perform effects on audio • You’re a control freakFriday, February 1, 13
  • 41. Core Audio APIs • Almost always plain C • Little use of Core Foundation. Major CA objects are not CF opaque types (e.g., AudioQueueDispose() instead of CFRelease()). • Optional “PublicUtility” is C++Friday, February 1, 13
  • 42. Core Audio Patterns • Format agnosticism • Heavily property-driven • Can introspect to find what properties are supported, their types and size, etc. • Some properties have listenersFriday, February 1, 13
  • 43. Digital Audio Concepts • Sample: a value for one channel at one instant • Frame: collection of all the samples for one instant • Packet: a collection of framesFriday, February 1, 13
  • 44. AudioStreamBasicDescription struct AudioStreamBasicDescription { Float64 mSampleRate; UInt32 mFormatID; UInt32 mFormatFlags; UInt32 mBytesPerPacket; UInt32 mFramesPerPacket; UInt32 mBytesPerFrame; UInt32 mChannelsPerFrame; UInt32 mBitsPerChannel; UInt32 mReserved; }; typedef struct AudioStreamBasicDescription AudioStreamBasicDescription;Friday, February 1, 13
  • 45. ASBD • Describes an audio stream • Retrieved as a property of input objects (e.g., files), set on streams, units, output objects • Define only the values you know and care about; zero out the rest. • memset (&outputDesc, 0, sizeof (outputDesc));Friday, February 1, 13
  • 46. Format Notes • Some stream formats can be read but not written (MP3) • Significant differences in CPU, memory costs among formats • File “container” formats may be format-agnostic (CAF) or work with only one stream format (MP3)Friday, February 1, 13
  • 47. Core Audio APIsFriday, February 1, 13
  • 48. Audio Helpers • Audio File • Audio Converters • ExtAudioFile • Audio File Stream • Audio SessionFriday, February 1, 13
  • 49. Audio Engines • Audio Queue — convenience for recording, playback • Wrapped by AVAudioPlayer/ Recorder • Audio Units — capture, mixing, effects • OpenAL — 3D audio for gamesFriday, February 1, 13
  • 50. Audio QueuesFriday, February 1, 13
  • 51. Audio QueuesFriday, February 1, 13
  • 52. Audio QueuesFriday, February 1, 13
  • 53. Audio QueuesFriday, February 1, 13
  • 54. Audio Queue Recording • Feed empty buffers to queue • When buffer is returned to your callback, it contains captured packets • Write to file, process samples, whatever (…or just use AVAudioRecorder)Friday, February 1, 13
  • 55. Audio Queue Playback • Fill buffers with data (from file, network, synthesis, etc.), then enqueue • Queue plays them in order received • Callback returns drained buffer for your app to refill or dispose (…or just use AVAudioPlayer)Friday, February 1, 13
  • 56. Putting It Together: Web RadioFriday, February 1, 13
  • 57. Putting It Together: Web Radio NSURLConnection delivers NSData buffers, containing audio and framing info. We pass it to NSData NSData Audio File Services. Packets Packets Packets Packets PacketsFriday, February 1, 13
  • 58. Putting It Together: Web Radio NSURLConnection delivers NSData buffers, containing audio and framing info. We pass it to NSData NSData Audio File Services. Packets Packets Packets Packets Packets Packets Packets Audio File Services calls us back with parsed packets of audio data. Packets Packets PacketsFriday, February 1, 13
  • 59. Putting It Together: Web Radio NSURLConnection delivers NSData buffers, containing audio and framing info. We pass it to NSData NSData Audio File Services. Packets Packets Packets Packets Packets Packets Packets Audio File Services calls us back with parsed packets of audio data. Packets Packets Packets We create an AudioQueueBuffer Packets Packets with those packets and enqueue it Packets 2 Packets 1 0 for play-out. Packets PacketsFriday, February 1, 13
  • 60. Demo Web Radio AppFriday, February 1, 13
  • 61. Audio Units • Lowest level for iOS audio developers • Real-time capture, processing, play AUSomethingFriday, February 1, 13
  • 62. Types of Audio Units • Output (which also do input) • Generator • Converter • Effect • Mixer • MusicFriday, February 1, 13
  • 63. Pull Model AUSomethingFriday, February 1, 13
  • 64. Pull Model AUSomething AudioUnitRender()Friday, February 1, 13
  • 65. Pull Model (connected) AUSomethingElse AUSomethingFriday, February 1, 13
  • 66. Buses (aka, Elements) AUSomethingElse AUSomething AUSomethingElseFriday, February 1, 13
  • 67. AUGraph AUSomethingElse AUSomething AUSomethingElseFriday, February 1, 13
  • 68. Render Callbacks OSStatus converterInputRenderCallback (void *inRefCon, AudioUnitRenderActionFlags *ioActionFlags, const AudioTimeStamp *inTimeStamp, UInt32 inBusNumber, UInt32 inNumberFrames, AudioBufferList * ioData) { CCFWebRadioPlayer *player = (__bridge CCFWebRadioPlayer*) inRefCon; // read from buffer ioData->mBuffers[0].mData = player.preRenderData; return noErr; } AUSomething AUSomethingElseFriday, February 1, 13
  • 69. AURemoteIO • Output unit used for play-out, capture • A Core Audio thread repeatedly and automatically calls AudioUnitRender() • Must set EnableIO property to explicitly enable capture and/or play- out • Capture requires setting appropriate AudioSession categoryFriday, February 1, 13
  • 70. Create AURemoteIO CheckError(NewAUGraph(&_auGraph), ! ! "couldnt create au graph"); ! CheckError(AUGraphOpen(_auGraph), ! ! "couldnt open au graph"); ! AudioComponentDescription componentDesc; componentDesc.componentType = kAudioUnitType_Output; componentDesc.componentSubType = kAudioUnitSubType_RemoteIO; componentDesc.componentManufacturer = kAudioUnitManufacturer_Apple; ! AUNode remoteIONode; CheckError(AUGraphAddNode(_auGraph, ! ! ! ! ! ! &componentDesc, ! ! ! ! ! ! &remoteIONode), ! ! "couldnt add remote io node");Friday, February 1, 13
  • 71. AURemoteIO Buses AURemoteIOFriday, February 1, 13
  • 72. AURemoteIO Buses AURemoteIO bus 0 to output H/WFriday, February 1, 13
  • 73. AURemoteIO Buses AURemoteIO bus 0 bus 0 from app to output H/WFriday, February 1, 13
  • 74. AURemoteIO Buses bus 1from input H/W AURemoteIO bus 0 bus 0 from app to output H/WFriday, February 1, 13
  • 75. AURemoteIO Buses bus 1 bus 1from input H/W to app AURemoteIO bus 0 bus 0 from app to output H/WFriday, February 1, 13
  • 76. Pass Through bus 1 from input H/W AURemoteIO bus 0 to output H/WFriday, February 1, 13
  • 77. Pass Through with Effect AUEffect bus 1 from input H/W AURemoteIO bus 0 to output H/WFriday, February 1, 13
  • 78. Demo Delay EffectFriday, February 1, 13
  • 79. AUNewTimePitch • New in iOS 6! • Allows you to change pitch independent of time, or time independent of pitch • How do you use it?Friday, February 1, 13
  • 80. AudioUnitParameters.h // Parameters for AUNewTimePitch enum { ! ! // Global, rate, 1/32 -> 32.0, 1.0 ! kNewTimePitchParam_Rate! ! ! ! ! ! = ! 0, ! ! // Global, Cents, -2400 -> 2400, 1.0 ! kNewTimePitchParam_Pitch! ! ! ! ! ! = 1, ! ! // Global, generic, 3.0 -> 32.0, 8.0 ! kNewTimePitchParam_Overlap! ! ! ! ! ! = 4, ! ! // Global, Boolean, 0->1, 1 ! kNewTimePitchParam_EnablePeakLocking! ! ! = 6 }; This is the entire documentation for the AUNewTimePitch parametersFriday, February 1, 13
  • 81. AUNewTimePitch Parameters • Rate: kNewTimePitchParam_Rate takes a Float32 rate from 1/32 speed to 32x speed. • Use powers of 2: 1/32, 1/16, …, 2, 4, 8… • Pitch: kNewTimePitchParam_Pitch takes a Float32 representing cents, meaning 1/100 of a musical semitoneFriday, February 1, 13
  • 82. Pitch Shifting • Pitch can vary, time does not • Suitable for real-time sources, such as audio captureFriday, February 1, 13
  • 83. Demo Pitch ShiftingFriday, February 1, 13
  • 84. Rate Shifting • Rate can vary, pitch does not • Think of 1.5x and 2x speed modes in Podcasts app • Not suitable for real-time sources, as data will be consumed faster. • Sources must be able to map time systems with kAudioUnitProperty_InputSamplesInOutputFriday, February 1, 13
  • 85. Demo Rate ShiftingFriday, February 1, 13
  • 86. Putting It Together • iOS 6’s new AudioQueueProcessingTap can call us back with decoded data right before it’s going to be played. • Can process that data and put it back in queue • Oh, what shall we do?Friday, February 1, 13
  • 87. Web Radio w/Effects MP3 MP3 MP3Friday, February 1, 13
  • 88. Web Radio w/Effects MP3 MP3 PCMFriday, February 1, 13
  • 89. Web Radio w/Effects MP3 MP3 PCMFriday, February 1, 13
  • 90. Web Radio w/Effects MP3 MP3 PCMFriday, February 1, 13
  • 91. Web Radio w/Effects MP3 MP3 PCMFriday, February 1, 13
  • 92. Demo Pitch-Shifted Web RadioFriday, February 1, 13
  • 93. Friday, February 1, 13
  • 94. Takeaways • You use Core Audio only when you need to do real-time processing • Audio Queue is easier, Audio Units are more powerful • OpenAL for gamesFriday, February 1, 13
  • 95. Want More?Friday, February 1, 13

×