Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Forward Swift 2017: Media Frameworks and Swift: This Is Fine

3,349 views

Published on

Swift is great for writing iOS and Mac apps, and its creators also mean for it to be used as a systems programming language. However, certain traits about Swift make it officially off-limits for use in some audio/video-processing scenarios. What's the deal, is it not fast enough or what? We'll look at what media apps can and can't do in Swift, and what you're supposed to do instead. We'll also look at strategies for knowing what responsibilities to dole out to Swift and to C, and how to make those parts of your code play nicely with each other.

Published in: Technology
  • Be the first to like this

Forward Swift 2017: Media Frameworks and Swift: This Is Fine

  1. 1. Media Frameworks and Swift: This is Fine Chris Adamson • @invalidname Forward Swift, March 2017
  2. 2. Who the what, now? @invalidname
  3. 3. import Cocoa import AVFoundation import CoreMediaIO if let devices = AVCaptureDevice.devices(), let avDevices = devices.filter( {$0 is AVCaptureDevice}) as? [AVCaptureDevice] { for device in avDevices { print("(device.description)") } }
  4. 4. <AVCaptureHALDevice: 0x100b16ab0 [Loopback Simulator][com.rogueamoeba.Loopback:E8577B20-0806-4472-A5E6-426CABCD6C8E]> <AVCaptureHALDevice: 0x100c1a7c0 [Loopback Line-In][com.rogueamoeba.Loopback:A00F38FD-C2B6-43FD-98B7-23BAA6FACB03]> <AVCaptureHALDevice: 0x100c16910 [iMic USB audio system][AppleUSBAudioEngine:Griffin Technology, Inc:iMic USB audio system:220000:2,1]> <AVCaptureHALDevice: 0x100d13900 [Loopback Keynote][com.rogueamoeba.Loopback:1936D2A3-6D0B-428E-899E-0ABE46628EA4]> <AVCaptureHALDevice: 0x100a26850 [Soundflower (64ch)][SoundflowerEngine:1]> <AVCaptureHALDevice: 0x100a26310 [HD Pro Webcam C920][AppleUSBAudioEngine:Unknown Manufacturer:HD Pro Webcam C920:1218B05F:3]> <AVCaptureHALDevice: 0x100d13660 [Soundflower (2ch)][SoundflowerEngine:0]> <AVCaptureDALDevice: 0x100a348f0 [iGlasses][iGlasses]> <AVCaptureDALDevice: 0x100a28d00 [HD Pro Webcam C920][0x244000046d082d]> Program ended with exit code: 0
  5. 5. CMIOObjectPropertyAddress prop = { kCMIOHardwarePropertyAllowScreenCaptureDevices, kCMIOObjectPropertyScopeGlobal, kCMIOObjectPropertyElementMaster }; UInt32 allow = 1; CMIOObjectSetPropertyData( kCMIOObjectSystemObject, &prop, 0, NULL, sizeof(allow), &allow );
  6. 6. var prop = CMIOObjectPropertyAddress( mSelector: CMIOObjectPropertySelector( kCMIOHardwarePropertyAllowScreenCaptureDevices), mScope: CMIOObjectPropertyScope(kCMIOObjectPropertyScopeGlobal), mElement: CMIOObjectPropertyElement( kCMIOObjectPropertyElementMaster)) var allow : UInt32 = 1 CMIOObjectSetPropertyData(CMIOObjectID(kCMIOObjectSystemObject), &prop, 0, nil, UInt32(MemoryLayout<UInt32>.size), &allow)
  7. 7. CMIOObjectPropertyAddress prop = { kCMIOHardwarePropertyAllowScreenCaptureDevices, kCMIOObjectPropertyScopeGlobal, kCMIOObjectPropertyElementMaster }; UInt32 allow = 1; CMIOObjectSetPropertyData( kCMIOObjectSystemObject, &prop, 0, NULL, sizeof(allow), &allow ); var prop = CMIOObjectPropertyAddress( mSelector: CMIOObjectPropertySelector( kCMIOHardwarePropertyAllowScreenCaptureDevices), mScope: CMIOObjectPropertyScope(kCMIOObjectPropertyScopeGlobal), mElement: CMIOObjectPropertyElement( kCMIOObjectPropertyElementMaster)) var allow : UInt32 = 1 CMIOObjectSetPropertyData(CMIOObjectID(kCMIOObjectSystemObject), &prop, 0, nil, UInt32(MemoryLayout<UInt32>.size), &allow) This is fine
  8. 8. var prop = CMIOObjectPropertyAddress( mSelector: CMIOObjectPropertySelector( kCMIOHardwarePropertyAllowScreenCaptureDevices), mScope: CMIOObjectPropertyScope( kCMIOObjectPropertyScopeGlobal), mElement: CMIOObjectPropertyElement( kCMIOObjectPropertyElementMaster)) var prop = CMIOObjectPropertyAddress( mSelector: CMIOObjectPropertySelector( kCMIOHardwarePropertyAllowScreenCaptureDevices), mScope: CMIOObjectPropertyScope( kCMIOObjectPropertyScopeGlobal), mElement: CMIOObjectPropertyElement( kCMIOObjectPropertyElementMaster))
  9. 9. public typealias CMIOObjectPropertySelector = UInt32 public typealias CMIOObjectPropertyScope = UInt32 public typealias CMIOObjectPropertyElement = UInt32 public struct CMIOObjectPropertyAddress { public var mSelector: CMIOObjectPropertySelector public var mScope: CMIOObjectPropertyScope public var mElement: CMIOObjectPropertyElement public init() public init(mSelector: CMIOObjectPropertySelector, mScope: CMIOObjectPropertyScope, mElement: CMIOObjectPropertyElement) }
  10. 10. extension CMIOObjectPropertySelector { static let allowScreenCaptureDevices = CMIOObjectPropertySelector( kCMIOHardwarePropertyAllowScreenCaptureDevices) } extension CMIOObjectPropertyScope { static let global = CMIOObjectPropertyScope(kCMIOObjectPropertyScopeGlobal) } extension CMIOObjectPropertyElement { static let master = CMIOObjectPropertyElement(kCMIOObjectPropertyElementMaster) }
  11. 11. var prop = CMIOObjectPropertyAddress( mSelector: .allowScreenCaptureDevices, mScope: .global, mElement: .master) var prop = CMIOObjectPropertyAddress( mSelector: CMIOObjectPropertySelector( kCMIOHardwarePropertyAllowScreenCaptureDevices), mScope: CMIOObjectPropertyScope( kCMIOObjectPropertyScopeGlobal), mElement: CMIOObjectPropertyElement( kCMIOObjectPropertyElementMaster))
  12. 12. Demo http://github.com/invalidstream/audio-reverser
  13. 13. Reversing Audio 1. Decode the MP3/AAC to LPCM2. Grab a buffer from the end3. Reverse its samples in memory4. Write it to the front of a new file5. Repeat until fully baked
  14. 14. API Needs • Convert from MP3/AAC to LPCM • Write sequentially to audio file (.caf, .aif, .wav) • Random-access read from audio file
  15. 15. Plan A (Swift) • AV Foundation • AVAssetReader/Writer can do format conversion while reading/writing audio files • Can’t (easily) read from arbitrary packet offsets; meant to process everything forward
  16. 16. Plan B (C, Swift?) • Audio Toolbox (part of Core Audio) • ExtAudioFile can do format conversions while reading/writing audio files • AudioFile can read from arbitrary packet offset
  17. 17. // declare LPCM format we are converting to AudioStreamBasicDescription format = {0}; format.mSampleRate = 44100.0; format.mFormatID = kAudioFormatLinearPCM; format.mFormatFlags = kAudioFormatFlagIsPacked | kAudioFormatFlagIsSignedInteger; format.mBitsPerChannel = 16; format.mChannelsPerFrame = 2; format.mBytesPerFrame = 4; format.mFramesPerPacket = 1; format.mBytesPerPacket = 4;
  18. 18. // declare LPCM format we are converting to var format = AudioStreamBasicDescription( mSampleRate: 44100.0, mFormatID: kAudioFormatLinearPCM, mFormatFlags: kAudioFormatFlagIsPacked + kAudioFormatFlagIsSignedInteger, mBytesPerPacket: 4, mFramesPerPacket: 1, mBytesPerFrame: 4, mChannelsPerFrame: 2, mBitsPerChannel: 16, mReserved: 0)
  19. 19. // open AudioFile for output AudioFileID forwardAudioFile; err = AudioFileCreateWithURL(forwardURL, kAudioFileCAFType, &format, kAudioFileFlags_EraseFile, &forwardAudioFile); IF_ERR_RETURN #define IF_ERR_RETURN if (err != noErr) { return err; }
  20. 20. // open AudioFile for output var forwardAudioFile: AudioFileID? err = AudioFileCreateWithURL(forwardURL, kAudioFileCAFType, &format, AudioFileFlags.eraseFile, &forwardAudioFile) if err != noErr { return err }
  21. 21. // open AudioFile for output var forwardAudioFile: AudioFileID? err = AudioFileCreateWithURL(forwardURL, kAudioFileCAFType, &format, AudioFileFlags.eraseFile, &forwardAudioFile) if err != noErr { return err }
  22. 22. // open AudioFile for output var forwardAudioFile: AudioFileID? err = AudioFileCreateWithURL(forwardURL, kAudioFileCAFType, &format, AudioFileFlags.eraseFile, &forwardAudioFile) if err != noErr { return err } 1. Uses a free function, rather than a method on AudioFile
  23. 23. // open AudioFile for output var forwardAudioFile: AudioFileID? err = AudioFileCreateWithURL(forwardURL, kAudioFileCAFType, &format, AudioFileFlags.eraseFile, &forwardAudioFile) if err != noErr { return err } 2. Errors are communicated via the return value, rather than throws
  24. 24. // open AudioFile for output var forwardAudioFile: AudioFileID? err = AudioFileCreateWithURL(forwardURL, kAudioFileCAFType, &format, AudioFileFlags.eraseFile, &forwardAudioFile) if err != noErr { return err } 3. Some parameters are UInt32 constants, some are enums
  25. 25. // open AudioFile for output var forwardAudioFile: AudioFileID? err = AudioFileCreateWithURL(forwardURL, kAudioFileCAFType, &format, AudioFileFlags.eraseFile, &forwardAudioFile) if err != noErr { return err } 4. Audio format is passed as an UnsafePointer<AudioStreamBasicDescription>
  26. 26. // open AudioFile for output var forwardAudioFile: AudioFileID? err = AudioFileCreateWithURL(forwardURL, kAudioFileCAFType, &format, AudioFileFlags.eraseFile, &forwardAudioFile) if err != noErr { return err } 5. Created object is returned via an in-out parameter
  27. 27. To say nothing of…
  28. 28. Pointer arithmetic! // swap packets inside transfer buffer for i in 0..<packetsToTransfer/2 { let swapSrc = transferBuffer.advanced(by: Int(i) * Int(format.mBytesPerPacket)) let swapDst = transferBuffer.advanced(by: transferBufferSize - (Int(i+1) * Int(format.mBytesPerPacket))) memcpy(swapBuffer, swapSrc, Int(format.mBytesPerPacket)) memcpy(swapSrc, swapDst, Int(format.mBytesPerPacket)) memcpy(swapDst, swapBuffer, Int(format.mBytesPerPacket)) }
  29. 29. Couldn’t you just…
  30. 30. extension AudioFileID { init? (url: URL, fileType: UInt32, format: AudioStreamBasicDescription, flags: AudioFileFlags) { var fileId : AudioFileID? var format = format let err = AudioFileCreateWithURL(url as CFURL, fileType, &format, flags, &fileId) guard err != noErr, let createdFile = fileId else { return nil } self = createdFile } }
  31. 31. Been there, done that • The Amazing Audio Engine 💀 • Novocaine (💀?) • EZAudio 💀 • AudioKit • Superpowered • etc…
  32. 32. /** Convert a source audio file (using any Core Audio-supported codec) and create LPCM .caf files for its forward and backward versions. - parameter sourceURL: A file URL containing the source audio to be read from - parameter forwardURL: A file URL with the destination to write the decompressed (LPCM) forward file - parameter backwardURL: A file URL with the destination to write the backward file */ OSStatus convertAndReverse(CFURLRef sourceURL, CFURLRef forwardURL, CFURLRef backwardURL); AudioReversingC.h // // Use this file to import your target's public headers that you would like to expose to Swift. // #import <CoreFoundation/CoreFoundation.h> #import <AudioToolbox/AudioToolbox.h> OSStatus convertAndReverse(CFURLRef sourceURL, CFURLRef forwardURL, CFURLRef backwardURL); AudioReverser-Bridging-Header.h
  33. 33. if USE_SWIFT_CONVERTER { err = convertAndReverseSwift(sourceURL: source as CFURL, forwardURL: self.forwardURL as! CFURL, backwardURL: self.backwardURL as! CFURL) } else { err = convertAndReverse(source as! CFURL, self.forwardURL as! CFURL, self.backwardURL as! CFURL) }
  34. 34. C APIs on iOS/macOS • Core Foundation • Core Audio • Core Media • Video Toolbox • Keychain • IOKit • OpenGL • SQLite • Accelerate • XPC • BSD, Mach • etc…
  35. 35. Going deeper…
  36. 36. Audio Units • Discrete software objects for working with audio • Generators, I/O, Filters/Effects, Mixers, Converters • Typically combined in a “graph” model • Used by Garage Band, Logic, etc.
  37. 37. Wait, what?
  38. 38. -(instancetype)initWithComponentDescription:(AudioComponentDescription)componentDescription
 options:(AudioComponentInstantiationOptions)options
 error:(NSError **)outError { self = [super initWithComponentDescription:componentDescription
 options:options error:outError]; if (self == nil) { return nil; } // ... return self; } MyAudioUnit.m
  39. 39. XPC (macOS only)
  40. 40. Swift 3Swift 4
  41. 41. https://github.com/apple/swift/blob/master/docs/ ABIStabilityManifesto.md
  42. 42. “Given the importance of getting the core ABI and the related fundamentals correct, we are going to defer the declaration of ABI stability out of Swift 4 while still focusing the majority of effort to get to the point where the ABI can be declared stable.” —Ted Kremenek, Feb. 16, 2017
 “Swift 4, stage 2 starts now” https://lists.swift.org/pipermail/swift-evolution/Week-of- Mon-20170213/032116.html
  43. 43. // Block which subclassers must provide to implement rendering. - (AUInternalRenderBlock)internalRenderBlock { // Capture in locals to avoid Obj-C member lookups. // If "self" is captured in render, we're doing it wrong. See sample code. return ^AUAudioUnitStatus(AudioUnitRenderActionFlags *actionFlags, const AudioTimeStamp *timestamp, AVAudioFrameCount frameCount, NSInteger outputBusNumber, AudioBufferList *outputData, const AURenderEvent *realtimeEventListHead, AURenderPullInputBlock pullInputBlock) { // Do event handling and signal processing here. return noErr; }; }
  44. 44. Don’t do this • An audio unit’s render block is called on a realtime thread • Therefore it cannot perform any action that could block: • I/O (file or network) • Waiting on a mutex or semaphore
  45. 45. Also, don’t do this • Call objc_msg_send() • Capture any Objective-C or Swift object • Allocate memory Basically, if you touch anything in the block other than a pre- allocated C struct, you’re asking for trouble.
  46. 46. https://github.com/apple/swift/blob/master/docs/ OwnershipManifesto.md
  47. 47. Certain kinds of low-level programming require stricter performance guarantees. Often these guarantees are less about absolute performance than predictable performance. For example, keeping up with an audio stream is not a taxing job for a modern processor, even with significant per- sample overheads, but any sort of unexpected hiccup is immediately noticeable by users. —“Swift Ownership Manifesto”,
 February 2017
  48. 48. We believe that these problems can be addressed with an opt-in set of features that we collectively call ownership. […] Swift already has an ownership system, but it's "under the covers": it's an implementation detail that programmers have little ability to influence. What we are proposing here is easy to summarize: • We should add a core rule to the ownership system, called the Law of Exclusivity […] • We should add features to give programmers more control over the ownership system […] • We should add features to allow programmers to express types with unique ownership […]
  49. 49. And yet…
  50. 50. “[Swift] is the first industrial-quality systems programming language that is as expressive and enjoyable as a scripting language.” https://developer.apple.com/library/content/documentation/ Swift/Conceptual/Swift_Programming_Language/
  51. 51. So… when?
  52. 52. Waiting… • ABI stability — will not be in Swift 4 • Ownership — unclear • Are these traits sufficient?
  53. 53. Strategies • Use AV Foundation if you can • Learn to balance C and Swift • “Render undo Caesar what is Caesar’s…” • The goal is to have idiomatic Swift, not Swift that may work but looks like C
  54. 54. Media Frameworks and Swift: This is Fine Chris Adamson • @invalidname Forward Swift, March 2017

×