Composing &Editing Media with AV Foundation http://bobmccune.com
About...Bob McCune‣ MN Developer and Instructor‣ Owner of TapHarmonic, LLC.‣ Founded Minnesota CocoaHeads in 2008
AgendaWhat will I learn? ‣ AV Foundation Overview ‣ Media Playback ‣ Media Editing ‣ Composing Media ‣ Mixing Audio ‣ Building Video Transitions ‣ Layering Visual Content
AV FoundationOverview ‣ Apple’s advanced Objective-C framework for working with timed-media ‣ High performance, asynchronous processing ‣ Hardware accelerated handling of AV media ‣ Available in its current form since iOS 4 ‣ Significant additions and enhancements iOS 5 and 6 ‣ Default media framework on Mac OS X since 10.7 Lion ‣ Apple’s focus for media apps on both iOS and Mac
Where does it fit?iOS Media Options Simple Awesome Complex MediaPlayer CoreAudio UIKit CoreMedia AVFoundation CoreVideo CoreAnimation
Media AssetsUnderstanding Assets‣ AVAsset is an abstract representation of media resource modeling the static aspects of the media. ‣ Abstracts away the type and location‣ AVAssetTrack models the static aspects of the individual media streams within an asset ‣ Tracks are of a uniform type (video, audio, etc.)AVAssetTrack (Video)AVAssetTrack (Audio)
Using AVAssetsWhat can I do with an asset? ‣ Inspect ‣ Generate Images ‣ Transcode and Export ‣ Playback
AVPlayerPlayback Controller‣ AVPlayer is a controller for managing playback ‣ play ‣ pause ‣ rate ‣ seekToTime:‣ Use KVO to observe playback readiness and state ‣ status‣ Timed Observations ‣ addPeriodicTimeObserverForInterval:queue:usingBlock ‣ addBoundaryTimeObserverForInterval:queue:usingBlock
Playing MediaStatic vs Dynamic Models ‣ AV Foundation distinguishes between static and dynamic aspects of media AVAsset AVAsset AVAsset AVAssetTrack St ati c AVPlayerItemTrack AVPlayerItem AVPlayerItemTrack AVPlayerItemTrack D y n amic
Understanding TimeCore Media EssentialsCMTime‣ Rational number representing time ‣ 64-bit integer time value (numerator) ‣ 32-bit integer time scale (denominator) CMTime fiveSeconds = CMTimeMake(5, 1); CMTime oneSample = CMTimeMake(1, 44100); CMTime zeroTime = kCMTimeZero;‣ Large number of utility functions in Core Media: ‣ CMTimeAdd,CMTimeSubtract,CMTimeCompare, etc.
Understanding TimeCore Media EssentialsCMTimeRange‣ Core Media struct containing start time and duration CMTimeRange assetRange = CMTimeRangeMake(kCMTimeZero, asset.duration); CMTimeRange zeroRange = kCMTimeRangeZero;‣ Like CMTime, there are many Core Media functions: ‣ CMTimeRangeEqual, CMTimeRangeContainsTime, CMTimeRangeGetEnd, CMTIMERANGE_ISVALID, etc.
Audio MixingAVAudioMix‣ Composition tracks play at their natural volume‣ AVAudioMix applies track-level volume adjustments ‣ Composed of AVAudioMixInputParameters ‣ Parameters control individual track volume over time CMTime CMTimeRange
Video TransitionsAVVideoComposition AVVideoComposition Deﬁnes how two or more video tracks are composited together AVVideoCompositionInstruction Conﬁgured through collection of AVVideoCompositionInstruction AVVideoCompositionInstruction composition instructions describing compositing behavior
Video TransitionsAVVideoCompositionInstruction AVVideoComposition AVVideoCompositionInstruction AVVideoCompositionInstruction AVVideoCompositionInstruction Deﬁnes the time range of compositing behavior Composed of layer instructions AVAsset AVAsset AVVideoCompositionLayerInstruction describing compositing behavior
Video TransitionsAVVideoCompositionLayerInstruction AVVideoComposition AVVideoCompositionInstruction AVVideoCompositionInstruction AVVideoCompositionInstruction AVAsset AVAsset Deﬁnes the transform and AVVideoCompositionLayerInstruction opacity ramps of input layers Transform and opacity changes modiﬁed over given time range
Building Transitions 2Define Overlapping RegionsAB
Building Transitions 3Define Time RangesA Passthrough Passthrough PassthroughB Transition Transition * Time ranges must not have gaps or overlap * Total duration must not be shorter than composition
Building Transitions 5Set sizes and apply#define FRAME_RATE CMTimeMake(1, 30)#define RENDER_SIZE CGSizeMake(1280, 720)AVMutableVideoComposition *videoComposition = [AVMutableVideoComposition videoComposition];// Set instructions on AVVideoComposition instancevideoComposition.instructions = instructions;videoComposition.frameDuration = FRAME_RATE;videoComposition.renderSize = RENDER_SIZE;AVPlayerItem *playerItem = [AVPlayerItem playerItemWithAsset:[composition copy]];playerItem.videoComposition = videoComposition;
New in iOS 6AVVideoComposition videoCompositionWithPropertiesOfAsset: Automagic Setup:• Calculates all required passthrough and transition time ranges• Builds appropriate composition and layer instructions for time ranges• Sets the appropriate render size• Sets the appropriate frame rate
Layering ContentCore AnimationCore Animation a natural choice ‣ High performance, inherently time-based ‣ CALayer subclasses used for all video rendering CALayer: used to layer images and text CAAnimation: used to animate layered content CABasicAnimation CAKeyframeAnimation
Animation TimingAVSynchronizedLayer‣ Core Animation operates on host time ‣ Starts at boot, marches towards infinity‣ Timeline animations need to use movie time ‣ Starts at kCMTimeZero and runs to duration ‣ Can be started, stopped, rewound, etc.‣ Use AVSynchronizedLayer to use movie time ‣ Confers player item timing on to its sublayer tree AVPlayerItem AVSynchronizedLayer CATextLayer CABasicAnimation
Core Animation Timeline vs Realtime Animations‣ Exactly the same, but diﬀerent: ‣ Animations with zero beginTime won’t be seen ‣ Set beginTime = AVCoreAnimationBeginTimeZero ‣ Animations removed by default ‣ Set removedOnCompletion = NO ‣ Unable to use CAAnimationGroup?
Core AnimationNatural Choice, Awkward Implementation‣ Diﬀerent conceptual models for timeline editing ‣ CMTime and CMTimeRange for asset items ‣ Seconds and milliseconds for layers ‣ Build abstraction to help bridge the gap‣ Usage diﬀers in playback and export scenarios ‣ AVSynchronizedLayer for playback ‣ Attach to player’s view hierarchy/layer tree ‣ AVVideoCompositionCoreAnimationTool for export
SummaryAV Foundation Rocks!‣ Powerful tools for audio and video playback ‣ AVPlayer, AVPlayerItem, AVPlayerLayer‣ Powerful tools for composing/editing media: ‣ AVComposition, AVAudioMix, AVVideoComposition, AVSynchronizedLayer‣ Powerful utility classes: ‣ AVAssetImageGenerator ‣ AVExportSession‣ Steep learning curve, but worth the investment!
ResourcesPresentation Materialshttp://www.speakerdeck.com/bobmccune/http://www.slideshare.net/bobmccune/https://github.com/tapharmonic/AVFoundationEditorLearning AV Foundationhttp://www.speakerdeck.com/bobmccune/https://github.com/tapharmonic/AVFoundationDemosWWDC 2011: Exploring AV Foundationhttps://developer.apple.com/videos/wwdc/2011/?id=405WWDC 2011: Working with Media in AV Foundationhttps://developer.apple.com/videos/wwdc/2011/?id=415 BobMcCune.com @bobmccune