Composing and Editing Media with AV Foundation


Published on

Presentation given at the CocoaConf DC.

  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Composing and Editing Media with AV Foundation

  1. 1. Composing &Editing Media with AV Foundation
  2. 2. About...Bob McCune‣ MN Developer and Instructor‣ Owner of TapHarmonic, LLC.‣ Founded Minnesota CocoaHeads in 2008
  3. 3. AgendaWhat will I learn? ‣ AV Foundation Overview ‣ Media Playback ‣ Media Editing ‣ Composing Media ‣ Mixing Audio ‣ Building Video Transitions ‣ Layering Visual Content
  4. 4. AV FoundationOverview ‣ Apple’s advanced Objective-C framework for working with timed-media ‣ High performance, asynchronous processing ‣ Hardware accelerated handling of AV media ‣ Available in its current form since iOS 4 ‣ Significant additions and enhancements iOS 5 and 6 ‣ Default media framework on Mac OS X since 10.7 Lion ‣ Apple’s focus for media apps on both iOS and Mac
  5. 5. Where does it fit?iOS Media Options Simple Awesome Complex MediaPlayer CoreAudio UIKit CoreMedia AVFoundation CoreVideo CoreAnimation
  6. 6. AV Essentials
  7. 7. Media AssetsUnderstanding Assets‣ AVAsset is an abstract representation of media resource modeling the static aspects of the media. ‣ Abstracts away the type and location‣ AVAssetTrack models the static aspects of the individual media streams within an asset ‣ Tracks are of a uniform type (video, audio, etc.)AVAssetTrack (Video)AVAssetTrack (Audio)
  8. 8. Using AVAssetsWhat can I do with an asset? ‣ Inspect ‣ Generate Images ‣ Transcode and Export ‣ Playback
  9. 9. Media Playback
  10. 10. AVPlayerPlayback Controller‣ AVPlayer is a controller for managing playback ‣ play ‣ pause ‣ rate ‣ seekToTime:‣ Use KVO to observe playback readiness and state ‣ status‣ Timed Observations ‣ addPeriodicTimeObserverForInterval:queue:usingBlock ‣ addBoundaryTimeObserverForInterval:queue:usingBlock
  11. 11. Playing MediaStatic vs Dynamic Models ‣ AV Foundation distinguishes between static and dynamic aspects of media AVAsset AVAsset AVAsset AVAssetTrack St ati c AVPlayerItemTrack AVPlayerItem AVPlayerItemTrack AVPlayerItemTrack D y n amic
  12. 12. Understanding TimeCore Media EssentialsCMTime‣ Rational number representing time ‣ 64-bit integer time value (numerator) ‣ 32-bit integer time scale (denominator) CMTime fiveSeconds = CMTimeMake(5, 1); CMTime oneSample = CMTimeMake(1, 44100); CMTime zeroTime = kCMTimeZero;‣ Large number of utility functions in Core Media: ‣ CMTimeAdd,CMTimeSubtract,CMTimeCompare, etc.
  13. 13. Understanding TimeCore Media EssentialsCMTimeRange‣ Core Media struct containing start time and duration CMTimeRange assetRange = CMTimeRangeMake(kCMTimeZero, asset.duration); CMTimeRange zeroRange = kCMTimeRangeZero;‣ Like CMTime, there are many Core Media functions: ‣ CMTimeRangeEqual, CMTimeRangeContainsTime, CMTimeRangeGetEnd, CMTIMERANGE_ISVALID, etc.
  14. 14. Video PlaybackAVPlayerLayer AVAsset AVPlayerItem AVPlayer AVAsset AVAsset AVPlayerItemTrack AVPlayerItemTrack AVAssetTrack AVPlayerItemTrack
  15. 15. Video PlaybackAVPlayerLayer AVPlayerItem AVPlayer AVPlayerLayer AVPlayerItemTrack AVPlayerItemTrack AVPlayerItemTrack
  16. 16. Demo
  17. 17. Composing Media
  18. 18. Composing AssetsAVComposition‣ Concrete extension of AVAsset‣ Composes asset segments on a timeline
  19. 19. Composing AssetsTracks and SegmentsAVMutableComposition *composition = [AVMutableComposition composition]; AVComposition
  20. 20. Composing AssetsTracks and SegmentsCMPersistentTrackID trackID = kCMPersistentTrackID_Invalid;AVMutableCompositionTrack *videoTrack = [composition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:trackID];AVMutableCompositionTrack *audioTrack = [composition addMutableTrackWithMediaType:AVMediaTypeAudio preferredTrackID:trackID]; AVComposition AVCompositionTrack (Video) AVCompositionTrack (Audio)
  21. 21. Composing AssetsTracks and SegmentsAVAssetTrack *srcVideoTrack1 = // source video track 1[videoTrack insertTimeRange:timeRange ofTrack:srcVideoTrack1 atTime:startTime error:&error];AVAssetTrack *srcVideoTrack2 = // source video track 2[videoTrack insertTimeRange:timeRange ofTrack:srcVideoTrack2 atTime:startTime error:&error];AVAssetTrack *srcAudioTrack = // source audio track[audioTrack insertTimeRange:timeRange ofTrack:srcAudioTrack atTime:startTime error:&error]; AVComposition AVCompositionTrack (Video) AVCompositionTrackSegment AVCompositionTrackSegment Seconds 10-30 of “redpanda.m4v” Seconds 20-60 of “waves.m4v” AVCompositionTrack (Audio) AVCompositionTrackSegment Seconds 0-60 of “soundtrack.mp3”
  22. 22. Demo
  23. 23. Mixing Audio
  24. 24. Audio MixingAVAudioMix‣ Composition tracks play at their natural volume‣ AVAudioMix applies track-level volume adjustments ‣ Composed of AVAudioMixInputParameters ‣ Parameters control individual track volume over time CMTime CMTimeRange
  25. 25. AVAudioMixExampleCMTime fadeTime = CMTimeMake(5, 1);CMTime fadeInStartTime = kCMTimeZero;CMTime fadeOutStartTime = CMTimeSubtract(asset.duration, fadeTime);CMTimeRange fadeInRange = CMTimeRangeMake(fadeInStartTime, fadeTime);CMTimeRange fadeOutRange = CMTimeRangeMake(fadeOutStartTime, fadeTime);AVMutableAudioMixInputParameters *parameters = [AVMutableAudioMixInputParameters audioMixInputParameters];parameters.trackID = assetTrack.trackID;[parameters setVolumeRampFromStartVolume:0.0 toEndVolume:1.0 timeRange:fadeInRange];[parameters setVolumeRampFromStartVolume:1.0 toEndVolume:0.0 timeRange:fadeOutRange];AVMutableAudioMix *audioMix = [AVMutableAudioMix audioMix];audioMix.inputParameters = @[parameters];playerItem.audioMix = audioMix;
  26. 26. Demo
  27. 27. Video Transitions
  28. 28. Video TransitionsAVVideoComposition AVVideoComposition Defines how two or more video tracks are composited together AVVideoCompositionInstruction Configured through collection of AVVideoCompositionInstruction AVVideoCompositionInstruction composition instructions describing compositing behavior
  29. 29. Video TransitionsAVVideoCompositionInstruction AVVideoComposition AVVideoCompositionInstruction AVVideoCompositionInstruction AVVideoCompositionInstruction Defines the time range of compositing behavior Composed of layer instructions AVAsset AVAsset AVVideoCompositionLayerInstruction describing compositing behavior
  30. 30. Video TransitionsAVVideoCompositionLayerInstruction AVVideoComposition AVVideoCompositionInstruction AVVideoCompositionInstruction AVVideoCompositionInstruction AVAsset AVAsset Defines the transform and AVVideoCompositionLayerInstruction opacity ramps of input layers Transform and opacity changes modified over given time range
  31. 31. Building TransitionsConceptual Steps
  32. 32. Building Transitions 1Stagger LayoutAB
  33. 33. Building Transitions 2Define Overlapping RegionsAB
  34. 34. Building Transitions 3Define Time RangesA Passthrough Passthrough PassthroughB Transition Transition * Time ranges must not have gaps or overlap * Total duration must not be shorter than composition
  35. 35. Building Transitions 4Configure Instructions// Build transition instructionsAVMutableVideoCompositionInstruction *transitionInstruction = ...;transitionInstruction.timeRange = transitionTimeRange;AVMutableVideoCompositionLayerInstruction *fromLayerInstruction = ...;AVMutableVideoCompositionLayerInstruction *toLayerInstruction = ...;// Cross Disolve[fromLayerInstruction setOpacityRampFromStartOpacity:1.0 toEndOpacity:0.0 timeRange:transitionTimeRange];NSArray *instructions = @[fromLayerInstruction, toLayerInstruction];transitionInstruction.layerInstructions = instructions;[instructions addObject:transitionInstruction];
  36. 36. Building Transitions 5Set sizes and apply#define FRAME_RATE CMTimeMake(1, 30)#define RENDER_SIZE CGSizeMake(1280, 720)AVMutableVideoComposition *videoComposition = [AVMutableVideoComposition videoComposition];// Set instructions on AVVideoComposition instancevideoComposition.instructions = instructions;videoComposition.frameDuration = FRAME_RATE;videoComposition.renderSize = RENDER_SIZE;AVPlayerItem *playerItem = [AVPlayerItem playerItemWithAsset:[composition copy]];playerItem.videoComposition = videoComposition;
  37. 37. New in iOS 6AVVideoComposition videoCompositionWithPropertiesOfAsset: Automagic Setup:• Calculates all required passthrough and transition time ranges• Builds appropriate composition and layer instructions for time ranges• Sets the appropriate render size• Sets the appropriate frame rate
  38. 38. Demo
  39. 39. Layering Content
  40. 40. Layering ContentCore AnimationCore Animation a natural choice ‣ High performance, inherently time-based ‣ CALayer subclasses used for all video rendering CALayer: used to layer images and text CAAnimation: used to animate layered content CABasicAnimation CAKeyframeAnimation
  41. 41. Animation TimingAVSynchronizedLayer‣ Core Animation operates on host time ‣ Starts at boot, marches towards infinity‣ Timeline animations need to use movie time ‣ Starts at kCMTimeZero and runs to duration ‣ Can be started, stopped, rewound, etc.‣ Use AVSynchronizedLayer to use movie time ‣ Confers player item timing on to its sublayer tree AVPlayerItem AVSynchronizedLayer CATextLayer CABasicAnimation
  42. 42. Core Animation Timeline vs Realtime Animations‣ Exactly the same, but different: ‣ Animations with zero beginTime won’t be seen ‣ Set beginTime = AVCoreAnimationBeginTimeZero ‣ Animations removed by default ‣ Set removedOnCompletion = NO ‣ Unable to use CAAnimationGroup?
  43. 43. Core AnimationNatural Choice, Awkward Implementation‣ Different conceptual models for timeline editing ‣ CMTime and CMTimeRange for asset items ‣ Seconds and milliseconds for layers ‣ Build abstraction to help bridge the gap‣ Usage differs in playback and export scenarios ‣ AVSynchronizedLayer for playback ‣ Attach to player’s view hierarchy/layer tree ‣ AVVideoCompositionCoreAnimationTool for export
  44. 44. Demo
  45. 45. SummaryAV Foundation Rocks!‣ Powerful tools for audio and video playback ‣ AVPlayer, AVPlayerItem, AVPlayerLayer‣ Powerful tools for composing/editing media: ‣ AVComposition, AVAudioMix, AVVideoComposition, AVSynchronizedLayer‣ Powerful utility classes: ‣ AVAssetImageGenerator ‣ AVExportSession‣ Steep learning curve, but worth the investment!
  46. 46. ResourcesPresentation Materials AV Foundation 2011: Exploring AV Foundation 2011: Working with Media in AV Foundation @bobmccune