Presentation given at the CocoaConf DC.

1. Composing &
Editing Media with
AV Foundation
http://bobmccune.com

2. About...
Bob McCune
‣ MN Developer and Instructor
‣ Owner of TapHarmonic, LLC.
‣ Founded Minnesota CocoaHeads in 2008

3. Agenda
What will I learn?
‣ AV Foundation Overview
‣ Media Playback
‣ Media Editing
‣ Composing Media
‣ Mixing Audio
‣ Building Video Transitions
‣ Layering Visual Content

4. AV Foundation
Overview
‣ Apple’s advanced Objective-C framework for
working with timed-media
‣ High performance, asynchronous processing
‣ Hardware accelerated handling of AV media
‣ Available in its current form since iOS 4
‣ Significant additions and enhancements iOS 5 and 6
‣ Default media framework on Mac OS X since 10.7 Lion
‣ Apple’s focus for media apps on both iOS and Mac

5. Where does it fit?
iOS Media Options
Simple Awesome Complex
MediaPlayer CoreAudio
UIKit CoreMedia
AVFoundation
CoreVideo
CoreAnimation

6. AV Essentials

7. Media Assets
Understanding Assets
‣ AVAsset is an abstract representation of media resource
modeling the static aspects of the media.
‣ Abstracts away the type and location
‣ AVAssetTrack models the static aspects of the
individual media streams within an asset
‣ Tracks are of a uniform type (video, audio, etc.)
AVAssetTrack (Video)
AVAssetTrack (Audio)

8. Using AVAssets
What can I do with an asset?
‣ Inspect
‣ Generate Images
‣ Transcode and Export
‣ Playback

9. Media Playback

10. AVPlayer
Playback Controller
‣ AVPlayer is a controller for managing playback
‣ play
‣ pause
‣ rate
‣ seekToTime:
‣ Use KVO to observe playback readiness and state
‣ status
‣ Timed Observations
‣ addPeriodicTimeObserverForInterval:queue:usingBlock
‣ addBoundaryTimeObserverForInterval:queue:usingBlock

11. Playing Media
Static vs Dynamic Models
‣ AV Foundation distinguishes between static
and dynamic aspects of media
AVAsset
AVAsset AVAsset
AVAssetTrack
St ati c
AVPlayerItemTrack
AVPlayerItem AVPlayerItemTrack
AVPlayerItemTrack
D y n amic

12. Understanding Time
Core Media Essentials
CMTime
‣ Rational number representing time
‣ 64-bit integer time value (numerator)
‣ 32-bit integer time scale (denominator)
CMTime fiveSeconds = CMTimeMake(5, 1);
CMTime oneSample = CMTimeMake(1, 44100);
CMTime zeroTime = kCMTimeZero;
‣ Large number of utility functions in Core Media:
‣ CMTimeAdd,CMTimeSubtract,CMTimeCompare, etc.

13. Understanding Time
Core Media Essentials
CMTimeRange
‣ Core Media struct containing start time and duration
CMTimeRange assetRange = CMTimeRangeMake(kCMTimeZero, asset.duration);
CMTimeRange zeroRange = kCMTimeRangeZero;
‣ Like CMTime, there are many Core Media functions:
‣ CMTimeRangeEqual, CMTimeRangeContainsTime,
CMTimeRangeGetEnd, CMTIMERANGE_ISVALID, etc.

14. Video Playback
AVPlayerLayer
AVAsset AVPlayerItem AVPlayer
AVAsset
AVAsset AVPlayerItemTrack
AVPlayerItemTrack
AVAssetTrack AVPlayerItemTrack

15. Video Playback
AVPlayerLayer
AVPlayerItem AVPlayer AVPlayerLayer
AVPlayerItemTrack
AVPlayerItemTrack
AVPlayerItemTrack

16. Demo

17. Composing Media

18. Composing Assets
AVComposition
‣ Concrete extension of AVAsset
‣ Composes asset segments on a timeline

19. Composing Assets
Tracks and Segments
AVMutableComposition *composition = [AVMutableComposition composition];
AVComposition

20. Composing Assets
Tracks and Segments
CMPersistentTrackID trackID = kCMPersistentTrackID_Invalid;
AVMutableCompositionTrack *videoTrack =
[composition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:trackID];
AVMutableCompositionTrack *audioTrack =
[composition addMutableTrackWithMediaType:AVMediaTypeAudio preferredTrackID:trackID];
AVComposition
AVCompositionTrack (Video)
AVCompositionTrack (Audio)

21. Composing Assets
Tracks and Segments
AVAssetTrack *srcVideoTrack1 = // source video track 1
[videoTrack insertTimeRange:timeRange ofTrack:srcVideoTrack1 atTime:startTime error:&error];
AVAssetTrack *srcVideoTrack2 = // source video track 2
[videoTrack insertTimeRange:timeRange ofTrack:srcVideoTrack2 atTime:startTime error:&error];
AVAssetTrack *srcAudioTrack = // source audio track
[audioTrack insertTimeRange:timeRange ofTrack:srcAudioTrack atTime:startTime error:&error];
AVComposition
AVCompositionTrack (Video)
AVCompositionTrackSegment AVCompositionTrackSegment
Seconds 10-30 of “redpanda.m4v” Seconds 20-60 of “waves.m4v”
AVCompositionTrack (Audio)
AVCompositionTrackSegment
Seconds 0-60 of “soundtrack.mp3”

22. Demo

23. Mixing Audio

24. Audio Mixing
AVAudioMix
‣ Composition tracks play at their natural volume
‣ AVAudioMix applies track-level volume adjustments
‣ Composed of AVAudioMixInputParameters
‣ Parameters control individual track volume over time
CMTime CMTimeRange

25. AVAudioMix
Example
CMTime fadeTime = CMTimeMake(5, 1);
CMTime fadeInStartTime = kCMTimeZero;
CMTime fadeOutStartTime = CMTimeSubtract(asset.duration, fadeTime);
CMTimeRange fadeInRange = CMTimeRangeMake(fadeInStartTime, fadeTime);
CMTimeRange fadeOutRange = CMTimeRangeMake(fadeOutStartTime, fadeTime);
AVMutableAudioMixInputParameters *parameters =
[AVMutableAudioMixInputParameters audioMixInputParameters];
parameters.trackID = assetTrack.trackID;
[parameters setVolumeRampFromStartVolume:0.0 toEndVolume:1.0 timeRange:fadeInRange];
[parameters setVolumeRampFromStartVolume:1.0 toEndVolume:0.0 timeRange:fadeOutRange];
AVMutableAudioMix *audioMix = [AVMutableAudioMix audioMix];
audioMix.inputParameters = @[parameters];
playerItem.audioMix = audioMix;

26. Demo

27. Video Transitions

28. Video Transitions
AVVideoComposition
AVVideoComposition
Defines how two or more video
tracks are composited together
AVVideoCompositionInstruction
Configured through collection of
AVVideoCompositionInstruction
AVVideoCompositionInstruction composition instructions describing
compositing behavior

29. Video Transitions
AVVideoCompositionInstruction
AVVideoComposition
AVVideoCompositionInstruction
AVVideoCompositionInstruction
AVVideoCompositionInstruction
Defines the time range of
compositing behavior
Composed of layer instructions
AVAsset
AVAsset
AVVideoCompositionLayerInstruction describing compositing behavior

30. Video Transitions
AVVideoCompositionLayerInstruction
AVVideoComposition
AVVideoCompositionInstruction
AVVideoCompositionInstruction
AVVideoCompositionInstruction
AVAsset
AVAsset Defines the transform and
AVVideoCompositionLayerInstruction
opacity ramps of input layers
Transform and opacity changes
modified over given time range

31. Building Transitions
Conceptual Steps

32. Building Transitions 1
Stagger Layout
A
B

33. Building Transitions 2
Define Overlapping Regions
A
B

34. Building Transitions 3
Define Time Ranges
A
Passthrough Passthrough Passthrough
B
Transition Transition
* Time ranges must not have gaps or overlap
* Total duration must not be shorter than composition

35. Building Transitions 4
Configure Instructions
// Build transition instructions
AVMutableVideoCompositionInstruction *transitionInstruction = ...;
transitionInstruction.timeRange = transitionTimeRange;
AVMutableVideoCompositionLayerInstruction *fromLayerInstruction = ...;
AVMutableVideoCompositionLayerInstruction *toLayerInstruction = ...;
// Cross Disolve
[fromLayerInstruction setOpacityRampFromStartOpacity:1.0
toEndOpacity:0.0
timeRange:transitionTimeRange];
NSArray *instructions = @[fromLayerInstruction, toLayerInstruction];
transitionInstruction.layerInstructions = instructions;
[instructions addObject:transitionInstruction];

36. Building Transitions 5
Set sizes and apply
#define FRAME_RATE CMTimeMake(1, 30)
#define RENDER_SIZE CGSizeMake(1280, 720)
AVMutableVideoComposition *videoComposition =
[AVMutableVideoComposition videoComposition];
// Set instructions on AVVideoComposition instance
videoComposition.instructions = instructions;
videoComposition.frameDuration = FRAME_RATE;
videoComposition.renderSize = RENDER_SIZE;
AVPlayerItem *playerItem =
[AVPlayerItem playerItemWithAsset:[composition copy]];
playerItem.videoComposition = videoComposition;

37. New in iOS 6
AVVideoComposition
videoCompositionWithPropertiesOfAsset:
Automagic Setup:
• Calculates all required passthrough and transition time ranges
• Builds appropriate composition and layer instructions for time ranges
• Sets the appropriate render size
• Sets the appropriate frame rate

38. Demo

39. Layering Content

40. Layering Content
Core Animation
Core Animation a natural choice
‣ High performance, inherently time-based
‣ CALayer subclasses used for all video rendering
CALayer: used to layer images and text
CAAnimation: used to animate layered content
CABasicAnimation
CAKeyframeAnimation

41. Animation Timing
AVSynchronizedLayer
‣ Core Animation operates on host time
‣ Starts at boot, marches towards infinity
‣ Timeline animations need to use movie time
‣ Starts at kCMTimeZero and runs to duration
‣ Can be started, stopped, rewound, etc.
‣ Use AVSynchronizedLayer to use movie time
‣ Confers player item timing on to its sublayer tree
AVPlayerItem AVSynchronizedLayer
CATextLayer CABasicAnimation

42. Core Animation
Timeline vs Realtime Animations
‣ Exactly the same, but different:
‣ Animations with zero beginTime won’t be seen
‣ Set beginTime = AVCoreAnimationBeginTimeZero
‣ Animations removed by default
‣ Set removedOnCompletion = NO
‣ Unable to use CAAnimationGroup?

43. Core Animation
Natural Choice, Awkward Implementation
‣ Different conceptual models for timeline editing
‣ CMTime and CMTimeRange for asset items
‣ Seconds and milliseconds for layers
‣ Build abstraction to help bridge the gap
‣ Usage differs in playback and export scenarios
‣ AVSynchronizedLayer for playback
‣ Attach to player’s view hierarchy/layer tree
‣ AVVideoCompositionCoreAnimationTool for export

44. Demo

45. Summary
AV Foundation Rocks!
‣ Powerful tools for audio and video playback
‣ AVPlayer, AVPlayerItem, AVPlayerLayer
‣ Powerful tools for composing/editing media:
‣ AVComposition, AVAudioMix,
AVVideoComposition, AVSynchronizedLayer
‣ Powerful utility classes:
‣ AVAssetImageGenerator
‣ AVExportSession
‣ Steep learning curve, but worth the investment!

46. Resources
Presentation Materials
http://www.speakerdeck.com/bobmccune/
http://www.slideshare.net/bobmccune/
https://github.com/tapharmonic/AVFoundationEditor
Learning AV Foundation
http://www.speakerdeck.com/bobmccune/
https://github.com/tapharmonic/AVFoundationDemos
WWDC 2011: Exploring AV Foundation
https://developer.apple.com/videos/wwdc/2011/?id=405
WWDC 2011: Working with Media in AV Foundation
https://developer.apple.com/videos/wwdc/2011/?id=415
BobMcCune.com @bobmccune