Some of my favourite bits of AVFoundation. Topics include capture, composition, a custom player and scrubber interface, synchronized CAAnimations, and real-time VFX.

1. Neat things to do on a rainy day with
AVFoundation
Ryder Mackay
@rydermackay
TACOW
May 14, 2013

2. Don’t worry, I’m an “expert”

3. AVFoundation
• Mid-level Objective-C framework for playing,
recording and editing time-based media
• Available on iOS 4.0+ and Mac OS X 10.7+
AV Foundation
Core Audio Core Media Core Animation
Media Player
UIKit

4. Additional Frameworks
• QuartzCore
• Layer trees, animation
• Core Media
• CMTime, CMTimeRange, etc.
• CoreVideo
• CVPixelBufferRef, kCVPixelFormatTypes

5. A Brief History
iOS AVFoundation Features
2.2 AVAudioPlayer
3.0 AVAudioRecorder,AVAudioSession
4.0 Capture, playback and editing
4.1 Read/write sample buffers, queue player
5.0 OpenGLES compatibility,AirPlay

6. New in iOS 6.0
• Real-time access to video buffers
• Face tracking during capture
• Better support for encrypted streams
• Advanced synchronization features

7. AVAsset (abstract base class)
• AVURLAsset: local or remote
• AVComposition
• AVMutableComposition
AVAsset
AVAssetTrack
AVAssetTrackSegment
audio
video

8. @protocol AVAsynchronousKeyValueLoading
• Handler invoked on arbitrary thread;
dispatch_async to main queue
- (void)loadValuesAsynchronouslyForKeys:(NSArray *)keys
completionHandler:(void (^)())handler;
- (AVKeyValueStatus)statusOfValueForKey:(NSString *)key
error:(NSError **)outError;

9. AVURLAsset
+ (AVURLAsset *)assetWithURL:(NSURL *)URL options:
(NSDictionary *)options;
- (CMTime)duration;
- (NSArray *)tracks;
- (BOOL)isPlayable;
- (BOOL)isExportable;
- (BOOL)isReadable;
- (BOOL)isComposable;
AVURLAssetPreferPreciseDurationAndTimingKey

10. CMTime
• C struct representing rational number
• numerator: value, denominator: scale
• time in seconds = value / scale
• Flags: valid, +/-ve infinity, has been rounded
• Time scale of 600 can conveniently
represents 24, 25 and 30 fps exactly

11. Playback

12. AVPlayerItem
AVPlayerItemTrack
AVPlayerItemTrack
AVAsset
AVAssetTrack
AVAssetTrack
AVPlayerLayer
AVPlayer
AVPlayerItem
AVPlayerItemTrack
AVPlayer & item manage asset presentation state

13. AVPlayer
@property (nonatomic) float rate;
- (void)play;
- (void)seekToTime:(CMTime)time;
- (CMTime)currentTime;
- (AVPlayerStatus)status;
- (NSError *)error;

14. Playback Notifications
– (id)addPeriodicTimeObserverForInterval:
(CMTime)interval queue:(dispatch_queue_t)queue
usingBlock:(void (^)(CMTime))block;
-(id)addBoundaryTimeObserverForTimes:(NSArray *)times
queue:(dispatch_queue_t)queue usingBlock:(void (^)
(CMTime))block;
– (void)removeTimeObserver:(id)observer;
typedef NSInteger AVPlayerActionAtItemEnd;
NSString * const AVPlayerItemDidPlayToEndNotification;

15. AVPlayerLayer
+ (AVPlayerLayer *)playerLayerWithPlayer:
(AVPlayer *)player;
@property (copy) NSString *videoGravity;
@property (nonatomic,readonly, getter =
isReadyForDisplay) BOOL readyForDisplay;

16. Playback Summary
• Load asset tracks
• Create player and player item
• Create player layer
• Observe “readyForDisplay” property
• Insert layer into subtree

17. Generating Thumbnails

18. AVAssetImageGenerator
- (id)initWithAsset:(AVAsset *)asset;
- (CGImageRef)copyCGImageAtTime:(CMTime)requestedTime
actualTime:(CMTime *)actualTime
error:(NSError **)outError;
- (void)generateCGImagesAsynchronouslyForTimes:(NSArray
*)requestedTimes completionHandler:(CMTime requestedTime,
CGImageRef image, CMTime actualTime,
AVAssetImageGeneratorResult result, NSError
*error)handler;
- (void)cancelAllCGImageGeneration;

19. Demo:
Custom Player

20. Editing

21. AVAsset
AVAssetTrack AVAssetTrackSegment
AVComposition
AVCompositionTrack AVCompositionTrackSegment

22. AVMutableComposition
- (BOOL)insertTimeRange:
(CMTimeRange)timeRange ofAsset:(AVAsset
*)asset atTime:(CMTime)startTime error:
(NSError **)outError;
- (void)scaleTimeRange:(CMTimeRange)timeRange
toDuration:(CMTime)duration;
- (void)removeTimeRange:
(CMTimeRange)timeRange;
timeRange must be valid or you’re gonna have a bad time

23. Demo:
Capture, Compose & Export

24. AVSynchronizedLayer
• Confers timing state upon sublayers
• Timing synced with AVPlayerItem instance
• +synchronizedLayerWithPlayerItem:
• When creating CAAnimations:
• Use AVCoreAnimationBeginTimeAtZero
• -setRemovedOnCompletion:NO

25. Demo:
AVSynchronizedLayer

26. Real-time Processing

27. Now Playing in 3D
• iOS 5:
• CVOpenGLESTextureCacheRef
• CVOpenGLESTextureRef
• +[CIImage imageWithCVPixelBuffer:options:]
• Binding between CVPixelBufferRef and GL textures
• Bypasses copying to/from CPU-controlled memory
• iOS 6:
• +[CIImage imageWithTexture:size:flipped:colorSpace:]

28. CVOpenGLESTextureCacheCreate(
CFAllocatorRef allocator,
CFDictionaryRef cacheAttributes,
CVEAGLContext eaglContext,
CFDictionaryRef textureAttributes,
CVOpenGLESTextureCacheRef *cacheOut)
CVOpenGLESTextureCacheCreateTextureFromImage(
CFAllocatorRef allocator,
CVOpenGLESTextureCacheRef textureCache,
CVImageBufferRef sourceImage,
CFDictionaryRef textureAttributes,
GLenum target, // GL_TEXTURE_2D
GLint internalFormat, // GL_RGBA
GLsizei width,
GLsizei height,
GLenum format, // GL_BGRA
GLenum type, // GL_UNSIGNED_BYTE
size_t planeIndex, // 0
CVOpenGLESTextureRef *textureOut)

29. Obtaining Pixel Buffers
Real-time:
• AVCaptureVideoDataOutput
• AVPlayerItemVideoOutput iOS 6
Offline:
• AVAssetReaderTrackOutput
• AVAssetReaderVideoCompositionOutput

30. AVPlayerItemVideoDataOutput
• Access pixel buffers during playback
• Request wakeup, poll w/ display link
running pausedCADisplayLink
request change
notification
media data
will change
buffer for
time?
NO
YES
process &
display
entry
point

31. Demo:
Real-timeVFX

32. Sample Code
Apple:
AVBasicVideoOutput
AVSimpleEditor
RosyWriter
Bob McCune – AVFoundationEditor:
https://github.com/tapharmonic/AVFoundationEditor
Bill Dudney – AVCoreImageIntegration:
https://github.com/bdudney/Experiments

33. Twitter: @rydermackay
ADN: @ryder
github.com/rydermackay
while (self.retainCount > 0) {
[self release];
}