Your SlideShare is downloading. ×
AVFoundation @ TACOW 2013 05 14
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

AVFoundation @ TACOW 2013 05 14


Published on

Some of my favourite bits of AVFoundation. Topics include capture, composition, a custom player and scrubber interface, synchronized CAAnimations, and real-time VFX.

Some of my favourite bits of AVFoundation. Topics include capture, composition, a custom player and scrubber interface, synchronized CAAnimations, and real-time VFX.

Published in: Technology, Art & Photos

  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Neat things to do on a rainy day withAVFoundationRyder Mackay@rydermackayTACOWMay 14, 2013
  • 2. Don’t worry, I’m an “expert”
  • 3. AVFoundation• Mid-level Objective-C framework for playing,recording and editing time-based media• Available on iOS 4.0+ and Mac OS X 10.7+AV FoundationCore Audio Core Media Core AnimationMedia PlayerUIKit
  • 4. Additional Frameworks• QuartzCore• Layer trees, animation• Core Media• CMTime, CMTimeRange, etc.• CoreVideo• CVPixelBufferRef, kCVPixelFormatTypes
  • 5. A Brief HistoryiOS AVFoundation Features2.2 AVAudioPlayer3.0 AVAudioRecorder,AVAudioSession4.0 Capture, playback and editing4.1 Read/write sample buffers, queue player5.0 OpenGLES compatibility,AirPlay
  • 6. New in iOS 6.0• Real-time access to video buffers• Face tracking during capture• Better support for encrypted streams• Advanced synchronization features
  • 7. AVAsset (abstract base class)• AVURLAsset: local or remote• AVComposition• AVMutableCompositionAVAssetAVAssetTrackAVAssetTrackSegmentaudiovideo
  • 8. @protocol AVAsynchronousKeyValueLoading• Handler invoked on arbitrary thread;dispatch_async to main queue- (void)loadValuesAsynchronouslyForKeys:(NSArray *)keyscompletionHandler:(void (^)())handler;- (AVKeyValueStatus)statusOfValueForKey:(NSString *)keyerror:(NSError **)outError;
  • 9. AVURLAsset+ (AVURLAsset *)assetWithURL:(NSURL *)URL options:(NSDictionary *)options;- (CMTime)duration;- (NSArray *)tracks;- (BOOL)isPlayable;- (BOOL)isExportable;- (BOOL)isReadable;- (BOOL)isComposable;AVURLAssetPreferPreciseDurationAndTimingKey
  • 10. CMTime• C struct representing rational number• numerator: value, denominator: scale• time in seconds = value / scale• Flags: valid, +/-ve infinity, has been rounded• Time scale of 600 can convenientlyrepresents 24, 25 and 30 fps exactly
  • 11. Playback
  • 12. AVPlayerItemAVPlayerItemTrackAVPlayerItemTrackAVAssetAVAssetTrackAVAssetTrackAVPlayerLayerAVPlayerAVPlayerItemAVPlayerItemTrackAVPlayer & item manage asset presentation state
  • 13. AVPlayer@property (nonatomic) float rate;- (void)play;- (void)seekToTime:(CMTime)time;- (CMTime)currentTime;- (AVPlayerStatus)status;- (NSError *)error;
  • 14. Playback Notifications– (id)addPeriodicTimeObserverForInterval:(CMTime)interval queue:(dispatch_queue_t)queueusingBlock:(void (^)(CMTime))block;-(id)addBoundaryTimeObserverForTimes:(NSArray *)timesqueue:(dispatch_queue_t)queue usingBlock:(void (^)(CMTime))block;– (void)removeTimeObserver:(id)observer;typedef NSInteger AVPlayerActionAtItemEnd;NSString * const AVPlayerItemDidPlayToEndNotification;
  • 15. AVPlayerLayer+ (AVPlayerLayer *)playerLayerWithPlayer:(AVPlayer *)player;@property (copy) NSString *videoGravity;@property (nonatomic,readonly, getter =isReadyForDisplay) BOOL readyForDisplay;
  • 16. Playback Summary• Load asset tracks• Create player and player item• Create player layer• Observe “readyForDisplay” property• Insert layer into subtree
  • 17. Generating Thumbnails
  • 18. AVAssetImageGenerator- (id)initWithAsset:(AVAsset *)asset;- (CGImageRef)copyCGImageAtTime:(CMTime)requestedTimeactualTime:(CMTime *)actualTimeerror:(NSError **)outError;- (void)generateCGImagesAsynchronouslyForTimes:(NSArray*)requestedTimes completionHandler:(CMTime requestedTime,CGImageRef image, CMTime actualTime,AVAssetImageGeneratorResult result, NSError*error)handler;- (void)cancelAllCGImageGeneration;
  • 19. Demo:Custom Player
  • 20. Editing
  • 21. AVAssetAVAssetTrack AVAssetTrackSegmentAVCompositionAVCompositionTrack AVCompositionTrackSegment
  • 22. AVMutableComposition- (BOOL)insertTimeRange:(CMTimeRange)timeRange ofAsset:(AVAsset*)asset atTime:(CMTime)startTime error:(NSError **)outError;- (void)scaleTimeRange:(CMTimeRange)timeRangetoDuration:(CMTime)duration;- (void)removeTimeRange:(CMTimeRange)timeRange;timeRange must be valid or you’re gonna have a bad time
  • 23. Demo:Capture, Compose & Export
  • 24. AVSynchronizedLayer• Confers timing state upon sublayers• Timing synced with AVPlayerItem instance• +synchronizedLayerWithPlayerItem:• When creating CAAnimations:• Use AVCoreAnimationBeginTimeAtZero• -setRemovedOnCompletion:NO
  • 25. Demo:AVSynchronizedLayer
  • 26. Real-time Processing
  • 27. Now Playing in 3D• iOS 5:• CVOpenGLESTextureCacheRef• CVOpenGLESTextureRef• +[CIImage imageWithCVPixelBuffer:options:]• Binding between CVPixelBufferRef and GL textures• Bypasses copying to/from CPU-controlled memory• iOS 6:• +[CIImage imageWithTexture:size:flipped:colorSpace:]
  • 28. CVOpenGLESTextureCacheCreate(CFAllocatorRef allocator,CFDictionaryRef cacheAttributes,CVEAGLContext eaglContext,CFDictionaryRef textureAttributes,CVOpenGLESTextureCacheRef *cacheOut)CVOpenGLESTextureCacheCreateTextureFromImage(CFAllocatorRef allocator,CVOpenGLESTextureCacheRef textureCache,CVImageBufferRef sourceImage,CFDictionaryRef textureAttributes,GLenum target, // GL_TEXTURE_2DGLint internalFormat, // GL_RGBAGLsizei width,GLsizei height,GLenum format, // GL_BGRAGLenum type, // GL_UNSIGNED_BYTEsize_t planeIndex, // 0CVOpenGLESTextureRef *textureOut)
  • 29. Obtaining Pixel BuffersReal-time:• AVCaptureVideoDataOutput• AVPlayerItemVideoOutput iOS 6Offline:• AVAssetReaderTrackOutput• AVAssetReaderVideoCompositionOutput
  • 30. AVPlayerItemVideoDataOutput• Access pixel buffers during playback• Request wakeup, poll w/ display linkrunning pausedCADisplayLinkrequest changenotificationmedia datawill changebuffer fortime?NOYESprocess &displayentrypoint
  • 31. Demo:Real-timeVFX
  • 32. Sample CodeApple:AVBasicVideoOutputAVSimpleEditorRosyWriterBob McCune – AVFoundationEditor: Dudney – AVCoreImageIntegration:
  • 33. Twitter: @rydermackayADN: (self.retainCount > 0) {[self release];}