AVFoundation @ TACOW 2013 05 14

2,131 views

Published on

Some of my favourite bits of AVFoundation. Topics include capture, composition, a custom player and scrubber interface, synchronized CAAnimations, and real-time VFX.

Published in: Technology, Art & Photos
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,131
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
41
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide

AVFoundation @ TACOW 2013 05 14

  1. 1. Neat things to do on a rainy day withAVFoundationRyder Mackay@rydermackayTACOWMay 14, 2013
  2. 2. Don’t worry, I’m an “expert”
  3. 3. AVFoundation• Mid-level Objective-C framework for playing,recording and editing time-based media• Available on iOS 4.0+ and Mac OS X 10.7+AV FoundationCore Audio Core Media Core AnimationMedia PlayerUIKit
  4. 4. Additional Frameworks• QuartzCore• Layer trees, animation• Core Media• CMTime, CMTimeRange, etc.• CoreVideo• CVPixelBufferRef, kCVPixelFormatTypes
  5. 5. A Brief HistoryiOS AVFoundation Features2.2 AVAudioPlayer3.0 AVAudioRecorder,AVAudioSession4.0 Capture, playback and editing4.1 Read/write sample buffers, queue player5.0 OpenGLES compatibility,AirPlay
  6. 6. New in iOS 6.0• Real-time access to video buffers• Face tracking during capture• Better support for encrypted streams• Advanced synchronization features
  7. 7. AVAsset (abstract base class)• AVURLAsset: local or remote• AVComposition• AVMutableCompositionAVAssetAVAssetTrackAVAssetTrackSegmentaudiovideo
  8. 8. @protocol AVAsynchronousKeyValueLoading• Handler invoked on arbitrary thread;dispatch_async to main queue- (void)loadValuesAsynchronouslyForKeys:(NSArray *)keyscompletionHandler:(void (^)())handler;- (AVKeyValueStatus)statusOfValueForKey:(NSString *)keyerror:(NSError **)outError;
  9. 9. AVURLAsset+ (AVURLAsset *)assetWithURL:(NSURL *)URL options:(NSDictionary *)options;- (CMTime)duration;- (NSArray *)tracks;- (BOOL)isPlayable;- (BOOL)isExportable;- (BOOL)isReadable;- (BOOL)isComposable;AVURLAssetPreferPreciseDurationAndTimingKey
  10. 10. CMTime• C struct representing rational number• numerator: value, denominator: scale• time in seconds = value / scale• Flags: valid, +/-ve infinity, has been rounded• Time scale of 600 can convenientlyrepresents 24, 25 and 30 fps exactly
  11. 11. Playback
  12. 12. AVPlayerItemAVPlayerItemTrackAVPlayerItemTrackAVAssetAVAssetTrackAVAssetTrackAVPlayerLayerAVPlayerAVPlayerItemAVPlayerItemTrackAVPlayer & item manage asset presentation state
  13. 13. AVPlayer@property (nonatomic) float rate;- (void)play;- (void)seekToTime:(CMTime)time;- (CMTime)currentTime;- (AVPlayerStatus)status;- (NSError *)error;
  14. 14. Playback Notifications– (id)addPeriodicTimeObserverForInterval:(CMTime)interval queue:(dispatch_queue_t)queueusingBlock:(void (^)(CMTime))block;-(id)addBoundaryTimeObserverForTimes:(NSArray *)timesqueue:(dispatch_queue_t)queue usingBlock:(void (^)(CMTime))block;– (void)removeTimeObserver:(id)observer;typedef NSInteger AVPlayerActionAtItemEnd;NSString * const AVPlayerItemDidPlayToEndNotification;
  15. 15. AVPlayerLayer+ (AVPlayerLayer *)playerLayerWithPlayer:(AVPlayer *)player;@property (copy) NSString *videoGravity;@property (nonatomic,readonly, getter =isReadyForDisplay) BOOL readyForDisplay;
  16. 16. Playback Summary• Load asset tracks• Create player and player item• Create player layer• Observe “readyForDisplay” property• Insert layer into subtree
  17. 17. Generating Thumbnails
  18. 18. AVAssetImageGenerator- (id)initWithAsset:(AVAsset *)asset;- (CGImageRef)copyCGImageAtTime:(CMTime)requestedTimeactualTime:(CMTime *)actualTimeerror:(NSError **)outError;- (void)generateCGImagesAsynchronouslyForTimes:(NSArray*)requestedTimes completionHandler:(CMTime requestedTime,CGImageRef image, CMTime actualTime,AVAssetImageGeneratorResult result, NSError*error)handler;- (void)cancelAllCGImageGeneration;
  19. 19. Demo:Custom Player
  20. 20. Editing
  21. 21. AVAssetAVAssetTrack AVAssetTrackSegmentAVCompositionAVCompositionTrack AVCompositionTrackSegment
  22. 22. AVMutableComposition- (BOOL)insertTimeRange:(CMTimeRange)timeRange ofAsset:(AVAsset*)asset atTime:(CMTime)startTime error:(NSError **)outError;- (void)scaleTimeRange:(CMTimeRange)timeRangetoDuration:(CMTime)duration;- (void)removeTimeRange:(CMTimeRange)timeRange;timeRange must be valid or you’re gonna have a bad time
  23. 23. Demo:Capture, Compose & Export
  24. 24. AVSynchronizedLayer• Confers timing state upon sublayers• Timing synced with AVPlayerItem instance• +synchronizedLayerWithPlayerItem:• When creating CAAnimations:• Use AVCoreAnimationBeginTimeAtZero• -setRemovedOnCompletion:NO
  25. 25. Demo:AVSynchronizedLayer
  26. 26. Real-time Processing
  27. 27. Now Playing in 3D• iOS 5:• CVOpenGLESTextureCacheRef• CVOpenGLESTextureRef• +[CIImage imageWithCVPixelBuffer:options:]• Binding between CVPixelBufferRef and GL textures• Bypasses copying to/from CPU-controlled memory• iOS 6:• +[CIImage imageWithTexture:size:flipped:colorSpace:]
  28. 28. CVOpenGLESTextureCacheCreate(CFAllocatorRef allocator,CFDictionaryRef cacheAttributes,CVEAGLContext eaglContext,CFDictionaryRef textureAttributes,CVOpenGLESTextureCacheRef *cacheOut)CVOpenGLESTextureCacheCreateTextureFromImage(CFAllocatorRef allocator,CVOpenGLESTextureCacheRef textureCache,CVImageBufferRef sourceImage,CFDictionaryRef textureAttributes,GLenum target, // GL_TEXTURE_2DGLint internalFormat, // GL_RGBAGLsizei width,GLsizei height,GLenum format, // GL_BGRAGLenum type, // GL_UNSIGNED_BYTEsize_t planeIndex, // 0CVOpenGLESTextureRef *textureOut)
  29. 29. Obtaining Pixel BuffersReal-time:• AVCaptureVideoDataOutput• AVPlayerItemVideoOutput iOS 6Offline:• AVAssetReaderTrackOutput• AVAssetReaderVideoCompositionOutput
  30. 30. AVPlayerItemVideoDataOutput• Access pixel buffers during playback• Request wakeup, poll w/ display linkrunning pausedCADisplayLinkrequest changenotificationmedia datawill changebuffer fortime?NOYESprocess &displayentrypoint
  31. 31. Demo:Real-timeVFX
  32. 32. Sample CodeApple:AVBasicVideoOutputAVSimpleEditorRosyWriterBob McCune – AVFoundationEditor:https://github.com/tapharmonic/AVFoundationEditorBill Dudney – AVCoreImageIntegration:https://github.com/bdudney/Experiments
  33. 33. Twitter: @rydermackayADN: @rydergithub.com/rydermackaywhile (self.retainCount > 0) {[self release];}

×