Introduction to AV Foundation

Introduction to AV Foundation Chris Adamson CocoaHeads Ann Arbor — Aug. 12, 2010

Media on iPhone / iOS

iPhone 2 Media Frameworks Core Audio Low-level audio streaming Media Player Full-screen video player Obj-C wrapper for audio ﬁle AV Foundation playback (2.2 only)

iPhone 3 Media Frameworks Core Audio Low-level audio streaming Media Player iPod library search/playback Obj-C wrapper for audio ﬁle AV Foundation playback, recording

iOS 4 Media Frameworks Core Audio Low-level audio streaming Media Player iPod library search/playback Audio/video capture, editing, AV Foundation playback, export… Quartz effects on moving Core Video images Objects for representing media Core Media times, formats, buffers

AV Foundation in iPhone 2.2

AV Foundation in iPhone 3.2

AV Foundation in iOS 4

Size is relative AV android. QuickTime QT Kit Foundation media for Java Classes 56 40 24 576 Methods 460 280 360 >10,000

AV Foundation Classes • Capture • Assets and compositions • Playback, editing, and export • Legacy classes

Media Capture

Capture: Old and Busted • UIImagePickerController • Takes user out of your UI • Low conﬁgurability • No capture-time data access • AVAudioRecorder • Audio only, to ﬁle only

Capture: New Hotness • AV Foundation capture classes • Highly conﬁgurable • Live callbacks with capture data • Image/video preview to a CALayer

Capture Classes Seem Familiar? QT Kit AV Foundation QTCaptureAudioPreviewOutput AVCaptureAudioDataOutput QTCaptureConnection AVCaptureConnection QTCaptureDecompressedAudioOutput AVCaptureDevice QTCaptureDecompressedVideoOutput AVCaptureFileOutput QTCaptureDevice AVCaptureInput QTCaptureDeviceInput AVCaptureMovieFileOutput QTCaptureFileOutput AVCaptureOutput QTCaptureInput AVCaptureSession QTCaptureLayer AVCaptureStillImageOutput QTCaptureMovieFileOutput AVCaptureVideoDataOutput QTCaptureOutput AVCaptureVideoPreviewLayer QTCaptureSession QTCaptureVideoPreviewOutput QTCaptureView

AVCaptureDevice • Represents an input (camera, microphone) or output (speakers) device • Discover with +[devices], + [devicesWithMediaType], + [defaultDeviceWithMediaType:], … • Flash, white balance, exposure, focus settings for camera devices

AVCaptureOutput • A destination for captured data • Files: AVCaptureFileOutput, AVCaptureMovieFileOutput • Images: AVCaptureStillImageOutput • Live data: AVCaptureAudioDataOutput, AVCaptureVideoDataOutput

AVCaptureSession • Coordinates the activity of audio and video capture devices • Allows you to connect/disconnect inputs and outputs • startRunning/stopRunning

AVCaptureVideoPreview Layer • Subclass of CALayer • +[layerWithSession:]

Data Output Callbacks • Audio and video data outputs provide -[setSampleBufferDelegate:queue:] • Delegates get -[captureOutput: didOutputSampleBuffer: fromConnection]; • Sample buffer is a CMSampleBufferRef

Core Media • New in iOS 4 • Core Foundation opaque types for wrapping sample buffers, format descriptions, time structures • Functions convert video samples to CVImageBuffer, audio to Core Audio AudioBufferList

Core Media Time • CMTime: value, timescale, ﬂags, epoch • Timescale is n-ths of a second • Set timescale to a resolution appropriate to your media (e.g., 44100 for CD audio). QT convention is 600 for video (ask Chris why!) • CMTimeConvertScale()

WWDC 2010 Session 409 Using the Camera with AV Foundation Overview and best practices Brad Ford iPhone Engineering 2

Assets and Composition

“iMovie is built entirely on exactly the same public API in AV Foundation that we’re presenting to you in iPhone 4.”

“Boom Box” APIs • Simple API for playback, sometimes recording • Little or no support for editing, mixing, metadata, etc. • Example: HTML 5 <audio> tag

“Streaming” APIs • Use “stream of audio” metaphor • Strong support for mixing, effects, other real-time operations • Example: Core Audio

“Document” APIs • Use “media document” metaphor • Strong support for editing • Mixing may be a special case of editing • Example: QuickTime and AV Foundation

Assets and Movies • AVAsset: Collection of tracks representing timed media data • QTMovie: Collection of tracks representing timed media data

QuickTime movies Movie Track Track Media Media

AVFoundation assets Movie Track Track

AVAsset • Superclass of all “movie”-like structures in AVFoundation • Represents traits of all tracks taken together: size, duration • Build your own with AVURLAsset

AVComposition • Subclass of AVAsset representing a combination of multiple ﬁle-based assets • Tracks are AVCompositionTracks • For editing, AVMutableComposition and AVMutableCompositionTracks

Effects 1 • AVAudioMix, AVMutableAudioMix: set volumes or audio ramps at speciﬁc times • AVVideoCompositionInstructions: provide a set of layer-based instructions for performing time- based opacity or afﬁne transform ramps

Playback • AVPlayer: Playback controller • play, pause, seekToTime, etc. • AVPlayerLayer: CALayer for presenting video from an AVLayer

Effects 2 • AVSynchronizedLayer: CALayer that synchronizes with a AVPlayerItem’s playback timing • Use for overlays, titles, rendered images, Ken Burns effects, etc.

Export • AVAssetExportSession • Must be created with a canned preset • -[exportAsynchronouslyWith CompletionHandler:] • Takes a block! • Exporting CA effects is tricky…

Other stuff • AVAssetStillImageGenerator: used for generating thumbnails • Not suitable for getting individual frames (but no GetMediaSample() equivalent either!) • NSCoder, NSValue additions for wrapping CMTimes, CMTimeRanges

WWDC 2010 Session 407 Editing Media with AV Foundation Overview and best practices Eric Lee iPhone Engineering 2

What’s Next?

What’s Next Next? AVAssetExportSession.h: extern NSString *const AVAssetExportPreset1280x720 __OSX_AVAILABLE_STARTING(__MAC_10_7,__IPHONE_4_0); CMTime.h: CM_EXPORT const CFStringRef kCMTimeScaleKey __OSX_AVAILABLE_STARTING(__MAC_10_7,__IPHONE_4_0);

Contact Info • http://www.subfurther.com/blog • @invalidname • invalidname [at] gmail [dot] com

Introduction to AV Foundation

by Chris Adamson on Aug 13, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

3 Embeds 3

Statistics

Introduction to AV Foundation — Presentation Transcript

http://www.slideshare.net	1
http://confluence.moleque.com.br	1
http://a0.twimg.com	1