1.
Introduction to AV
Foundation
Chris Adamson
CocoaHeads Ann Arbor — Aug. 12, 2010

2.
Media on iPhone / iOS

3.
iPhone 2 Media
Frameworks
Core Audio Low-level audio streaming
Media Player Full-screen video player
Obj-C wrapper for audio file
AV Foundation
playback (2.2 only)

4.
iPhone 3 Media
Frameworks
Core Audio Low-level audio streaming
Media Player iPod library search/playback
Obj-C wrapper for audio file
AV Foundation
playback, recording

5.
iOS 4 Media Frameworks
Core Audio Low-level audio streaming
Media Player iPod library search/playback
Audio/video capture, editing,
AV Foundation
playback, export…
Quartz effects on moving
Core Video
images
Objects for representing media
Core Media
times, formats, buffers

6.
AV Foundation in iPhone
2.2

7.
AV Foundation in iPhone
3.2

8.
AV Foundation in iOS 4

9.
Size is relative
AV android. QuickTime
QT Kit
Foundation media for Java
Classes 56 40 24 576
Methods 460 280 360 >10,000

10.
AV Foundation Classes
• Capture
• Assets and compositions
• Playback, editing, and export
• Legacy classes

11.
Media Capture

12.
Capture: Old and Busted
• UIImagePickerController
• Takes user out of your UI
• Low configurability
• No capture-time data access
• AVAudioRecorder
• Audio only, to file only

13.
Capture: New Hotness
• AV Foundation capture classes
• Highly configurable
• Live callbacks with capture data
• Image/video preview to a CALayer

14.
Capture Classes Seem
Familiar?
QT Kit AV Foundation
QTCaptureAudioPreviewOutput AVCaptureAudioDataOutput
QTCaptureConnection AVCaptureConnection
QTCaptureDecompressedAudioOutput AVCaptureDevice
QTCaptureDecompressedVideoOutput AVCaptureFileOutput
QTCaptureDevice AVCaptureInput
QTCaptureDeviceInput AVCaptureMovieFileOutput
QTCaptureFileOutput AVCaptureOutput
QTCaptureInput AVCaptureSession
QTCaptureLayer AVCaptureStillImageOutput
QTCaptureMovieFileOutput AVCaptureVideoDataOutput
QTCaptureOutput AVCaptureVideoPreviewLayer
QTCaptureSession
QTCaptureVideoPreviewOutput
QTCaptureView

15.
AVCaptureDevice
• Represents an input (camera,
microphone) or output (speakers)
device
• Discover with +[devices], +
[devicesWithMediaType], +
[defaultDeviceWithMediaType:], …
• Flash, white balance, exposure, focus
settings for camera devices

16.
AVCaptureOutput
• A destination for captured data
• Files: AVCaptureFileOutput,
AVCaptureMovieFileOutput
• Images: AVCaptureStillImageOutput
• Live data:
AVCaptureAudioDataOutput,
AVCaptureVideoDataOutput

17.
AVCaptureSession
• Coordinates the activity of audio and
video capture devices
• Allows you to connect/disconnect
inputs and outputs
• startRunning/stopRunning

18.
AVCaptureVideoPreview
Layer
• Subclass of CALayer
• +[layerWithSession:]

19.
Data Output Callbacks
• Audio and video data outputs provide
-[setSampleBufferDelegate:queue:]
• Delegates get -[captureOutput:​
didOutputSampleBuffer:​
fromConnection];
• Sample buffer is a
CMSampleBufferRef

20.
Core Media
• New in iOS 4
• Core Foundation opaque types for
wrapping sample buffers, format
descriptions, time structures
• Functions convert video samples to
CVImageBuffer, audio to Core Audio
AudioBufferList

21.
Core Media Time
• CMTime: value, timescale, flags, epoch
• Timescale is n-ths of a second
• Set timescale to a resolution
appropriate to your media (e.g.,
44100 for CD audio). QT convention
is 600 for video (ask Chris why!)
• CMTimeConvertScale()

22.
WWDC 2010 Session 409
Using the Camera
with AV Foundation
Overview and best practices
Brad Ford
iPhone Engineering
2

23.
Assets and
Composition

24.
“iMovie is built entirely on exactly the same public API in
AV Foundation that we’re presenting to you in iPhone 4.”

25.
“Boom Box” APIs
• Simple API for playback,
sometimes recording
• Little or no support for
editing, mixing, metadata,
etc.
• Example: HTML 5 <audio> tag

26.
“Streaming” APIs
• Use “stream of audio”
metaphor
• Strong support for mixing,
effects, other real-time
operations
• Example: Core Audio

27.
“Document” APIs
• Use “media document”
metaphor
• Strong support for editing
• Mixing may be a special case
of editing
• Example: QuickTime
and AV Foundation

28.
Assets and Movies
• AVAsset: Collection of tracks
representing timed media data
• QTMovie: Collection of tracks
representing timed media data

29.
QuickTime movies
Movie
Track Track
Media Media

30.
AVFoundation assets
Movie
Track Track

31.
AVAsset
• Superclass of all “movie”-like
structures in AVFoundation
• Represents traits of all tracks taken
together: size, duration
• Build your own with AVURLAsset

32.
AVComposition
• Subclass of AVAsset representing a
combination of multiple file-based
assets
• Tracks are AVCompositionTracks
• For editing, AVMutableComposition
and AVMutableCompositionTracks

33.
Effects 1
• AVAudioMix, AVMutableAudioMix: set
volumes or audio ramps at specific
times
• AVVideoCompositionInstructions:
provide a set of layer-based
instructions for performing time-
based opacity or affine transform
ramps

34.
Playback
• AVPlayer: Playback controller
• play, pause, seekToTime, etc.
• AVPlayerLayer: CALayer for
presenting video from an AVLayer

35.
Effects 2
• AVSynchronizedLayer: CALayer that
synchronizes with a AVPlayerItem’s
playback timing
• Use for overlays, titles, rendered
images, Ken Burns effects, etc.

36.
Export
• AVAssetExportSession
• Must be created with a canned
preset
• -[exportAsynchronouslyWith​
CompletionHandler:]
• Takes a block!
• Exporting CA effects is tricky…

37.
Other stuff
• AVAssetStillImageGenerator: used for
generating thumbnails
• Not suitable for getting individual
frames (but no GetMediaSample()
equivalent either!)
• NSCoder, NSValue additions for
wrapping CMTimes, CMTimeRanges

38.
WWDC 2010 Session 407
Editing Media with AV Foundation
Overview and best practices
Eric Lee
iPhone Engineering
2

39.
Demo

40.
What’s Next?

41.
What’s Next Next?
AVAssetExportSession.h:
extern NSString *const AVAssetExportPreset1280x720
__OSX_AVAILABLE_STARTING(__MAC_10_7,__IPHONE_4_0);
CMTime.h:
CM_EXPORT const CFStringRef kCMTimeScaleKey
__OSX_AVAILABLE_STARTING(__MAC_10_7,__IPHONE_4_0);

42.
Contact Info
• http://www.subfurther.com/blog
• @invalidname
• invalidname [at] gmail [dot] com