Your SlideShare is downloading. ×

Mastering Media with AV Foundation

26,501

Published on

The AV Foundation has grown from a simple audio player quietly added in iPhone OS 2.2 to an extraordinarily ambitious media-creation framework in iOS 4. The latest version provides highly-customizable …

The AV Foundation has grown from a simple audio player quietly added in iPhone OS 2.2 to an extraordinarily ambitious media-creation framework in iOS 4. The latest version provides highly-customizable audio and video capture, deep support for editing and Core Animation-based effects, and export. It is far more comprehensive than nearly any other platform's media framework, mobile or desktop. As a developer, this gives you incredible power… and one heck of a learning curve. In this session, we will take a ground-up look at AV Foundation, starting with its core concepts and patterns, and moving through its most practical and powerful capabilities. Along the way, we'll see how AV Foundation works with iOS' other media APIs -- Core Audio, Media Player, and Core Media -- and how it aggressively uses new iOS 4 programming paradigms like blocks and Grand Central Dispatch. Attendees will come away understanding the straight-arrow paths to AV Foundation's most important features like capture, editing, and export, and how its pieces might be combined in interesting ways to create even more powerful media applications.

Published in: Technology, Business
0 Comments
17 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
26,501
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
17
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Mastering Media with AV Foundation Chris Adamson — @invalidname — http://www.subfurther.com/blog Voices That Matter IPhone Developer Conference — October 17, 2010 Wednesday, October 20, 2010
  • 2. Road Map ✤ Fundamentals of Dynamic Media ✤ iOS Media Frameworks ✤ Playback ✤ Capture ✤ Editing ✤ Advanced Stuff Wednesday, October 20, 2010
  • 3. The Fundamentals Wednesday, October 20, 2010
  • 4. Analog ✤ Having a measurable value that is continuously variable ✤ Contrast to digital or “discrete” signals Wednesday, October 20, 2010
  • 5. Audio ✤ Phonograph — Grooves vibrate a needle, which is amplified to a speaker ✤ Telephone — Voice vibrates microphone membrane, vibration is transmitted as voltage and reproduced by vibrating headset speaker ✤ Radio — Audio signal modulated on a carrier wave Wednesday, October 20, 2010
  • 6. Film ✤ Light projected through translucent film frames onto screen ✤ Each frame held in place briefly (≅ 1/24 sec) ✤ Eye sees a moving image due to “persistence of vision” ✤ Sound may be out-of-band Wednesday, October 20, 2010
  • 7. Television ✤ Chrominance and luminance sent as continuous AM signal ✤ CRT gun sweeps across screen in zig-zag pattern, illuminating phosphors ✤ Sound is FM in adjacent spectrum Wednesday, October 20, 2010
  • 8. Digital Media ✤ Represents a continuous signal numerically ✤ Audio — sample the signal at some frequency ✤ Video — digital images as frames ✤ Other kinds of samples — text (captions, subtitles), metadata, web links, executable code, etc. Wednesday, October 20, 2010
  • 9. Encoding ✤ How do we turn a digital signal into numbers ✤ Audio — Pulse Code Modulation (PCM). Each sample represents amplitude of audio signal at a specific time. ✤ Compressed audio — Lossless and lossy transformations to and from PCM ✤ Video — Series of images (e.g., M-JPEG), or keyframe image (i-frame) followed by deltas (p-frames and b-frames) ✤ Other media — Text samples are just strings Wednesday, October 20, 2010
  • 10. Containers ✤ Transport and/or storage of encoded streams ✤ Examples: MP3, AIFF, QuickTime Movie, Core Audio Format, .mp4, MPEG-2 transport stream ✤ Containers may be optimized for streaming, editing, end-user delivery, etc. Wednesday, October 20, 2010
  • 11. Wednesday, October 20, 2010
  • 12. Wednesday, October 20, 2010
  • 13. Metadata ✤ Information related to the audio data other than the signal itself ✤ Song title/album/artist, movie title, TV episode title/series, etc. ✤ Some containers support metadata, otherwise it is provided out-of- band Wednesday, October 20, 2010
  • 14. Keep in mind… ✤ Different codecs may go in different containers ✤ A network stream is a container ✤ A media stream and a network stream are two different things ✤ Containers can contain multiple media streams ✤ A stream’s data is not necessarily in the container file ✤ Media samples may be in distinct places, or interleaved Wednesday, October 20, 2010
  • 15. Media frameworks on iPhone OS / iOS Wednesday, October 20, 2010
  • 16. iPhone 2 Media Frameworks Core Audio / Low-level audio streaming OpenAL Media Player Full-screen video player Obj-C wrapper for audio AV Foundation playback (2.2 only) Wednesday, October 20, 2010
  • 17. iPhone 3 Media Frameworks Core Audio / Low-level audio streaming OpenAL Media Player iPod library search/playback Obj-C wrapper for audio AV Foundation playback, recording Wednesday, October 20, 2010
  • 18. iOS 4 Media Frameworks Core Audio / Low-level audio streaming OpenAL Media Player iPod library search/playback Audio / Video capture, editing, AV Foundation playback, export… Core Video Quartz effects on moving images Objects for representing media Core Media times, formats, buffers Wednesday, October 20, 2010
  • 19. AV Foundation in iPhone 2.2 Wednesday, October 20, 2010
  • 20. AV Foundation in iPhone 3.2 Wednesday, October 20, 2010
  • 21. AV Foundation in iOS 4 Wednesday, October 20, 2010
  • 22. Size is relative AV android. QuickTime QT Kit Foundation media for Java Classes 56 40 24 576 Methods 460 280 360 >10,000 Wednesday, October 20, 2010
  • 23. How do media frameworks work? Wednesday, October 20, 2010
  • 24. Wednesday, October 20, 2010
  • 25. “Boom Box” APIs ✤ Simple API for playback, sometimes recording ✤ Little or no support for editing, mixing, metadata, etc. ✤ Example: HTML 5 <audio> tag Wednesday, October 20, 2010
  • 26. “Streaming” APIs ✤ Use “stream of audio” metaphor ✤ Strong support for mixing, effects, other real-time operations ✤ Example: Core Audio and AV Foundation (capture) Wednesday, October 20, 2010
  • 27. “Document” APIs ✤ Use “media document” metaphor ✤ Strong support for editing ✤ Mixing may be a special case of editing ✤ Example: QuickTime and AV Foundation (playback and editing) Wednesday, October 20, 2010
  • 28. AV Foundation Classes ✤ Capture ✤ Assets and compositions ✤ Playback, editing, and export ✤ Legacy classes Wednesday, October 20, 2010
  • 29. AVAsset ✤ A collection of time-based media data ✤ Sound, video, text (closed captions, subtitles, etc.) ✤ Each distinct media type is contained in a track ✤ An asset represents the arrangement of the tracks. The tracks represent the traits of the media’s presentation (volume, pan, affine transforms, opacity, etc.). ✤ Asset ≠ media. Track ≠ media. Media = media. ✤ Also contains metadata (where common to all tracks) Wednesday, October 20, 2010
  • 30. AVAsset subclasses ✤ AVURLAsset — An asset created from a URL, such as a song or movie file or network document/stream ✤ AVComposition — An asset created from assets in multiple files, used to combine and present media together. ✤ Used for editing Wednesday, October 20, 2010
  • 31. AVPlayer ✤ Provides the ability to play an asset ✤ play, pause, seekToTime: methods; currentTime, rate properties ✤ Init with URL or with AVPlayerItem NSURL *url = [NSURL URLWithString: @"http://www.subfurther.com/video/running-start- iphone.m4v"]; AVURLAsset *asset = [AVURLAsset URLAssetWithURL:url ! ! ! ! ! ! ! ! options:nil]; AVPlayerItem *playerItem = [AVPlayerItem playerItemWithAsset:asset]; player = [[AVPlayer playerWithPlayerItem:playerItem] retain]; Wednesday, October 20, 2010
  • 32. AVPlayerLayer (or not) ✤ CALayer used to display video from a player ✤ Check that the media has video NSArray *visualTracks = [asset tracksWithMediaCharacteristic: AVMediaCharacteristicVisual]; if ((!visualTracks) || ! ([visualTracks count] == 0)) { ! playerView.hidden = YES; ! noVideoLabel.hidden = NO; } Wednesday, October 20, 2010
  • 33. AVPlayerLayer (no really) ✤ If you have video, create AVPlayerLayer from AVPlayer. ✤ Set bounds and video “gravity” (bounds-filling behavior) else { ! playerView.hidden = NO; ! noVideoLabel.hidden = YES; ! AVPlayerLayer *playerLayer = [AVPlayerLayer playerLayerWithPlayer:player]; ! [playerView.layer addSublayer:playerLayer]; ! playerLayer.frame = playerView.layer.bounds; ! playerLayer.videoGravity = AVLayerVideoGravityResizeAspect; } Wednesday, October 20, 2010
  • 34. Demo VTM_AVPlayer Wednesday, October 20, 2010
  • 35. A brief sidetrack… Wednesday, October 20, 2010
  • 36. HTTP Live Streaming Wednesday, October 20, 2010
  • 37. HTTP Live Streaming ✤ Audio / Video network streaming standard developed by Apple ✤ Replaces RTP/RTSP ✤ Built-in support in iOS (AV Framework, Media Player) and Mac OS X 10.6 (QTKit) ✤ Required for apps that stream more than 10 MB over cellular network Wednesday, October 20, 2010
  • 38. How HTTP Live Streaming works ✤ Segmenting server splits source media into separate files (usually .m4a for audio-only, .ts for A/V), usually of about 10 seconds each, and creates an .m3u8 playlist file ✤ Playlist may point to bandwidth-appropriate playlists ✤ Clients download the playlist, fetch the segments, queue them up ✤ Server updates playlist periodically with latest segments; clients refresh playlist, fetch and queue new segments Wednesday, October 20, 2010
  • 39. Steve’s .m3u8 playlist #EXTM3U #EXT-X-TARGETDURATION:8 #EXT-X-MEDIA-SEQUENCE:0 #EXTINF:8, 0640/0640_090110_120505_0.ts #EXTINF:8, 0640/0640_090110_120505_1.ts #EXTINF:8, 0640/0640_090110_120505_2.ts #EXTINF:8, 0640/0640_090110_120505_3.ts #EXTINF:8, 0640/0640_090110_120505_4.ts Wednesday, October 20, 2010
  • 40. HTTP Live Streaming wins ✤ Works with existing file servers and content delivery networks ✤ Port 80 is never blocked ✤ Adapts to changes in available bandwidth ✤ Can be encrypted ✤ Has been submitted as a proposed IETF standard ✤ http://tools.ietf.org/htmłdraft-pantos-http-live-streaming-04 Wednesday, October 20, 2010
  • 41. HTTP Live Streaming fails ✤ Not really “live” when buffer can be a minute long ✤ Can’t watch a game on TV and listen to HLS web radio for the audio ✤ No meaningful adoption outside of the Apple world ✤ This may change before the next Stevenote. It’s an easy protocol to implement Wednesday, October 20, 2010
  • 42. Back to AV Foundation… Wednesday, October 20, 2010
  • 43. Media Capture ✤ AV Foundation capture classes for audio / video capture, along with still image capture ✤ Programmatic control of white balance, autofocus, zoom, etc. ✤ Does not exist on the simulator. AV Foundation capture apps can only be compiled for and run on the device. ✤ API design is borrowed from QTKit on the Mac Wednesday, October 20, 2010
  • 44. Capture Classes Seem Familiar? QT Kit AV Foundation QTCaptureAudioPreviewOutput AVCaptureAudioDataOutput QTCaptureConnection AVCaptureConnection QTCaptureDecompressedAudioOutput AVCaptureDevice QTCaptureDecompressedVideoOutput AVCaptureFileOutput QTCaptureDevice AVCaptureInput QTCaptureDeviceInput AVCaptureMovieFileOutput QTCaptureFileOutput AVCaptureOutput QTCaptureInput AVCaptureSession QTCaptureLayer AVCaptureStillImageOutput QTCaptureMovieFileOutput AVCaptureVideoDataOutput QTCaptureOutput AVCaptureVideoPreviewLayer QTCaptureSession QTCaptureVideoPreviewOutput QTCaptureView Wednesday, October 20, 2010
  • 45. Capture basics ✤ Create an AVCaptureSession to coordinate the capture ✤ Investigate available AVCaptureDevices ✤ Create AVCaptureDeviceInput and connect it to the session ✤ Optional: set up an AVCaptureVideoPreviewLayer ✤ Optional: connect AVCaptureOutputs ✤ Tell the session to start recording Wednesday, October 20, 2010
  • 46. Getting capture device and input AVCaptureDevice *videoDevice = [AVCaptureDevice defaultDeviceWithMediaType: AVMediaTypeVideo]; if (videoDevice) { ! NSLog (@"got videoDevice"); ! AVCaptureDeviceInput *videoInput = [AVCaptureDeviceInput deviceInputWithDevice:videoDevice ! ! ! ! ! ! ! ! error:&setUpError]; ! if (videoInput) { ! ! [captureSession addInput: videoInput]; ! } } Note 1: You may also want to check for AVMediaTypeMuxed Note 2: Do not assume devices based on model (c.f. iPad Camera Connection Kit) Wednesday, October 20, 2010
  • 47. Creating a video preview layer AVCaptureVideoPreviewLayer *previewLayer = [AVCaptureVideoPreviewLayer layerWithSession:captureSession]; previewLayer.frame = captureView.layer.bounds; previewLayer.videoGravity = AVLayerVideoGravityResizeAspect; [captureView.layer addSublayer:previewLayer]; Keep in mind that the iPhone cameras have a portrait orientation Wednesday, October 20, 2010
  • 48. Setting an output captureMovieOutput = [[AVCaptureMovieFileOutput alloc] init]; if (! captureMovieURL) { ! captureMoviePath = [getCaptureMoviePath() retain]; ! captureMovieURL = [[NSURL alloc] initFileURLWithPath:captureMoviePath]; } NSLog (@"recording to %@", captureMovieURL); [captureSession addOutput:captureMovieOutput]; We’ll use the captureMovieURL later… Wednesday, October 20, 2010
  • 49. Start capturing [captureSession startRunning]; recordButton.selected = YES; if ([[NSFileManager defaultManager] fileExistsAtPath:captureMoviePath]) { ! [[NSFileManager defaultManager] removeItemAtPath:captureMoviePath error:nil]; } // note: must have a delegate [captureMovieOutput startRecordingToOutputFileURL:captureMovieURL ! ! ! ! ! ! ! ! recordingDelegate:self]; Wednesday, October 20, 2010
  • 50. Capture delegate callbacks - (void)captureOutput:(AVCaptureFileOutput *)captureOutput didStartRecordingToOutputFileAtURL:(NSURL *)fileURL fromConnections:(NSArray *)connections { ! NSLog (@"started recording to %@", fileURL); } - (void)captureOutput:(AVCaptureFileOutput *)captureOutput didFinishRecordingToOutputFileAtURL:(NSURL *)outputFileURL fromConnections:(NSArray *)connections error:(NSError *)error { ! if (error) { ! ! NSLog (@"failed to record: %@", error); ! } else { ! ! NSLog (@"finished recording to %@", outputFileURL); ! } } Wednesday, October 20, 2010
  • 51. Demo VTM_AVRecPlay Wednesday, October 20, 2010
  • 52. More fun with capture ✤ Can analyze video data coming off the camera with the AVCaptureVideoDataOutput class ✤ Can provide uncompressed frames to your AVCaptureVideoDataOutputSampleBufferDelegate ✤ The callback provides you with a CMSampleBufferRef Wednesday, October 20, 2010
  • 53. Surprise! Core Media! Wednesday, October 20, 2010
  • 54. Core Media ✤ C-based framework containing structures that represent media samples and media timing ✤ Opaque types: CMBlockBuffer, CMBufferQueue, CMFormatDescription, CMSampleBuffer, CMTime, CMTimeRange ✤ Handful of convenience functions to work with these ✤ Buffer types provide wrappers around possibly-fragmented memory, time types provide timing at arbitrary precision Wednesday, October 20, 2010
  • 55. Media Editing Wednesday, October 20, 2010
  • 56. Video Editing? On iPhone? Really? 1999: 2010: Power Mac G4 500 AGP iPhone 4 CPU: 500 MHz G4 CPU: 800 MHz Apple A4 RAM: 256 MB RAM: 512 MB Storage: 20 GB HDD Storage: 16 GB Flash Comparison specs from everymac.com Wednesday, October 20, 2010
  • 57. AVComposition ✤ An AVAsset that gets its tracks from multiple file-based sources ✤ To create a movie, you typically use an AVMutableComposition composition = [[AVMutableComposition alloc] init]; Wednesday, October 20, 2010
  • 58. Copying from another asset ✤ -[AVMutableComposition insertTimeRange:ofAsset:atTime:error:] CMTime inTime = CMTimeMakeWithSeconds(inSeconds, 600); CMTime outTime = CMTimeMakeWithSeconds(outSeconds, 600); CMTime duration = CMTimeSubtract(outTime, inTime); CMTimeRange editRange = CMTimeRangeMake(inTime, duration); NSError *editError = nil; [targetController.composition insertTimeRange:editRange ! ! ! ! ! ! ! ofAsset:sourceAsset atTime:targetController.composition.duration ! ! ! ! ! ! ! error:&editError]; Wednesday, October 20, 2010
  • 59. CMTime ✤ CMTime contains a value and a timescale (similar to QuickTime) ✤ Time scale is how the time is measured: “nths of a second” ✤ Time in seconds = value / timescale ✤ Allows for exact timing of any kind of media ✤ Different tracks of an asset can and will have different timescales ✤ Convert with CMTimeConvertScale() Wednesday, October 20, 2010
  • 60. Track-level editing ✤ Add an AVMutableCompositionTrack to the composition ✤ Call -[AVMutableCompositionTrack insertTimeRange:ofTrack:atTime:error] musicAsset = [AVURLAsset URLAssetWithURL: musicURL ! ! ! ! ! ! ! ! options: nil]; CMTimeRange musicRange = CMTimeRangeMake( CMTimeMakeWithSeconds (0,600), composition.duration); AVAssetTrack *musicAssetTrack = [[musicAsset tracksWithMediaType:AVMediaTypeAudio] objectAtIndex: 0]; [musicTrack insertTimeRange:musicRange ! ! ! ! ! ofTrack:musicAssetTrack ! ! ! ! ! atTime:CMTimeMakeWithSeconds(0,600) ! ! ! ! ! error:&exportError]; Wednesday, October 20, 2010
  • 61. Export ✤ Create an AVAssetExportSession ✤ Must set outputURL and outputFileType properties ✤ Inspect possible types with supportedFileTypes property (list of AVFileType… strings in docs) ✤ Begin export with exportAsynchronouslyWithCompletionHandler: ✤ This takes a block, which will be called on completion, failure, cancellation, etc. Wednesday, October 20, 2010
  • 62. Export AVAssetExportSession *exportSession = [[AVAssetExportSession alloc] initWithAsset:composition ! ! presetName:AVAssetExportPresetMediumQuality]; exportSession.outputURL = exportURL; exportSession.outputFileType = @"com.apple.quicktime-movie"; [exportSession exportAsynchronouslyWithCompletionHandler:^{ ! NSLog (@"i is in your block, exportin. status is %d", ! ! exportSession.status); ! switch (exportSession.status) { ! ! case AVAssetExportSessionStatusFailed: ! ! case AVAssetExportSessionStatusCompleted: { ! ! ! [self performSelectorOnMainThread: @selector (doPostExportUICleanup:) ! ! ! ! ! ! ! ! withObject:nil ! ! ! ! ! ! ! ! waitUntilDone:NO]; ! ! ! break; ! ! } ! }; }]; Wednesday, October 20, 2010
  • 63. Demo VTM_AVEditor Wednesday, October 20, 2010
  • 64. Effects 1 ✤ AVAudioMix, AVMutableAudioMix: set volumes or audio ramps at specific times ✤ AVVideoCompositionInstructions: provide a set of layer-based instructions for performing time-based opacity or affine transform ramps Wednesday, October 20, 2010
  • 65. Effects 2 ✤ AVSynchronizedLayer: CALayer that synchronizes with a AVPlayerItem’s playback timing ✤ Use for overlays, titles, rendered images, Ken Burns effects, etc. ✤ Exporting any of these effects is tricky: have to create, configure, and set AVAudioMix and AVVideoComposition properties on the exporter Wednesday, October 20, 2010
  • 66. Sample-level access ✤ AVAssetReader and AVAssetWriter introduced in iOS 4.1 ✤ Similar to capture: you add AVAssetReaderOutputs and AVAssetWriterInputs to the readers and writers, respectively ✤ Reads and writes use CMSampleBufferRef pointers ✤ You can re-encode media by setting up the AVAssetWriter with an NSDictionary of output settings ✤ Read through AVAudioSettings.h and AVVideoSettings.h to find the appropriate keys. Be prepared for lots of trial and error. Wednesday, October 20, 2010
  • 67. Stuff to watch for… Wednesday, October 20, 2010
  • 68. Interfaces to other iOS frameworks ✤ MPMediaItem now has a MPMediaItemPropertyAssetURL that allows you to open iPod Library songs as AVURLAssets ✤ ALAssetLibrary URLs can also be opened with AV Foundation ✤ Core Media has functions to convert CMSampleBuffers to and from Core Audio AudioBufferLists ✤ Accelerate framework’s vDSP functions may be useful when processing samples from AVCaptureDataOutput or AVAssetReader Wednesday, October 20, 2010
  • 69. You can’t help but speculate… AVAssetExportSession.h: extern NSString *const AVAssetExportPreset1280x720 __OSX_AVAILABLE_STARTING(__MAC_10_7,__IPHONE_4_0); CMTime.h: CM_EXPORT const CFStringRef kCMTimeScaleKey __OSX_AVAILABLE_STARTING(__MAC_10_7,__IPHONE_4_0); Wednesday, October 20, 2010
  • 70. Q&A Time ✤ Cleaned-up code will be available on my blog: ✤ http://www.subfurther.com/blog ✤ invalidname [at] gmail.com ✤ @invalidname ✤ Check out the Rough Cut of my Core Audio book too. ✤ Get the WWDC 2010 sessions (there are 3 on AV Foundation!) Wednesday, October 20, 2010

×