• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Capturing Stills, Sounds, and Scenes with AV Foundation
 

Capturing Stills, Sounds, and Scenes with AV Foundation

on

  • 7,442 views

AV Foundation -- introduced in iOS 4, ported to Lion, and enhanced further in iOS 5 -- delivers a comprehensive framework for audio and video capture and playback. The capture functionality is so ...

AV Foundation -- introduced in iOS 4, ported to Lion, and enhanced further in iOS 5 -- delivers a comprehensive framework for audio and video capture and playback. The capture functionality is so good, it's now the preferred option for still photography applications. In this session, we'll focus squarely on AV Foundation as a media capture framework. Attendees will learn:
* How to get the most out of the device for still photography, by using AV Foundation to access the flash, white-balance, and image resolution.
* How to capture audio and video to the file system
* How to process incoming audio and video capture buffers in memory, to create real-time effects or pick out interesting parts of the scene on the fly

Statistics

Views

Total Views
7,442
Views on SlideShare
7,441
Embed Views
1

Actions

Likes
7
Downloads
0
Comments
2

1 Embed 1

http://www.verious.com 1

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

12 of 2 previous next

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • Hi Chris Adamson,
    This is really very helpful document to learn AV Foudation framework.
    Thanks a lot.
    But My Question is:- Can I connect external device (camera) as input Device as AVCaptureInpute Device?
    Please reply me...
    Are you sure you want to
    Your message goes here
    Processing…
  • This document is very useful for me to learn how to use AV Foundation framework, Thanks a lot, in addition, how can i to contact with you if i have any question about AV Foundation?
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Capturing Stills, Sounds, and Scenes with AV Foundation Capturing Stills, Sounds, and Scenes with AV Foundation Presentation Transcript

    • Capturing Stills, Sounds, and Scenes with AV Foundation Chris Adamson • @invalidname Voices That Matter: iOS Developer Conference Nov. 12, 2011 • Boston, MATuesday, November 15, 11
    • Road Map • Media capture technologies in iOS • AV Foundation capture concepts • Device-specific concerns • Doing stuff with captured mediaTuesday, November 15, 11
    • Capture? • Digital media encoding of some real-world source, such as still images, moving images, and/or sound • Contrast with synthetic media: musical synthesizers, CG animation • Not the same as "recording", which implies storage • Capture devices include cameras and microphonesTuesday, November 15, 11
    • iOS Capture DevicesTuesday, November 15, 11
    • Accessing Capture Devices • Simple shoot-and-save - UIImagePickerController • Core Audio - low level capture and real-time processing • More info in my talk tomorrow • AV FoundationTuesday, November 15, 11
    • AV Foundation • Introduced in iPhone OS 2.3 as Obj-C wrapper for Core Audio playback, added capture in 3.0 • Repurposed in iOS 4 as audio/video capture, editing, export, and playback framework • Ported to OS X in Lion, heir apparent to QuickTimeTuesday, November 15, 11
    • #import this! • AVFoundation.framework • CoreMedia.framework • Possibly also: • CoreVideo, CoreImage, CoreGraphics • AudioToolbox, AudioUnitsTuesday, November 15, 11
    • Core Media • C-based helper framework for AVF • Structures to represent media buffers and queues of buffers, media times and time ranges • Low-level conversion and calculation functions • Does not provide capture, editing, or playback functionalityTuesday, November 15, 11
    • AV Foundation • Editing / Playback classes • Assets, compositions, and tracks. Player and player layer. Asset readers and writers • Capture classes • Devices, inputs, outputs, and the sessionTuesday, November 15, 11
    • How it fits together…Tuesday, November 15, 11
    • Tuesday, November 15, 11
    • AVCaptureSessionTuesday, November 15, 11
    • AVCaptureDevice AVCaptureSessionTuesday, November 15, 11
    • AVCaptureDevice AVCaptureInput AVCaptureSessionTuesday, November 15, 11
    • AVCaptureDevice AVCaptureDevice AVCaptureInput AVCaptureSessionTuesday, November 15, 11
    • AVCaptureDevice AVCaptureDevice AVCaptureInput AVCaptureInput AVCaptureSessionTuesday, November 15, 11
    • AVCaptureDevice AVCaptureDevice AVCaptureInput AVCaptureInput AVCaptureVideo AVCaptureSession PreviewLayerTuesday, November 15, 11
    • AVCaptureDevice AVCaptureDevice AVCaptureInput AVCaptureInput AVCaptureVideo AVCaptureSession PreviewLayer AVCaptureOutputTuesday, November 15, 11
    • AVCaptureDevice AVCaptureDevice AVCaptureInput AVCaptureInput AVCaptureVideo AVCaptureSession PreviewLayer AVCaptureOutputTuesday, November 15, 11
    • AVCaptureDevice AVCaptureDevice AVCaptureInput AVCaptureInput AVCaptureVideo AVCaptureSession PreviewLayer AVCaptureOutput AVCaptureOutputTuesday, November 15, 11
    • AVCaptureDevice AVCaptureDevice AVCaptureInput AVCaptureInput AVCaptureVideo AVCaptureSession PreviewLayer AVCaptureOutput AVCaptureOutputTuesday, November 15, 11
    • AVCaptureSession • Coordinates the flow of capture from inputs to outputs • Create, add inputs and outputs, start running captureSession = [[AVCaptureSession alloc] init];Tuesday, November 15, 11
    • AVCaptureDevice • Represents a device that can perform media capture (cameras, microphones) • Could be connected as external accessory or Bluetooth • You cannot make assumptions based on device modelTuesday, November 15, 11
    • Discovering Devices • AVCaptureDevice class methods devices, deviceWithUniqueID, devicesWithMediaType, defaultDeviceWithMediaType • Media types include audio, video, muxed (audio and video in one stream), plus some outliers (timecode, etc.)Tuesday, November 15, 11
    • Inspecting Devices • position (property): is the camera on the front or the back? • supportsAVCaptureSessionPreset: allows you to inspect whether it can copy at one of several predefined image resolutionsTuesday, November 15, 11
    • Photo traits • Focus & exposure • isFocusModeSupported:, focusMode, focusPointOfInterestSupported, focusPointOfInterest, focusAdjusting • isExposureModeSupported:, exposureMode, exposurePointOfInterestSupported, etc. • White balance • isWhiteBalanceModeSupported:, whiteBalanceMode, whiteBalanceModeAdjustingTuesday, November 15, 11
    • Light up • Flash and Torch • hasFlash, isFlashModeSupported: flashMode, flashActive, flashAvailable • hasTorch, isTorchModeSupported:, torchMode, torchLevel, torchAvailableTuesday, November 15, 11
    • AVCaptureSession sessionPreset • Constants for video capture quality. Allows you to inspect capabilities, trade performance/framerate for resolution • Default is AVCaptureSessionPresetHigh • For still photos: AVCaptureSessionPresetPhotoTuesday, November 15, 11
    • iFrame • Session presets for use when capturing video intended for subsequent editing • AVCaptureSessionPresetiFrame960x540, AVCaptureSessionPresetiFrame1280x720 • No P- or B-frames; files are much larger than typical H.264. http://en.wikipedia.org/wiki/Video_compression_picture_typesTuesday, November 15, 11
    • Capture inputs • Connect a device to the capture session • Instances of AVCaptureDeviceInput • create with -initWithDevice:error: or deviceInputForDevice:errorTuesday, November 15, 11
    • AVCaptureDevice *videoDevice = ! [AVCaptureDevice defaultDeviceWithMediaType: ! AVMediaTypeVideo]; if (videoDevice) { ! AVCaptureDeviceInput *videoInput = ! [AVCaptureDeviceInput ! ! deviceInputWithDevice:videoDevice ! ! ! ! ! ! ! ! error:&setUpError]; ! if (videoInput) { ! ! [captureSession addInput: videoInput]; ! } }Tuesday, November 15, 11
    • Capture preview • AVCapturePreviewLayer: A CALayer that shows whats currently being captured from video input • Remember: CALayer, not UIView • videoGravity property determines how it will deal with preview that doesnt match bounds: aspect, fill, or resizeTuesday, November 15, 11
    • AVCaptureVideoPreviewLayer *previewLayer = ! [AVCaptureVideoPreviewLayer ! ! layerWithSession:captureSession]; previewLayer.frame = captureView.layer.bounds; previewLayer.videoGravity = ! AVLayerVideoGravityResizeAspect; [captureView.layer addSublayer:previewLayer];Tuesday, November 15, 11
    • Capture Outputs • File output: AVCaptureMovieFileOutput and AVCaptureAudioFileOutput • Photo output: AVCaptureStillImageOutput • Image processing: AVCaptureDataOutput • More on this one later…Tuesday, November 15, 11
    • AVCaptureFileOutput • startRecordingToOutputURL:recordingDelegate: • The delegate must be set and must implement two callbacks: • captureOutput:didStartRecordingToOutputFileAt  URL:fromConnections: • captureOutput:didFinishRecordingToOutputFileAt  URL:fromConnections: • Then connect to capture sessionTuesday, November 15, 11
    • captureMovieOutput = ! [[AVCaptureMovieFileOutput alloc] init]; if (! captureMovieURL) { ! captureMoviePath = [getCaptureMoviePath() retain]; ! captureMovieURL = [[NSURL alloc] ! ! ! initFileURLWithPath:captureMoviePath]; } NSLog (@"recording to %@", captureMovieURL); [captureSession addOutput:captureMovieOutput];Tuesday, November 15, 11
    • Cranking it up • -[AVCaptureSession startRunning] starts capturing from all connected inputs • If you have a preview layer, it will start getting updated • File outputs do not start writing to filesystem until you call startRecording on themTuesday, November 15, 11
    • Demo AVRecPlay http://dl.dropbox.com/u/12216224/conferences/vtm10/mastering-media-with-av- foundation/VTM_AVRecPlay.zipTuesday, November 15, 11
    • Orientation issues • Default orientation of an iOS device is portrait • The AVCaptureConnections between the device inputs and the session have a read- write videoOrientation property. • Capture layers orientation property should matchTuesday, November 15, 11
    • Capture Processing • Analyzing or manipulating capture data as it comes in • Audio: real-time effects ("I Am T-Pain"), oscilloscopes, etc. • May make more sense to use Audio Units • Video: bar code readers, face-finders, etc.Tuesday, November 15, 11
    • Data Outputs • Connects your code to the capture session via a delegate callback • Delegate callback occurs on a serial GCD queue that you provide (can be dispatch_get_main_queue(), should not be dispatch_get_current_queue(), must not be NULL).Tuesday, November 15, 11
    • Creating the data output AVCaptureVideoDataOutput *captureOutput = ! [[AVCaptureVideoDataOutput alloc] init]; captureOutput.alwaysDiscardsLateVideoFrames = ! YES; [captureOutput setSampleBufferDelegate:self ! ! ! ! queue:dispatch_get_main_queue()];Tuesday, November 15, 11
    • Configuring the data output NSString* key = ! (NSString*)kCVPixelBufferPixelFormatTypeKey; NSNumber* value = ! [NSNumber numberWithUnsignedInt: ! ! kCVPixelFormatType_32BGRA]; NSDictionary* videoSettings = [NSDictionary ! dictionaryWithObject:value forKey:key]; [captureOutput setVideoSettings:videoSettings];Tuesday, November 15, 11
    • Analyzing the data • You get the callback captureOutput: didOutputSampleBuffer:fromConnection: • Second parameter is a CMSampleBufferRef, Core Medias opaque type for sample buffers • Could be video… could be audio… (but you can tell from the connection and its input and output ports)Tuesday, November 15, 11
    • Analyzing frames with Core Video CVImageBufferRef imageBuffer = ! CMSampleBufferGetImageBuffer(sampleBuffer); /*Lock the image buffer*/ CVPixelBufferLockBaseAddress(imageBuffer,0); /*Get information about the image*/ size_t bytesPerRow = ! CVPixelBufferGetBytesPerRow(imageBuffer); size_t width = CVPixelBufferGetWidth(imageBuffer); size_t height = CVPixelBufferGetHeight(imageBuffer); This example is from the ZXing barcode reader http://code.google.com/p/zxing/Tuesday, November 15, 11
    • Demo ZXing http://code.google.com/p/zxing/Tuesday, November 15, 11
    • Audio considerations • Can process CMSampleBufferRef by using CMSampleBufferGetAudioStreamPacket Descriptions() and CMSampleBufferGet AudioBufferListWithRetainedBlockBuffer() • Then use Core Audio call that take these types • May make more sense to just capture in Core Audio in the first place, especially if youre playing captured data through an audio queue or audio unitsTuesday, November 15, 11
    • Face Finding in iOS 5 • iOS 5 introduces Core Image, which allows us to chain effects on images • Also includes some interesting image processing classesTuesday, November 15, 11
    • CIDetector • Core Image class to find features in a Core Image buffer • Only supported detector type in iOS 5 is CIDetectorTypeFace • featuresInImage: returns an NSArray of all detected features in the imageTuesday, November 15, 11
    • Convert CM to CV to CI CVPixelBufferRef cvPixelBuffer = ! CMSampleBufferGetImageBuffer(sampleBuffer); CFDictionaryRef attachmentsDict = ! CMCopyDictionaryOfAttachments( ! ! kCFAllocatorSystemDefault, ! ! sampleBuffer, ! ! kCMAttachmentMode_ShouldPropagate); CIImage *ciImage = [[CIImage alloc] ! ! initWithCVPixelBuffer:cvPixelBuffer ! ! options:(__bridge NSDictionary*) ! ! ! ! attachmentsDict];Tuesday, November 15, 11
    • Creating the CIDetector NSDictionary *faceDetectorDict = ! ! [NSDictionary dictionaryWithObjectsAndKeys: ! ! ! CIDetectorAccuracyHigh, ! ! ! CIDetectorAccuracy, ! ! ! nil]; CIDetector *faceDetector = [CIDetector detectorOfType:CIDetectorTypeFace context:nil options:faceDetectorDict]; NSArray *faces = [faceDetector ! ! ! featuresInImage:ciImage];Tuesday, November 15, 11
    • Demo VTMFaceFinder http://dl.dropbox.com/u/12216224/conferences/vtm11/VTMFaceFinder.zipTuesday, November 15, 11
    • Boxing the faces for (CIFaceFeature *faceFeature in self.facesArray) { CGRect boxRect = CGRectMake( faceFeature.bounds.origin.x * self.scaleToApply, faceFeature.bounds.origin.y * self.scaleToApply, faceFeature.bounds.size.width * self.scaleToApply, faceFeature.bounds.size.height * self.scaleToApply); CGContextSetStrokeColorWithColor(cgContext, [UIColor yellowColor].CGColor); CGContextStrokeRect(cgContext, boxRect); }Tuesday, November 15, 11
    • CIFaceFeature • Inherits bounds from CIFeature • Adds CGPoint properties leftEyePosition, rightEyePosition, and mouthPosition (with "has" properties for each of these)Tuesday, November 15, 11
    • Image Processing on the fly • New CVOpenGLESTextureCache makes it possible to render Core Video buffers in real time • These are what you get in the callback • See ChromaKey example from WWDC 2011 session 419. Requires mad OpenGL ES skillz.Tuesday, November 15, 11
    • Ericas "Face Pong"Tuesday, November 15, 11
    • Recap • Start with an AVCaptureSession • Discover devices and create inputs • Create and configure outputs • Start the session • Start recording or wait to start handling callbacksTuesday, November 15, 11
    • Recap: Easy parts • Basic capture apps (preview-only or record to file) will require little or no Core Media or other C APIs. • Default devices are usually the one you want (back megapixel camera on the iPhone, best available microphone, etc.) • Capture API is pretty easy to understand and remember (compare to the editing API)Tuesday, November 15, 11
    • Recap: Hard parts • Core Media calls require high comfort level with C, Core Foundation, functions that take 8 or more parameters, etc. • Lots of bit-munging when you parse a CV buffer (pixel formats, strides) • Callbacks do not have an infinite amount of time or resources to finish their workTuesday, November 15, 11
    • Resources • devforums.apple.com • No mailing list at lists.apple.com • WWDC session videos and slides (four in 2011, three in 2010) • Stack OverflowTuesday, November 15, 11
    • Q&A Watch my blog for updated sample code: http://www.subfurther.com/blog @invalidnameTuesday, November 15, 11