The document discusses using Apple's AV Foundation framework to work with audiovisual media on iOS and OS X, including playing, editing, and composing media through techniques like adding transitions between video clips, adjusting audio levels over time, and layering visual elements. It provides an overview of core AV Foundation concepts like assets, tracks, and playback before demonstrating examples of common media tasks.
The AV Foundation has grown from a simple audio player quietly added in iPhone OS 2.2 to an extraordinarily ambitious media-creation framework in iOS 4. The latest version provides highly-customizable audio and video capture, deep support for editing and Core Animation-based effects, and export. It is far more comprehensive than nearly any other platform's media framework, mobile or desktop. As a developer, this gives you incredible power… and one heck of a learning curve. In this session, we will take a ground-up look at AV Foundation, starting with its core concepts and patterns, and moving through its most practical and powerful capabilities. Along the way, we'll see how AV Foundation works with iOS' other media APIs -- Core Audio, Media Player, and Core Media -- and how it aggressively uses new iOS 4 programming paradigms like blocks and Grand Central Dispatch. Attendees will come away understanding the straight-arrow paths to AV Foundation's most important features like capture, editing, and export, and how its pieces might be combined in interesting ways to create even more powerful media applications.
Core Image: The Most Fun API You're Not Using, CocoaConf Atlanta, December 2014Chris Adamson
Graphics on iOS and OS X isn't just about stroking shapes and paths in Core Graphics and trying to figure out OpenGL. The Core Image framework gives you access to about 100 built-in filters, providing everything from photographic effects and color manipulation to face-finding and QR Code generation. It can leverage the power of the GPU to provide performance fast enough to perform complex effects work on real-time video capture. But even if you're not writing the next Final Cut Pro or Photoshop, it's easy to call in Core Image for simple tasks, like putting a blur in part of your UI for transitions or privacy reasons. In this session, we'll explore the many ways Core Image can make your app sizzle
The AV Foundation has grown from a simple audio player quietly added in iPhone OS 2.2 to an extraordinarily ambitious media-creation framework in iOS 4. The latest version provides highly-customizable audio and video capture, deep support for editing and Core Animation-based effects, and export. It is far more comprehensive than nearly any other platform's media framework, mobile or desktop. As a developer, this gives you incredible power… and one heck of a learning curve. In this session, we will take a ground-up look at AV Foundation, starting with its core concepts and patterns, and moving through its most practical and powerful capabilities. Along the way, we'll see how AV Foundation works with iOS' other media APIs -- Core Audio, Media Player, and Core Media -- and how it aggressively uses new iOS 4 programming paradigms like blocks and Grand Central Dispatch. Attendees will come away understanding the straight-arrow paths to AV Foundation's most important features like capture, editing, and export, and how its pieces might be combined in interesting ways to create even more powerful media applications.
Core Image: The Most Fun API You're Not Using, CocoaConf Atlanta, December 2014Chris Adamson
Graphics on iOS and OS X isn't just about stroking shapes and paths in Core Graphics and trying to figure out OpenGL. The Core Image framework gives you access to about 100 built-in filters, providing everything from photographic effects and color manipulation to face-finding and QR Code generation. It can leverage the power of the GPU to provide performance fast enough to perform complex effects work on real-time video capture. But even if you're not writing the next Final Cut Pro or Photoshop, it's easy to call in Core Image for simple tasks, like putting a blur in part of your UI for transitions or privacy reasons. In this session, we'll explore the many ways Core Image can make your app sizzle
Stupid Video Tricks (CocoaConf DC, March 2014)Chris Adamson
AV Foundation makes it reasonably straightforward to capture video from the camera and edit together a nice family video. This session is not about that stuff. This session is about the nooks and crannies where AV Foundation exposes what's behind the curtain. Instead of letting AVPlayer read our video files, we can grab the samples ourselves and mess with them. AVCaptureVideoPreviewLayer, meet the CGAffineTransform. And instead of dutifully passing our captured video frames to the preview layer and an output file, how about if we instead run them through a series of Core Image filters? Record your own screen? Oh yeah, we can AVAssetWriter that. With a few pointers, a little experimentation, and a healthy disregard for safe coding practices, Core Media and Core Video let you get away with some neat stuff.
Grand Central Dispatch (GCD) dispatch queues are a powerful tool for performing tasks. Dispatch queues let you execute arbitrary blocks of code either asynchronously or synchronously with respect to the caller.
Refresh what you know about AssetDatabase.Refresh()- Unite Copenhagen 2019Unity Technologies
The AssetDatabase has been rewritten. The more you know about how this API works, the stronger your code will be. This information can guide your decision-making for your own Asset Management strategies. For example, you do not need to reimport assets when you jump between platforms. In this session, you'll gain a deeper understanding of importing modified assets and tracking dependencies to improve your workflow and iteration time significantly.
Speaker: Javier Abud Chavez- Unity
Watch the session on Youtube: https://youtu.be/S2P9n5U9xVw
Updated version using Swift 3 is available here:
https://www.slideshare.net/bobmccune/quartz-2d-with-swift-3/
Presentation by Bob McCune of TapHarmonic detailing how to use the Quartz framework to perform 2D drawing on the iOS platform. Quartz Demos:
https://github.com/tapharmonic/QuartzDemos
Slides for my Master Video session at Renaissance 2014. This session provided a high-level overview of some of AV Foundation's video playback and editing capabilities.
The demo app for this talk can be found at:
https://github.com/tapharmonic/AVFoundationEditor
Stupid Video Tricks (CocoaConf DC, March 2014)Chris Adamson
AV Foundation makes it reasonably straightforward to capture video from the camera and edit together a nice family video. This session is not about that stuff. This session is about the nooks and crannies where AV Foundation exposes what's behind the curtain. Instead of letting AVPlayer read our video files, we can grab the samples ourselves and mess with them. AVCaptureVideoPreviewLayer, meet the CGAffineTransform. And instead of dutifully passing our captured video frames to the preview layer and an output file, how about if we instead run them through a series of Core Image filters? Record your own screen? Oh yeah, we can AVAssetWriter that. With a few pointers, a little experimentation, and a healthy disregard for safe coding practices, Core Media and Core Video let you get away with some neat stuff.
Grand Central Dispatch (GCD) dispatch queues are a powerful tool for performing tasks. Dispatch queues let you execute arbitrary blocks of code either asynchronously or synchronously with respect to the caller.
Refresh what you know about AssetDatabase.Refresh()- Unite Copenhagen 2019Unity Technologies
The AssetDatabase has been rewritten. The more you know about how this API works, the stronger your code will be. This information can guide your decision-making for your own Asset Management strategies. For example, you do not need to reimport assets when you jump between platforms. In this session, you'll gain a deeper understanding of importing modified assets and tracking dependencies to improve your workflow and iteration time significantly.
Speaker: Javier Abud Chavez- Unity
Watch the session on Youtube: https://youtu.be/S2P9n5U9xVw
Updated version using Swift 3 is available here:
https://www.slideshare.net/bobmccune/quartz-2d-with-swift-3/
Presentation by Bob McCune of TapHarmonic detailing how to use the Quartz framework to perform 2D drawing on the iOS platform. Quartz Demos:
https://github.com/tapharmonic/QuartzDemos
Slides for my Master Video session at Renaissance 2014. This session provided a high-level overview of some of AV Foundation's video playback and editing capabilities.
The demo app for this talk can be found at:
https://github.com/tapharmonic/AVFoundationEditor
Presentation by Bob McCune of TapHarmonic detailing how to use the Quartz (CoreGraphics) framework with Swift 3.
Sample code available here: https://github.com/tapharmonic/QuartzDemosSwift
Forward Swift 2017: Media Frameworks and Swift: This Is FineChris Adamson
Swift is great for writing iOS and Mac apps, and its creators also mean for it to be used as a systems programming language. However, certain traits about Swift make it officially off-limits for use in some audio/video-processing scenarios. What's the deal, is it not fast enough or what? We'll look at what media apps can and can't do in Swift, and what you're supposed to do instead. We'll also look at strategies for knowing what responsibilities to dole out to Swift and to C, and how to make those parts of your code play nicely with each other.
Some of my favourite bits of AVFoundation. Topics include capture, composition, a custom player and scrubber interface, synchronized CAAnimations, and real-time VFX.
, AV Foundation moves to center stage as the essential media framework on the device, offering support for playing, capturing, and even editing audio and video. Borrowing some of the core ideas from the Mac's QuickTime, while adding many new concepts of its own, AV Foundation offers extraordinary capabilities for application programmers. This talk will offer a high-level overview of what's in AV Foundation, and a taste of what it can do.
AV Foundation makes it reasonably straightforward to capture video from the camera and edit together a nice family video. This session is not about that stuff. This session is about the nooks and crannies where AV Foundation exposes what's behind the curtain. Instead of letting AVPlayer read our video files, we can grab the samples ourselves and mess with them. AVCaptureVideoPreviewLayer, meet the CGAffineTransform. And instead of dutifully passing our captured video frames to the preview layer and an output file, how about if we instead run them through a series of Core Image filters? Record your own screen? Oh yeah, we can AVAssetWriter that. With a few pointers, a little experimentation, and a healthy disregard for safe coding practices, Core Media and Core Video let you get away with some neat stuff.
This presentation reviews the various approaches you can take to creating animations in your iOS apps.
It discusses UIKit animations, including curling pages up (as in the Maps app), flipping back and forth (as in the Stocks app) and image view animations.
It also describes Core Animation, including its implicit animations that occur when you change the value of certain animatable properties, as well as how to create explicit animations.
So you are a young, inexperienced iOS developer preparing for your first ever interview. But you don’t know what they are going to throw at you during the interview. During your student’s life you have been searching for the answers, this is the first time you are looking for the questions. This power point presentation is the right answer for your quest of looking for the interview questions. These are the probable questions interviewee may ask you for the post of iOS developer. Good luck.
Created by: Cygnis Media
Stupid Video Tricks, CocoaConf Las VegasChris Adamson
AV Foundation makes it reasonably straightforward to capture video from the camera and edit together a nice family video. This session is not about that stuff. This session is about the nooks and crannies where AV Foundation exposes what's behind the curtain. Instead of letting AVPlayer read our video files, we can grab the samples ourselves and mess with them. AVCaptureVideoPreviewLayer, meet the CGAffineTransform. And instead of dutifully passing our captured video frames to the preview layer and an output file, how about if we instead run them through a series of Core Image filters? Record your own screen? Oh yeah, we can AVAssetWriter that. With a few pointers, a little experimentation, and a healthy disregard for safe coding practices, Core Media and Core Video let you get away with some neat stuff.
Advanced AV Foundation (CocoaConf, Aug '11)Chris Adamson
The iOS version of iMovie uses the AV Foundation framework, and indications are that Final Cut Pro X will be using the Mac OS X version of AVF. And if AV Foundation is powerful enough to provide the core functionality of Final Cut, it must have some great stuff going on, right? In this session, we'll dig into the more powerful (and more challenging) APIs in AV Foundation, including reading and writing raw samples, performing live processing of incoming data at capture time, and advanced editing features like mixing audio and video tracks and adding Core Animation-based titles.
Stupid Video Tricks, CocoaConf Seattle 2014Chris Adamson
AV Foundation makes it reasonably straightforward to capture video from the camera and edit together a nice family video. This session is not about that stuff. This session is about the nooks and crannies where AV Foundation exposes what's behind the curtain. Instead of letting AVPlayer read our video files, we can grab the samples ourselves and mess with them. AVCaptureVideoPreviewLayer, meet the CGAffineTransform. And instead of dutifully passing our captured video frames to the preview layer and an output file, how about if we instead run them through a series of Core Image filters? Record your own screen? Oh yeah, we can AVAssetWriter that. With a few pointers, a little experimentation, and a healthy disregard for safe coding practices, Core Media and Core Video let you get away with some neat stuff
Anton Venema describes how Frozen Mountain is using Netflix VMAF to assess perceptual video quality in automated testing for LiveSwitch and IceLink.
Presented at CommCon 2019:
https://www.youtube.com/watch?v=uM_ZpIS-lO
References:
https://medium.com/netflix-techblog/toward-a-practical-perceptual-video-quality-metric-653f208b9652
https://medium.com/netflix-techblog/vmaf-the-journey-continues-44b51ee9ed12
https://github.com/netflix/vmaf
https://ece.uwaterloo.ca/~z70wang/research/ssim
https://peach.blender.org
https://media.xiph.org/video/derf
https://frozenmountain.com
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2023/11/building-accelerated-gstreamer-applications-for-video-and-audio-ai-a-presentation-from-wave-spectrum/
Abdo Babukr, Accelerated Computing Consultant at Wave Spectrum, presents the “Building Accelerated GStreamer Applications for Video and Audio AI,” tutorial at the May 2023 Embedded Vision Summit.
GStreamer is a popular open-source framework for creating streaming media applications. Developers often use GStreamer to streamline the development of computer vision and audio perception applications. Since perceptual algorithms are often quite demanding in terms of processing performance, in many cases developers need to find ways to accelerate key GStreamer building blocks, taking advantage of specialized features of their target processor or co-processor.
In this talk, Babukr introduces GStreamer and shows how to use it to build computer vision and audio perception applications. He also shows how to create efficient, high-performance GStreamer applications that utilize specialized hardware features.
WebRTC Standards & Implementation Q&A - All about browser interoperabilityAmir Zmora
Hosting guest speaker Jan-Ivar Bruaroey of Mozilla Firefox WebRTC R&D team and contributor to adapter.js talking about Chrome and Firefox interoperability.
Web Standards for AR workshop at ISMAR13Rob Manson
This work was presented at the Open Standards session at the IEEE ISMAR 2013 event. It provides a detailed overview and working examples that show exactly where Augmented Reality and Computer Vision are up to on the Web Platform.
This presentation also provides a detailed description of how to define exactly what the Augmented Web is.
Optimising video delivery - Brightcove PLAY 2019Jeremy Brown
7plus has taken advantage of the flexibility of the Brightcove video platform using the latest features Context Aware Encoding (CAE), Delivery Rules driven when driven by the 7plus automated quality assessment (QA) ingest workflow to mitigate common compromises to deliver better video experiences for our users
I presented at Brightcove PLAY May 2019 (suffering severe jet lag), thanks to everyone involved this was a fantastic event
A brief rollerskate along HTML5 multimedia beach, in which we pop into the soda shop of subtitling and the ice-cream parlour of synchronised media, before we incongruously pop into the igloo of JavaScript access to the camera (because I pulled in from slides from another presso after we talked about it in an earlier presentation).
You can gain a lot from a stable suite of end-to-end tests, but not a lot of developers know how easy it is to setup these tests. Dive into this crash course on end-to-end testing, you'll first learn the basics and then some advanced techniques that will have you testing like a pro in no time.
This presentation illustrates a solution for rapid deployment and validation of VoLTE services in the NFV ecosystem. It uses Juju as the deployment tool and ABot as Test Orchestration solution to achieve this objective.
Similar to Composing and Editing Media with AV Foundation (20)
2. About...
Bob McCune
‣ MN Developer and Instructor
‣ Owner of TapHarmonic, LLC.
‣ Founded Minnesota CocoaHeads in 2008
3. Agenda
What will I learn?
‣ AV Foundation Overview
‣ Media Playback
‣ Media Editing
‣ Composing Media
‣ Mixing Audio
‣ Building Video Transitions
‣ Layering Visual Content
4. AV Foundation
Overview
‣ Apple’s advanced Objective-C framework for
working with timed-media
‣ High performance, asynchronous processing
‣ Hardware accelerated handling of AV media
‣ Available in its current form since iOS 4
‣ Significant additions and enhancements iOS 5 and 6
‣ Default media framework on Mac OS X since 10.7 Lion
‣ Apple’s focus for media apps on both iOS and Mac
5. Where does it fit?
iOS Media Options
Simple Awesome Complex
MediaPlayer CoreAudio
UIKit CoreMedia
AVFoundation
CoreVideo
CoreAnimation
7. Media Assets
Understanding Assets
‣ AVAsset is an abstract representation of media resource
modeling the static aspects of the media.
‣ Abstracts away the type and location
‣ AVAssetTrack models the static aspects of the
individual media streams within an asset
‣ Tracks are of a uniform type (video, audio, etc.)
AVAssetTrack (Video)
AVAssetTrack (Audio)
8. Using AVAssets
What can I do with an asset?
‣ Inspect
‣ Generate Images
‣ Transcode and Export
‣ Playback
10. AVPlayer
Playback Controller
‣ AVPlayer is a controller for managing playback
‣ play
‣ pause
‣ rate
‣ seekToTime:
‣ Use KVO to observe playback readiness and state
‣ status
‣ Timed Observations
‣ addPeriodicTimeObserverForInterval:queue:usingBlock
‣ addBoundaryTimeObserverForInterval:queue:usingBlock
11. Playing Media
Static vs Dynamic Models
‣ AV Foundation distinguishes between static
and dynamic aspects of media
AVAsset
AVAsset AVAsset
AVAssetTrack
St ati c
AVPlayerItemTrack
AVPlayerItem AVPlayerItemTrack
AVPlayerItemTrack
D y n amic
12. Understanding Time
Core Media Essentials
CMTime
‣ Rational number representing time
‣ 64-bit integer time value (numerator)
‣ 32-bit integer time scale (denominator)
CMTime fiveSeconds = CMTimeMake(5, 1);
CMTime oneSample = CMTimeMake(1, 44100);
CMTime zeroTime = kCMTimeZero;
‣ Large number of utility functions in Core Media:
‣ CMTimeAdd,CMTimeSubtract,CMTimeCompare, etc.
13. Understanding Time
Core Media Essentials
CMTimeRange
‣ Core Media struct containing start time and duration
CMTimeRange assetRange = CMTimeRangeMake(kCMTimeZero, asset.duration);
CMTimeRange zeroRange = kCMTimeRangeZero;
‣ Like CMTime, there are many Core Media functions:
‣ CMTimeRangeEqual, CMTimeRangeContainsTime,
CMTimeRangeGetEnd, CMTIMERANGE_ISVALID, etc.
24. Audio Mixing
AVAudioMix
‣ Composition tracks play at their natural volume
‣ AVAudioMix applies track-level volume adjustments
‣ Composed of AVAudioMixInputParameters
‣ Parameters control individual track volume over time
CMTime CMTimeRange
28. Video Transitions
AVVideoComposition
AVVideoComposition
Defines how two or more video
tracks are composited together
AVVideoCompositionInstruction
Configured through collection of
AVVideoCompositionInstruction
AVVideoCompositionInstruction composition instructions describing
compositing behavior
29. Video Transitions
AVVideoCompositionInstruction
AVVideoComposition
AVVideoCompositionInstruction
AVVideoCompositionInstruction
AVVideoCompositionInstruction
Defines the time range of
compositing behavior
Composed of layer instructions
AVAsset
AVAsset
AVVideoCompositionLayerInstruction describing compositing behavior
30. Video Transitions
AVVideoCompositionLayerInstruction
AVVideoComposition
AVVideoCompositionInstruction
AVVideoCompositionInstruction
AVVideoCompositionInstruction
AVAsset
AVAsset Defines the transform and
AVVideoCompositionLayerInstruction
opacity ramps of input layers
Transform and opacity changes
modified over given time range
34. Building Transitions 3
Define Time Ranges
A
Passthrough Passthrough Passthrough
B
Transition Transition
* Time ranges must not have gaps or overlap
* Total duration must not be shorter than composition
36. Building Transitions 5
Set sizes and apply
#define FRAME_RATE CMTimeMake(1, 30)
#define RENDER_SIZE CGSizeMake(1280, 720)
AVMutableVideoComposition *videoComposition =
[AVMutableVideoComposition videoComposition];
// Set instructions on AVVideoComposition instance
videoComposition.instructions = instructions;
videoComposition.frameDuration = FRAME_RATE;
videoComposition.renderSize = RENDER_SIZE;
AVPlayerItem *playerItem =
[AVPlayerItem playerItemWithAsset:[composition copy]];
playerItem.videoComposition = videoComposition;
37. New in iOS 6
AVVideoComposition
videoCompositionWithPropertiesOfAsset:
Automagic Setup:
• Calculates all required passthrough and transition time ranges
• Builds appropriate composition and layer instructions for time ranges
• Sets the appropriate render size
• Sets the appropriate frame rate
40. Layering Content
Core Animation
Core Animation a natural choice
‣ High performance, inherently time-based
‣ CALayer subclasses used for all video rendering
CALayer: used to layer images and text
CAAnimation: used to animate layered content
CABasicAnimation
CAKeyframeAnimation
41. Animation Timing
AVSynchronizedLayer
‣ Core Animation operates on host time
‣ Starts at boot, marches towards infinity
‣ Timeline animations need to use movie time
‣ Starts at kCMTimeZero and runs to duration
‣ Can be started, stopped, rewound, etc.
‣ Use AVSynchronizedLayer to use movie time
‣ Confers player item timing on to its sublayer tree
AVPlayerItem AVSynchronizedLayer
CATextLayer CABasicAnimation
42. Core Animation
Timeline vs Realtime Animations
‣ Exactly the same, but different:
‣ Animations with zero beginTime won’t be seen
‣ Set beginTime = AVCoreAnimationBeginTimeZero
‣ Animations removed by default
‣ Set removedOnCompletion = NO
‣ Unable to use CAAnimationGroup?
43. Core Animation
Natural Choice, Awkward Implementation
‣ Different conceptual models for timeline editing
‣ CMTime and CMTimeRange for asset items
‣ Seconds and milliseconds for layers
‣ Build abstraction to help bridge the gap
‣ Usage differs in playback and export scenarios
‣ AVSynchronizedLayer for playback
‣ Attach to player’s view hierarchy/layer tree
‣ AVVideoCompositionCoreAnimationTool for export