Core ImageMark PavlidisCo-founderFlixel Photos Inc.Toronto Cocoa and WebObjects Developers Group2013/01/08
MotivationWhy use Core Image? •   Image processing and analysis •   Fast, efficient filters •   Auto enhancement •   Feature ...
Definitions• Image Filter • a single transform or effect • built-in or loaded from an Image Unit plugin (OS X only)• Framewo...
OverviewApply an image filter to the source pixel data                       Monochrome                          Filter • P...
Core Image RuntimeHow it works
Core Image RuntimeHow it works • Operates on image data types from:   • Core Graphics   • Core Video   • Image I/O
Core Image RuntimeHow it works • Built-in or plugin filters apply effects • Written in the Core Image Kernel Language • JIT ...
Core Image RuntimeHow it works • Executed on the GPU or CPU • Low-level graphics processing encapsulated by CI • Performan...
Core Image Inputs• Photo Library and Files   imageWithContentsOfURL:• Live Video Capture   imageWithCVPixelBuffer:   image...
Core Image Outputs• CGImageRef • Output to UIImage, ImageIO, ALAssetLibrary   createCGImage:fromRect:• CAEAGLLayer   drawI...
Simple Filter// Create a CIImageCIImage *ciImage = [CIImage imageWithContentsOfURL:myURL];// Create a CIFilterCIFilter *fi...
Filter Chains// Filter 1CIImage *output = [CIFilter                   filterWithName:@”CIColorMonochrome”]                ...
Math!• An image filter is kernel• Output is the convolution of the source & kernel      Source                             ...
Demo - QuartzComposer
Demo - Simple filter
Tips and Best Practices • CIImage and CIFilter are autoreleased    • Use autorelease pools • CIImage, CIContext are immuta...
Face DetectionCIDetector• Identifies rectangles that contain human faces• Feature detection    • eyes & mouth position, tra...
Demo - Redacted Faces
OS X and iOS DifferencesPortability • Some CIImage create methods differ • Has more built-in filters • Permits custom filter p...
Alternatives• GPUImage (iOS)   ✓Hand-tuned OpenGL ES shaders that are much faster than CI   ✓Custom filters   - Lacks advan...
SummaryCore Image • Fast and efficient image processing framework • Simple Objective-C API • Encapsulates the low-level inst...
More Information• Core Image Programming Guide• WWDC 2012 Videos    •   Session 510 - Getting Started with Core Image    •...
[self release]; Mark Pavlidis mark@pavlidis.ca Twitter: @mhp ADN: @mhp
Core Image
Upcoming SlideShare
Loading in …5
×

Core Image

3,018 views

Published on

Core Image talk at Toronto Area Cocoa and WebObjects Developers Group

Published in: Technology
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,018
On SlideShare
0
From Embeds
0
Number of Embeds
13
Actions
Shares
0
Downloads
36
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide

Core Image

  1. 1. Core ImageMark PavlidisCo-founderFlixel Photos Inc.Toronto Cocoa and WebObjects Developers Group2013/01/08
  2. 2. MotivationWhy use Core Image? • Image processing and analysis • Fast, efficient filters • Auto enhancement • Feature detection • Still and “real-time” video • Simple Objective-C API
  3. 3. Definitions• Image Filter • a single transform or effect • built-in or loaded from an Image Unit plugin (OS X only)• Framework <CoreImage/CoreImage.h>• Key Classes CIImage, CIFilter, CIContext
  4. 4. OverviewApply an image filter to the source pixel data Monochrome Filter • Pixel Accurate • Non-destructive • Filters can be chained together
  5. 5. Core Image RuntimeHow it works
  6. 6. Core Image RuntimeHow it works • Operates on image data types from: • Core Graphics • Core Video • Image I/O
  7. 7. Core Image RuntimeHow it works • Built-in or plugin filters apply effects • Written in the Core Image Kernel Language • JIT complier assembles instruction pipeline
  8. 8. Core Image RuntimeHow it works • Executed on the GPU or CPU • Low-level graphics processing encapsulated by CI • Performance: GLSL capabilities of the GPU or processing power of the CPU
  9. 9. Core Image Inputs• Photo Library and Files imageWithContentsOfURL:• Live Video Capture imageWithCVPixelBuffer: imageWithCVImageBuffer:• In Memory imageWithCGImage:• GL Texture imageWithTexture:size:flipped:colorSpace:
  10. 10. Core Image Outputs• CGImageRef • Output to UIImage, ImageIO, ALAssetLibrary createCGImage:fromRect:• CAEAGLLayer drawImage:inRect:fromRect:• CVPixelBufferRef render:toCVPixelBuffer:• Bitmap render:toBitmap:rowBytes:bounds:format:colorSpace:
  11. 11. Simple Filter// Create a CIImageCIImage *ciImage = [CIImage imageWithContentsOfURL:myURL];// Create a CIFilterCIFilter *filter = [CIFilter filterWithName:@”CIColorMonochrome”];[filter setValue:ciImage forKey:kCIInputImageKey];[filter setValue:@(0.5) forKey:@”inputIntensity];// Render the filter output image into a UIImageUIImage *uiImage = [UIImage imageWithCIImage:filter.outputImage];
  12. 12. Filter Chains// Filter 1CIImage *output = [CIFilter filterWithName:@”CIColorMonochrome”] keysAndValues: kCIInputImageKey, ciImage, @”inputIntensity, @(0.5), nil].outputImage;// Filter 2output = [CIFilter filterWithName:@”CIVignette”] keysAndValues: kCIInputImageKey, output, @”inputIntensity, @(0.5), @”inputRadius, @(1.5), nil].outputImage;•CI defers pixel processing until render is requested•CI optimizes the render graph (e.g., sepia then scale)
  13. 13. Math!• An image filter is kernel• Output is the convolution of the source & kernel Source Output Kernel 2 2 2 1 1 1 2 1 3 -1 2 1 6 2 2 1 -1 -1 1• Commutative, associative, distributive, associative with scalar multiplication, multiplicative identity
  14. 14. Demo - QuartzComposer
  15. 15. Demo - Simple filter
  16. 16. Tips and Best Practices • CIImage and CIFilter are autoreleased • Use autorelease pools • CIImage, CIContext are immutable and thread-safe, CIFilter is not • CIContext is expensive • Stores a lot of state information so reuse them • Avoid CA animations when rendering on GPU • Use smaller images when possible • Use Core Animation to upscale view, texture, framebuffer • Disable colour management • If real-time performance and/or reduced quality is unnoticeable
  17. 17. Face DetectionCIDetector• Identifies rectangles that contain human faces• Feature detection • eyes & mouth position, tracking ID and frame count (video) CIContext *context = [CIContext contextWithOptions:nil]; NSDictionary *opts = @{CIDetectorAccuracy, ! ! ! ! ! ! ! CIDetectorAccuracyHigh}; CIDetector *detector = [CIDetector ! ! ! ! ! ! detectorOfType:CIDetectorTypeFace ! ! ! ! ! ! ! ! context:context ! ! ! ! ! ! ! ! options:opts]; NSArray *features = [detector featuresInImage:myImage ! ! ! ! ! ! options:opts];
  18. 18. Demo - Redacted Faces
  19. 19. OS X and iOS DifferencesPortability • Some CIImage create methods differ • Has more built-in filters • Permits custom filter plugins • filter.outputImage is iOS only, use [filter valueForKey:kCIOutputImage] • Remember to [filter setDefaults] • All input parameter keys have strings defined
  20. 20. Alternatives• GPUImage (iOS) ✓Hand-tuned OpenGL ES shaders that are much faster than CI ✓Custom filters - Lacks advanced features of Core Image• Aviary ✓ Everyone is using it (Twitter, Flickr) - Everyone is using it • Only still images• Hardcore - Write your own GL shaders • Don’t be silly
  21. 21. SummaryCore Image • Fast and efficient image processing framework • Simple Objective-C API • Encapsulates the low-level instruction details • Many built-in filters • Advanced processing features: • Face Detection • Auto Enhancement • Real-time video filters
  22. 22. More Information• Core Image Programming Guide• WWDC 2012 Videos • Session 510 - Getting Started with Core Image • Session 511 - Core Image Techniques• Sample Code • https://github.com/mhpavl/FilterDemo • https://github.com/mhpavl/RedactedFaces • Core Image Funhouse (TBD)
  23. 23. [self release]; Mark Pavlidis mark@pavlidis.ca Twitter: @mhp ADN: @mhp

×