Core Image is an image processing framework that provides fast and efficient filters. It encapsulates low-level graphics processing to apply effects like monochrome, vignette, and face detection. Filters can be chained and applied to images from photos, video, or GL textures. Core Image renders processing to the GPU or CPU and has a simple Objective-C API. It is portable across iOS and OS X with some differences in creation methods and built-in filters.

1. Core Image
Mark Pavlidis
Co-founder
Flixel Photos Inc.
Toronto Cocoa and WebObjects Developers Group
2013/01/08

2. Motivation
Why use Core Image?
• Image processing and analysis
• Fast, efficient filters
• Auto enhancement
• Feature detection
• Still and “real-time” video
• Simple Objective-C API

3. Definitions
• Image Filter
• a single transform or effect
• built-in or loaded from an Image Unit plugin (OS X only)
• Framework
<CoreImage/CoreImage.h>
• Key Classes
CIImage, CIFilter, CIContext

4. Overview
Apply an image filter to the source pixel data
Monochrome
Filter
• Pixel Accurate
• Non-destructive
• Filters can be chained together

5. Core Image Runtime
How it works

6. Core Image Runtime
How it works
• Operates on image data types from:
• Core Graphics
• Core Video
• Image I/O

7. Core Image Runtime
How it works
• Built-in or plugin filters apply effects
• Written in the Core Image Kernel Language
• JIT complier assembles instruction pipeline

8. Core Image Runtime
How it works
• Executed on the GPU or CPU
• Low-level graphics processing encapsulated by CI
• Performance: GLSL capabilities of the GPU or
processing power of the CPU

9. Core Image Inputs
• Photo Library and Files
imageWithContentsOfURL:
• Live Video Capture
imageWithCVPixelBuffer:
imageWithCVImageBuffer:
• In Memory
imageWithCGImage:
• GL Texture
imageWithTexture:size:flipped:colorSpace:

10. Core Image Outputs
• CGImageRef
• Output to UIImage, ImageIO, ALAssetLibrary
createCGImage:fromRect:
• CAEAGLLayer
drawImage:inRect:fromRect:
• CVPixelBufferRef
render:toCVPixelBuffer:
• Bitmap
render:toBitmap:rowBytes:bounds:format:colorSpace:

11. Simple Filter
// Create a CIImage
CIImage *ciImage = [CIImage imageWithContentsOfURL:myURL];
// Create a CIFilter
CIFilter *filter = [CIFilter
filterWithName:@”CIColorMonochrome”];
[filter setValue:ciImage forKey:kCIInputImageKey];
[filter setValue:@(0.5) forKey:@”inputIntensity];
// Render the filter output image into a UIImage
UIImage *uiImage = [UIImage
imageWithCIImage:filter.outputImage];

12. Filter Chains
// Filter 1
CIImage *output = [CIFilter
filterWithName:@”CIColorMonochrome”]
keysAndValues:
kCIInputImageKey, ciImage,
@”inputIntensity, @(0.5),
nil].outputImage;
// Filter 2
output = [CIFilter
filterWithName:@”CIVignette”] keysAndValues:
kCIInputImageKey, output,
@”inputIntensity, @(0.5),
@”inputRadius, @(1.5),
nil].outputImage;
•CI defers pixel processing until render is requested
•CI optimizes the render graph (e.g., sepia then scale)

13. Math!
• An image filter is kernel
• Output is the convolution of the source & kernel
Source Output
Kernel
2 2 2 1 1 1
2 1 3 -1 2 1 6
2 2 1 -1 -1 1
• Commutative, associative, distributive, associative
with scalar multiplication, multiplicative identity

14. Demo - QuartzComposer

15. Demo - Simple filter

16. Tips and Best Practices
• CIImage and CIFilter are autoreleased
• Use autorelease pools
• CIImage, CIContext are immutable and thread-safe,
CIFilter is not
• CIContext is expensive
• Stores a lot of state information so reuse them
• Avoid CA animations when rendering on GPU
• Use smaller images when possible
• Use Core Animation to upscale view, texture, framebuffer
• Disable colour management
• If real-time performance and/or reduced quality is
unnoticeable

17. Face Detection
CIDetector
• Identifies rectangles that contain human faces
• Feature detection
• eyes & mouth position, tracking ID and frame count (video)
CIContext *context = [CIContext
contextWithOptions:nil];
NSDictionary *opts = @{CIDetectorAccuracy,
! ! ! ! ! ! ! CIDetectorAccuracyHigh};
CIDetector *detector = [CIDetector
! ! ! ! ! ! detectorOfType:CIDetectorTypeFace
! ! ! ! ! ! ! ! context:context
! ! ! ! ! ! ! ! options:opts];
NSArray *features = [detector featuresInImage:myImage
! ! ! ! ! ! options:opts];

18. Demo - Redacted Faces

19. OS X and iOS Differences
Portability
• Some CIImage create methods differ
• Has more built-in filters
• Permits custom filter plugins
• filter.outputImage is iOS only, use
[filter valueForKey:kCIOutputImage]
• Remember to [filter setDefaults]
• All input parameter keys have strings defined

20. Alternatives
• GPUImage (iOS)
✓Hand-tuned OpenGL ES shaders that are much faster than CI
✓Custom filters
- Lacks advanced features of Core Image
• Aviary
✓ Everyone is using it (Twitter, Flickr)
- Everyone is using it
• Only still images
• Hardcore - Write your own GL shaders
• Don’t be silly

21. Summary
Core Image
• Fast and efficient image processing framework
• Simple Objective-C API
• Encapsulates the low-level instruction details
• Many built-in filters
• Advanced processing features:
• Face Detection
• Auto Enhancement
• Real-time video filters

22. More Information
• Core Image Programming Guide
• WWDC 2012 Videos
• Session 510 - Getting Started with Core Image
• Session 511 - Core Image Techniques
• Sample Code
• https://github.com/mhpavl/FilterDemo
• https://github.com/mhpavl/RedactedFaces
• Core Image Funhouse (TBD)

23. [self release];
Mark Pavlidis
mark@pavlidis.ca
Twitter: @mhp
ADN: @mhp