1.
Core Image
The Most Fun API
You’re Not Using
Chris Adamson • @invalidname
CocoaConf Atlanta, December 2014

2.
“Core Image is an image processing and
analysis technology designed to provide near
real-time processing for still and video images.”

3.
Agenda
• Images, Filters, and Contexts
• The Core Image Filter Gallery
• Neat Tricks with Built-In Filters
• Core Image on OS X

4.
Core Image, Core Concepts
• Core Image is not about pixels… at least not most
of the time
• A chain of filters describes a “recipe” of processing
steps to be applied to one or more images
• “Stringly typed”
• You only get pixels when you render

5.
Typical Workflow
• Start with a source CIImage
• Apply one or more filters
• Render resulting CIImage to a CIContext, or
convert CIImage out to another type
• A few filters take or produce types other than
CIImage (CIQRCodeGenerator)

6.
CIImage
• An image provided to or produced by Core Image
• But no bitmap of pixel data!
• Immutable
• -imageByCroppingToRect,
-imageByApplyingTransform
• -extent — a CGRect of the image’s size

7.
CIImage sources
• NSURL
• CGImageRef
• Bitmap or JPEG/PNG/TIFF in NSData
• OpenGL texture
• Core Video image/pixel buffer

8.
CIContext
• Rendering destination for a CIImage (-
[drawImage:inRect:fromRect:])
• This is where you get pixels (also, this is the
processor-intenstive part)
• On iOS, must be created from an EAGLContext. On
Mac, can be created with CGContextRef
• Can also produce output as a CGImageRef, bitmap
data, or a CVPixelBuffer (iOS only)

9.
????

10.
CIFilter
• Performs an image processing operation
• Typically takes and produces a CIImage
• All parameters are provided via -[setValue:forKey:]
• Stringly-typed!
• Output is retrieved -[outputImage] (or -
[valueForKey:kCIOutputImageKey])

11.
“I can haz filters?”
–Core Image Cat

12.
Yes, you can has Filterz!

13.
Core Image Filter Reference
Filter Name
Parameters
Note the type & number to
provide
Categories
Watch for CICategoryBuiltIn
and CICategoryVideo
Example Figure
Availability
Watch for versioning and
OS X-only filters

14.
Filter Categories
• Group filters by functionality: CICategoryBlur,
CICategoryGenerator,
CICategoryCompositeOperation, etc.
• Also group filters by availability and
appropriateness: CICategoryBuiltIn,
CICategoryVideo, CICategoryNonSquarePixels

15.
CICategoryGenerator
• No input image, just produces an output
• CICategoryGradient is also output-only
• Example: CICheckerboardGenerator

16.
CICategoryBlur
• Algorithmically spreads/blends pixels
• CICategorySharpen offers an opposite effect
• Example: CIGaussianBlur

17.
CICategoryColorAdjustement
• Changes distribution of color throughout an image
• Example: CIColorControls (adjusts saturation,
brightness, contrast)

18.
CICategoryColorEffect
• Color changes that affect the subjective nature of
the image
• Example: CIPhotoEffectNoir

19.
CICategoryDistortionEffect
• Moves pixels to achieve an effect
• Example: CITorusLensDistortion

20.
CICategoryStylize
• Various stylistic effects
• Example: CIPointillize

21.
CICategoryGeometryAdjust
ment
• Moves pixels via cropping, affine transforms, etc.
• Example: CICrop

22.
CICategoryTileEffect
• Repeatedly copies all or part of an image
• Example: CIAffineTile

23.
CICategoryCompositeOpera
tion
• Combines multiple images
• Example: CISourceOverCompositing

24.
Demo

25.
Creating CIColorControls
Filter
_colorControlsFilter = [CIFilter
filterWithName:@"CIColorControls"];

26.
Setting input values
[self.colorControlsFilter
setValue:@(self.saturationSlider.value)
forKey:kCIInputSaturationKey];
[self.colorControlsFilter
setValue:@(self.brightnessSlider.value)
forKey:kCIInputBrightnessKey];
[self.colorControlsFilter
setValue:@(self.contrastSlider.value)
forKey:kCIInputContrastKey];

27.
Setting input image
CIImage *ciImage =
[CIImage imageWithCGImage:
self.imageView.image.CGImage];
[self.colorControlsFilter
setValue:ciImage
forKey:kCIInputImageKey];
Note: source image is 3264 × 2448 pixels

28.
Getting output image
ciImage = [self.colorControlsFilter
outputImage];
UIImage *filteredUIImage =
[UIImage imageWithCIImage:ciImage];
self.imageView.image = filteredUIImage;
Can also use CIFilter outputImage property instead of
valueForKey:

29.
API Modernizations
• iOS 8 and Mac OS X 10.10
• Can provide input parameters when creating a filter
with -[CIFilter filterWithName:withInputParameters:]
• Can apply a filter to an image in a one-off fashion
with -[CIImage
imageByApplyingFilter:withInputParameters:]

30.
Other output options
• Use a CIContext
• -[drawImage:inRect:fromRect:] draws pixels to
the EAGLContext (iOS) or CGContextRef (OS X)
that the CIContext was created from.
• CIContext can also render to a void* bitmap
• On iOS, can create a CVPixelBufferRef, typically
used for writing to a file with AVAssetWriter

31.
Chaining filters
• Use the output of one filter as the input to the next
• This doesn’t cost anything, because the CIImages
just hold state, not pixels

32.
Demo

33.
Creating CIContext
if (self.context.API != kEAGLRenderingAPIOpenGLES2) {
EAGLContext *eagl2Context = [[EAGLContext alloc]
initWithAPI:kEAGLRenderingAPIOpenGLES2];
self.context = eagl2Context;
}
// make CIContext from GL context,
// clearing out default color space
self.ciContext =
[CIContext contextWithEAGLContext:self.context
options: @{
kCIContextWorkingColorSpace : [NSNull null]
}];
Note: This is in a subclass of GLKView

34.
Set up Sepia Tone filter
_sepiaToneFilter = [CIFilter
filterWithName:@"CISepiaTone"];
[_sepiaToneFilter setValue:@(1.0)
forKey:@"inputIntensity"];

35.
Set up Hole Distortion Filter
_holeDistortionFilter = [CIFilter
filterWithName:@"CIHoleDistortion"];
[_holeDistortionFilter
setValue:[CIVector vectorWithX:100.0
Y:100.0]
forKey:kCIInputCenterKey];
[_holeDistortionFilter
setValue:@(50.0)
forKey:kCIInputRadiusKey];

36.
Set up Mask to Alpha filter
UIImage *circleImageUI = [UIImage
imageNamed:@"circle-mask-200x200"];
_circleMaskFilter = [CIFilter
filterWithName:@"CIMaskToAlpha"];
CIImage *circleImageCI = [CIImage
imageWithCGImage: circleImageUI.CGImage];
[_circleMaskFilter setValue:circleImageCI
forKey:kCIInputImageKey];
_circleMask = [_circleMaskFilter
valueForKey:kCIOutputImageKey];
circle-mask-200x200.png

37.
Set up Blend with Mask filter
_blendWithMaskFilter = [CIFilter
filterWithName:@"CIBlendWithMask"];
[_blendWithMaskFilter setValue:circleImageCI
forKey:kCIInputMaskImageKey];
[_blendWithMaskFilter setValue:_backgroundAlphaFill
forKey:kCIInputBackgroundImageKey];

38.
redrawAtOrigin (1/3)
// Get CIImage from source image
CGImageRef loupeImageCG =
CGImageCreateWithImageInRect(
self.sourceImage.CGImage, fromRect);
loupeImage = [CIImage imageWithCGImage:loupeImageCG];

39.
redrawAtOrigin (2/3)
// Apply sepia filter
[self.sepiaToneFilter setValue:loupeImage
forKey:kCIInputImageKey];
loupeImage = [self.sepiaToneFilter outputImage];
// Apply hole distortion filter
[self.holeDistortionFilter setValue:loupeImage
forKey:kCIInputImageKey];
loupeImage = [self.holeDistortionFilter outputImage];
// Set double-filtered image as input to blend-with-mask
[self.blendWithMaskFilter setValue:loupeImage
forKey:kCIInputImageKey];
loupeImage = [_blendWithMaskFilter outputImage];

40.
redrawAtOrigin (3/3)
if ([EAGLContext currentContext] != self.context) {
[EAGLContext setCurrentContext: self.context];
}
[self bindDrawable];
// GL-on-Retina fix
CGRect drawBoundsInPoints = self.glDrawBounds;
drawBoundsInPoints.size.width /= self.contentScaleFactor;
drawBoundsInPoints.size.height /= self.contentScaleFactor;
// drawing to CIContext draws to the
// EAGLESContext it's based on
[self.ciContext drawImage:loupeImage
inRect:self.glDrawBounds
fromRect:drawBoundsInPoints];
// Refresh GLKView contents immediately
[self display];

41.
Working with Video
• AVFoundation AVCaptureVideoDataOutput and
AVAssetReader deliver CMSampleBuffers
• CMSampleBuffers have timing information and
CVImageBuffers/CVPixelBuffers
• +[CIImage imageWithCVPixelBuffer:]

42.
Demo

43.
Chroma Key (“green screen”
recipe
• Use a CIColorCube to map green-ish colors to
transparent
• Use CISourceOverCompositing to draw this
alpha’ed image over another image

44.
CIColorCube
Maps colors from one RGB “cube” to another
http://en.wikipedia.org/wiki/RGB_color_space

45.
Using CIColorCube
CIColorCube maps green(-ish) colors to 0.0 alpha, all other
colors pass through

46.
CISourceOverCompositing
inputBackgroundImage inputImage
outputImage

47.
CIColorCube Data const unsigned int size = 64;
size_t cubeDataSize = size * size * size * sizeof (float) * 4;
float *keyCubeData = (float *)malloc (cubeDataSize);
// float *alphaMatteCubeData = (float *)malloc (cubeDataSize);
// float rgb[3], hsv[3], *keyC = keyCubeData, *alphaC = alphaMatteCubeData;
float rgb[3], hsv[3], *keyC = keyCubeData;
// Populate cube with a simple gradient going from 0 to 1
for (int z = 0; z < size; z++){
rgb[2] = ((double)z)/(size-1); // Blue value
for (int y = 0; y < size; y++){
rgb[1] = ((double)y)/(size-1); // Green value
for (int x = 0; x < size; x ++){
rgb[0] = ((double)x)/(size-1); // Red value
// Convert RGB to HSV
// You can find publicly available rgbToHSV functions on the Internet
RGBtoHSV(rgb[0], rgb[1], rgb[2],
&hsv[0], &hsv[1], &hsv[2]);
// RGBtoHSV uses 0 to 360 for hue, while UIColor (used above) uses 0 to 1.
hsv[0] /= 360.0;
// Use the hue value to determine which to make transparent
// The minimum and maximum hue angle depends on
// the color you want to remove
bool keyed = (hsv[0] > minHueAngle && hsv[0] < maxHueAngle) &&
(hsv[1] > minSaturation && hsv[1] < maxSaturation) &&
(hsv[2] > minBrightness && hsv[2] < maxBrightness);
float alpha = keyed ? 0.0f : 1.0f;
// re-calculate c pointer
keyC = (((z * size * size) + (y * size) + x) * sizeof(float)) + keyCubeData;
// Calculate premultiplied alpha values for the cube
keyC[0] = rgb[0] * alpha;
keyC[1] = rgb[1] * alpha;
keyC[2] = rgb[2] * alpha;
keyC[3] = alpha;
}
}
}
See “Chroma Key Filter Recipe” in Core Image Programming Guide

48.
Create CIColorCube from
mapping data
// build the color cube filter and set its data to above
self.colorCubeFilter = [CIFilter
filterWithName:@"CIColorCube"];
[self.colorCubeFilter setValue:[NSNumber numberWithInt:size]
forKey:@"inputCubeDimension"];
NSData *data = [NSData dataWithBytesNoCopy:keyCubeData
length:cubeDataSize
freeWhenDone:YES];
[self.colorCubeFilter setValue:data
forKey:@"inputCubeData"];

49.
Create
CISourceOverCompositing
// source over filter
self.backgroundImage = [UIImage imageNamed:
@"img_washington_small_02.jpg"];
self.backgroundCIImage = [CIImage imageWithCGImage:
self.backgroundImage.CGImage];
self.sourceOverFilter = [CIFilter filterWithName:
@"CISourceOverCompositing"];
[self.sourceOverFilter setValue:self.backgroundCIImage
forKeyPath:@"inputBackgroundImage"];

50.
Apply Filters in Capture
Callback
CIImage *bufferCIImage = [CIImage
imageWithCVPixelBuffer:cvBuffer];
[self.colorCubeFilter setValue:bufferCIImage
forKey:kCIInputImageKey];
CIImage *keyedCameraImage =
[self.colorCubeFilter outputImage];
[self.sourceOverFilter setValue:keyedCameraImage
forKeyPath:kCIInputImageKey];
CIImage *compositedImage =
[self.sourceOverFilter outputImage];
Then draw compositedImage to CIContext as before

51.
Other Points of Interest
• CIQRCodeGenerator filter — Converts data to a QR Code
• CILenticularHaloGenerator filter — aka, lens flare
• CIDetector — Class (not a filter) to find features in images.
iOS 7 / Lion only support face finding (returned as an array
of CIFeatures). Optionally detects smiles and eye blinks
within faces.
• iOS 8 / Yosemite add rectangle and QR code detection
• CIImage has a red-eye enhancement that takes the array
of face CIFeatures to tell it where to apply the effect

52.
Core Image on OS X
• Core Image is part of QuartzCore (or Image Kit), so
you don’t @import CoreImage
• Many more filters are available
• Filters can be set on CALayers

53.
CALayer Filters on OS X
• Views must be layer-backed (obviously)
• Must also call -[NSView
setLayerUsesCoreImageFilters:] on 10.9+
• CALayer has properties: filters, compositingFilter,
backgroundFilters, minificationFilter,
magnificationFilter
• These exist on iOS, but do nothing

54.
Demo

55.
Adding CIPixellate to layer’s
filters
self.pixellateFilter = [CIFilter filterWithName:
@"CIPixellate"];
self.pixellateFilter.name = @"myPixellateFilter";
[self.pixellateFilter setValue:
[CIVector vectorWithX:100.0 Y:100.0]
forKey:@“inputCenter"];
[self.pixellateFilter setValue:
@([self.pixellationScaleSlider floatValue])
forKey:@"inputScale"];
self.someTextField.layer.filters =
@[self.pixellateFilter];

56.
Updating a layer’s filters
-(void) updatePixellationScale {
[self.someTextField.layer setValue:
@([self.pixellationScaleSlider floatValue])
forKeyPath:
@"filters.myPixellateFilter.inputScale"];
}

57.
Building Your Own Filter

58.
CIKernel (new in iOS 8)
• Write per-pixel image processing code in Core
Image Kernel Language (subset of OpenGL + CI
extensions)
• -[CIKernel kernelWithString:]
• Subclass CIFilter, call apply:arguments:options: in
outputImage
• apply: takes your CIKernel as argument

59.
Wrap Up: Stuff to Remember
• Get psyched about filters, but remember to check
that they’re on your targeted platform/version.
• Drawing to a CIContext on iOS must be GL-backed
(e.g., with a GLKView)
• Not the only game in town: GPUImage offers an
open-source alternative

60.
Q&A
Slides and code will be posted to:
http://www.slideshare.net/invalidname/
@invalidname
http://subfurther.com/blog