This document introduces the GPUImage framework, an open source iOS library for GPU-based image and video processing. It provides advantages over Core Image such as improved performance for real-time previews through OpenGL ES shader code and greater customization abilities. The framework uses frame buffer objects and textures to process video frames on the GPU, applying filters through shader programs before saving results back to the CPU. It employs Grand Central Dispatch for multi-threaded processing across CPU and GPU.

1. Aug. 5, 2014
GPUImage Introduction
Bruce Tsai

2. Core Image
✤ Included in
iOS from 2011!
✤ 100+ built-in
filters!
✤ Multithread
by CPU

3. Drawbacks
✤ Customization limited!
- Subclass CIFilter in iOS!
- Create custom filters (shader) only in OSX!
✤ Performance not tuned!
- Address tuning tips in WWDC 2012 (session 511)
3

4. GPUImage
✤ Open source (BSD) iOS framework!
✤ GPU-based image and video processing!
✤ From GPU-accelerated video processing project (2010)!
✤ https://github.com/BradLarson/GPUImage
4

5. Advantages
✤ Performance!
- High FPS in real-time preview!
✤ Customization!
- OpenGL ES shader code
5

6. Buffer Process Flow
VideoCamera Sepia
View
MovieWriter
BlendPicture
Source Filter Output

7. Code Snippet
GPUImageVideoCamera *videoCamera = [[GPUImageVideoCamera
alloc] initWithSessionPreset:AVCaptureSessionPreset640x480
cameraPosition:AVCaptureDevicePositionBack];
GPUImageFilter *customFilter = [[GPUImageFilter alloc]
initWithFragmentShaderFromFile:@“CustomShader”];
GPUImageView *filteredVideoView = [[GPUImageView alloc]
initWithFrame:CGRectMake(0.0, 0.0, viewWidth, viewHeight)];
[videoCamera addTarget:customFilter];
[customFilter addTarget:filteredVideoView];
[videoCamera startCameraCapture];

8. Class Diagram

9. Procedures
1. Add effect to input buffer !
2. Save result to output buffer!
3. Set targets’ input buffer attributes!
4. Set targets’ input buffer!
5. Invoke targets’ operation
9

11. Frame Buffer Object (FBO)
glBindFramebuffer(GL_FRAMEBUFFER,
framebuffer);
glBindTexture(GL_TEXTURE_2D, renderTexture);
glFramebufferTexture2D(GL_FRAMEBUFFER,
GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D,
renderTexture, 0);

12. Buffer Binding in FBO
Frame
Buffer
TexturePixel Buffer
Render destination
Actual render
target
Main memory
buffer

13. Conversion to and fromTexture
CVPixelBufferRef pxBufferRef =
CMSampleBufferGetImageBuffer(imageBuffer);
CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDe
fault, videoTextureCache, pxBufferRef, NULL, GL_TEXTURE_2D,
GL_RGBA, width, height, GL_BGRA, GL_UNSIGNED_BYTE, 0,
&videoTextureRef);
...
// renderTargetRef is corresponding to renderTexture of FBO
[writerAdaptor appendPixelBuffer:renderTargetRef
withPresentationTime:pts];

14. Threading Model
✤ Grand Central Dispatch (GCD)!
- Dispatch queue!
✤ runSynchronousOnVideoProcessingQueue!
✤ runAsynchronousOnVideoProcessingQueue!
✤ runSynchronousOnContextQueue!
✤ runAsynchronousOnContextQueue
14

16. Demo

17. Reference
✤ https://github.com/BradLarson/GPUImage!
✤ http://blog.db-in.com/all-about-opengl-es-2-x-part-2/!
✤ https://developer.apple.com/library/ios/documentation/
3DDrawing/Conceptual/OpenGLES_ProgrammingGuide/
Introduction/Introduction.html#//apple_ref/doc/uid/
TP40008793!
✤ https://developer.apple.com/library/ios/documentation/
GraphicsImaging/Conceptual/CoreImaging/ci_intro/
ci_intro.html#//apple_ref/doc/uid/TP30001185