This document provides an overview of GPU programming with GPUImage and Metal frameworks. It discusses what a GPU is and how it enables parallel computing. It then covers shader basics and graphics programming options in iOS like UIKit, SpriteKit, OpenGL ES, and Metal. GPUImage is presented as an image processing framework that utilizes the GPU efficiently. Metal is introduced as a new low-level graphics framework that promises improved performance over OpenGL ES by reducing expensive CPU tasks. Examples are provided to demonstrate GPUImage and Metal.

1. GPU PROGRAMMING WITH
GPUIMAGE AND METAL
JANIE CLAYTON

2. About Me
Janie Clayton
Co-Author of “iOS 8 SDK
Development”
Software engineer at
SonoPlot
@redqueencoder
http://redqueencoder.com

3. What is a GPU?
A Graphics Processing Unit (GPU) is a small super
computer that does one thing really really well. That
one thing is processing floating point math in parallel.
There are several applications for being able to do
really fast floating point math: Graphics processing,
bioinformatics, molecular dynamics, etc…
Most people are going to primarily focus on graphics
processing, as we will today. For GPGPU
programming, go see Jeff Biggus speak about
OpenCL!

4. What is Parallel Computing
The default processes in a project is serialized
computing. One instruction is processed at a time
and then the CPU moves on to the next one.
Parallel computing is the process of allowing
multiple instructions to be carried out at once.
Can be done on different threads, cores, and even
at the bit level.

5. But I Have Concurrency!
Concurrency is about dealing with a lot of things
at once.
Parallelism is about doing lots of things at once.

6. SERIES CIRCUIT
LESS EFFICIENT

7. PARALLEL CIRCUIT
MORE EFFICIENT

8. Shader Basics
Shaders are the programs that determine
what gets displayed on your screen.
Shaders determine shapes, colors,
textures, lighting…

9. GRAPHICS ON IOS DEVICES

10. There are many levels of abstraction
for graphics on iOS.
Some frameworks are more
abstracted than others.

11. UIKit
Sprite Kit
Core Animation/Graphics
OpenGL ES/GLKit

12. A BRIEF HISTORY OF TIME,
UH, OPENGL…

13. OpenGL Origins
First released in 1992
Was an attempt to formalize a 3D graphic
specification across platforms
John Carmack was instrumental for the adoption
of OpenGL as a cross-platform 3D graphic
specification.

14. Problems with OpenGL
Was created back when GPUs were not very
powerful and existed on external graphics cards
that could be swapped out
The computer system architecture was vastly
different when OpenGL was created. Things that
were not very efficient then, like the GPU, are vastly
more efficient now.
Nothing is ever deprecated (Don’t ask Java
programmers what that means, they don’t know)

16. Creation of OpenGL ES
ES: Embedded Systems
Wanted to strip out all of the dead code from
OpenGL
Was specifically tailored to work on less powerful
devices like mobile phones

17. We don’t need a dozen turtles that all do the same thing

18. OpenGL ES Specifics
Streamlined version of OpenGL
Everything you can do in OpenGL ES can directly
be ported to OpenGL
Basically an optimized version of OpenGL

19. CPU VS GPU PROGRAMMING

20. CPU Expensive Tasks
Sending hardware commands to the GPU
(Changing State Vectors)
Confirming that API usage is valid
Compiling the shaders
Interaction between the state and the shaders

21. How does the CPU Send
tasks to the GPU?
Try to envision a client-server process. Instead of
your program sending an instruction over the
network to a server and getting data back, you are
sending instructions from your CPU to your GPU
to be executed. Since you are sending instructions
away from your client to be done elsewhere, you
want to minimize this as much as possible.

22. How does the CPU Send
tasks to the GPU?
For example, in most Twitter client applications the
client batches 20 or more Tweets in one call. This
allows the application to feed tweets to the user
without them having to wait for the network to
deliver each and every tweet individually.

23. What Actually Sends
Commands to the GPU?
glGenBuffers(): Creates the buffer
glBindBuffers(): Tells OpenGL to use this buffer.
glBufferData(): Allocate this much continuous memory
glVertexAttribPointer(): What kind of data do we have?
glDrawArrays(): Render the data in the buffer
glDeleteBuffer(): We don’t need the buffer anymore,
get rid of it.

25. Fixed Function Pipeline
Present in OpenGL ES 1.1
Shaders were hard-coded into OpenGL
Easier to use, but were very limited

26. Programmable Pipeline
Introduced in OpenGL ES 2.0
Shaders are now the responsibility of the
programmer
Harder to do, but provides far more flexibility and
options for effects

27. OpenGL ES 1.1 vs 2.0
1.1 2.0
http://www.sunsetlakesoftware.com/molecules

28. What Frameworks are
Hardware Accelerated?
Core Animation
GLKit
SpriteKit
SceneKit

29. What About Core
Graphics/Quartz?
Core Graphics/Quartz is NOT performed on the
GPU. It is performed on the CPU.
Core Graphics is off on its own. UIKit is written on
top of Core Animation, which is written on top of
the GPU.
Core Graphics utilizes offscreen drawing. Anything
using offscreen drawing is not hardware
accelerated.

30. Offscreen Drawing
Core Graphics (anything starting with “CG”)
Every “drawRect()” method
Anything using Core Text
CALayers using masks and shadows
CALayers with “shouldRasterize” set to YES
Do not animate anything using offscreen drawing!
It is horribly inefficient!!

31. GLSL SHADER BUILDING

32. GLSL
OpenGL Shading Language (GLSL)
Introduced in OpenGL 2.0 in 2004
C-like language for building shaders, which are
small, efficient programs to run on the GPU
Includes some specific data types and methods
for processing geometry and graphics math that
are not included in C

33. GLSL
Two shader components: Vertex and Fragment
Both are necessary to create a completed shader
program
Vertex shaders deal with how geometry is handled
on the screen
Fragment shaders calculate what each individual
pixel will look like

34. Vertex Shaders

35. Fragment Shaders

36. GPU IMAGE

37. Creating GPUImage
GPUImage dates back to iOS 5.
Unlike Core Image (at the time), GPUImage utilized
shaders more efficiently to make image processing
faster. Core Image has been improved over the
years and they are now comparable.

38. Why is GPUImage so
Efficient?
OpenGL ES tasks must be performed on one
thread
Many people utilize locks to manage the thread or,
God forbid, only use the main thread. <shudder>
NSLock is expensive to the CPU
GPUImage utilizes a serial dispatch queue through
GCD to manage anything that touches the GPU to
keep everything happy and thread safe.

39. Demo

40. METAL: THE NEW KID IN
TOWN

41. What does Metal Promise?
Deep hardware integration between Apple chips
and Apple frameworks
General Purpose GPU programming (GPGPU)
Precompiled Shaders
up to 10 times more draw calls per frame
Being able to perform draw calls on multiple
threads

42. What Specifically are the
CPU Expensive Tasks?
Compiling Shaders
Validating State
Start Work on the GPU

43. Life Before Metal
All three of these expensive tasks were done on
each and every single draw call.
All of these tasks don’t have to be done thousands
of times a frame. Many can be done once, as long
as the program knows that it does not have to
continually check them.

44. Life After Metal
Compiling Shaders: Now done when the
applications builds
Validating State: Now done when the content
loads
Start Work on the GPU: Still happens on each
draw call. We can’t win them all…

45. Where Does Metal Help
You?
Metal helps you when you have a lot of objects
that need to work independently of one another.
Certain tasks, like image processing, do not
involve a lot of objects, so you aren’t going to gain
much with Metal.

46. Why is This Important?
Before Metal, you would have to balance CPU time
with GPU time. Tasks were so expensive that the
GPU would usually not be used to capacity.
Now that the CPU tasks are less expensive, you can
take that time to generate more AI and do more
programming logic.
Also, when people were generating art assets, they
had to make convoluted versions of assets to work
around OpenGL ES limitations. Now the same
assets can be used in all places. Not everything is
about code.

47. ZEN GARDEN DEMO
EPIC GAMES

48. SO, WHAT DO I THINK
ABOUT METAL?

49. Why Metal is Scary
You have to control EVERYTHING!!!
You have to have a deep understanding of how
the computer works that I have not seen
demonstrated by a large number of people.
Metal assumes you know more than the computer
does, which in my experience is usually a bad
move.

50. BAD WOLF
PROJECT
DATE CLIENT
3/27/15
WHAT HAPPENS WHEN YOU LOOK
INTO THE HEART OF THE GPU

51. Why Metal is Exciting
Metal, along with Swift, signals a shift to figuring
out how to do more parallel programming.
I believe Metal is not going anywhere. It will take a
while for people to learn how to fully utilize it, but I
believe it has the potential to be a game changer.
Metal, like Swift, is still partly baked. It gives early
adopters an opportunity to master something
extraordinary.

52. IS THERE ANY POINT IN LEARNING
OPENGL ES ANYMORE?

53. “Easy things should be easy.
Hard things should be possible.”
–Larry Wall

54. Yes, absolutely. Metal’s API is very similar to OpenGL ES.
It will take a while for everyone to transition over
to devices with A7 chips.
Apple will continue to support its developers who
work with OpenGL ES, especially since the
Mac uses OpenGL and won’t be able to use Metal (yet).
Also, Metal is new. It usually takes Apple a few
years to work the kinks out of their new frameworks.
Also, with Metal’s incredibly steep learning curve,
very few people could work with it now.

55. Take Aways
Whether you learn GLSL or Metal Shading Language,
the value comes from the algorithms. The languages
are not complicated and are similar. If you don’t know
how the math on a shader works, knowing the language
won’t really help you.
There are lots of books on GPU programming out there
explaining how to create effects, not to mention the
shaders included in GPUImage. You will need to
understand the math, but there are great resources
online out there for this stuff.
Be tenacious. This takes a lot of time to master. It is
worth it. Be patient.

57. Think Different.

58. The End

59. • http://www.objc.io/issue-21/gpu-accelerated-image-
processing.html
• https://developer.apple.com/videos/wwdc/2011/#414
• https://developer.apple.com/videos/wwdc/2014/#603
• http://www.sunsetlakesoftware.com/2011/05/08/
enhancing-molecules-using-opengl-es-20
Links