GPU PROGRAMMING WITH
GPUIMAGE AND METAL
What is a GPU?
A Graphics Processing Unit (GPU) is a small super
computer that does one thing really really well. That
one thing is processing ﬂoating point math in
There are several applications for being able to do
really fast ﬂoating point math: Graphics processing,
bioinformatics, molecular dynamics, etc…
Most people are going to primarily focus on
graphics processing, as we will today
What is Parallel Computing
The default processes in a project is serialized
computing. One instruction is processed at a time
and then the CPU moves on to the next one.
Parallel computing is the process of allowing
multiple instructions to be carried out at once.
Can be done on diﬀerent threads, cores, and even
at the bit level.
But I Have Concurrency!
Concurrency is about dealing with a lot of things
Parallelism is about doing lots of things at once.
Shaders are the programs that determine what
gets displayed on your screen.
Shaders determine shapes, colors, textures,
Everything you see and use comes down to
First released in 1992
Was an attempt to formalize a 3D graphic
speciﬁcation across platforms
Problems with OpenGL
Was created back when GPUs were not very
powerful and existed on external graphics cards
that could be swapped out
The computer system architecture was vastly
diﬀerent when OpenGL was created. Things that
were not very eﬃcient then, like the GPU, are vastly
more eﬃcient now.
Nothing is ever deprecated (Don’t ask Java
programmers what that means, they don’t know)
Creation of OpenGL ES
ES: Embedded Systems
Wanted to strip out all of the dead code from
Was speciﬁcally tailored to work on less powerful
devices like mobile phones
We don’t need a dozen turtles that all do the same thing
OpenGL ES Speciﬁcs
Streamlined version of OpenGL
Everything you can do in OpenGL ES can directly
be ported to OpenGL
Basically an optimized version of OpenGL
CPU Expensive Tasks
Sending hardware commands to the GPU
(Changing State Vectors)
Conﬁrming that API usage is valid
Compiling the shaders
Interaction between the state and the shaders
What the Heck is “State”??
Try to envision a client-server process. Instead of
your program sending an instruction over the
network to a server and getting data back, you are
sending instructions from your CPU to your GPU
to be executed. Since you are sending instructions
away from your client to be done elsewhere, you
want to minimize this as much as possible.
What the Heck is “State”??
For example, in most Twitter client applications the
client batches 20 or more Tweets in one call. This
allows the application to feed tweets to the user
without them having to wait for the network to
deliver each and every tweet individually.
Fixed Function Pipeline
Present in OpenGL ES 1.1
Shaders were hard-coded into OpenGL
Easier to use, but were very limited
Introduced in OpenGL ES 2.0
Shaders are now the responsibility of the
Harder to do, but provides far more ﬂexibility and
options for eﬀects
OpenGL Shading Language (GLSL)
Introduced in OpenGL 2.0 in 2004
C-like language for building shaders, which are
small, eﬃcient programs to run on the GPU
Includes some speciﬁc data types and methods
for processing geometry and graphics math that
are not included in C
The Vertex Shader would record the vertices of the
star (which would be broken down into a series of
The Vertex Shader would also specify that the area
between the vertices is yellow. If there was a
texture instead of a color, the shader would keep
track of the texture coordinates.
Fragment Shaders determine what pixels receive
If you look carefully at the star, there are areas
outside the star that are yellow and areas inside
that are white.
If there is a gradient, the Fragment Shader will
calculate what speciﬁc color each individual pixel
Inputs and Outputs
Values that don’t change during rendering
Available in both Vertex and Fragment Shaders
Vertex Shader only
Input values that change with every vertex, like
their position, color, and texture coordinates
Used to pass data between the Vertex and the
Read-only in the Fragment Shader
Read-Write in the Vertex Shader
Varyings are the variables that determine the pixel
color for the Fragment Shader
GPUImage dates back to iOS 5.
Unlike Core Image (at the time), GPUImage utilized
shaders more eﬃciently to make image processing
faster. Core Image has been improved over the
years and they are now comparable.
Why is GPUImage so
OpenGL ES tasks must be performed on one
Many people utilize locks to manage the thread or,
God forbid, only use the main thread. <shudder>
NSLock is expensive to the CPU
GPUImage utilizes a serial dispatch queue through
GCD to manage anything that touches the GPU to
keep everything happy and thread safe.
What does Metal Promise?
Deep hardware integration between Apple chips
and Apple frameworks
General Purpose GPU programming (GPGPU)
up to 10 times more draw calls per frame
Being able to perform draw calls on multiple
What Speciﬁcally are the
CPU Expensive Tasks?
Start Work on the GPU
Life Before Metal
All three of these expensive tasks were done on
each and every single draw call.
All of these tasks don’t have to be done thousands
of times a frame. Many can be done once, as long
as the program knows that it does not have to
continually check them.
Life After Metal
Compiling Shaders: Now done when the
Validating State: Now done when the content
Start Work on the GPU: Still happens on each
draw call. We can’t win them all…
Why is This Important?
Before Metal, you would have to balance CPU
time with GPU time. Tasks were so expensive that
the GPU would usually not be used to capacity.
Now that the CPU tasks are less expensive, you
can take that time to generate more AI and do
more programming logic.
Where Does Metal Help
Metal helps you when you have a lot of objects
that need to work independently of one another.
Certain tasks, like image processing, do not
involve a lot of objects, so you aren’t going to gain
much with Metal.
Okay, so one of the big, obscure black boxes in
this scheme is the promise of deep software/
hardware integration. One thing I have not had
the chance to study in depth is kernel
programming and chip architecture. Knowing
the idiosyncrasies of the chips and only having
to support one type allows for more targeted
IS THERE ANY POINT IN LEARNING
OPENGL ES ANYMORE?
Yes, absolutely. Metal’s API is very similar to OpenGL ES.
It will take a while for everyone to transition over
to devices with A7 chips.
Apple will continue to support its developers who
work with OpenGL ES, especially since the
Mac uses OpenGL and won’t be able to use Metal (yet).
It isn’t like they would
just throw out an older technology that works
perfectly well and replace it with something
that barely works, right??