Shader X³: Image Space - Color Grading

IMAGE SPACE: COLOR GRADING AUGUST 22, 2014
©2004 RONNY BURKERSRODA PAGE - 1 -
THEORY
Since the introduction of hardware shaders we were able to recognize a new level of real time graphic effects. Many of them can be classified as post-processing effects because they are done after rendering the scene. Examples are soft focus, blooming and screen blurring. Color grading is also such an effect and has its origin in image and film processing. A well known movie trilogy is “Lord of the Rings”, in which it was heavily used for nearly every frame. The creators wanted to match the atmosphere of a scene with their imaginations by changing the colors of the shots.
Different sets of colors transport one image to different atmospheres like warm or cold ones, realistic or fantastic and so on. Grading is used to do that and with an implementation in games it can bring some feeling of movies to those.
There are also other ways to change the colors of real time rendered images but these are not so flexible. For example gamma correction is not only able to adjust brightness but it is limited to color channels, so blue components influence only the blue channel, red only red and green only green. With blending you are able to add, subtract or modulate images with constant colors like gradient filters. In this case the source color cannot be used to control the target color completely like you may want. Both variants do not allow us to transport a yellow to light blue and let green and red like they are for example. If you want complete control over the modification you can use color grading, which is explained in this article and implemented for Direct3D.
If you ask, why to use color grading as post-processing effect and not to modify the colors of the object textures, then here are the answers: First you cannot control the results particularly good because lighting, blending and other effects are influencing the colors. Secondly dynamic changes of large textures cause a lot of memory bandwidth and thus a bad performance.
To change a color of a pixel we can use functions but those can cost a lot of time and the count of different operations and the maximum length of pixel shader limit our possibilities. Especially shaders of the first major version can only do few things. But we can pre-calculate all color projections and store them into a table, so various functions can be combined. Because RGB colors have got three channels we need a three dimensional table. The values can be set as shader constants but for the standard color range one channel can have up to 256 different values. That means there are 2563 (= 16,777,216) possible combinations, which cannot all stored into constant registers. Because we have to use a pixel shader we are able to access textures and because we are working with colors, textures are a perfect format to store the colors to. Fortunately simultaneous to the introduction of shaders three dimensional textures were introduced, too.
If you want to set the graded color for all RGB combinations, then you have to create a volume texture with a width, height and depth of 256. Because we need a RGB destination color, every element must be at least 24 bits in size. Modern graphic cards do not support this any more so it has to be set to 32 bits. This results in 256 * 256 * 256 * 4 byte (= 64 MB) texture. It is not only a lot of video memory but also a lot of AGP bandwidth, if the texture is updated dynamically. Because we are using textures we also can use all their features like filtering and compression. Instead of using the XRGB format using a DXT1 texture results in only 8 MB for same dimension. Or we decrease the dimension to a width, height and

depth of 64 and activate linear filtering for the volume map to interpolate between the 64 steps. Then the texture takes only 1 MB.
Both variants can decrease the quality of an image. Texture compression produces artifacts and those are distorting the destination color. And with decreasing the dimension the artist looses control of every RGB combination. But because many color manipulation functions are calculating graphs, which have got a relatively regular course, filtering produces very similar results. So this is a good ratio between performance respectively memory usage and quality.
These are screenshots from a bowling-like game, which is developed by Media Seasons, Leipzig, Germany. The upper left is the original scene, where the textures have got a very warm tone. Right it is graded to a cold atmosphere only using the algorithm, which is described in the article. Below the original there are a sepia- graded image and an experimental one. As you can see the GUI does not have to be affected by the grading, too.
To grade an image in real time a pixel shader is needed. Fortunately the lowest version (1.1) can be used to do that. But this version is not able to use a temporary register as texture coordinate to sample the volume map. Therefore we render the scenery to a texture and take the texel colors as coordinates. The sampled color of the 3D texture is the graded one for the input texel of the render target scene texture.
From pixel shader version 1.4 on it is possible to include color grading directly to the shader of an object. In version 1.4 we use the first phase to calculate the original pixel color and in the second phase the temporary register, which contains

the color, is taken as texture coordinate for the volume map. Version 2 and higher is able to use such registers directly to sample textures. But this results in grading the colors per face and not per final pixel or texel. That means when two graded colors are blended together there can be a more unintentional result as when the two original colors are blended and the result is graded afterwards. In this case we are loosing some control and the post-processing character of the effect. Moreover all existing pixel shaders have to be enhanced, which can be much work. Also long and many two-phase shaders can be at their maximum length and cannot be enhanced by the color grading feature. Finally we will save fill rate, if projected scenes include overlapping regions, which is always the case in games.
IMPLEMENTATION
First of all we need the volume texture, which contains the color projection table. For a frequently changed table the texture should be created in default pool with staid dynamic flag. The texture can be calculated in the application or loaded from a preset. and are useful for the first option, but you have to take care for texture coordinates, which are given as parameter. Because those are centered at the texels, we are not getting values between 0 and 1 but texel size / 2 and 1 – texel size / 2. The texel size is calculate by 1 / texture size. functions are able to load previously stored volumes from DDS files. To store a texture to such a DirectDraw Surface file functions are provided. So an artist can specify presets, which are loaded later into the application.
To post-process the drawn scene we need it in a texture. Here we have two possibilities: For both we have to create a render target texture. In the first version we use the texture instead of the back buffer to render the scene to. But textures do not allow multi-sampling, so all objects are drawn without anti- aliasing. This should be no option for modern games because anti-aliasing is a standard, which everybody should be able to choose. Since DirectX 9b there is an alternative: In the second version we do not render the scene directly to the texture. First the scene is drawn normally to the back buffer with multi-sampling optionally enabled and then we copy the back buffer to the texture surface. DirectX 9b and newer versions are able to convert between multi-sampled and non- multi-sampled surfaces using . It is a very fast instruction and we do not need to change render target or depth stencil buffer.
The render target texture is set to stage 0. A vertex buffer with a rectangle that covers the whole screen is used to render the texture to screen. Now it is only needed that we grade to original texture colors to the projection ones. For a pixel shader 1.1 the volume texture has to be set at least to stage 3 because a sampled texel color cannot be directly used as texture coordinate. The shader instruction performs a multiplication of the sampled color as vector with a 3x3 matrix. The first two rows are multiplied by and

multiplies the third row and samples the graded color. Every row is a dot-3-
multiplication that results in one component of the volume texture coordinate:
T C
T C
T C
3
2
1
 
 
 
w'
v'
u'
Tx is the texture coordinate at stage x, C the source color with the three channels
as components and u’, v’ and w’ are building the destination texture coordinate T’
to sample the graded color. Because we want Cr as u’, Cg as v’ and Cb as w’ we
have to set T1 to (1,0,0), T2 to (0,1,0) and T3 to (0,0,1). So we are getting: T’=C.
Here is the pixel shader to do that:
Listing 1. Color grading with pixel shader 1.1
After the declaration of the shader version the original color is loaded from the
screen texture into . Then this register is multiplied with the three input Tx
vectors ( - ) and the three resulting float values are used as texture coordinate
to sample the graded color into . At last this color is copied into , which is
used as output of the pixel shader.
Using a pixel shader version 1.4 or higher is much easier because the original
color can be directly used as texture coordinate:

For the version 1.4 the screen texture is set to stage one and the volume map to stage zero. In this case we save a instruction. After the original color is loaded into the temporary register is set, so can be used as source register for the next instruction, which samples the graded color.
A pixel shader of version 2 and higher needs to declare all input registers and samplers. So it is done for the texture coordinate of screen texture and for the samplers of that and the volume texture. Then the original color is loaded and used as texture coordinate to load the graded one next. At last it is copied to the output color register. The partial precision modifier is used for nearly all instructions because it can run faster on some hardware and the full precision is not needed.
As you can see version 1.4 needs only two instructions so it is the fastest way. I prefer to use it primarily and to implement a version 1.1 fallback for DirectX 8.0 hardware.
Using a low resolution volume map and point filtering results in a toon effect.

ENHANCEMENTS
To get the best result of this effect, artists should be able to use a powerful editing tool. What could be better than the favorite program?
1. USING A REFERENCE IMAGE
Every professional image processing program has got the ability to adjust colors. To use this feature for our grading algorithm we have to use reference images. The effect class can produce a picture file including all 2563 colors. After an in-game screenshot is adjusted the way the artist wanted the same operations have to be applied on the reference image, which graded colors can be loaded back to the effect class to create the volume of the 3D texture.
When you grade the colors of the scene, from which the screenshot was taken, using that volume texture the result is the same or similar to the picture the artist has adjusted. The accuracy depends on the dimension and color format of the volume.
2. GRADING A FRAME PARTICIALLY
To grade only parts of a scene stencils can be set while rendering it. For every group of objects that should be adjusted by one grading table we change the pixels of the stencil buffer to one unique value using for and setting that value to .
In the post-processing phase is set to . So only the pixels of the stencil reference value are graded by one render call. At every render call the reference value and the gradient texture can be changed.
CONCLUSION
You can use the color grading algorithm to get more atmosphere or special effects to your game. It is also possible to use it for common color operations like inversion, comic, grayscale or sepia shading.
At last here is the summary of advantages and drawbacks:
Pro Contra  Very fast real time rendering  Quick updates for smaller volumes, which are keeping good quality for most settings  Artists can use familiar programs to adjust colors  Different operations can be merged into one table  Large volumes, which are increasing the quality, consume a lot of memory and need much time to be updated  Needs pixel shader hardware

You can look at the CD to find a library, which can easily be included into any Direct3D 9 project, to grade the colors of a texture, which is projected to the screen.
Here you can see a level from another Media Seasons production. The upper left image is the non-graded scene with directional light and blooming. All other renderings are graded from it without changing the light or blooming effect. At the upper right one the original is projected to a day-light setting by increasing the brightness of mid-tones and shadows. The lower left image is a change to a sunrise or sunset scenario by increasing red in the bright areas and blue in the dark ones. Yellow tones are decreased at the last figure for non-bright pixels to create blue moon light.
If you want to learn more about Media Seasons or the games, which are presented in the figures, then visit the web site www.mediaseasons.com.

Shader X³: Image Space - Color Grading

More Related Content

What's hot

Viewers also liked

Similar to Shader X³: Image Space - Color Grading

Recently uploaded

Shader X³: Image Space - Color Grading