Around the World in 80 Shaders


Published on

Talk given in the programming track at Develop in Liverpool 2010 on 25th November.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • At Bizarre, our environment and character artists primarily work in Maya whereas our vehicle team use XSI.

    For example, in our Bangkok levels in Blood Stone we had a dynamically generated ripple texture used as a normal map on puddles. The artist exposed the normal map parameter on that shader with the name “Ripple” and we were then able to set the ripple texture to that exposed parameter.
  • Semantics such as POSITION and NORMAL have default settings so there is no need to annotate them. As for the texture coordinate, we are telling it to use a format of four F16s, put UV set 0 into the xy coordinates and put our lightmap UV set into the zw coordinates. (Lux is our lighting tool.)

    Vertex declarations are now therefore fully data driven.
  • This Lua script is run per polygon, which although is slow, isn’t as slow as you think. There are so many ways we can use this functionality. For example, for shaders where the vertex colours are used to blend between two textures, we could read the vertex colours in script, detect where no blending is taking place and fallback to a permutation with only one texture instead of two. Or where vertex alpha is used to alpha fade, we could move any polygons with a vertex alpha of 1 into the solid render pass.
  • F0 should never be zero, the minimum real world value is 0.02.
  • We call our specular texture IPR (just look at the initials).

    We don’t just have sun highlights for specular, we also have an environment map. We specify the mip map level we look up with the green channel. Make sure that the blurriness of the cube map and the size of the specular highlight match!

    (My advice on doing this is to avoid changing the equations, as they’re simple and you don’t want to complicate things. Instead, adjust the blurriness of the cube map mip levels when you generate it. CubeMapGen is great for this. We realised that we needed to blur each mip level a little bit more than normal – if you just downsize to create the mips, you’re left with an ugly blocky reflection instead of a blurry reflection.)
  • Almost everything on this shot is using just the default shader, including the car! As you can see, it’s really powerful and easy for the artists to set up.
  • Some other approximations are described by John Hable in his talk on Uncharted 2 Character Lighting and Shading in the Advances in Real-Time Rendering in 3D Graphics and Games course at SIGGRAPH 2010.
  • This is the reference photo our art director gave me!
  • Note:
    We offset the lookup into the noise texture by the corruption amount, so that as the corruption blends in and out it animates.
    We also brighten and darken the colour of each block by a small amount, to give more variation in the effect.
  • The normal and specular textures are sampled with the standard uv, whereas the albedo is sampled with the newly calculated refracted uv.
  • We calculate the tangent view vector (the view vector in tangent space) in the vertex shader and upload via an interpolator to the pixel shader.
  • Had to dig around in the holiday albums for these… the picture on the left is from a little stream in the Forest of Bowland (I fell into this at one point…), whereas on the right is Lake Michigan, as seen from Muskegon. The deep water on the right looks opaque, whereas on the left you can see the rocks underneath the water.
  • This is for diffuse lighting only. We simulate specular with a standard physically-correct Blinn-Phong model. Remember that the normal specular reflectance of water is 0.02.
  • L0 is the colour at the surface at the water, the water surface colour modulated by the diffuse lighting. LZ is the colour underneath the water, you’ll need to look this up from either your frame buffer or a specifically generated refraction texture. I’m using L to denote that both these points have already been lit before being fed into this equation.
  • The original equation has two exponentials in it, which given that they are scalar instructions and we need to operate on a vector, amounts to six whole scalar instructions. That’s really going to block up your scalar pipe, so minimising the exponentials is really key.

    Observe that 1 – exp(-cR)exp(-KH) is always between 0 and 1 (given that c, K, R and H are all positive, so exp(-cR) and exp(-KH) are both between 0 and 1) and when H is large, R is large (although the reverse isn’t necessarily true), which takes the exponential terms to 0.
  • This now falls out into a lerp!
  • Although you can notice the visual difference by removing this term, this is enough of an approximation anyway that it would be hard for you to say that one looks more realistic than the other. So for performance reasons, it’s best to remove it.
  • Having a real-time reflection really helps!
  • Can we treat the aquarium tank in the same way we treat shallow water? Rather than having a horizontal water plane, now we just have a vertical one.
  • The answer is no, because the aquarium tank just looks far too dark.
  • If we look at real aquarium tanks (here’s a huge one in Okinawa, Japan), they clearly have light filtering down from the top of the tank. They’re definitely not as dark as the implementation I just showed!
  • Can we treat the aquarium tank in the same way we treat shallow water? Rather than having a horizontal water plane, now we just have a vertical one.
  • To calculate the distance under the water, you do need to have the height of the light (i.e. the world space position of the top of the aquarium tank) passed up as a shader constant.
  • Around the World in 80 Shaders

    1. 1. Around the World in 80 Shaders Stephen McAuley Bizarre Creations
    2. 2. Overview • Introduction: – The FX system • Around the world tour: – Default – Skin – MPEG corruption – Refraction mapping – Shallow water – Aquarium
    3. 3. FX System: Problem • After finishing PGR4 and The Club we needed a new solution to handling shaders. – Two different systems. – Extremely complicated to add new shaders. • Have to add new vertex declarations, new exporter code and new rendering code as well. – Not data driven. • Shaders treated as code not data. • Could never run on an SPU. – No Maya previewing for artists.
    4. 4. FX System: Solution • Our solution was the FX system: – Based on .fx files. – Entirely data driven. – Automatic Maya previews for artists. – All rendering code now runs on the SPUs.
    5. 5. FX System: Maya and XSI • Compile Maya and XSI shaders alongside PC shaders. • Annotations control how the parameters appear to the artists: • Artists can “expose” parameters they want to be set within game. texture2D g_tAlbedo < string SasUiLabel = “Colour”; >; float g_fScale < string SasUiLabel = “Scale”; > = 0.0;
    6. 6. FX System: Exporting • Scene converter: – Reads vertex input structure • With the help of annotations. – Exports relevant vertex data in the correct format. struct C_VertexInput { float3 m_vPosition : POSITION; float3 m_vNormal : NORMAL; float4 m_vTexCoord : TEXCOORD0 annotation(xy(uv:0) | zw(uv:luxlightmap) | format(F16_4)); };
    7. 7. FX System: Exporting • Lua script within each shader helps build shader permutations: • If a bumpmap is present, game will search for ShaderB.fx instead of Shader.fx. • Make sure you have compiled that permutation! texture2D g_tBumpmap < string SasUiLabel = “Bumpmap”; >; #if defined(FXFILE_PARSER) if (g_tBumpmap) then append(“B”, 10) end #endif
    8. 8. FX System: In Game • Game sets exposed and shared parameters. • Techniques are changed dynamically: • e.g. shadow mapping technique, prepass technique… • Shader parameters can be overridden via our debugging tool: – Currently only on all instances of a shader.
    9. 9. Default Athens
    10. 10. Default Shader • This is our base material for everything in game. • Applied when artists use the Lambert shader in Maya. – Easy to set up. – Artists just set an albedo texture. – Other textures are picked up in the exporter using a naming convention.
    11. 11. Default Shader: Material • The following material options are supported: – Albedo (_DIFF) – Normal (_NORM) – Specular (_IPR0) – Emissive (_NEON) • These are all controlled by textures.
    12. 12. Default Shader: Lighting • We support these lighting methods: – Light maps – Vertex lighting – Vertex PRT – SH lighting • In Blood Stone, they contained the following information: – Colour, sun occlusion and ambient occlusion
    13. 13. Default Shader: Lighting Our BRDF for sun lighting: 𝜌 = 𝑅 𝑑 𝜋 1 − 𝐼𝐹 𝐹0 + 𝐼𝐹(𝐹0) 𝑛 + 2 2𝜋 (𝑅. 𝑉) 𝑛 (𝑁. 𝐿) where 𝑅 𝑑 is the surface albedo and 𝐹(𝐹0) is Schlick’s fresnel approximation: 𝐹(𝐹0) = 𝐹0 + (1 − 𝐹0)(𝑁. 𝑉)5
    14. 14. Default Shader: Specular We have three parameters controlling specular: 𝜌 = 𝑅 𝑑 𝜋 1 − 𝐼𝐹 𝐹0 + 𝐼𝐹(𝐹0) 𝑛 + 2 2𝜋 (𝑅. 𝑉) 𝑛 (𝑁. 𝐿) • 𝐼 is the specular intensity • 𝑛 is the specular power • 𝐹0 is the normal specular reflectance
    15. 15. Default Shader: Specular • These three specular properties come from our specular texture: – Red channel: Intensity (𝐼) – Green channel: Power (𝑛) • 𝑛 = 4 + 8188𝑔2 • 𝑚𝑖𝑝𝑀𝑎𝑝𝐿𝑜𝑑 = 7(1 − 𝑔) – Blue channel: Reflectance (𝐹0)
    16. 16. Skin Istanbul
    17. 17. Skin • Standard diffuse lighting gives unrealistic results when applied to skin. • It looks hard and dry compared to skin’s soft appearance. • To achieve this look, we need to model subsurface scattering.
    18. 18. Skin • The best approach is described by Eugene d’Eon and David Luebke in [1]: – Observed that red light scatters in skin more than green and blue. – Simulate subsurface scattering by lighting in texture space, then performing separate gaussian blurs for red, green and blue channels. • This is expensive and intrusive. Are there any cheaper ways?
    19. 19. Skin: Approximations • Standard diffuse lighting: D = N.L • Wrapped diffuse lighting: D = N.L * w + (1 – w) • i.e. D = N.L * 0.5 + 0.5 • What if we wrap different colours by different amounts?
    20. 20. Skin: Our solution • Coloured wrapped diffuse lighting: D = N.L * W + (1 – W) • i.e. D = N.L * { 0.5, 0.8, 0.9 } + { 0.5, 0.2, 0.1 } • Simulates subsurface scattering by giving skin a softer look. • Simulates red light scattering further in skin. • Cheap and easy to implement.
    21. 21. Standard Diffuse
    22. 22. Wrapped Diffuse
    23. 23. Coloured Wrapped Diffuse
    24. 24. MPEG Corruption Monaco
    25. 25. MPEG Corruption • For the AR phone, we wanted an effect that simulated MPEG corruption. • “Blocks” in the image don’t update and remain from the previous frame.
    26. 26. MPEG Corruption • Algorithm: – For each pixel, point sample a noise texture. • Scaled so each texel is the size of a “block”. – If sample is less than a threshold, take pixel from previous frame buffer. • Instead of the previous frame buffer, use a scaled and offset version of the current frame. – Else, take pixel from current frame buffer.
    27. 27. MPEG Corruption half2 vRUV = vUV * g_vCorruptionBlockSize; vRUV += half2(g_fCorruptionAmount * 7.123h, 0.0h); half fRand = h4tex2D(g_tNoise, vRUV).r; if(fRand < g_fCorruptionAmount) { real2 vDUV = input.m_TexCoord * g_vCorruptionUVScale; vDUV += g_vCorruptionUVOffset; vColour = h4tex2D(g_tBackbuffer, vDUV).rgb; vColour *= (1.0h + fRand - 0.5h * g_fCorruptionAmount); } else { vColour = h4tex2D(g_tBackbuffer, vUV).rgb; }
    28. 28. Refraction Mapping Siberia
    29. 29. Refraction Mapping • We wanted to simulate surfaces that are coated in a layer of ice. • Requirements: – These surfaces have two layers: • Ice layer • Base layer – The base layer is refracted by the ice layer.
    30. 30. Refraction Mapping • Our in-game materials use three textures: – albedo – normal – specular • We split these textures across the two layers. Ice layer: • normal • specular Base layer: • albedo uv refracted uv
    31. 31. Refraction Mapping • Calculate the refracted texture coordinate: • Albedo texture uses vRefractUV. • Normal and specular maps use vUV. half3 vRefract = refract(vTangentView, vTangentNormal, 1.0f / g_fRefractiveIndex); float2 vUVOffset = g_fDisplacement * vRefract.xy / vRefract.z; float2 vRefractUV = vUV + vUVOffset;
    32. 32. Default
    33. 33. Displacement
    34. 34. Refraction
    35. 35. Refraction Mapping: Extensions • Vary the displacement across the surface: – Height map. – Vertex colours. • Add a frost layer in between the top layer and the base layer.
    36. 36. Default
    37. 37. Refraction
    38. 38. Frost
    39. 39. Refraction Mapping: Problems • Make sure your tangent space is correct! – Any errors will very quickly become apparent. • Displacement breaks as you look at the surface edge on.
    40. 40. Shallow Water Burma
    41. 41. Shallow water Deep water
    42. 42. Shallow Water • Why does shallow water appear more transparent than deep water? – Water molecules absorb and scatter light. – The further light travels, the higher probability it is absorbed or scattered. – Absorption and scattering are dependent on wavelength: • e.g. red light is absorbed more than green or blue, giving water its characteristic colour.
    43. 43. Shallow Water • Premoze and Ashikhmin suggested the following model for the absorption and scattering of light in water [2]: 𝐿 0, 𝜃, 𝜑 = 𝐿 𝑍, 𝜃, 𝜑 𝑒−𝑐𝑅 + 𝐿 𝑑𝑓 0 1 − 𝑒 −𝑐+𝐾 𝑑 𝑐𝑜𝑠𝜃 𝑅
    44. 44. Shallow Water 𝐿 = 𝐿 𝑧 𝑒−𝑐𝑅 + 𝐿0(1 − 𝑒−𝑐𝑅 𝑒−𝐾 𝑑 𝐻 ) R H LZ L0 extinction inscattering c and Kd are wavelength dependent absorption and scattering coefficients
    45. 45. Shallow Water 𝐿 = 𝐿 𝑧 𝑒−𝑐𝑅 + 𝐿0 1 − 𝑒−𝑐𝑅 𝑒−𝐾 𝑑 𝐻 • Optimisation: – Remove the 𝑒−𝐾 𝑑 𝐻term. – Only changes the falloff of the inscattering, not the intensity.
    46. 46. Shallow Water 𝐿 = 𝐿 𝑧 𝑒−𝑐𝑅 + 𝐿0 1 − 𝑒−𝑐𝑅 • Optimisation: – Remove the 𝑒−𝐾 𝑑 𝐻term. – Only changes the falloff of the inscattering, not the intensity.
    47. 47. Shallow Water 𝐿 = 𝐿 𝑧 𝑒−𝑐𝑅 + 𝐿0 1 − 𝑒−𝑐𝑅 • Optimisation: – Remove the 𝑒−𝐾 𝑑 𝐻term. – Only changes the falloff of the inscattering, not the intensity. • Visual artefact: – Very shallow water is slightly darker.
    48. 48. Shallow Water • The final equation is a lerp between the water colour and the underwater colour, based on the extinction: 𝐿 = lerp(𝐿0, 𝐿 𝑧, 𝑒−𝑐𝑅 ) • Final shader code: half3 vExtinction = exp(-g_vExtinctionCoeffs * fDistUnderwater); half3 vDiffuse = lerp(vSceneColour, vWaterColour, vExtinction);
    49. 49. No extinction
    50. 50. Red extinction
    51. 51. Green/blue extinction
    52. 52. Aquarium Bangkok
    53. 53. Aquarium • Is an aquarium tank equivalent to shallow water? R LZ L0
    54. 54. With Extinction
    55. 55. Image courtesy Jon Rawlinson
    56. 56. Aquarium • Simulate a light positioned directly above the tank. • This is our new inscattering term. R LZ L0 M0 H MZ
    57. 57. Aquarium • Inscattering equation: 𝐼 = 𝑀0 𝑒−𝑐1 𝐻 (1 − 𝑒−𝑐0 𝑅 ) • Light colour, 𝑀0, should be the colour of the light as it hits the surface of the water. – i.e. light colour modulated by water colour • Use a different attenuation coefficient to the extinction. – Allows for more interesting hue shifts.
    58. 58. Aquarium • Shader code: half3 vExtinction = exp(-g_vExtinctionCoeffs * fDistanceThroughWater); half3 vInscattering = exp(-g_vInscatteringCoeffs * fDistanceUnderWater); vInscattering *= g_vLightColour.rgb; half3 vDiffuse = lerp(vSceneColour, vInscattering, vExtinction);
    59. 59. Aquarium • For the final touch, add light shafts: – Project two scrolling textures in the world x-z plane. • Sample each texture at a fixed distance behind the glass front of the aquarium tank. – Combine texture samples into light shaft term, 𝑘. – Modify inscattering term: inscattering = 𝑀0 𝑒−𝑐1 𝐻(1−𝑘) (1 − 𝑒−𝑐0 𝑅 )
    60. 60. With Light
    61. 61. With Light Shafts
    62. 62. References • [1] Advanced Techniques for Real-Time Skin Rendering, Eugene d’Eon and David Luebke, GPU Gems 3, 2008 • [2] Rendering Natural Waters, Simon Premoze & Michael Ashikhmin, 2001
    63. 63. Credits • For working on the technology described in this presentation: – Paul Malin – Jan van Valburg – David Hampson