The document outlines advancements in real-time rendering techniques from games like Battlefield 3 and Need for Speed: The Run, focusing on methods such as separable bokeh depth-of-field, hi-z culling, and tiled-based deferred shading. It discusses how these techniques enhance performance and optimize resource usage on gaming consoles like Xbox 360. Various algorithms and their implementations are explored to improve lighting, shadow mapping, and image processing efficiency.

1. More Performance!Five Rendering Ideas from Battlefield 3 and Need For Speed: The RunJohn White (NFS)Colin Barré-Brisebois (BF3)Advances in Real-Time Rendering in Games

2. AgendaMotivations

3. The Techniques

4. Separable Bokeh Depth-of-Field

5. Hi-Z / Z-Cull Reverse Reload Tricks

6. Chroma Sub-Sampled Image Processing

7. Tiled-Based Deferred Shading on Xbox 360

8. Temporally-Stable Screen-Space Ambient Occlusion

9. Q&AAdvances in Real-Time Rendering in Games

10. MotivationsFrostbite 2 is DICE’s Next-Generation Engine for Current Generation Platforms

11. 5 Pillars:

12. Animation

13. Audio

14. Scale

15. Destruction

16. Rendering

17. Powers Battlefield 3 and Need For Speed: The RunAdvances in Real-Time Rendering in Games

18. More InfoLots of FB2 papers on DICE website

19. publications.dice.se

20. Also see Alex Ferrier’s talk on “1000 points of light” Tues, 2pm, East Building, Ballroom A/BAdvances in Real-Time Rendering in Games

21. Advances in Real-Time Rendering in Games

22. Advances in Real-Time Rendering in Games

23. Separable BokehDepth-of-FieldAdvances in Real-Time Rendering in Games

24. Real World Bokeh - DiscPhoto Courtesy of MohsinHasan 2011Advances in Real-Time Rendering in Games

25. Real World Bokeh – PentagonalPhoto Courtesy of MohsinHasan 2011Advances in Real-Time Rendering in Games

26. Circle of Confusion CalculationCalc per pixel CoC from real world camera parameters

27. Lens Length (derive from FOV)

28. F/Stop

29. Focal Plane

30. CoC is a simple MADD on the raw Z Depth [Demers04][Jones08]CoC = abs(z * CoCScale + CoCBias) CoCScale = (A * focallength * focalplane * (zfar - znear)) / ((focalplane - focallength) * znear * zfar) CoCBias = (A * focallength * (znear - focalplane )) / ((focalplane * focallength) * znear)Advances in Real-Time Rendering in Games

31. Pre Multiplied CoCFor 16-bit source pre multiply the CoC by the colour

32. Store CoC in alpha

33. Recover colour by doing col.rgb /= col.a

34. Ensure CoC always has a small number so colour can always be recoveredAdvances in Real-Time Rendering in Games

35. Blur ProcessGaussian blur

36. Common in DX9 games. Cheap

37. 2D Area samples

38. Limited kernel size before texture tap explosion

39. GS expanded Point Sprites

40. Heavy fill rate

41. CryEngine3 DX11 and Unreal Engine 3 Samaritan demoAdvances in Real-Time Rendering in Games

42. Gaussian vs. real world bokehArbitrary blurs in image space are O(N^2)

43. Gaussian blurs can be made separable O(N)

44. What 2D blurs can be made separable?

45. Gaussian

46. Box

47. Skewed BoxAdvances in Real-Time Rendering in Games

48. Other separable blursGaussian, Box and Skewed BoxAdvances in Real-Time Rendering in Games

49. Hexagonal BlursDecompose a hexagon into 3 rhombi

50. Each rhombi can be computed via separable blur

51. 7 Passes in total. 3 shapes x 2 blurs + 1 combine12First PassSecond Pass3Advances in Real-Time Rendering in Games

52. Hexagonal Blurs – Pass ReductionHexagonal blur using a separable filter

53. But 7 passes and 6 blurs is not competitive

54. Need to reduce passes12First PassSecond Pass3Advances in Real-Time Rendering in Games

55. Hexagonal Blurs – Pass Reduction 1Pass 1Pass 2Pass 1 Up Down LeftPass 2 Down Left + Down Right + Down Right+Advances in Real-Time Rendering in Games

56. Hexagonal blurs – Pass reduction 2Pass 2Pass 1++Advances in Real-Time Rendering in Games

57. Hexagonal BokehAdvances in Real-Time Rendering in Games

58. Hexagonal BokehAdvances in Real-Time Rendering in Games

59. Hexagonal BokehAdvances in Real-Time Rendering in Games

60. Hexagonal BokehAdvances in Real-Time Rendering in Games

61. Hexagonal vsGaussianGaussian

62. 2 Passes with a total of 2 blurs

63. Hexagonal

64. 2 Passes (3 resolves) with a total of 4 blurs

65. BUT each blur only needs half the taps therefore same #taps

66. BUT each tap contributes equally unlike Gaussian so need less taps for a given aesthetic filter kernel width!

67. PLUS We can improve furtherAdvances in Real-Time Rendering in Games

68. Iterative RefinementBecause we have equal weighted blurs can use iterative refinement on the blurring [Sousa08]

69. Multiple passes fill in the under-sampling

70. Dual iteration blur needs a total of 5 passes with a total of 8 half blurs.Advances in Real-Time Rendering in Games

71. Iterative RefinementPass 5Pass 1Pass 2Pass 3++Pass 4Advances in Real-Time Rendering in Games

72. Pseudo Scatter filterProper bokeh should have its blur scattered to its neighbours

73. However pixel shaders are designed to gather they can’t scatter the results

74. Typical blurs default the filter kernel to the CoC of pixel

75. Instead, default to big CoC and reject based on the sampled texelCoCExtra method to stop bleeding artifacts and can sharpen up smooth gradientsAdvances in Real-Time Rendering in Games

76. Hi Z cullingWhen downsampling the premultipliedCoC buffer output the computed CoC as depth

77. You can then draw the plane at Z depth of 0.001f

78. In focus pixels will be quickly rejected by Hi Z

79. Same for iterative refinement

80. Draw undersample pass at higher Z value, fine at small Z valueRequires an explicit copy afterwards to re-fillAdvances in Real-Time Rendering in Games

81. Hi-Z / Z-Cull Reverse Reload TricksAdvances in Real-Time Rendering in Games

82. Hi-Z (1/)Ubiquitous on modern hardware

83. Stores a Low Res version of the Z buffer

84. Can use this to conservatively reject groups of pixels

85. Saves fragment shading known occluded pixelsAdvances in Real-Time Rendering in Games

86. Hi-Z (2/)Advances in Real-Time Rendering in Games

87. Hi-Z (3/)Solid SpaceEmpty SpaceAdvances in Real-Time Rendering in Games

88. Volume Rendering (1/)In Deferred Renderers it is common to reproject screen pixels back into world spaceCommon for lights (point, spot, line)Shadow volumesDraw a convex bounding polyhedron projected in screens spaceIn shader, reject pixels which are not in the volume boundsAdvances in Real-Time Rendering in Games

89. Volume Rendering (2/)BACAdvances in Real-Time Rendering in Games

90. Reverse Hi-Z Reload (X360) Alias a Render Target on existing depth buffer

91. Init aliased RT to D3DHIZFUNC_GREATER_EQUAL

92. Draw Full screen quadNULL Pixel ShaderZfunc == NeverHi-Z is now primed in reverse

93. Similar technique on Playstation3 (See DevNet)Advances in Real-Time Rendering in Games

94. Reverse Hi-Z (1/)Solid SpaceEmpty SpaceEmpty SpaceSolid SpaceAdvances in Real-Time Rendering in Games

95. Reverse Hi-Z (2/)The GPU will now cull fragments if they are closer than the value in the depth buffer

96. By rendering the backfaces of convex polyhedra, pixels beyond the faces will quickly reject

97. If the camera is inside the volume then only pixels inside the volume will pass

98. Perfect for cascaded shadow mapsAdvances in Real-Time Rendering in Games

99. Reverse Hi-Z (3/)BACAdvances in Real-Time Rendering in Games

100. Reverse Hi-Z for CSM renderingEach cascade for a directional light is bounded by a cuboid in world space

101. Only world space pixels inside the cuboid will project onto the shadow map

102. By drawing the cuboid backfaces only these pixels will pass the reverse Z testAdvances in Real-Time Rendering in Games

103. CSM CuboidsCSM1CSM0CABAdvances in Real-Time Rendering in Games

104. CSM CuboidsEvaluate as a separate pass [Sousa08]Input is depth bufferCreates a L8 mask texture input into directional light passCan do a prior full screen pass to tag back facing pixels wrt to light source in stencilHeuristic on sun angle with cameraPotential for ¼ res with bilateral upsample

105. Stencil is updated to denote already processed pixelsAdvances in Real-Time Rendering in Games

106. Reverse Hi-Z CSM (1/)Cascade 0Advances in Real-Time Rendering in Games

107. Reverse Hi-Z CSM (2/)Cascade 1Advances in Real-Time Rendering in Games

108. Reverse Hi-Z CSM (3/)Cascade 2Advances in Real-Time Rendering in Games

109. Reverse Hi-Z CSM (4/)Cascade 3Advances in Real-Time Rendering in Games

110. Min/Max Shadow Maps (1/)Downsample and dilate SM, keeping track of min and max depthsAdvances in Real-Time Rendering in Games

111. Min/Max Shadow Maps (2/)Dilated min/max SM allow us to know if a pixel is ...Fully In shadowFully out of shadowPartially in shadow (conservatively)Draw each cuboid twice

112. First PassSingle simple tap shaderSecond PassHigh quality PCF shaderAdvances in Real-Time Rendering in Games

113. Min/Max Shadow Maps Simple Pass If Z < MinMax.minreturn (1,1,1,1)If Z > MinMax.max return (0,0,0,1)If Z < MinMax.min return (0,0,0,0) StencilMaskAdvances in Real-Time Rendering in Games

114. Min/Max Shadow Maps PCF Pass (1/)Second pass is standard PCF filterMaskStencilAdvances in Real-Time Rendering in Games

115. Min/Max Shadow Maps (3/)Do for all cascadesAdvances in Real-Time Rendering in Games

116. Min/Max Shadow Maps PCF Pass (2/)Final OverdrawAdvances in Real-Time Rendering in Games

117. Conditional TestsWhen we render the cuboid we can count how many pixels pass

118. If Zero then no pixels will sample from the shadow mapSo why even render the shadow map!Draw the cuboid first and only if pixels pass draw the actual shadow mapZero passed pixels occur for two reasonsAll pixels are further awayAll pixels have been touched by a closer cascade (stencil cleared)Advances in Real-Time Rendering in Games

119. Chroma Sub-Sampled Image ProcessingAdvances in Real-Time Rendering in Games

120. Chroma Sub-SamplingNot a new idea. Used in TV broadcasts as well as Jpeg/Mpeg compression

121. Decompose image into luminance and chroma

122. Store Luma at full res only. Chroma at lowerAdvances in Real-Time Rendering in Games

123. Chroma Sub-Sampling - MotivationPost processing requires lots of bandwidth

124. Easy to optimise ALU down

125. Quickly hit a performance ceiling, especially for 16bpp pixels

126. Reading and writing a 720p image with 16bpp components is 14MB bandwidth

127. Assuming 14GB/Sec Bandwidth and perfect cache usage this is 1ms for a single passAdvances in Real-Time Rendering in Games

128. Chroma Sub Sampling - MotivationInstead reduce down to Luma only

129. ¼ of the bandwidth required

130. Requires extra processing on the Colour

131. But this can be 2 channel at ¼ res ( 1/8 original size )

132. Also can get away with less tapsAdvances in Real-Time Rendering in Games

133. Chroma Sub Sampling Bandwidth is reduced to 1/4

134. So, are the shaders now 4X quicker?

135. No. We are ALU bound again

136. Texture units and ALU are designed for 4 component SIMD

137. We are only using 1 component

138. Need to pack 4 luma values together and process togetherAdvances in Real-Time Rendering in Games

139. Chroma Sub Sampling Pack 4 adjacent pixels together into one RGBA pixel

140. Only need 1 texread to get 4 luma values

141. So a 1280x720 luma buffer is a 320x720 ARGB buffer

142. With the packed buffer bilinear filtering is not correct

143. Have to manually filter horizontally using DOTP

144. Colin will go into this laterAdvances in Real-Time Rendering in Games

145. Chroma Sub Sampling Advances in Real-Time Rendering in Games

146. Chroma Sub Sampling Advances in Real-Time Rendering in Games

147. Butterfly PackingOverlay each quadrant into ARGB

148. Mirror around the image center point

149. Bilinear now works except across the boundaries Re-draw a strip with additive blend and swizzlingR<->G and B<->A for horizontal R<->B and G<->A for verticalRadial blurs just workAdvances in Real-Time Rendering in Games

150. Butterfly UnpackingWhen rendering fullscreen quad, need two attributesUse UV in Mirror ModeDot with saturated second component0,0640,-640,-360,-3602,0-640,640,-360,-3600,2-640,-640,360,-3602,2-640,-640,-360,360Advances in Real-Time Rendering in Games

151. Future WorkUse for hexagonal blurs

152. Output packed tonemapOnly perform temporal AA for lumaPacked luma used for MLAA passesAdvances in Real-Time Rendering in Games

153. Tiled-based Deferred Shading on Xbox 360Advances in Real-Time Rendering in Games

154. Tiled-based Deferred Shading? (1/)Want more interesting lighting with more dynamic lights!

155. Platform is fixed  better usage of rendering resources

156. [Swoboda09] and [Coffin11] on Playstation3™, by [Andersson09] in DirectCompute, and other hybrids

157. Christina, Johan and I teamed-up for this version on 360

158. Load-balance and compute lighting where it matters:Divide the screen in screen-space tilesCull analytical lights (point, cone, line), per tile Compute lighting for all contributing lights, per tileAdvances in Real-Time Rendering in Games

159. Tiled-based Deferred Shading? (2/)Advances in Real-Time Rendering in Games

160. Tiled-based Deferred Shading? (3/)Advances in Real-Time Rendering in Games

161. Tiled-based Deferred Shading? (4/)Advances in Real-Time Rendering in Games

162. Tiled-based Deferred Shading? (5/)Advances in Real-Time Rendering in Games

163. How Does This Fit on Xbox 360?We don't have DirectCompute nor SPUs on 360...

164. Fortunately, Xenos is powerful, and will crunch ALU

165. For maximal throughput, data at rendering time has to be cleverly pre-digested

166. If timed properly, we can also use the CPUs to help the GPU along the way...

167. GPU is better at analyzing a scene than CPUs…

168. Let’s use it to classify the sceneAdvances in Real-Time Rendering in Games

169. GPGPU Culling (1/)Our screen is divided in 920 tiles of 32x32 pixels

170. Downsample and classify the scene from 720p to 40x23 (1 pixel == 1 tile)

171. Find each tile’s Min/Max depth

172. Find each tile’s material permutations

173. Downsampling is done in multi-pass and via MRTs

174. Similar to [Hutchinson10]Advances in Real-Time Rendering in Games

175. GPGPU Culling (2/)Advances in Real-Time Rendering in Games

176. GPGPU Culling (3/)Advances in Real-Time Rendering in Games

177. GPGPU Culling (4/)Advances in Real-Time Rendering in Games

178. GPGPU Culling (5/)Build mini-frustas for each tile

179. Cull lights against sky-free tiles in a shader

180. Store the culling results in a texture:

181. Column == Light ID

182. Row == Tile ID

183. Actually, 4 lights can be processed at once (A-R-G-B)

184. Read back the contribution results on the CPU and prepare for lighting! Advances in Real-Time Rendering in Games

185. I Need a LightParse the culling results texture on CPU

186. For each light type, For each tile, For each material permutation,

187. Regroup & set the light parameters for the PS constants

188. Setup the shader loop counter*

189. Additively render lights with a single draw call (to the final HDR lighting buffer)Advances in Real-Time Rendering in Games

190. Results (1/)Advances in Real-Time Rendering in Games

191. Results (2/)Advances in Real-Time Rendering in Games

192. TimelineGPUG-BufferDSCOtherLightCPUG-BufferOtherPrepareDS: Downsample / Classify

193. C: Cull

194. Light: Lighting pass

195. We kick CPU jobs from the GPU using a MEMEXPORT shader (i.e.: write token at specific address, job starts)Advances in Real-Time Rendering in Games

196. Don’t Upset The GPU (1/)Constant Waterfall sucks!

197. This WILL kill performance

198. To prevent, use the aL register when iterating over lights [Pritchard10]

199. If set properly, ALU / lighting will run at 100% efficiency

200. In C++ CodeintlightCounter[4] = { count, start, step, 0 };pDevice->SetPixelShaderConstantI(0, lightCounter, 1);Advances in Real-Time Rendering in Games

201. Don’t Upset The GPU (2/)inttileLightCount: register(i0); float4 lightParams[NUM_LIGHT_PARAMS] : register(c0);[loop] for (intiLight = 0; iLight< tileLightCount; iLight++) { float4 params1 = lightParams[iLight + 0]; // mov r0 c0[0+aL] float4 params2 = lightParams[iLight + 1]; // mov r1 c0[1+aL] float4 params3 = lightParams[iLight + 2]; // mov r2 c0[2+aL] …}stepstartcount*stepAdvances in Real-Time Rendering in Games

202. Don’t Upset The GPU (3/)Use Dr.PIX, and check shader disassembly!

203. These shaders are ALU bound

204. Simplify your math, especially in the loops!

205. Get rid of complicated non 1:1 instructions (e.g. smoothstep)

206. Play with microcode: -normalize(v) is faster than normalize(-v)

207. Move code around to help with dual-issuing: /* 14 */ mul r5.xyz, r4.yzx, r4.yzx this + mulsc r0.w, c254.y, r0.zUse shader predicates to help the compiler ([flatten], [branch], [isolate], [ifAny], [ifAll]), and tweak GPRs!Advances in Real-Time Rendering in Games

208. Don’t Upset The GPU (4/)Use GPU freebies

209. Texture sampler scale/bias (*2-1)

210. Simplify / remove unneeded code via permutations

211. Upload constants via the constant buffer pointers

212. We use async pre-compiled command buffers (APCBs)

213. Keep them lean & mean (check contents in PIX)

214. For more info, check out Ivan’s awesome presentation from Gamefest 2011 [Nevraev11]Advances in Real-Time Rendering in Games

215. PerformanceClassification: 1.35 ms (with resolves)Advances in Real-Time Rendering in Games

216. Temporally-stableScreen-Space Ambient OcclusionAdvances in Real-Time Rendering in Games

217. SSAO in Frostbite 2 (1/)SSAO for mid-range PC & consoles, HBAO for high-end PC

218. Line sample [Loos10], with linear depth in a texture

219. Linearize depth for better precision/distributionkZ = – far * near / (far – near); kW = far / (far – near)linearDepth = kZ / (z - kW)

220. Sample linear depth texture with linear sampling

221. Scale SSAO parameters over distance

222. Final compositing with Hi-Stencil, reject sky pixels

223. 4x4 random noise texture is sufficient, 1:1 (texel:pixel)Advances in Real-Time Rendering in Games

224. SSAO in Frostbite 2 (2/)Line sampling, from Volumetric Obscurance[Loos10]

225. HBAO in Frostbite 2Advances in Real-Time Rendering in Games

226. SSAO in Frostbite 2 (1/)Advances in Real-Time Rendering in Games

227. Blurring the line (1/)Dynamic AO is done best with edge-preserving blur / bilateral filtering

228. On consoles, we have really tight budgets

229. Scenes are pretty action-packed, halos not too noticeable

230. AO should be a subtle effect

231. We need to find the fastest way to blur AO, and has to look soft! (e.g.: 9x9 Gaussian, with bilinear)Advances in Real-Time Rendering in Games

232. Fast Grayscale Blur - 8 as 8888 (1/)Reduce the number of taps: aliasing the AO results from R8 as A8R8G8B8

233. 1 horizontal tap == 4 taps (ARGB)

234. Combine with bilinear sampling (vertical pass only)

235. 9x9 Gaussian = 3 horizontal taps and 5 vertical taps

236. On PS3: alias the memory directly

237. On 360: Formats are different in memory, use resolve remap textures. See FastUntile XDK sample.Advances in Real-Time Rendering in Games

238. Fast Grayscale Blur (1/)Vertical9 “samples  5 bilinear tapsHorizontal9 “samples”  3 point tapsAdvances in Real-Time Rendering in Games

239. Fast Grayscale Blur (2/)For a 640x360 SSAO Buffer (720p / 2)Average (total) AO performance (compute + blur + blit) : (360: 1.25-1.5 ms ; PS3: 1.5-2.0 ms)Advances in Real-Time Rendering in Games

240. Thank YouChristina Coffin

241. Johan Andersson

242. Ivan Nevraev

243. Daniel Collin

244. Khalid Khalkhouli

245. Andrew Routledge

246. Stephen Hill

247. Aurelio Reis

248. Alex Ferrier

249. Fredrik Seehussen

250. Alex Fry

251. Natalya Tatarchuk

252. Mohsin HasanAdvances in Real-Time Rendering in Games

253. QuestionsJohn White – Bokeh, Z Cull Reverse Reload, Chroma SubsamplingColin Barré-Brisebois – Tile-based Deferred Shading, SSAOAdvances in Real-Time Rendering in Games

254. References (1/)ANDERSSON, J., “Parallel Graphics in Frostbite - Current & Future”, Beyond Programmable Shading, SIGGRAPH 2009.BAVOIL, L., SAINZ, M., and DIMITROV, R., “Image-space horizon-based ambient occlusion”, SIGGRAPH 2008.COFFIN, C., “SPU Based Deferred Shading for Battlefield 3 on Playstation 3”, GDC 2011.DEMERS, J., “Depth of Field : A Survey of Techniques”, GPU Gems, Ch.23.HUTCHINSON, N. et al., “Screen Space Classification for Efficient Deferred Shading”, SIGGRAPH 2010.JONES, M., Optimal CoC Calculation, Rendering @ EA internal mail, 2008.Advances in Real-Time Rendering in Games

255. References (2/)LOOS, B. J., and SLOAN, P-P., “Volumetric Obscurance”, 2010.NEVRAEV, I., “Xbox 360 Precompiled Command Buffers”, Microsoft Gamefest London 2011.PRITCHARD, C., “Xbox 360 Shaders and Performance: How Not to Upset the GPU”, Microsoft Gamefest Seattle 2010.SOUSA, T., “Crysis Next Gen Effects”, GDC 2008.SWOBODA, M., “Deferred Lighting and Post-Processing on PLAYSTATION®3”, GDC 2009.KAWASE, M., “Frame Buffer Postprocessing Effects in DOUBLE-S.T.E.A.L (Wreckless), GDC 2003.Advances in Real-Time Rendering in Games