Downloaded 13 times
More Related Content
Similar to Deferred Lighting and Post Processing on PLAYSTATION®3
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
Deferred Lighting and Post Processing on PLAYSTATION®3
- 1. Deferred Lighting andPost Processing on PLAYSTATION®3 Matt Swoboda PhyreEngine™ Team Sony Computer Entertainment Europe (SCEE) R&D
- 2. 2 Where Are WeNow? • PS3 into its 3rd year • Many developers on their 2nd generation engines • Solved the basic problems • SPUs STILL underused – But it’s improving
- 3. 3 But.. • GPU nowthe most common bottleneck • Usually limited by fragment operations • Many titles take > 1/3 of their time in post processing • Most developers want to do even more fragment work – More / heavier post processing effects – Better lighting techniques / more lights / softer shadows – Longer shaders – Features ported from PC / other console hardware
- 4. 4 “We fixed thevertex bottleneck..” • Many possible solutions to improve geometry performance beyond just “optimising the shader” – LOD – Occlusion culling & visibility culling – Move large vertex operations to SPU, e.g. skinning – SPU triangle culling
- 5. 5 What About Pixels? •Fragment operations / post processing rarely optimised like geometry operations – Throw whole operation at the GPU – Same operation done for every pixel – Spatial optimization / branching considered too slow • SPU not considered: “too slow”, “uses too much bandwidth”
- 6. 6 SPU pixel processing •Yes, the SPU is fast enough to process pixels • Won’t beat the GPU in a brute force race • GPU specialises in rasterising triangles and sampling textures – has dedicated hardware • SPU is a general purpose processor – Use flexibility to your advantage – Choose different code branches and fast paths
- 7. Post Processing Effectson SPU A Whirlwind Tour
- 8. 8 What to doon SPU • Options: • Offload whole processes from GPU to SPU • Or use SPU and GPU together to do one process
- 9. 9 Depth Of FieldPre-Process • High quality depth of field requires a long fragment shader – Read depth samples and colour samples in a kernel / disc – Check depths against centre pixel depth – Weight colours by depth check results • Wasteful for “most” of the screen – All depth checks pass (out of focus) or all fail (in focus) – All fail == pass through original buffer – All pass == use pre-blurred buffer – separable gaussian blur • Categorise the screen for these cases on SPU
- 10. 10 Depth Of FieldClassification Results • Post process depth buffer • Classify by min/max depth • Green: fully in focus • Blue: fully out of focus • Red: neither fully in or out
- 11. 11 Depth Of FieldPre-process results • Pre-process only on SPU, blur operations on GPU – Goal: minimise overall frame time and latency • Large blur w.r.t. depth • 15 ms+ on GPU alone • 1.5-2ms on SPU + 3 ms on GPU
- 12. 12 Screen Tile Classification •Categorise the screen using the range of depth values within a tile • Powerful technique with many applications – Full screen effect optimization - DOF, SSAO.. – Soft particles – Affecting lights – Occluder information
- 13. 13 Screen Space AmbientOcclusion (SSAO) • Generate an ambient occlusion approximation using the depth buffer alone • Perform a large kernel-based series of depth comparisons and sum the results • Downsample output to ½ size for performance – Output normals for bilateral upsampling
- 14. 14 SPU Screen SpaceAmbient Occlusion Results • GPU version: 10ms+ • SPU version: 6ms on 2 SPUs • Used in “Donkey Trader” PhyreEngine game template
- 15.
- 16. 16 Deferred Shading Overview •Rasterise geometry information to multiple “GBuffers” (geometry buffers) • Apply lighting and shading in a post process
- 17.
- 18. 18 Deferred Lighting onSPU • The SPU can handle the deferred lighting process • The GPU renders the geometry to GBuffers • SPU and GPU execute in parallel – Total time : max( geometry, lighting )
- 19. 19 Deferred Lighting onSPU: Implementation (1) • Process each pixel once • Work out which lights affect each pixel • Apply the N affecting lights in a loop • Process the screen in tiles • Use classification techniques per tile to optimise
- 20. 20 Deferred Lighting onSPU: Implementation (2) • Calculate affecting lights per tile – Build a frustum around the tile using the min and max depth values in that tile – Perform frustum check with each light’s bounding volume – Compare light direction with tile average normal value • Choose fast paths based on tile contents – No lights affect the tile? Use fast path – Check material values to see if any pixels are marked as lit
- 21. 21 Deferred Lighting onSPU: Implementation (3) • Choose whether to process MSAA per tile – If no sample pair values differ, light only one sample from the pair, otherwise light both samples separately – Typically quite few tiles need both MSAA samples lit Tiles requiring MSAA
- 22. 22 Deferred Lighting onSPU: Results • 3 shadow casting lights, 100 point lights • 2x MSAA, 720p – Lighting performed per sample • Apply tone mapping on SPU – Virtually free • Performance: > 60 fps, 3 SPUs for 11ms each – No MSAA: 2 SPUs for 11ms
- 23. 23 Deferred Lighting onSPU: Issues • Potential latency – Must keep GPU busy while SPU process is running – Render something else or add a frame of latency • Main memory requirements • Shadows – Requires “random” texture access – not ideal for SPU – Can render shadows on GPU to a full screen buffer and use it on SPU
- 24. 24 Flavours of DeferredLighting on SPU • Full deferred render on SPU – Input all GBuffers, output final composited result • Light pre-pass render on SPU – Input normal and depth only; calculate light result; sample in 2nd geometry pass • Light tile classification data output? – SPU outputs information per tile about affecting lights – Do lighting calculations on GPU
- 25.
- 26. 26 Volumetric Lighting • Alsoknown as “god rays” or “light beams” • Simulates the effect of light illuminating dust particles in the air • Numerous fakes exist – Artist-placed geometry – Artist-placed particles • Better: generate using the shadow map – Works in a “general case”
- 27. 27 Volumetric Lighting • Raymarch through the shadow map – Trace one ray per pixel in screen space – Sample the depth buffer to determine the end of the ray • Sample the shadow map at N points along the ray – N ~= 50 – Attenuate and sum up the number of samples that passed • Blur and add noise
- 28. 28 Volumetric Lighting • Effectis a bit too slow to be practical on GPU: ~5ms • Do it on SPU instead • Parallelises with GPU easily – Result needed late in the render at compositing stage – Only needs depth and shadow map inputs • Problem: must randomly sample from the shadow map
- 29. 29 Texture sampling onSPU • “Random access” texture sampling is bad for SPU • It’s bad for GPU, too, but sometimes you just have to do it • GPU: – Fast access from texture cache; cache miss is slow – Dedicated hardware handles lookups, filtering and wrapping • SPU: – Fast access from “texture cache” (SPU local memory) – Slow access on cache miss (DMA from main memory) – Cache lookups slow (no dedicated hardware) – Must manually handle filtering and wrapping (again, slow)
- 30. 30 Texture sampling onSPU • Either: – Make the texture entirely fit in SPU local memory – Problem solved! – Still inefficient: random accesses reduce register parallelism • Or – Write a very good software cache – Locate potential cache misses early - long before you need the values – Avoid branches in sampling code
- 31. 31 Volumetric Lighting onSPU • Volumetric light result will be blurred – Don’t need full shadow map accuracy – No filtering on texture samples needed • Downsample shadow map from 1024x1024, 32 bit to 256x256, 16 bit – 128k – fits in SPU local memory • Fast enough to sample on SPU
- 32. 32 Volumetric Lighting onSPU: Results • Takes ~11 ms on 1 SPU
- 33. 33 Shadow Mapping onSPU (1) • Needs the full-size shadow map – 1024x1024x32 bit == 4mb : won’t fit in SPU local memory – We’ll have to write that “very good software cache”, then • Pre-process the shadow map on SPU – Calculate min and max depth for each tile – Store in a low resolution depth hierarchy map – Output high resolution shadow map as cache tiles
- 34. 34 Shadow Mapping onSPU (2) • Software cache with 32 entries – Each entry is a shadow map tile – Branchless determination of cache entry index for tile index • Locate cache misses early – While detiling depth data – work out required shadow tiles – Pull in all cache-missed tiles • Sample shadow map during lighting calculations – All required shadow tiles are now definitely in cache – lookup is branchless • It’s quite slow – Locate tile in cache per pixel
- 35. 35 Shadow Mapping onSPU (3) • Optimise via special cases to win back performance • Use the low resolution shadow tile map – Always in SPU local memory – If pixel shadow z > tile max Z : definitely in shadow – If pixel shadow z < tile min Z : definitely not in shadow • Check low resolution map before triggering cache fetches • Classify whole screen tiles as in or out of shadow – Don’t need to sample high resolution shadow map at all for those tiles Tiles requiring high resolution shadow samples
- 36.
- 37. 37 Conclusion • New additionsto your toolbox: – Tile-based classification techniques on SPU – Deferred lighting on SPU – Texture sampling on SPU • Rendering is no longer just a GPU problem – Use general purpose nature of the SPU to your advantage • Rethink fragment processing optimisation strategies – Make the GPU work smarter, not harder
- 38. 38 Conclusion • Some titlesare already using SPU post processing – Killzone 2 • PhyreEngine™ is here to help – (If you’re a registered PS3 developer) it’s on DevNet now – Not just an engine: also a reference – Comes with full source – Download it, learn from it, steal bits of the code – Check out the PhyreEngine™ SPU Post Processing Library
Editor's Notes
- #4 The Typical performance limitation on PS3 titles has moved from a CPU bottleneck to a GPU bottleneck - specifically fragment operations.
- #5 There’s a large range of techniques that can be applied to optimise vertex performance, often focusing on only drawing what you can see, and spending the most time drawing the most important things. In addition the SPU has successfully been used to perform vertex processing.
- #6 Fragment operations are usually applied in a brute force manner. Techniques for spending the most time only where it makes the most difference – e.g. edge cases – are rarely used because of the need for branching and potentially complex pre-passes. The SPU is rarely used for pixel processing because it’s perceived as being too slow or needing too much bandwidth.
- #7 The SPU is fast enough to perform some pixel processing tasks. Bandwidth is rarely an issue – every post process we’ve ever developed so far on SPU has been cycle limited, not bandwidth limited – the PS3 bus really is that fast. The GPU is very good at specific tasks, such as rasterisation and texture sampling, but it has strict limitations. The SPU is a general purpose processor – you can read and write data in any order you like, apply branches anywhere you like, and use fast paths to optimise the process.
- #8 A brief run-through of some post processes we’ve implemented on SPU, to contrast the differing approaches used.
- #9 Use SPU to optimise GPU operations Depth buffer tile classification DXT compress render targets on SPU Use SPU to perform processes more suitable for SPU architecture Summed area table generation – much easier on SPU than GPU Deferred lighting – SPU can pick fast paths per block of pixels Use SPU to perform processes to offload from GPU Screen space ambient occlusion, volumetric lighting Operations on SPU work in parallel with GPU doing other work – minimise time on critical path
- #10 Depth of field is a very desirable but potentially slow post process. The process works by performing a blur where each weight in the kernel is scaled by a function of the difference between the kernel sample depth and pixel depth. As such the kernel can’t be separated, and the blur shader is therefore quite long and slow. This is in fact a waste of time for much of the screen. For most pixels, the depth differences are such that the weights are all 1 or 0. If we can detect areas like this we can run those areas through considerably shorter shaders and greatly reduce execution time.
- #11 We can perform this classification process on SPU. The process reads the depth buffer, processes it in tiles, classifies the results and outputs three lists of point sprites – one for each classification type. The lists are then rendered on the GPU using different shaders to perform the effect. For more detail about this process please refer to my SCEE DevStation 2008 presentation on SPU post processing.
- #12 The result is that using a version of the effect with classification techniques is massively faster than one without. The performance timings can be scaled down if the original shader used is simpler.
- #13 Soft particles: determine which particles need to be handled as “soft” by checking depth min/max tiles. Only those intersecting need to be handled as “soft”. The rest can be handled as regular particles or even avoided completely if they are totally obscured by the depth.
- #14 For SSAO we perform the entire process on the SPU. The effect is generated and output to a texture which is sampled during rendering on the GPU.
- #15 The process was able to be kicked off after the depth pre pass, and parallelised with the shadow rendering pass on the GPU. The results were then available to be read during the main render pass.
- #17 Rasterise geometry information to multiple “GBuffers” (geometry buffers) colour, normal, depth, specular and material information Apply lighting and shading as post processes Multiple lights and spatial optimizations easy Fewer shader combinations required Has some negative points GBuffers consume memory and bandwidth MSAA is problematic Fixed BRDFs
- #18 Demo showing deferred lighting on the SPU.
- #19 The SPU handles the deferred lighting process usually done on GPU. Parallelise the lighting process across multiple SPUs to improve performance. The GPU handles rasterisation processes - like rendering GBuffers, shadow maps, alpha passes, reflection geometry etc. SPU may be slower than GPU at light processing, but it’s faster than doing it all in serial on GPU. By moving such a large body of work off the GPU, we can greatly increase the overall frame rate if the GPU is the bottleneck.
- #20 To handle multiple lighting models, calculate all the different models and select based on light type. Branch per tile to optimise the set of light types used.
- #21 Comparing the light direction with tile average normal value can avoid lights behind walls and so on.
- #22 MSAA - why does this work? If there are no triangle edges or intersections, both samples rendered from the colour output will contain the same value.
- #23 The resulting performance is massively increased compared to a GPU-only equivalent. Tone mapping is virtually free on SPU – accumulation is just a sum of all pixel luminance, easily rolled into lighting calculations. The current frame’s tone mapping is applied using the eye adapted value from the previous frame. This maps much better to SPU than GPU. Why not roll in colour correction and other post processing operations – e.g. a depth of field pre-process - to the full deferred render solution?
- #24 GBuffers must be in main memory to be read by SPU – potentially requiring a lot of main memory. Find something else for the GPU to do that doesn’t depend on SPU results - alpha geometry, reflections, shadow maps, effects? Otherwise, add a frame of latency.
- #25 Different options exist depending on your limitations.
- #28 Potentially very slow. 50 texture samples times 1280x720 pixels? Downsample to ¼ width and height – result is blurred anyway. There is a demo of this effect running on GPU in the NVIDIA SDK.
- #29 Even after considerable down-sampling the effect still takes over 5ms on the GPU – too slow for our situation. So we decided to implement it on SPU instead. Unfortunately the effect requires random sampling of a shadow map texture – something which is difficult to map to the SPU.
- #30 To work out the best way to do SPU texture sampling, consider the GPU. The GPU has a texture cache which stores a small portion of the texture in fast-to-access memory, and the rest in a slower, larger memory buffer elsewhere (main memory or VRAM). On the SPU, that texture cache is the SPU’s local store. Accesses to this are fast, but if the data is not in cache it must be pulled in from main memory by DMA – which is slower. Also there is no dedicated hardware to manage the texture lookup – everything must be emulated in software.
- #31 If the texture can fit entirely in the SPU’s local store, we can avoid the whole texture cache issue. If not, we have to write a software cache that can handle it. This software cache must be branchless for lookups, otherwise performance of the calling code will be destroyed. This implies that cache misses must be caught and resolved early, so there are no DMAs in the main processing loop either.
- #32 Fortunately for volumetric lighting, the whole shadow map can be made to fit in SPU local memory by downsampling and reducing it.
- #33 The effect is fast and parallel enough to run on an SPU in the background while other work is done on the GPU.
- #34 Low resolution format: 64x64 for a 1k x 1k shadow map. Output the high resolution shadow map in a series of 16x16 tiles – they map to cache pages 1k in size.
- #36 The low resolution shadow min/max depth map can be used to greatly optimise the process by skipping high resolution shadow reads where the whole shadow tile is all in or out of shadow, and skipping shadow lookups entirely where the whole screen tile is in or out of shadow. By doing all this we can achieve good performance – good enough to make it practical to sample shadow maps on the SPU. This screen tile information can be output in a pre-process step similar to the depth of field classification and used for deferred shadowing on the GPU – only sampling the shadow map on GPU for the edge cases where the tile falls on the border of in and out of shadow. The rest of the screen can run through a fast path. This can greatly optimise the performance of deferred shadowing.
- #38 Key takeaways: Reconsider your approach to fragment processing operations. Use tile-based classification on the SPU to optimise heavy fragment processes. Move 2D fragment processing operations such as post process effects or deferred lighting to the SPU. Texture sampling on the SPU is possible too.
- #39 Much of the work you need for SPU post processing has already been done for you – download PhyreEngine and you’ll find a complete engine with full source code which implements the effects in this presentation. It also provides the necessary GPU/SPU sync framework and many useful utilities to aid post processing – such as de-tiling of main memory render targets.