Killzone Shadow Fall:
Threading the Entity
Update on PS4
Jorrit Rouwé
Lead Game Tech, Guerrilla Games
Introduction
• Killzone Shadow Fall is a First Person Shooter
• PlayStation 4 launch title
• In SP up to 60 characters @ 3...
What do we cover?
• What is an Entity?
• What we did on PS3
• Multi threading on PS4
• Balancing Entities across frames
• ...
What is an Entity?
What is an Entity?
• Base class for most game objects
• E.g. Player, Enemy, Weapon, Door
• Not used for static world
• Has...
What is a Representation?
• Entities and Components have Representation
• Controls rendering, audio and VFX
• State is int...
Movie
Multi Threading Approach
PS3: 1 Entity = 1 Fiber
• Most time spent on PPU
• No clear concurrency model
• Read partial updated state
• Entities dead...
PS4: 1 Entity = 1 Job
• No fibers
• Entity updates as a whole
• How to solve race conditions?
Time
CPU 1
CPU 2
CPU 3
Entit...
• Make update order explicit:
Strong Dependencies
Missile
MissileMissile LauncherMissile Launcher Soldier
Time
Soldier
str...
• No (indirect) dependency = no access
• Works two ways: Weapon can access Soldier too
• Create dependency has 1 frame lat...
• A few entities cause huge bottleneck
What about this?
Time
Soldier1
Pistol1
Soldier2
Pistol2
Bullet System
Soldier1 Pist...
Non-exclusive Dependencies
• Access to ‘Bullet System’ must be lock protected
Soldier1 Soldier2
Bullet System
non-exclusiv...
Weak Dependencies
• 2 tanks fire at each other:
• Update order reversed when circular dependency occurs
• Not used very of...
Non-updating Entities
• Entity can skip updates (LOD)
• Entity can update in other frame
• Do normal scheduling!
Entity1
E...
Summarizing Dependencies
Strong
Exclusive
Weak
Exclusive
Strong
Non-excl.
Weak
Non-excl.
Symbol
Two way access ✓ ✓ ✓ ✓
Ord...
Referencing Entities
• Dev build: CheckedPtr<Entity>
• Acts as normal pointer
• Check dependency on dereference
• Retail b...
Working with Entities without dependency
• ThreadSafeEntityInterface
• Mostly read only
• Often used state (name, type, po...
Scheduling Algorithm
Scheduling Algorithm
• Entities with exclusive dependencies merged to 1 job
• Dependencies determine sorting
• Non-exclusi...
Scheduling Algorithm – Edge Case
Entity1 Entity3
Entity4Entity2
non-exclusive
exclusive job dependency
Entity1 Entity2
Ent...
Scheduling Algorithm – Edge Case
• Non cyclic dependency becomes cyclic job dependency
• Job1 and Job2 need to be merged
E...
Balancing Entities Across Frames
Balancing Entities Across Frames
• Prevent all 15 Hz entities from updating in same frame
• Entity can move to other frame...
Balancing Entities – In Action
Time in even frame
(sum across cores,
running average)
Balancing Entities – In Action
Time in odd frame
(should be equal)
Balancing Entities – In Action
Civilian @ 15Hz
update even frame
Balancing Entities – In Action
Civilian @ 15Hz
update odd frame
Balancing Entities – In Action
Flash on switch odd / even
Movie
Movie
Performance Issues
Performance Issues
• Memory allocation mutex
• Eliminated many dynamic allocations
• Use stack allocator
• Locking physics...
Cut Scene - Problem
• Cut scene entity requires dependencies
• 10+ characters in cut scene creates huge job!
Civilian 1
Cu...
Cut Scene - Solution
• Create sub cut scenes for non-interacting entities
• Master cut scene determines time and flow
• Sc...
Using an Object
• Dependencies on useable objects not possible (too many)
• Get list of usable objects
• Global system pro...
Grenade
• Explosion damages many entities
• Creating dependencies not option (too many)
• ThreadSafeEntityInterface not an...
Performance Results
5000 Crates
Performance Results - Synthetic
100 Soldiers 500 Flags
Level
Counts Dependencies Max
Entities
in Job
SpeedupNu...
Performance Results - Levels
Level
Counts Dependencies Max
Entities
in Job
SpeedupNumber
Entities
Updating
Entities
Number...
Game Frame - The Patriot
Game Frame - Global
Time
18 ms
Game Frame - Global
CPU 1 (main thread)
Game Frame - Global
CPU 2
Game Frame - Global
CPU 3
Game Frame - Global
AI
Physics
Entity Update VFX & Draw
Barrier
Game Frame – Entity Update
Fix Weak Dependency Cycles
Game Frame – Entity Update
Prepare Jobs
Game Frame – Entity Update
Link and Order Jobs
Game Frame – Entity Update
Execute Jobs
Game Frame – Entity Update
Single Threaded Callbacks
Game Frame – Entity Update
Player Job
Game Frame – Entity Update
Player Entity
Game Frame – Entity Update
Animation
Game Frame – Entity Update
Capsule Collision
Game Frame – Entity Update
Player Representation
Game Frame – Entity Update
OWL (Helper Robot)
Game Frame – Entity Update
Inventory
Game Frame – Entity Update
AI Soldier
Game Frame – Entity Update
Cloth Simulation
Game Frame – Entity Update
Cut Scene
Game Frame – Entity Update
Sub Cut Scenes
Game Frame – Entity Update
Cheap Entities (Destructibles)
Debug Tools
Debug Tools
• Dependencies get complex, we use yEd to visualize!
Debug Tools
Player
Debug Tools
Bullet System
Debug ToolsCut scenes
Conclusions
• Easy to implement in existing engine
• Game programmers can program as if single threaded
• Very few multith...
Questions?
jorrit@guerrilla-games.com
Upcoming SlideShare
Loading in...5
×

Killzone Shadow Fall: Threading the Entity Update on PS4

6,912

Published on

On the PS3 the SPUs were too limited for doing typical entity logic (e.g. humanoids walking around and shooting), so all of our entity logic was running on a single thread. For the PS4 we adopted the "entity as a job" approach and developed a set of rules that entities have to adhere to. We also developed a way to validate these rules at runtime so that no race conditions occur. This presentation will also show how we spread out entity updates across frames. The resulting system has proven itself on the PS4 -- and there have been surprisingly few race conditions that were not easily caught.

Published in: Engineering
0 Comments
14 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
6,912
On Slideshare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
54
Comments
0
Likes
14
Embeds 0
No embeds

No notes for slide

Killzone Shadow Fall: Threading the Entity Update on PS4

  1. 1. Killzone Shadow Fall: Threading the Entity Update on PS4 Jorrit Rouwé Lead Game Tech, Guerrilla Games
  2. 2. Introduction • Killzone Shadow Fall is a First Person Shooter • PlayStation 4 launch title • In SP up to 60 characters @ 30 FPS • In MP up to 24 players @ 60 FPS • Gameplay logic has lots of • Branches • Virtual functions • Cache misses • Not suitable for PS3 SPU’s but PS4 has 6 x86 cores
  3. 3. What do we cover? • What is an Entity? • What we did on PS3 • Multi threading on PS4 • Balancing Entities across frames • Performance issues found • Performance results achieved • Debug tools
  4. 4. What is an Entity?
  5. 5. What is an Entity? • Base class for most game objects • E.g. Player, Enemy, Weapon, Door • Not used for static world • Has Components • E.g. Model, Mover, Destructibility • Update at a fixed frequency • 15, 30, 60 Hz • Almost everything at 15 Hz • Player updated at higher frequency to avoid lag
  6. 6. What is a Representation? • Entities and Components have Representation • Controls rendering, audio and VFX • State is interpolated in frames where entity not updated • Cheaper to interpolate than to update • Introduces latency • Always blend towards last update
  7. 7. Movie
  8. 8. Multi Threading Approach
  9. 9. PS3: 1 Entity = 1 Fiber • Most time spent on PPU • No clear concurrency model • Read partial updated state • Entities deadlock waiting for each other yield fiber resume fiber Time PPU SPU 1 SPU 2 Entity 1 Entity 2 Entity 3 Entity 1 Animation Ray Cast Entity 2
  10. 10. PS4: 1 Entity = 1 Job • No fibers • Entity updates as a whole • How to solve race conditions? Time CPU 1 CPU 2 CPU 3 Entity 1 Entity 2 Entity 3 Entity 1Animation Ray Cast Entity 2  job
  11. 11. • Make update order explicit: Strong Dependencies Missile MissileMissile LauncherMissile Launcher Soldier Time Soldier strong dependency
  12. 12. • No (indirect) dependency = no access • Works two ways: Weapon can access Soldier too • Create dependency has 1 frame latency • Global systems need locks Non-dependent Entities can Execute Concurrently Soldier1 Weapon1 Time Soldier2 Weapon2 CPU 2 CPU 1 Soldier1 Weapon1 Soldier2 Weapon2
  13. 13. • A few entities cause huge bottleneck What about this? Time Soldier1 Pistol1 Soldier2 Pistol2 Bullet System Soldier1 Pistol1 Soldier2 Pistol2 Bullet System
  14. 14. Non-exclusive Dependencies • Access to ‘Bullet System’ must be lock protected Soldier1 Soldier2 Bullet System non-exclusive Time CPU 2 CPU 1 Soldier1 Pistol1 Soldier2 Pistol2 Bullet System Barrier Pistol1 Pistol2
  15. 15. Weak Dependencies • 2 tanks fire at each other: • Update order reversed when circular dependency occurs • Not used very often (< 10 per frame) Time Missile2Missile1 Tank1 Tank2 Missile2Missile1 Tank2 Tank1 Missile1Missile2 or weak strong Tank1 Tank2
  16. 16. Non-updating Entities • Entity can skip updates (LOD) • Entity can update in other frame • Do normal scheduling! Entity1 Entity3 Entity1 Entity2 Entity3 Time not updating
  17. 17. Summarizing Dependencies Strong Exclusive Weak Exclusive Strong Non-excl. Weak Non-excl. Symbol Two way access ✓ ✓ ✓ ✓ Order guaranteed ✓ ✓ Allow concurrency + + ++ ++ Require lock ✓ ✓
  18. 18. Referencing Entities • Dev build: CheckedPtr<Entity> • Acts as normal pointer • Check dependency on dereference • Retail build: Entity * • No overhead! • Doesn’t catch everything • Can use pointers to members • Bugs were easy to find
  19. 19. Working with Entities without dependency • ThreadSafeEntityInterface • Mostly read only • Often used state (name, type, position, …) • Mutex per Entity • Send message (expensive) • Processed single threaded when no dependency • Schedule single threaded callback (expensive) • Everything can be accessed
  20. 20. Scheduling Algorithm
  21. 21. Scheduling Algorithm • Entities with exclusive dependencies merged to 1 job • Dependencies determine sorting • Non-exclusive dependencies become job dependencies • Expensive jobs kicked first! Missile Missile Launcher Tank Bullet System Pistol Soldier Missile Tank Bullet System PistolSoldier Job1 Job2 job dependency not updating weak non-exclusive strong
  22. 22. Scheduling Algorithm – Edge Case Entity1 Entity3 Entity4Entity2 non-exclusive exclusive job dependency Entity1 Entity2 Entity3 Entity4 Job1 Job2
  23. 23. Scheduling Algorithm – Edge Case • Non cyclic dependency becomes cyclic job dependency • Job1 and Job2 need to be merged Entity1 Entity3 Entity4Entity2 non-exclusive exclusive Entity1 Entity2Entity3 Entity4 Job1
  24. 24. Balancing Entities Across Frames
  25. 25. Balancing Entities Across Frames • Prevent all 15 Hz entities from updating in same frame • Entity can move to other frame • Smaller delta time for 1 update • Keep parent-child linked entities together • Weapon of soldier • Soldier on mounted gun • Locked close combat
  26. 26. Balancing Entities – In Action Time in even frame (sum across cores, running average)
  27. 27. Balancing Entities – In Action Time in odd frame (should be equal)
  28. 28. Balancing Entities – In Action Civilian @ 15Hz update even frame
  29. 29. Balancing Entities – In Action Civilian @ 15Hz update odd frame
  30. 30. Balancing Entities – In Action Flash on switch odd / even
  31. 31. Movie
  32. 32. Movie
  33. 33. Performance Issues
  34. 34. Performance Issues • Memory allocation mutex • Eliminated many dynamic allocations • Use stack allocator • Locking physics world • R/W mutex for main simulation world • Second ‘bullet collision’ broadphase + lock • Large groups of dependent entities • Player update very expensive
  35. 35. Cut Scene - Problem • Cut scene entity requires dependencies • 10+ characters in cut scene creates huge job! Civilian 1 Cut Scene Civilian 2 Player Camera
  36. 36. Cut Scene - Solution • Create sub cut scenes for non-interacting entities • Master cut scene determines time and flow • Scan 1 frame ahead in timeline to create dependency Civilian 1 Cut Scenenon-exclusive Sub Cut Scene 1 Sub Cut Scene 2 Civilian 2 Sub Cut Scene 3 Player Camera
  37. 37. Using an Object • Dependencies on useable objects not possible (too many) • Get list of usable objects • Global system protected by lock • ‘Use’ icon appears on screen • Player selects • Create dependency • Start ‘use’ animation • Start interaction 1 frame later (dependency valid) • Hides 1 frame delay!
  38. 38. Grenade • Explosion damages many entities • Creating dependencies not option (too many) • ThreadSafeEntityInterface not an option • Need knowledge of parts • Run line of sight checks inside update • Uses scheduled callback to apply damage
  39. 39. Performance Results
  40. 40. 5000 Crates Performance Results - Synthetic 100 Soldiers 500 Flags Level Counts Dependencies Max Entities in Job SpeedupNumber Entities Updating Entities Number Humans Strong Excl Strong Non-Excl 5000 Crates (20 µs each) 5019 5008 1 12 4 13 2.8X 100 Soldiers (700 µs each) 326 214 105 212 204 19 4.2X 500 Flags (160 µs each) 519 508 1 12 4 13 5.2X 6 cores!
  41. 41. Performance Results - Levels Level Counts Dependencies Max Entities in Job SpeedupNumber Entities Updating Entities Number Humans Strong Excl Strong Non-Excl The Helghast (You Owe Me) 1141 206 32 71 23 20 4.1X The Patriot (On Vectan Soil) 435 257 44 199 107 15 4.3X The Remains (12p Botzone) 450 128 14 97 44 18 3.7X The Helghast The RemainsThe Patriot
  42. 42. Game Frame - The Patriot
  43. 43. Game Frame - Global Time 18 ms
  44. 44. Game Frame - Global CPU 1 (main thread)
  45. 45. Game Frame - Global CPU 2
  46. 46. Game Frame - Global CPU 3
  47. 47. Game Frame - Global AI Physics Entity Update VFX & Draw Barrier
  48. 48. Game Frame – Entity Update Fix Weak Dependency Cycles
  49. 49. Game Frame – Entity Update Prepare Jobs
  50. 50. Game Frame – Entity Update Link and Order Jobs
  51. 51. Game Frame – Entity Update Execute Jobs
  52. 52. Game Frame – Entity Update Single Threaded Callbacks
  53. 53. Game Frame – Entity Update Player Job
  54. 54. Game Frame – Entity Update Player Entity
  55. 55. Game Frame – Entity Update Animation
  56. 56. Game Frame – Entity Update Capsule Collision
  57. 57. Game Frame – Entity Update Player Representation
  58. 58. Game Frame – Entity Update OWL (Helper Robot)
  59. 59. Game Frame – Entity Update Inventory
  60. 60. Game Frame – Entity Update AI Soldier
  61. 61. Game Frame – Entity Update Cloth Simulation
  62. 62. Game Frame – Entity Update Cut Scene
  63. 63. Game Frame – Entity Update Sub Cut Scenes
  64. 64. Game Frame – Entity Update Cheap Entities (Destructibles)
  65. 65. Debug Tools
  66. 66. Debug Tools • Dependencies get complex, we use yEd to visualize!
  67. 67. Debug Tools Player
  68. 68. Debug Tools Bullet System
  69. 69. Debug ToolsCut scenes
  70. 70. Conclusions • Easy to implement in existing engine • Game programmers can program as if single threaded • Very few multithreading issues
  71. 71. Questions? jorrit@guerrilla-games.com
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×