THQ/Gas Powered Games Supreme Commander and Supreme Commander: Forged Alliance Thread for Performance

Supreme Commander runs best on 4 cores - let’s see how! Threading in mid-project can be done! Decoupled threads give great performance Memory management extends the gains Lessons learned

Threading in mid-project can be done!

Decoupled threads give great performance

Memory management extends the gains

Lessons learned

Threading was a mid-stream change Code was initially single-threaded Game demanded more performance Changed mid-project (6-12 months into development) Separate render/sim threads to run at different rates Support multiple cores Limited architecture choices due to existing code Using Boost thread library Portable, open-source thread library

Code was initially single-threaded

Game demanded more performance

Changed mid-project (6-12 months into development)

Separate render/sim threads to run at different rates

Support multiple cores

Limited architecture choices due to existing code

Using Boost thread library

Portable, open-source thread library

Render split is essential to speed Lots of “little” threads: sound, loading, etc. Sim thread: All simulation Render thread: Full speed, <=10x per sim tick Sync phase: Once frame is ready to render Sync render and sim Fully queued in and out of sim Fast

Lots of “little” threads: sound, loading, etc.

Sim thread: All simulation

Render thread: Full speed, <=10x per sim tick

Sync phase: Once frame is ready to render

Sync render and sim

Fully queued in and out of sim

Fast

Decoupled architecture is built for speed Ready to start a frame and a simulation tick Issue

Ready to start a frame and a simulation tick

Decoupled architecture is built for speed Simulation Render Issue Run decoupled sim and render Fully buffered input to sim, call via Sim Thread Interface Sim Thread Interface

Decoupled architecture is built for speed Render can run repeatedly Depends on sim duration Simulation Render Issue Up to 10x per sim tick Render …

Render can run repeatedly

Depends on sim duration

Decoupled architecture is built for speed Fully decoupled? No. A few low level systems have locks. No major performance impact! Simulation Render Issue Up to 10x per sim tick Render … Locks

Fully decoupled? No.

A few low level systems have locks.

No major performance impact!

Decoupled architecture is built for speed Sync sim thread out to render thread, via STI again Simulation Render Issue Up to 10x per sim tick Render Render … Sim Thread Interface Issue

Sync sim thread out to render thread,

via STI again

Decoupled architecture is built for speed Multiplayer: Record everything going through STI Send over network Simulation Render Issue Up to 10x per sim tick Render Render … Sim Thread Interface Issue Sim Thread Interface

Multiplayer: Record everything going through STI

Send over network

Decoupled architecture is built for speed And so on… Simulation Render Issue Up to 10x per sim tick Render Sim Render Render Issue … … Issue

And so on…

Thread model adapts to varying loads Architecture scales well with loads Render load will often dominate Re-render to keep frame rates up Sim-heavy map will try to be sim-dominated

Architecture scales well with loads

Render load will often dominate

Re-render to keep frame rates up

Sim-heavy map will try to be sim-dominated

Displaying frame times – cool! Thread stats in real time

Thread stats in real time

Sometimes, there’s more to render Render Runs as fast as possible Simulation Sim/render sync Both threads synced, fully queued in and out of sim

Render

Runs as fast as possible

Simulation

Sim/render sync

Both threads synced, fully queued in and out of sim

Other times, there’s more to simulate Sim runs across many rendered frames

Sim runs across many rendered frames

A little sync doesn’t slow this code down Frame n Frame n+1 Sync Busy Waiting Threads are busy most of the time! Mostly waiting

Memory manager gives an additional boost Memory: If you’re not careful in a threaded game… Memory use can thrash cache – but not a problem here! Memory alloc/free can be slow Suspected memory management was problem Doing lots of small allocations Built code to make it easy to switch mem managers Custom mem manager outperforms default malloc/free Can cause some debugging questions Purchased commercial one for Supreme Commander Wrote new one for Forged Alliance

Memory: If you’re not careful in a threaded game…

Memory use can thrash cache – but not a problem here!

Memory alloc/free can be slow

Suspected memory management was problem

Doing lots of small allocations

Built code to make it easy to switch mem managers

Custom mem manager outperforms default malloc/free

Can cause some debugging questions

Purchased commercial one for Supreme Commander

Wrote new one for Forged Alliance

What are some current bottlenecks? Multiplayer: all sims run concurrently Limited by least-common-denominator machine That’s the RTS way Monolithic render thread Multiple monitors, typically different views Possibly split off top part of render for second monitor? Too expensive/complex for niche feature

Multiplayer: all sims run concurrently

Limited by least-common-denominator machine

That’s the RTS way

Monolithic render thread

Multiple monitors, typically different views

Possibly split off top part of render for second monitor?

Too expensive/complex for niche feature

This was a great learning experience! Good intermediate step Especially for threading mid-project Would do it differently if doing it from scratch Target more processor cores General worker threads w/dispatch system Templates to define an interface to common semantics Directed work graph/node graph (hard to express) Or …? The engine is so good, it’ll be back in Demigod! Demigod team using modified Supreme Commander engine

Good intermediate step

Especially for threading mid-project

Would do it differently if doing it from scratch

Target more processor cores

General worker threads w/dispatch system

Templates to define an interface to common semantics

Directed work graph/node graph (hard to express)

Or …?

The engine is so good, it’ll be back in Demigod!

Demigod team using modified Supreme Commander engine

We learned some DOs and DON’Ts Do: Architect for threading from the start, if you can Thread single-threaded code, if you must Decouple threads where possible Don’t: Be afraid to thread single-threaded code

Do:

Architect for threading from the start, if you can

Thread single-threaded code, if you must

Decouple threads where possible

Don’t:

Be afraid to thread single-threaded code

Supreme Commander runs best on 4 cores – that’s how! Threading in mid-project can be done! Decoupled threads give great performance Memory management extends the gains Lessons learned

Threading in mid-project can be done!

Decoupled threads give great performance

Memory management extends the gains

Lessons learned

So, what do you think? Have you tried something like this? Successes? Failures? Have you rejected trying something like this? Why?

Have you tried something like this?

Successes?

Failures?

Have you rejected trying something like this?

Why?