Stackless Python In Eve

Slide 1: Stackless Python in EVE Kristján Valur Jónsson kristjan@ccpgames.com CCP Games inc.

Slide 2: EVE • MMORPG Space game • Client / server • Single shard massive server • 120.000 active players, >24.000 concurrent users • World concurrency record on a shard • Relies on Stackless Python

Slide 3: The Tranquility cluster • 400 GHz CPU / 200 Gb RAM • 2 Routers (CISCO Alteon) • 14 Proxy servers (IBM Blade) • 55 Sol servers (IBM x335) • 2 DB servers (clustered, IBM Brick x445) • FastT600 Fiber, 56 x FC 15k disks, DS4300 + 3*EXP700 • Windows 2000, MS SQL Server • Currently being upgraded • AMD x64

Slide 4: EVE Architecture • COM-like basic architecture • Python tighly integrated at an early stage • Home-grown wrapping of BLUE objects blue modules python modules blue.dll Pythonlib.lib ExeFile.exe autoexec.py Netclient.dll foo.py Trinity.dll bar.py ... ...

Slide 5: Stackless Python • Tasklets • Threads of execution. Not OS threads • Lightweight • No pre-emption • Channels • Tasklet rendezvous point • Data passing • Scheduling • Synchronization

Slide 6: Stackless? • No C stack • Python stack in linked main() frame objects • Tasklet switching by PyStackless _CallMethod _Main(“MyMain” ) swapping frame chain def MyMain (self): cmodule .Frobnicate (self.Callback ) • Compromise PyFrobnicate () { • stackless where PyObject _CallObject (cb, …); } possible. def Callback (self): self.channel.send(42) • C stack whisked away Stack switch if necessary

Slide 7: Channels Readers Writers Tasklet 2 Tasklet 1 Channel

Slide 8: Channel semantics • Send on a channel with no receiver blocks tasklet. • Send on a channel with a (blocked) receiver, suspends tasklet and runs receiver immediately. Sender runs again in due course. • Symmetric wrt. Send and Receive. • “balance”, can have a queue of readers or writers. • Conceptually similar to Unix pipes

Slide 9: Channel semantics, cont. • Scheduling semantics are precise: • A blocked tasklet is run immediately • Usable as a building block: • semaphores • mutex • critical section • condition variables

Slide 10: Stackless in EVE • BLUE foundation: robust, but cumbersome • RAD • Stackless Python: Python and so much more • EVE is inconceivable without Stackless • Everyone is a programmer

Slide 11: The main loop • Establish stackless context int WinMain(...) { PyObject *myApp = new EveApp(); PyObject *r = PyStackless_CallMethod_Main(MyApp, “WinMain”, 0); return PyInt_AsLong( r );

Slide 12: The main loop cont. PyObject* EveApp::WinMain(PyObject *self, PyObject *args) { PyOS->ExecFile(\"script:/sys/autoexec.py\"); MSG msg; while(PeekMessage(&msg, 0, 0, 0, PM_REMOVE)){ TranslateMessage(&msg); DispatchMessage(&msg); } for (TickIt i = mTickers.begin(; i != mTickers.end(); i++) i->mCb->OnTick(mTime, (void*)taskname); } • Regular Windows message loop • Runs in Stackless context • The “Main Tasklet”

Slide 13: Autoexec.py import blue def Startup(): import service srvMng = service.ServiceManager() run = [\"dataconfig\", \"godma\", “ui\", …] srvMng.Run(run) #Start up the client in a tasklet! if CheckDXVersion(): import blue blue.pyos.CreateTasklet(Startup, (), {})

Slide 14: Tickers • Tickers are BLUE modules: • Trinity (the renderer) • Netclient • DB (on the server) • Audio • PyOS (special python services) • …

Slide 15: The PyOS tick: • Runs fresh tasklets • (sleepers awoken elsewhere) Tick() { … mSynchro->Tick() PyObject *watchdogResult; do { watchdogResult = PyStackless_RunWatchdog(20000000); if (!watchdogResult) PyFlushError(\"PumpPython::Watchdog\"); Py_XDECREF(watchdogresult); } while (!watchdogResult);

Slide 16: blue.pyos.synchro • Synchro: • Provides Thread-like tasklet utilities: • Sleep(ms) • Yield() • BeNice()

Slide 17: blue.pyos.synchro cont. 1. Sleep: A python script makes the call blue.pyos.Sleep(200) 2. C++ code runs: 1. Main tasklet check 2. sleeper = New Sleeper(); mSleepers.insert(sleeper); PyObject *r = PyChannel_Receive(sleeper- >mChannel); 3. Another tasklet runs

Slide 18: blue.pyos.synchro, ctd. 1. Main tasklet in windows loop enters PyOS::Tick() 2. mSleepers are examined for all that are due we do: mSleepers.remove(sleeper); PyChannel_Send(sleepers.mChannel, Py_NONE); 3. Main tasklet is suspended (but runnable), sleeper runs.

Slide 19: Points to note: • A tasklet goes to sleep by calling PyChannel_Receive() on a channel which has no pending sender. • It will sleep there (block) until someone sends • Typically the main tasklet does this, doing PyChannel_Send() on a channel with a reader • Ergo: The main tasklet may not block

Slide 20: Socket Receive • Use Windows asynchronous file API • Provide a synchronous python API. A python script calls Read(). • Tasklet may be blocked for a long time, (many frames) other tasklets continue running. • Do this using channels.

Slide 21: Receive, cont. 1. Python script runs: foo, bar = socket.Read() 2. C code executes the request: Request *r = new Request(this); WSAReceive(mSocket, …); mServe->insert( r ); PyChannel_Receive(r->mChannel); 3. Tasklet is suspended

Slide 22: Receive, cont. 1. Socket server is ticked from main loop 2. For all requests that are marked completed, it transfers the data to the sleeping tasklets: PyObject *r = PyString_FromStringAndSize(req->mData, req->mDataLen); PyChannel_Send(req->mChannel, r); Py_DECREF(data); delete req; 4. The sleeping tasklet wakes up, main tasklet is suspended (but runnable)

Slide 23: Receive completed def TaskletMain (): def ProcessStuff : Cmodule ::Work(PyObject *cb) EveApp ::WinMain () def Callback (): Netclient ::OnTick () Socket ::Rece ive(PyObject *cb) PyChannel _Send (mReceiver , data); d = PyChannel _Receive (mReceiver ); Stack switch Stack switch

Slide 24: Main Tasklet • The one running the windows loop • Can be suspended, allowing other tasklets to run • Can be blocked, as long as there is another tasklet to unblock it (dangerous) • Is responsible for waking up Sleepers, Yielders, IO tasklets, etc. therefore cannot be one of them • Is flagged as non-blockable (stackless.get_current().block_trap = True)

Slide 25: Channel magic • Channels perform the stackless context switch. • If there is a C stack in the call chain, it will magically swap the stacks. • Your entire C stack (with C and python invocations) is whisked away and stored, to be replaced with a new one. • This allows stackless to simulate cooperative multi- threading

Slide 26: Co-operative multitasking • Context is switched only at known points. • In Stakcless, this is channel.send() and channel.receive() • Also synchro.Yield(), synchro.Sleep(), BeNice(), socket and DB ops, etc. • No unexpected context switches • Almost no race conditions • Program like you are single-threaded • Very few exceptions. • This extends to C state too!

Slide 27: Tasklets • Tasklets are cheap • Used liberally to reduce perceived lag • UI events forked out to tasklets • A click can have heavy consequences. • Heavy logic • DB Access • Networks access • special rendering tasks forked out to tasklets. • controlling an audio track • “tasklet it out” • Use blue.pyos.synchro.BeNice() in large loops

Slide 28: Example: UI Event: • Main tasklet receives window messages such as WM_CLICK • Trinity invokes handler on UI elements or global handler • Handler “tasklets out” any action to allow main thread to continue immediately. def OnGlobalUp(self, *args): if not self or self.destroyed: return mo = eve.triapp.uilib.mouseOver if mo in self.children: uthread.new(mo._OnClick) class Action(xtriui.QuickDeco): def _OnClick(self, *args): pass

Slide 29: That’s all • For more info: • http://www.ccpgames.com • http://www.eve-online.com • http://www.stackless.com • kristjan@ccpgames.com