Real-Time Stats for Candy Box


Published on

Blog with clickable links:

We like games here at PubNub, but not as much as we like real-time. Combine the two, and you’ve got pure mega-awesome. During the PubNub Hackathon, I took a popular text adventure game called Candy Box, and updated its stats page to provide a real-time overview of the global game statistics.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Real-Time Stats for Candy Box

  1. 1. Real-TimeStats forCandy Box!by Jay Osterwww. .com
  2. 2. We like games here at PubNub, but not as much as we like real-time. Combine the two, and you’ve got pure mega-awesome.During the PubNub Hackathon, I took a popular text adventuregame called Candy Box, and updated its stats page to provide areal-time overview of the global game statistics. The updatedgame can be played at, and the new real-time stats page is here. Full source code is available on github.In this article, we’ll guide you through how the game wasmodified, and how to build a very simple, yet hyper-scalableserver infrastructure to serve real-time statistics. Today we’reusing JavaScript on the client-side, and Python on the server-side. So let’s dig in!Since Candy Box is a very new game, there isn’t a public sourcecode repository available. So I started this project by mirroringthe original website,, with wget. It’s alsopossible to load the URL into a browser and use File->Save PageAs… to get a complete local copy of the game.www. .com
  3. 3. Download the Game CodeWith all of the JavaScriptdownloaded, I had to makesome minor adjustments insome places to get a few thingsworking correctly. Loading andsaving had to be modified;moving some of the originalserver-side logic right intoJavaScript. I’ll spare the details,and the curious among you canhave a look at load.js for someinsight. To allow saving, the statsserver just proxies save requeststo the original server. The proxyis required because the originalserver does not support CORS,meaning newer browsers willreject the save attempt.www. .com
  4. 4. Building the Stats ServerWith the baseline game ready, it’s time to start thinking about the server-side, particularly how to handle the highload that such a popular game will generate. High load is the reason the original stats page is updated onlyinfrequently. The challenge is to record and report game statistics for thousands of simultaneous players in real-time, with very frequent updates (refreshing stats about once every second). With this in mind, I came up with thefollowing list of requirements while designing the server:•  In-memory statistics aggregation•  Horizontally scalable•  Stateful-design, but distributedSounds simple, right? Do calculations in memory to make it fast, and scale horizontally to handle any load. Apanacea! But how can we sum and average hundreds of thousands of accounts per second? Well, we can cheat abit by doing the summing on the client-side, and only keep the result of the sums on the server. In other words,client logic will be responsible for calculating the deltas between each status update, so the server just needs toworry about adding those deltas into its internal state. Therefore, each status update is a snapshot in time, and theserver records only the sum over all snapshots.To share state between servers, we’ll use the same idea; deltas between updates, and periodically publish to acommon PubNub channel to distribute changes to the internal state. This method of replication introduces its ownchallenges, such as update latencies with multiple servers. This is acceptable because the statistics in this gameare quite volatile anyway. No one will notice any latencies.www. .com
  5. 5. Server Implementation DetailsFor the server, I decided to use Bottle to handle the RESTinterface, and gevent for non-blocking sockets. This willgive us a great deal of flexibility for the server.After writing a few stubs for the REST interface, I startedon the distribution mechanism, which is just a PubNubsubscribe that handles messages from other servers.Ideally, each server will periodically share the updates theyhave received from clients. The server only needs to adddeltas from the user to its own internal delta fordistribution to other servers. It can’t distribute every clientrequest, because traffic would be much too high. And itwould defeat the purpose of horizontally scaling, anyway.This is also where distribution latency comes in; a deltafrom a client may take a few seconds to reach everyserver.www. .com
  6. 6. www. .comIntegrating PubNubgevent makes the subscribe super easy; just the normal Pubnub.subscribe(), wrapped in a call to gevent.spawn(). Thereare initially two such subscribes: a “control” subscription, which will recursively re-subscribe after handling eachmessage, and a “sync” subscription to initially synchronize with other servers.The “control” subscription does three things:1.  Respond to “sync” requests; providing the current game state to other servers2.  Update config; allowing you to remotely reconfigure all servers3.  Update game state; receive deltas from other serversI’ll explain more about the config updating later. After Both subscriptions are established, a “sync” request ispublished to any listening servers. The first “sync” response that comes within 5 minutes will cause the game stateto be updated with the provided values.The server then goes to sleep until a user makes an “update” request. That will start a recursive timer (the interval isconfigurable) which will send stat updates to clients, and stat delta updates to other servers. The timer recursionautomatically stops 5 minutes after the last user “update” request.And that’s about all there really is to the server. You could also do some other fancy things like persisting the stateto disk periodically. It isn’t a lot of data to store, but these stats are also not critical, especially for the demo.
  7. 7. Running the Stats ServerStarting the server is easy as well, just start the main script with python, or run it directly in a shell (it has execpermissions and a shebang). The script accepts three optional arguments for listening IP, listening port, and configfile. The config file is just JSON, in the same format as in config.pyHere’s an example:$ python 8999 ~/config.json www. .com
  8. 8. Hacking the ClientBack on the client-side, we just need a function to record the deltas, and another to send the “update” requests tothe server. I decided to use the localStorage API to record the update state between each request, allowing thedeltas to be calculated correctly even after restarting the browser.As far as security goes, I will be ignoring the possibility of cheaters for the demo. Stats can also be skewed bysaves that have completed the game, because SPOILER ALERT the computer tab grants access to generatingcandies and lollipops at an impossible rate, and changing pretty much every variable in strange ways. SPOILERALERTClient requirements are as follows:•  Turn a blind eye to cheaters (simplifies everything)•  Periodically send “update” requests (once every 5 seconds is a good start)•  Do not send “update” requests after the game has been completedThe update interval will be once every 5 seconds by default, which will be quick enough to affect the stats updatesthat users end up seeing, and slow enough to handle a large number of simultaneous players with low serverresources; With 2,000 users, the server only needs to handle 400 requests per second. The gevent-based server willeasily handle that without a hiccup, even on commodity hardware. In fact, each server should handle about 1,000concurrent connections. If more than 5,000 users are playing, just launch another stats server and put it behindnginx (reverse proxy) as a load balancer. More on that later.www. .com
  9. 9. // Save to PubNub CandyBox stats server periodically if ((this.nbrOfSecondsSinceLastMinInterval % 5) === 0) { stats.update(); } The Hookmain.js is where the game loop runs. It’s implemented as a simple interval that fires once per second. This is the placeto add the stats updates. The code is very simple; just throttle a function call to once every 5 seconds:The stats.update() function is where the magic happens. It records the interesting bits of game state, calculates thedelta, and sends the request to the stat server.www. .com
  10. 10. $.each(currentUpdate, function (k, v) { if (typeof(v) !== "string") delta[k] = lastUpdate[k] - v; }); Delta CalculationThe delta calculation is very easy (as you might imagine). I just keep a record of the last game stat after a successful“update” request (and save this object to localStorage), and the delta is calculated with a small iterator:Should be self-explanatory, but basically the difference between values in lastUpdate and values in currentUpdate arerecorded as the delta, with a safety net for the code key (not shown) which is a string value. The delta is then sent to thestats server in an “update” request.The server does its work, and periodically publishes a message for the stats page. The listener code is in stats.js andyou can see it does the percentage calculation client-side. It is otherwise incredibly basic.www. .com
  11. 11. upstream stats_server { server localhost:8999; #server localhost:8998; #server localhost:8997; #server localhost:8996; keepalive 32; } server { listen 80; root /home/ubuntu/pn-candybox/public; index index.html; server_name; location /ping { proxy_pass http://stats_server; } Server ConfigurationWith the client and server ready to go, it’s time to start thinking about the operational side of the project; configuringservers, DNS, an even dynamically scaling and remote-control reconfiguration.I’m using nginx as a host for the client code and it also doubles as a front-end load balancer for the stats server. Thenginx config looks like this:location /save { proxy_pass http://stats_server; } location /update { proxy_pass http://stats_server; } location / { try_files $uri $uri/ /index.html; } } www. .com
  12. 12. I did some load testing with ApacheBench and found that nginx with a single stats server can handle about 763requests per second with 100 concurrent connections, or about 305 requests per second with 200 concurrent. Alltests were done on a t1.micro AWS instance (E5507 @ 2.27GHz, 589MB RAM) running Ubuntu 13.04 with no TCPkernel tuning. This setup is good enough for our “2,000 simultaneous players” requirement.Dynamically ScalingWith the server config in place, we can easily scale up by adding more upstream stats servers (commented in theconfig above). Then reloading nginx. The stats servers will automatically synchronize with one another over PubNub.We can also reconfigure the servers at runtime to tune the message publishing rates. I just have to open thePubNub console and publish a specially constructed message to the “candybox_update” channel. Here’s anexample message that reconfigures the servers to publish only once every 5 seconds:Publish that message, and all servers will instantly adjust their message publishing interval to 5 seconds. This is justone example of what makes PubNub truly awesome. { "uuid" : "master", "action" : "config", "data" : { "update_interval" : 5 } } www. .com
  13. 13. Wrapping UpWith all of that, we now have Candy Box sendingperiodic updates to our stats server, and our statsservers periodically sending updates to the stats page.And it’s all done in a dynamically scalable way, with aridiculously small memory footprint, and low bandwidthrequirements.All done! Now you should play the game,check the stats, and fork me!www. .com