This document describes how the author modified the Candy Box game to provide real-time global game statistics. The game was mirrored locally. The client was modified to periodically send statistic updates to a Python server using Bottle and gevent. The server aggregates statistics in memory and shares updates with other servers over PubNub. The servers are horizontally scalable and can be reconfigured in real-time via PubNub messages. Real-time global statistics are now viewable to players.

1. Real-Time
Stats for
Candy Box!
by Jay Oster
www. .com

2. We like games here at PubNub, but not as much as we like real-
time. Combine the two, and you’ve got pure mega-awesome.
During the PubNub Hackathon, I took a popular text adventure
game called Candy Box, and updated its stats page to provide a
real-time overview of the global game statistics. The updated
game can be played at candybox.pubnub.co, and the new real-
time stats page is here. Full source code is available on github.
In this article, we’ll guide you through how the game was
modified, and how to build a very simple, yet hyper-scalable
server infrastructure to serve real-time statistics. Today we’re
using JavaScript on the client-side, and Python on the server-
side. So let’s dig in!
Since Candy Box is a very new game, there isn’t a public source
code repository available. So I started this project by mirroring
the original website, candies.aniwey.net, with wget. It’s also
possible to load the URL into a browser and use File->Save Page
As… to get a complete local copy of the game.
www. .com

3. Download the Game Code
With all of the JavaScript
downloaded, I had to make
some minor adjustments in
some places to get a few things
working correctly. Loading and
saving had to be modified;
moving some of the original
server-side logic right into
JavaScript. I’ll spare the details,
and the curious among you can
have a look at load.js for some
insight. To allow saving, the stats
server just proxies save requests
to the original server. The proxy
is required because the original
server does not support CORS,
meaning newer browsers will
reject the save attempt.
www. .com

4. Building the Stats Server
With the baseline game ready, it’s time to start thinking about the server-side, particularly how to handle the high
load that such a popular game will generate. High load is the reason the original stats page is updated only
infrequently. The challenge is to record and report game statistics for thousands of simultaneous players in real-
time, with very frequent updates (refreshing stats about once every second). With this in mind, I came up with the
following list of requirements while designing the server:
• In-memory statistics aggregation
• Horizontally scalable
• Stateful-design, but distributed
Sounds simple, right? Do calculations in memory to make it fast, and scale horizontally to handle any load. A
panacea! But how can we sum and average hundreds of thousands of accounts per second? Well, we can cheat a
bit by doing the summing on the client-side, and only keep the result of the sums on the server. In other words,
client logic will be responsible for calculating the deltas between each status update, so the server just needs to
worry about adding those deltas into its internal state. Therefore, each status update is a snapshot in time, and the
server records only the sum over all snapshots.
To share state between servers, we’ll use the same idea; deltas between updates, and periodically publish to a
common PubNub channel to distribute changes to the internal state. This method of replication introduces its own
challenges, such as update latencies with multiple servers. This is acceptable because the statistics in this game
are quite volatile anyway. No one will notice any latencies.
www. .com

5. Server Implementation Details
For the server, I decided to use Bottle to handle the REST
interface, and gevent for non-blocking sockets. This will
give us a great deal of flexibility for the server.
After writing a few stubs for the REST interface, I started
on the distribution mechanism, which is just a PubNub
subscribe that handles messages from other servers.
Ideally, each server will periodically share the updates they
have received from clients. The server only needs to add
deltas from the user to its own internal delta for
distribution to other servers. It can’t distribute every client
request, because traffic would be much too high. And it
would defeat the purpose of horizontally scaling, anyway.
This is also where distribution latency comes in; a delta
from a client may take a few seconds to reach every
server.
www. .com

6. www. .com
Integrating PubNub
gevent makes the subscribe super easy; just the normal Pubnub.subscribe(), wrapped in a call to gevent.spawn(). There
are initially two such subscribes: a “control” subscription, which will recursively re-subscribe after handling each
message, and a “sync” subscription to initially synchronize with other servers.
The “control” subscription does three things:
1. Respond to “sync” requests; providing the current game state to other servers
2. Update config; allowing you to remotely reconfigure all servers
3. Update game state; receive deltas from other servers
I’ll explain more about the config updating later. After Both subscriptions are established, a “sync” request is
published to any listening servers. The first “sync” response that comes within 5 minutes will cause the game state
to be updated with the provided values.
The server then goes to sleep until a user makes an “update” request. That will start a recursive timer (the interval is
configurable) which will send stat updates to clients, and stat delta updates to other servers. The timer recursion
automatically stops 5 minutes after the last user “update” request.
And that’s about all there really is to the server. You could also do some other fancy things like persisting the state
to disk periodically. It isn’t a lot of data to store, but these stats are also not critical, especially for the demo.

7. Running the Stats Server
Starting the server is easy as well, just start the main script with python, or run it directly in a shell (it has exec
permissions and a shebang). The script accepts three optional arguments for listening IP, listening port, and config
file. The config file is just JSON, in the same format as in config.py
Here’s an example:
$ python main.py 0.0.0.0 8999 ~/config.json
www. .com

8. Hacking the Client
Back on the client-side, we just need a function to record the deltas, and another to send the “update” requests to
the server. I decided to use the localStorage API to record the update state between each request, allowing the
deltas to be calculated correctly even after restarting the browser.
As far as security goes, I will be ignoring the possibility of cheaters for the demo. Stats can also be skewed by
saves that have completed the game, because SPOILER ALERT the computer tab grants access to generating
candies and lollipops at an impossible rate, and changing pretty much every variable in strange ways. SPOILER
ALERT
Client requirements are as follows:
• Turn a blind eye to cheaters (simplifies everything)
• Periodically send “update” requests (once every 5 seconds is a good start)
• Do not send “update” requests after the game has been completed
The update interval will be once every 5 seconds by default, which will be quick enough to affect the stats updates
that users end up seeing, and slow enough to handle a large number of simultaneous players with low server
resources; With 2,000 users, the server only needs to handle 400 requests per second. The gevent-based server will
easily handle that without a hiccup, even on commodity hardware. In fact, each server should handle about 1,000
concurrent connections. If more than 5,000 users are playing, just launch another stats server and put it behind
nginx (reverse proxy) as a load balancer. More on that later.
www. .com

9. // Save to PubNub CandyBox stats server periodically
if ((this.nbrOfSecondsSinceLastMinInterval % 5) === 0) {
stats.update();
}
The Hook
main.js is where the game loop runs. It’s implemented as a simple interval that fires once per second. This is the place
to add the stats updates. The code is very simple; just throttle a function call to once every 5 seconds:
The stats.update() function is where the magic happens. It records the interesting bits of game state, calculates the
delta, and sends the request to the stat server.
www. .com

10. $.each(currentUpdate, function (k, v) {
if (typeof(v) !== "string")
delta[k] = lastUpdate[k] - v;
});
Delta Calculation
The delta calculation is very easy (as you might imagine). I just keep a record of the last game stat after a successful
“update” request (and save this object to localStorage), and the delta is calculated with a small iterator:
Should be self-explanatory, but basically the difference between values in lastUpdate and values in currentUpdate are
recorded as the delta, with a safety net for the code key (not shown) which is a string value. The delta is then sent to the
stats server in an “update” request.
The server does its work, and periodically publishes a message for the stats page. The listener code is in stats.js and
you can see it does the percentage calculation client-side. It is otherwise incredibly basic.
www. .com

11. upstream stats_server {
server localhost:8999;
#server localhost:8998;
#server localhost:8997;
#server localhost:8996;
keepalive 32;
}
server {
listen 80;
root /home/ubuntu/pn-candybox/public;
index index.html;
server_name candybox.pubnub.co;
location /ping {
proxy_pass http://stats_server;
}
Server Configuration
With the client and server ready to go, it’s time to start thinking about the operational side of the project; configuring
servers, DNS, an even dynamically scaling and remote-control reconfiguration.
I’m using nginx as a host for the client code and it also doubles as a front-end load balancer for the stats server. The
nginx config looks like this:
location /save {
proxy_pass http://stats_server;
}
location /update {
proxy_pass http://stats_server;
}
location / {
try_files $uri $uri/ /index.html;
}
}
www. .com

12. I did some load testing with ApacheBench and found that nginx with a single stats server can handle about 763
requests per second with 100 concurrent connections, or about 305 requests per second with 200 concurrent. All
tests were done on a t1.micro AWS instance (E5507 @ 2.27GHz, 589MB RAM) running Ubuntu 13.04 with no TCP
kernel tuning. This setup is good enough for our “2,000 simultaneous players” requirement.
Dynamically Scaling
With the server config in place, we can easily scale up by adding more upstream stats servers (commented in the
config above). Then reloading nginx. The stats servers will automatically synchronize with one another over PubNub.
We can also reconfigure the servers at runtime to tune the message publishing rates. I just have to open the
PubNub console and publish a specially constructed message to the “candybox_update” channel. Here’s an
example message that reconfigures the servers to publish only once every 5 seconds:
Publish that message, and all servers will instantly adjust their message publishing interval to 5 seconds. This is just
one example of what makes PubNub truly awesome.
{
"uuid" : "master",
"action" : "config",
"data" : {
"update_interval" : 5
}
}
www. .com

13. Wrapping Up
With all of that, we now have Candy Box sending
periodic updates to our stats server, and our stats
servers periodically sending updates to the stats page.
And it’s all done in a dynamically scalable way, with a
ridiculously small memory footprint, and low bandwidth
requirements.
All done! Now you should play the game,
check the stats, and fork me!
www. .com