The document outlines the architectural overview of Dragoncraft by Freeverse Inc., focusing on the use of technologies like Node.js, MongoDB, and EC2 for developing RESTful HTTP-based games for handheld clients. It also discusses the infrastructure management, including deployment strategies, backup processes, and a centralized logging system, while addressing challenges related to scaling and performance testing. Additionally, various services and libraries used to support the platform are highlighted, emphasizing the importance of efficient job scheduling and execution, particularly with tools like Beanstalkd and Megaphone.

1. DragonCraft
Architectural Overview
Freeverse Inc.

3. Jesse Sanford, Joshua Kehn
Freeverse / ngmoco:) / DeNA
● Web guys brought in to design RESTful
HTTP based games for handheld clients.
● Platform concurrently being built by Ngmoco
team out in San Francisco.
● First games in company's history built
entirely on
○ EC2
○ Node.js
○ MongoDB
● There is a lot of firsts here!

4. Why Node.js?
● Already using javascript. Knowledge share!
● Fast growing ecosystem.
● Reasonable to bring libraries from client to
server and vice versa.
● Lots of javascript patterns and best practices
to follow.
● Growing talent pool.

5. Why MongoDB?
● Ever changing schemas make document
stores attractive.
● Easy path to horizontal scalability.
● 10gen is very easy to work with.
● Lots of best practice patterns for running on
ec2 infrastructure.
● Javascript is a friendly interface.

6. Handling the change.
● Lots of patience.
● Many proof of concepts.
● Dedicated poc's for different puzzle pieces.
(Platform services, Game Libraries)
● Developer training and evangelists.
● Performance testing and open source library
vetting.
● Lots of patience. Seriously.

7. Building from scratch.
● Lots of testing.
● Pre-flight environment for content.
● Duplicate of production for release staging.
● Full stack developer sandboxes on every
workstation.
○ Individual MongoDB and Node.js instances running.
○ Full client stack available as a browser based
handheld simulator for client interface.

8. "Physical" Infrastructure
● EC2 fabric managed by Rightscale
● Extensive library of "Rightscripts" and
"Server templates".
● Different deployments for each environment.
● Deployments a mix of single service
machines and arrays of machines.
● Arrays load balanced by HA proxy not ELB's
● Mongo clusters are largest expense.

9. Logical Diagram

10. Physical/Logical Diagram

11. Mongo Infrastructure
● Mongo cluster per environment.
● 3 config nodes split between 2 availability
zones.
● Currently only 1 shard.
● 3 db nodes split between 2 availability
● mongos processes running directly on app
servers.

12. Mongo Infrastructure cont.
● Config nodes on t1-micros.
● DB nodes on m1-xlarges.
● DB nodes running raid 10 on ebs.
● XFS with LVM.
● Snapshots taken after forcing fsync and lock
on db and then XFS freeze.
● Backups always done on secondary.

13. Shrinking Mongo
● Staging and testing environments too costly.
● Logically the application knows no
MongoD/S differences.
● Still single shard.
● Spinning instances is quick.
● Only used for smoke testing at the end of
every dev cycle.
● Moving to single master -> slave replication.
● Cost savings of 60% in these environments.

14. Other Services
● HA-proxy 2 m1-small
● Memcached - 1 m1-large
● PHP+Apache (cms), Flume/Syslog - 1 m1-
large
● Ejabberd - 1 m1-large
● Beanstalkd - 1 m1-large
● Nodejs - (currently 3) c1-xlarge

15. Log4js-syslog, Flume
● Centralized logging from all application
servers in the cluster.
● Configurable log levels at both the
application layer and filters on the stream
after that.
● Flume speaks syslog fluently
● Flume allows us to point the firehose
wherever we want.
● It's trivial to ingest the Flume ouput from s3
into Hadoop/Elastic Map Reduce

16. Daida, Beanstalkd
● Needed fast worker queue for push
messaging and out-of-band computation.
● Considered Redis and Resque
● Considered RabbitMQ/AMPQ
● Beanstalkd was built for work queues.
● Beanstalkd is very simple.
● No real support for HA
● Workers needed to be written in javascript.
● No upfront knowledge about the runtime
activities of workers.

17. Daida, Beanstalkd cont.
● Developers define jobs (payload contains
variables needed for job to execute)
● Developers schedule jobs.
● Developers create "strategies" which know
how to execute the jobs.
● At runtime using some functional magic
Daida closes the developer defined strategy
around the payload variables that came with
the job.
● This is somewhat similar to the job being run
by a worker inside a container with a

18. Daida handler example.
var handlers = {
bar: function(data, cb) {
var callback = cb || function() { /* noOp */ }; //if callback wasn't passed
console.log('test job passed data: ' + JSON.stringify(data));
callback(); //always make sure to callback!!!!
},
foo: function(data, cb) {
var callback = cb || function() { /* noOp */ };
console.log('foo job passed name'+ data.name);
callback(); //again never forget to callback!!!
},
};
exports.handlers = handlers;
exports.bar = handlers.bar;
exports.foo = handlers.foo;
//taken from https://github.com/ngmoco/daida.js

19. Ejabberd
● Best multi-user-chat solution for the money.
● Considered IRC and other more custom
solutions.
● Javascript handhelds can use javascript chat
client libraries!
● Capable of being run over plain HTTP.
(Comet/long-poll/BOSH)
● Widely used.
● Fine grained control over users and rooms.
● A little complex for our needs.
● Erlang/OTP is solid.

20. Other Nodejs libraries
● Connect
● Express
● mongoose
● oauth
● connect-auth
● connect-cookie-session

21. Megaphone load tester
● Written in erlang/otp to make use of it's
lightweight processes and distributed nature.
● SSL Capable HTTP Reverse proxy.
● Records sessions from handhelds.
● Proxy is transparent and handhelds are
stupid.
● Choose which sessions to replay.
● Write small scripts to manipulate req/resp
during replay. OAuth handshakes?
● Interact with replay in console.
● Record results of replay.

22. Megaphone load tester cont.
● Replay in bulk! (Load test).
● Centralized console can spawn http replay
processes on many headless machines.
Similar to headless Jmeter.
● A single session (some number of individual
requests) is sent to the client process when
spawned
● Responses are sent back to the centralized
databases as clients receive them.
● The same session can be sent to multiple
clients and played back concurrently.

23. %% This module contains the functions to manipulate req/resp for the dcraft_session1 playback
-module(dcraft_session1).
EX. Session handler
-include("blt_otp.hrl"). script for
-export([ create_request/1, create_request/2, create_request/3]).
-record(request, { url, verb, body_vars}). manipulating
-record(response, {request_number, response_obj}).
create_request(Request) -> create_request(Request, []).
requests at runtime
create_request(Request, Responses) -> create_request(Request, Responses, 0).
create_request(#request{url="http://127.0.0.1:8080/1.2.1/dragoncraft/player/sanford/mission/"++OldMissionId} = Request, Responses, RequestNumber) ->
?DEBUG_MSG("~p Request for wall Found!~n", [?MODULE]),
[LastResponseRecord|RestResponses] = Responses,
{{_HttpVer, _ResponseCode, _ResponseDesc}, _Headers, ResponseBodyRaw} = LastResponseRecord#response.response_obj,
{ok, ResponseBodyObj} = json:decode(ResponseBodyRaw),
ResponseKVs = element(1, ResponseBodyObj),
[Response_KV1 | [Response_KV2 | Response_KV_Rest ]] = ResponseKVs,
Response_KV2_Key = element(1, Response_KV2),
Response_KV2_Val = element(2, Response_KV2),
ResponseDataObj = element(1, Response_KV2_Val),
[ResponseDataKV | ResponseDataKVRest ] = ResponseDataObj,
ResponseData_KV_Key = element(1, ResponseDataKV), %<<"identifier">>
ResponseData_KV_Val = element(2, ResponseDataKV),
MissionId = binary_to_list(ResponseData_KV_Val),
Replaced = re:replace(Request#request.url, OldMissionId, MissionId++"/wall"),
[ReHead|ReRest] = Replaced,
[ReTail] = ReRest,
?DEBUG_MSG("~p replaced head is ~p and tail ~p ~n", [?MODULE, ReHead, ReTail]),
NewUrl = binary_to_list(ReHead)++binary_to_list(ReTail),
NewRequest = Request#request{url=NewUrl};
create_request(Request, Responses, RequestNumber) -> Request.

24. Other notables
● Recently started using python's fabric library
for rolling releases.
● Node cluster for multiprocess node.
● Node ipc with linux signals to raise and lower
logging levels and content updates.

25. Screenshots

26. Demo

27. Links
● http://dragoncraftthegame.com/
● http://freeverse.com/
● http://blog.ngmoco.com/
● https://developer.mobage.com/
● http://dena.jp/intl/

28. Repos
● https://github.com/ngmoco/daida.js.git
● https://github.com/ngmoco/daida-beanstalk.
git
● https://github.com/ngmoco/daida-local.git
● https://github.com/ngmoco/Megaphone
(coming very soon)