DragonCraftArchitectural Overview       Freeverse Inc.
Jesse Sanford, Joshua KehnFreeverse / ngmoco:) / DeNA● Web guys brought in to design RESTful  HTTP based games for handhel...
Why Node.js?● Already using javascript. Knowledge share!● Fast growing ecosystem.● Reasonable to bring libraries from clie...
Why MongoDB?● Ever changing schemas make document  stores attractive.● Easy path to horizontal scalability.● 10gen is very...
Handling the change.● Lots of patience.● Many proof of concepts.● Dedicated pocs for different puzzle pieces.  (Platform s...
Building from scratch.●   Lots of testing.●   Pre-flight environment for content.●   Duplicate of production for release s...
"Physical" Infrastructure● EC2 fabric managed by Rightscale● Extensive library of "Rightscripts" and  "Server templates".●...
Logical Diagram
Physical/Logical Diagram
Mongo Infrastructure● Mongo cluster per environment.● 3 config nodes split between 2 availability  zones.● Currently only ...
Mongo Infrastructure cont.● Config nodes on t1-micros.● DB nodes on m1-xlarges.● DB nodes running raid 10 on ebs.● XFS wit...
Shrinking Mongo● Staging and testing environments too costly.● Logically the application knows no  MongoD/S differences.● ...
Other Services● HA-proxy 2 m1-small● Memcached - 1 m1-large● PHP+Apache (cms), Flume/Syslog - 1 m1-  large● Ejabberd - 1 m...
Log4js-syslog, Flume● Centralized logging from all application  servers in the cluster.● Configurable log levels at both t...
Daida, Beanstalkd● Needed fast worker queue for push  messaging and out-of-band computation.● Considered Redis and Resque●...
Daida, Beanstalkd cont.● Developers define jobs (payload contains  variables needed for job to execute)● Developers schedu...
Daida handler example. var handlers = {    bar: function(data, cb) {       var callback = cb || function() { /* noOp */ };...
Ejabberd● Best multi-user-chat solution for the money.● Considered IRC and other more custom  solutions.● Javascript handh...
Other Nodejs libraries●   Connect●   Express●   mongoose●   oauth●   connect-auth●   connect-cookie-session
Megaphone load tester● Written in erlang/otp to make use of its  lightweight processes and distributed nature.● SSL Capabl...
Megaphone load tester cont.● Replay in bulk! (Load test).● Centralized console can spawn http replay  processes on many he...
%% This module contains the functions to manipulate req/resp for the dcraft_session1 playback-module(dcraft_session1).    ...
Other notables● Recently started using pythons fabric library  for rolling releases.● Node cluster for multiprocess node.●...
Screenshots
Demo
Links●   http://dragoncraftthegame.com/●   http://freeverse.com/●   http://blog.ngmoco.com/●   https://developer.mobage.co...
Repos● https://github.com/ngmoco/daida.js.git● https://github.com/ngmoco/daida-beanstalk.  git● https://github.com/ngmoco/...
Dragoncraft Architectural Overview
Upcoming SlideShare
Loading in …5
×

Dragoncraft Architectural Overview

947 views

Published on

Presentation given to NYC Tech Talks Meetup group on June 26 2012. More info here: http://www.meetup.com/NYC-Tech-Talks/events/69478562/

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
947
On SlideShare
0
From Embeds
0
Number of Embeds
63
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Dragoncraft Architectural Overview

  1. 1. DragonCraftArchitectural Overview Freeverse Inc.
  2. 2. Jesse Sanford, Joshua KehnFreeverse / ngmoco:) / DeNA● Web guys brought in to design RESTful HTTP based games for handheld clients.● Platform concurrently being built by Ngmoco team out in San Francisco.● First games in companys history built entirely on ○ EC2 ○ Node.js ○ MongoDB● There is a lot of firsts here!
  3. 3. Why Node.js?● Already using javascript. Knowledge share!● Fast growing ecosystem.● Reasonable to bring libraries from client to server and vice versa.● Lots of javascript patterns and best practices to follow.● Growing talent pool.
  4. 4. Why MongoDB?● Ever changing schemas make document stores attractive.● Easy path to horizontal scalability.● 10gen is very easy to work with.● Lots of best practice patterns for running on ec2 infrastructure.● Javascript is a friendly interface.
  5. 5. Handling the change.● Lots of patience.● Many proof of concepts.● Dedicated pocs for different puzzle pieces. (Platform services, Game Libraries)● Developer training and evangelists.● Performance testing and open source library vetting.● Lots of patience. Seriously.
  6. 6. Building from scratch.● Lots of testing.● Pre-flight environment for content.● Duplicate of production for release staging.● Full stack developer sandboxes on every workstation. ○ Individual MongoDB and Node.js instances running. ○ Full client stack available as a browser based handheld simulator for client interface.
  7. 7. "Physical" Infrastructure● EC2 fabric managed by Rightscale● Extensive library of "Rightscripts" and "Server templates".● Different deployments for each environment.● Deployments a mix of single service machines and arrays of machines.● Arrays load balanced by HA proxy not ELBs● Mongo clusters are largest expense.
  8. 8. Logical Diagram
  9. 9. Physical/Logical Diagram
  10. 10. Mongo Infrastructure● Mongo cluster per environment.● 3 config nodes split between 2 availability zones.● Currently only 1 shard.● 3 db nodes split between 2 availability● mongos processes running directly on app servers.
  11. 11. Mongo Infrastructure cont.● Config nodes on t1-micros.● DB nodes on m1-xlarges.● DB nodes running raid 10 on ebs.● XFS with LVM.● Snapshots taken after forcing fsync and lock on db and then XFS freeze.● Backups always done on secondary.
  12. 12. Shrinking Mongo● Staging and testing environments too costly.● Logically the application knows no MongoD/S differences.● Still single shard.● Spinning instances is quick.● Only used for smoke testing at the end of every dev cycle.● Moving to single master -> slave replication.● Cost savings of 60% in these environments.
  13. 13. Other Services● HA-proxy 2 m1-small● Memcached - 1 m1-large● PHP+Apache (cms), Flume/Syslog - 1 m1- large● Ejabberd - 1 m1-large● Beanstalkd - 1 m1-large● Nodejs - (currently 3) c1-xlarge
  14. 14. Log4js-syslog, Flume● Centralized logging from all application servers in the cluster.● Configurable log levels at both the application layer and filters on the stream after that.● Flume speaks syslog fluently● Flume allows us to point the firehose wherever we want.● Its trivial to ingest the Flume ouput from s3 into Hadoop/Elastic Map Reduce
  15. 15. Daida, Beanstalkd● Needed fast worker queue for push messaging and out-of-band computation.● Considered Redis and Resque● Considered RabbitMQ/AMPQ● Beanstalkd was built for work queues.● Beanstalkd is very simple.● No real support for HA● Workers needed to be written in javascript.● No upfront knowledge about the runtime activities of workers.
  16. 16. Daida, Beanstalkd cont.● Developers define jobs (payload contains variables needed for job to execute)● Developers schedule jobs.● Developers create "strategies" which know how to execute the jobs.● At runtime using some functional magic Daida closes the developer defined strategy around the payload variables that came with the job.● This is somewhat similar to the job being run by a worker inside a container with a
  17. 17. Daida handler example. var handlers = { bar: function(data, cb) { var callback = cb || function() { /* noOp */ }; //if callback wasnt passed console.log(test job passed data: + JSON.stringify(data)); callback(); //always make sure to callback!!!! }, foo: function(data, cb) { var callback = cb || function() { /* noOp */ }; console.log(foo job passed name+ data.name); callback(); //again never forget to callback!!! }, }; exports.handlers = handlers; exports.bar = handlers.bar; exports.foo = handlers.foo;//taken from https://github.com/ngmoco/daida.js
  18. 18. Ejabberd● Best multi-user-chat solution for the money.● Considered IRC and other more custom solutions.● Javascript handhelds can use javascript chat client libraries!● Capable of being run over plain HTTP. (Comet/long-poll/BOSH)● Widely used.● Fine grained control over users and rooms.● A little complex for our needs.● Erlang/OTP is solid.
  19. 19. Other Nodejs libraries● Connect● Express● mongoose● oauth● connect-auth● connect-cookie-session
  20. 20. Megaphone load tester● Written in erlang/otp to make use of its lightweight processes and distributed nature.● SSL Capable HTTP Reverse proxy.● Records sessions from handhelds.● Proxy is transparent and handhelds are stupid.● Choose which sessions to replay.● Write small scripts to manipulate req/resp during replay. OAuth handshakes?● Interact with replay in console.● Record results of replay.
  21. 21. Megaphone load tester cont.● Replay in bulk! (Load test).● Centralized console can spawn http replay processes on many headless machines. Similar to headless Jmeter.● A single session (some number of individual requests) is sent to the client process when spawned● Responses are sent back to the centralized databases as clients receive them.● The same session can be sent to multiple clients and played back concurrently.
  22. 22. %% This module contains the functions to manipulate req/resp for the dcraft_session1 playback-module(dcraft_session1). EX. Session handler-include("blt_otp.hrl"). script for-export([ create_request/1, create_request/2, create_request/3]).-record(request, { url, verb, body_vars}). manipulating-record(response, {request_number, response_obj}).create_request(Request) -> create_request(Request, []). requests at runtimecreate_request(Request, Responses) -> create_request(Request, Responses, 0).create_request(#request{url="http://127.0.0.1:8080/1.2.1/dragoncraft/player/sanford/mission/"++OldMissionId} = Request, Responses, RequestNumber) -> ?DEBUG_MSG("~p Request for wall Found!~n", [?MODULE]), [LastResponseRecord|RestResponses] = Responses, {{_HttpVer, _ResponseCode, _ResponseDesc}, _Headers, ResponseBodyRaw} = LastResponseRecord#response.response_obj, {ok, ResponseBodyObj} = json:decode(ResponseBodyRaw), ResponseKVs = element(1, ResponseBodyObj), [Response_KV1 | [Response_KV2 | Response_KV_Rest ]] = ResponseKVs, Response_KV2_Key = element(1, Response_KV2), Response_KV2_Val = element(2, Response_KV2), ResponseDataObj = element(1, Response_KV2_Val), [ResponseDataKV | ResponseDataKVRest ] = ResponseDataObj, ResponseData_KV_Key = element(1, ResponseDataKV), %<<"identifier">> ResponseData_KV_Val = element(2, ResponseDataKV), MissionId = binary_to_list(ResponseData_KV_Val), Replaced = re:replace(Request#request.url, OldMissionId, MissionId++"/wall"), [ReHead|ReRest] = Replaced, [ReTail] = ReRest, ?DEBUG_MSG("~p replaced head is ~p and tail ~p ~n", [?MODULE, ReHead, ReTail]), NewUrl = binary_to_list(ReHead)++binary_to_list(ReTail), NewRequest = Request#request{url=NewUrl};create_request(Request, Responses, RequestNumber) -> Request.
  23. 23. Other notables● Recently started using pythons fabric library for rolling releases.● Node cluster for multiprocess node.● Node ipc with linux signals to raise and lower logging levels and content updates.
  24. 24. Screenshots
  25. 25. Demo
  26. 26. Links● http://dragoncraftthegame.com/● http://freeverse.com/● http://blog.ngmoco.com/● https://developer.mobage.com/● http://dena.jp/intl/
  27. 27. Repos● https://github.com/ngmoco/daida.js.git● https://github.com/ngmoco/daida-beanstalk. git● https://github.com/ngmoco/daida-local.git● https://github.com/ngmoco/Megaphone (coming very soon)

×