Erlang can help with game backends even if you don't know Erlang. The presentation covered three talks: 1. Noah Gift advocated for using Erlang in a game company. 2. Mario Izquierdo discussed using RabbitMQ in the game company DIO's architecture to improve scalability and flexibility over their previous ZeroMQ-based system. 3. James Mayfield described the evolution of Versu, an interactive storytelling platform, from a single C process simulator to a distributed Erlang-based architecture using supervisors, ports, load balancing, and Riak.

1. Believe in Erlang
in Games!!!
Noah Gift
Mario Izquierdo
James Mayfield
High Level Overview of
Erlang in Game Backends

2. Overview of Our Talk: 3 for 1
Special
● Advocating Erlang in A Game Company:
Noah Gift
● RabbitMQ in DIO: Mario Izquierdo
● Versu an AI Game Evolution: James
Mayfield

3. You Can Get Fired For Using Erlang
● Erlang is "functional" and
this is "complicated" for
"regular" programmers
● It has been around for too
long
● It is not Java, or Python or
Ruby or C++ or "X".
● Your boss is an idiot who
can't code like this guy???

4. Always Be Delivering
● On day 1 at work don't suggest rewriting the
billing systems in Haskell
● Show up early, stay late, underpromise
and overdeliver. Rinse and Repeat * N
● Then suggest Erlang on a small project...

5. Ubiquitous Multiprocessing in
Erlang: Not TBD...
● PyCon 2013 Keynote: Async
Multiprocessing...TBD
● Event Loops in Ruby, Node and Python can
be simple at first, then exponentially difficult
●

6. More on Event Loops in
Scripting Languages
● What do you do when your one processor
gets hot and starts to consume a lot of CPU?
● Start to write Erlang in a scripting language
on a tight deadline?
● Amdah's Law (sucks)
● Energy Efficiency

7. More on Event Loops in Scripting
Languages
● Maybe you simply use RabbitMQ to dance
around it...notice how the "X" language's
largest website always has a secret
ingredient...(hint not their language)
● Erlang (RabbitMQ): script language crack.

8. RabbitMQ
in dio
Mario Izquierdo
Erlang can help you even if
you don't know erlang!

9. What is RabbitMQ?

10. What is RabbitMQ?
● A service for messaging that implements the
AMQP protocol
● Erlang nodes running Erlang applications
(so it must be good)
Think of it as a service (like MySQL), where you can:
● Open a connection
● Define Exchanges, Queues and Bindings
(instead of tables and indexes)
● Send and Receive Messages
(instead of reading/adding records)

11. Queues and Consumers
Round Robin
queue
Deliver
subscribe
Consumer Consumer
Consumer

12. Exchanges and Producers
Producer
Producer
Producer
Publish
to any exchange
Exchange Exchange

13. Bindings and Routing
Consumer
Producer
queue
Publish
Consumer
Bind Subscribe
routing rules
Exchange
Consumer
queue

14. Types of Exchanges and bindings
Direct Fanout
uses routing_key broadcasts to all
dog
Exchange Exchange
dog
cat
dog
queue queue queue queue

15. Typical usage of RabbitMQ
● Connect with RabbitMQ
● Define an exchange
○ Start publishing messages
● Define a queue and its bindings
○ Subscribe to the queue and listen for incoming
messages
● Add more consumers (workers) to the same
queue for load balancing
● Connect applications written in different
languages
● "Send and forget", i.e. logging

16. Real usage example: dio.com
dio is a web app built on
ruby, optimized to support
multiplayer experiences in
real time.
A place (world) instance
runs in memory on one of
the available GameServers.

17. Dio Architecture
● Browsers send HTTP requests to Rails and
Connection Servers
● Connection Servers use HTML5
EventSource for push notifications
● Connection Servers send in-game
messages to world instances that are
running in Game Servers
● Connection and Game servers need to talk
to each other constantly, and they need to
scale

18. Dio before RabbitMQ
Browser Browser Browser
http
Push
Notifications
NGINX
Rails Rails Connection Connection
Server Server Server Server
Workers AMQP
Zmq
Dio Broker
Game Game Game
WebAPI
Server Server Server

19. Dio before RabbitMQ
Relies in ZeroMQ to send messages
Browser Browser Browser
http
Push
Notifications
NGINX
Keep a ZeroMQ socket for each GameServer, and
other for the Broker. EventSource with the Browsers.
Rails Rails Connection Connection
Server Server Server Server
Rails and workers
RabbitMQ is used
can not talk with
only for Chat
other components
Workers AMQP
Zmq
Knows about all
GameServers and
worlds, decides
Dio Broker about where to run
the next world.
Receives
GameServers
Heartbeats.Routes
Connection Server
Instance 0 used only Game Game Game requests to the right
for WebAPI Server Server Server GameServer.

20. Dio before RabbitMQ
Relies in ZeroMQ to send messages
Browser Browser Browser
http
NGINX
notice our friend
Rails Rails Connection Connection
was already here
Server Server Server Server
But it's ok, omnipresent
Workers AMQP and omnipotent Broker
will keep the evil away,
right?
Integrate other
Dio Broker
systems is
really hard
GOD
Game Game Game
Server Server Server

21. Pentagram Antipattern
Build a complex system with a lot of
interdependencies and you will cast the devil
● Dependence on black magic
● Dependence on priests
● Multiple global-aware Gods
Fix it with SCIENCE!
● Study the problem
● Burn Churches
● Use the right tool for the right problem

22. Dio before RabbitMQ: Problems
● ZeroMQ has bad support in ruby, the libraries are outdated.
● ZeroMQ sockets are very low level, actually they are a very weird mix of
a low level library with black magic, they are never disconnected.
● Too many sockets: it needs an all-to-all socket connections between all
game servers, connection servers and broker (Pentagram).
● Hard to scale
● Limited communications: for example if we want to send a message
from Rails to a Game Server, or from a Worker to a ConnectionServer. Too
hard that seems impossible, so we end up doing hacks like short polling
mongodb or adding HTTP methods in the ConnectionServers (Gandalf).
● Broker is too complex. It is needed for routing and also to decide where
to run new world instances. Needs to know about every GameServer and
needs to be in sync, which makes it very error prone and unstable (GOD)

23. Dio with
RabbitMQ Browser Browser Browser
http
Push
NGINX Notifi.
Rails Rails Connection Connection
Server Server Server Server
Workers
AMQP
Game Game
Server Server

24. HTML and JavaScript client.
Dio with Send messages through HTTP.
Receive messages through
HTML5 SSE.
RabbitMQ Browser Browser Browser
http
Push
NGINX Notifi.
Serves HTML and
assets.
Holds the connection with
Authentication.
the browsers to push
Authorization.
notifications.
Pages outside stories.
Rails Rails Connection Connection Also works as ClientProxy
Not a RabbitMQ
consumer. Server Server Server Server
RabbitMQ exchanges and queues.
Workers
Used for messaging, load balance
world instances in the GameServers
AMQP and allows any component to talk with
any other component.
For example send a push notification
Run world instances (several to the browser from a Sidekiq worker.
instances on each GameServer), in
real time, in memory.
Subscribes to RabbitMQ by Game Game
world_id channel. Server Server
Publish to avatar_id channel

25. Dio with
RabbitMQ Browser Browser Browser
http
Push
NGINX Notifi.
Rails Rails Connection Connection
Server Server Server Server
GO TO
HELL
Workers
AMQP Dio
GOD Brok
er
Game Game
Server Server

26. Who would you prefer to worship?
Dio Broker
RabbitMQ
RabbitMQ GOD ARCANE GODS
supported by a big community, defined by some dudes
widely used, in a reactive way
known practices, to remove the fears
and written in erlang of the people of their time

27. Good choice
RabbitMQ
ROCKS
RabbitMQ
ME TOO
AGREE

28. Dio with RabbitMQ: Advantages
● Consistency: AMQP is going to be the protocol for messaging.
● Flexibility: is easy to add more components when needed, and they instantly are connected
with any other component.
● Visibility: RabbitMQ is more easy to monitor than a lot bunch of sockets.
● Less restrictions: Any system can be a producer and send messages to RabbitMQ, so if we
need to send a message to a world from Rails is easy, and if we need to send a push notification
to the user from rails or from a worker is also easy.
● Support: RabbitMQ has much better support in ruby than ZeroMQ.
● More Features: RabbitMQ can accomplish a lot more. ZeroMQ is like a simple abstraction over
sockets whether RabbitMQ is a full messaging system (similar to compare the file system with a
database).
● Routing out of the box: for example a Game Server can subscribe for messages to the words
that are running on it, and the Connection Server just publishes to that world, no need to know
what Game Server is running that world.
● No complex Broker and automatic load balancing: We can use RabbitMQ queues to round-
robin new worlds, and the Game Servers can subscribe/unsubscribe to that queue when they
are available or overloaded. So they don't need to know about other Game Servers.

29. Messaging Browser Browser Browser
examples http
NGINX
Rails Rails Connection Connection
Server Server Server Server
Workers
AMQP
Game Game
Server Server

30. Game Messages
Av1 Av3
Connection Connection
Av2 Server Server
Av1
avatar_clients.direct Av3
Av2
World1
world_instances.
direct
World1
Game Game
Server Server

31. Av2
Load a world instance (1) Av3
Av1
Av1
Av3
Av2 avatar_clients.direct
Connection Connection
Server Server
non_loaded_worlds.
fanout REDIS
loading_world
locks
World1
World1
world_instances.
Game
direct Game
Server Server

32. Av2
Load a world instance (2) Av3
Av1
Av1
Av2 Av3
avatar_clients.direct
Connection Av4 Connection
Server Server
Av4
wants to play in
World2
non_loaded_worlds.
fanout REDIS
loading_world
locks
World1
World1
world_instances.
Game
direct Game
Server Server

33. Av2
Load a world instance (3) Av3
Av1
Av1
Av2 Av3
avatar_clients.direct
Connection Av4 Connection
Server Server
Av4
non_loaded_worlds.
Message is fanout REDIS
returned because can loading_world
not route to World2 locks
connect world 2
World1
World1
world_instances.
Game
direct Game
Server Server

34. Av2
Load a world instance (4) Av3
Av1
Av1
Av2 Av3
avatar_clients.direct
Connection Av4 Connection
Server Server
connect
Av4
non_loaded_worlds.
fanout REDIS
loading_world
locks
connect
round robin
connect
World1
World1
world_instances.
Game
direct Game
Server Server

35. Av2
Load a world instance (5) Av3
Av1
Av1
Av2 Av3
avatar_clients.direct
Connection Av4 Connection
Server Server
Av4
non_loaded_worlds.
fanout REDIS
loading_world
locks
World 2
World1 World2
World1 World 2
world_instances.
Game
direct Game
Server Server

36. Av2
Load a world instance (6) Av3
Av1
Av1
Av2 Av3
avatar_clients.direct
Connection Av4 Connection
Server Server
Av4 connect ACK
non_loaded_worlds.
fanout REDIS
loading_world
locks
World1 World2
World1 World 2
world_instances.
Game
direct Game
Server Server

37. Thanks
Questions for the end
Have fun!

38. Versu: Evolution of
an Erlang Project
James Mayfield
Erlang can help you even if
you don't know erlang!

39. What is Versu?

40. What is Versu?
"Versu is an interactive storytelling platform that builds
experiences around characters and social interaction"

41. What is Versu?
● Simulator

42. What is Versu?
● Simulator
○ Written in C

43. What is Versu?
● Simulator
○ Written in C
○ Robust, but can crash (content errors)

44. What is Versu?
● "Business Logic"

45. What is Versu?
● "Business Logic"
○ Talks to C process via ports

46. What is Versu?
● "Business Logic"
○ Talks to C process via ports
○ Tracks players and games

47. What is Versu?
● "Business Logic"
○ Talks to C process via ports
○ Tracks players and games
○ Some data transformation of simulator I/O

48. What is Versu?
● Web App
tm

49. What is Versu?
● Web App
○ Expose Rest-like interface to simulator functions
tm

50. What is Versu?
● Web App
○ Expose Rest-like interface to simulator functions
○ UI
tm

51. What is Versu?
● Data Store

52. What is Versu?
● Data Store
○ Saved games

53. What is Versu?
● Data Store
○ Saved games
○ User generated content

54. The Evolution of Versu
The early days...

55. The Evolution of Versu
The early days...
Misultin
Process Simulator
Port
Process

56. The Evolution of Versu
First Steps...
Cowboy
Supervisor
Simulator
Game Logic
Interface
Simulator

57. The Evolution of Versu
Many sims per node
Cowboy
Supervisor
Supervisor Supervisor
Port Port
Supervisor Supervisor
Owner Owner
Sim.. Sim..
Game Simulator Game Simulator
Logic Interface Logic Interface

58. The Evolution of Versu
Increased Awesomeness
Load balancer
Erlang VM Node Erlang VM Node
YeeHaw YeeHaw
S S
S S S S
S W S W S W S W
C C C C
W W W W W W W W

59. The Evolution of Versu
More Increased Awesomeness
Riak
Load balancer Riak Riak
Erlang VM Node Erlang VM Node
YeeHaw YeeHaw
S S
S S S S
S W S W S W S W
C C C C
W W W W W W W W

60. Take away

61. Take away
● Happy devs

62. Take away
● Happy devs
● Happy ops

63. Take away
● Happy devs
● Happy ops
● much to learn!

64. Thank You

65. References
● http://www.ibm.
com/developerworks/cloud/library/cl-
optimizepythoncloud2/index.html
●