You have an application that works well on a single node, and you’ve heard that Erlang lets you scale out in a cluster. How do you go about doing that? We’ll walk through the steps I took to turn ExVenture (a multiplayer game server) into a distributed application. Starting with connecting nodes in development and production, to picking a cluster leader via the Raft protocol, and dealing with process groups to fan calls throughout the cluster. Finally we’ll see some of the hurdles I encountered when spanning multiple nodes.

1. Going Multi-Node
Eric Oestrich
SmartLogic

2. Eric Oestrich
SmartLogic - smartlogic.io
GitHub: oestrich
Twitter: ericoestrich

3. Going Multi-Node
● What is a MUD?
● Clustering
● Leader Selection
● Spanning the Cluster
● Problems Encountered

4. ExVenture
exventure.org

5. Gossip
gossip.haus

6. midmud.com
MidMUD

7. What is a M.U.D.?

8. What is a MUD?
● Multi-User Dungeon
● Text based multiplayer game
● Started in the late 70s
● World of Warcraft or EverQuest, but with no graphics

9. What is a MUD?

10. What is a MUD?

11. Supervision Tree
● ExVenture
○ Cluster.Supervisor
○ Raft
○ Data.Repo
○ Web.Supervisor
○ Game.Supervisor
○ :ranch

12. Supervision Tree
● Game.Supervisor
○ Game.Session.Registry
○ Game.Config
○ Game.Caches
■ Game.Items
■ …
○ Game.Session.Supervisor
○ Game.World

13. Supervision Tree
● Game.World
○ Game.World.Master
○ Game.World.ZoneController
○ Game.ZoneSupervisor
○ Game.ZoneSupervisor
○ ...

14. Supervision Tree
● Game.ZoneSupervisor
○ Game.Zone
○ Game.Room.Supervisor
■ Game.Room
■ ...
○ Game.NPC.Supervisor
■ Game.NPC
■ ….

15. Starting Point

16. Everything Assumed One Node
● Heavy geared for a single node
● Most processes were local Registry
● Data stored in local ETS tables
● Lots of in process state (a virtual world)

17. Paths Forward
● Umbrella App
● Single Application

18. Single Application that is Cluster Aware

19. Clustering

20. libcluster
bitwalker/libcluster

21. libcluster
config :libcluster,
topologies: [
local: [
strategy: Cluster.Strategy.Epmd,
config: [
hosts: [:"world1@host", :"world2@host"]
]
]
]

22. Start the nodes
iex --sname world1 -S mix
iex --sname world2 -S mix

23. Start the nodes

24. Picking a leader

25. Raft Protocol
raft.github.io

26. Raft Basics

27. Raft Basics

28. Raft Basics

29. Raft Basics

30. Raft Basics

31. Picking a Leader
● Each node waits a random amount of time and picks itself
as leader
● Each other node votes for the first node it sees ask
● Once a majority is chosen the leader is picked and starts
the world

32. What the leader does
● Pushes zones out across the cluster
● When a node dies
○ Looks at zones that should be online
○ Spins them up across the cluster

33. Squabble
● ExVenture’s leader election has been pulled out
● GitHub: oestrich/squabble

34. Squabble Callbacks
defmodule MyApp.Leader do
@behaviour Squabble.Leader
@impl true
def leader_selected(term) do
end
@impl true
def node_down() do
end
end

35. Why a majority of the cluster?

36. Removing local Registry
Switch to :global registry

37. Spanning the Cluster
● Cache updates need to be updated on each node
● Process groups to handle this
● Might be a nicer way to handle these

38. Join the pg2 group on cache start
defmodule Game.Items do
@ets_key :items
def init(_) do
:ok = :pg2.create(@ets_key)
:ok = :pg2.join(@ets_key, self())
#...
end

39. Client API
def insert(item) do
members = :pg2.get_members(@ets_key)
Enum.map(members, fn member ->
GenServer.call(member, {:insert, item})
end)
end

40. Problems Encountered

41. Calls timing out
● Might be a network blip and a previously stable call is no
longer stable

42. Circuit Breakers

43. Circuit Breakers

44. Circuit Breakers

45. Things that should happen at most once
● Attached to Gossip, a chat service
● Each node connects as a websocket to Gossip
● New messages posted at most once to the local channel
● Only the leader node should handle these actions

46. Performance Tweaks
The journey from 230 to 3500

47. Single process being overloaded by messages
● Room processes became a bottleneck
● Create a side process that handles notifications
● PR #72
● ~230 -> ~600

48. Single process being overloaded by data size
● Session registry was overloaded with size of user data
● Pushing a large preloaded Ecto struct around
● Massively simplify what is stored
● PR #73
● ~600 -> ~1200

49. Too large messages
● The data being passed around was too large
● Same huge User structs
● Ran out of ram at 50GB
● Use same simplification in messages
● PR #74
● ~1200 -> ~3500

51. Specs
- Intel Core i7-6700K
- Quad Core, with Hyperthreading
- 64GB of RAM

52. Demo

53. ExVenture Discord
https://discord.gg/GPEa6dB

54. Image Credits
http://discworld.starturtle.net/lpc/
http://www.majormud.com/

55. Thanks!
● GitHub: oestrich/ex_venture
● Documentation: exventure.org