The document discusses the implementation of a multi-node architecture for a multi-user dungeon (MUD) game called Exventure, focusing on aspects such as clustering, leader selection using the Raft protocol, and challenges faced during the transition from a single node to a multi-node system. Key topics include maintaining data consistency across nodes, optimizing performance, and utilizing chat services for messaging. Specific performance improvements and technical solutions are also mentioned, highlighting the project's evolution and development resources available on GitHub and its documentation site.

1. Going Multi-Node
Eric Oestrich
SmartLogic

2. Eric Oestrich
SmartLogic - smartlogic.io
GitHub: oestrich
Twitter: ericoestrich

3. Going Multi-Node
● What is a MUD?
● Clustering
● Leader Selection
● Spanning the Cluster
● Problems Encountered

4. ExVenture
exventure.org

5. Gossip
gossip.haus

6. midmud.com
MidMUD

7. What is a M.U.D.?

8. What is a MUD?
● Multi-User Dungeon
● Text based multiplayer game
● Started in the late 70s
● World of Warcraft or EverQuest, but with no graphics

9. What is a MUD?

10. What is a MUD?

11. Supervision Tree
● ExVenture
○ Cluster.Supervisor
○ Raft
○ Data.Repo
○ Web.Supervisor
○ Game.Supervisor
○ :ranch

12. Supervision Tree
● Game.Supervisor
○ Game.Session.Registry
○ Game.Config
○ Game.Caches
■ Game.Items
■ …
○ Game.Session.Supervisor
○ Game.World

13. Supervision Tree
● Game.World
○ Game.World.Master
○ Game.World.ZoneController
○ Game.ZoneSupervisor
○ Game.ZoneSupervisor
○ ...

14. Supervision Tree
● Game.ZoneSupervisor
○ Game.Zone
○ Game.Room.Supervisor
■ Game.Room
■ ...
○ Game.NPC.Supervisor
■ Game.NPC
■ ….

15. Starting Point

16. Everything Assumed One Node
● Heavy geared for a single node
● Most processes were local Registry
● Data stored in local ETS tables
● Lots of in process state (a virtual world)

17. Paths Forward
● Umbrella App
● Single Application

18. Single Application that is Cluster Aware

19. Clustering

20. libcluster
bitwalker/libcluster

21. libcluster
config :libcluster,
topologies: [
local: [
strategy: Cluster.Strategy.Epmd,
config: [
hosts: [:"world1@host", :"world2@host"]
]
]
]

22. Start the nodes
iex --sname world1 -S mix
iex --sname world2 -S mix

23. Start the nodes

24. Picking a leader

25. Raft Protocol
raft.github.io

26. Raft Basics

27. Raft Basics

28. Raft Basics

29. Raft Basics

30. Raft Basics

31. Picking a Leader
● Each node waits a random amount of time and picks itself
as leader
● Each other node votes for the first node it sees ask
● Once a majority is chosen the leader is picked and starts
the world

32. What the leader does
● Pushes zones out across the cluster
● When a node dies
○ Looks at zones that should be online
○ Spins them up across the cluster

33. Squabble
● ExVenture’s leader election has been pulled out
● GitHub: oestrich/squabble

34. Squabble Callbacks
defmodule MyApp.Leader do
@behaviour Squabble.Leader
@impl true
def leader_selected(term) do
end
@impl true
def node_down() do
end
end

35. Why a majority of the cluster?

36. Removing local Registry
Switch to :global registry

37. Spanning the Cluster
● Cache updates need to be updated on each node
● Process groups to handle this
● Might be a nicer way to handle these

38. Join the pg2 group on cache start
defmodule Game.Items do
@ets_key :items
def init(_) do
:ok = :pg2.create(@ets_key)
:ok = :pg2.join(@ets_key, self())
#...
end

39. Client API
def insert(item) do
members = :pg2.get_members(@ets_key)
Enum.map(members, fn member ->
GenServer.call(member, {:insert, item})
end)
end

40. Problems Encountered

41. Calls timing out
● Might be a network blip and a previously stable call is no
longer stable

42. Circuit Breakers

43. Circuit Breakers

44. Circuit Breakers

45. Things that should happen at most once
● Attached to Gossip, a chat service
● Each node connects as a websocket to Gossip
● New messages posted at most once to the local channel
● Only the leader node should handle these actions

46. Performance Tweaks
The journey from 230 to 3500

47. Single process being overloaded by messages
● Room processes became a bottleneck
● Create a side process that handles notifications
● PR #72
● ~230 -> ~600

48. Single process being overloaded by data size
● Session registry was overloaded with size of user data
● Pushing a large preloaded Ecto struct around
● Massively simplify what is stored
● PR #73
● ~600 -> ~1200

49. Too large messages
● The data being passed around was too large
● Same huge User structs
● Ran out of ram at 50GB
● Use same simplification in messages
● PR #74
● ~1200 -> ~3500

51. Specs
- Intel Core i7-6700K
- Quad Core, with Hyperthreading
- 64GB of RAM

52. Demo

53. ExVenture Discord
https://discord.gg/GPEa6dB

54. Image Credits
http://discworld.starturtle.net/lpc/
http://www.majormud.com/

55. Thanks!
● GitHub: oestrich/ex_venture
● Documentation: exventure.org