Building And Releasing A Massively Multiplayer Online Game
•
3 likes•2,487 views
When a small startup of experienced game developers in Seattle finished their first game, an open-world single player zombie survival game, to rave reviews they continuously received the same feedback - "No multiplayer?". Building the necessary software infrastructure for a massively multiplayer online game historically requires large engineering teams, even larger operational support organizations, and can take years to finish. With a big success under their belts the organization had a choice to make: Go through a massive growth period, potentially ruining the culture they worked so hard to cultivate or hire smart, work even smarter, and use the right tools for the job. In this talk you will experience this story from to back. Come hear how Undead Labs grew from a small 20 person game development shop creating a single player Xbox 360 Live Arcade Game, State of Decay, to a multi-game studio bringing State of Decay and more to the online space. You will hear how we leveraged Elixir, the OTP ecosystem, and DevOps to create the foundations of Undead's online platform while simultaneously developing multiple games and the personnel required for the studio's future.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Building And Releasing A Massively Multiplayer Online Game
Similar to Building And Releasing A Massively Multiplayer Online Game (20)
Recently uploaded
Recently uploaded (20)
Building And Releasing A Massively Multiplayer Online Game
- 3. The$Story
- 13. The$Problem
- 14. How$Do$We • Stay&development&focused • Transi6on&from&single&player&to&massively&mul6player&online& games • Build&a&posi6ve&rela6onship&with&our&publishers • Develop&and&maintain&a&strong&company&culture
- 19. Interview)For • IQ$%$intelligence • EQ$%$emo0onal$intelligence • Personality
- 24. Team%Language%Evalua-on What%technology%should%we%adopt? • C/C++ • C#&mono • Python • Ruby • Erlang
- 30. You$Could$But...
- 33. Erlang/OTP+Network+Programming+Perks • Easily(encode(and(decode(binary(message • Monitor(/(subscribe(to(open(sockets • "Free"(internal(rou:ng(of(messages • A(sane(threading(model • Crash?Resistant
- 36. Elixir&Has&The&Same&Core&Concepts • Processes(and(Mailboxes • Supervision(Trees • Pa6ern(Matching • Tail(Recursion
- 40. Erlang'Calling'Elixir defmodule Time do def now do :erlang.now() end end 'Elixir.Time':now().
- 42. (Except(It(Does)
- 44. Have%Compassion
- 45. Dev$Tooling
- 46. Mix • Build'Tool • Create/Compile/Test'projects • Dependency'management $ mix new hello_world $ cd hello_world $ mix deps.get
- 47. Hex
- 50. DevOps Make%opera)ons%every%engineer's%responsibility • Protect(your(culture • Control(your(live(service • Don't(become(beholden(to(a(service(organiza:on(with(a( monopoly Operate'without'the'extra'headcount
- 53. Terraform • Abstrac)on,for,Cloud,providers • Allows,you,to,describe,network/server,resources,in,code • Security,Groups • VPCs,,Subnets • Virtual,Instances • And,more
- 56. Chef • Configura*on+Management+Tool • Allows+you+to+describe+node+resources+in+code • Files • Directories • Services
- 59. Foster'Adop+on For$Your$Technology$Choices • Make&on)ramps&for&your&teammates • Create&a&safe&to&fail&environment • Create&and&evangelize&pa9erns&and&standardiza:ons.&Allow&for& collabora:on.&(Illusion&of&choice?)
- 60. Code%Reviews • Assist&with&learning • Keep&a&posi2ve&tone&in&your&code&reviews.&Stay&away&from&an& authorata2ve&tone&(I&love&you&but&I&push&you&in&the&dirt) • Generic&praise&is¬&enough • Avoid&syntax/style&discussions
- 62. Three%Components%of%An%Online%Game • Game&Client • Game&Server(s) • Service&Pla4orm
- 63. Game%Client • Executable+on+your+device+(game.app,+game.exe,+etc) • Draws+images+at+30++fps • Sends+and+receives+data+to+game+server
- 64. Game%Server(s) • Simulate*the*game • Nego/ates*with*clients • Communicate*with*pla6orm
- 65. Pla$orm • Game&agnos+c • Highly&available • Game&experience&s+tching • Provides&services&to&games • Auth,&Chat,&Presence,&Leaderboards,&Guilds,&Player&Profile,&and& more
- 66. Tubes (Pla%orm)
- 74. Undead&Server&Message&Anatomy • 32$96'bit'message'envelope • Message'ID'with'a'fixed'max'length • Variable'size'body'with'a'fixed'max'length
- 75. Message&Evelope 3"Parts"("32"Bits"each • Header'Info • Rou.ng'Info'(op.onal) • Transac.on'Info'(op.onal)
- 76. Header&Info • Rou%ng(Info((1(bit) • Transac%on(Info((1(bit) • Reserved((4(bits) • Message(ID(Length((6(bits) • Body(Length((20(bits)
- 79. Routed'Messages
- 83. Transac'ons • Complete)when)client)receives)a)message)with)the)transac4on) complete)flag)set • Client)automa4cally)4mes8out)long)running)transac4ons • Client)can)receive)a)transac4on)extension)message)to)avoid) 4meout • Client)can)receive)a)par4al)message)to)build)a)complete)response
- 84. Transac'on)Info)(op'onal) • Is$Response$(1$bit) • Is$Par3al$(1$bit) • Transac3on$ID$(30$bits)
- 87. Protocol'Message'Defini1on defmodule TUProtocol.Chat do use TUProtocol.Behaviour, protocol_id: 1 protocol_message "WhisperSend" do field :sender_id, :integer field :receiver_name, :string field :receiver_tag, :integer field :content, :string end # ... other definitions end
- 89. Lobby%Server • Provides*a*persistent*connec0on*for*clients • Nego0ates*with*"core"*pla8orm*services • Auth,*Chat,*Presence • Communicates*with*pla8orm*to*get*a*connec0on*to*a*suitable* game*server • Mul0@tenant*service • Connects*as*client*to*Route
- 90. Route&Server • Ranch'Listener • Peek'at'message'envelop'and'route'message'accordingly • Tag'messages'with'route'acceptor'pid • Disconnects'misbehaving'clients • Rate'limit • Hanging'/'Par?al'connec?on'flooding
- 91. Listener defmodule Route.Listener do alias Route.Config def start do :ranch.start_listener(__MODULE__, Config.acceptors, :ranch_tcp, [port: Config.port], Route.ConnHandler, []) end end Ranch&Documenta-on: h"p://ninenines.eu/docs/en/ranch/HEAD/guide/introduc7on/
- 93. Client'To'Lobby 1. Client)connects)to)versioner 2. Versioner)1>)Match)for)the)loca7on)of)an)appropriate)Lobby a.)Match)1>)Spawn)to)create)an)appropriate)Lobby)server)if)not) exis7ng b.)Versioner)caches)response 3. Versioner)1>)Client)with)Lobby's)address 4. Client)connects)to)Lobby
- 94. Client'To'Game 1. Lobby'(>'Match'with'a'game'join'request 2. Match'(>'Spawner'if'appropriate'game'server'is'not'running'for' game'type'and'version 3. Match'(>'Lobby'with'address'of'game'server'and'reservaAon' token 4. Lobby'(>'Client'with'address'of'game'server'and'reservaAon' token 5. Client'connects'to'game'server'with'reservaAon'token
- 97. Rou$ng'The'Message %ChatMsg.Login{protocol_id: 1, message_id: "Login", acceptor: route_acceptor_pid, account_id: id, account_name: "reset", account_tag: 1984} = message • Send&to&any&known&Chat&server • handle_message/1&matching&%ChatMsg.Login{}&will& handle&message
- 98. Message&Dispatcher defmodule Chat.Dispatcher do alias Protocol.Chat.Message, as: ChatMsg use Net.Dispatcher def handle_message(%ChatMsg.Login{} = request) do case Chat.User.login(request) do :ok -> {:reply, NetError.success} {:error, :banned} -> {:reply, NetError.banned("ch:login:1")} end end # ... other handle_message/1 clauses end
- 99. The$Net.Dispatcher$Behaviour • Re$usable*behaviour • Simply:*use Net.Dispatcher • Creates*a*registered*Erlang*process • Implement*handle_message/1*callbacks*matching*protocol* messages • Quickly*dispatches*to*worker*processes*to*run*matching* handle_message/1*callback
- 100. def handle_message(%ChatMsg.Login{} = request) do case Chat.User.login(request) do :ok -> {:reply, NetError.success} # <--- NetError {:error, :banned} -> {:reply, NetError.banned("ch:login:1")} # <--- NetError end end
- 101. NetError • Actually)kind)of)named)poorly...)Should)be)NetResponse? • Special)type)of)Protocol)Message • Contains)two)parts • Code)A)Unique)error)code)to)Undead)servers • Message)A)Unique)error)string)to)help)idenEfy)source)of)issue
- 102. Login&(And&Subscribe) def init([user, %ChatMsg.Login{} = txn]) do {:ok, feed} = Chat.PresenceFeed.start_link account = %Account{id: user.id, name: user.name, tag: user.tag} state = %{login_txn: txn, socket: txn.acceptor, account: account, presence_feed: feed} {:ok, state, 0} end def handle_info(:timeout, %{login_txn: %ChatMsg.Login{} = request, account: account} = state) do complete_login_txn(request, account) {:noreply, %{state | login_txn: nil}, 0} end def handle_info(:timeout, %{socket: socket, account: account, entity: entity} = state) do monitor_ref = Process.monitor(socket) Enum.each friends(account), fn(%{account_id: receiver}) -> broadcast_online(receiver, {account, self}) end {:noreply, %{state | monitor_ref: monitor_ref, active_sockets: [{socket, monitor_ref}]}} end
- 103. Lose%Connec)on%To%Lobby? def handle_info({:DOWN, ref, _, _, _}, %{monitor_ref: ref} = state) do {:stop, :normal, state} end def terminate(_, %{presence_feed: feed}) do PresenceFeed.update(feed, @offline) :ok end
- 104. Presence • Add$PresenceFeed$to$user$process • A0empt$broadcast$when$coming$online • Users$receiving$broadcast$subscribe$to$your$presencefeed • You$subscribe$to$users$subscribing$to$your$presencefeed • Handlers$are$removed$if$user$process$is$shutdown • User$process$is$shutdown$if • Primary$connec?on$is$terminated
- 105. Sending'A'Presence'No/fica/on defmodule Protocol.Route do import Protocol.Serializer, only: [encode!: 2] def client_send(message, handler) when is_map(message) and is_pid(handler) do encoded = encode!(message, routing: false) case message do %{time_start: start, txn_response: 1, txn_partial: 0} -> GenServer.cast(handler, {:send, {:reply, encoded, start}}) _ -> GenServer.cast(handler, {:send, {:message, encoded}}) end end end
- 106. Forwarding*A*Presence*No1fica1on defmodule Route.ConnHandler do # .... def handle_cast({:send, message}, %{wait_timeout: wait, last_heard: heard} = state) do case message do {:reply, data, nil} -> send_data(data, state) {:reply, data, start} -> time_end = :timer.now_diff(:os.timestamp, start) Route.Metrics.report("tu_route.transaction_response_ms", (time_end / 1_000)) send_data(data, state) {:message, data} -> send_data(data, state) end new_timeout = wait - (Time.tick_count - heard) {:noreply, state, :erlang.max(0, new_timeout)} end end
- 107. Distribu(ng+Tubes
- 108. Sharding • Pre%shard*data*into*X*buckets • Buckets*are*PostgreSQL*schemas • Elixir*applica>ons*service*shards*(buckets) • Broadcast*the*shards*to*route*servers
- 109. Internally)Rou-ng)Messages • Route'messages'to'applica0on'servicing'desired'shard • Hide'complexity'from'engineers • Make'it'look'like'Elixir/Erlang • Send'Erlang'messages'to'Net.Dispatchers • One'process'(dispatch'worker)'per'request
- 110. Tubes&Net defmodule Net do @spec call(atom, integer | binary, atom, atom, list) def call(protocol, route_key, module, function, arguments) do # ... end @spec cast(atom, integer | binary, atom, atom, list) def cast(protocol, route_key, module, function, arguments) do # ... end end
- 111. defmodule Chat.User do alias Protocol.Chat.Message, as: ChatMsg def whisper(sender_id, receiver_id, body) do Net.cast(Protocol.Chat, receiver_id, __MODULE__, :rpc_whisper, [receiver_id, sender_id, body]) end def rpc_whisper(receiver, sender, body) do case lookup(receiver) do {:ok, acceptor} -> msg = %ChatMsg{receiver: receiver, sender: sender, body: body} client_send(acceptor, msg) _ -> :ok end end end
- 113. Embedded&Shard&/&Create&Date&In&En11es CREATE SEQUENCE next_id_seq; CREATE OR REPLACE FUNCTION next_id(OUT result bigint) AS $$ DECLARE our_epoch bigint := 1409266191000; seq_id bigint; now_millis bigint; shard_id int := 1; BEGIN SELECT nextval('next_id_seq') % 1024 INTO seq_id; SELECT FLOOR(EXTRACT(EPOCH FROM clock_timestamp()) * 1000) INTO now_millis; result := (now_millis - our_epoch) << 23; result := result | (seq_id << 13); result := result | (shard_id); END;
- 115. Extract'Shard'ID def extract(id) when is_integer(id) do extract(<<id::size(64)>>) end def extract(<<_::size(51), shard_id::size(13)>>) do shard_id end
- 117. Hash/Mod)Shard)Value defmodule Tubes.Shard do use Bitwise @divisor 128 def hash(value) when is_binary(value) do String.downcase(value) |> Fnv1a.hash end def mod(value) when is_integer(value) do (value &&& (@divisor - 1)) end end
- 118. defmodule Fnv1a do use Bitwise @offset 2166136261 @prime 16777619 def hash(term) when is_binary(term) do _hash(@offset, 0, term) end def hash(term) do :erlang.term_to_binary(term) |> hash end defp _hash(hash, byte_offset, bin) when byte_size(bin) == byte_offset, do: hash defp _hash(hash, byte_offset, bin) do <<_::size(byte_offset)-binary, octet::size(8), _::binary>> = bin xord = hash ^^^ octet hash = rem((xord * @prime), (2 <<< 31)) _hash(hash, byte_offset + 1, bin) end end
- 123. Discovery*Breakdown • Poller'('Discover'nodes'running'an'OTP'applica6on • PollerEvent'('Callbacks'to'run'when'poller'finds'something • Handler'('Behavior,'callbacks'to'run'when'poller'event'is'fired • Heartbeat'('Broadcast'availability
- 124. Chat%Supervisor%init/1%Snippet children = [ worker(Discovery.Poller, ["tu_route", [ {Discovery.Handler.NodeConnect, [ {Tubes.RouteConnector, [:tu_chat, Chat.Repo.shard_ids]} ]} ]], id: Poller.Route), worker(Discovery.Heartbeat, [@heartbeat_check, Config.heartbeat_ttl]) ] supervise(children, strategy: :one_for_one)
- 125. RouteConnector • When&connec)on&is&made • Register&serviced&shards&with&Route&server's&shard&directory • Retry®istra)on&on&failure
- 128. World&Server • "Town" • Quest,hub • Player,interac6on • Chat • Store
- 130. Other&Services
- 131. Admin • Powered(by • Phoenix,(Ecto,(Ember.js • Administra9ve(console(for(game(masters • Gi?(items • Kick/Ban(players
- 132. Catalogue • Provides*a*catalogue*of*purchaseable*items*with • Price*"8er" • Friendly*name • Internal*item*ID • Automa8cally*download/transforms*store*data*output*from*build* server
- 133. Fulfillment • Mediates)purchasing)from)client)to • Steam/iOS/Android/Kindle)AppStore • All)transac:ons)are)idempotent • Delivers)goods)to)user's)mailbox
- 134. Mail • Asynchronous+message+storage • A1achment+support+to+deliver+goods+to+players • Used+for • Gi:ing+and+purchasing • Sending+messages/items+between+players
- 135. No#fy • Push&no)fica)on&mediator • Send&push&no)fica)ons&to&devices • Supports&X&number&of&push&no)fica)on&services
- 136. Vault • Completely*game*agnos/c • Player*Data*persistence*layer • Storage*for • Character*data • Inventory*Data • Shared*storage*for*guilds
- 138. Fun$Addi(onal$Problems$Solved • Building)OTP)Releases)For)Elixir)Apps • Account)Name)+)Tag • (Brutally))Hacking)in)schema)support)into)Ecto • Game)agnosDc)item)fulfillment
- 139. What's'Next? • Automa(c*shard*migra(on • Expanding*and*contrac(ng • Automa(c*shard*re6balancing • Addi(onal*PvP/E6Sports*features • Guilds/Community*features • Always*improving*performance
- 141. Over%me!
- 142. Monitoring(Your(App • Wombat(Erlang(Solu0ons • Metrics(6(Instrumental • Log(Collec0on(6(Sumologic