Speaker: Jamie Winsor Language: English When a small startup of experienced game developers in Seattle finished their first game, an open-world single player zombie survival game, to rave reviews they continuously received the same feedback - "No multiplayer?". Building the necessary software infrastructure for a massively multiplayer online game historically requires large engineering teams, even larger operational support organizations, and can take years to finish. With a big success under their belts the organization had a choice to make: Go through a massive growth period, potentially ruining the culture they worked so hard to cultivate or hire smart, work even smarter, and use the right tools for the job. In this talk you will experience this story from to back. Come hear how Undead Labs grew from a small 20 person game development shop creating a single player Xbox 360 Live Arcade Game, State of Decay, to a multi-game studio bringing State of Decay and more to the online space. You will hear how Undead Lab's leveraged Chef, Elixir, and fully embraced the DevOps development pattern to create the foundations of their new online platform while simultaneously developing multiple titles while cultivating the personnel required to lock in a bright future of the studio. Visit our website: http://atmosphere-conference.com/

1. We#will#begin#shortly...

2. Stay%Awhile%And%Fiddle%With%Your%
Smartphone
By#Jamie#Winsor

3. The$Story

4. Building(And(Suppor.ng
Online&Experiences

8. Architec(ng,,Planning,,and,Implemen(ng,is,40%,of,an,
Engineer's,job
The$other$60%$is$team$building$and$communica6on.

9. So#ware(doesn't(solve(problems.(
People(do.

12. Disclaimer
"Be$er"&Within&Context

13. The$Problem

14. How$Do$We
• Stay&development&focused
• Transi6on&from&single&player&to&massively&mul6player&online&
games
• Build&a&posi6ve&rela6onship&with&our&publishers
• Develop&and&maintain&a&strong&company&culture

15. Develop'Your'Company,'Not'Just'
Your'So6ware

16. Hire%Engineers,%Not%Programmers

17. Don't&Be&Afraid&To&Contract&Work&
Out

18. Never%Higher%A%Maybe

19. Interview)For
• IQ$%$intelligence
• EQ$%$emo0onal$intelligence
• Personality

20. Develop'Smart
Be#Pragma)c#And#Iterate
Leverage'Open'Source

21. Use$The$Right$Tools$And$Pa2erns

22. Building(A(Distributed(System?

24. Team%Language%Evalua-on
What%technology%should%we%adopt?
• C/C++
• C#&mono
• Python
• Ruby
• Erlang

25. Game%engineers%are%generally%experienced%in%
C2like%languages

27. OTP
A"Framework"For"Building"Distributed"Systems

28. Would&You&Build&Your&Own&Web&Framework?

29. Or#Your#Own#Game#Engine?

30. You$Could$But...

31. Some%Of%Us%Have%Shit%To%Do

32. Erlang/OTP+Saves+Time
A"Lot"Of"Time

33. Erlang/OTP+Network+Programming+Perks
• Easily(encode(and(decode(binary(message
• Monitor(/(subscribe(to(open(sockets
• "Free"(internal(rou:ng(of(messages
• A(sane(threading(model
• Crash?Resistant

34. Erlang'isn't'very'approachable
—"Concerned"Engineer

36. Elixir&Has&The&Same&Core&Concepts
• Processes(and(Mailboxes
• Supervision(Trees
• Pa6ern(Matching
• Tail(Recursion

37. Started'Using'Elixir'at'0.9.0
We#Asked#Ourselves,#"Is#This#A#Risk?"

38. Solid&Two)Way&Interoperability

39. Elixir&Calling&Erlang
:erlang.now()

40. Erlang'Calling'Elixir
defmodule Time do
def now do
:erlang.now()
end
end
'Elixir.Time':now().

41. Syntax'Doesn't'Ma.er

42. (Except(It(Does)

43. Elixir's(Syntax(Makes
Erlang's)Ecosystem)More)
Approachable
For$Your$Colleagues

44. Have%Compassion

45. Dev$Tooling

46. Mix
• Build'Tool
• Create/Compile/Test'projects
• Dependency'management
$ mix new hello_world
$ cd hello_world
$ mix deps.get

47. Hex

48. How$Are$We$Gonna$Run$This?

49. Live%In%The%Cloud
• On$demand$test$environments
• Expand$or$contract$your$live$cluster

50. DevOps
Make%opera)ons%every%engineer's%responsibility
• Protect(your(culture
• Control(your(live(service
• Don't(become(beholden(to(a(service(organiza:on(with(a(
monopoly
Operate'without'the'extra'headcount

51. No#Opera)ons?
Who$Will$Build$Our$Machines?

53. Terraform
• Abstrac)on,for,Cloud,providers
• Allows,you,to,describe,network/server,resources,in,code
• Security,Groups
• VPCs,,Subnets
• Virtual,Instances
• And,more

54. Who$Will$Configure$Our$Machines?

56. Chef
• Configura*on+Management+Tool
• Allows+you+to+describe+node+resources+in+code
• Files
• Directories
• Services

57. Terraform(to(provision(your(nodes
Chef%to%bootstrap%your%nodes

58. No#Such#Thing#As#DevOps#Team
Every&server&engineer&needs&to&understand&Elixir,&OTP,&Chef,&and&
the&automa:on&tooling&to&leverage&the&cloud.

59. Foster'Adop+on
For$Your$Technology$Choices
• Make&on)ramps&for&your&teammates
• Create&a&safe&to&fail&environment
• Create&and&evangelize&pa9erns&and&standardiza:ons.&Allow&for&
collabora:on.&(Illusion&of&choice?)

60. Code%Reviews
• Assist&with&learning
• Keep&a&posi2ve&tone&in&your&code&reviews.&Stay&away&from&an&
authorata2ve&tone&(I&love&you&but&I&push&you&in&the&dirt)
• Generic&praise&is&not&enough
• Avoid&syntax/style&discussions

61. Architec(ng+The+Service

62. Three%Components%of%An%Online%Game
• Game&Client
• Game&Server(s)
• Service&Pla4orm

63. Game%Client
• Executable+on+your+device+(game.app,+game.exe,+etc)
• Draws+images+at+30++fps
• Sends+and+receives+data+to+game+server

64. Game%Server(s)
• Simulate*the*game
• Nego/ates*with*clients
• Communicate*with*pla6orm

65. Pla$orm
• Game&agnos+c
• Highly&available
• Game&experience&s+tching
• Provides&services&to&games
• Auth,&Chat,&Presence,&Leaderboards,&Guilds,&Player&Profile,&and&
more

66. Tubes
(Pla%orm)

67. Q.#Where#to#start?
A.#With#only#exactly#what#you#need.

68. Iden%fy(A('Proving'(Feature

69. Presence'Updates'From'One'Player'
To'Another

70. Set$Expecta*ons$With$Your$
Publisher
And$Help$Inform$Milestones

71. Building(The(Main(Players
Protocol,(Route,(Lobby,(and(Chat

72. The$Suppor*ng$Cast
Spawn,'Match,'and'Versioner

73. Protocol'Layer
• TCP
• Binary+messages

74. Undead&Server&Message&Anatomy
• 32$96'bit'message'envelope
• Message'ID'with'a'fixed'max'length
• Variable'size'body'with'a'fixed'max'length

75. Message&Evelope
3"Parts"("32"Bits"each
• Header'Info
• Rou.ng'Info'(op.onal)
• Transac.on'Info'(op.onal)

76. Header&Info
• Rou%ng(Info((1(bit)
• Transac%on(Info((1(bit)
• Reserved((4(bits)
• Message(ID(Length((6(bits)
• Body(Length((20(bits)

77. Header&Binary
<<header_field::size(32)>> = <<has_routing::size(1),
has_txn::size(1),
reserved::size(4),
byte_size(message_id)::size(6),
byte_size(body)::size(20)>>

78. Binary'Pa)ern'Matching
Is#More#Intui,ve#Than#Bit#Shi3ing

79. Routed'Messages

80. Rou$ng'Info'(op$onal)
• Route'Hash'(22'bits)
• Protocol'ID'(10'bits)

81. Rou$ng'Binary
<<routing_field::size(32)>> = <<route_hash::size(22),
protocol_id::size(10)>>

82. Non$Transac*onal,Versus,
Transac*onal,Messages

83. Transac'ons
• Complete)when)client)receives)a)message)with)the)transac4on)
complete)flag)set
• Client)automa4cally)4mes8out)long)running)transac4ons
• Client)can)receive)a)transac4on)extension)message)to)avoid)
4meout
• Client)can)receive)a)par4al)message)to)build)a)complete)response

84. Transac'on)Info)(op'onal)
• Is$Response$(1$bit)
• Is$Par3al$(1$bit)
• Transac3on$ID$(30$bits)

85. Transac'on)Binary
<<txn_field::size(32)>> = <<txn_response::size(1),
txn_partial::size(1),
txn_id::size(30)>>

86. Make%It%Easy%To
Define%And%Handle%New%Messages

87. Protocol'Message'Defini1on
defmodule TUProtocol.Chat do
use TUProtocol.Behaviour, protocol_id: 1
protocol_message "WhisperSend" do
field :sender_id, :integer
field :receiver_name, :string
field :receiver_tag, :integer
field :content, :string
end
# ... other definitions
end

88. Elixir&Macros
Used%macros%to%make%it%easy%to%define%a%new%protocol%and%protocol%
message

89. Lobby%Server
• Provides*a*persistent*connec0on*for*clients
• Nego0ates*with*"core"*pla8orm*services
• Auth,*Chat,*Presence
• Communicates*with*pla8orm*to*get*a*connec0on*to*a*suitable*
game*server
• Mul0@tenant*service
• Connects*as*client*to*Route

90. Route&Server
• Ranch'Listener
• Peek'at'message'envelop'and'route'message'accordingly
• Tag'messages'with'route'acceptor'pid
• Disconnects'misbehaving'clients
• Rate'limit
• Hanging'/'Par?al'connec?on'flooding

91. Listener
defmodule Route.Listener do
alias Route.Config
def start do
:ranch.start_listener(__MODULE__, Config.acceptors,
:ranch_tcp, [port: Config.port], Route.ConnHandler, [])
end
end
Ranch&Documenta-on:
h"p://ninenines.eu/docs/en/ranch/HEAD/guide/introduc7on/

92. Establishing+Connec0ons

93. Client'To'Lobby
1. Client)connects)to)versioner
2. Versioner)1>)Match)for)the)loca7on)of)an)appropriate)Lobby
a.)Match)1>)Spawn)to)create)an)appropriate)Lobby)server)if)not)
exis7ng
b.)Versioner)caches)response
3. Versioner)1>)Client)with)Lobby's)address
4. Client)connects)to)Lobby

94. Client'To'Game
1. Lobby'(>'Match'with'a'game'join'request
2. Match'(>'Spawner'if'appropriate'game'server'is'not'running'for'
game'type'and'version
3. Match'(>'Lobby'with'address'of'game'server'and'reservaAon'
token
4. Lobby'(>'Client'with'address'of'game'server'and'reservaAon'
token
5. Client'connects'to'game'server'with'reservaAon'token

95. Lobby%&%Game%+>%Route

96. Explore(Login(&(Whisper(To(Chat

97. Rou$ng'The'Message
%ChatMsg.Login{protocol_id: 1,
message_id: "Login",
acceptor: route_acceptor_pid,
account_id: id,
account_name: "reset",
account_tag: 1984} = message
• Send&to&any&known&Chat&server
• handle_message/1&matching&%ChatMsg.Login{}&will&
handle&message

98. Message&Dispatcher
defmodule Chat.Dispatcher do
alias Protocol.Chat.Message, as: ChatMsg
use Net.Dispatcher
def handle_message(%ChatMsg.Login{} = request) do
case Chat.User.login(request) do
:ok ->
{:reply, NetError.success}
{:error, :banned} ->
{:reply, NetError.banned("ch:login:1")}
end
end
# ... other handle_message/1 clauses
end

99. The$Net.Dispatcher$Behaviour
• Re$usable*behaviour
• Simply:*use Net.Dispatcher
• Creates*a*registered*Erlang*process
• Implement*handle_message/1*callbacks*matching*protocol*
messages
• Quickly*dispatches*to*worker*processes*to*run*matching*
handle_message/1*callback

100. def handle_message(%ChatMsg.Login{} = request) do
case Chat.User.login(request) do
:ok ->
{:reply, NetError.success} # <--- NetError
{:error, :banned} ->
{:reply, NetError.banned("ch:login:1")} # <--- NetError
end
end

101. NetError
• Actually)kind)of)named)poorly...)Should)be)NetResponse?
• Special)type)of)Protocol)Message
• Contains)two)parts
• Code)A)Unique)error)code)to)Undead)servers
• Message)A)Unique)error)string)to)help)idenEfy)source)of)issue

102. Login&(And&Subscribe)
def init([user, %ChatMsg.Login{} = txn]) do
{:ok, feed} = Chat.PresenceFeed.start_link
account = %Account{id: user.id, name: user.name, tag: user.tag}
state = %{login_txn: txn, socket: txn.acceptor, account: account, presence_feed: feed}
{:ok, state, 0}
end
def handle_info(:timeout, %{login_txn: %ChatMsg.Login{} = request, account: account} = state) do
complete_login_txn(request, account)
{:noreply, %{state | login_txn: nil}, 0}
end
def handle_info(:timeout, %{socket: socket, account: account, entity: entity} = state) do
monitor_ref = Process.monitor(socket)
Enum.each friends(account), fn(%{account_id: receiver}) ->
broadcast_online(receiver, {account, self})
end
{:noreply, %{state | monitor_ref: monitor_ref, active_sockets: [{socket, monitor_ref}]}}
end

103. Lose%Connec)on%To%Lobby?
def handle_info({:DOWN, ref, _, _, _}, %{monitor_ref: ref} = state) do
{:stop, :normal, state}
end
def terminate(_, %{presence_feed: feed}) do
PresenceFeed.update(feed, @offline)
:ok
end

104. Presence
• Add$PresenceFeed$to$user$process
• A0empt$broadcast$when$coming$online
• Users$receiving$broadcast$subscribe$to$your$presencefeed
• You$subscribe$to$users$subscribing$to$your$presencefeed
• Handlers$are$removed$if$user$process$is$shutdown
• User$process$is$shutdown$if
• Primary$connec?on$is$terminated

105. Sending'A'Presence'No/fica/on
defmodule Protocol.Route do
import Protocol.Serializer, only: [encode!: 2]
def client_send(message, handler) when is_map(message) and is_pid(handler) do
encoded = encode!(message, routing: false)
case message do
%{time_start: start, txn_response: 1, txn_partial: 0} ->
GenServer.cast(handler, {:send, {:reply, encoded, start}})
_ ->
GenServer.cast(handler, {:send, {:message, encoded}})
end
end
end

106. Forwarding*A*Presence*No1fica1on
defmodule Route.ConnHandler do
# ....
def handle_cast({:send, message}, %{wait_timeout: wait, last_heard: heard} = state) do
case message do
{:reply, data, nil} ->
send_data(data, state)
{:reply, data, start} ->
time_end = :timer.now_diff(:os.timestamp, start)
Route.Metrics.report("tu_route.transaction_response_ms", (time_end / 1_000))
send_data(data, state)
{:message, data} ->
send_data(data, state)
end
new_timeout = wait - (Time.tick_count - heard)
{:noreply, state, :erlang.max(0, new_timeout)}
end
end

107. Distribu(ng+Tubes

108. Sharding
• Pre%shard*data*into*X*buckets
• Buckets*are*PostgreSQL*schemas
• Elixir*applica>ons*service*shards*(buckets)
• Broadcast*the*shards*to*route*servers

109. Internally)Rou-ng)Messages
• Route'messages'to'applica0on'servicing'desired'shard
• Hide'complexity'from'engineers
• Make'it'look'like'Elixir/Erlang
• Send'Erlang'messages'to'Net.Dispatchers
• One'process'(dispatch'worker)'per'request

110. Tubes&Net
defmodule Net do
@spec call(atom, integer | binary, atom, atom, list)
def call(protocol, route_key, module, function, arguments) do
# ...
end
@spec cast(atom, integer | binary, atom, atom, list)
def cast(protocol, route_key, module, function, arguments) do
# ...
end
end

111. defmodule Chat.User do
alias Protocol.Chat.Message, as: ChatMsg
def whisper(sender_id, receiver_id, body) do
Net.cast(Protocol.Chat, receiver_id,
__MODULE__, :rpc_whisper, [receiver_id, sender_id, body])
end
def rpc_whisper(receiver, sender, body) do
case lookup(receiver) do
{:ok, acceptor} ->
msg = %ChatMsg{receiver: receiver, sender: sender, body: body}
client_send(acceptor, msg)
_ -> :ok
end
end
end

112. Determinis)c+Rou)ng
We#Know#What#Shard#The#Message#Goes#To

113. Embedded&Shard&/&Create&Date&In&En11es
CREATE SEQUENCE next_id_seq;
CREATE OR REPLACE FUNCTION next_id(OUT result bigint) AS $$
DECLARE
our_epoch bigint := 1409266191000;
seq_id bigint;
now_millis bigint;
shard_id int := 1;
BEGIN
SELECT nextval('next_id_seq') % 1024 INTO seq_id;
SELECT FLOOR(EXTRACT(EPOCH FROM clock_timestamp()) * 1000) INTO now_millis;
result := (now_millis - our_epoch) << 23;
result := result | (seq_id << 13);
result := result | (shard_id);
END;

114. h"p://instagram.
engineering.tumblr.com/post/
10853187575/sharding.ids.at.
instagram

115. Extract'Shard'ID
def extract(id) when is_integer(id) do
extract(<<id::size(64)>>)
end
def extract(<<_::size(51), shard_id::size(13)>>) do
shard_id
end

116. Randomly)Determinis0c)Rou0ng
We#Don't#Know#What#Shard#The#Message#
Goes#To,#But#It#Will#Always#Go#There

117. Hash/Mod)Shard)Value
defmodule Tubes.Shard do
use Bitwise
@divisor 128
def hash(value) when is_binary(value) do
String.downcase(value) |> Fnv1a.hash
end
def mod(value) when is_integer(value) do
(value &&& (@divisor - 1))
end
end

118. defmodule Fnv1a do
use Bitwise
@offset 2166136261
@prime 16777619
def hash(term) when is_binary(term) do
_hash(@offset, 0, term)
end
def hash(term) do
:erlang.term_to_binary(term) |> hash
end
defp _hash(hash, byte_offset, bin) when byte_size(bin) == byte_offset, do: hash
defp _hash(hash, byte_offset, bin) do
<<_::size(byte_offset)-binary, octet::size(8), _::binary>> = bin
xord = hash ^^^ octet
hash = rem((xord * @prime), (2 <<< 31))
_hash(hash, byte_offset + 1, bin)
end
end

119. FNV$1a'Hashing
h"p://www.isthe.com/chongo/tech/comp/fnv
h"p://programmers.stackexchange.com/ques5ons/49550/which<
hashing<algorithm<is<best<for<uniqueness<and<speed

120. Shard&Directory
• Net%func*ons%hash/mod%route%key%for%shard%ID
• Peek%in%ShardDirectory%to%iden*fy%Erlang%node%servicing%shard%&%
protocol%combina*on'

121. Discovery
Automa'cally+Discover+And+Connect+To+Erlang+
Nodes
github.com/undeadlabs/discovery

123. Discovery*Breakdown
• Poller'('Discover'nodes'running'an'OTP'applica6on
• PollerEvent'('Callbacks'to'run'when'poller'finds'something
• Handler'('Behavior,'callbacks'to'run'when'poller'event'is'fired
• Heartbeat'('Broadcast'availability

124. Chat%Supervisor%init/1%Snippet
children = [
worker(Discovery.Poller, ["tu_route", [
{Discovery.Handler.NodeConnect, [
{Tubes.RouteConnector, [:tu_chat, Chat.Repo.shard_ids]}
]}
]], id: Poller.Route),
worker(Discovery.Heartbeat, [@heartbeat_check, Config.heartbeat_ttl])
]
supervise(children, strategy: :one_for_one)

125. RouteConnector
• When&connec)on&is&made
• Register&serviced&shards&with&Route&server's&shard&directory
• Retry&registra)on&on&failure

126. Discovery,+Thanks+to+Erlang
Will$automa*cally$re/connect$nodes$which$
become$disconnected

127. Game%Servers
Instanced)And)Distributed

128. World&Server
• "Town"
• Quest,hub
• Player,interac6on
• Chat
• Store

129. Combat'Server
• Player(vs(Player
• Player(vs(AI

130. Other&Services

131. Admin
• Powered(by
• Phoenix,(Ecto,(Ember.js
• Administra9ve(console(for(game(masters
• Gi?(items
• Kick/Ban(players

132. Catalogue
• Provides*a*catalogue*of*purchaseable*items*with
• Price*"8er"
• Friendly*name
• Internal*item*ID
• Automa8cally*download/transforms*store*data*output*from*build*
server

133. Fulfillment
• Mediates)purchasing)from)client)to
• Steam/iOS/Android/Kindle)AppStore
• All)transac:ons)are)idempotent
• Delivers)goods)to)user's)mailbox

134. Mail
• Asynchronous+message+storage
• A1achment+support+to+deliver+goods+to+players
• Used+for
• Gi:ing+and+purchasing
• Sending+messages/items+between+players

135. No#fy
• Push&no)fica)on&mediator
• Send&push&no)fica)ons&to&devices
• Supports&X&number&of&push&no)fica)on&services

136. Vault
• Completely*game*agnos/c
• Player*Data*persistence*layer
• Storage*for
• Character*data
• Inventory*Data
• Shared*storage*for*guilds

137. All#Services#Exposed#Through#Same#
API

138. Fun$Addi(onal$Problems$Solved
• Building)OTP)Releases)For)Elixir)Apps
• Account)Name)+)Tag
• (Brutally))Hacking)in)schema)support)into)Ecto
• Game)agnosDc)item)fulfillment

139. What's'Next?
• Automa(c*shard*migra(on
• Expanding*and*contrac(ng
• Automa(c*shard*re6balancing
• Addi(onal*PvP/E6Sports*features
• Guilds/Community*features
• Always*improving*performance

140. Jamie&Winsor
@resetexistence
github.com/reset

141. Over%me!

142. Monitoring(Your(App
• Wombat(Erlang(Solu0ons
• Metrics(6(Instrumental
• Log(Collec0on(6(Sumologic