1.
Clusters
and where to find
them

2.
Eugene Pirogov
gmile

3.
Databases
Tools
Theory
Takeaways

4.
Databases
Tools
Theory
Takeaways

5.
What is
a cluster?

6.
set of loosely or tightly
connected computers that work
together so that, in many
respects, they can be viewed as
a single system

7.
When to use
a cluster?

8.
1. Fail-over
clusters

9.
2. Load balancing
clusters

10.
What typical
Erlang cluster
is built with?

11.
1. A node

12.
node()

13.
2. An RPC call

14.
:rpc.call(:nodex, M, :f, [“a”])

15.
3. send call

16.
send({MyProcess, :mynode}, :msg)

17.
Example 1:
Starting a node

18.
iex

19.
~> iex
Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async-
threads:10] [hipe] [kernel-poll:false] [dtrace]
Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h()
ENTER for help)
iex(1)> node()
:nonode@nohost
iex(2)>

20.
iex --name eugene

21.
~> iex --name eugene
Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async-
threads:10] [hipe] [kernel-poll:false] [dtrace]
Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h()
ENTER for help)
iex(eugene@Eugenes-MacBook-Pro-2.local)1>

22.
iex --sname eugene

23.
~> iex --sname eugene
Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async-
threads:10] [hipe] [kernel-poll:false] [dtrace]
Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h()
ENTER for help)
iex(eugene@Eugenes-MacBook-Pro-2)1>

24.
iex --name eugene@host

25.
~> iex --name eugene@host
Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async-
threads:10] [hipe] [kernel-poll:false] [dtrace]
Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h()
ENTER for help)
iex(eugene@host)1>

26.
Example 2:
starting two nodes

27.
iex --name node1@127.0.0.1
iex --name node2@127.0.0.1

28.
~> iex --name node1@127.0.0.1
iex(node1@127.0.0.1)1>
~> iex --name node2@127.0.0.1
iex(node2@127.0.0.1)1>
# On node1
iex(1)> :net_adm.ping(:’node2@127.0.0.1’)
:pong

29.
Example 3:
sending a message

30.
iex --name node1
iex --name node2

31.
# On node2
iex(1)> Process.register(Terminal, self())
# On node1
iex(1)> send({Terminal, :’node2@127.0.0.1’}, “Hello!”)
# On node2
iex(2)> flush()
“Hello!”

32.
Example 4:
calling remotely

33.
# On node1
iex(node1@127.0.0.1)1> :rpc.call(:'node2@127.0.0.1', Enum, :reverse, [100..1])
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21,
22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40,
41, 42, 43, 44, 45, 46, 47, 48, 49, 50, …]
iex(node1@127.0.0.1)2>

34.
REST/JSON/XML
Binary protocol

35.
REST/JSON/XML
Binary protocol

36.
REST/JSON/XML
Binary protocol

37.
Bonus track:
Magic cookie!

38.
cat ~/.erlang.cookie

39.
iex --name node1 --cookie abc
iex --name node2 --cookie xyz

40.
# On node1
iex(1)> :erlang.get_cookie()
:abc
# On node2
iex(1)> :erlang.get_cookie()
:xyz
# On node1
iex(2)> :net_kernel.connect(:'node2@127.0.01')
false
# On node1
iex(3)> :erlang.set_cookie(:’node1@127.0.01’, :xyz)
true
# On node1
iex(4)> :net_kernel.connect(:'node2@127.0.01')
true

41.
Databases
Tools
Theory
Takeaways

42.
epmd

43.
Erlang Port
Mapper Daemon

44.
runs on system
startup

45.
~> ps ax | grep epmd
25502 ?? S 0:11.53 /usr/local/Cellar/erlang/19.1/lib/
erlang/erts-8.1/bin/epmd -daemon

46.
maps ports
to node names

47.
net_kernel

48.
Example 5:
starting a
distributed node

49.
iex

50.
~> iex
Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async-
threads:10] [hipe] [kernel-poll:false] [dtrace]
Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h()
ENTER for help)
iex(1)>

51.
~> iex
Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async-
threads:10] [hipe] [kernel-poll:false] [dtrace]
Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h()
ENTER for help)
iex(1)> node()
:nonode@nohost
iex(2)>

52.
~> iex
Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async-
threads:10] [hipe] [kernel-poll:false] [dtrace]
Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h()
ENTER for help)
iex(1)> node()
:nonode@nohost
iex(2)> Process.registered() |> Enum.find(&(&1
== :net_kernel))
nil

53.
~> iex
Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async-
threads:10] [hipe] [kernel-poll:false] [dtrace]
Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h()
ENTER for help)
iex(1)> node()
:nonode@nohost
iex(2)> Process.registered() |> Enum.find(&(&1
== :net_kernel))
nil
iex(3)> :net_kernel.start([:’mynode@127.0.0.1’])
{:ok, #PID<0.84.0>}
iex(mynode@127.0.0.1)4>

54.
~> iex
Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async-
threads:10] [hipe] [kernel-poll:false] [dtrace]
Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h()
ENTER for help)
iex(1)> node()
:nonode@nohost
iex(2)> Process.registered() |> Enum.find(&(&1
== :net_kernel))
nil
iex(3)> :net_kernel.start([:’mynode@127.0.0.1’])
{:ok, #PID<0.84.0>}
iex(mynode@127.0.0.1)4> Process.registered() |>
Enum.find(&(&1 == :net_kernel))
:net_kernel

55.
Example 6:
monitoring a
node

56.
iex --name node1@127.0.0.1
iex --name node2@127.0.0.1

57.
iex(node1@127.0.0.1)1>

58.
iex(node1@127.0.0.1)1>
iex(node1@127.0.0.1)2> :net_kernel.monitor_nodes(true)
:ok

59.
iex(node1@127.0.0.1)1>
iex(node1@127.0.0.1)2> :net_kernel.monitor_nodes(true)
:ok
iex(node2@127.0.0.1)3> :net_kernel.connect(:'node1@127.0.0.1')
true

60.
iex(node1@127.0.0.1)1>
iex(node1@127.0.0.1)2> :net_kernel.monitor_nodes(true)
:ok
iex(node1@127.0.0.1)3> :net_kernel.connect(:'node2@127.0.0.1')
true
iex(node1@127.0.0.1)4> flush()
{:nodeup, :"node2@127.0.0.1"}
:ok

61.
iex(node1@127.0.0.1)1>
iex(node1@127.0.0.1)2> :net_kernel.monitor_nodes(true)
:ok
iex(node1@127.0.0.1)3> :net_kernel.connect(:'node2@127.0.0.1')
true
iex(node1@127.0.0.1)4> flush()
{:nodeup, :"node2@127.0.0.1"}
:ok
# Ctrl+C on node2

62.
iex(node1@127.0.0.1)1>
iex(node1@127.0.0.1)2> :net_kernel.monitor_nodes(true)
:ok
iex(node1@127.0.0.1)3> :net_kernel.connect(:'node2@127.0.0.1')
true
iex(node1@127.0.0.1)4> flush()
{:nodeup, :"node2@127.0.0.1"}
:ok
# Ctrl+C on node2
iex(node1@127.0.0.1)5> flush()
{:nodedown, :"node2@127.0.0.1"}
:ok
iex(node1@127.0.0.1)5>

63.
net_adm

64.
Example 7:
ping

65.
iex --name node1@127.0.0.1
iex --name node2@127.0.0.1

66.
iex(node1@127.0.0.1)1>

67.
iex(node1@127.0.0.1)1>
iex(node1@127.0.0.1)2> :net_adm.ping(:'node3@127.0.0.1')
pang

68.
iex(node1@127.0.0.1)1>
iex(node1@127.0.0.1)2> :net_adm.ping(:'node3@127.0.0.1')
pang
iex(node1@127.0.0.1)2> :net_adm.ping(:'node2@127.0.0.1')
pong

69.
Example 8:
names

70.
iex --name node1@127.0.0.1
iex --name node2@127.0.0.1

71.
iex(node1@127.0.0.1)1>

72.
iex(node1@127.0.0.1)1>
iex(node1@127.0.0.1)2> :net_adm.names()
{:ok, [{'rabbit', 25672}, {'node1', 51813}, {'node2', 51815}]}
iex(node1@127.0.0.1)3>

73.
iex(node1@127.0.0.1)1>
iex(node1@127.0.0.1)2> :net_adm.names()
{:ok, [{'rabbit', 25672}, {'node1', 51813}, {'node2', 51815}]}
iex(node1@127.0.0.1)3> Node.list()
[]
iex(node1@127.0.0.1)4>

74.
Example 9:
world

75.
# /etc/hosts
127.0.0.1 host1.com
127.0.0.1 host2.com
127.0.0.1 host3.com

76.
# /Users/gmile/.hosts.erlang
host1.com.
host2.com.
host3.com.

77.
iex --name node1@host1.com
iex --name node2@host1.com
iex --name node3@host2.com
iex --name node4@host2.com
iex --name node5@host3.com
iex --name node6@host3.com

78.
iex(node1@host1.com)1>

79.
iex(node1@host1.com)1>
iex(node1@host1.com)1> :net_adm.world()
[:"node1@host1.com", :"node2@host1.com", :"node3@host2.com", :”no
de4@host2.com", :"node5@host3.com", :"node6@host3.com"]
iex(node1@host1.com)2>

80.
Bonus track:
Connecting
to a node running
in production

81.
iex --name node1@127.0.0.1 --cookie abc

82.
$ iex --remsh foo@127.0.0.1 --cookie abc --
name bar@localhost
Erlang/OTP 19 [erts-8.1] [source] [64-bit]
[smp:8:8] [async-threads:10] [hipe] [kernel-
poll:false] [dtrace]
Interactive Elixir (1.3.4) - press Ctrl+C to
exit (type h() ENTER for help)
iex(foo@127.0.0.1)1>

83.
$ kubectl get pods -l app=matcher -o template --
template="{{range.items}}{{.metadata.name}}{{end}}" | xargs -o
-I my_pod kubectl exec my_pod -i -t -- iex --name
debugging@127.0.0.1 --remsh marketplace@127.0.0.1 --cookie
marketplace

84.
$ kubectl get pods -l app=matcher -o template --
template="{{range.items}}{{.metadata.name}}{{end}}" | xargs -o
-I my_pod kubectl exec my_pod -i -t -- iex --name
debugging@127.0.0.1 --remsh marketplace@127.0.0.1 --cookie
marketplace

85.
$ kubectl get pods -l app=matcher -o template --
template="{{range.items}}{{.metadata.name}}{{end}}" | xargs -o
-I my_pod kubectl exec my_pod -i -t -- iex --name
debugging@127.0.0.1 --remsh marketplace@127.0.0.1 --cookie
marketplace

86.
slave

87.
3. Transfer configuration
to slave nodes
2. Add code path to slave nodes
4. Ensure apps
are started on slave
1. Start slave

88.
What else?

89.
Node

90.
bitwalker/swarm

91.
Easy clustering, registration, and
distribution of worker processes for
Erlang/Elixir

92.
…a simple case where workers are
dynamically created in response to
some events under a supervisor, and
we want them to be distributed across
the cluster and be discoverable by
name from anywhere in the cluster

93.
bitwalker/
libcluster

94.
What next?

95.
…I didn’t want to resort to
something like Docker,
because I wanted to see how
far I could push Elixir and its
tooling.

96.
Databases
Tools
Theory
Takeaways

97.
mnesia

98.
Example 10:
initialize mnesia

99.
iex --name node1@127.0.0.1
iex --name node2@127.0.0.1
iex --name node3@127.0.0.1

100.
iex(node1@127.0.0.1)1> :mnesia.create_schema([:'node1@127.0.0.1'])
:ok

101.
iex(node1@127.0.0.1)1> :mnesia.create_schema([:'node1@127.0.0.1'])
:ok
iex(node1@127.0.0.1)2> :mnesia.start()
:ok

102.
iex(node1@127.0.0.1)1> :mnesia.create_schema([:'node1@127.0.0.1'])
:ok
iex(node1@127.0.0.1)2> :mnesia.start()
:ok
iex(node1@127.0.0.1)3> :mnesia.info()
---> Processes holding locks <---
---> Processes waiting for locks <---
---> Participant transactions <---
---> Coordinator transactions <---
---> Uncertain transactions <---
---> Active tables <---
schema : with 1 records occupying 413 words of mem
===> System info in version "4.14.1", debug level = none <===
opt_disc. Directory "/Users/gmile/Mnesia.node1@127.0.0.1" is used.
use fallback at restart = false
running db nodes = ['node1@127.0.0.1']
stopped db nodes = []
master node tables = []
remote = []
ram_copies = []
disc_copies = [schema]
disc_only_copies = []
[{'node1@127.0.0.1',disc_copies}] = [schema]
2 transactions committed, 0 aborted, 0 restarted, 0 logged to disc
0 held locks, 0 in queue; 0 local transactions, 0 remote
0 transactions waits for other nodes: []
:ok
iex(node1@127.0.0.1)4>

103.
iex(node1@127.0.0.1)1> :mnesia.create_schema([:'node1@127.0.0.1'])
:ok
iex(node1@127.0.0.1)2> :mnesia.start()
:ok
iex(node1@127.0.0.1)3> :mnesia.info()
---> Processes holding locks <---
---> Processes waiting for locks <---
---> Participant transactions <---
---> Coordinator transactions <---
---> Uncertain transactions <---
---> Active tables <---
schema : with 1 records occupying 413 words of mem
===> System info in version "4.14.1", debug level = none <===
opt_disc. Directory "/Users/gmile/Mnesia.node1@127.0.0.1" is used.
use fallback at restart = false
running db nodes = ['node1@127.0.0.1']
stopped db nodes = []
master node tables = []
remote = []
ram_copies = []
disc_copies = [schema]
disc_only_copies = []
[{'node1@127.0.0.1',disc_copies}] = [schema]
2 transactions committed, 0 aborted, 0 restarted, 0 logged to disc
0 held locks, 0 in queue; 0 local transactions, 0 remote
0 transactions waits for other nodes: []
:ok
iex(node1@127.0.0.1)4>
“schema” table exists
as a disk_copy (RAM + disk)
on node1@127.0.0.1

104.
iex(node2@127.0.0.1)2> :mnesia.start()
:ok

105.
iex(node2@127.0.0.1)2> :mnesia.start()
:ok
iex(node3@127.0.0.1)2> :mnesia.start()
:ok

106.
iex(node2@127.0.0.1)2> :mnesia.start()
:ok
iex(node3@127.0.0.1)2> :mnesia.start()
:ok
iex(node1@127.0.0.1)2> :mnesia.change_config(:extra_db_nodes, [:’node2@127.0.0.1’])
{:ok, [:"node2@127.0.0.1"]}

107.
iex(node2@127.0.0.1)2> :mnesia.start()
:ok
iex(node3@127.0.0.1)2> :mnesia.start()
:ok
iex(node1@127.0.0.1)2> :mnesia.change_config(:extra_db_nodes, [:’node2@127.0.0.1’])
{:ok, [:"node2@127.0.0.1"]}
iex(node1@127.0.0.1)3> :mnesia.change_config(:extra_db_nodes, [:’node3@127.0.0.1’])
{:ok, [:"node3@127.0.0.1"]}

108.
iex(node2@127.0.0.1)2> :mnesia.start()
:ok
iex(node3@127.0.0.1)2> :mnesia.start()
:ok
iex(node1@127.0.0.1)2> :mnesia.change_config(:extra_db_nodes, [:’node2@127.0.0.1’])
{:ok, [:"node2@127.0.0.1"]}
iex(node1@127.0.0.1)3> :mnesia.change_config(:extra_db_nodes, [:’node3@127.0.0.1’])
{:ok, [:"node3@127.0.0.1"]}
iex(node1@127.0.0.1)1> :mnesia.create_table(:books, [disc_copies: [:'node1@127.0.0.1'],
attributes: [:id, :title, :year]])
:ok

109.
iex(node2@127.0.0.1)2> :mnesia.start()
:ok
iex(node3@127.0.0.1)2> :mnesia.start()
:ok
iex(node1@127.0.0.1)2> :mnesia.change_config(:extra_db_nodes, [:’node2@127.0.0.1’])
{:ok, [:"node2@127.0.0.1"]}
iex(node1@127.0.0.1)3> :mnesia.change_config(:extra_db_nodes, [:’node3@127.0.0.1’])
{:ok, [:"node3@127.0.0.1"]}
iex(node1@127.0.0.1)1> :mnesia.create_table(:books, [disc_copies: [:'node1@127.0.0.1'],
attributes: [:id, :title, :year]])
:ok
iex(node1@127.0.0.1)4> :mnesia.add_table_copy(:books, :'node2@127.0.0.1', :ram_copies)
:ok

110.
iex(node2@127.0.0.1)2> :mnesia.start()
:ok
iex(node3@127.0.0.1)2> :mnesia.start()
:ok
iex(node1@127.0.0.1)2> :mnesia.change_config(:extra_db_nodes, [:’node2@127.0.0.1’])
{:ok, [:"node2@127.0.0.1"]}
iex(node1@127.0.0.1)3> :mnesia.change_config(:extra_db_nodes, [:’node3@127.0.0.1’])
{:ok, [:"node3@127.0.0.1"]}
iex(node1@127.0.0.1)1> :mnesia.create_table(:books, [disc_copies: [:'node1@127.0.0.1'],
attributes: [:id, :title, :year]])
:ok
iex(node1@127.0.0.1)4> :mnesia.add_table_copy(:books, :'node2@127.0.0.1', :ram_copies)
:ok
iex(node1@127.0.0.1)5> :mnesia.add_table_copy(:books, :'node3@127.0.0.1', :ram_copies)
:ok

111.
iex(node1@127.0.0.1)6> :mnesia.info()
---> Processes holding locks <---
---> Processes waiting for locks <---
---> Participant transactions <---
---> Coordinator transactions <---
---> Uncertain transactions <---
---> Active tables <---
books : with 0 records occupying 304 words of mem
schema : with 2 records occupying 566 words of mem
===> System info in version "4.14.1", debug level = none <===
opt_disc. Directory "/Users/gmile/Mnesia.node1@127.0.0.1" is used.
use fallback at restart = false
running db nodes = ['node3@127.0.0.1','node2@127.0.0.1','node1@127.0.0.1']
stopped db nodes = []
master node tables = []
remote = []
ram_copies = []
disc_copies = [books,schema]
disc_only_copies = []
[{'node1@127.0.0.1',disc_copies},
{'node2@127.0.0.1',ram_copies},
{'node3@127.0.0.1',ram_copies}] = [schema,books]
12 transactions committed, 0 aborted, 0 restarted, 10 logged to disc
0 held locks, 0 in queue; 0 local transactions, 0 remote
0 transactions waits for other nodes: []
:ok
iex(node1@127.0.0.1)32>

112.
iex(node1@127.0.0.1)6> :mnesia.info()
---> Processes holding locks <---
---> Processes waiting for locks <---
---> Participant transactions <---
---> Coordinator transactions <---
---> Uncertain transactions <---
---> Active tables <---
books : with 0 records occupying 304 words of mem
schema : with 2 records occupying 566 words of mem
===> System info in version "4.14.1", debug level = none <===
opt_disc. Directory "/Users/gmile/Mnesia.node1@127.0.0.1" is used.
use fallback at restart = false
running db nodes = ['node3@127.0.0.1','node2@127.0.0.1','node1@127.0.0.1']
stopped db nodes = []
master node tables = []
remote = []
ram_copies = []
disc_copies = [books,schema]
disc_only_copies = []
[{'node1@127.0.0.1',disc_copies},
{'node2@127.0.0.1',ram_copies},
{'node3@127.0.0.1',ram_copies}] = [schema,books]
12 transactions committed, 0 aborted, 0 restarted, 10 logged to disc
0 held locks, 0 in queue; 0 local transactions, 0 remote
0 transactions waits for other nodes: []
:ok
iex(node1@127.0.0.1)32>
“schema” + “books” tables exist
on 3 different nodes
3 nodes are running
current node (node1)
keeps 2 tables as RAM + disk

113.
Before we
proceed…

114.
CAP theorem!

115.
@b0rk

116.
Consistency
Every read receives the
most recent write or an error

117.
Availability
Every request receives a response,
without guarantee that
it contains the most recent
version of the information

118.
Partition tolerance
The system continues to operate despite
an arbitrary number of messages being
dropped by the network between nodes

119.
Pick two!
AP or AC or CP

120.
AC
is kind of
pointless

121.
Mnesia chooses…

122.
AC!

123.
If in your cluster the network
connection between two nodes
goes bad, then each one
will think that the other node is down,
and continue to write data.
Recovery from this is complicated.

124.
AXD 301
switch

125.
“…measures are taken such
that network reliability is very high”

126.
“…In such a highly specialized
environment, the reliability of the control
backplane essentially removes some of
the worries which the CAP theorem
introduces.”

127.
Databases
Tools
Theory
Takeaways

128.
1. Elixir lowers the
barrier of entrance
in building clusters

129.
…via productivity
batteries!

130.
And yet it’s all
about Erlang

131.
2. “Hello world” for
clusters is simple

132.
3. Deciding what
matters is hard

133.
Understahd your
values when
building a cluster!

134.
4. Releasing &
deploying stuff
may get tricky

135.
5. Building stateful
clusters is really
challanging

136.
6. An Erlang app
can be your little
universe