Successfully reported this slideshow.

Magic Clusters and Where to Find Them - Eugene Pirogov

1

Share

1 of 167
1 of 167

Magic Clusters and Where to Find Them - Eugene Pirogov

1

Share

Download to read offline

The talk focuses on Erlang & Elixir facilities for distributed computing.
We'll explore how to build clusters and look at how testing a clustered setup might looks like. We'll create a Mnesia cluster and will talk about CAP theorem. We'll answer the questions of what distributed computing is for, and what Erlang has to offer with regards to balancing the load between the nodes in a cluster.
Experience this and much more in the "Magic Clusters and Where to Find Them" talk!
Elixir Meetup 4 Lviv

The talk focuses on Erlang & Elixir facilities for distributed computing.
We'll explore how to build clusters and look at how testing a clustered setup might looks like. We'll create a Mnesia cluster and will talk about CAP theorem. We'll answer the questions of what distributed computing is for, and what Erlang has to offer with regards to balancing the load between the nodes in a cluster.
Experience this and much more in the "Magic Clusters and Where to Find Them" talk!
Elixir Meetup 4 Lviv

More Related Content

More from Elixir Club

Related Books

Free with a 14 day trial from Scribd

See all

Related Audiobooks

Free with a 14 day trial from Scribd

See all

Magic Clusters and Where to Find Them - Eugene Pirogov

  1. 1. Clusters and where to find them
  2. 2. Eugene Pirogov gmile
  3. 3. Nebo15 Were hiring!
  4. 4. Databases Tools Theory Takeaways
  5. 5. Databases Tools Theory Takeaways
  6. 6. What is a cluster?
  7. 7. set of loosely or tightly connected computers that work together so that, in many respects, they can be viewed as a single system
  8. 8. When to use a cluster?
  9. 9. 1. Fail-over clusters
  10. 10. 2. Load balancing clusters
  11. 11. What typical Erlang cluster is built with?
  12. 12. 1. A node
  13. 13. node()
  14. 14. 2. An RPC call
  15. 15. :rpc.call(:nodex, M, :f, [“a”])
  16. 16. 3. send call
  17. 17. send({MyProcess, :mynode}, :msg)
  18. 18. Example 1: Starting a node
  19. 19. iex
  20. 20. ~> iex Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async- threads:10] [hipe] [kernel-poll:false] [dtrace] Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help) iex(1)> node() :nonode@nohost iex(2)>
  21. 21. iex --name eugene
  22. 22. ~> iex --name eugene Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async- threads:10] [hipe] [kernel-poll:false] [dtrace] Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help) iex(eugene@Eugenes-MacBook-Pro-2.local)1>
  23. 23. iex --sname eugene
  24. 24. ~> iex --sname eugene Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async- threads:10] [hipe] [kernel-poll:false] [dtrace] Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help) iex(eugene@Eugenes-MacBook-Pro-2)1>
  25. 25. iex --name eugene@host
  26. 26. ~> iex --name eugene@host Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async- threads:10] [hipe] [kernel-poll:false] [dtrace] Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help) iex(eugene@host)1>
  27. 27. Example 2: starting two nodes
  28. 28. iex --name node1@127.0.0.1 iex --name node2@127.0.0.1
  29. 29. ~> iex --name node1@127.0.0.1 iex(node1@127.0.0.1)1> ~> iex --name node2@127.0.0.1 iex(node2@127.0.0.1)1> # On node1 iex(1)> :net_adm.ping(:’node2@127.0.0.1’) :pong
  30. 30. Example 3: sending a message
  31. 31. iex --name node1 iex --name node2
  32. 32. # On node2 iex(1)> Process.register(Terminal, self()) # On node1 iex(1)> send({Terminal, :’node2@127.0.0.1’}, “Hello!”) # On node2 iex(2)> flush() “Hello!”
  33. 33. Example 4: calling remotely
  34. 34. # On node1 iex(node1@127.0.0.1)1> :rpc.call(:'node2@127.0.0.1', Enum, :reverse, [100..1]) [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, …] iex(node1@127.0.0.1)2>
  35. 35. REST/JSON/XML Binary protocol
  36. 36. REST/JSON/XML Binary protocol
  37. 37. REST/JSON/XML Binary protocol
  38. 38. Bonus track: Magic cookie!
  39. 39. cat ~/.erlang.cookie
  40. 40. iex --name node1 --cookie abc iex --name node2 --cookie xyz
  41. 41. # On node1 iex(1)> :erlang.get_cookie() :abc # On node2 iex(1)> :erlang.get_cookie() :xyz # On node1 iex(2)> :net_kernel.connect(:'node2@127.0.01') false # On node1 iex(3)> :erlang.set_cookie(:’node1@127.0.01’, :xyz) true # On node1 iex(4)> :net_kernel.connect(:'node2@127.0.01') true
  42. 42. Databases Tools Theory Takeaways
  43. 43. epmd
  44. 44. Erlang Port Mapper Daemon
  45. 45. runs on system startup
  46. 46. ~> ps ax | grep epmd 25502 ?? S 0:11.53 /usr/local/Cellar/erlang/19.1/lib/ erlang/erts-8.1/bin/epmd -daemon
  47. 47. maps ports to node names
  48. 48. net_kernel
  49. 49. Example 5: starting a distributed node
  50. 50. iex
  51. 51. ~> iex Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async- threads:10] [hipe] [kernel-poll:false] [dtrace] Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help) iex(1)>
  52. 52. ~> iex Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async- threads:10] [hipe] [kernel-poll:false] [dtrace] Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help) iex(1)> node() :nonode@nohost iex(2)>
  53. 53. ~> iex Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async- threads:10] [hipe] [kernel-poll:false] [dtrace] Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help) iex(1)> node() :nonode@nohost iex(2)> Process.registered() |> Enum.find(&(&1 == :net_kernel)) nil
  54. 54. ~> iex Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async- threads:10] [hipe] [kernel-poll:false] [dtrace] Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help) iex(1)> node() :nonode@nohost iex(2)> Process.registered() |> Enum.find(&(&1 == :net_kernel)) nil iex(3)> :net_kernel.start([:’mynode@127.0.0.1’]) {:ok, #PID<0.84.0>} iex(mynode@127.0.0.1)4>
  55. 55. ~> iex Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async- threads:10] [hipe] [kernel-poll:false] [dtrace] Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help) iex(1)> node() :nonode@nohost iex(2)> Process.registered() |> Enum.find(&(&1 == :net_kernel)) nil iex(3)> :net_kernel.start([:’mynode@127.0.0.1’]) {:ok, #PID<0.84.0>} iex(mynode@127.0.0.1)4> Process.registered() |> Enum.find(&(&1 == :net_kernel)) :net_kernel
  56. 56. Example 6: monitoring a node
  57. 57. iex --name node1@127.0.0.1 iex --name node2@127.0.0.1
  58. 58. iex(node1@127.0.0.1)1>
  59. 59. iex(node1@127.0.0.1)1> iex(node1@127.0.0.1)2> :net_kernel.monitor_nodes(true) :ok
  60. 60. iex(node1@127.0.0.1)1> iex(node1@127.0.0.1)2> :net_kernel.monitor_nodes(true) :ok iex(node2@127.0.0.1)3> :net_kernel.connect(:'node1@127.0.0.1') true
  61. 61. iex(node1@127.0.0.1)1> iex(node1@127.0.0.1)2> :net_kernel.monitor_nodes(true) :ok iex(node1@127.0.0.1)3> :net_kernel.connect(:'node2@127.0.0.1') true iex(node1@127.0.0.1)4> flush() {:nodeup, :"node2@127.0.0.1"} :ok
  62. 62. iex(node1@127.0.0.1)1> iex(node1@127.0.0.1)2> :net_kernel.monitor_nodes(true) :ok iex(node1@127.0.0.1)3> :net_kernel.connect(:'node2@127.0.0.1') true iex(node1@127.0.0.1)4> flush() {:nodeup, :"node2@127.0.0.1"} :ok # Ctrl+C on node2
  63. 63. iex(node1@127.0.0.1)1> iex(node1@127.0.0.1)2> :net_kernel.monitor_nodes(true) :ok iex(node1@127.0.0.1)3> :net_kernel.connect(:'node2@127.0.0.1') true iex(node1@127.0.0.1)4> flush() {:nodeup, :"node2@127.0.0.1"} :ok # Ctrl+C on node2 iex(node1@127.0.0.1)5> flush() {:nodedown, :"node2@127.0.0.1"} :ok iex(node1@127.0.0.1)5>
  64. 64. net_adm
  65. 65. Example 7: ping
  66. 66. iex --name node1@127.0.0.1 iex --name node2@127.0.0.1
  67. 67. iex(node1@127.0.0.1)1>
  68. 68. iex(node1@127.0.0.1)1> iex(node1@127.0.0.1)2> :net_adm.ping(:'node3@127.0.0.1') pang
  69. 69. iex(node1@127.0.0.1)1> iex(node1@127.0.0.1)2> :net_adm.ping(:'node3@127.0.0.1') pang iex(node1@127.0.0.1)2> :net_adm.ping(:'node2@127.0.0.1') pong
  70. 70. Example 8: names
  71. 71. iex --name node1@127.0.0.1 iex --name node2@127.0.0.1
  72. 72. iex(node1@127.0.0.1)1>
  73. 73. iex(node1@127.0.0.1)1> iex(node1@127.0.0.1)2> :net_adm.names() {:ok, [{'rabbit', 25672}, {'node1', 51813}, {'node2', 51815}]} iex(node1@127.0.0.1)3>
  74. 74. iex(node1@127.0.0.1)1> iex(node1@127.0.0.1)2> :net_adm.names() {:ok, [{'rabbit', 25672}, {'node1', 51813}, {'node2', 51815}]} iex(node1@127.0.0.1)3> Node.list() [] iex(node1@127.0.0.1)4>
  75. 75. Example 9: world
  76. 76. # /etc/hosts 127.0.0.1 host1.com 127.0.0.1 host2.com 127.0.0.1 host3.com
  77. 77. # /Users/gmile/.hosts.erlang host1.com. host2.com. host3.com.
  78. 78. iex --name node1@host1.com iex --name node2@host1.com iex --name node3@host2.com iex --name node4@host2.com iex --name node5@host3.com iex --name node6@host3.com
  79. 79. iex(node1@host1.com)1>
  80. 80. iex(node1@host1.com)1> iex(node1@host1.com)1> :net_adm.world() [:"node1@host1.com", :"node2@host1.com", :"node3@host2.com", :”no de4@host2.com", :"node5@host3.com", :"node6@host3.com"] iex(node1@host1.com)2>
  81. 81. Bonus track: Connecting to a node running in production
  82. 82. iex --name node1@127.0.0.1 --cookie abc
  83. 83. $ iex --remsh foo@127.0.0.1 --cookie abc -- name bar@localhost Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel- poll:false] [dtrace] Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help) iex(foo@127.0.0.1)1>
  84. 84. slave
  85. 85. 3. Transfer configuration to slave nodes 2. Add code path to slave nodes 4. Ensure apps are started on slave 1. Start slave
  86. 86. What else?
  87. 87. Node
  88. 88. phoenixframework/ firenest
  89. 89. bitwalker/swarm
  90. 90. Easy clustering, registration, and distribution of worker processes for Erlang/Elixir
  91. 91. …a simple case where workers are dynamically created in response to some events under a supervisor, and we want them to be distributed across the cluster and be discoverable by name from anywhere in the cluster
  92. 92. bitwalker/ libcluster
  93. 93. What next?
  94. 94. …I didn’t want to resort to something like Docker, because I wanted to see how far I could push Elixir and its tooling.
  95. 95. Databases Tools Theory Takeaways
  96. 96. mnesia
  97. 97. Example 10: initialize mnesia
  98. 98. iex --name node1@127.0.0.1 iex --name node2@127.0.0.1 iex --name node3@127.0.0.1
  99. 99. iex(node1@127.0.0.1)1> :mnesia.create_schema([:'node1@127.0.0.1']) :ok
  100. 100. iex(node1@127.0.0.1)1> :mnesia.create_schema([:'node1@127.0.0.1']) :ok iex(node1@127.0.0.1)2> :mnesia.start() :ok
  101. 101. iex(node1@127.0.0.1)1> :mnesia.create_schema([:'node1@127.0.0.1']) :ok iex(node1@127.0.0.1)2> :mnesia.start() :ok iex(node1@127.0.0.1)3> :mnesia.info() ---> Processes holding locks <--- ---> Processes waiting for locks <--- ---> Participant transactions <--- ---> Coordinator transactions <--- ---> Uncertain transactions <--- ---> Active tables <--- schema : with 1 records occupying 413 words of mem ===> System info in version "4.14.1", debug level = none <=== opt_disc. Directory "/Users/gmile/Mnesia.node1@127.0.0.1" is used. use fallback at restart = false running db nodes = ['node1@127.0.0.1'] stopped db nodes = [] master node tables = [] remote = [] ram_copies = [] disc_copies = [schema] disc_only_copies = [] [{'node1@127.0.0.1',disc_copies}] = [schema] 2 transactions committed, 0 aborted, 0 restarted, 0 logged to disc 0 held locks, 0 in queue; 0 local transactions, 0 remote 0 transactions waits for other nodes: [] :ok iex(node1@127.0.0.1)4>
  102. 102. iex(node1@127.0.0.1)1> :mnesia.create_schema([:'node1@127.0.0.1']) :ok iex(node1@127.0.0.1)2> :mnesia.start() :ok iex(node1@127.0.0.1)3> :mnesia.info() ---> Processes holding locks <--- ---> Processes waiting for locks <--- ---> Participant transactions <--- ---> Coordinator transactions <--- ---> Uncertain transactions <--- ---> Active tables <--- schema : with 1 records occupying 413 words of mem ===> System info in version "4.14.1", debug level = none <=== opt_disc. Directory "/Users/gmile/Mnesia.node1@127.0.0.1" is used. use fallback at restart = false running db nodes = ['node1@127.0.0.1'] stopped db nodes = [] master node tables = [] remote = [] ram_copies = [] disc_copies = [schema] disc_only_copies = [] [{'node1@127.0.0.1',disc_copies}] = [schema] 2 transactions committed, 0 aborted, 0 restarted, 0 logged to disc 0 held locks, 0 in queue; 0 local transactions, 0 remote 0 transactions waits for other nodes: [] :ok iex(node1@127.0.0.1)4> “schema” table exists as a disk_copy (RAM + disk) on node1@127.0.0.1
  103. 103. iex(node2@127.0.0.1)2> :mnesia.start() :ok
  104. 104. iex(node2@127.0.0.1)2> :mnesia.start() :ok iex(node3@127.0.0.1)2> :mnesia.start() :ok
  105. 105. iex(node2@127.0.0.1)2> :mnesia.start() :ok iex(node3@127.0.0.1)2> :mnesia.start() :ok iex(node1@127.0.0.1)2> :mnesia.change_config(:extra_db_nodes, [:’node2@127.0.0.1’]) {:ok, [:"node2@127.0.0.1"]}
  106. 106. iex(node2@127.0.0.1)2> :mnesia.start() :ok iex(node3@127.0.0.1)2> :mnesia.start() :ok iex(node1@127.0.0.1)2> :mnesia.change_config(:extra_db_nodes, [:’node2@127.0.0.1’]) {:ok, [:"node2@127.0.0.1"]} iex(node1@127.0.0.1)3> :mnesia.change_config(:extra_db_nodes, [:’node3@127.0.0.1’]) {:ok, [:"node3@127.0.0.1"]}
  107. 107. iex(node2@127.0.0.1)2> :mnesia.start() :ok iex(node3@127.0.0.1)2> :mnesia.start() :ok iex(node1@127.0.0.1)2> :mnesia.change_config(:extra_db_nodes, [:’node2@127.0.0.1’]) {:ok, [:"node2@127.0.0.1"]} iex(node1@127.0.0.1)3> :mnesia.change_config(:extra_db_nodes, [:’node3@127.0.0.1’]) {:ok, [:"node3@127.0.0.1"]} iex(node1@127.0.0.1)1> :mnesia.create_table(:books, [disc_copies: [:'node1@127.0.0.1'], attributes: [:id, :title, :year]]) :ok
  108. 108. iex(node2@127.0.0.1)2> :mnesia.start() :ok iex(node3@127.0.0.1)2> :mnesia.start() :ok iex(node1@127.0.0.1)2> :mnesia.change_config(:extra_db_nodes, [:’node2@127.0.0.1’]) {:ok, [:"node2@127.0.0.1"]} iex(node1@127.0.0.1)3> :mnesia.change_config(:extra_db_nodes, [:’node3@127.0.0.1’]) {:ok, [:"node3@127.0.0.1"]} iex(node1@127.0.0.1)1> :mnesia.create_table(:books, [disc_copies: [:'node1@127.0.0.1'], attributes: [:id, :title, :year]]) :ok iex(node1@127.0.0.1)4> :mnesia.add_table_copy(:books, :'node2@127.0.0.1', :ram_copies) :ok
  109. 109. iex(node2@127.0.0.1)2> :mnesia.start() :ok iex(node3@127.0.0.1)2> :mnesia.start() :ok iex(node1@127.0.0.1)2> :mnesia.change_config(:extra_db_nodes, [:’node2@127.0.0.1’]) {:ok, [:"node2@127.0.0.1"]} iex(node1@127.0.0.1)3> :mnesia.change_config(:extra_db_nodes, [:’node3@127.0.0.1’]) {:ok, [:"node3@127.0.0.1"]} iex(node1@127.0.0.1)1> :mnesia.create_table(:books, [disc_copies: [:'node1@127.0.0.1'], attributes: [:id, :title, :year]]) :ok iex(node1@127.0.0.1)4> :mnesia.add_table_copy(:books, :'node2@127.0.0.1', :ram_copies) :ok iex(node1@127.0.0.1)5> :mnesia.add_table_copy(:books, :'node3@127.0.0.1', :ram_copies) :ok
  110. 110. iex(node1@127.0.0.1)6> :mnesia.info() ---> Processes holding locks <--- ---> Processes waiting for locks <--- ---> Participant transactions <--- ---> Coordinator transactions <--- ---> Uncertain transactions <--- ---> Active tables <--- books : with 0 records occupying 304 words of mem schema : with 2 records occupying 566 words of mem ===> System info in version "4.14.1", debug level = none <=== opt_disc. Directory "/Users/gmile/Mnesia.node1@127.0.0.1" is used. use fallback at restart = false running db nodes = ['node3@127.0.0.1','node2@127.0.0.1','node1@127.0.0.1'] stopped db nodes = [] master node tables = [] remote = [] ram_copies = [] disc_copies = [books,schema] disc_only_copies = [] [{'node1@127.0.0.1',disc_copies}, {'node2@127.0.0.1',ram_copies}, {'node3@127.0.0.1',ram_copies}] = [schema,books] 12 transactions committed, 0 aborted, 0 restarted, 10 logged to disc 0 held locks, 0 in queue; 0 local transactions, 0 remote 0 transactions waits for other nodes: [] :ok iex(node1@127.0.0.1)32>
  111. 111. iex(node1@127.0.0.1)6> :mnesia.info() ---> Processes holding locks <--- ---> Processes waiting for locks <--- ---> Participant transactions <--- ---> Coordinator transactions <--- ---> Uncertain transactions <--- ---> Active tables <--- books : with 0 records occupying 304 words of mem schema : with 2 records occupying 566 words of mem ===> System info in version "4.14.1", debug level = none <=== opt_disc. Directory "/Users/gmile/Mnesia.node1@127.0.0.1" is used. use fallback at restart = false running db nodes = ['node3@127.0.0.1','node2@127.0.0.1','node1@127.0.0.1'] stopped db nodes = [] master node tables = [] remote = [] ram_copies = [] disc_copies = [books,schema] disc_only_copies = [] [{'node1@127.0.0.1',disc_copies}, {'node2@127.0.0.1',ram_copies}, {'node3@127.0.0.1',ram_copies}] = [schema,books] 12 transactions committed, 0 aborted, 0 restarted, 10 logged to disc 0 held locks, 0 in queue; 0 local transactions, 0 remote 0 transactions waits for other nodes: [] :ok iex(node1@127.0.0.1)32> “schema” + “books” tables exist on 3 different nodes 3 nodes are running current node (node1) keeps 2 tables as RAM + disk
  112. 112. Before we proceed…
  113. 113. CAP theorem!
  114. 114. @b0rk
  115. 115. Consistency Every read receives the most recent write or an error
  116. 116. Availability Every request receives a response, without guarantee that it contains the most recent version of the information
  117. 117. Partition tolerance The system continues to operate despite an arbitrary number of messages being dropped by the network between nodes
  118. 118. Pick two! AP or AC or CP
  119. 119. AC is kind of pointless
  120. 120. Mnesia chooses…
  121. 121. AC!
  122. 122. If in your cluster the network connection between two nodes goes bad, then each one will think that the other node is down, and continue to write data. Recovery from this is complicated.
  123. 123. AXD 301 switch
  124. 124. “…measures are taken such that network reliability is very high”
  125. 125. “…In such a highly specialized environment, the reliability of the control backplane essentially removes some of the worries which the CAP theorem introduces.”
  126. 126. Databases Tools Theory Takeaways
  127. 127. 1. Elixir doesn’t bring anything new to the table…
  128. 128. …apart from productivity!
  129. 129. It’s all about Erlang
  130. 130. 2. “Hello world” for clusters is simple
  131. 131. 3. Deciding what matters is hard
  132. 132. Understahd your values when building a cluster!
  133. 133. 4. Releasing & deploying stuff is tricky
  134. 134. 5. An Erlang app can be your little universe

×