Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Magic Clusters and Where to Find Them 2.0 - Eugene Pirogov

228 views

Published on

Elixir Club 5

Published in: Software
  • Be the first to comment

  • Be the first to like this

Magic Clusters and Where to Find Them 2.0 - Eugene Pirogov

  1. 1. Clusters and where to find them
  2. 2. Eugene Pirogov gmile
  3. 3. Databases Tools Theory Takeaways
  4. 4. Databases Tools Theory Takeaways
  5. 5. What is a cluster?
  6. 6. set of loosely or tightly connected computers that work together so that, in many respects, they can be viewed as a single system
  7. 7. When to use a cluster?
  8. 8. 1. Fail-over clusters
  9. 9. 2. Load balancing clusters
  10. 10. What typical Erlang cluster is built with?
  11. 11. 1. A node
  12. 12. node()
  13. 13. 2. An RPC call
  14. 14. :rpc.call(:nodex, M, :f, [“a”])
  15. 15. 3. send call
  16. 16. send({MyProcess, :mynode}, :msg)
  17. 17. Example 1: Starting a node
  18. 18. iex
  19. 19. ~> iex Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async- threads:10] [hipe] [kernel-poll:false] [dtrace] Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help) iex(1)> node() :nonode@nohost iex(2)>
  20. 20. iex --name eugene
  21. 21. ~> iex --name eugene Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async- threads:10] [hipe] [kernel-poll:false] [dtrace] Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help) iex(eugene@Eugenes-MacBook-Pro-2.local)1>
  22. 22. iex --sname eugene
  23. 23. ~> iex --sname eugene Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async- threads:10] [hipe] [kernel-poll:false] [dtrace] Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help) iex(eugene@Eugenes-MacBook-Pro-2)1>
  24. 24. iex --name eugene@host
  25. 25. ~> iex --name eugene@host Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async- threads:10] [hipe] [kernel-poll:false] [dtrace] Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help) iex(eugene@host)1>
  26. 26. Example 2: starting two nodes
  27. 27. iex --name node1@127.0.0.1 iex --name node2@127.0.0.1
  28. 28. ~> iex --name node1@127.0.0.1 iex(node1@127.0.0.1)1> ~> iex --name node2@127.0.0.1 iex(node2@127.0.0.1)1> # On node1 iex(1)> :net_adm.ping(:’node2@127.0.0.1’) :pong
  29. 29. Example 3: sending a message
  30. 30. iex --name node1 iex --name node2
  31. 31. # On node2 iex(1)> Process.register(Terminal, self()) # On node1 iex(1)> send({Terminal, :’node2@127.0.0.1’}, “Hello!”) # On node2 iex(2)> flush() “Hello!”
  32. 32. Example 4: calling remotely
  33. 33. # On node1 iex(node1@127.0.0.1)1> :rpc.call(:'node2@127.0.0.1', Enum, :reverse, [100..1]) [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, …] iex(node1@127.0.0.1)2>
  34. 34. REST/JSON/XML Binary protocol
  35. 35. REST/JSON/XML Binary protocol
  36. 36. REST/JSON/XML Binary protocol
  37. 37. Bonus track: Magic cookie!
  38. 38. cat ~/.erlang.cookie
  39. 39. iex --name node1 --cookie abc iex --name node2 --cookie xyz
  40. 40. # On node1 iex(1)> :erlang.get_cookie() :abc # On node2 iex(1)> :erlang.get_cookie() :xyz # On node1 iex(2)> :net_kernel.connect(:'node2@127.0.01') false # On node1 iex(3)> :erlang.set_cookie(:’node1@127.0.01’, :xyz) true # On node1 iex(4)> :net_kernel.connect(:'node2@127.0.01') true
  41. 41. Databases Tools Theory Takeaways
  42. 42. epmd
  43. 43. Erlang Port Mapper Daemon
  44. 44. runs on system startup
  45. 45. ~> ps ax | grep epmd 25502 ?? S 0:11.53 /usr/local/Cellar/erlang/19.1/lib/ erlang/erts-8.1/bin/epmd -daemon
  46. 46. maps ports to node names
  47. 47. net_kernel
  48. 48. Example 5: starting a distributed node
  49. 49. iex
  50. 50. ~> iex Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async- threads:10] [hipe] [kernel-poll:false] [dtrace] Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help) iex(1)>
  51. 51. ~> iex Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async- threads:10] [hipe] [kernel-poll:false] [dtrace] Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help) iex(1)> node() :nonode@nohost iex(2)>
  52. 52. ~> iex Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async- threads:10] [hipe] [kernel-poll:false] [dtrace] Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help) iex(1)> node() :nonode@nohost iex(2)> Process.registered() |> Enum.find(&(&1 == :net_kernel)) nil
  53. 53. ~> iex Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async- threads:10] [hipe] [kernel-poll:false] [dtrace] Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help) iex(1)> node() :nonode@nohost iex(2)> Process.registered() |> Enum.find(&(&1 == :net_kernel)) nil iex(3)> :net_kernel.start([:’mynode@127.0.0.1’]) {:ok, #PID<0.84.0>} iex(mynode@127.0.0.1)4>
  54. 54. ~> iex Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async- threads:10] [hipe] [kernel-poll:false] [dtrace] Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help) iex(1)> node() :nonode@nohost iex(2)> Process.registered() |> Enum.find(&(&1 == :net_kernel)) nil iex(3)> :net_kernel.start([:’mynode@127.0.0.1’]) {:ok, #PID<0.84.0>} iex(mynode@127.0.0.1)4> Process.registered() |> Enum.find(&(&1 == :net_kernel)) :net_kernel
  55. 55. Example 6: monitoring a node
  56. 56. iex --name node1@127.0.0.1 iex --name node2@127.0.0.1
  57. 57. iex(node1@127.0.0.1)1>
  58. 58. iex(node1@127.0.0.1)1> iex(node1@127.0.0.1)2> :net_kernel.monitor_nodes(true) :ok
  59. 59. iex(node1@127.0.0.1)1> iex(node1@127.0.0.1)2> :net_kernel.monitor_nodes(true) :ok iex(node2@127.0.0.1)3> :net_kernel.connect(:'node1@127.0.0.1') true
  60. 60. iex(node1@127.0.0.1)1> iex(node1@127.0.0.1)2> :net_kernel.monitor_nodes(true) :ok iex(node1@127.0.0.1)3> :net_kernel.connect(:'node2@127.0.0.1') true iex(node1@127.0.0.1)4> flush() {:nodeup, :"node2@127.0.0.1"} :ok
  61. 61. iex(node1@127.0.0.1)1> iex(node1@127.0.0.1)2> :net_kernel.monitor_nodes(true) :ok iex(node1@127.0.0.1)3> :net_kernel.connect(:'node2@127.0.0.1') true iex(node1@127.0.0.1)4> flush() {:nodeup, :"node2@127.0.0.1"} :ok # Ctrl+C on node2
  62. 62. iex(node1@127.0.0.1)1> iex(node1@127.0.0.1)2> :net_kernel.monitor_nodes(true) :ok iex(node1@127.0.0.1)3> :net_kernel.connect(:'node2@127.0.0.1') true iex(node1@127.0.0.1)4> flush() {:nodeup, :"node2@127.0.0.1"} :ok # Ctrl+C on node2 iex(node1@127.0.0.1)5> flush() {:nodedown, :"node2@127.0.0.1"} :ok iex(node1@127.0.0.1)5>
  63. 63. net_adm
  64. 64. Example 7: ping
  65. 65. iex --name node1@127.0.0.1 iex --name node2@127.0.0.1
  66. 66. iex(node1@127.0.0.1)1>
  67. 67. iex(node1@127.0.0.1)1> iex(node1@127.0.0.1)2> :net_adm.ping(:'node3@127.0.0.1') pang
  68. 68. iex(node1@127.0.0.1)1> iex(node1@127.0.0.1)2> :net_adm.ping(:'node3@127.0.0.1') pang iex(node1@127.0.0.1)2> :net_adm.ping(:'node2@127.0.0.1') pong
  69. 69. Example 8: names
  70. 70. iex --name node1@127.0.0.1 iex --name node2@127.0.0.1
  71. 71. iex(node1@127.0.0.1)1>
  72. 72. iex(node1@127.0.0.1)1> iex(node1@127.0.0.1)2> :net_adm.names() {:ok, [{'rabbit', 25672}, {'node1', 51813}, {'node2', 51815}]} iex(node1@127.0.0.1)3>
  73. 73. iex(node1@127.0.0.1)1> iex(node1@127.0.0.1)2> :net_adm.names() {:ok, [{'rabbit', 25672}, {'node1', 51813}, {'node2', 51815}]} iex(node1@127.0.0.1)3> Node.list() [] iex(node1@127.0.0.1)4>
  74. 74. Example 9: world
  75. 75. # /etc/hosts 127.0.0.1 host1.com 127.0.0.1 host2.com 127.0.0.1 host3.com
  76. 76. # /Users/gmile/.hosts.erlang host1.com. host2.com. host3.com.
  77. 77. iex --name node1@host1.com iex --name node2@host1.com iex --name node3@host2.com iex --name node4@host2.com iex --name node5@host3.com iex --name node6@host3.com
  78. 78. iex(node1@host1.com)1>
  79. 79. iex(node1@host1.com)1> iex(node1@host1.com)1> :net_adm.world() [:"node1@host1.com", :"node2@host1.com", :"node3@host2.com", :”no de4@host2.com", :"node5@host3.com", :"node6@host3.com"] iex(node1@host1.com)2>
  80. 80. Bonus track: Connecting to a node running in production
  81. 81. iex --name node1@127.0.0.1 --cookie abc
  82. 82. $ iex --remsh foo@127.0.0.1 --cookie abc -- name bar@localhost Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel- poll:false] [dtrace] Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help) iex(foo@127.0.0.1)1>
  83. 83. $ kubectl get pods -l app=matcher -o template -- template="{{range.items}}{{.metadata.name}}{{end}}" | xargs -o -I my_pod kubectl exec my_pod -i -t -- iex --name debugging@127.0.0.1 --remsh marketplace@127.0.0.1 --cookie marketplace
  84. 84. $ kubectl get pods -l app=matcher -o template -- template="{{range.items}}{{.metadata.name}}{{end}}" | xargs -o -I my_pod kubectl exec my_pod -i -t -- iex --name debugging@127.0.0.1 --remsh marketplace@127.0.0.1 --cookie marketplace
  85. 85. $ kubectl get pods -l app=matcher -o template -- template="{{range.items}}{{.metadata.name}}{{end}}" | xargs -o -I my_pod kubectl exec my_pod -i -t -- iex --name debugging@127.0.0.1 --remsh marketplace@127.0.0.1 --cookie marketplace
  86. 86. slave
  87. 87. 3. Transfer configuration to slave nodes 2. Add code path to slave nodes 4. Ensure apps are started on slave 1. Start slave
  88. 88. What else?
  89. 89. Node
  90. 90. bitwalker/swarm
  91. 91. Easy clustering, registration, and distribution of worker processes for Erlang/Elixir
  92. 92. …a simple case where workers are dynamically created in response to some events under a supervisor, and we want them to be distributed across the cluster and be discoverable by name from anywhere in the cluster
  93. 93. bitwalker/ libcluster
  94. 94. What next?
  95. 95. …I didn’t want to resort to something like Docker, because I wanted to see how far I could push Elixir and its tooling.
  96. 96. Databases Tools Theory Takeaways
  97. 97. mnesia
  98. 98. Example 10: initialize mnesia
  99. 99. iex --name node1@127.0.0.1 iex --name node2@127.0.0.1 iex --name node3@127.0.0.1
  100. 100. iex(node1@127.0.0.1)1> :mnesia.create_schema([:'node1@127.0.0.1']) :ok
  101. 101. iex(node1@127.0.0.1)1> :mnesia.create_schema([:'node1@127.0.0.1']) :ok iex(node1@127.0.0.1)2> :mnesia.start() :ok
  102. 102. iex(node1@127.0.0.1)1> :mnesia.create_schema([:'node1@127.0.0.1']) :ok iex(node1@127.0.0.1)2> :mnesia.start() :ok iex(node1@127.0.0.1)3> :mnesia.info() ---> Processes holding locks <--- ---> Processes waiting for locks <--- ---> Participant transactions <--- ---> Coordinator transactions <--- ---> Uncertain transactions <--- ---> Active tables <--- schema : with 1 records occupying 413 words of mem ===> System info in version "4.14.1", debug level = none <=== opt_disc. Directory "/Users/gmile/Mnesia.node1@127.0.0.1" is used. use fallback at restart = false running db nodes = ['node1@127.0.0.1'] stopped db nodes = [] master node tables = [] remote = [] ram_copies = [] disc_copies = [schema] disc_only_copies = [] [{'node1@127.0.0.1',disc_copies}] = [schema] 2 transactions committed, 0 aborted, 0 restarted, 0 logged to disc 0 held locks, 0 in queue; 0 local transactions, 0 remote 0 transactions waits for other nodes: [] :ok iex(node1@127.0.0.1)4>
  103. 103. iex(node1@127.0.0.1)1> :mnesia.create_schema([:'node1@127.0.0.1']) :ok iex(node1@127.0.0.1)2> :mnesia.start() :ok iex(node1@127.0.0.1)3> :mnesia.info() ---> Processes holding locks <--- ---> Processes waiting for locks <--- ---> Participant transactions <--- ---> Coordinator transactions <--- ---> Uncertain transactions <--- ---> Active tables <--- schema : with 1 records occupying 413 words of mem ===> System info in version "4.14.1", debug level = none <=== opt_disc. Directory "/Users/gmile/Mnesia.node1@127.0.0.1" is used. use fallback at restart = false running db nodes = ['node1@127.0.0.1'] stopped db nodes = [] master node tables = [] remote = [] ram_copies = [] disc_copies = [schema] disc_only_copies = [] [{'node1@127.0.0.1',disc_copies}] = [schema] 2 transactions committed, 0 aborted, 0 restarted, 0 logged to disc 0 held locks, 0 in queue; 0 local transactions, 0 remote 0 transactions waits for other nodes: [] :ok iex(node1@127.0.0.1)4> “schema” table exists as a disk_copy (RAM + disk) on node1@127.0.0.1
  104. 104. iex(node2@127.0.0.1)2> :mnesia.start() :ok
  105. 105. iex(node2@127.0.0.1)2> :mnesia.start() :ok iex(node3@127.0.0.1)2> :mnesia.start() :ok
  106. 106. iex(node2@127.0.0.1)2> :mnesia.start() :ok iex(node3@127.0.0.1)2> :mnesia.start() :ok iex(node1@127.0.0.1)2> :mnesia.change_config(:extra_db_nodes, [:’node2@127.0.0.1’]) {:ok, [:"node2@127.0.0.1"]}
  107. 107. iex(node2@127.0.0.1)2> :mnesia.start() :ok iex(node3@127.0.0.1)2> :mnesia.start() :ok iex(node1@127.0.0.1)2> :mnesia.change_config(:extra_db_nodes, [:’node2@127.0.0.1’]) {:ok, [:"node2@127.0.0.1"]} iex(node1@127.0.0.1)3> :mnesia.change_config(:extra_db_nodes, [:’node3@127.0.0.1’]) {:ok, [:"node3@127.0.0.1"]}
  108. 108. iex(node2@127.0.0.1)2> :mnesia.start() :ok iex(node3@127.0.0.1)2> :mnesia.start() :ok iex(node1@127.0.0.1)2> :mnesia.change_config(:extra_db_nodes, [:’node2@127.0.0.1’]) {:ok, [:"node2@127.0.0.1"]} iex(node1@127.0.0.1)3> :mnesia.change_config(:extra_db_nodes, [:’node3@127.0.0.1’]) {:ok, [:"node3@127.0.0.1"]} iex(node1@127.0.0.1)1> :mnesia.create_table(:books, [disc_copies: [:'node1@127.0.0.1'], attributes: [:id, :title, :year]]) :ok
  109. 109. iex(node2@127.0.0.1)2> :mnesia.start() :ok iex(node3@127.0.0.1)2> :mnesia.start() :ok iex(node1@127.0.0.1)2> :mnesia.change_config(:extra_db_nodes, [:’node2@127.0.0.1’]) {:ok, [:"node2@127.0.0.1"]} iex(node1@127.0.0.1)3> :mnesia.change_config(:extra_db_nodes, [:’node3@127.0.0.1’]) {:ok, [:"node3@127.0.0.1"]} iex(node1@127.0.0.1)1> :mnesia.create_table(:books, [disc_copies: [:'node1@127.0.0.1'], attributes: [:id, :title, :year]]) :ok iex(node1@127.0.0.1)4> :mnesia.add_table_copy(:books, :'node2@127.0.0.1', :ram_copies) :ok
  110. 110. iex(node2@127.0.0.1)2> :mnesia.start() :ok iex(node3@127.0.0.1)2> :mnesia.start() :ok iex(node1@127.0.0.1)2> :mnesia.change_config(:extra_db_nodes, [:’node2@127.0.0.1’]) {:ok, [:"node2@127.0.0.1"]} iex(node1@127.0.0.1)3> :mnesia.change_config(:extra_db_nodes, [:’node3@127.0.0.1’]) {:ok, [:"node3@127.0.0.1"]} iex(node1@127.0.0.1)1> :mnesia.create_table(:books, [disc_copies: [:'node1@127.0.0.1'], attributes: [:id, :title, :year]]) :ok iex(node1@127.0.0.1)4> :mnesia.add_table_copy(:books, :'node2@127.0.0.1', :ram_copies) :ok iex(node1@127.0.0.1)5> :mnesia.add_table_copy(:books, :'node3@127.0.0.1', :ram_copies) :ok
  111. 111. iex(node1@127.0.0.1)6> :mnesia.info() ---> Processes holding locks <--- ---> Processes waiting for locks <--- ---> Participant transactions <--- ---> Coordinator transactions <--- ---> Uncertain transactions <--- ---> Active tables <--- books : with 0 records occupying 304 words of mem schema : with 2 records occupying 566 words of mem ===> System info in version "4.14.1", debug level = none <=== opt_disc. Directory "/Users/gmile/Mnesia.node1@127.0.0.1" is used. use fallback at restart = false running db nodes = ['node3@127.0.0.1','node2@127.0.0.1','node1@127.0.0.1'] stopped db nodes = [] master node tables = [] remote = [] ram_copies = [] disc_copies = [books,schema] disc_only_copies = [] [{'node1@127.0.0.1',disc_copies}, {'node2@127.0.0.1',ram_copies}, {'node3@127.0.0.1',ram_copies}] = [schema,books] 12 transactions committed, 0 aborted, 0 restarted, 10 logged to disc 0 held locks, 0 in queue; 0 local transactions, 0 remote 0 transactions waits for other nodes: [] :ok iex(node1@127.0.0.1)32>
  112. 112. iex(node1@127.0.0.1)6> :mnesia.info() ---> Processes holding locks <--- ---> Processes waiting for locks <--- ---> Participant transactions <--- ---> Coordinator transactions <--- ---> Uncertain transactions <--- ---> Active tables <--- books : with 0 records occupying 304 words of mem schema : with 2 records occupying 566 words of mem ===> System info in version "4.14.1", debug level = none <=== opt_disc. Directory "/Users/gmile/Mnesia.node1@127.0.0.1" is used. use fallback at restart = false running db nodes = ['node3@127.0.0.1','node2@127.0.0.1','node1@127.0.0.1'] stopped db nodes = [] master node tables = [] remote = [] ram_copies = [] disc_copies = [books,schema] disc_only_copies = [] [{'node1@127.0.0.1',disc_copies}, {'node2@127.0.0.1',ram_copies}, {'node3@127.0.0.1',ram_copies}] = [schema,books] 12 transactions committed, 0 aborted, 0 restarted, 10 logged to disc 0 held locks, 0 in queue; 0 local transactions, 0 remote 0 transactions waits for other nodes: [] :ok iex(node1@127.0.0.1)32> “schema” + “books” tables exist on 3 different nodes 3 nodes are running current node (node1) keeps 2 tables as RAM + disk
  113. 113. Before we proceed…
  114. 114. CAP theorem!
  115. 115. @b0rk
  116. 116. Consistency Every read receives the most recent write or an error
  117. 117. Availability Every request receives a response, without guarantee that it contains the most recent version of the information
  118. 118. Partition tolerance The system continues to operate despite an arbitrary number of messages being dropped by the network between nodes
  119. 119. Pick two! AP or AC or CP
  120. 120. AC is kind of pointless
  121. 121. Mnesia chooses…
  122. 122. AC!
  123. 123. If in your cluster the network connection between two nodes goes bad, then each one will think that the other node is down, and continue to write data. Recovery from this is complicated.
  124. 124. AXD 301 switch
  125. 125. “…measures are taken such that network reliability is very high”
  126. 126. “…In such a highly specialized environment, the reliability of the control backplane essentially removes some of the worries which the CAP theorem introduces.”
  127. 127. Databases Tools Theory Takeaways
  128. 128. 1. Elixir lowers the barrier of entrance in building clusters
  129. 129. …via productivity batteries!
  130. 130. And yet it’s all about Erlang
  131. 131. 2. “Hello world” for clusters is simple
  132. 132. 3. Deciding what matters is hard
  133. 133. Understahd your values when building a cluster!
  134. 134. 4. Releasing & deploying stuff may get tricky
  135. 135. 5. Building stateful clusters is really challanging
  136. 136. 6. An Erlang app can be your little universe

×