Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Consul and Complex Networks

A systematic overview of Consul's different network models, how they work, what kind of use cases they serve, and how prepared queries can help provide glue to keep service discovery simple across all.

  • Login to see the comments

Consul and Complex Networks

  1. 1. Copyright © 2017 HashiCorp Consul and Complex Networks HashiConf US 2017, Austin Texas James Phillips, Consul Lead Engineer @slackpad
  2. 2. s Copyright © 2017 HashiCorp 2 Consul Overview
  3. 3. Copyright © 2017 HashiCorp ▪ Register services (IP:port + health checks) via config files or HTTP ▪ Discover services via DNS or HTTP ▪ Perform load balancing via DNS result shuffling ▪ Manage dynamic runtime configuration via key/value store ▪ Orchestrate via key/value store, events, watches, … ▪ Integrate with existing apps via DNS, HTTP, consul-template, envconsul, Fabio, … Service Discovery, Configuration, and Orchestration 3
  4. 4. Copyright © 2017 HashiCorp ▪ Run node- and service-specific health checks via Nagios compatible scripts, Docker, HTTP, and TCP ▪ TTL “dead person switch” checks Monitoring at Scale 4
  5. 5. Copyright © 2017 HashiCorp ▪ Fault tolerant via Raft consensus ▪ Federate multiple clusters easily ▪ Look up services locally or in other federated clusters ▪ Automatically apply failover policies across federated clusters ▪ Coordinate complex distributed systems via key/value locking primitives and sessions (eg. Vault) High Availability 5
  6. 6. Copyright © 2017 HashiCorp ▪ Consul agent is a single Go binary ▪ Agent runs on every machine in a cluster ▪ Applications always talk to their local Consul agent ▪ 3, 5, or 7 agents in a cluster run as servers, all others are clients ▪ Servers maintain a consistent view of the cluster state Running Consul 6
  7. 7. Copyright © 2017 HashiCorp ▪ Automated backups ▪ Automated upgrades ▪ Enhanced read scalability ▪ Redundancy zones ▪ Advanced network models Consul Enterprise 7
  8. 8. s Copyright © 2017 HashiCorp 8 Network Models
  9. 9. Copyright © 2017 HashiCorp ▪ Connect agents inside a datacenter ▪ One set of servers connected to a cluster of clients ▪ Two models available ▪ LAN Gossip ▪ Network Segments (Enterprise) Clustering 9
  10. 10. Copyright © 2017 HashiCorp ▪ Connect servers in different datacenters ▪ Clients don’t participate directly ▪ Two models available ▪ WAN Gossip ▪ Network Areas (Enterprise) Federation 10
  11. 11. Copyright © 2017 HashiCorp ▪ Set of agents with full mesh connectivity via UDP* and TCP ▪ SWIM algorithm via hashicorp/memberlist ▪ Distributed failure detector ▪ Fast gossip-based broadcast mechanism ▪ Shared list of all agents, anti-entropy combats drift ▪ Used all over Consul ▪ Clients finding available servers ▪ Edge-triggered push-updated health checks ▪ AP events feed CP Raft system on the leader update catalog ▪ ... Gossip Pools 11
  12. 12. Copyright © 2017 HashiCorp Gossip Pools In Action 12 B A X C Probe (UDP)
  13. 13. Copyright © 2017 HashiCorp Gossip Pools In Action 13 B A X C Ack (UDP)
  14. 14. Copyright © 2017 HashiCorp Gossip Pools In Action 14 B A X C Probe (TCP) Probe X ProbeX
  15. 15. Copyright © 2017 HashiCorp Gossip Pools In Action 15 B A X C Probe (TCP) Probe (UDP) Probe (UDP)
  16. 16. Copyright © 2017 HashiCorp Gossip Pools In Action 16 B A X C Nack Nack Probe (TCP) Probe (UDP) Probe (UDP)
  17. 17. Copyright © 2017 HashiCorp Gossip Pools In Action 17 X is suspect
  18. 18. Copyright © 2017 HashiCorp Gossip Pools In Action 18
  19. 19. Copyright © 2017 HashiCorp Gossip Pools In Action 19 X has failed
  20. 20. Copyright © 2017 HashiCorp Gossip Pools In Action 20 X is gone
  21. 21. Copyright © 2017 HashiCorp Gossip Pools In Action 21 Whoa
  22. 22. Copyright © 2017 HashiCorp Further Reading 22
  23. 23. Copyright © 2017 HashiCorp Concept of Operations 23 CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT SERVER SERVER SERVER
  24. 24. Copyright © 2017 HashiCorp Concept of Operations 24 CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT SERVER SERVER SERVER DC1
  25. 25. Copyright © 2017 HashiCorp Concept of Operations 25 CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT SERVER SERVER SERVER DC1
  26. 26. Copyright © 2017 HashiCorp Concept of Operations 26 CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT SERVER SERVER SERVER REPLICATION REPLICATION DC1
  27. 27. Copyright © 2017 HashiCorp Concept of Operations 27 CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT SERVER SERVER SERVER REPLICATION REPLICATION DC1 consul kv put hello world
  28. 28. Copyright © 2017 HashiCorp Concept of Operations 28 CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT SERVER SERVER SERVER REPLICATION DC1 consul kv put hello world
  29. 29. Copyright © 2017 HashiCorp Concept of Operations 29 CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT SERVER SERVER SERVER REPLICATION REPLICATION DC1 consul kv get hello
  30. 30. Copyright © 2017 HashiCorp Concept of Operations 30 CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT SERVER SERVER SERVER REPLICATION DC1 consul kv get hello
  31. 31. Copyright © 2017 HashiCorp Concept of Operations 31 CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT SERVER SERVER SERVER REPLICATION REPLICATION DC1
  32. 32. Copyright © 2017 HashiCorp Concept of Operations 32 CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT SERVER SERVER SERVER SERVER SERVER SERVER REPLICATION REPLICATION CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT REPLICATION REPLICATION DC1 DC2
  33. 33. Copyright © 2017 HashiCorp Concept of Operations 33 CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT SERVER SERVER SERVER SERVER SERVER SERVER REPLICATION REPLICATION CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT REPLICATION REPLICATION DC1 DC2
  34. 34. Copyright © 2017 HashiCorp Concept of Operations 34 CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT SERVER SERVER SERVER SERVER SERVER SERVER REPLICATION REPLICATION CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT REPLICATION REPLICATION DC1 DC2 consul kv put -datacenter=dc2 hello world
  35. 35. Copyright © 2017 HashiCorp Concept of Operations 35 CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT SERVER SERVER SERVER SERVER SERVER SERVER REPLICATION REPLICATION CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT REPLICATION REPLICATION DC1 DC2 consul kv put -datacenter=dc2 hello world
  36. 36. Copyright © 2017 HashiCorp Concept of Operations 36 CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT SERVER SERVER SERVER SERVER SERVER SERVER REPLICATION REPLICATION CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT REPLICATION REPLICATION DC1 DC2 consul kv put -datacenter=dc2 hello world
  37. 37. Copyright © 2017 HashiCorp Concept of Operations 37 CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT SERVER SERVER SERVER SERVER SERVER SERVER REPLICATION REPLICATION CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT REPLICATION REPLICATION DC1 DC2
  38. 38. s Copyright © 2017 HashiCorp 38 Simple Clustering: LAN Gossip
  39. 39. Copyright © 2017 HashiCorp ▪ Basic clustering for homogeneous networks of agents (small to massive scale) ▪ Services discoverable via DNS and HTTP ▪ Shared KV store with sessions ▪ Prepared queries support nearest neighbor routing within the datacenter via RTT LAN Gossip 39
  40. 40. Copyright © 2017 HashiCorp Use Case 40 Web Application with Multiple Services CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT SERVER SERVER SERVER REPLICATION REPLICATION DC1 web.service (Go) user.service (Rails) search.service (Java) order.service (Go) cache.service (Redis) db.service (Postgres)
  41. 41. Copyright © 2017 HashiCorp Use Case 41 Web Application with Multiple Services SERVER SERVER SERVER REPLICATION REPLICATION DC1 web.service user.servicce search.service order.service cache.service CLIENT db.service (Postgres)
  42. 42. Copyright © 2017 HashiCorp Use Case 42 Massive Compute Cluster with Nomad CLIENT CLIENT CLIENT CLIENT SERVER SERVER SERVER REPLICATION REPLICATION DC1 NOMAD SERVER NOMAD SERVER NOMAD SERVER NOMAD CLIENT 1000 CLIENT NOMAD CLIENT 1 …
  43. 43. Copyright © 2017 HashiCorp ▪ High speed, low latency network (also needed for Raft) ▪ Full mesh for all clients and servers on 8301/udp and 8301/tcp ▪ Gossip uses AES and shared key ▪ RPC uses TLS ▪ ACLs protect server state ▪ RPC rate limiter (new in 0.9.3) Network Topology, Security, and Isolation 43
  44. 44. Copyright © 2017 HashiCorp ▪ Manually via “consul join” command ▪ Automatic at agent startup via list of IPs or a DNS name ▪ Automatic at agent startup via cloud provider instance metadata ▪ Support for AWS, GCE, Azure, and SoftLayer ▪ Community developing more in hashicorp/go-discover Forming Clusters 44
  45. 45. Copyright © 2017 HashiCorp Manual Join 45 $ consul join 1.2.3.4 Terminal
  46. 46. Copyright © 2017 HashiCorp Manual Join 46 $ consul join 1.2.3.4 Successfully joined cluster by contacting 3 nodes. Terminal
  47. 47. Copyright © 2017 HashiCorp Automatic Join via DNS 47 $ consul agent -retry-join=consul.domain.internal Terminal
  48. 48. Copyright © 2017 HashiCorp Automatic Join via DNS 48 $ consul agent -retry-join=consul.domain.internal ==> Starting Consul agent... ==> Consul agent running! Version: 'v0.9.3-10-g7a2cd047' Node ID: 'df505b71-cd19-4051-8e40-e492392155fa' Node name: 'node-2' Datacenter: 'dc1' (Segment: '') Server: false (Bootstrap: false) Client Addr: 127.0.0.2 (HTTP: 8500, HTTPS: -1, DNS: 8600) Cluster Addr: 127.0.0.2 (LAN: 8301, WAN: 8302) Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false ==> Log data will now stream in as it occurs: 2017/09/16 23:12:37 [INFO] agent: Join LAN completed. Synced with 3 initial agents Terminal
  49. 49. Copyright © 2017 HashiCorp Automatic Join via Cloud Provider Metadata 49 $ consul agent -retry-join=‘provider=aws tag_key=consul tag_value=joiner’ Terminal
  50. 50. Copyright © 2017 HashiCorp Automatic Join via Cloud Provider Metadata 50 $ consul agent -retry-join=‘provider=aws tag_key=consul tag_value=joiner’ ==> Starting Consul agent... ==> Consul agent running! Version: 'v0.9.3-10-g7a2cd047' Node ID: 'df505b71-cd19-4051-8e40-e492392155fa' Node name: 'node-2' Datacenter: 'dc1' (Segment: '') Server: false (Bootstrap: false) Client Addr: 127.0.0.2 (HTTP: 8500, HTTPS: -1, DNS: 8600) Cluster Addr: 127.0.0.2 (LAN: 8301, WAN: 8302) Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false ==> Log data will now stream in as it occurs: 2017/09/16 23:12:33 [INFO] agent: Discovered LAN servers: 10.1.1.11 10.1.1.91 10.1.1.74 10.1.2.197 2017/09/16 23:12:37 [INFO] agent: Join LAN completed. Synced with 3 initial agents Terminal
  51. 51. Copyright © 2017 HashiCorp Completed Cluster 51 $ consul members Terminal
  52. 52. Copyright © 2017 HashiCorp Completed Cluster 52 $ consul members Node Address Status Type Build Protocol DC Segment consul-server-nyc3-1 104.131.14.250:8301 alive server 0.9.3rc1 2 nyc3 <all> consul-server-nyc3-2 104.131.180.172:8301 alive server 0.9.3rc1 2 nyc3 <all> consul-server-nyc3-3 104.131.31.159:8301 alive server 0.9.3rc1 2 nyc3 <all> consul-client-nyc3-1 104.131.86.98:8301 alive client 0.9.3rc1 2 nyc3 <default> consul-client-nyc3-2 104.131.75.130:8301 alive client 0.9.3rc1 2 nyc3 <default> consul-client-nyc3-3 104.236.9.114:8301 alive client 0.9.3rc1 2 nyc3 <default> Terminal
  53. 53. Copyright © 2017 HashiCorp ▪ SWIM’s gossip protocol has agents randomly probe each other ▪ This gives a nice set of RTT samples that are fed into a simple physics model ▪ The model allows the Consul servers to calculate estimated RTTs between agents, and to sort results by estimated RTT ▪ Exposed in APIs as ?near= parameter ▪ Exposed in DNS interface via prepared queries Gossip and Round Trip Times 53
  54. 54. Copyright © 2017 HashiCorp Nearest Neighbor Routing 54 $ curl -X POST -d ‘{ "Name": “nearest-", "Template": { "Type": "name_prefix_match" }, "Service": { "Service": "${name.suffix}" }, “Near”: “_agent” }’ 127.0.0.1:8500/v1/query {“ID”:"a4ef7a27-e189-9e5c-ae3a-e586eac88f00"} Terminal
  55. 55. Copyright © 2017 HashiCorp Nearest Neighbor Routing 55 $ dig +short cache.service.consul Terminal
  56. 56. Copyright © 2017 HashiCorp Nearest Neighbor Routing 56 $ dig +short cache.service.consul 10.1.0.1 10.1.0.2 10.1.0.3 Terminal
  57. 57. Copyright © 2017 HashiCorp Nearest Neighbor Routing 57 $ dig +short cache.service.consul 10.1.0.1 10.1.0.2 10.1.0.3 $ dig +short nearest-cache.query.consul Terminal
  58. 58. Copyright © 2017 HashiCorp Nearest Neighbor Routing 58 $ dig +short cache.service.consul 10.1.0.1 10.1.0.2 10.1.0.3 $ dig +short nearest-cache.query.consul 10.1.0.2 10.1.0.1 10.1.0.3 Terminal
  59. 59. s Copyright © 2017 HashiCorp 59 Simple Federation: WAN Gossip
  60. 60. Copyright © 2017 HashiCorp ▪ Basic federation for homogeneous networks of servers (small to massive scale) ▪ Service and KV state are local to each datacenter (no replication) ▪ Service and KV operations can be made to remote datacenters joined to the WAN Gossip pool ▪ Prepared queries support nearest neighbor routing between datacenters via RTT WAN Gossip 60
  61. 61. Copyright © 2017 HashiCorp Use Case 61 Geo Redundancy with Multiple Datacenters CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT SERVER SERVER SERVER nyc3 web.service user.service search.service order.service cache.service db.service CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT ams2web.service user.service search.service order.service cache.service db.service SERVER SERVER SERVER
  62. 62. Copyright © 2017 HashiCorp Use Case 62 Geo Redundancy with Multiple Datacenters CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT SERVER SERVER SERVER nyc3 web.service user.service search.service order.service cache.service db.service CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT ams2web.service user.service search.service order.service cache.service db.service SERVER SERVER SERVER
  63. 63. Copyright © 2017 HashiCorp Use Case 63 Isolation Using Multiple Datacenters CLIENT CLIENT VAULT SERVER VAULT SERVER CLIENT CLIENT PAYMENTS SERVICE PAYMENTS DB CLIENT CLIENT REPORTS SERVICE REPORTS DB vault payments reports SERVER SERVER SERVER SERVER SERVER SERVER SERVER SERVER SERVER
  64. 64. Copyright © 2017 HashiCorp Use Case 64 Isolation Using Multiple Datacenters CLIENT CLIENT VAULT SERVER VAULT SERVER CLIENT CLIENT PAYMENTS SERVICE PAYMENTS DB CLIENT CLIENT REPORTS SERVICE REPORTS DB vault payments reports SERVER SERVER SERVER SERVER SERVER SERVER SERVER SERVER SERVER
  65. 65. Copyright © 2017 HashiCorp Use Case 65 Isolation Using Multiple Datacenters CLIENT CLIENT VAULT SERVER VAULT SERVER CLIENT CLIENT PAYMENTS SERVICE PAYMENTS DB CLIENT CLIENT REPORTS SERVICE REPORTS DB vault payments reports SERVER SERVER SERVER SERVER SERVER SERVER SERVER SERVER SERVER
  66. 66. Copyright © 2017 HashiCorp Use Case 66 Isolation Using Multiple Datacenters CLIENT CLIENT VAULT SERVER VAULT SERVER CLIENT CLIENT PAYMENTS SERVICE PAYMENTS DB CLIENT CLIENT REPORTS SERVICE REPORTS DB vault payments reports X SERVER SERVER SERVER SERVER SERVER SERVER SERVER SERVER SERVER
  67. 67. Copyright © 2017 HashiCorp ▪ Based on same SWIM implementation that LAN Gossip uses, but tuned for a (potentially) low speed, high latency network ▪ Full mesh for all servers on 8302/udp and 8302/tcp ▪ Gossip uses AES and shared key ▪ RPC uses TLS ▪ Each datacenter is an independent failure domain ▪ Soft fail uses RPC feedback to control online/offline status for a remote cluster; circuit breaks when all servers in a cluster are failed ▪ ACLs protect server state ▪ RPC rate limiter (new in 0.9.3) Network Topology, Security, and Isolation 67
  68. 68. Copyright © 2017 HashiCorp ▪ Similar to LAN Gossip ▪ Manually via “consul join” command ▪ Automatic at agent startup via list of IPs or a DNS name ▪ Automatic at agent startup via cloud provider instance metadata (new in 0.9.3) ▪ Consul keeps the servers from the LAN Gossip pool synced with the WAN Gossip pool with flood join (new in 0.8.0) Forming Federations 68
  69. 69. Copyright © 2017 HashiCorp Completed Federation 69 (nyc3) $ consul members -wan Terminal
  70. 70. Copyright © 2017 HashiCorp Completed Federation 70 (nyc3) $ consul members -wan Node Address Status Type Build Protocol DC Segment consul-server-ams2-1.ams2 37.139.0.54:8302 alive server 0.9.3rc1 2 ams2 <all> consul-server-ams2-2.ams2 37.139.4.179:8302 alive server 0.9.3rc1 2 ams2 <all> consul-server-ams2-3.ams2 198.211.118.201:8302 alive server 0.9.3rc1 2 ams2 <all> consul-server-nyc3-1.nyc3 104.131.14.250:8302 alive server 0.9.3rc1 2 nyc3 <all> consul-server-nyc3-2.nyc3 104.131.180.172:8302 alive server 0.9.3rc1 2 nyc3 <all> consul-server-nyc3-3.nyc3 104.131.31.159:8302 alive server 0.9.3rc1 2 nyc3 <all> consul-server-sfo1-1.sfo1 192.241.204.248:8302 alive server 0.9.3rc1 2 sfo1 <all> consul-server-sfo1-2.sfo1 198.199.119.154:8302 alive server 0.9.3rc1 2 sfo1 <all> consul-server-sfo1-3.sfo1 192.241.223.22:8302 alive server 0.9.3rc1 2 sfo1 <all> Terminal
  71. 71. Copyright © 2017 HashiCorp Completed Federation 71 (nyc3) $ consul catalog datacenters Terminal
  72. 72. Copyright © 2017 HashiCorp Completed Federation 72 (nyc3) $ consul catalog datacenters nyc3 sfo1 ams2 Terminal
  73. 73. Copyright © 2017 HashiCorp Remote Catalog 73 (nyc3) $ consul catalog services -datacenter=ams2 Terminal
  74. 74. Copyright © 2017 HashiCorp Remote Catalog 74 (nyc3) $ consul catalog services -datacenter=ams2 consul redis web Terminal
  75. 75. Copyright © 2017 HashiCorp Remote Catalog 75 (nyc3) $ consul catalog services -datacenter=ams2 consul redis web (nyc3) $ consul catalog nodes -datacenter=ams2 -detailed -service=redis Terminal
  76. 76. Copyright © 2017 HashiCorp Remote Catalog 76 (nyc3) $ consul catalog services -datacenter=ams2 consul redis web (nyc3) $ consul catalog nodes -datacenter=ams2 -detailed -service=redis Node ID Address DC TaggedAddresses consul-client-ams2-1 94fba7d4 37.139.11.13 ams2 lan=37.139.11.13, wan=10.1.10.11 consul-client-ams2-2 ee539288 37.139.4.172 ams2 lan=37.139.4.172, wan=10.1.10.12 consul-client-ams2-3 f629a56c 37.139.2.208 ams2 lan=37.139.2.208, wan=10.1.10.13 Terminal
  77. 77. Copyright © 2017 HashiCorp Remote Catalog 77 (nyc3) $ dig +short redis.service.ams2.consul Terminal
  78. 78. Copyright © 2017 HashiCorp Remote Catalog 78 (nyc3) $ dig +short redis.service.ams2.consul 37.139.4.172 37.139.2.208 37.139.11.13 Terminal
  79. 79. Copyright © 2017 HashiCorp Remote KV 79 (nyc3) $ consul kv put -datacenter=ams2 hello world Terminal
  80. 80. Copyright © 2017 HashiCorp Remote KV 80 (nyc3) $ consul kv put -datacenter=ams2 hello world Success! Data written to: hello Terminal
  81. 81. Copyright © 2017 HashiCorp Remote KV 81 (nyc3) $ consul kv put -datacenter=ams2 hello world Success! Data written to: hello (nyc3) $ consul kv get hello Terminal
  82. 82. Copyright © 2017 HashiCorp Remote KV 82 (nyc3) $ consul kv put -datacenter=ams2 hello world Success! Data written to: hello (nyc3) $ consul kv get hello Error! No key exists at: hello Terminal
  83. 83. Copyright © 2017 HashiCorp Remote KV 83 (nyc3) $ consul kv put -datacenter=ams2 hello world Success! Data written to: hello (nyc3) $ consul kv get hello Error! No key exists at: hello (nyc3) $ consul kv get --datacenter=ams2 hello Terminal
  84. 84. Copyright © 2017 HashiCorp Remote KV 84 (nyc3) $ consul kv put -datacenter=ams2 hello world Success! Data written to: hello (nyc3) $ consul kv get hello Error! No key exists at: hello (nyc3) $ consul kv get --datacenter=ams2 hello world Terminal
  85. 85. Copyright © 2017 HashiCorp Static Service Failover 85 (nyc3) $ curl -X POST -d ‘{ "Name": “ha-", "Template": { "Type": "name_prefix_match" }, "Service": { "Service": “${name.suffix}”, “Failover: { “Datacenters”: [“nyc3”, “ams2”] } } }’ 127.0.0.1:8500/v1/query {“ID”:"a4ef7a27-e189-9e5c-ae3a-e586eac88f00"} Terminal
  86. 86. Copyright © 2017 HashiCorp Static Service Failover 86 (nyc3) $ dig +short ha-redis.query.consul Terminal
  87. 87. Copyright © 2017 HashiCorp Static Service Failover 87 (nyc3) $ dig +short ha-redis.query.consul 104.131.86.98 104.131.75.130 104.236.9.114 Terminal
  88. 88. Copyright © 2017 HashiCorp Static Service Failover 88 (nyc3) $ dig +short ha-redis.query.consul 104.131.86.98 104.131.75.130 104.236.9.114 (nyc3) $ echo “something bad happens to Redis in nyc3” Terminal
  89. 89. Copyright © 2017 HashiCorp Static Service Failover 89 (nyc3) $ dig +short ha-redis.query.consul 104.131.86.98 104.131.75.130 104.236.9.114 (nyc3) $ echo “something bad happens to Redis in nyc3” (nyc3) $ dig +short ha-redis.query.consul Terminal
  90. 90. Copyright © 2017 HashiCorp Static Service Failover 90 (nyc3) $ dig +short ha-redis.query.consul 104.131.86.98 104.131.75.130 104.236.9.114 (nyc3) $ echo “something bad happens to Redis in nyc3” (nyc3) $ dig +short ha-redis.query.consul 37.139.11.13 37.139.4.172 37.139.2.208 Terminal
  91. 91. s Copyright © 2017 HashiCorp 91 Advanced Clustering: Network Segments
  92. 92. Copyright © 2017 HashiCorp ▪ Available in Consul Enterprise ▪ Advanced clustering for heterogeneous networks of agents that can’t be in a full mesh ▪ Shared set of servers (for cost or to share cluster state like KV) ▪ Clients belong to a particular segment ▪ Servers belong to all segments ▪ Services discoverable across all segments via DNS and HTTP ▪ Shared KV store with sessions ▪ Prepared queries support finding services in a given segment Network Segments 92
  93. 93. Copyright © 2017 HashiCorp Use Case 93 Isolation Using Multiple Datacenters (WAN Gossip) CLIENT CLIENT VAULT SERVER VAULT SERVER CLIENT CLIENT PAYMENTS SERVICE PAYMENTS DB CLIENT CLIENT REPORTS SERVICE REPORTS DB vault payments reports SERVER SERVER SERVER SERVER SERVER SERVER SERVER SERVER SERVER
  94. 94. Copyright © 2017 HashiCorp Use Case 94 Isolation Inside a Single Datacenter (Network Segments) CLIENT CLIENT SERVER SERVER SERVER VAULT SERVER VAULT SERVER CLIENT CLIENT PAYMENTS SERVICE PAYMENTS DB CLIENT CLIENT REPORTS SERVICE REPORTS DB DC1:vault DC1:payments DC1:reports DC1:default
  95. 95. Copyright © 2017 HashiCorp Use Case 95 Isolation Inside a Single Datacenter CLIENT CLIENT SERVER SERVER SERVER REPLICATION REPLICATION VAULT SERVER VAULT SERVER CLIENT CLIENT PAYMENTS SERVICE PAYMENTS DB CLIENT CLIENT REPORTS SERVICE REPORTS DB DC1:vault DC1:payments DC1:reports DC1:default
  96. 96. Copyright © 2017 HashiCorp Use Case 96 Isolation Inside a Single Datacenter CLIENT CLIENT SERVER SERVER SERVER REPLICATION REPLICATION VAULT SERVER VAULT SERVER CLIENT CLIENT PAYMENTS SERVICE PAYMENTS DB CLIENT CLIENT REPORTS SERVICE REPORTS DB DC1:vault DC1:payments DC1:reports DC1:default
  97. 97. Copyright © 2017 HashiCorp Use Case 97 Isolation Inside a Single Datacenter CLIENT CLIENT SERVER SERVER SERVER REPLICATION REPLICATION VAULT SERVER VAULT SERVER CLIENT CLIENT PAYMENTS SERVICE PAYMENTS DB CLIENT CLIENT REPORTS SERVICE REPORTS DB DC1:vault DC1:payments DC1:reports DC1:default
  98. 98. Copyright © 2017 HashiCorp Use Case 98 Isolation Inside a Single Datacenter CLIENT CLIENT SERVER SERVER SERVER REPLICATION REPLICATION VAULT SERVER VAULT SERVER CLIENT CLIENT PAYMENTS SERVICE PAYMENTS DB CLIENT CLIENT REPORTS SERVICE REPORTS DB DC1:vault DC1:payments DC1:reports DC1:default X
  99. 99. Copyright © 2017 HashiCorp ▪ High speed, low latency network (also needed for Raft) ▪ Full mesh for all servers on 8301/udp and 8301/tcp ▪ Full mesh for all clients in segmented gossip pools (xxxx/udp and xxxx/tcp); servers in all pools (xxxx/udp and xxxx/tcp) ▪ Gossip uses AES and shared key ▪ RPC uses TLS ▪ Client agents don’t need any connectivity outside of their segment, other than to servers ▪ ACLs protect server state ▪ RPC rate limiter (new in 0.9.3) Network Topology, Security, and Isolation 99
  100. 100. Copyright © 2017 HashiCorp Forming Clusters ▪ Servers specify list of available segments in their configuration 100 { "segments": [ { "name": “vault", "bind": “{{GetPrivateIP}}”, "port": 8303 }, { "name": “payments", "bind": “{{GetPrivateIP}}”, "port": 8304 }, { "name": “reports", "bind": “{{GetPrivateIP}}”, "port": 8305 } ] } server.json
  101. 101. Copyright © 2017 HashiCorp Forming Clusters ▪ Clients specify which segment they belong to in their configuration ▪ Clients join just like LAN Gossip 101 { “segment”: “vault”, “retry_join”: [“consul.domain.internal:8303”] } client.json
  102. 102. Copyright © 2017 HashiCorp Completed Cluster 102 $ consul members Terminal
  103. 103. Copyright © 2017 HashiCorp Completed Cluster 103 $ consul members Node Address Status Type Build Protocol DC Segment server1 192.168.0.4:8301 alive server 0.9.3+ent 2 dc1 <all> server1 192.168.0.5:8301 alive server 0.9.3+ent 2 dc1 <all> server1 192.168.0.6:8301 alive server 0.9.3+ent 2 dc1 <all> client1 192.168.10.1:8303 alive client 0.9.3+ent 2 dc1 vault client2 192.168.11.1:8304 alive client 0.9.3+ent 2 dc1 payments client3 192.168.12.1:8305 alive client 0.9.3+ent 2 dc1 reports Terminal
  104. 104. Copyright © 2017 HashiCorp Completed Cluster 104 $ consul members Node Address Status Type Build Protocol DC Segment server1 192.168.0.4:8301 alive server 0.9.3+ent 2 dc1 <all> server1 192.168.0.5:8301 alive server 0.9.3+ent 2 dc1 <all> server1 192.168.0.6:8301 alive server 0.9.3+ent 2 dc1 <all> client1 192.168.10.1:8303 alive client 0.9.3+ent 2 dc1 vault client2 192.168.11.1:8304 alive client 0.9.3+ent 2 dc1 payments client3 192.168.12.1:8305 alive client 0.9.3+ent 2 dc1 reports $ consul members -segment=vault Terminal
  105. 105. Copyright © 2017 HashiCorp Completed Cluster 105 $ consul members Node Address Status Type Build Protocol DC Segment server1 192.168.0.4:8301 alive server 0.9.3+ent 2 dc1 <all> server1 192.168.0.5:8301 alive server 0.9.3+ent 2 dc1 <all> server1 192.168.0.6:8301 alive server 0.9.3+ent 2 dc1 <all> client1 192.168.10.1:8303 alive client 0.9.3+ent 2 dc1 vault client2 192.168.11.1:8304 alive client 0.9.3+ent 2 dc1 payments client3 192.168.12.1:8305 alive client 0.9.3+ent 2 dc1 reports $ consul members -segment=vault Node Address Status Type Build Protocol DC Segment server1 192.168.0.4:8303 alive server 0.9.3+ent 2 dc1 vault server1 192.168.0.5:8303 alive server 0.9.3+ent 2 dc1 vault server1 192.168.0.6:8303 alive server 0.9.3+ent 2 dc1 vault client1 192.168.10.1:8303 alive client 0.9.3+ent 2 dc1 vault Terminal
  106. 106. Copyright © 2017 HashiCorp ▪ KV works the same as LAN Gossip (all state is shared) ▪ Catalog operations can work the same as well to discover a service in any segment (vault in the use case example) ▪ Catalog operations can also be scoped to discover services inside of an agent’s segment Working with Network Segments 106
  107. 107. Copyright © 2017 HashiCorp Query for Service in Same Segment 107 $ curl -X POST -d ‘{ "Name": “local-", "Template": { "Type": "name_prefix_match" }, "Service": { "Service": “${name.suffix}”, "NodeMeta": {"consul-network-segment": "${agent.segment}"} } }’ 127.0.0.1:8500/v1/query {“ID”:"a4ef7a27-e189-9e5c-ae3a-e586eac88f00"} Terminal
  108. 108. Copyright © 2017 HashiCorp Query for Service in Same Segment 108 (payments) $ dig +short local-db.service.consul Terminal
  109. 109. Copyright © 2017 HashiCorp Query for Service in Same Segment 109 (payments) $ dig +short local-db.service.consul 192.168.11.2 Terminal
  110. 110. Copyright © 2017 HashiCorp Query for Service in Same Segment 110 (payments) $ dig +short local-db.service.consul 192.168.11.2 (reports) $ dig +short local-db.service.consul 192.168.12.2 Terminal
  111. 111. s Copyright © 2017 HashiCorp 111 Advanced Federation: Network Areas
  112. 112. Copyright © 2017 HashiCorp ▪ Available in Consul Enterprise ▪ Advanced federation for heterogeneous networks of servers (small to massive scale) ▪ Explicit network areas defined between pairs of datacenters ▪ Service and KV state are local to each datacenter (no replication) ▪ Service and KV operations can be made to remote datacenters with a Network Area in common ▪ Prepared queries support nearest neighbor routing between datacenters via RTT Network Areas 112
  113. 113. Copyright © 2017 HashiCorp Use Case 113 Central Management Hub Datacenter with Isolated Spoke Datacenters tenant-1 tenant-2 tenant-3 hub tenant-4
  114. 114. Copyright © 2017 HashiCorp Use Case 114 Central Management Hub Datacenter with Isolated Spoke Datacenters tenant-1 tenant-2 tenant-3 hub tenant-4 CLIENT CLIENT VAULT SERVER VAULT SERVER SERVER SERVER SERVER hub
  115. 115. Copyright © 2017 HashiCorp Use Case 115 Central Management Hub Datacenter with Isolated Spoke Datacenters tenant-1 tenant-2 tenant-3 hub tenant-4 CLIENT CLIENT UNTRUSTED THING 1 UNTRUSTED THING 2 SERVER SERVER SERVER tenant-3
  116. 116. Copyright © 2017 HashiCorp Use Case 116 Central Management Hub Datacenter with Isolated Spoke Datacenters tenant-1 tenant-2 tenant-3 hub tenant-4
  117. 117. Copyright © 2017 HashiCorp Use Case 117 Central Management Hub Datacenter with Isolated Spoke Datacenters tenant-1 tenant-2 tenant-3 hub tenant-4 X
  118. 118. Copyright © 2017 HashiCorp Use Case 118 Massive Geo-Distributed Datacenters with Partial Connectivity sfo1 nyc3 ams2 sgp1 tor1
  119. 119. Copyright © 2017 HashiCorp ▪ Based on same SWIM implementation that WAN Gossip uses, but tuned for a (potentially) low speed, high latency network ▪ Full mesh for all servers in each area ▪ RPC and gossip use TLS, and only 8300/tcp ▪ Each datacenter is an independent failure domain ▪ ACLs protect server state ▪ RPC rate limiter (new in 0.9.3) Network Topology, Security, and Isolation 119
  120. 120. Copyright © 2017 HashiCorp ▪ Operators explicitly define areas in each datacenter to be joined as a pair ▪ Once defined, areas can be joined using CLI or HTTP interfaces Forming Federations 120
  121. 121. Copyright © 2017 HashiCorp Forming Federations 121 (nyc3) $ consul operator area create -peer-datacenter=ams2 Terminal
  122. 122. Copyright © 2017 HashiCorp Forming Federations 122 (nyc3) $ consul operator area create -peer-datacenter=ams2 Created area "cbd364ae-3710-1770-911b-7214e98016c0" with peer datacenter “ams2"! Terminal
  123. 123. Copyright © 2017 HashiCorp Forming Federations 123 (nyc3) $ consul operator area create -peer-datacenter=ams2 Created area "cbd364ae-3710-1770-911b-7214e98016c0" with peer datacenter “ams2”! (ams2) $ consul operator area create -peer-datacenter=nyc3 Terminal
  124. 124. Copyright © 2017 HashiCorp Forming Federations 124 (nyc3) $ consul operator area create -peer-datacenter=ams2 Created area "cbd364ae-3710-1770-911b-7214e98016c0" with peer datacenter “ams2”! (ams2) $ consul operator area create -peer-datacenter=nyc3 Created area "2aea3145-f1e3-cb1d-a775-67d15ddd89bf" with peer datacenter “nyc3"! Terminal
  125. 125. Copyright © 2017 HashiCorp Forming Federations 125 (nyc3) $ consul operator area create -peer-datacenter=ams2 Created area "cbd364ae-3710-1770-911b-7214e98016c0" with peer datacenter “ams2”! (ams2) $ consul operator area create -peer-datacenter=nyc3 Created area "2aea3145-f1e3-cb1d-a775-67d15ddd89bf" with peer datacenter “nyc3"! (ams2) $ consul operator area join -peer-datacenter=nyc3 nyc.consul.internal Terminal
  126. 126. Copyright © 2017 HashiCorp Forming Federations 126 (nyc3) $ consul operator area create -peer-datacenter=ams2 Created area "cbd364ae-3710-1770-911b-7214e98016c0" with peer datacenter “ams2”! (ams2) $ consul operator area create -peer-datacenter=nyc3 Created area "2aea3145-f1e3-cb1d-a775-67d15ddd89bf" with peer datacenter “nyc3"! (ams2) $ consul operator area join -peer-datacenter=nyc3 nyc.consul.internal Address Joined Error nyc.consul.internal true (none) Terminal
  127. 127. Copyright © 2017 HashiCorp (ams2) $ consul operator area members Area Node Address Status Build Protocol DC RTT 2aea3145 consul-server-ams2-1.ams2 37.139.0.54:8302 alive 0.9.3rc1 2 ams2 0s 2aea3145 consul-server-ams2-2.ams2 37.139.4.179:8302 alive 0.9.3rc1 2 ams2 581.649µs 2aea3145 consul-server-ams2-3.ams2 198.211.118.201:8302 alive 0.9.3rc1 2 ams2 789.121µs 2aea3145 consul-server-nyc3-1.nyc3 104.131.14.250:8302 alive 0.9.3rc1 2 nyc3 79.753 ms 2aea3145 consul-server-nyc3-2.nyc3 104.131.180.172:8302 alive 0.9.3rc1 2 nyc3 80.140 ms 2aea3145 consul-server-nyc3-3.nyc3 104.131.31.159:8302 alive 0.9.3rc1 2 nyc3 83.346 ms Completed Federation 127 Terminal
  128. 128. Copyright © 2017 HashiCorp ▪ Once Network Areas are defined and joined, federation works the same as WAN Gossip ▪ Management is simplified because there are no shared gossip keys (TLS is used for gossip and RPC) ▪ Soft fail uses RPC feedback to control online/offline status for a remote cluster; circuit breaks when all servers in a cluster are failed Working with Network Areas 128
  129. 129. Copyright © 2017 HashiCorp Dynamic Service Failover 129 $ curl -X POST -d ‘{ "Name": “geo-", "Template": { "Type": "name_prefix_match" }, "Service": { "Service": “${name.suffix}”, “Failover: { "NearestN": 2 } } }’ 127.0.0.1:8500/v1/query {“ID”:"a4ef7a27-e189-9e5c-ae3a-e586eac88f00"} Terminal
  130. 130. Copyright © 2017 HashiCorp Dynamic Service Failover 130 (nyc3) $ dig +short geo-redis.query.consul Terminal
  131. 131. Copyright © 2017 HashiCorp Dynamic Service Failover 131 (nyc3) $ dig +short geo-redis.query.consul 104.131.86.98 104.131.75.130 104.236.9.114 Terminal
  132. 132. Copyright © 2017 HashiCorp Dynamic Service Failover 132 (nyc3) $ dig +short geo-redis.query.consul 104.131.86.98 104.131.75.130 104.236.9.114 (nyc3) $ echo “something bad happens to Redis in nyc3” Terminal
  133. 133. Copyright © 2017 HashiCorp Dynamic Service Failover 133 (nyc3) $ dig +short geo-redis.query.consul 104.131.86.98 104.131.75.130 104.236.9.114 (nyc3) $ echo “something bad happens to Redis in nyc3” (nyc3) $ dig +short geo-redis.query.consul Terminal
  134. 134. Copyright © 2017 HashiCorp Dynamic Service Failover 134 (nyc3) $ dig +short geo-redis.query.consul 104.131.86.98 104.131.75.130 104.236.9.114 (nyc3) $ echo “something bad happens to Redis in nyc3” (nyc3) $ dig +short geo-redis.query.consul 37.139.11.13 37.139.4.172 37.139.2.208 Terminal
  135. 135. s Copyright © 2017 HashiCorp (on any network topology) 135 Provision, secure, connect, and run any infrastructure for any application.
  136. 136. Copyright © 2017 HashiCorp ▪ Consul’s four network models map to a large set of use cases learned from years of developing and supporting Consul on the road to Consul 1.0 ▪ Models allow nuanced tradeoffs around cost, features, connectivity requirements, and levels of isolation ▪ Models can be in use simultaneously, enabling incremental adoption and rollout ▪ Gossip provides a unique set of primitives that operate across all network models ▪ Prepared queries allow any application to benefit, even if they are only using Consul’s DNS interface Consul and Complex Networks 136
  137. 137. Thank you. hello@hashicorp.comwww.hashicorp.com

×