Consul and Complex Networks

Copyright © 2017 HashiCorp
Consul and Complex
Networks
HashiConf US 2017, Austin Texas
James Phillips, Consul Lead Engineer
@slackpad

s
Copyright © 2017 HashiCorp 2
Consul Overview

▪ Register services (IP:port + health checks) via config files or HTTP
▪ Discover services via DNS or HTTP
▪ Perform load balancing via DNS result shuffling
▪ Manage dynamic runtime configuration via key/value store
▪ Orchestrate via key/value store, events, watches, …
▪ Integrate with existing apps via DNS, HTTP, consul-template,
envconsul, Fabio, …
Service Discovery, Configuration, and Orchestration
3

▪ Run node- and service-specific health checks via Nagios compatible
scripts, Docker, HTTP, and TCP
▪ TTL “dead person switch” checks
Monitoring at Scale
4

▪ Fault tolerant via Raft consensus
▪ Federate multiple clusters easily
▪ Look up services locally or in other federated clusters
▪ Automatically apply failover policies across federated clusters
▪ Coordinate complex distributed systems via key/value locking
primitives and sessions (eg. Vault)
High Availability
5

▪ Consul agent is a single Go binary
▪ Agent runs on every machine in a cluster
▪ Applications always talk to their local Consul agent
▪ 3, 5, or 7 agents in a cluster run as servers, all others are clients
▪ Servers maintain a consistent view of the cluster state
Running Consul
6

▪ Automated backups
▪ Automated upgrades
▪ Enhanced read scalability
▪ Redundancy zones
▪ Advanced network models
Consul Enterprise
7

s
Network Models

▪ Connect agents inside a datacenter
▪ One set of servers connected to a cluster of clients
▪ Two models available
▪ LAN Gossip
▪ Network Segments (Enterprise)
Clustering
9

▪ Connect servers in different datacenters
▪ Clients don’t participate directly
▪ Two models available
▪ WAN Gossip
▪ Network Areas (Enterprise)
Federation
10

▪ Set of agents with full mesh connectivity via UDP* and TCP
▪ SWIM algorithm via hashicorp/memberlist
▪ Distributed failure detector
▪ Fast gossip-based broadcast mechanism
▪ Shared list of all agents, anti-entropy combats drift
▪ Used all over Consul
▪ Clients finding available servers
▪ Edge-triggered push-updated health checks
▪ AP events feed CP Raft system on the leader update catalog
▪ ...
Gossip Pools
11

Gossip Pools In Action
12
B
A
X
C
Probe (UDP)

13
B
A
X
C
Ack (UDP)

14
B
A
X
C
Probe (TCP)
Probe X
ProbeX

15
B
A
X
C
Probe (TCP)
Probe (UDP)
Probe (UDP)

16
B
A
X
C
Nack
Nack
Probe (TCP)
Probe (UDP)
Probe (UDP)

17
X is
suspect

18

19
X has
failed

20
X is gone

21
Whoa

Further Reading
22

Concept of Operations
23
CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT
SERVER SERVER SERVER

24
DC1

25
DC1

26
REPLICATION REPLICATION
DC1

27
DC1
consul kv put hello world

28
REPLICATION
DC1
consul kv put hello world

29
DC1
consul kv get hello

30
REPLICATION
DC1
consul kv get hello

31
DC1

32
DC1
DC2

33
DC1
DC2

34
DC1
DC2
consul kv put -datacenter=dc2 hello world

35
DC1
DC2

36
DC1
DC2

37
DC1
DC2

s
Simple Clustering:
LAN Gossip

▪ Basic clustering for homogeneous networks of agents (small to
massive scale)
▪ Services discoverable via DNS and HTTP
▪ Shared KV store with sessions
▪ Prepared queries support nearest neighbor routing within the
datacenter via RTT
LAN Gossip
39

Use Case
40
Web Application with Multiple Services
DC1
web.service
(Go)
user.service
(Rails)
search.service
(Java)
order.service
(Go)
cache.service
(Redis)
db.service
(Postgres)

Use Case
41
Web Application with Multiple Services
DC1
web.service
user.servicce
search.service
order.service
cache.service
CLIENT
db.service
(Postgres)

Use Case
42
Massive Compute Cluster with Nomad
CLIENT CLIENT CLIENT CLIENT
DC1
NOMAD
SERVER
NOMAD
SERVER
NOMAD
SERVER
NOMAD
CLIENT
1000
CLIENT
NOMAD
CLIENT
1
…

▪ High speed, low latency network (also needed for Raft)
▪ Full mesh for all clients and servers on 8301/udp and 8301/tcp
▪ Gossip uses AES and shared key
▪ RPC uses TLS
▪ ACLs protect server state
▪ RPC rate limiter (new in 0.9.3)
Network Topology, Security, and Isolation
43

▪ Manually via “consul join” command
▪ Automatic at agent startup via list of IPs or a DNS name
▪ Automatic at agent startup via cloud provider instance metadata
▪ Support for AWS, GCE, Azure, and SoftLayer
▪ Community developing more in hashicorp/go-discover
Forming Clusters
44

Manual Join
45
$ consul join 1.2.3.4
Terminal

Manual Join
46
$ consul join 1.2.3.4
Successfully joined cluster by contacting 3 nodes.
Terminal

Automatic Join via DNS
47
$ consul agent -retry-join=consul.domain.internal
Terminal

Automatic Join via DNS
48
$ consul agent -retry-join=consul.domain.internal
==> Starting Consul agent...
==> Consul agent running!
Version: 'v0.9.3-10-g7a2cd047'
Node ID: 'df505b71-cd19-4051-8e40-e492392155fa'
Node name: 'node-2'
Datacenter: 'dc1' (Segment: '')
Server: false (Bootstrap: false)
Client Addr: 127.0.0.2 (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 127.0.0.2 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2017/09/16 23:12:37 [INFO] agent: Join LAN completed. Synced with 3 initial agents
Terminal

Automatic Join via Cloud Provider Metadata
49
$ consul agent -retry-join=‘provider=aws tag_key=consul tag_value=joiner’
Terminal

Automatic Join via Cloud Provider Metadata
50
$ consul agent -retry-join=‘provider=aws tag_key=consul tag_value=joiner’
==> Starting Consul agent...
==> Consul agent running!
Version: 'v0.9.3-10-g7a2cd047'
Node ID: 'df505b71-cd19-4051-8e40-e492392155fa'
Node name: 'node-2'
Datacenter: 'dc1' (Segment: '')
Server: false (Bootstrap: false)
Client Addr: 127.0.0.2 (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 127.0.0.2 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2017/09/16 23:12:33 [INFO] agent: Discovered LAN servers: 10.1.1.11 10.1.1.91 10.1.1.74 10.1.2.197
2017/09/16 23:12:37 [INFO] agent: Join LAN completed. Synced with 3 initial agents
Terminal

Completed Cluster
51
$ consul members
Terminal

Completed Cluster
52
$ consul members
Node Address Status Type Build Protocol DC Segment
consul-server-nyc3-1 104.131.14.250:8301 alive server 0.9.3rc1 2 nyc3 <all>
consul-client-nyc3-1 104.131.86.98:8301 alive client 0.9.3rc1 2 nyc3 <default>
Terminal

▪ SWIM’s gossip protocol has agents randomly probe each other
▪ This gives a nice set of RTT samples that are fed into a simple
physics model
▪ The model allows the Consul servers to calculate estimated RTTs
between agents, and to sort results by estimated RTT
▪ Exposed in APIs as ?near= parameter
▪ Exposed in DNS interface via prepared queries
Gossip and Round Trip Times
53

Nearest Neighbor Routing
54
$ curl -X POST -d
‘{
"Name": “nearest-",
"Template": {
"Type": "name_prefix_match"
},
"Service": {
"Service": "${name.suffix}"
},
“Near”: “_agent”
}’ 127.0.0.1:8500/v1/query
{“ID”:"a4ef7a27-e189-9e5c-ae3a-e586eac88f00"}
Terminal

55
$ dig +short cache.service.consul
Terminal

56
10.1.0.1
10.1.0.2
10.1.0.3
Terminal

57
10.1.0.1
10.1.0.2
10.1.0.3
$ dig +short nearest-cache.query.consul
Terminal

58
10.1.0.1
10.1.0.2
10.1.0.3
$ dig +short nearest-cache.query.consul
10.1.0.2
10.1.0.1
10.1.0.3
Terminal

s
Simple Federation:
WAN Gossip

▪ Basic federation for homogeneous networks of servers (small to
massive scale)
▪ Service and KV state are local to each datacenter (no replication)
▪ Service and KV operations can be made to remote datacenters
joined to the WAN Gossip pool
▪ Prepared queries support nearest neighbor routing between
datacenters via RTT
WAN Gossip
60

Use Case
61
Geo Redundancy with Multiple Datacenters
nyc3
web.service user.service search.service order.service cache.service db.service
ams2web.service user.service search.service order.service cache.service db.service

Use Case
62
Geo Redundancy with Multiple Datacenters
nyc3
web.service user.service search.service order.service cache.service db.service
ams2web.service user.service search.service order.service cache.service db.service

Use Case
63
Isolation Using Multiple Datacenters
CLIENT CLIENT
VAULT
SERVER
VAULT
SERVER
CLIENT CLIENT
PAYMENTS
SERVICE
PAYMENTS
DB
CLIENT CLIENT
REPORTS
SERVICE
REPORTS
DB
vault payments reports
SERVER
SERVER
SERVER
SERVER
SERVER
SERVER
SERVER
SERVER
SERVER

Use Case
64
CLIENT CLIENT
VAULT
SERVER
VAULT
SERVER
CLIENT CLIENT
PAYMENTS
SERVICE
PAYMENTS
DB
CLIENT CLIENT
REPORTS
SERVICE
REPORTS
DB
SERVER
SERVER
SERVER
SERVER
SERVER
SERVER
SERVER
SERVER
SERVER

Use Case
65
CLIENT CLIENT
VAULT
SERVER
VAULT
SERVER
CLIENT CLIENT
PAYMENTS
SERVICE
PAYMENTS
DB
CLIENT CLIENT
REPORTS
SERVICE
REPORTS
DB
SERVER
SERVER
SERVER
SERVER
SERVER
SERVER
SERVER
SERVER
SERVER

Use Case
66
CLIENT CLIENT
VAULT
SERVER
VAULT
SERVER
CLIENT CLIENT
PAYMENTS
SERVICE
PAYMENTS
DB
CLIENT CLIENT
REPORTS
SERVICE
REPORTS
DB
X
SERVER
SERVER
SERVER
SERVER
SERVER
SERVER
SERVER
SERVER
SERVER

▪ Based on same SWIM implementation that LAN Gossip uses, but
tuned for a (potentially) low speed, high latency network
▪ Full mesh for all servers on 8302/udp and 8302/tcp
▪ RPC uses TLS
▪ Each datacenter is an independent failure domain
▪ Soft fail uses RPC feedback to control online/offline status for a
remote cluster; circuit breaks when all servers in a cluster are failed
67

▪ Similar to LAN Gossip
▪ Manually via “consul join” command
▪ Automatic at agent startup via list of IPs or a DNS name
▪ Automatic at agent startup via cloud provider instance metadata
(new in 0.9.3)
▪ Consul keeps the servers from the LAN Gossip pool synced with the
WAN Gossip pool with flood join (new in 0.8.0)
Forming Federations
68

Completed Federation
69
(nyc3) $ consul members -wan
Terminal

70
(nyc3) $ consul members -wan
consul-server-ams2-1.ams2 37.139.0.54:8302 alive server 0.9.3rc1 2 ams2 <all>
consul-server-nyc3-1.nyc3 104.131.14.250:8302 alive server 0.9.3rc1 2 nyc3 <all>
consul-server-sfo1-1.sfo1 192.241.204.248:8302 alive server 0.9.3rc1 2 sfo1 <all>
Terminal

71
(nyc3) $ consul catalog datacenters
Terminal

72
(nyc3) $ consul catalog datacenters
nyc3
sfo1
ams2
Terminal

Remote Catalog
73
(nyc3) $ consul catalog services -datacenter=ams2
Terminal

Remote Catalog
74
consul
redis
web
Terminal

Remote Catalog
75
consul
redis
web
(nyc3) $ consul catalog nodes -datacenter=ams2 -detailed -service=redis
Terminal

Remote Catalog
76
consul
redis
web
(nyc3) $ consul catalog nodes -datacenter=ams2 -detailed -service=redis
Node ID Address DC TaggedAddresses
consul-client-ams2-1 94fba7d4 37.139.11.13 ams2 lan=37.139.11.13, wan=10.1.10.11
consul-client-ams2-2 ee539288 37.139.4.172 ams2 lan=37.139.4.172, wan=10.1.10.12
consul-client-ams2-3 f629a56c 37.139.2.208 ams2 lan=37.139.2.208, wan=10.1.10.13
Terminal

Remote Catalog
77
(nyc3) $ dig +short redis.service.ams2.consul
Terminal

Remote Catalog
78
(nyc3) $ dig +short redis.service.ams2.consul
37.139.4.172
37.139.2.208
37.139.11.13
Terminal

Remote KV
79
(nyc3) $ consul kv put -datacenter=ams2 hello world
Terminal

Remote KV
80
Success! Data written to: hello
Terminal

Remote KV
81
(nyc3) $ consul kv get hello
Terminal

Remote KV
82
Error! No key exists at: hello
Terminal

Remote KV
83
(nyc3) $ consul kv get --datacenter=ams2 hello
Terminal

Remote KV
84
(nyc3) $ consul kv get --datacenter=ams2 hello
world
Terminal

Static Service Failover
85
(nyc3) $ curl -X POST -d
‘{
"Name": “ha-",
"Template": {
},
"Service": {
"Service": “${name.suffix}”,
“Failover: {
“Datacenters”: [“nyc3”, “ams2”]
}
}
}’ 127.0.0.1:8500/v1/query
Terminal

86
(nyc3) $ dig +short ha-redis.query.consul
Terminal

87
104.131.86.98
104.131.75.130
104.236.9.114
Terminal

88
104.131.86.98
104.131.75.130
104.236.9.114
(nyc3) $ echo “something bad happens to Redis in nyc3”
Terminal

89
104.131.86.98
104.131.75.130
104.236.9.114
Terminal

90
104.131.86.98
104.131.75.130
104.236.9.114
37.139.11.13
37.139.4.172
37.139.2.208
Terminal

s
Advanced Clustering:
Network Segments

▪ Available in Consul Enterprise
▪ Advanced clustering for heterogeneous networks of agents that
can’t be in a full mesh
▪ Shared set of servers (for cost or to share cluster state like KV)
▪ Clients belong to a particular segment
▪ Servers belong to all segments
▪ Services discoverable across all segments via DNS and HTTP
▪ Shared KV store with sessions
▪ Prepared queries support finding services in a given segment
Network Segments
92

Use Case
93
Isolation Using Multiple Datacenters (WAN Gossip)
CLIENT CLIENT
VAULT
SERVER
VAULT
SERVER
CLIENT CLIENT
PAYMENTS
SERVICE
PAYMENTS
DB
CLIENT CLIENT
REPORTS
SERVICE
REPORTS
DB
SERVER
SERVER
SERVER
SERVER
SERVER
SERVER
SERVER
SERVER
SERVER

Use Case
94
Isolation Inside a Single Datacenter (Network Segments)
CLIENT CLIENT
VAULT
SERVER
VAULT
SERVER
CLIENT CLIENT
PAYMENTS
SERVICE
PAYMENTS
DB
CLIENT CLIENT
REPORTS
SERVICE
REPORTS
DB
DC1:vault DC1:payments DC1:reports
DC1:default

Use Case
95
Isolation Inside a Single Datacenter
CLIENT CLIENT
VAULT
SERVER
VAULT
SERVER
CLIENT CLIENT
PAYMENTS
SERVICE
PAYMENTS
DB
CLIENT CLIENT
REPORTS
SERVICE
REPORTS
DB
DC1:default

Use Case
96
CLIENT CLIENT
VAULT
SERVER
VAULT
SERVER
CLIENT CLIENT
PAYMENTS
SERVICE
PAYMENTS
DB
CLIENT CLIENT
REPORTS
SERVICE
REPORTS
DB
DC1:default

Use Case
97
CLIENT CLIENT
VAULT
SERVER
VAULT
SERVER
CLIENT CLIENT
PAYMENTS
SERVICE
PAYMENTS
DB
CLIENT CLIENT
REPORTS
SERVICE
REPORTS
DB
DC1:default

Use Case
98
CLIENT CLIENT
VAULT
SERVER
VAULT
SERVER
CLIENT CLIENT
PAYMENTS
SERVICE
PAYMENTS
DB
CLIENT CLIENT
REPORTS
SERVICE
REPORTS
DB
DC1:default
X

▪ High speed, low latency network (also needed for Raft)
▪ Full mesh for all servers on 8301/udp and 8301/tcp
▪ Full mesh for all clients in segmented gossip pools (xxxx/udp and
xxxx/tcp); servers in all pools (xxxx/udp and xxxx/tcp)
▪ RPC uses TLS
▪ Client agents don’t need any connectivity outside of their segment,
other than to servers
99

Forming Clusters
▪ Servers specify list of
available segments in
their configuration
100
{
"segments": [
{
"name": “vault",
"bind": “{{GetPrivateIP}}”,
"port": 8303
},
{
"name": “payments",
"port": 8304
},
{
"name": “reports",
"port": 8305
}
]
}
server.json

Forming Clusters
▪ Clients specify which
segment they belong
to in their
configuration
▪ Clients join just like
LAN Gossip
101
{
“segment”: “vault”,
“retry_join”: [“consul.domain.internal:8303”]
}
client.json

Completed Cluster
102
$ consul members
Terminal

Completed Cluster
103
$ consul members
server1 192.168.0.4:8301 alive server 0.9.3+ent 2 dc1 <all>
client1 192.168.10.1:8303 alive client 0.9.3+ent 2 dc1 vault
client2 192.168.11.1:8304 alive client 0.9.3+ent 2 dc1 payments
client3 192.168.12.1:8305 alive client 0.9.3+ent 2 dc1 reports
Terminal

Completed Cluster
104
$ consul members
$ consul members -segment=vault
Terminal

Completed Cluster
105
$ consul members
$ consul members -segment=vault
server1 192.168.0.4:8303 alive server 0.9.3+ent 2 dc1 vault
Terminal

▪ KV works the same as LAN Gossip (all state is shared)
▪ Catalog operations can work the same as well to discover a service
in any segment (vault in the use case example)
▪ Catalog operations can also be scoped to discover services inside of
an agent’s segment
Working with Network Segments
106

Query for Service in Same Segment
107
$ curl -X POST -d
‘{
"Name": “local-",
"Template": {
},
"Service": {
"NodeMeta": {"consul-network-segment": "${agent.segment}"}
}
}’ 127.0.0.1:8500/v1/query
Terminal

108
(payments) $ dig +short local-db.service.consul
Terminal

109
192.168.11.2
Terminal

110
192.168.11.2
(reports) $ dig +short local-db.service.consul
192.168.12.2
Terminal

s
Advanced Federation:
Network Areas

▪ Available in Consul Enterprise
▪ Advanced federation for heterogeneous networks of servers (small
to massive scale)
▪ Explicit network areas defined between pairs of datacenters
▪ Service and KV state are local to each datacenter (no replication)
▪ Service and KV operations can be made to remote datacenters with
a Network Area in common
▪ Prepared queries support nearest neighbor routing between
datacenters via RTT
Network Areas
112

Use Case
113
Central Management Hub Datacenter with Isolated Spoke Datacenters
tenant-1
tenant-2 tenant-3
hub
tenant-4

Use Case
114
tenant-1
tenant-2 tenant-3
hub
tenant-4
CLIENT CLIENT
VAULT
SERVER
VAULT
SERVER
SERVER
SERVER
SERVER
hub

Use Case
115
tenant-1
tenant-2 tenant-3
hub
tenant-4
CLIENT CLIENT
UNTRUSTED
THING 1
UNTRUSTED
THING 2
SERVER
SERVER
SERVER
tenant-3

Use Case
116
tenant-1
tenant-2 tenant-3
hub
tenant-4

Use Case
117
tenant-1
tenant-2 tenant-3
hub
tenant-4
X

Use Case
118
Massive Geo-Distributed Datacenters with Partial Connectivity
sfo1
nyc3
ams2
sgp1
tor1

▪ Based on same SWIM implementation that WAN Gossip uses, but
tuned for a (potentially) low speed, high latency network
▪ Full mesh for all servers in each area
▪ RPC and gossip use TLS, and only 8300/tcp
▪ Each datacenter is an independent failure domain
119

▪ Operators explicitly define areas in each datacenter to be joined as
a pair
▪ Once defined, areas can be joined using CLI or HTTP interfaces
Forming Federations
120

Forming Federations
121
(nyc3) $ consul operator area create -peer-datacenter=ams2
Terminal

Forming Federations
122
Created area "cbd364ae-3710-1770-911b-7214e98016c0" with peer datacenter “ams2"!
Terminal

Forming Federations
123
Created area "cbd364ae-3710-1770-911b-7214e98016c0" with peer datacenter “ams2”!
(ams2) $ consul operator area create -peer-datacenter=nyc3
Terminal

Forming Federations
124
Created area "2aea3145-f1e3-cb1d-a775-67d15ddd89bf" with peer datacenter “nyc3"!
Terminal

Forming Federations
125
(ams2) $ consul operator area join -peer-datacenter=nyc3 nyc.consul.internal
Terminal

Forming Federations
126
(ams2) $ consul operator area join -peer-datacenter=nyc3 nyc.consul.internal
Address Joined Error
nyc.consul.internal true (none)
Terminal

(ams2) $ consul operator area members
Area Node Address Status Build Protocol DC RTT
2aea3145 consul-server-ams2-1.ams2 37.139.0.54:8302 alive 0.9.3rc1 2 ams2 0s
2aea3145 consul-server-ams2-2.ams2 37.139.4.179:8302 alive 0.9.3rc1 2 ams2 581.649µs
2aea3145 consul-server-ams2-3.ams2 198.211.118.201:8302 alive 0.9.3rc1 2 ams2 789.121µs
2aea3145 consul-server-nyc3-1.nyc3 104.131.14.250:8302 alive 0.9.3rc1 2 nyc3 79.753 ms
127
Terminal

▪ Once Network Areas are defined and joined, federation works the
same as WAN Gossip
▪ Management is simplified because there are no shared gossip keys
(TLS is used for gossip and RPC)
▪ Soft fail uses RPC feedback to control online/offline status for a
remote cluster; circuit breaks when all servers in a cluster are failed
Working with Network Areas
128

Dynamic Service Failover
129
$ curl -X POST -d
‘{
"Name": “geo-",
"Template": {
},
"Service": {
“Failover: {
"NearestN": 2
}
}
}’ 127.0.0.1:8500/v1/query
Terminal

130
(nyc3) $ dig +short geo-redis.query.consul
Terminal

131
104.131.86.98
104.131.75.130
104.236.9.114
Terminal

132
104.131.86.98
104.131.75.130
104.236.9.114
Terminal

133
104.131.86.98
104.131.75.130
104.236.9.114
Terminal

134
104.131.86.98
104.131.75.130
104.236.9.114
37.139.11.13
37.139.4.172
37.139.2.208
Terminal

s
(on any network topology)
135
Provision, secure, connect,
and run any infrastructure
for any application.

▪ Consul’s four network models map to a large set of use cases
learned from years of developing and supporting Consul on the road
to Consul 1.0
▪ Models allow nuanced tradeoffs around cost, features, connectivity
requirements, and levels of isolation
▪ Models can be in use simultaneously, enabling incremental adoption
and rollout
▪ Gossip provides a unique set of primitives that operate across all
network models
▪ Prepared queries allow any application to benefit, even if they are
only using Consul’s DNS interface
Consul and Complex Networks
136

Thank you.
hello@hashicorp.comwww.hashicorp.com

Consul and Complex Networks

More Related Content

What's hot

Viewers also liked

Similar to Consul and Complex Networks

Recently uploaded

Consul and Complex Networks