Presentation about Service Discovery for Puppetconf 2016 in San Diego, California

1. Service Discovery and Puppet
Marc Cluet

2. Who am I?
@lynxman
Engineer based in London
Co-Founder of Ukon Cherry
Working at TrainLine
19 years of experience as a SysAdmin
Founding member of Juju and MAAS while at Canonical
Built a DevOps Engineering Team at Rackspace
Been DevOps’in for the last 7 years
2

3. Meet other people in London!
3
http://www.meetup.com/London-DevOps/

4. Running in the Cloud?
4
https://www.flickr.com/photos/pontla/11879018534/

5. The Problem
Cloud is Hard!
Old Style DNS
TTL a problem for auto-scaling
Non-reactive health checks
Where do I keep my metadata?
5
https://www.flickr.com/photos/qchristopher/5038229030/

6. The Solution!
You were expecting this weren't you
6

7. Service Discovery
Automatically define your services
Active Health Checking
Dynamically updated service lists
Can be DNS accessible (if needed)
API accessible (win!)
7
https://www.flickr.com/photos/marknye/12992319144/

8. Service Discovery
8
Service
Publication
Service
Node A
Service
Node B
Service
Node C

9. Service Discovery
9
Service
Publication
Service
Node A
Health
Check
Discovery
Agent
Discovery
Agent
Service
Node B
Health
Check

10. Service Discovery
10
Service
Publication
Service
Node A
Health
Check
Discovery
Agent
Discovery
Agent
Service
Node B
Health
Check

11. Service Discovery
11
Service
Publication
Service
Node A
Health
Check
Discovery
Agent
Discovery
Agent
Service
Node B
Health
Check

12. Service Discovery
12
Service
Publication
Service
Node A
Health
Check
Discovery
Agent
Discovery
Agent
Service
Node B
Health
Check

13. Amazeballs!
13
https://www.flickr.com/photos/neilmartin/31519343/

14. Amazeballs!
14
https://www.flickr.com/photos/neilmartin/31519343/

15. Service Discovery
15
Service
Node A
Service: web
10.10.10.1
10.10.10.2
10.10.10.3
Service
Node B
Service
Node C

16. Service Discovery
16
Service
Node A
Service: web
10.10.10.1
10.10.10.2
10.10.10.3
Service
Node B
Service
Node C

17. Service Discovery
17
Service
Node A
Service: web
10.10.10.1
10.10.10.2
10.10.10.3
Service
Node B
Service
Node C

18. Amazeballs!
18
https://www.flickr.com/photos/neilmartin/31519343/

19. Service Discovery
19
Agent Master Agent Agent Agent Agent
Agent Agent Agent Agent Master Agent
Agent Agent Master Agent Agent Agent
Agent Agent Agent Agent Agent Agent

20. Service Discovery
20
Agent Master Agent Agent Agent Agent
Agent Agent Agent Agent Master Agent
Agent Agent Master Agent Agent Agent
Agent Agent Agent Agent Agent Agent

21. Service Discovery
21
Agent Master Agent Agent Agent Agent
Agent Agent Agent Agent Master Agent
Agent Agent Master Agent Agent Agent
Agent Agent Agent Agent Agent Agent

22. Service Discovery
22
Agent Master Agent Agent Agent Agent
Agent Agent Agent Agent Master Agent
Agent Agent Master Agent Agent
Agent Agent Agent Agent Agent Agent

23. Service Discovery
23
Agent Master Agent Agent Agent Agent
Agent Agent Agent Agent Master Agent
Agent Agent Master Agent Agent
Agent Agent Agent Agent Agent Agent
! !
!

24. Service Discovery
24
Agent Master Agent Agent Agent Agent
Agent Agent Agent Agent Master Agent
Agent Agent Master Agent Agent Agent
Agent Agent Agent Agent Agent Agent

25. Amazeballs!
25
https://www.flickr.com/photos/neilmartin/31519343/

26. Service Discovery Solutions
Choices!
26

27. Service Discovery Solutions
CoreOS Fleet
Uses the sidekick
model, uses separate
agents to orchestrate
service discovery.
Kubernetes Service
All pods declare
services to be
discoverable by the
cluster.
27
Consul
All services are
declared by the
agents, the agents
themselves are
responsible.

28. Service Discovery Solutions
CoreOS Fleet
API publication
k/v Strongly Consistent
Container checks
Kubernetes Service
API publication
Container checks
Auto-Heal
28
Consul
API + DNS publication
k/v Strongly Consistent
Host + Service checks
Puppet modulePuppet module
Hiera access
Puppet module
Hiera access

29. Service Discovery Solutions
29

30. What is Consul?
It's a Service Discovery System (duh!)
Service Publications (DNS + API)
k/v Storage (strongly consistent)
Health Checks
With encryption! (whaaaaaat)
30

31. What is Consul?
31

32. Consul Concepts
Datacenter
Node
Service
Health Check
Watches
ACLs
32

33. Consul Architecture
33
Consul
Master
Consul
Master
Consul
Master

34. Consul Architecture
34
Consul
Master
Consul
Master
Consul
Master
Consul
Agent
Consul
Agent
Consul
Agent
Consul
Agent
Consul
Agent
Consul
Agent
Consul
Agent

35. Consul Architecture
35
Consul
Master
Consul
Master
Consul
Master
Consul
Agent
Consul
Agent
Consul
Agent
Consul
Agent
Consul
Agent
Consul
Agent
Consul
Agent
Consul
Master
WAN DC

36. Consul DNS publisher
Port 8600
Use dnsmasq:
server=/consul/127.0.0.1#8600
36

37. Consul DNS publisher
service name: web
zone: yourDC
<servicename>.service.<zone>.consul
web.service.yourDC.consul
37

38. Consul Catalog API
API http://localhost:8500
/v1/catalog/
/v1/catalog/datacenters/
/v1/catalog/nodes/
/v1/catalog/services/<servicename>
/v1/catalog/service/<name>
/v1/catalog/node/<nodename>
38

39. Consul Health Checks
{
"check": {
"id": "mem-util",
"name": "Memory utilization",
"script": "/usr/local/bin/check_mem.py",
"interval": "10s"
}
}
39

40. Consul Health Checks
40
Server

41. Consul Health Checks
41
Server
Memory
Disk
CPU
Load
Logs

42. Consul Health Checks
42
Server
Service Service

43. Consul Restful API
API http://localhost:8500
/v1/kv/
/v1/agent/
/v1/catalog/
/v1/health/
/v1/session/
/v1/acl/
/v1/status/
43

44. Consul Restful API
API http://localhost:8500
/v1/kv/
/v1/agent/
/v1/catalog/
/v1/health/
/v1/session/
/v1/acl/
/v1/status/
44

45. Now comes the Puppet stuff!
All of this applied!
45

46. You can do all this after the talk
https://github.com/lynxman/consul-first-steps
46
https://www.flickr.com/photos/hortlander/6245707871/

47. Puppet + Consul
47
https://www.flickr.com/photos/hortlander/6245707871/in/
You can integrate at several levels
Puppet Module
KyleAnderson-consul
Hiera
lynxman-hiera_consul

48. Puppet + Consul
48
https://www.flickr.com/photos/hortlander/6245707871/in/
You can integrate at several levels
k/v access
venmo-consulr
Templates with Consul
ghdbaston-consul_template

49. Puppet Module - Install
49
https://www.flickr.com/photos/hortlander/6245707871/in/
Puppet Forge makes it very easy to install
$ puppet module install KyleAnderson-consul

50. Puppet Module - Dnsmasq Install
50
https://www.flickr.com/photos/hortlander/6245707871/in/
$ puppet module install saz-dnsmasq
include dnsmasq
dnsmasq::conf { 'consul':
ensure => present,
content => 'server=/consul/127.0.0.1#8600',
}

51. Puppet Module - Health Checks
51
https://www.flickr.com/photos/hortlander/6245707871/in/
consul::check { 'disk_space':
script => 'check_disk -w 5% -c 1%',
interval => '30s',
}

52. Puppet Module - Services
52
https://www.flickr.com/photos/hortlander/6245707871/in/
consul::service { 'nginx':
port => '80',
checks => [
{
script => 'check_http -H localhost -w 20 -c 60',
interval => '30s',
},
],
}

53. Puppet Hiera
53
Puppet Hiera
Consul
Hiera
Yaml

54. Puppet Hiera Module - Install
54
https://www.flickr.com/photos/hortlander/6245707871/in/
Puppet Forge makes it very easy to install
$ puppet module install lynxman-hiera_consul

55. Puppet Hiera Module - Config
55
https://www.flickr.com/photos/hortlander/6245707871/in/
:backends:
- yaml
- consul
:yaml:
:datadir: /etc/puppetlabs/hieradata
:consul:
:host: 127.0.0.1
:port: 8500
:failure: graceful
:paths:
- /v1/catalog/service
- /v1/catalog/node

56. Puppet Hiera Module - Config
56
https://www.flickr.com/photos/hortlander/6245707871/in/
:consul:
:host: 127.0.0.1
:port: 8500
:failure: graceful
:paths:
- /v1/catalog/service
- /v1/catalog/node

57. Puppet Hiera Module - Arrays
57
https://www.flickr.com/photos/hortlander/6245707871/in/
notice('Generating rabbitmq cluster members based on Consul
information')
$consul_service_array = hiera('rabbitmq',[])
$mq_cluster_nodes = consul_info($consul_service_array, 'Address')
notice("Result: ${mq_cluster_nodes}")

58. Puppet Hiera Module - Arrays
58
https://www.flickr.com/photos/hortlander/6245707871/in/
notice("Generating neo4j_ha cluster members based on Consul
information")
$consul_service_array = hiera('neo4j_ha',[])
$consul_fields = [ 'Address', 'ServicePort' ]
$consul_ha_initial_hosts = consul_info($consul_service_array,
$consul_fields, ':')
$ha_initial_hosts = join($consul_ha_initial_hosts, ',')
notice("Result: ${ha_initial_hosts}")

59. Puppet Hiera Module - Accessing the k/v
59
https://www.flickr.com/photos/hortlander/6245707871/in/
:consul:
:host: 127.0.0.1
:port: 8500
:failure: graceful
:paths:
- /v1/kv/

60. Puppet Hiera Module - Accessing the k/v
60
https://www.flickr.com/photos/hortlander/6245707871/in/
:consul:
:host: 127.0.0.1
:port: 8500
:failure: graceful
:paths:
- /v1/kv/mystuff/

61. Puppet Hiera Module - Accessing the k/v
61
https://www.flickr.com/photos/hortlander/6245707871/in/
:consul:
:host: 127.0.0.1
:port: 8500
:failure: graceful
:paths:
- "/v1/kv/%{env}/"

62. Puppet Security
Don't get your a** on the line!
62

63. Puppet Security - Github
63
https://www.flickr.com/photos/hortlander/6245707871/in/
$ git commit -m "All my passwords"
$ git push

64. Puppet Hiera Module - Arrays
64
https://www.flickr.com/photos/hortlander/6245707871/in/
https://www.flickr.com/photos/tomukas/3554360505/

65. Puppet Security - Hiera Modules
65
https://www.flickr.com/photos/hortlander/6245707871/in/
hiera-eyaml
hiera-gpg

66. Puppet Security - Hiera
66
Puppet Hiera
Consul
Hiera
eyaml

67. Puppet Security - Module Install
67
https://www.flickr.com/photos/hortlander/6245707871/in/
Puppet Forge makes it very easy to install
$ puppet module install hiera-eyaml

68. Puppet Security - Hiera eyaml
68
https://www.flickr.com/photos/hortlander/6245707871/in/
$ eyaml createkeys
keys/public_key.pkcs7.pem
keys/private_key.pkcs7.pem

69. Puppet Security - Hiera eyaml
69
https://www.flickr.com/photos/hortlander/6245707871/in/
Usage: eyaml <subcommand>
Please use one of the following subcommands
or help for more help:
createkeys, decrypt, edit, encrypt,
recrypt, version

70. Puppet Security - Hiera eyaml
70
https://www.flickr.com/photos/hortlander/6245707871/in/
mysecret: DEC::PKCS7[mypassword]!

71. Puppet Security - Hiera eyaml
71
https://www.flickr.com/photos/hortlander/6245707871/in/
mysecret:
ENC[PKCS7,MIIBmQYJKoZIhvcNAQcDoIIBijCCAYYCAQAxggEhMIIBHQI
BADAFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEAAikKizydVh0wXQrvtoMC
7vM9NxfksqwOX2jtajDYMMJwXXP/5zKHjnnGmr
+LSXFVkL52FuGentCdityjF0zZEvbZ2D95TWnRTinO9htteC8ZiwqpTeu
NJkTJikOEEZvHbNlp6eX381ppKoatV1k0EmIHwsnqeRJN5T9TVScoXOb/
1Fre4H7TxSvvaFqo02MWUBaKkWECoEu2PLiuXWEoiLrkDq8pxhjYADvGU
JLWC8PUSWT/94075z5UKHYBQgLlFrzG+89Rhm5keTy/
cuHsOK9d0nUScjd4m6duCEsvRT5SG/n6GwTEk/
cDMqIuvAwNETv2fdepu4z5nR383zlngDBcBgkqhkiG9w0BBwEwHQYJYIZ
IAWUDBAEqBBAJCDkds8PbXeBUMZhFPxWTgDDH1pvUCbCLtWDNVFkW2yZ1
NYF06RuqsSTxofHfMwajC+BSPcTu7heMKQnbKP/KE6o=]

72. Puppet Security Extra time!
This is beyond awesome
72

73. Puppet Security - Extra
73
https://www.flickr.com/photos/hortlander/6245707871/in/

74. Puppet Security - Extra
Friday, October 21 • 11:15am - 12:00pm
Using HashiCorp's Vault With Puppet
Seth Vargo, HashiCorp
74