Puppet and your Metadata - PuppetCamp London 2015

1. Marc Cluet https://www.flickr.com/photos/jsprig/5106529710/

2. Checklist https://www.flickr.com/photos/timothytsuihin/2743062701/ https://www.flickr.com/photos/zachinglis/5507648594/ https://www.flickr.com/photos/epsos/8270256961/ https://www.flickr.com/photos/mini82/1361976303/

3. Who am I? Marc Cluet (@lynxman) Grumpy Engineer based in London Co-‐Founder of Ukon Cherry 17 years of experience as a SysAdmin Founding member of Juju and MAAS while at Canonical Built a DevOps Engineering Team at Rackspace Been DevOps’in for the last 6 years

4. Who am I? http://www.meetup.com/London-‐DevOps/

5. What is Metadata? https://www.flickr.com/photos/annarbor/4349876203/

6. What is Metadata? Metadata is “data about data” Structural Metadata Descriptive Metadata

7. What is Metadata? Structural Metadata architecture => amd64 ipaddress => 10.0.115.197 Descriptive Metadata $puppetversion = 3.7.5 $apacheversion = 2.4

8. Metadata purpose? Abstract Definitions Unique Data

9. Metadata purpose? Abstract Definitions Unique Data

10. Metadata in Puppet https://www.flickr.com/photos/jimmcd/4859841581/

11. Metadata in Puppet Variables Facts Hiera PuppetDB

12. Variables class apples { $apples = 5 $string = “I have ${apples} apples” } $apples::apples $apples::string global

13. Scope web1 mail web2 apache global

14. Exported resources Class newclient { @@sshkey { $::hostname: type => dsa, key => $mykey, } }

15. Exported resources Server SSH bastion class bastion { Sshkey <<| |>> }

16. Facts Facter Puppet

17. Facts $ facter interfaces => eth0,lo ipaddress => 10.0.115.197 ipaddress_eth0 => 10.0.115.197 ipaddress_lo => 127.0.0.1 is_virtual => true kernel => Linux kernelmajversion => 3.13

18. Custom Facts Where /etc/facter/facts.d What Text File Script Output role=webserver

19. Custom Facts $ facter role webserver

20. Scope web1 facts web2 apache mail global

21. Metadata Distribution https://www.flickr.com/photos/wallyg/6271443142/

22. PuppetDB Facter Puppet PuppetDB

23. PuppetDB Install Debian/Ubuntu $ apt-‐get install puppetdb RedHat $ yum install puppetdb

24. PuppetDB Data Keeps Reports Facts Exported Resources (searchable) Used by Puppet Client (Exp Resources) Reporting

25. Hiera Facter Puppet Hiera PuppetDB

26. Hiera Install $ gem install hiera

27. Hiera Config /etc/hiera.yaml -‐-‐-‐ :backends: -‐ yaml :hierarchy: -‐ "%{environment}/nodes/%{fqdn}” -‐ "%{environment}/roles/%{role}” -‐ "%{environment}/common” -‐ common :yaml: :datadir: /etc/puppet/hieradata

28. Hiera Data /etc/puppet/hieradata/common.yaml # Define our variable example: foo

29. Hiera Data in Puppet class test { $hieravar = hiera(‘example’) }

30. Hiera Data in Puppet class test { $hieravar = hiera(‘example’,’bar’) }

31. Hiera Data Hierarchy /etc/puppet/hieradata/common.yaml # Apache variables apache::rootdir: /www apache::user: www-‐data

32. Hiera Loops /etc/puppet/hieradata/roles/apache.yaml # Apache variables vhost: site rootdir: "%{hiera(’apache::rootdir')}/site" user: "%{hiera(’apache::user')}"

33. Modules consume Hiera https://forge.puppetlabs.com/mthibaut/users /etc/puppet/hieradata/common.yaml # Define our users users_sysadmins: marc: ensure: present uid: 10001 gid: staff groups: sysadmin comment: Marc Cluet shell: /bin/bash

34. Hiera Security

35. Hiera Security Hiera Security Projects Hiera GPG https://github.com/crayfishx/hiera-‐gpg Hiera eyaml https://github.com/TomPoulton/hiera-‐eyaml

36. Hiera eyaml Install $ gem install hiera_eyaml eyaml {createkeys decrypt edit encrypt recrypt}

37. Hiera eyaml /etc/hiera.yaml -‐-‐-‐ :backends: -‐ eyaml :hierarchy: -‐ "%{environment}/nodes/%{fqdn}” -‐ "%{environment}/roles/%{role}” -‐ "%{environment}/common” -‐ common :eyaml: :datadir: /etc/puppet/hieradata :pkcs7_private_key: "/secure/private.pkcs7.pem” :pkcs7_public_key: "/secure/public.pkcs7.pem"

38. Hiera eyaml secretpassword: DEC::PCSK7[potato]!

39. Hiera eyaml secretpassword: ENC[PKCS7,MIIBiQYJKoZIhvcNAQcDoII BejCCAXYCAQAxggEhMIIBHQIBADAFMAACAQEwDQYJKoZIhvcNA QEBBQAEggEAGPLCXn8goh27KO3msvCz6GJydTHTNrplQ23dR/d fV0MuSmYwnASFa9RDpqe9K5fhs5XO5TfyQ5Uf4IAZxhnuDx+3Z SiVY1lbXY6C3x6XeXUBN0jfB5FkdrR+mMYCzGRbVB3gPlM0I8g 2Wq2397h5zMHRkFizPr16vhKuQDxMeGlq8yfoZ4FqwhUteeYxP MnCV7lx0I6Z/e8I4UZek4FQ7dMfuXNerdDAxx+UMZjRhK3trQl R/x2TLS4vgh8m0Ml3F5q851W4s5O4ZE7/wvq//nvlr1RoH5EnP pPQ6shv6R6I7lBfViqIRJalRdyTWVLpFgUknXKLRzoxH8S7IlD 5PzBMBgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBA66G5+tBaaF 9g8Lq4dTrwNgCCVP77OWTPoSWGyfxNkuTrmoOANAL98kVWXioO re612Vw==]

40. Connecting Metadata https://www.flickr.com/photos/lightspectrals/1447079898/

41. Automation metadata Provisioner Puppet Monitoring Services

42. Automation metadata Provisioner Puppet Monitoring Services ?

43. Metadata Partitioning Service Node A Load Balancer Service Node B Service Node C

44. Metadata Partitioning Service Node A Load Balancer Service Node C

45. Metadata partitioning https://www.flickr.com/photos/genista/4346973713/

46. Detour! https://www.flickr.com/photos/bionicteaching/14586204543/

47. Service Discovery

48. What is Service Discovery? Service Node A Health Check Discovery Agent Discovery Agent Service Node B Health Check Service Publication Node A Node B

49. Metadata Unification https://www.flickr.com/photos/bangorfuji9500/7152182001/

50. Metadata Unification hiera_mysql https://forge.puppetlabs.com/crayfishx/hiera_mysql hiera_redis https://forge.puppetlabs.com/rubyisbeautiful/hiera_redis hiera_http https://forge.puppetlabs.com/crayfishx/hiera_http hiera_etcd https://forge.puppetlabs.com/garethr/hiera_etcd hiera_consul https://forge.puppetlabs.com/lynxman/hiera_consul

51. Metadata Unification Puppet yaml Hiera consul consul k/v 1 2 3 2

52. Hiera Consul /etc/hiera.yaml -‐-‐-‐ :backends: -‐ consul :consul: :host: 127.0.0.1 :port: 8500 :failure: graceful :paths: -‐ /v1/catalog/service -‐ /v1/catalog/node

53. Hiera Consul http://localhost:8500/v1/catalog/service/sensu [ { "ServicePort": 0, "ServiceTags": [], "ServiceName": "sensu", "ServiceID": "sensu", "Address": "10.0.115.197", "Node": "ip-‐10-‐0-‐115-‐197“ } ]

54. Hiera Consul http://localhost:8500/v1/catalog/service/sensu [ { "ServicePort": 0, "ServiceTags": [], "ServiceName": "sensu", "ServiceID": "sensu", "Address": "10.0.115.197", "Node": "ip-‐10-‐0-‐115-‐197“ } ]

55. Hiera Consul $sensu_service = hiera('sensu',[]) $sensu_service[ServicePort] $sensu_service[ServiceTags] $sensu_service[ServiceName] $sensu_service[ServiceID] $sensu_service[Address] $sensu_service[Node]

56. Hiera Consul $sensu_service = hiera('sensu',[]) $s_address = consul_info($sensu_service, 'Address')

57. Warnings & Advice https://www.flickr.com/photos/catherinekolodziej/8866489274/

58. Facter Facter needs to be FAST Don’t put here long running scripts Cron it up! Text always works better

59. Hiera Hiera is awesome! Be careful of long query loops

60. Who’s the brain? Provisioner Puppet Orchestration

64. Who’s the brain? It’s your decision! Keep coherency Unified metadata is powerful

65. https://www.flickr.com/photos/mikko_luntiala/12691267935 https://www.flickr.com/photos/dullhunk/202872717 @lynxman http://slideshare.net/lynxmanuk/ https://github.com/lynxman/ https://devroot.org/