Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Remco Overdijk
LEAD OPERATIONS ENGINEER
Automating the Cloud
Integrating Puppet with
Cloud Infrastructures
@MaxServ @RemzJ...
Automating
the Cloud
3AUTOMATING THE CLOUD
Traditional Website Hosting
MySQL
AUTOMATING THE CLOUD
Single Server Infrastructure Issues
PROBLEM
Limited headroom
Service issues affect other services
One...
Image credit: https://commons.wikimedia.org/wiki/File:Grumpy-Cat.jpg - Rjommabolli (CC 4.0)
AUTOMATING THE CLOUD
Single Server Infrastructure Issues
6
Scalability
Service Isolation
Redundancy
Zero-Downtime Deployme...
• AWS specific, but applies to most (if not any) Cloud platforms.
AUTOMATING THE CLOUD
Scope
7
• LAMP stack, but works for...
AUTOMATING THE CLOUD
AWS Infrastructure Principles
Read more
https://media.amazonwebservices.com/AWS_Cloud_Best_Practices....
AUTOMATING THE CLOUD
AWS Infra: A lot of moving parts
9
Provisioning
Infrastructure
• IAM Server Certificates
• IAM Instance Profiles
• IAM Roles
• IAM Policies
• IAM Role Policies
• CodeDeploy Apps
• CodeD...
Provisioning Infrastructure
That’s a lot of clicking
13
• Manual changes are extremely error-prone.
• Manual changes resul...
Provisioning Infrastructure
Provisioning
14
• Uses the same DSL as your ”regular” Puppet code.
Provisioning Infrastructure
Puppetlabs-AWS
15
• Most benefits from ”reg...
`AWS_PROFILE=my-aws-account AWS_REGION=eu-west-1 puppet apply
aws-stack.pp --templatedir ./templates/`
Provisioning Infrastructure
Hashicorp Terraform
• Uses HCL, which looks a lot like Puppet DSL.
• Maintains a dependency gr...
Provisioning Infrastructure
Hashicorp Terraform
18
View & try the full source
https://github.com/MaxServ/Terraform-Puppet-Demo/tree/master/Terraform
Provisioning Infrastructure
Puppet?
20
• Automated Infrastructure is awesome.
• We like Puppet too.
• Empty EC2 instances ...
Bootstrapping
Puppet
Bootstrapping Puppet
Machine Configuration States
22
SLIDE CREDIT: Tim Bell, CERN – PuppetConf 2012
https://speakerdeck.com/puppetlabs/cern-accelerating-science-with-puppet
ht...
Image credit: Geert Orye, https://flic.kr/p/7zDqXY (CC BY 2.0)
Bootstrapping Puppet
So, Cattle.
• Configuration should be at type level. Not node level.
• Nodes should be replaceable.
•...
Bootstrapping Puppet
Puppet: Autosigning
• Automates indoctrination for new nodes.
• Multiple mechanisms available:
Naïve,...
Bootstrapping Puppet
Puppet: Autosigning
27
Bootstrapping Puppet
Node Manifests using Regex
28
Read more
https://docs.puppetlabs.com/puppet/latest/reference/lang_node...
Bootstrapping Puppet
Hiera
• Configuration at `type` level instead of `clientcert`.
• Use node-level overrides when requir...
Bootstrapping Puppet
Hiera
30
Bootstrapping Puppet
ENC’s in Foreman
31
Read more
https://docs.puppetlabs.com/guides/external_nodes.html
http://theforema...
Bootstrapping Puppet
Foreman: Default Host Groups
• The same idea as Autosigning in Puppet.
• Uses a Foreman Plugin.
• Use...
Bootstrapping Puppet
Foreman: Default Host Groups
33
Bootstrapping Puppet
Assigning Configuration: Facter
34
View & try
https://github.com/MaxServ/Terraform-Puppet-Demo/blob/m...
Bootstrapping Puppet
AWS: User data
• User data allows you to pass configuration data to an instance
during launch.
• User...
Bootstrapping Puppet
AWS: User data
36
Bootstrapping Puppet
Cloud-init
37
• Handles early initialization of cloud instances.
• Supports EC2, CloudStack, OpenStac...
Bootstrapping Puppet
Cloud-init with Puppetlabs-Aws
38
Read more
https://docs.puppetlabs.com/puppet/latest/reference/lang_...
Bootstrapping Puppet
Cloud-init with Terraform
39
View & try
https://github.com/MaxServ/Terraform-Puppet-Demo/blob/master/...
Bootstrapping Puppet
Don’t forget to clean up!
• Revoke Puppet-CA certificates for decommissioned nodes.
• Clean up Salt k...
Demo
Demo
Terraform & Puppet
THIS DEMONSTRATION INCLUDES:
VIEW & TRY THE FULL SOURCE OF THIS DEMO
https://github.com/MaxServ/Te...
Demo
Virtual Private Cloud
43
44
0416 - 30 10 00
Remco Overdijk
LEAD OPERATIONS ENGINEER
info@maxserv.com
Questions?
@MaxServ @RemzJay
Integrating Puppet with Cloud Infrastructures-Remco Overdijk
Upcoming SlideShare
Loading in …5
×

Integrating Puppet with Cloud Infrastructures-Remco Overdijk

867 views

Published on

Presentation Puppet Camp Amsterdam 2016

Published in: Technology
  • Be the first to comment

Integrating Puppet with Cloud Infrastructures-Remco Overdijk

  1. 1. Remco Overdijk LEAD OPERATIONS ENGINEER Automating the Cloud Integrating Puppet with Cloud Infrastructures @MaxServ @RemzJay
  2. 2. Automating the Cloud
  3. 3. 3AUTOMATING THE CLOUD Traditional Website Hosting MySQL
  4. 4. AUTOMATING THE CLOUD Single Server Infrastructure Issues PROBLEM Limited headroom Service issues affect other services One outage means downtime Maintenance during deployment 4
  5. 5. Image credit: https://commons.wikimedia.org/wiki/File:Grumpy-Cat.jpg - Rjommabolli (CC 4.0)
  6. 6. AUTOMATING THE CLOUD Single Server Infrastructure Issues 6 Scalability Service Isolation Redundancy Zero-Downtime Deployments SOLUTIONPROBLEM Limited headroom Affected services One outage means downtime Maintenance during deployment
  7. 7. • AWS specific, but applies to most (if not any) Cloud platforms. AUTOMATING THE CLOUD Scope 7 • LAMP stack, but works for most stacks. • Mix & match for best results. • The method that works best depends on your own setup. • Based on Puppet Open Source. • Things may be different in Puppet Enterprise (Orchestrator). • Ready-to-go AMI’s may be faster to launch, but harder to maintain.
  8. 8. AUTOMATING THE CLOUD AWS Infrastructure Principles Read more https://media.amazonwebservices.com/AWS_Cloud_Best_Practices.pdf 8 • Infrastructure is failure-prone; Service robustness is achieved through redundancy. • EC2 instances should be considered volatile. • Scaling should be both vertical and horizontal. • Legacy applications aren’t magically cloud-ready. • Loose coupling improves scalability. • Isolation increases security and decreases dependencies.
  9. 9. AUTOMATING THE CLOUD AWS Infra: A lot of moving parts 9
  10. 10. Provisioning Infrastructure
  11. 11. • IAM Server Certificates • IAM Instance Profiles • IAM Roles • IAM Policies • IAM Role Policies • CodeDeploy Apps • CodeDeploy Deployment Groups • EC2 Instances • EC2 Elastic IP’s • CloudWatch Metrics • CloudWatch Alarms • EIP Attachments • ElastiCache Subnet Groups Provisioning Infrastructure AWS Resources • ElastiCache Clusters • ElastiCache Parameter Groups • EC2 Elastic Load Balancers • ELB Health Checks • ELB App Cookie Stickiness Policies • Key Pairs • RDS Subnet Groups • RDS Parameter Groups • RDS DB Instances • Route53 Zones • Route53 Records • S3 Buckets • S3 Policies • S3 CORS Configuration • Security Groups • SNS Topics • SNS Topic Subscriptions • VPC’s • VPC Subnets • VPC Internet Gateways • VPC (S3) Endpoints • VPC Route Tables • VPC Customer Gateways • VPC VPN Gateways • VPC DHCP Option Sets • VPC VPN Routes 11
  12. 12. Provisioning Infrastructure That’s a lot of clicking 13 • Manual changes are extremely error-prone. • Manual changes result in an inconsistent platform. • Collaboration is difficult; People are scared to break things. • Changes are scattered throughout the AWS console.
  13. 13. Provisioning Infrastructure Provisioning 14
  14. 14. • Uses the same DSL as your ”regular” Puppet code. Provisioning Infrastructure Puppetlabs-AWS 15 • Most benefits from ”regular” Puppet; like relationships, defined types and the graph. • At the moment this module only supports a few of the resources in the AWS API. • Does NOT run using agents / puppet masters, but manually from your workstation using `puppet apply`. • Not as idempotent as you’d like at times. Read more https://github.com/puppetlabs/puppetlabs-aws https://forge.puppetlabs.com/puppetlabs/aws
  15. 15. `AWS_PROFILE=my-aws-account AWS_REGION=eu-west-1 puppet apply aws-stack.pp --templatedir ./templates/`
  16. 16. Provisioning Infrastructure Hashicorp Terraform • Uses HCL, which looks a lot like Puppet DSL. • Maintains a dependency graph, just like Puppet. • Runs from your workstation. • Is not AWS API feature complete, but covers most services. • Allows flexible scaling and destruction. 17 Read more https://github.com/hashicorp/terraform https://www.terraform.io/
  17. 17. Provisioning Infrastructure Hashicorp Terraform 18
  18. 18. View & try the full source https://github.com/MaxServ/Terraform-Puppet-Demo/tree/master/Terraform
  19. 19. Provisioning Infrastructure Puppet? 20 • Automated Infrastructure is awesome. • We like Puppet too. • Empty EC2 instances don’t serve websites. • How do we connect Puppet to the infrastructure we’ve just provisioned?
  20. 20. Bootstrapping Puppet
  21. 21. Bootstrapping Puppet Machine Configuration States 22
  22. 22. SLIDE CREDIT: Tim Bell, CERN – PuppetConf 2012 https://speakerdeck.com/puppetlabs/cern-accelerating-science-with-puppet https://youtu.be/-Ykb2j2ojYU?t=19m33s
  23. 23. Image credit: Geert Orye, https://flic.kr/p/7zDqXY (CC BY 2.0)
  24. 24. Bootstrapping Puppet So, Cattle. • Configuration should be at type level. Not node level. • Nodes should be replaceable. • Data on the nodes should be considered volatile. • Only versioned and cached content should be present. 25 • Provisioning should be automated; No manual intervention should be required.
  25. 25. Bootstrapping Puppet Puppet: Autosigning • Automates indoctrination for new nodes. • Multiple mechanisms available: Naïve, Basic and Policy-based Autosigning. 26 View & try https://github.com/MaxServ/Terraform-Puppet-Demo/blob/master/Puppet/puppet.conf Read more https://docs.puppetlabs.com/puppet/latest/reference/ssl_autosign.html Be very careful with naïve autosigning. Don’t do it in production without strict firewalls.
  26. 26. Bootstrapping Puppet Puppet: Autosigning 27
  27. 27. Bootstrapping Puppet Node Manifests using Regex 28 Read more https://docs.puppetlabs.com/puppet/latest/reference/lang_node_definitions.html#regular-expression-names
  28. 28. Bootstrapping Puppet Hiera • Configuration at `type` level instead of `clientcert`. • Use node-level overrides when required. • Combine with host `%H` mount points for master/slave. 29 View & try https://github.com/MaxServ/Terraform-Puppet-Demo/tree/master/Puppet/hieradata Read more https://docs.puppetlabs.com/hiera/latest/ https://docs.puppetlabs.com/guides/file_serving.html#file-server-configuration
  29. 29. Bootstrapping Puppet Hiera 30
  30. 30. Bootstrapping Puppet ENC’s in Foreman 31 Read more https://docs.puppetlabs.com/guides/external_nodes.html http://theforeman.org/manuals/1.10/index.html#2.Quickstart
  31. 31. Bootstrapping Puppet Foreman: Default Host Groups • The same idea as Autosigning in Puppet. • Uses a Foreman Plugin. • Use facts for assigning host groups. 32 Read more https://github.com/theforeman/foreman_default_hostgroup
  32. 32. Bootstrapping Puppet Foreman: Default Host Groups 33
  33. 33. Bootstrapping Puppet Assigning Configuration: Facter 34 View & try https://github.com/MaxServ/Terraform-Puppet-Demo/blob/master/Terraform/templates/userdata.tpl https://github.com/MaxServ/Terraform-Puppet-Demo/blob/master/Puppet/Vagrantfile Read more https://docs.puppetlabs.com/facter/2.4/custom_facts.html#structured-data-facts
  34. 34. Bootstrapping Puppet AWS: User data • User data allows you to pass configuration data to an instance during launch. • User data can be used to provide cloud-init configuration. 35
  35. 35. Bootstrapping Puppet AWS: User data 36
  36. 36. Bootstrapping Puppet Cloud-init 37 • Handles early initialization of cloud instances. • Supports EC2, CloudStack, OpenStack, OnApp, OpenNebula, RHEVm, vSphere & more. • Can install packages, do basic config and more. • Is able to bootstrap Puppet. Read more http://cloudinit.readthedocs.org/en/latest/index.html http://cloudinit.readthedocs.org/en/latest/topics/examples.html#setup-and-run-puppet
  37. 37. Bootstrapping Puppet Cloud-init with Puppetlabs-Aws 38 Read more https://docs.puppetlabs.com/puppet/latest/reference/lang_template_erb.html
  38. 38. Bootstrapping Puppet Cloud-init with Terraform 39 View & try https://github.com/MaxServ/Terraform-Puppet-Demo/blob/master/Terraform/ec2.tf Read more https://docs.puppetlabs.com/puppet/latest/reference/lang_template_erb.html
  39. 39. Bootstrapping Puppet Don’t forget to clean up! • Revoke Puppet-CA certificates for decommissioned nodes. • Clean up Salt keys. • Remove old reports, exported resources and catalogs from PuppetDB. • Clean connected resources like Load Balancers. 40 Sloppiness will catch up with you when it hurts the most.
  40. 40. Demo
  41. 41. Demo Terraform & Puppet THIS DEMONSTRATION INCLUDES: VIEW & TRY THE FULL SOURCE OF THIS DEMO https://github.com/MaxServ/Terraform-Puppet-Demo A B C Docker container running a Puppetmaster. AWS Stack with 4 webservers using Terraform. Webserver provisioning using Puppet based on type. 42
  42. 42. Demo Virtual Private Cloud 43
  43. 43. 44
  44. 44. 0416 - 30 10 00 Remco Overdijk LEAD OPERATIONS ENGINEER info@maxserv.com Questions? @MaxServ @RemzJay

×