Downloaded 12 times
Uploaded byMaxServ
Integrating Puppet with Cloud Infrastructures-Remco Overdijk
This document discusses automating cloud infrastructure using Puppet. It begins by describing issues with traditional single server infrastructure like limited scalability and redundancy. It then introduces using tools like AWS, Puppet, and Terraform to provision infrastructure in the cloud with improved scalability, isolation, and zero-downtime deployments. It discusses using Puppet and Terraform to define and provision AWS resources declaratively. It also covers bootstrapping Puppet onto new instances using techniques like autosigning, ENCs, Hiera lookups, AWS user data, and Cloud-init to automate configuration. The document concludes with a demonstration of provisioning a stack of web servers on AWS using Terraform and Puppet.
More Related Content
![[OpenInfra Days Korea 2018] Day 2 - E5-1: "Invited Talk: Kubicorn - Building ...](https://cdn.slidesharecdn.com/ss_thumbnails/e60955buildingsimplekubernetes-180705043459-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Integrating Puppet with Cloud Infrastructures-Remco Overdijk
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
Integrating Puppet with Cloud Infrastructures-Remco Overdijk
- 1. Remco Overdijk LEAD OPERATIONSENGINEER Automating the Cloud Integrating Puppet with Cloud Infrastructures @MaxServ @RemzJay
- 2.
- 3. 3AUTOMATING THE CLOUD TraditionalWebsite Hosting MySQL
- 4. AUTOMATING THE CLOUD SingleServer Infrastructure Issues PROBLEM Limited headroom Service issues affect other services One outage means downtime Maintenance during deployment 4
- 5. Image credit: https://commons.wikimedia.org/wiki/File:Grumpy-Cat.jpg- Rjommabolli (CC 4.0)
- 6. AUTOMATING THE CLOUD SingleServer Infrastructure Issues 6 Scalability Service Isolation Redundancy Zero-Downtime Deployments SOLUTIONPROBLEM Limited headroom Affected services One outage means downtime Maintenance during deployment
- 7. • AWS specific,but applies to most (if not any) Cloud platforms. AUTOMATING THE CLOUD Scope 7 • LAMP stack, but works for most stacks. • Mix & match for best results. • The method that works best depends on your own setup. • Based on Puppet Open Source. • Things may be different in Puppet Enterprise (Orchestrator). • Ready-to-go AMI’s may be faster to launch, but harder to maintain.
- 8. AUTOMATING THE CLOUD AWSInfrastructure Principles Read more https://media.amazonwebservices.com/AWS_Cloud_Best_Practices.pdf 8 • Infrastructure is failure-prone; Service robustness is achieved through redundancy. • EC2 instances should be considered volatile. • Scaling should be both vertical and horizontal. • Legacy applications aren’t magically cloud-ready. • Loose coupling improves scalability. • Isolation increases security and decreases dependencies.
- 9. AUTOMATING THE CLOUD AWSInfra: A lot of moving parts 9
- 10.
- 11. • IAM ServerCertificates • IAM Instance Profiles • IAM Roles • IAM Policies • IAM Role Policies • CodeDeploy Apps • CodeDeploy Deployment Groups • EC2 Instances • EC2 Elastic IP’s • CloudWatch Metrics • CloudWatch Alarms • EIP Attachments • ElastiCache Subnet Groups Provisioning Infrastructure AWS Resources • ElastiCache Clusters • ElastiCache Parameter Groups • EC2 Elastic Load Balancers • ELB Health Checks • ELB App Cookie Stickiness Policies • Key Pairs • RDS Subnet Groups • RDS Parameter Groups • RDS DB Instances • Route53 Zones • Route53 Records • S3 Buckets • S3 Policies • S3 CORS Configuration • Security Groups • SNS Topics • SNS Topic Subscriptions • VPC’s • VPC Subnets • VPC Internet Gateways • VPC (S3) Endpoints • VPC Route Tables • VPC Customer Gateways • VPC VPN Gateways • VPC DHCP Option Sets • VPC VPN Routes 11
- 13. Provisioning Infrastructure That’s alot of clicking 13 • Manual changes are extremely error-prone. • Manual changes result in an inconsistent platform. • Collaboration is difficult; People are scared to break things. • Changes are scattered throughout the AWS console.
- 14.
- 15. • Uses thesame DSL as your ”regular” Puppet code. Provisioning Infrastructure Puppetlabs-AWS 15 • Most benefits from ”regular” Puppet; like relationships, defined types and the graph. • At the moment this module only supports a few of the resources in the AWS API. • Does NOT run using agents / puppet masters, but manually from your workstation using `puppet apply`. • Not as idempotent as you’d like at times. Read more https://github.com/puppetlabs/puppetlabs-aws https://forge.puppetlabs.com/puppetlabs/aws
- 16. `AWS_PROFILE=my-aws-account AWS_REGION=eu-west-1 puppetapply aws-stack.pp --templatedir ./templates/`
- 17. Provisioning Infrastructure Hashicorp Terraform •Uses HCL, which looks a lot like Puppet DSL. • Maintains a dependency graph, just like Puppet. • Runs from your workstation. • Is not AWS API feature complete, but covers most services. • Allows flexible scaling and destruction. 17 Read more https://github.com/hashicorp/terraform https://www.terraform.io/
- 18.
- 19. View & trythe full source https://github.com/MaxServ/Terraform-Puppet-Demo/tree/master/Terraform
- 20. Provisioning Infrastructure Puppet? 20 • AutomatedInfrastructure is awesome. • We like Puppet too. • Empty EC2 instances don’t serve websites. • How do we connect Puppet to the infrastructure we’ve just provisioned?
- 21.
- 22.
- 23. SLIDE CREDIT: TimBell, CERN – PuppetConf 2012 https://speakerdeck.com/puppetlabs/cern-accelerating-science-with-puppet https://youtu.be/-Ykb2j2ojYU?t=19m33s
- 24. Image credit: GeertOrye, https://flic.kr/p/7zDqXY (CC BY 2.0)
- 25. Bootstrapping Puppet So, Cattle. •Configuration should be at type level. Not node level. • Nodes should be replaceable. • Data on the nodes should be considered volatile. • Only versioned and cached content should be present. 25 • Provisioning should be automated; No manual intervention should be required.
- 26. Bootstrapping Puppet Puppet: Autosigning •Automates indoctrination for new nodes. • Multiple mechanisms available: Naïve, Basic and Policy-based Autosigning. 26 View & try https://github.com/MaxServ/Terraform-Puppet-Demo/blob/master/Puppet/puppet.conf Read more https://docs.puppetlabs.com/puppet/latest/reference/ssl_autosign.html Be very careful with naïve autosigning. Don’t do it in production without strict firewalls.
- 27.
- 28. Bootstrapping Puppet Node Manifestsusing Regex 28 Read more https://docs.puppetlabs.com/puppet/latest/reference/lang_node_definitions.html#regular-expression-names
- 29. Bootstrapping Puppet Hiera • Configurationat `type` level instead of `clientcert`. • Use node-level overrides when required. • Combine with host `%H` mount points for master/slave. 29 View & try https://github.com/MaxServ/Terraform-Puppet-Demo/tree/master/Puppet/hieradata Read more https://docs.puppetlabs.com/hiera/latest/ https://docs.puppetlabs.com/guides/file_serving.html#file-server-configuration
- 30.
- 31. Bootstrapping Puppet ENC’s inForeman 31 Read more https://docs.puppetlabs.com/guides/external_nodes.html http://theforeman.org/manuals/1.10/index.html#2.Quickstart
- 32. Bootstrapping Puppet Foreman: DefaultHost Groups • The same idea as Autosigning in Puppet. • Uses a Foreman Plugin. • Use facts for assigning host groups. 32 Read more https://github.com/theforeman/foreman_default_hostgroup
- 33. Bootstrapping Puppet Foreman: DefaultHost Groups 33
- 34. Bootstrapping Puppet Assigning Configuration:Facter 34 View & try https://github.com/MaxServ/Terraform-Puppet-Demo/blob/master/Terraform/templates/userdata.tpl https://github.com/MaxServ/Terraform-Puppet-Demo/blob/master/Puppet/Vagrantfile Read more https://docs.puppetlabs.com/facter/2.4/custom_facts.html#structured-data-facts
- 35. Bootstrapping Puppet AWS: Userdata • User data allows you to pass configuration data to an instance during launch. • User data can be used to provide cloud-init configuration. 35
- 36.
- 37. Bootstrapping Puppet Cloud-init 37 • Handlesearly initialization of cloud instances. • Supports EC2, CloudStack, OpenStack, OnApp, OpenNebula, RHEVm, vSphere & more. • Can install packages, do basic config and more. • Is able to bootstrap Puppet. Read more http://cloudinit.readthedocs.org/en/latest/index.html http://cloudinit.readthedocs.org/en/latest/topics/examples.html#setup-and-run-puppet
- 38. Bootstrapping Puppet Cloud-init withPuppetlabs-Aws 38 Read more https://docs.puppetlabs.com/puppet/latest/reference/lang_template_erb.html
- 39. Bootstrapping Puppet Cloud-init withTerraform 39 View & try https://github.com/MaxServ/Terraform-Puppet-Demo/blob/master/Terraform/ec2.tf Read more https://docs.puppetlabs.com/puppet/latest/reference/lang_template_erb.html
- 40. Bootstrapping Puppet Don’t forgetto clean up! • Revoke Puppet-CA certificates for decommissioned nodes. • Clean up Salt keys. • Remove old reports, exported resources and catalogs from PuppetDB. • Clean connected resources like Load Balancers. 40 Sloppiness will catch up with you when it hurts the most.
- 41.
- 42. Demo Terraform & Puppet THISDEMONSTRATION INCLUDES: VIEW & TRY THE FULL SOURCE OF THIS DEMO https://github.com/MaxServ/Terraform-Puppet-Demo A B C Docker container running a Puppetmaster. AWS Stack with 4 webservers using Terraform. Webserver provisioning using Puppet based on type. 42
- 43.
- 44.
- 45. 0416 - 3010 00 Remco Overdijk LEAD OPERATIONS ENGINEER info@maxserv.com Questions? @MaxServ @RemzJay