WELCOME
Puppet and AWS
Getting the Best of BothWorlds
Mike Ryan - Epitech BV
23
August
2013
CONTACT:
mike@epitech.nl
www.e...
EPITECH BV
Hello, I’m Mike
2
CONTACT:
mike@epitech.nl
www.epitech.nl
Sysadmin	
  with	
  a	
  passion	
  for	
  automa2on
...
EPITECH BV
AWS - A very brief
introduction
3
CONTACT:
mike@epitech.nl
www.epitech.nl
EC2	
  -­‐	
  Elas2c	
  Compute	
  Cl...
EPITECH BV
EC2 - Auto Scaling
4
CONTACT:
mike@epitech.nl
www.epitech.nl
Saturday, August 24, 13
EPITECH BV
CloudFormation
5
CONTACT:
mike@epitech.nl
www.epitech.nl
{
"AWSTemplateFormatVersion" : "2010-09-09",
"Descript...
EPITECH BV
6
CONTACT:
mike@epitech.nl
www.epitech.nl
}
}
}
},
"asgMyAutoScalingGroup": {
"Type": "AWS::AutoScaling::AutoSc...
EPITECH BV
7
CONTACT:
mike@epitech.nl
www.epitech.nl
Bootstrapping an
Infrastructure
www.infrastructures.org
Saturday, Aug...
EPITECH BV
8
CONTACT:
mike@epitech.nl
www.epitech.nl
CloudFormation or Puppet
Saturday, August 24, 13
EPITECH BV
9
CONTACT:
mike@epitech.nl
www.epitech.nl
Files, Services and Packages
with cfn-init
"Metadata": {
"AWS::CloudF...
EPITECH BV
10
CONTACT:
mike@epitech.nl
www.epitech.nl
Run a script at launch time
with User Data
{ "AWSTemplateFormatVersi...
EPITECH BV
11
CONTACT:
mike@epitech.nl
www.epitech.nl
cfn-create-stack
cfn-create-stack puppetmaster 
--template-file=pupp...
EPITECH BV
12
CONTACT:
mike@epitech.nl
www.epitech.nl
Auto Scale = Autosign
Saturday, August 24, 13
EPITECH BV
13
CONTACT:
mike@epitech.nl
www.epitech.nl
The Hostname Issue
mike@ip-10-32-34-116:~$ hostname -f
ip-10-32-34-1...
EPITECH BV
14
CONTACT:
mike@epitech.nl
www.epitech.nl
User Data - Web Console
Saturday, August 24, 13
EPITECH BV
15
CONTACT:
mike@epitech.nl
www.epitech.nl
User Data - CloudFormation},
"UserData": {
"Fn::Base64": {
"Fn::Join...
EPITECH BV
16
CONTACT:
mike@epitech.nl
www.epitech.nl
include stdlib
node default {
  $userdata = parsejson($ec2_userdata)...
EPITECH BV
17
CONTACT:
mike@epitech.nl
www.epitech.nl
Vagrant -> EC2
Saturday, August 24, 13
EPITECH BV
18
CONTACT:
mike@epitech.nl
www.epitech.nl
Saturday, August 24, 13
EPITECH BV
19
CONTACT:
mike@epitech.nl
www.epitech.nl
Vagrant::Config.run do |config|
config.vm.provision :puppet do |pupp...
EPITECH BV
20
CONTACT:
mike@epitech.nl
www.epitech.nl
Decoupling
Saturday, August 24, 13
EPITECH BV
21
CONTACT:
mike@epitech.nl
www.epitech.nl
Broken Puppet Master =
Broken Auto Scaling
No
Saturday, August 24, 13
EPITECH BV
22
CONTACT:
mike@epitech.nl
www.epitech.nl
#!/bin/bash
/usr/local/bin/update.sh
/usr/bin/puppet apply site.pp
/...
EPITECH BV
23
CONTACT:
mike@epitech.nl
www.epitech.nl
But you lose...
Puppet	
  Dashboard
Exported	
  Resources
Stored	
  ...
EPITECH BV
24
CONTACT:
mike@epitech.nl
www.epitech.nl
env.roledefs = {
'web': ['www1', 'www2', 'www3'],
'db': ['db1', 'db1...
EPITECH BV
25
CONTACT:
mike@epitech.nl
www.epitech.nl
def configure_roles():
 tags = EC2TagManager(AWS_KEY, AWS_SECRET,
  ...
THANK YOU
BACK X
CONTACT:
Mike Ryan - Epitech BV
mike@epitech.nl
mike@epitech.nl
www.epitech.nl
Saturday, August 24, 13
Upcoming SlideShare
Loading in …5
×

Puppet and AWS: Getting the Best of Both Worlds

9,235 views

Published on

This talk will be a modified and updated version of the talk given at Puppet Camp Amsterdam. I will discuss some technical examples of how to use Puppet to manage large infrastructures in Amazon's cloud, as well as giving some background in to how Puppet fits in to the AWS ecosystem. Further topics include: - OpsWorks (Amazon's Chef-based configuration management offering) and what it means for Puppet - using Puppet in conjunction with CloudFormation - using Puppet to automate common AWS tasks, such as building AMIs - using Vagrant and Puppet to create an easy path from local development to production - common pitfalls and workarounds

Mike Ryan
Cloud Infrastructure Consultant, Epitech BV
Mike Ryan is the founder of Epitech BV, a cloud technology consultancy based in Amsterdam, and author of the upcoming O'Reilly book AWS System Administration. He is also the founder of Cloudfrag, an internet gaming startup, and is focussed on using efficient engineering practices to help companies accelerate their growth.

Published in: Spiritual, Technology

Puppet and AWS: Getting the Best of Both Worlds

  1. 1. WELCOME Puppet and AWS Getting the Best of BothWorlds Mike Ryan - Epitech BV 23 August 2013 CONTACT: mike@epitech.nl www.epitech.nl Saturday, August 24, 13
  2. 2. EPITECH BV Hello, I’m Mike 2 CONTACT: mike@epitech.nl www.epitech.nl Sysadmin  with  a  passion  for  automa2on Londoner  in  Amsterdam Epitech.nl  -­‐  sysadmin  as  a  service Saturday, August 24, 13
  3. 3. EPITECH BV AWS - A very brief introduction 3 CONTACT: mike@epitech.nl www.epitech.nl EC2  -­‐  Elas2c  Compute  Cloud AMI  -­‐  Amazon  Machine  Images User  Data Saturday, August 24, 13
  4. 4. EPITECH BV EC2 - Auto Scaling 4 CONTACT: mike@epitech.nl www.epitech.nl Saturday, August 24, 13
  5. 5. EPITECH BV CloudFormation 5 CONTACT: mike@epitech.nl www.epitech.nl { "AWSTemplateFormatVersion" : "2010-09-09", "Description" : "EC2 instance", "Resources" : { "MyEC2Instance" : { "Type" : "AWS::EC2::Instance", "Properties" : { "ImageId" : "ami-79fd7eee", "KeyName" : "my-ssh-key", } } } } Saturday, August 24, 13
  6. 6. EPITECH BV 6 CONTACT: mike@epitech.nl www.epitech.nl } } } }, "asgMyAutoScalingGroup": { "Type": "AWS::AutoScaling::AutoScalingGroup", "Properties": { "AvailabilityZones": [ "us-east-1b", "us-east-1c" ], "Cooldown": "300", "DesiredCapacity": "1", "MaxSize": "1", "MinSize": "1", "LaunchConfigurationName": { "Ref": "lcMyLC" }, "LoadBalancerNames": [ { "Ref": "elbMyLB" } ] } }, "s3webapplication": { "Type": "AWS::S3::Bucket" }, "sgwebappsecuritygroup": { "Type": "AWS::EC2::SecurityGroup", "Properties": { "GroupDescription": "for web app", "SecurityGroupIngress": [ Saturday, August 24, 13
  7. 7. EPITECH BV 7 CONTACT: mike@epitech.nl www.epitech.nl Bootstrapping an Infrastructure www.infrastructures.org Saturday, August 24, 13
  8. 8. EPITECH BV 8 CONTACT: mike@epitech.nl www.epitech.nl CloudFormation or Puppet Saturday, August 24, 13
  9. 9. EPITECH BV 9 CONTACT: mike@epitech.nl www.epitech.nl Files, Services and Packages with cfn-init "Metadata": { "AWS::CloudFormation::Init": { "config": { "sources" : { "/etc/puppet" : "http://example.com/puppet.tar.gz" }, "packages": { "yum": { "puppet-server": [], }, }, "services": { "sysvinit": { "puppetmaster": { "ensureRunning": "true", "enabled": "true" Saturday, August 24, 13
  10. 10. EPITECH BV 10 CONTACT: mike@epitech.nl www.epitech.nl Run a script at launch time with User Data { "AWSTemplateFormatVersion" : "2010-09-09", "Description" : "EC2 instance", "Resources" : { "PuppetMasterInstance": { "Type": "AWS::EC2::Instance", "Metadata": { }, "Properties": { "UserData": { "Fn::Base64": { "Fn::Join": [ "", [ "#!/bin/bashn", "/opt/aws/bin/cfn-init --region ", "n", "/usr/bin/puppet apply site.pp, "n" ] ... Saturday, August 24, 13
  11. 11. EPITECH BV 11 CONTACT: mike@epitech.nl www.epitech.nl cfn-create-stack cfn-create-stack puppetmaster --template-file=puppetmaster.json http://aws.amazon.com/developertools/ 2555753788650372 http://aws.amazon.com/cli/ Saturday, August 24, 13
  12. 12. EPITECH BV 12 CONTACT: mike@epitech.nl www.epitech.nl Auto Scale = Autosign Saturday, August 24, 13
  13. 13. EPITECH BV 13 CONTACT: mike@epitech.nl www.epitech.nl The Hostname Issue mike@ip-10-32-34-116:~$ hostname -f ip-10-32-34-116.eu-west-1.compute.internal Saturday, August 24, 13
  14. 14. EPITECH BV 14 CONTACT: mike@epitech.nl www.epitech.nl User Data - Web Console Saturday, August 24, 13
  15. 15. EPITECH BV 15 CONTACT: mike@epitech.nl www.epitech.nl User Data - CloudFormation}, "UserData": { "Fn::Base64": { "Fn::Join": [ "", [ “{”role”: ”web”, ”, “”env”: ”staging”} ” ] ] } } Saturday, August 24, 13
  16. 16. EPITECH BV 16 CONTACT: mike@epitech.nl www.epitech.nl include stdlib node default {   $userdata = parsejson($ec2_userdata) $role = userdata[‘role’] $environment = userdata[‘env’] case $role { ‘web’: { include nginx } ‘db’: { include postgresql } } } Saturday, August 24, 13
  17. 17. EPITECH BV 17 CONTACT: mike@epitech.nl www.epitech.nl Vagrant -> EC2 Saturday, August 24, 13
  18. 18. EPITECH BV 18 CONTACT: mike@epitech.nl www.epitech.nl Saturday, August 24, 13
  19. 19. EPITECH BV 19 CONTACT: mike@epitech.nl www.epitech.nl Vagrant::Config.run do |config| config.vm.provision :puppet do |puppet| puppet.manifests_path = "../puppet/manifests" puppet.module_path = "../puppet/modules" puppet.manifest_file = "site.pp" puppet.options = "--verbose --debug" puppet.facter = { :ec2_userdata => { :role => "database", :env => "vagrant", }.to_json, :vagrant => "true" } end end Saturday, August 24, 13
  20. 20. EPITECH BV 20 CONTACT: mike@epitech.nl www.epitech.nl Decoupling Saturday, August 24, 13
  21. 21. EPITECH BV 21 CONTACT: mike@epitech.nl www.epitech.nl Broken Puppet Master = Broken Auto Scaling No Saturday, August 24, 13
  22. 22. EPITECH BV 22 CONTACT: mike@epitech.nl www.epitech.nl #!/bin/bash /usr/local/bin/update.sh /usr/bin/puppet apply site.pp /etc/rc.local Saturday, August 24, 13
  23. 23. EPITECH BV 23 CONTACT: mike@epitech.nl www.epitech.nl But you lose... Puppet  Dashboard Exported  Resources Stored  configs Saturday, August 24, 13
  24. 24. EPITECH BV 24 CONTACT: mike@epitech.nl www.epitech.nl env.roledefs = { 'web': ['www1', 'www2', 'www3'], 'db': ['db1', 'db1'] } @roles('db', ‘web’) def run_puppet(): sudo('puppet apply site.pp') $fab run_puppet #run puppet everywhere $fab run_puppet --roles db # role-specific Triggering Puppet with Fabric Saturday, August 24, 13
  25. 25. EPITECH BV 25 CONTACT: mike@epitech.nl www.epitech.nl def configure_roles():  tags = EC2TagManager(AWS_KEY, AWS_SECRET,    regions=['eu-west-1'])  roles = {} for role in [‘db’, ‘web’]: roles[role] = tags.get_instances(role=role) return roles Triggering Puppet with Fabric https://github.com/mikery/fabric-ec2 Saturday, August 24, 13
  26. 26. THANK YOU BACK X CONTACT: Mike Ryan - Epitech BV mike@epitech.nl mike@epitech.nl www.epitech.nl Saturday, August 24, 13

×