Immutable servers provided by Vagrant/Chef/AWS


Published on

Madrid DevOps Meeting in September 2013 about how to create a whole environment of immutable servers in AWS with Vagrant and Chef Server. More information about this topic available in

Published in: Technology, Self Improvement
1 Comment
No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Immutable servers provided by Vagrant/Chef/AWS

  1. 1. DevOps Toolchain: Vagrant/Chef/AWS
  2. 2. Web Twitter LinkedIn GitHub Slideshare About me
  3. 3. Index 1. Requisites 2. Vagrant Box 3. Vagrantfile 4. Vagrant Settings 5. Vagrant Commands 6. Provisioning with Chef 7. Multi machines 8. Version control 9. Cleanup 10.Conclusions 11.References
  4. 4. Why a common toolchain? • Consistency • Efficiency • Safety • Automation
  5. 5. Requisites Knife Vagrant $ gem install vagrant (ruby needed) Vagrant plugins $ vagrant plugin install vagrant-aws $ vagrant plugin install vagrant-omnibus $ vagrant plugin install vagrant-butcher AWS credentials Chef server running and chef client certificates
  6. 6. Quick view Chef Server AMI base Dev/Ops workstation (Vagrant+knife) VPC Subnet App instances VPC Subnet Virtual Private Cloud AWS Cloud 1. Create instances from AMI base and make chef bootstrap 2. Apply roles
  7. 7. Vagrant Box Base Box: The base box is simply a saved hard-disk of a Virtual Machine created with VirtualBox. It can contain anything but it needs at least : • Ruby • VirtualBox guest additions • Puppet or Chef Vagrant AWS Example Box Vagrant providers each require a custom provider-specific box format. These files compose the contents of a box for the AWS provider. • • Vagrantfile • metadata.json To turn this into a box: $ tar cvzf ./metadata.json ./Vagrantfile
  8. 8. Vagrantfile EC2 Vagrant.configure("2") do |config| # Version 2 Vagrant syntax = "dummy" config.omnibus.chef_version = :latest config.vm.provider :aws do |aws, override| aws.keypair_name = "bq” aws.access_key_id = ”xxxxxxxxx" aws.secret_access_key = ”xxxxxxxxxxxxxxxxxx" aws.ami = "ami-3ad1af53” override.ssh.username = "ec2-user” override.ssh.private_key_path = "/Users/juanvi/keypairs/bq.pem" end end
  9. 9. Vagrant Settings Creating Vagrant File mkdir [project name] cd [project name] Vagrant init -> creates Vagrantfile Vagrant ec2 test 2 ec2 without chef management $ vagrant plugin install vagrant-aws $ vagrant box add dummy aws/raw/master/ $ vagrant up --provider=aws This box works by using Vagrant's built-in Vagrantfile merging to setup defaults for AWS. These defaults can easily be overwritten by higher- level Vagrantfiles (such as project root Vagrantfiles).
  10. 10. Vagrant Commands vagrant box # Commands to manage system boxes vagrant destroy # Destroy the environment, deleting the created virtual machines vagrant halt # Halt the running VMs in the environment vagrant help [TASK] # Describe available tasks or one specific task vagrant init [box_name] [box_url] # Initializes the current folder for Vagrant usage vagrant package # Package a Vagrant environment for distribution vagrant provision # Rerun the provisioning scripts on a running VM vagrant reload # Reload the environment, halting it then restarting it. vagrant resume # Resume a suspended Vagrant environment. vagrant ssh # SSH into the currently running Vagrant environment. vagrant ssh_config # outputs .ssh/config valid syntax for connecting to this environment via ssh vagrant status # Shows the status of the current Vagrant environment. vagrant suspend # Suspend a running Vagrant environment. vagrant up # Creates the Vagrant environment vagrant version # Prints the Vagrant version information
  11. 11. Provisioning with Chef web.vm.provision :chef_client do |chef| chef.chef_server_url = "" chef.validation_key_path = "/Users/juanvi/chef- repo/.chef/juanvi-validator-new.pem" chef.validation_client_name = "juanvi-validator" # Provision with the database role chef.add_role("webserver") # Set the environment for the chef server chef.environment = ”prod"
  12. 12. Multi Machines Vagrant.configure("2") do |config| config.vm.define :web do |web| = "apache" end web.vm.provision :chef_client do |chef| chef.chef_server_url = chef_server_url chef.validation_key_path = validation_key_path chef.validation_client_name = validation_client_name ……. end config.vm.define :db do |db| = "mysql" end db.vm.provision :chef_client do |chef| chef.chef_server_url = chef_server_url chef.validation_key_path = validation_key_path chef.validation_client_name = validation_client_name … end end
  13. 13. Provisioning with Chef Provisioning only one kind of server executing chef_client $ vagrant provision web --provision-with chef_client Provisioning the whole platform executing chef_client $ vagrant provision --provision-with chef_client After editing the Vagrantfile you need to 'reboot' the machine to take this settings $ vagrant reload
  14. 14. Version control with git Version control Now is a good time to version control our awesome- vagrant project $ cd $ git init $ git add Vagrantfile $ git commit -m "This was just my first commit"
  15. 15. Cleanup $ vagrant plugin install vagrant-butcher then you can terminate a instance and deregister in the Chef server executing: $ vagrant destroy -f [Butcher] knife.rb location set to '/path/to/knife.rb' [Butcher] Chef node 'node_name' successfully butchered from the server... [Butcher] Chef client 'node_name' successfully butchered from the server... [default] Forcing shutdown of VM... [default] Destroying VM and associated drives...
  16. 16. • The task of writing recipes is not picked up by all team members, and seem to stay the main job of the system oriented people on the team. • Reading Chef Cookbooks help people understand what is needed and makes it easy to point out what needs to be changed Conclusions
  17. 17. Conclusions • Before committing cookbooks-> destroy 'development' box and re-provision a new box to see if this works. (immutable servers) • The longer the provision takes, the less frequent people do it. It's important to keep that process as fast as possible.
  18. 18. Conclusions • Installation problems would get noticed far sooner in the process. • People would only do a full rebuild in the morning when getting their coffee.
  19. 19. Conclusions • It clearly helps everybody to have a consistent environment to develop against, the latest version is just one git pull away. • The central approach drives people to • do frequent commits • do stable commits.
  20. 20. Conclusions • Vagrant is an essential part of the DevOps process: it is the solution to developing, testing and deploying in the same environment. It thus ensures a smoother transition of your project from the dev team to the ops team. • Chef has to be the only way to manage the configuration; NO MORE MANUAL CHANGES
  21. 21. Hands On A typical situation in a company that makes software is the arrival of a new developer that has the tough task to install a lot of tools in his workstation that in the best case takes days. We are going to create a whole developer stack in AWS not for this new developer but for a complete new developer team! After make magic with Vagrant and Chef the unique task for all of the members of the team to begin to work with the environment is clone the git repository, receive the chef and EC2 certificates/keypairs from their project leader.
  22. 22. Hands On 1. Download project (private repository) 2. Configure ~/.chef/aws_credentials.rb with your AWS security credentials 3. Check that you have in ~/.chef the chef and aws credentials and keypairs. 4. Well you are ready to create the whole demo environment! 5. $ gem install vagrant 6. $ vagrant plugin install vagrant-aws 7. $ vagrant plugin install vagrant-omnibus 8. $ environment=prod vagrant up --provider=aws 9. Tweak in the /etc/hosts url pointing to elastic ip assigned to ws.(only for test, in prod env change dns entry or proxywebserver virtualhost) 10. Check the url 11. Congrats! IT’S MAGIC! You have got your first complete environment created with Vagrant and Chef! Notes: if you don’t want to import a initial db dump… 1. This step can be avoid changing node attribute db_load to 0 in environments/prod.json.
  23. 23. Create a new demo environment called demo 1. $ cp environments/prod.json environments/demo.json 2. Change id env and ips 3. $ knife environment from file environments/demo.json 4. Change values for all of the attributes with the appropiate demo values 5. $ vi Vagrantfile -> add the environment in the array of the allowed environments 6. if (environment_id =~ /^(prod|qa|dev|demo)$/) Add demo and app db password in encrypted databags for the new environment 7. $ knife data bag edit --secret-file ~/.chef/encrypted_data_bag_secret secrets php_app 8. $ knife data bag edit --secret-file ~/.chef/encrypted_data_bag_secret secrets postgresql 9. $ environment=demo vagrant up --provider=aws 10.Tweak in the /etc/hosts url pointing to elastic ip assigned to ws.(only for test, in prod env change dns entry or proxywebserver virtualhost) 11. Check the url Hands On
  24. 24. Video with the execution of the hands on available at: For more information about this hands on check servers-with-vagrant-and-chef-chapter-i/ Hands On
  25. 25. References • Video of this presentation recorded in Madrid DevOps’ September meeting • • • Vagrant: Up and Running, Mitchell Hashimoto Hashimoto/dp/1449335837/ref=sr_1_1?ie=UTF8&qid=1375433466&sr=8- 1&keywords=vagrant+up • and-chef-chapter-i/ • • • vagrant-chef-from-development-to-the-cloud/ •
  26. 26. irc channel #madrid-devops. Web Twitter Grupo de Google GitHub Group information