Building a Cauldron for Chef to Cook In


Published on

Regardless of whether you're using chef or any other automated devops tool, you still need to consider where you are going to host things. Redundancy is good, so in this talk I will describe the tools I used as well as how and why I set up my own chef+git server to provide my own cauldron in which to cook up server deployments.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Building a Cauldron for Chef to Cook In

  1. 1. Building a Cauldron for Chef to Cook In Jonathan Altman @async_io For DevOpsDC August 2013 Wednesday, August 14, 2013
  2. 2. Background • 12 years architecting at a leading SaaS software provider • Started 2 companies in August 2012 • One is a SaaS/product based company • One is a services firm • Customers have access to their source • No time to waste: automation good Wednesday, August 14, 2013
  3. 3. What is a Cauldron? • A complete (as you want) code/artifact management and build/deployment system • Chef server • Git server • Automated build system e.g. Jenkins (eventually) • JIRA server? • Group chat server? (IRC or XMPP) • With a real cert Wednesday, August 14, 2013
  4. 4. But SaaS is Awesome! • Yes. Yes it is. • Github • Hosted Chef • ShiningPanda, CloudBees, JenkinsHosting, • Jira OnDemand • HipChat (or for hosted IRC, or Google+ hangouts) Wednesday, August 14, 2013
  5. 5. The 3 R’s • Redundancy • Resiliency • Revelation Wednesday, August 14, 2013
  6. 6. Redundancy • Does a good backup strategy ever keep just one copy of something? • Disaster Recovery: how do you recover with your backups? • At least for git, chef, and CI you can use multiple servers • git “just works” in this model • The knife plugin has backup/restore capability for example • Not sure how to integrate with Opscode-hosted chef Wednesday, August 14, 2013
  7. 7. Resiliency • Tradeoff between your ability to deliver uptime and their motivation to address their issues that affect your uptime • Honeypot: tradeoff again. github and hosted chef server are well protected, but a big, attractive attack vector. Your server, not so much; but is it secure? • Ability to assign the amount of compute resources you need to deliver the performance, uptime, and redundancy you want Wednesday, August 14, 2013
  8. 8. Revelation • You will have the tools and ability to help yourself • Git != Github, for example Wednesday, August 14, 2013
  9. 9. A Cauldron: workstation • Client workstation with • omnibus installer ( • knife-server plugin ( • git client software installed on it • You could do this on your workstation’s host OS, but using aVM simplifies/isolates having multiple cauldrons Wednesday, August 14, 2013
  10. 10. Create Cauldron Workstation Cocytus:vagrant_servers jonathan$ mkdir cauldron_ws Cocytus:vagrant_servers jonathan$ cd cauldron_ws Cocytus:cauldron_ws jonathan$ vagrant init A `Vagrantfile` has been placed in this directory. You are now ready to `vagrant up` your first virtual environment! Please read the comments in the Vagrantfile as well as documentation on `` for more information on using Vagrant. Cocytus:cauldron_ws jonathan$ Wednesday, August 14, 2013
  11. 11. Bootstrap theVagrant Cocytus:cauldron_ws jonathan$ vi Vagrantfile # Set up your config here Cocytus:cauldron_ws jonathan$ vagrant up [default] Box base was not found. Fetching box from specified URL... [vagrant] Downloading with Vagrant::Downloaders::HTTP... [vagrant] Extracting box... [vagrant] Verifying box... [vagrant] Cleaning up downloaded box... [default] Importing base box 'base'... {bunch of lines deleted ...} [default] Booting VM... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! VM must be created before running this command. Run `vagrant up` first. Cocytus:cauldron_ws jonathan$ Wednesday, August 14, 2013
  12. 12. Install chef omnibus Cocytus:cauldron_ws jonathan$ vagrant ssh Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-23-generic x86_64) * Documentation: Welcome to your Vagrant-built virtual machine. Last login: Fri Sep 14 06:23:18 2012 from vagrant@precise64:~$ sudo apt-get install build-essential curl [...bunch of output deleted] vagrant@precise64:~$ curl -L | sudo bash % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 6790 100 6790 0 0 22442 0 --:--:-- --:--:-- --:--:-- 36902 Downloading Chef for ubuntu... Installing Chef Selecting previously unselected package chef. (Reading database ... 51127 files and directories currently installed.) Unpacking chef (from .../tmp.8PmNsIUQ/chef__amd64.deb) ... Setting up chef (11.6.0-1.ubuntu.12.04) ... Thank you for installing Chef! vagrant@precise64:~$ Wednesday, August 14, 2013
  13. 13. Knife-server plugin • knife plugin to create chef servers on the command line running on ubuntu: • on ec2 (requires the knife-ec2 plugin as well) • on linode (requires the knife-linode plugin) • on an already-running instance (“standalone”) Wednesday, August 14, 2013
  14. 14. Install knife-server Plugin vagrant@precise64:~$ sudo /opt/chef/embedded/bin/gem install knife-server --no-ri --no-rdoc Building native extensions. This could take a while... Fetching: ruby-hmac-0.4.0.gem (100%) Fetching: fog-1.14.0.gem (100%) Fetching: knife-server-1.1.0.gem (100%) Successfully installed nokogiri-1.6.0 Successfully installed ruby-hmac-0.4.0 Successfully installed fog-1.14.0 Successfully installed knife-server-1.1.0 4 gems installed vagrant@precise64:~$ Wednesday, August 14, 2013
  15. 15. Make a Server (Standalone) • Standalone there is a bunch of other stuff to get right: • Better if the server you’re installing on has root with ssh authorized_keys set up to your workstation’s ssh private key • Need both private and public part of key on workstation because of Wednesday, August 14, 2013
  16. 16. Server Bootstrapping... knife server bootstrap standalone --node-name cauldron.your.domain --host [hundreds of lines of output deleted ...] Recipe: chef-server::erchef * service[erchef] action restart - restart service service[erchef] Chef Client finished, 244 resources updated chef-server Reconfigured! Server reconfigured -----> Bootstrapping Chef Server on is complete. If you want the web UI enabled, include --web-ui-enable Wednesday, August 14, 2013
  17. 17. Voila? Wednesday, August 14, 2013
  18. 18. uhh...Voila! Wednesday, August 14, 2013
  19. 19. IfYou Used --web-ui-enable BTW, login right now and change the password! Or: turn off web-ui Wednesday, August 14, 2013
  20. 20. git server • Several (sane) choices for hosting your own git: • Gitolite: • Gitlab: • Gitblit: • gitweb: (please don’t) • On windows or OSX, hosted github ($$) • We are going to use gitolite Wednesday, August 14, 2013
  21. 21. Gitolite • Gitlab is cool, tries to reproduce as much of github other web-based git hosting as possible. Installation? Several pages of hand-invoking • Gitblit: same goal as gitlab, but built in java. So: easy install, but big and not using the official git binaries so compatibility? • Gitosis is dead if you run across it • Gitweb: just don’t Wednesday, August 14, 2013
  22. 22. Install • Put the key of the user you want to admin gitolite as on the cauldron server, but name it where username is the username you want to be known as on the gitolite server # get the software git clone git:// # install it gitolite/install -ln # setup the initial repos with your key gitolite setup -pk Wednesday, August 14, 2013
  23. 23. Configure gitolite • Back on your workstation machine: git clone git@host:gitolite-admin.git • Add your git repository setups in the cloned conf/gitolite.conf file • Example, jonathan is the owner of the heatNode repository (and there is a RSA public key): repo heatNode RW+ = jonathan Wednesday, August 14, 2013
  24. 24. You have a cauldron • You can git add remote your cauldron plus any other git servers such as github to any git repository you have git remote add origin git@cauldron.your.domain:name_of_your_repo.git git remote add github • The name after git remote add is arbitrary! “origin” is a convention but not required • You git push/pull from all external servers so you have redundant copies • Your cauldron is now a chef server Wednesday, August 14, 2013
  25. 25. Put a real cert on the box • As of omnibus Chef 11, webserver is nginx. Edit the ssl config to put a real server on to get rid of the big red untrusted cert warning • Check out for steps on how to do it with chef • Probably a good idea, as chef-ctl-reconfigure might blast manual changes Wednesday, August 14, 2013
  26. 26. Thank you. Questions? Also, thanks to @nathenharvey and @devopsdc for letting me present, @devopsdc and @fnichol (Fletcher Nichol) for the awesome real-time interactive improvements to my presentation, and @fnichol for the awesome knife-server plugin! Wednesday, August 14, 2013