This document discusses using Chef 0.8 to build a baseline Amazon EC2 AMI with the Chef client installed. It describes cloning Chef code from Git and building related gems. It then covers setting up a generic client.rb file, instantiating an EC2 instance with the new AMI, and installing the Chef server and related gems. Finally, it discusses the development process of creating roles and cookbooks locally, uploading to the Chef server, and debugging clients running on EC2.

1. Chef 0.8, Knife & EC2
The Bleeding Edge

2. Building Chef from Git
BTM’s GIST: chef 0.8 alpha installation
Git Clone & Build Gems ohai, mixlib-log,
mixlib-authentication, chef, chef-server,
chef-server-api, chef-server-webui, chef-
solar

3. Make a Baseline EC2
AMI with Chef Client
Set up Apt for multiverse, update, upgrade
Install fundamental packages and gems
emacs, ruby, rubygems, ec2_tools, merb...
Install Chef Client related gems you built
Set up /etc/chef/client.rb
Clean up and run Eric Hammond’s ec2 build
AMI ends up in S3 and registered as an AMI

4. Generic client.rb
require 'ohai'
unless File.exists?("/etc/chef/client.pem")
require 'json'
File.open("/etc/chef/validation.pem", "w") do |f|
f.print(chef_config["validation_key"])
o = Ohai::System.new
end
o.all_plugins
end
chef_config = JSON.parse(o[:ec2][:userdata])
if chef_config.kind_of?(Array)
if chef_config.has_key?("attributes")
chef_config = chef_config[o[:ec2][:ami_launch_index]]
File.open("/etc/chef/client-config.json", "w") do |f|
end
f.print(JSON.pretty_generate(chef_config["attributes"]))
end
log_level :info
json_attribs "/etc/chef/client-config.json"
log_location "/var/log/chef/client.rb"
end
chef_server_url chef_config["chef_server"]
registration_url chef_config["chef_server"]
validation_key "/etc/chef/validation.pem"
openid_url chef_config["chef_server"]
validation_client_name
template_url chef_config["chef_server"]
chef_config["validation_client_name"]
remotefile_url chef_config["chef_server"]
search_url chef_config["chef_server"]
Mixlib::Log::Formatter.show_time = true
role_url chef_config["chef_server"]
client_url chef_config["chef_server"]
node_name o[:ec2][:instance_id]

5. Instantiate a
Chef Server
Instantiate the new chef-client AMI
Install Chef Server Gems (chef-server, chef-server-
api, chef-server-webui, chef-solar)
Use Chef Solo and danielsdeleo (Dan
DeLeo)’s bootstrap cookbook to setup Server
Configures various /etc/chef files & certs
Sets up runit to run the various servers
Install couchdb

6. Startup the WebUI
Useful mainly for reality checking
Currently does not automatically set up the
WebUI
Have to manually start it on a port
Does not yet have an Apache/Passenger
frontend
sudo sh -c '/usr/bin/chef-server-webui -p 4002 > /
var/log/chef-server-webui.log' &

7. The Dev Process
Create/edit roles / cookbooks on dev machine
Use knife to upload to chef-server
Use knife to create user-data to pass to EC2
instantiate process to create customized
clients from your base chef client ami
Launch Instance
Debug

8. Hierarchy of Roles
We use a runa_base as a top level global
default configs
Then an environment role (staging, production,
etc) that “set”s the environment based overrides
Then a Function Role, like “dashboard” or
“runtime_db”
Still tend to need a recipe equivalent to allow
for aggregate attributes

9. Base Role
name "runa_base" Uses Json style
description "top level attributes"
recipes "users::env", "users::homes", "sudo", "emacs",
"git", "runa_base"
recipe
default_attributes(
"runa" => {
"home_base_dir" => "/home",
"mnt_point" => "/mnt",
"bin_dir" => "/usr/local/bin",
"upstart_event_dir" => "/etc/init",
"summarizer" => {"namespaces" => ['lotus','tesla']},
"target_user" => 'root',
"target_group"=> 'root',
"aws_access_key" => "secret",
"aws_secret_access_key" => "secret",
"availability_zone" => "us-west-1a"
}
)

10. Staging Role
Find that I end up
using the recipe
attributes instead of
name "staging"
these since you can’t description "staging environment"
refer to attributes in
recipes "staging"
other roles here
default_attributes({})
override_attributes ({})

11. Staging Recipes
Here we can access
the runa hash from
the runa_base role
set[:runa_dashboard][:environment] = "staging"
set[:runa_dashboard][:aws_access_key] = runa[:aws_access_key]
set[:runa_dashboard][:aws_secret_access_key] =
runa[:aws_secret_access_key]
set[:runa_dashboard][:availability_zone] = "us-west-1a"

12. Specific Function Role
Mainly the recipes
needed
name "runa_dashboard"
description "Use this role to make the node a runa_dashboard node"
recipes "runa_dashboard", "xfs", "aws", "mysql",
"runa_dashboard::ebs",
"mysql::server","runa_dashboard::configure_mysql",
"passenger_apache2", "runa_dashboard::setup",
"runa_dashboard::deploy"

13. Setting up Knife
Configure ~/.chef for personal config
~/.chef/knife.rb can be set up with editor or
a knife command
log, connection info, path to cookbooks,
Validation info for clients started by you
Your credentials (Can use the chef-webui
to start)

14. Knife Config File
log_level :info
log_location STDOUT
node_name 'rberger'
client_key '/Users/rberger/.chef/rberger.pem'
chef_server_url 'http://chef-server-staging.runa.com:4000/'
cache_type 'BasicFile'
cache_options( :path => '/Users/rberger/.chef/checksums' )
cookbook_path [ '/Users/rberger/work/runa/runa_repo/
cookbooks', '/Users/rberger/work/runa/runa_repo/site-
cookbooks' ]
validation_client_name "validator"

15. Upload Roles and
Cookbooks with Knife
Upload Roles
knife role from file ~/my_repo/roles/runa_base.rb
Upload Cookbooks all
knife cookbook upload -a
Upload Individual Cookbook
knife cookbook upload runa_dashboard

16. Validation Creds
Uploaded as part of user-data of starting ec2
instance
Used to authenticate the original connection between
a new chef client & chef-server
After initial validation specific client creds are
downloaded from chef-server
Having problems with this right now
Theoretically copy chef-server:/etc/chef/
validation.pem to your dev /etc/chef
Set the validator client name in ~/.chef/knife.rb

17. Starting a Specific
Client on EC2
Use Knife to generate user-data
knife ec2 instance data “role[runa_base”
“role[staging]” “ role[runa_dashboard]”
{
"attributes": {
"run_list": [
"role[runa_base]",
"role[staging]",
"role[runa_dashboard]"
]
},
"validation_key": "-----BEGIN RSA PRIVATE KEY-----
nMIIEowIBAAKCAQEAu9fKFLJz+JPSw5kWiRA6ywV1/omB73Juw9UyS/...
-----END CERTIFICATE-----n",
"validation_client_name": "validator",
"chef_server": "http://chef-server-staging.runa.com:4000/"
}

18. Start the EC2 instance
with user-data
Can use command line ec2 tools or elastic
fox to start an instance of your chef client
AMI
Pass in the user data
Between Ohai and user-data is enough to
bootstrap the connection to chef-server
Gets Roles & Cookbooks and converges

19. Debugging
ssh to client
ps and syslog to see if basic chef-client start
Chef-client log for chef debug (like Ruby)
Run chef-client in standalone in debug mode
Can also see whats up on the chef-server in /etc/sv/
chef-server/log/main/current
Ask questions on #chef-hacking
Fix bugs back in dev, knife upload, run chef-client, rince,
repeat