SlideShare a Scribd company logo
1 of 101
Download to read offline
Infrastructure Automation with Opscode Chef



                         http://opscode.com
                              @opscode
                               #opschef



Tuesday, June 14, 2011
Who are we?




                 •       Joshua Timberman
                 •       Adam Jacob
                 •       Christopher Brown
                 •       Aaron Peterson
                 •       Seth Chisamore
                 •       Matt Ray



Tuesday, June 14, 2011
Who are you?




                 •       System administrators?
                 •       Developers?
                 •       “Business” People?




                                                    http://www.flickr.com/photos/timyates/2854357446/sizes/l/

Tuesday, June 14, 2011

Hint, consultants, you’re “Business” people too.
What are we talking
                                    about?

                                                   http://www.flickr.com/photos/peterkaminski/2174679908/

Tuesday, June 14, 2011

Managing infrastructure in the Cloud. With Chef, hopefully.
Agenda




                 •       How’s and Why’s
                 •       Live Demo!
                 •       Getting Started with Chef
                 •       Anatomy of a Chef Run
                 •       Managing Cloud Infrastructure
                 •       Data Driven Shareable Cookbooks


                                                                                    http://www.flickr.com/photos/koalazymonkey/3590953001/
Tuesday, June 14, 2011

How’s and why’s of managing infrastructure with Chef.
We’re running a live demo!
We’ll walk through the things required to get started with Chef.
We will look at the anatomy of a Chef run in detail.
Since we’ve launched a cloud infrastructure, we’ll want to know how we manage it.
We’ll talk about our data driven sharable cookbooks.
Infrastructure as Code


Tuesday, June 14, 2011

The goal is fully automated infrastructure. In the cloud, anywhere. We get there with Infrastructure as Code.
A technical domain
                revolving around
                building and
                managing
                infrastructure
                programmatically
Tuesday, June 14, 2011
Enable the reconstruction
                            of the business from
                         nothing but a source code
                         repository, an application
                           data backup, and bare
                              metal resources.
Tuesday, June 14, 2011
Configuration
                                          Management


Tuesday, June 14, 2011

Keep track of all the steps required to take bare metal systems to doing their job in the infrastructure.

It is all about the policy.

And this needs to be available as a service in your infrastructure.
System Integration


                                                     http://www.flickr.com/photos/opalsson/3773629074/

Tuesday, June 14, 2011

Taking all the systems that have been configured to do their job, and make them work together to actually run the infrastructure.
Tuesday, June 14, 2011

Introducing Chef.

Maybe you’ve already met!

Stephen Nelson-Smith has a great way to introducing Chef, so with apologies to him, I’m going to reuse his descriptions.
The Chef Framework


                                         With thanks (and apologies) to Stephen Nelson-Smith
Tuesday, June 14, 2011

Chef provides a framework for fully automating infrastructure, and has some important design principles.
The Chef Framework




                 •       Reasonability
                 •       Flexibility
                 •       Library & Primitives
                 •       TIMTOWTDI




Tuesday, June 14, 2011

Chef makes it easy to reason about your infrastructure, at scale. The declarative Ruby configuration language is easy to read, and
the predictable ordering makes it easy to understand what’s going on.

Chef is flexible, and designed to allow you to build infrastructure using a sane set of libraries and primitives.

Just like Perl doesn’t tell programmers how to program, Chef doesn’t tell sysadmins how to manage infrastructure.
The Chef Tool(s)


                                           With thanks (and apologies) to Stephen Nelson-Smith
Tuesday, June 14, 2011

Since Chef is a framework with libraries and primitives for building and managing infrastructure, it only makes sense that it
comes with tools written for that purpose.
The Chef Tool(s)




                 •       ohai
                 •       chef-client
                 •       knife
                 •       shef




Tuesday, June 14, 2011

Ohai profiles the system to gather data about nodes and emits that data as JSON.
Chef client runs on your nodes to configure them.
Knife is used to access the API.
Shef is an interactive console debugger.
The Chef API


                                          With thanks (and apologies) to Stephen Nelson-Smith
Tuesday, June 14, 2011

The Chef API provides a client/server service for configuration management in your infrastructure.
The Chef API




                 •       RSA key authentication w/ Signed Headers
                 •       RESTful API w/ JSON
                 •       Search Service
                 •       Derivative Services




Tuesday, June 14, 2011

The API itself is RESTful with JSON responses.

Part of the API is a dynamic search service which can be queried to provide rich data about the objects stored on the server.

Because it is flexible and built as a service, it is easy to build derivative services on top, including integration with other tools and
services.
The Chef Community


                                          With thanks (and apologies) to Stephen Nelson-Smith
Tuesday, June 14, 2011

As an Open Source project, the Chef community is critical.
The Chef Community




                 •       Apache License, Version 2.0
                 •       360+ Individual contributors
                 •       70+ Corporate contributors
                         •   Dell, Rackspace,VMware, RightScale,
                             Heroku, and more
                 •       http://community.opscode.com
                 •       240+ cookbooks


Tuesday, June 14, 2011

Community is important.

http://apache.org/licenses/LICENSE-2.0.html
http://www.opscode.com/blog/2009/08/11/why-we-chose-the-apache-license/
http://wiki.opscode.com/display/chef/How+to+Contribute
http://wiki.opscode.com/display/chef/Approved+Contributors
Chef Enables Infrastructure as Code

                                                            package "haproxy" do
                                                              action :install
                                                            end

                                                            template "/etc/haproxy/haproxy.cfg" do
                                                              source "haproxy.cfg.erb"
                 •       Resources                            owner "root"
                                                              group "root"
                 •       Recipes                              mode 0644

                 •
                                                              notifies :restart, "service[haproxy]"
                         Roles                              end

                 •       Source Code                        service "haproxy" do
                                                              supports :restart => true
                                                              action [:enable, :start]
                                                            end



Tuesday, June 14, 2011

Declare system configuration as idempotent resources.
Put resources together in recipes.
Assign recipes to systems through roles.
Track it all like source code.
Chef Resources

                                                           package "haproxy" do
                                                             action :install
                                                           end

                 •       Have a type.                      template "/etc/haproxy/haproxy.cfg" do

                 •
                                                             source "haproxy.cfg.erb"
                         Have a name.                        owner "root"

                 •       Have parameters.                    group "root"
                                                             mode 0644

                 •       Take action to put the resource     notifies :restart, "service[haproxy]"
                                                           end
                         in the declared state.
                 •       Can send notifications to other
                                                           service "haproxy" do
                                                             supports :restart => true
                         resources.                          action [:enable, :start]
                                                           end



Tuesday, June 14, 2011
Resources take action
                            through Providers


Tuesday, June 14, 2011

Providers know how to actually configure the resources to be in the declared state
Chef Providers




                         package “haproxy”
                                                       {        yum install haproxy
                                                                apt-get install haproxy
                                                                pacman sync haproxy
                                                                pkg_add -r haproxy




Tuesday, June 14, 2011

The haproxy package resource may run any number of OS commands, depending on the node’s platform.
Recipes are collections
                              of Resources


Tuesday, June 14, 2011
Chef Recipes

                                                         package "haproxy" do
                                                           action :install
                                                         end

                                                         template "/etc/haproxy/haproxy.cfg" do
                                                           source "haproxy.cfg.erb"
                 •       Recipes are evaluated for         owner "root"
                         resources in the order they       group "root"
                                                           mode 0644
                         appear.                           notifies :restart, "service[haproxy]"

                 •       Each resource object is added   end

                         to the Resource Collection.     service "haproxy" do
                                                           supports :restart => true
                                                           action [:enable, :start]
                                                         end



Tuesday, June 14, 2011
Chef Recipes




                 •       Recipes can include other                   include_recipe
                                                                     include_recipe
                                                                                            "apache2"
                                                                                            "apache2::mod_rewrite"
                         recipes.                                    include_recipe         "apache2::mod_deflate"

                 •       Included recipes are
                                                                     include_recipe
                                                                     include_recipe
                                                                                            "apache2::mod_headers"
                                                                                            "apache2::mod_php5"
                         processed in order.




Tuesday, June 14, 2011

Just like recipes themselves are processed in order, the recipes included are processed in order, so when you include a recipe, all
its resources are added to the resource collection, then Chef continues to the next.
Chef Recipes




                 •       Extend recipes with        %w{ php5 php5-dev php5-cgi }.each do |pkg|
                         Ruby.
                                                      package pkg do
                 •       Iterate over an array of       action :install
                                                      end
                         package names to
                         install.                   end




Tuesday, June 14, 2011
Chef Recipes

                                                     template "/etc/haproxy/haproxy.cfg" do
                                                       source "haproxy.cfg.erb"
                                                       owner "root"
                                                       group "root"
                                                       mode 0644
                                                       notifies :restart, "service[haproxy]"
                                                     end
                 •       Good: Drop off a
                                                 pool_members = search("node", "role:mediawiki")
                         dynamic template.
                 •       Better: Discover data   template "/etc/haproxy/haproxy.cfg" do
                                                   source "haproxy.cfg.erb"
                         through search.           owner "root"
                                                   group "root"
                                                   mode 0644
                                                   variables :pool_members => pool_members
                                                   notifies :restart, "service[haproxy]"
                                                 end

Tuesday, June 14, 2011
Chef Roles

                                                      name "mediawiki"
                                                      description "mediawiki app server"
                                                      run_list(
                                                        "recipe[mysql::client]",
                                                        "recipe[application]",
                                                        "recipe[mediawiki::status]"
                                                      )

                 •       Roles describe nodes.
                                                      name "mediawiki_load_balancer"
                 •       Roles have a run list.       description "mediawiki load balancer"
                                                      run_list(
                 •       Roles can have attributes.
                                                      )
                                                        "recipe[haproxy::app_lb]"

                                                      override_attributes(
                                                        "haproxy" => {
                                                          "app_server_role" => "mediawiki"
                                                        }
                                                      )

Tuesday, June 14, 2011
Track it like source code...

                         % git log
                         commit d640a8c6b370134d7043991894107d806595cc35
                         Author: jtimberman <joshua@opscode.com>

                             Import nagios version 1.0.0

                         commit c40c818498710e78cf73c7f71e722e971fa574e7
                         Author: jtimberman <joshua@opscode.com>

                             installation and usage instruction docs

                         commit 99d0efb024314de17888f6b359c14414fda7bb91
                         Author: jtimberman <joshua@opscode.com>

                             Import haproxy version 1.0.1

                         commit c89d0975ad3f4b152426df219fee0bfb8eafb7e4
                         Author: jtimberman <joshua@opscode.com>

                             add mediawiki cookbook

                         commit 89c0545cc03b9be26f1db246c9ba4ce9d58a6700
                         Author: jtimberman <joshua@opscode.com>

                             multiple environments in data bag for mediawiki

Tuesday, June 14, 2011
LIVE DEMO!!!


                                git clone git://github.com/opscode/velocity2011-chef-repo
Tuesday, June 14, 2011

We thought we’d start with the live demo early on, since last year we were interrupted by a fire alarm.
Live Demo




                 •       Behind the scenes we’re building a
                         new infrastructure
                 •       Five nodes
                         •   Database master
                         •   Two App servers
                         •   Load Balanced
                         •   Monitored

                   git clone git://github.com/opscode/velocity2011-chef-repo
                                                                                             http://www.flickr.com/photos/takomabibelot/3787425422
Tuesday, June 14, 2011

During this workshop, we will build a cloud infrastructure before your very eyes (if we have multiple displays to show that while
the slides are up.)
How did we get here?


                                git clone git://github.com/opscode/velocity2011-chef-repo
Tuesday, June 14, 2011

How did we get to the point where we can build a multi-tiered, monitored infrastructure?
Getting Started




                 •       Opscode Hosted Chef
                 •       Authentication Credentials
                 •       Workstation Installation
                 •       Source Code Repository



                         git clone git://github.com/opscode/velocity2011-chef-repo
Tuesday, June 14, 2011

We signed up for Opscode Hosted Chef, downloaded our authentication credentials (RSA private keys), installed Chef on our
workstation and set up a source code repository.
Getting Started: Opscode Hosted Chef




                 •       Sign up for Opscode Hosted Chef
                         •   https://community.opscode.com/users/new

                 •       Sign into Management Console
                         •   https://manage.opscode.com

                 •       Create an Organization



                         git clone git://github.com/opscode/velocity2011-chef-repo
Tuesday, June 14, 2011

The workshop installation instructions describe how to go about the process.
Getting Started: Authentication
                                                                                          Credentials




                 •       Download User Private Key
                 •       Download Organization Validation Private
                         Key
                 •       Retrieve Cloud Credentials



                         git clone git://github.com/opscode/velocity2011-chef-repo
Tuesday, June 14, 2011

The signup process will provide instructions on how to retrieve your user private key and organization validation private key.

The examples in the chef repository will use Amazon EC2. You’ll need the cloud credentials.
Getting Started: Workstation Installation




                 •       Ruby (1.9.2 recommended)
                 •       RubyGems 1.3.7+
                 •       Chef
                 •       Git



                         git clone git://github.com/opscode/velocity2011-chef-repo
Tuesday, June 14, 2011

Ruby 1.9.2 is recommended. It is higher performance, Chef works well with it and it comes with a reasonable, stable version of
RubyGems, version 1.3.7.

Those that received the installation instructions will note that we’re currently recommending RVM for workstation setup. This is
not a recommendation for managed nodes.

We’re working diligently on a full-stack installer for Chef, its in testing and will be done soon.
Getting Started: Source Code Repository




                 •       Chef Repository for Velocity 2011
                         •     git://github.com/opscode/velocity2011-chef-repo

                 •       Upload to Opscode Hosted Chef server
                         •     roles
                         •     data bags
                         •     cookbooks
                         •     environments

                             git clone git://github.com/opscode/velocity2011-chef-repo
Tuesday, June 14, 2011

The repository has a README-velocity.md file that describes how to Upload the Repository to the Opscode Hosted Chef server.
Working in the Repository




                                      export ORGNAME="your_organization_name"
                                      export OPSCODE_USER="your_opscode_username"
                                      export AWS_ACCESS_KEY_ID="amazon aws access key id"
                                      export AWS_SECRET_ACCESS_KEY="amazon aws secret access key"
                                      export RACKSPACE_API_KEY="rackspace cloud api key"
                                      export RACKSPACE_API_USERNAME="rackspace cloud api username"
                                      % cd velocity2011-chef-repo
                                      % cat .chef/knife.rb
                                      % knife ec2 server list
                                      % knife rackspace server list
                                      % knife client list




                                                git clone git://github.com/opscode/velocity2011-chef-repo
Tuesday, June 14, 2011

Export these variables with your cloud credentials.

The README in the repository contains these instructions too.
knife ec2 server create
                                      OR!
                         knife rackspace server create

                                git clone git://github.com/opscode/velocity2011-chef-repo
Tuesday, June 14, 2011

With all that, we can run the series of knife ec2 server create commands. Nothing more than this to get fully automated
infrastructure launched.

The file README-velocity.md contains all the commands needed to get started with launching infrastructure for yourself.
Anatomy of a Chef Run

                         % knife ec2 server create -G default -I ami-7000f019 -f m1.small 
                           -S velocity-2011-aws -i ~/.ssh/velocity-2011-aws.pem -x ubuntu 
                           -E production -r 'role[base],role[mediawiki_database_master]'




Tuesday, June 14, 2011

What happens when we run the knife command?
Anatomy of a Chef Run: EC2 Create

                                     % knife ec2 server create -G default -I ami-7000f019 -f m1.small 
                                       -S velocity-2011-aws -i ~/.ssh/velocity-2011-aws.pem -x ubuntu 
                                       -E production -r 'role[base],role[mediawiki_database_master]'

                                     Instance ID: i-8157d9ef
                                     Flavor: m1.small
                                     Image: ami-7000f019
                                     Availability Zone: us-east-1a
                                     Security Groups: default
                                     SSH Key: velocity-2011-aws

                                     Waiting for server...............................
                                     Public DNS Name: ec2-50-17-117-98.compute-1.amazonaws.com
                                     Public IP Address: 50.17.117.98
                                     Private DNS Name: ip-10-245-87-117.ec2.internal
                                     Private IP Address: 10.245.87.117

                                     Waiting for sshd....done
                                     Bootstrapping Chef on ec2-50-17-117-98.compute-1.amazonaws.com


Tuesday, June 14, 2011

The knife ec2 server create command makes a call to the Amazon EC2 API through fog[0] and waits for SSH.

There’s a lot here to type, so you can copy/paste out of the README-velocity.md.

[0]: http://rubygems.org/gems/fog
Anatomy of a Chef Run: Bootstrap


                                       Successfully installed             mixlib-authentication-1.1.4
                                       Successfully installed             mime-types-1.16
                                       Successfully installed             rest-client-1.6.3
                                       Successfully installed             bunny-0.6.0
                                       Successfully installed             json-1.5.1
                                       Successfully installed             polyglot-0.3.1
                                       Successfully installed             treetop-1.4.9
                                       Successfully installed             net-ssh-2.1.4
                                       Successfully installed             net-ssh-gateway-1.1.0
                                       Successfully installed             net-ssh-multi-1.0.1
                                       Successfully installed             erubis-2.7.0
                                       Successfully installed             moneta-0.6.0
                                       Successfully installed             highline-1.6.2
                                       Successfully installed             uuidtools-2.1.2
                                       Successfully installed             chef-0.10.0
                                       15 gems installed



Tuesday, June 14, 2011

After the system is available in EC2 and SSH is up, the “bootstrap” process takes over. Chef is installed.
Anatomy of a Chef Run: Validation




                                      (
                                      cat <<'EOP'
                                      <%= validation_key %>
                                      EOP
                                      ) > /tmp/validation.pem
                                      awk NF /tmp/validation.pem > /etc/chef/validation.pem
                                      rm /tmp/validation.pem




Tuesday, June 14, 2011

The bootstrap will write out the validation certificate from the local workstation to the target system.
Anatomy of a Chef Run: Configuration




                                     (
                                     cat <<'EOP'
                                     <%= config_content %>
                                     EOP
                                     ) > /etc/chef/client.rb




Tuesday, June 14, 2011

The chef client configuration file is written based on values from the local system.

The bootstrap is done from a template you can customize, so you can change the content in the EOP to whatever client.rb you
want.
/etc/chef/client.rb




                                      log_level        :info
                                      log_location     STDOUT
                                      chef_server_url "https://api.opscode.com/organizations/velocitydemo"
                                      validation_client_name "velocitydemo-validator"
                                      node_name "i-138c137d"




Tuesday, June 14, 2011

For example, this is all it takes to configure the Chef Client on the new system.
Anatomy of a Chef Run: Run List




                         (
                         cat <<'EOP'
                         <%= { "run_list" => @run_list }.to_json %>
                         EOP
                         ) > /etc/chef/first-boot.json




Tuesday, June 14, 2011
Anatomy of a Chef Run: chef-client




                                       chef-client -j /etc/chef/first-boot.json

                                       # run with debug output for full detail:

                                       chef-client -j          /etc/chef/first-boot.json -l debug




Tuesday, June 14, 2011

Normally we just run chef-client with info level log output. To get more detail, I ran it with debug.

The -l debug option is available any time you want more detailed output from Chef.
Anatomy of a Chef Run: Ohai!



                                     INFO: *** Chef 0.10.0 ***
                                     DEBUG: Loading plugin os
                                     DEBUG: Loading plugin kernel
                                     DEBUG: Loading plugin ruby
                                     DEBUG: Loading plugin languages
                                     DEBUG: Loading plugin hostname
                                     DEBUG: Loading plugin linux::hostname
                                     ...
                                     DEBUG: Loading plugin ec2
                                     DEBUG: has_ec2_mac? == true
                                     DEBUG: can_metadata_connect? == true
                                     DEBUG: looks_like_ec2? == true
                                     DEBUG: Loading plugin rackspace
                                     ...
                                     DEBUG: Loading plugin cloud



Tuesday, June 14, 2011

Chef runs ohai, the system profiling and data gathering tool. Ohai automatically detects a number of attributes about the system
it is running on, including the kernel, operating system/platform, hostname and more.
Run Ohai




                 •       Run `ohai | less` on your system.
                 •       Marvel at the amount of data it returns.




Tuesday, June 14, 2011

You can run `ohai` on your local system with Chef installed to see what Chef discovers about it.
Anatomy of a Chef Run: Authenticate




                                       INFO: Client key /etc/chef/client.pem is not present -
                                       registering

                                       DEBUG: Signing the request as velocitydemo-validator

                                       DEBUG: Sending HTTP Request via POST to api.opscode.com:443/
                                       organizations/velocitydemo/clients

                                       DEBUG: Registration response: {"uri"=>"https://
                                       api.opscode.com/organizations/velocitydemo/clients/
                                       i-8157d9ef", "private_key"=>"SNIP!"}




Tuesday, June 14, 2011

If /etc/chef/client.pem is not present, the validation client is used to register a new client automatically.

The response comes back with the private key, which is written to /etc/chef/client.pem. All subsequent API requests to the
server will use the newly created client, and the /etc/chef/validation.pem file can be deleted (we have chef-
client::delete_validation for this).

Yes, the client’s private key is displayed. Be mindful of this when pasting debug output.

* http://tickets.opscode.com/browse/CHEF-2238
Anatomy of a Chef Run: Build Node



                                      DEBUG: Building node object for i-8157d9ef
                                      DEBUG: Signing the request as i-8157d9ef
                                      DEBUG: Sending HTTP Request via GET to api.opscode.com:443/
                                      organizations/velocitydemo/nodes/i-8157d9ef
                                      INFO: HTTP Request Returned 404 Not Found: Cannot load node
                                      i-8157d9ef
                                      DEBUG: Signing the request as i-8157d9ef
                                      DEBUG: Sending HTTP Request via POST to api.opscode.com:443/
                                      organizations/velocitydemo/nodes
                                      DEBUG: Extracting run list from JSON attributes provided on
                                      command line
                                      INFO: Setting the run_list to ["role[base]", "role
                                      [mediawiki_database_master]"] from JSON
                                      DEBUG: Applying attributes from json file
                                      DEBUG: Platform is ubuntu version 10.04



Tuesday, June 14, 2011

We have 3 important pieces of information about building the node object at this point. First, the instance ID is used as the node
name. This is automatically set up as the default node name by knife ec2 server create.

Second, the JSON file passed into chef-client determines the run list of the node.

Finally, during the ohai data gathering, it determined that the platform of the system is Ubuntu 10.04. This is important for how
our resources will be configured by the underlying providers.
Anatomy of a Chef Run: Sync Cookbooks




                                      INFO: Run List is [role[base], role
                                      [mediawiki_database_master]]

                                      INFO: Run List expands to [apt, zsh, users::sysadmins, sudo,
                                      git, build-essential, database::master]

                                      INFO: Starting Chef Run for i-8157d9ef

                                      DEBUG: Synchronizing cookbooks

                                      INFO: Loading cookbooks [apt, aws, build-essential,
                                      database, git, mysql, openssl, runit, sudo, users, xfs, zsh]




Tuesday, June 14, 2011

Once the run list is determined, it is expanded to find all the recipes that will be applied. The names of the recipes indicate which
cookbooks are required, and those cookbooks are downloaded.

Cookbooks are like packages, so sometimes they depend on another which may not show up in the run list. Dependencies can be
declared in cookbook metadata, similar to packaging system metadata for packages.
Anatomy of a Chef Run: Load Cookbooks




                 •       Chef loads cookbook components after
                         they are downloaded.
                 •       Libraries
                 •       Providers
                 •       Resources
                 •       Attributes
                 •       Definitions
                 •       Recipes

Tuesday, June 14, 2011

Once all the cookbooks have been downloaded, Chef will load the Ruby components of the cookbook. This is done in the order
above.
Anatomy of a Chef Run: Load Recipes




                                      DEBUG:     Loading Recipe zsh via include_recipe
                                      DEBUG:     Found recipe default in cookbook zsh
                                      DEBUG:     Loading Recipe users::sysadmins via include_recipe
                                      DEBUG:     Found recipe sysadmins in cookbook users

                                      DEBUG: Sending HTTP Request via GET to api.opscode.com:443/
                                      organizations/velocitydemo/search/users




Tuesday, June 14, 2011

When recipes are loaded, the Ruby code they contain is evaluated. This is where things like search will hit the server API. We’ll
see more of this later on.

Chef is building what we call the “resource collection”, an ordered list of all the resources that should be configured on the node.
Order Matters


Tuesday, June 14, 2011

The order of the run list and the order of resources in recipes is important, because it matters how your systems are configured.
A half configured system is a broken system, and a system configured out of order may be a broken system. Chef’s implicit
ordering makes it easy to reason about the way systems are built, so you can identify and troubleshoot this easier.
Anatomy of a Chef Run: Convergence

                                     user u['id'] do
                                       uid u['uid']
                                       gid u['gid']
                                       shell u['shell']
                                       comment u['comment']
                                       supports :manage_home => true
                                       home home_dir
                                     end

                                     directory "#{home_dir}/.ssh" do
                                       owner u['id']
                                       group u['gid'] || u['id']
                                       mode "0700"
                                     end

                                     template "#{home_dir}/.ssh/authorized_keys" do
                                       source "authorized_keys.erb"
                                       owner u['id']
                                       group u['gid'] || u['id']
                                       mode "0600"
                                       variables :ssh_keys => u['ssh_keys']
                                     end

Tuesday, June 14, 2011

For example, our users::sysadmins recipe creates some resources for each user it finds from the aforementioned search.

These resources are added to the resource collection in the specified order. This is repeated for every user.
Anatomy of a Chef Run: Convergence




                                     INFO: Processing user[velocity] action create
                                     (users::sysadmins line 41)

                                     INFO: Processing directory[/home/velocity/.ssh] action
                                     create (users::sysadmins line 51)

                                     INFO: Processing template[/home/velocity/.ssh/
                                     authorized_keys] action create (users::sysadmins line 57)




Tuesday, June 14, 2011

Convergence is the phase when the resources in the resource collection are configured. Providers take the appropriate action.
Users are created, packages are installed, services are started and so on.
Anatomy of a Chef Run: Save Node




                                       DEBUG: Saving the current state of node i-8157d9ef

                                       DEBUG: Signing the request as i-8157d9ef

                                       DEBUG: Sending HTTP Request via PUT to api.opscode.com:443/
                                       organizations/velocitydemo/nodes/i-8157d9ef




Tuesday, June 14, 2011

At the end of a run, the state of the node is saved, including all the attributes that were applied to the node from:

*   ohai
*   roles
*   cookbooks
*   environment

This data is also indexed by the server for search.
Anatomy of a Chef Run: Report Handlers




                                         INFO: Running report handlers
                                         INFO: Report handlers complete

                                         ... OR ...

                                         ERROR: Running exception handlers
                                         FATAL: Saving node information to /var/chef/cache/failed-
                                         run-data.json
                                         ERROR: Exception handlers complete
                                         FATAL: Stacktrace dumped to /var/chef/cache/chef-
                                         stacktrace.out
                                         FATAL: Some unhandled Ruby exception message here.




Tuesday, June 14, 2011

At the end of the Chef run, report and exception handlers are executed.

Report handlers are executed on a successful run.

Exception handlers are executed on an unsuccessful run.


      * stack trace data and state of the failed run are also saved to files on the filesystem, and reported.
I can haz cloud?


                             http://www.flickr.com/photos/felixmorgner/4347750467/

Tuesday, June 14, 2011
Configured systems are
                                Nodes.

                                                     http://www.flickr.com/photos/peterrosbjerg/3913766224/

Tuesday, June 14, 2011

Once a node is saved on the server, it is considered a managed system. In Chef, nodes do all the heavy lifting. All the above
happens on the node, the server just handles API requests and serves data/cookbooks.
knife node show




                                  % knife node show i-cda03aa3
                                  Node Name:   i-cda03aa3
                                  Environment: production
                                  FQDN:        ip-10-112-85-253.ec2.internal
                                  IP:          10.112.85.253
                                  Run List:    role[base], role[monitoring]
                                  Roles:       monitoring, base
                                  Recipes      apt, zsh, users::sysadmins, sudo, git, build-
                                  essential, nagios::client, nagios::server
                                  Platform:    ubuntu 10.04
                                  % knife node show i-cda03aa3 -m # non-automatic attributes
                                  % knife node show i-cda03aa3 -l # all attributes
                                  % knife node show i-cda03aa3 -Fj # JSON output




Tuesday, June 14, 2011

We can show the nodes we have configured!
Data Driven


Tuesday, June 14, 2011

The deployment is data driven. Besides the data that came from the roles which we’re about to see, we also have arbitrary data
about our infrastructure, namely the application we’re deploying and the users we’re creating.

We didn’t have to write or modify any code to get a fully functional infrastructure.
Writing Data Driven Cookbooks




                 •       Focus on primitives.
                 •       Apply the desired system state / behavior.
                 •       Don’t hardcode data.
                         •   Attributes
                         •   Data bags
                         •   Search



Tuesday, June 14, 2011
Data Driven Deployment




                                     data_bags
                                     ├── apps
                                     │   └── mediawiki.json
                                     └── users
                                         ├── nagiosadmin.json
                                         └── velocity.json




Tuesday, June 14, 2011

We encapsulate all the information about our application, including environment-specific details. We also have two users we’re
creating.
Each Instance Has a Role




                         roles
                         ├── base.rb
                         ├── mediawiki.rb                    Two app servers!
                         ├── mediawiki_database_master.rb
                         ├── mediawiki_load_balancer.rb
                         └── monitoring.rb




Tuesday, June 14, 2011
All Your Base...


Tuesday, June 14, 2011
Base Role


                                      % knife role show base
                                      chef_type:             role
                                      default_attributes:    {}
                                      description:           Base role applied to all nodes.
                                      env_run_lists:         {}
                                      json_class:            Chef::Role
                                      name:                  base
                                      override_attributes:
                                        authorization:
                                          sudo:
                                            passwordless: true
                                            users:         ["ubuntu"]
                                        nagios:
                                          server_role: monitoring
                                      run_list:             recipe[apt], recipe[zsh], recipe
                                      [users::sysadmins], recipe[sudo], recipe[git], recipe[build-
                                      essential]


Tuesday, June 14, 2011

The base role is going to apply some settings that are common across the entire infrastructure. For example, apt ensures apt
caches are updated, zsh installs the Z shell in case any users want it. Users::sysadmins creates all the system administrator users.
Sudo sets up sudo permissions. Git ensures that our favorite version control system is installed. Build essential ensures that we
can build our application, RubyGem native extensions, or other tools that should be installed by compilation.
Packages vs Source

                                                       Lean into it.


Tuesday, June 14, 2011

The base role installs build-essential. You may opt to only have packages. Build your infrastructure the way you want :).

We’re not going to have a holy war of packages vs source.

Come to DevOpsDays Mountain View for a panel discussion on this topic.
Nagios Server


Tuesday, June 14, 2011

Every well built infrastructure needs monitoring. We’ve set up Nagios for our monitoring system. We could also add another tool
such as munin to the mix if we wanted - there’s a munin cookbook that is data driven too.
Nagios Server




                                    % knife role show monitoring
                                    chef_type:            role
                                    default_attributes:
                                      nagios:
                                        server_auth_method: htauth
                                    description:          Monitoring Server
                                    env_run_lists:        {}
                                    json_class:           Chef::Role
                                    name:                 monitoring
                                    override_attributes: {}
                                    run_list:             recipe[nagios::server]




Tuesday, June 14, 2011

We’ve modified the default behavior of the cookbook to enable htauth authentication.
Load Balancer


Tuesday, June 14, 2011
Load Balancer




                                      % knife role show mediawiki_load_balancer
                                      chef_type:            role
                                      default_attributes:   {}
                                      description:          mediawiki load balancer
                                      env_run_lists:        {}
                                      json_class:           Chef::Role
                                      name:                 mediawiki_load_balancer
                                      override_attributes:
                                        haproxy:
                                          app_server_role: mediawiki
                                      run_list:             recipe[haproxy::app_lb]




Tuesday, June 14, 2011

We’re using haproxy, and we’ll search for a specific application to load balance. The recipe is written to search for the mediawiki
role to find systems that should be pool members.
MediaWiki App Servers
                                (two)


Tuesday, June 14, 2011

We actually have just the one system, we’ll add another one shortly :).
MediaWiki App Servers




                                       % knife role show mediawiki
                                       chef_type:            role
                                       default_attributes:   {}
                                       description:          mediawiki front end application
                                       server.
                                       env_run_lists:        {}
                                       json_class:           Chef::Role
                                       name:                 mediawiki
                                       override_attributes: {}
                                       run_list:             recipe[mysql::client], recipe
                                       [application], recipe[mediawiki::status]




Tuesday, June 14, 2011

The main thing in this role is the application recipe.

The recipe will read in data from the data bag (in a predefined format) to determine what kind of application to deploy, the
repository where it lives, details on where to put it, what roles to search for to find the database, and many more customizable
properties.

We launched two of these to have something to load balance :).
Application Data Bag Item

                         {
                           "id": "mediawiki",
                           "server_roles": [
                              "mediawiki"
                           ],
                           "type": {
                              "mediawiki": [
                                "php",
                                "mod_php_apache2"
                              ]
                           },
                           "database_master_role": [
                              "mediawiki_database_master"
                           ],
                           "repository": "git://github.com/mediawiki/mediawiki-trunk-
                         phase3.git",
                           "revision": {
                              "production": "master",
                              "staging": "master"
                           },
                         ...

Tuesday, June 14, 2011
Database Master


Tuesday, June 14, 2011

Every database backed application needs a master database. For this simple example we haven’t done any complex setup of
master/slave replication, but the recipes are built such that this would be relatively easy to add.
Database Master




                                     % knife role show mediawiki_database_master
                                     default_attributes:   {}
                                     description:          database master for the mediawiki
                                     application.
                                     env_run_lists:        {}
                                     json_class:           Chef::Role
                                     name:                 mediawiki_database_master
                                     override_attributes: {}
                                     run_list:             recipe[database::master]




Tuesday, June 14, 2011

The database master recipe will read the application information from the data bag and use it to create the database so the
application can store its data.
Cookbooks are easy to share.


Tuesday, June 14, 2011

Chef is designed such that cookbooks are easy to share. Data is easy to separate from logic in recipes by using Attributes and
Chef’s rich data discovery and look up features such as data bags.
Data Driven Cookbooks




                 •       application & database
                 •       nagios
                 •       users
                                                                                        http://www.flickr.com/photos/41176169@N00/2643328666/




Tuesday, June 14, 2011

Through data bag modification, role settings and Chef’s search feature, these cookbooks are data driven. No code was modified.
You didn’t have to understand Ruby (though we think its a good idea :)), and you can deploy an infrastructure quickly and easily.
Open Source Cookbooks




                                      knife    cookbook   site   install   nagios
                                      knife    cookbook   site   install   git
                                      knife    cookbook   site   install   application
                                      knife    cookbook   site   install   database
                                      knife    cookbook   site   install   haproxy
                                      knife    cookbook   site   install   sudo
                                      knife    cookbook   site   install   users
                                      knife    cookbook   site   install   zsh




Tuesday, June 14, 2011

The cookbooks directory contains all the cookbooks we need.

These do all kinds of things we didn’t have to write.

These cookbooks all came from community.opscode.com
Application-specific Cookbooks




                                    knife cookbook create mediawiki

                                    $EDITOR cookbooks/mediawiki/recipes/db_bootstrap.rb




Tuesday, June 14, 2011

Your application probably doesn’t have a specific cookbook already shared by the community.

We create our mediawiki cookbook for application specific purposes.
mediawiki::db_bootstrap



                                    app = data_bag_item("apps", "mediawiki")
                                    dbm = search(:node, "role:mediawiki_database_master")
                                    db = app['databases'][node.chef_environment]

                                    execute "db_bootstrap" do
                                      command <<-EOH
                                      /usr/bin/mysql 
                                        -u #{db['username']} 
                                        -p#{db['password']} 
                                        -h #{dbm['fqdn']} 
                                        #{db['database']} 
                                        < #{Chef::Config[:file_cache_path]}/schema.sql"
                                      EOH
                                      action :run
                                    end



Tuesday, June 14, 2011

We retrieve some data up front.

Then we use it to configure a resource.
Systems Integration
                               through Discovery.

                                                     http://www.flickr.com/photos/c0t0s0d0/2425404674/

Tuesday, June 14, 2011

The systems we manage are running their own services to fullfill their purpose in the infrastructure. Each of those services is
network accessible, and by expressing our systems through rich metadata, we can discover the systems that fullfill each role
through searching the chef server.
Search for Nodes with Knife




                         % knife search node role:mediawiki_database_master
                         1 items found

                         Node Name:   i-8157d9ef
                         Environment: production
                         FQDN:        ip-10-245-87-117.ec2.internal
                         IP:          10.245.87.117
                         Run List:    role[base], role[mediawiki_database_master]
                         Roles:       mediawiki_database_master, base
                         Recipes      apt, zsh, users::sysadmins, sudo, git, build-
                         essential, database::master
                         Platform:    ubuntu 10.04




Tuesday, June 14, 2011
Search for Nodes in Recipes




                                     results = search (:node, "role:mediawiki_database_master")

                                     template "/srv/mediawiki/shared/LocalSettings.php" do
                                       source "LocalSettings.erb"
                                       mode "644"
                                       variables(
                                         :path => "/srv/mediawiki/current",
                                         :host => results[0]['fqdn']
                                       )
                                     end




Tuesday, June 14, 2011

You no longer need to track which system has an IP that should be applied as the database master. We can just use its fqdn from
a search.
Managing Infrastructure: Knife SSH



                         % knife ssh 'role:mediawiki_database_master' 'sudo chef-
                         client' -a ec2.public_hostname -x ubuntu
                         ec2-50-17-117-98 INFO: *** Chef 0.10.0 ***
                         ec2-50-17-117-98 INFO: Run List is [role[base], role
                         [mediawiki_database_master]]
                         ec2-50-17-117-98 INFO: Run List expands to [apt, zsh,
                         users::sysadmins, sudo, git, build-essential,
                         database::master]
                         ec2-50-17-117-98 INFO: Starting Chef Run for i-8157d9ef
                         ec2-50-17-117-98 INFO: Loading cookbooks [apt, aws, build-
                         essential, database, git, mysql, openssl, runit, sudo,
                         users, xfs, zsh]
                         ec2-50-17-117-98 INFO: Chef Run complete in 9.471502 seconds
                         ec2-50-17-117-98 INFO: Running report handlers
                         ec2-50-17-117-98 INFO: Report handlers complete



Tuesday, June 14, 2011
What port is haproxy admin again?




                                      % knife ssh role:mediawiki_load_balancer -a ec2.public_hostname 
                                        'netstat -an | grep LISTEN'
                                      tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN
                                      tcp        0      0 0.0.0.0:22002           0.0.0.0:*               LISTEN
                                      tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
                                      tcp        0      0 0.0.0.0:5666            0.0.0.0:*               LISTEN
                                      tcp6       0      0 :::22                   :::*                    LISTEN




Tuesday, June 14, 2011

Oh that’s right. I always forget how many 2’s and 0’s.
Managing Nodes through an API




                                      knife node run list add NODE "recipe[mediawiki::api_update]"
                                      knife exec -E 'nodes.transform("role:mediawiki") 
                                        {|n| n.run_list << "recipe[mediawiki::api_update]"}'
                                      knife ssh 'role:mediawiki' -x velocity 'sudo chef-client' 
                                        -a cloud.public_hostname




Tuesday, June 14, 2011

We can programmatically add a recipe to the run list of all our nodes through the server API.
Manage Infrastructure: Knife SSH




                 •       “SSH In a For Loop” is bad right?
                 •       Parallel command execution.
                 •       SSH is industry standard.
                 •       Use sudo NOPASSWD.




Tuesday, June 14, 2011

“Best practice” suggests that ssh in a for loop is bad, because the prevailing idea is we’re doing “one-off” changes.

We’re actually working toward parallel command execution. Kick off a chef-client run on a set of nodes, or gather some kind of
command output.

SSH is an industry standard that everyone understands and knows how to set up.

A security best practice is to use sudo with NOPASSWD, which is e.g. how the Ubuntu AMIs are set up by Canonical.
Wrap-up




                 •       Infrastructure as Code
                 •       Getting Started with Chef
                 •       Anatomy of a Chef Run
                 •       Data Driven Shareable Cookbooks
                 •       Managing Cloud Infrastructure



                                                                      http://www.flickr.com/photos/villes/358790270/

Tuesday, June 14, 2011

We’ve covered a lot of topics today! I’m sure you have questions...
FAQ: Chef vs [Other Tool]


Tuesday, June 14, 2011
http://www.flickr.com/photos/gesika22/4458155541/

Tuesday, June 14, 2011

We can have that conversation over a pint :).
FAQ: How do you test
                               recipes?


Tuesday, June 14, 2011
FAQ: Testing




                 •       You launch cloud instances and watch
                         them converge.
                 •       You use Vagrant with a Chef
                         Provisioner




Tuesday, June 14, 2011

We test recipes by running chef-client. Chef environments prevent recipe errors from affecting production.

Or, you buy Stephen Nelson-Smith’s book!
FAQ: Testing




                 •       You buy Stephen Nelson-Smith’s book!




Tuesday, June 14, 2011
FAQ: How does Chef
                               scale?


Tuesday, June 14, 2011
FAQ: Scale




                 •       The Chef Server is a publishing
                         system.
                 •       Nodes do the heavy lifting.
                 •       Chef scales like a service-oriented
                         web application.
                 •       Opscode Hosted Chef was designed
                         and built for massive scale.


                                                               http://www.flickr.com/photos/amagill/61205408/

Tuesday, June 14, 2011
Questions?




                 •       http://opscode.com
                 •       http://wiki.opscode.com
                 •       @opscode, #opschef
                 •       irc.freenode.net, #chef, #chef-hacking
                 •       http://lists.opscode.com
                 •       We’re in the exhibit hall this week.
                 •       We’ll be at DevOpsDays Mountain View.

                                                http://www.flickr.com/photos/oberazzi/318947873/
Tuesday, June 14, 2011
Thanks!


                         http://opscode.com
                              @opscode
                               #opschef



Tuesday, June 14, 2011

More Related Content

What's hot

Chef Fundamentals Training Series Module 1: Overview of Chef
Chef Fundamentals Training Series Module 1: Overview of ChefChef Fundamentals Training Series Module 1: Overview of Chef
Chef Fundamentals Training Series Module 1: Overview of Chef
Chef Software, Inc.
 
Chef Fundamentals Training Series Module 4: The Chef Client Run and Expanding...
Chef Fundamentals Training Series Module 4: The Chef Client Run and Expanding...Chef Fundamentals Training Series Module 4: The Chef Client Run and Expanding...
Chef Fundamentals Training Series Module 4: The Chef Client Run and Expanding...
Chef Software, Inc.
 

What's hot (20)

Infrastructure Automation with Chef
Infrastructure Automation with ChefInfrastructure Automation with Chef
Infrastructure Automation with Chef
 
Chef Tutorial for DEVOPS Newbies
Chef Tutorial for DEVOPS NewbiesChef Tutorial for DEVOPS Newbies
Chef Tutorial for DEVOPS Newbies
 
Chef Fundamentals Training Series Module 1: Overview of Chef
Chef Fundamentals Training Series Module 1: Overview of ChefChef Fundamentals Training Series Module 1: Overview of Chef
Chef Fundamentals Training Series Module 1: Overview of Chef
 
Chef Fundamentals Training Series Module 6: Roles, Environments, Community Co...
Chef Fundamentals Training Series Module 6: Roles, Environments, Community Co...Chef Fundamentals Training Series Module 6: Roles, Environments, Community Co...
Chef Fundamentals Training Series Module 6: Roles, Environments, Community Co...
 
Chef Fundamentals Training Series Module 2: Workstation Setup
Chef Fundamentals Training Series Module 2: Workstation SetupChef Fundamentals Training Series Module 2: Workstation Setup
Chef Fundamentals Training Series Module 2: Workstation Setup
 
Chef ignited a DevOps revolution – BK Box
Chef ignited a DevOps revolution – BK BoxChef ignited a DevOps revolution – BK Box
Chef ignited a DevOps revolution – BK Box
 
Chef Fundamentals Training Series Module 3: Setting up Nodes and Cookbook Aut...
Chef Fundamentals Training Series Module 3: Setting up Nodes and Cookbook Aut...Chef Fundamentals Training Series Module 3: Setting up Nodes and Cookbook Aut...
Chef Fundamentals Training Series Module 3: Setting up Nodes and Cookbook Aut...
 
Chef introduction
Chef introductionChef introduction
Chef introduction
 
Chef Fundamentals Training Series Module 4: The Chef Client Run and Expanding...
Chef Fundamentals Training Series Module 4: The Chef Client Run and Expanding...Chef Fundamentals Training Series Module 4: The Chef Client Run and Expanding...
Chef Fundamentals Training Series Module 4: The Chef Client Run and Expanding...
 
Introduction to Chef
Introduction to ChefIntroduction to Chef
Introduction to Chef
 
Opscode tech festa july 2013
Opscode tech festa   july 2013Opscode tech festa   july 2013
Opscode tech festa july 2013
 
Chef fundamentals
Chef fundamentalsChef fundamentals
Chef fundamentals
 
Chef: Smart infrastructure automation
Chef: Smart infrastructure automationChef: Smart infrastructure automation
Chef: Smart infrastructure automation
 
Automating your infrastructure with Chef
Automating your infrastructure with ChefAutomating your infrastructure with Chef
Automating your infrastructure with Chef
 
AWS Meetup - Sydney - February
AWS Meetup - Sydney - February AWS Meetup - Sydney - February
AWS Meetup - Sydney - February
 
Understand Chef
Understand ChefUnderstand Chef
Understand Chef
 
Node object and roles - Fundamentals Webinar Series Part 3
Node object and roles - Fundamentals Webinar Series Part 3Node object and roles - Fundamentals Webinar Series Part 3
Node object and roles - Fundamentals Webinar Series Part 3
 
Overview of Chef - Fundamentals Webinar Series Part 1
Overview of Chef - Fundamentals Webinar Series Part 1Overview of Chef - Fundamentals Webinar Series Part 1
Overview of Chef - Fundamentals Webinar Series Part 1
 
Node setup, resource, and recipes - Fundamentals Webinar Series Part 2
Node setup, resource, and recipes - Fundamentals Webinar Series Part 2Node setup, resource, and recipes - Fundamentals Webinar Series Part 2
Node setup, resource, and recipes - Fundamentals Webinar Series Part 2
 
Community Cookbooks & further resources - Fundamentals Webinar Series Part 6
Community Cookbooks & further resources - Fundamentals Webinar Series Part 6Community Cookbooks & further resources - Fundamentals Webinar Series Part 6
Community Cookbooks & further resources - Fundamentals Webinar Series Part 6
 

Viewers also liked

Viewers also liked (12)

IT Automation with Chef
IT Automation with ChefIT Automation with Chef
IT Automation with Chef
 
Introduction to Chef
Introduction to ChefIntroduction to Chef
Introduction to Chef
 
Cook Infrastructure with chef -- Justeat.IN
Cook Infrastructure with chef  -- Justeat.INCook Infrastructure with chef  -- Justeat.IN
Cook Infrastructure with chef -- Justeat.IN
 
Chef
ChefChef
Chef
 
Devops madrid: successful case in AWS
Devops madrid: successful case in AWSDevops madrid: successful case in AWS
Devops madrid: successful case in AWS
 
Testable Infrastructure with Chef, Test Kitchen, and Docker
Testable Infrastructure with Chef, Test Kitchen, and DockerTestable Infrastructure with Chef, Test Kitchen, and Docker
Testable Infrastructure with Chef, Test Kitchen, and Docker
 
Introducing Chef | An IT automation for speed and awesomeness
Introducing Chef | An IT automation for speed and awesomenessIntroducing Chef | An IT automation for speed and awesomeness
Introducing Chef | An IT automation for speed and awesomeness
 
DevOps and Chef improve your life
DevOps and Chef improve your life DevOps and Chef improve your life
DevOps and Chef improve your life
 
Overview of chef ( Infrastructure as a Code )
Overview of chef ( Infrastructure as a Code )Overview of chef ( Infrastructure as a Code )
Overview of chef ( Infrastructure as a Code )
 
Chef Cookbook Testing and Continuous Integration
Chef Cookbook Testing and Continuous IntegrationChef Cookbook Testing and Continuous Integration
Chef Cookbook Testing and Continuous Integration
 
Jenkins and Chef: Infrastructure CI and Automated Deployment
Jenkins and Chef: Infrastructure CI and Automated DeploymentJenkins and Chef: Infrastructure CI and Automated Deployment
Jenkins and Chef: Infrastructure CI and Automated Deployment
 
Infrastructure Automation with Chef
Infrastructure Automation with ChefInfrastructure Automation with Chef
Infrastructure Automation with Chef
 

Similar to Velocity2011 chef-workshop

Xen Summit 2010 - globo.com
Xen Summit 2010 - globo.comXen Summit 2010 - globo.com
Xen Summit 2010 - globo.com
Marco Sinhoreli
 
Five Pillars of SharePoint Governance Supportability
Five Pillars of SharePoint Governance SupportabilityFive Pillars of SharePoint Governance Supportability
Five Pillars of SharePoint Governance Supportability
Sentri
 
SELF 2011: Deploying Django Application Stacks with Chef
SELF 2011: Deploying Django Application Stacks with ChefSELF 2011: Deploying Django Application Stacks with Chef
SELF 2011: Deploying Django Application Stacks with Chef
Chef Software, Inc.
 
Sneak Peek of Nuxeo 5.4
Sneak Peek of Nuxeo 5.4Sneak Peek of Nuxeo 5.4
Sneak Peek of Nuxeo 5.4
Nuxeo
 

Similar to Velocity2011 chef-workshop (20)

DevOps Introduction @Cegeka
DevOps Introduction @CegekaDevOps Introduction @Cegeka
DevOps Introduction @Cegeka
 
Devops what it means to me
Devops what it means to meDevops what it means to me
Devops what it means to me
 
AppScale Talk at SBonRails
AppScale Talk at SBonRailsAppScale Talk at SBonRails
AppScale Talk at SBonRails
 
Seattle OpenStack Meetup
Seattle OpenStack MeetupSeattle OpenStack Meetup
Seattle OpenStack Meetup
 
Xen summit2010 globocom
Xen summit2010 globocomXen summit2010 globocom
Xen summit2010 globocom
 
App Engine Meetup
App Engine MeetupApp Engine Meetup
App Engine Meetup
 
OSDC 2011 | Marionette - System Control Utility by Cody Herriges
OSDC 2011 | Marionette - System Control Utility by Cody HerrigesOSDC 2011 | Marionette - System Control Utility by Cody Herriges
OSDC 2011 | Marionette - System Control Utility by Cody Herriges
 
Xen Summit 2010 - globo.com
Xen Summit 2010 - globo.comXen Summit 2010 - globo.com
Xen Summit 2010 - globo.com
 
Deploying on the cutting edge
Deploying on the cutting edgeDeploying on the cutting edge
Deploying on the cutting edge
 
MongoDB at Sailthru: Scaling and Schema Design
MongoDB at Sailthru: Scaling and Schema DesignMongoDB at Sailthru: Scaling and Schema Design
MongoDB at Sailthru: Scaling and Schema Design
 
2011 June - Singapore GTUG presentation. App Engine program update + intro to Go
2011 June - Singapore GTUG presentation. App Engine program update + intro to Go2011 June - Singapore GTUG presentation. App Engine program update + intro to Go
2011 June - Singapore GTUG presentation. App Engine program update + intro to Go
 
Introduction to Puppet Scripting
Introduction to Puppet ScriptingIntroduction to Puppet Scripting
Introduction to Puppet Scripting
 
Five Pillars of SharePoint Governance Supportability
Five Pillars of SharePoint Governance SupportabilityFive Pillars of SharePoint Governance Supportability
Five Pillars of SharePoint Governance Supportability
 
SELF 2011: Deploying Django Application Stacks with Chef
SELF 2011: Deploying Django Application Stacks with ChefSELF 2011: Deploying Django Application Stacks with Chef
SELF 2011: Deploying Django Application Stacks with Chef
 
Shomi's Journey to Chef: Lessons Learned on Implementing Chef
Shomi's Journey to Chef: Lessons Learned on Implementing ChefShomi's Journey to Chef: Lessons Learned on Implementing Chef
Shomi's Journey to Chef: Lessons Learned on Implementing Chef
 
Shomi's Journey to Chef: Lessons Learned on Implementing Chef
Shomi's Journey to Chef: Lessons Learned on Implementing ChefShomi's Journey to Chef: Lessons Learned on Implementing Chef
Shomi's Journey to Chef: Lessons Learned on Implementing Chef
 
Red Dirt Ruby Conference
Red Dirt Ruby ConferenceRed Dirt Ruby Conference
Red Dirt Ruby Conference
 
Sneak Peek of Nuxeo 5.4
Sneak Peek of Nuxeo 5.4Sneak Peek of Nuxeo 5.4
Sneak Peek of Nuxeo 5.4
 
Debezium POC
Debezium POCDebezium POC
Debezium POC
 
STI Summit 2011 - Linked services
STI Summit 2011 - Linked servicesSTI Summit 2011 - Linked services
STI Summit 2011 - Linked services
 

More from jtimberman (12)

Socal piggies-app-deploy
Socal piggies-app-deploySocal piggies-app-deploy
Socal piggies-app-deploy
 
Oscon2011 tutorial
Oscon2011 tutorialOscon2011 tutorial
Oscon2011 tutorial
 
Agile services-dev opsdays
Agile services-dev opsdaysAgile services-dev opsdays
Agile services-dev opsdays
 
Cooking security sans@night
Cooking security sans@nightCooking security sans@night
Cooking security sans@night
 
Mwrc2011 cookbook design patterns
Mwrc2011 cookbook design patternsMwrc2011 cookbook design patterns
Mwrc2011 cookbook design patterns
 
tmux lightning talk mwrc
tmux lightning talk mwrctmux lightning talk mwrc
tmux lightning talk mwrc
 
Fosdem chef-101-app-deploy
Fosdem chef-101-app-deployFosdem chef-101-app-deploy
Fosdem chef-101-app-deploy
 
Data driven app deploys with chef frontdev
Data driven app deploys with chef frontdevData driven app deploys with chef frontdev
Data driven app deploys with chef frontdev
 
Understanding lwrp development
Understanding lwrp developmentUnderstanding lwrp development
Understanding lwrp development
 
Derailed chef update-oct2010
Derailed chef update-oct2010Derailed chef update-oct2010
Derailed chef update-oct2010
 
Chef in the cloud [dbccg]
Chef in the cloud [dbccg]Chef in the cloud [dbccg]
Chef in the cloud [dbccg]
 
Automated infrastructure is on the menu
Automated infrastructure is on the menuAutomated infrastructure is on the menu
Automated infrastructure is on the menu
 

Recently uploaded

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Victor Rentea
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Victor Rentea
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 

Recently uploaded (20)

Simplifying Mobile A11y Presentation.pptx
Simplifying Mobile A11y Presentation.pptxSimplifying Mobile A11y Presentation.pptx
Simplifying Mobile A11y Presentation.pptx
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
Platformless Horizons for Digital Adaptability
Platformless Horizons for Digital AdaptabilityPlatformless Horizons for Digital Adaptability
Platformless Horizons for Digital Adaptability
 
Stronger Together: Developing an Organizational Strategy for Accessible Desig...
Stronger Together: Developing an Organizational Strategy for Accessible Desig...Stronger Together: Developing an Organizational Strategy for Accessible Desig...
Stronger Together: Developing an Organizational Strategy for Accessible Desig...
 
How to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cfHow to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cf
 
WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...
WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...
WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...
 
Design Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxDesign Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptx
 
Modernizing Legacy Systems Using Ballerina
Modernizing Legacy Systems Using BallerinaModernizing Legacy Systems Using Ballerina
Modernizing Legacy Systems Using Ballerina
 
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage Intacct
 
The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...
The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...
The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...
 
WSO2's API Vision: Unifying Control, Empowering Developers
WSO2's API Vision: Unifying Control, Empowering DevelopersWSO2's API Vision: Unifying Control, Empowering Developers
WSO2's API Vision: Unifying Control, Empowering Developers
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
 
Choreo: Empowering the Future of Enterprise Software Engineering
Choreo: Empowering the Future of Enterprise Software EngineeringChoreo: Empowering the Future of Enterprise Software Engineering
Choreo: Empowering the Future of Enterprise Software Engineering
 
Six Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal OntologySix Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal Ontology
 
2024 May Patch Tuesday
2024 May Patch Tuesday2024 May Patch Tuesday
2024 May Patch Tuesday
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
CNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In PakistanCNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In Pakistan
 
Less Is More: Utilizing Ballerina to Architect a Cloud Data Platform
Less Is More: Utilizing Ballerina to Architect a Cloud Data PlatformLess Is More: Utilizing Ballerina to Architect a Cloud Data Platform
Less Is More: Utilizing Ballerina to Architect a Cloud Data Platform
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor Presentation
 

Velocity2011 chef-workshop

  • 1. Infrastructure Automation with Opscode Chef http://opscode.com @opscode #opschef Tuesday, June 14, 2011
  • 2. Who are we? • Joshua Timberman • Adam Jacob • Christopher Brown • Aaron Peterson • Seth Chisamore • Matt Ray Tuesday, June 14, 2011
  • 3. Who are you? • System administrators? • Developers? • “Business” People? http://www.flickr.com/photos/timyates/2854357446/sizes/l/ Tuesday, June 14, 2011 Hint, consultants, you’re “Business” people too.
  • 4. What are we talking about? http://www.flickr.com/photos/peterkaminski/2174679908/ Tuesday, June 14, 2011 Managing infrastructure in the Cloud. With Chef, hopefully.
  • 5. Agenda • How’s and Why’s • Live Demo! • Getting Started with Chef • Anatomy of a Chef Run • Managing Cloud Infrastructure • Data Driven Shareable Cookbooks http://www.flickr.com/photos/koalazymonkey/3590953001/ Tuesday, June 14, 2011 How’s and why’s of managing infrastructure with Chef. We’re running a live demo! We’ll walk through the things required to get started with Chef. We will look at the anatomy of a Chef run in detail. Since we’ve launched a cloud infrastructure, we’ll want to know how we manage it. We’ll talk about our data driven sharable cookbooks.
  • 6. Infrastructure as Code Tuesday, June 14, 2011 The goal is fully automated infrastructure. In the cloud, anywhere. We get there with Infrastructure as Code.
  • 7. A technical domain revolving around building and managing infrastructure programmatically Tuesday, June 14, 2011
  • 8. Enable the reconstruction of the business from nothing but a source code repository, an application data backup, and bare metal resources. Tuesday, June 14, 2011
  • 9. Configuration Management Tuesday, June 14, 2011 Keep track of all the steps required to take bare metal systems to doing their job in the infrastructure. It is all about the policy. And this needs to be available as a service in your infrastructure.
  • 10. System Integration http://www.flickr.com/photos/opalsson/3773629074/ Tuesday, June 14, 2011 Taking all the systems that have been configured to do their job, and make them work together to actually run the infrastructure.
  • 11. Tuesday, June 14, 2011 Introducing Chef. Maybe you’ve already met! Stephen Nelson-Smith has a great way to introducing Chef, so with apologies to him, I’m going to reuse his descriptions.
  • 12. The Chef Framework With thanks (and apologies) to Stephen Nelson-Smith Tuesday, June 14, 2011 Chef provides a framework for fully automating infrastructure, and has some important design principles.
  • 13. The Chef Framework • Reasonability • Flexibility • Library & Primitives • TIMTOWTDI Tuesday, June 14, 2011 Chef makes it easy to reason about your infrastructure, at scale. The declarative Ruby configuration language is easy to read, and the predictable ordering makes it easy to understand what’s going on. Chef is flexible, and designed to allow you to build infrastructure using a sane set of libraries and primitives. Just like Perl doesn’t tell programmers how to program, Chef doesn’t tell sysadmins how to manage infrastructure.
  • 14. The Chef Tool(s) With thanks (and apologies) to Stephen Nelson-Smith Tuesday, June 14, 2011 Since Chef is a framework with libraries and primitives for building and managing infrastructure, it only makes sense that it comes with tools written for that purpose.
  • 15. The Chef Tool(s) • ohai • chef-client • knife • shef Tuesday, June 14, 2011 Ohai profiles the system to gather data about nodes and emits that data as JSON. Chef client runs on your nodes to configure them. Knife is used to access the API. Shef is an interactive console debugger.
  • 16. The Chef API With thanks (and apologies) to Stephen Nelson-Smith Tuesday, June 14, 2011 The Chef API provides a client/server service for configuration management in your infrastructure.
  • 17. The Chef API • RSA key authentication w/ Signed Headers • RESTful API w/ JSON • Search Service • Derivative Services Tuesday, June 14, 2011 The API itself is RESTful with JSON responses. Part of the API is a dynamic search service which can be queried to provide rich data about the objects stored on the server. Because it is flexible and built as a service, it is easy to build derivative services on top, including integration with other tools and services.
  • 18. The Chef Community With thanks (and apologies) to Stephen Nelson-Smith Tuesday, June 14, 2011 As an Open Source project, the Chef community is critical.
  • 19. The Chef Community • Apache License, Version 2.0 • 360+ Individual contributors • 70+ Corporate contributors • Dell, Rackspace,VMware, RightScale, Heroku, and more • http://community.opscode.com • 240+ cookbooks Tuesday, June 14, 2011 Community is important. http://apache.org/licenses/LICENSE-2.0.html http://www.opscode.com/blog/2009/08/11/why-we-chose-the-apache-license/ http://wiki.opscode.com/display/chef/How+to+Contribute http://wiki.opscode.com/display/chef/Approved+Contributors
  • 20. Chef Enables Infrastructure as Code package "haproxy" do action :install end template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" • Resources owner "root" group "root" • Recipes mode 0644 • notifies :restart, "service[haproxy]" Roles end • Source Code service "haproxy" do supports :restart => true action [:enable, :start] end Tuesday, June 14, 2011 Declare system configuration as idempotent resources. Put resources together in recipes. Assign recipes to systems through roles. Track it all like source code.
  • 21. Chef Resources package "haproxy" do action :install end • Have a type. template "/etc/haproxy/haproxy.cfg" do • source "haproxy.cfg.erb" Have a name. owner "root" • Have parameters. group "root" mode 0644 • Take action to put the resource notifies :restart, "service[haproxy]" end in the declared state. • Can send notifications to other service "haproxy" do supports :restart => true resources. action [:enable, :start] end Tuesday, June 14, 2011
  • 22. Resources take action through Providers Tuesday, June 14, 2011 Providers know how to actually configure the resources to be in the declared state
  • 23. Chef Providers package “haproxy” { yum install haproxy apt-get install haproxy pacman sync haproxy pkg_add -r haproxy Tuesday, June 14, 2011 The haproxy package resource may run any number of OS commands, depending on the node’s platform.
  • 24. Recipes are collections of Resources Tuesday, June 14, 2011
  • 25. Chef Recipes package "haproxy" do action :install end template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" • Recipes are evaluated for owner "root" resources in the order they group "root" mode 0644 appear. notifies :restart, "service[haproxy]" • Each resource object is added end to the Resource Collection. service "haproxy" do supports :restart => true action [:enable, :start] end Tuesday, June 14, 2011
  • 26. Chef Recipes • Recipes can include other include_recipe include_recipe "apache2" "apache2::mod_rewrite" recipes. include_recipe "apache2::mod_deflate" • Included recipes are include_recipe include_recipe "apache2::mod_headers" "apache2::mod_php5" processed in order. Tuesday, June 14, 2011 Just like recipes themselves are processed in order, the recipes included are processed in order, so when you include a recipe, all its resources are added to the resource collection, then Chef continues to the next.
  • 27. Chef Recipes • Extend recipes with %w{ php5 php5-dev php5-cgi }.each do |pkg| Ruby. package pkg do • Iterate over an array of action :install end package names to install. end Tuesday, June 14, 2011
  • 28. Chef Recipes template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "root" group "root" mode 0644 notifies :restart, "service[haproxy]" end • Good: Drop off a pool_members = search("node", "role:mediawiki") dynamic template. • Better: Discover data template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" through search. owner "root" group "root" mode 0644 variables :pool_members => pool_members notifies :restart, "service[haproxy]" end Tuesday, June 14, 2011
  • 29. Chef Roles name "mediawiki" description "mediawiki app server" run_list( "recipe[mysql::client]", "recipe[application]", "recipe[mediawiki::status]" ) • Roles describe nodes. name "mediawiki_load_balancer" • Roles have a run list. description "mediawiki load balancer" run_list( • Roles can have attributes. ) "recipe[haproxy::app_lb]" override_attributes( "haproxy" => { "app_server_role" => "mediawiki" } ) Tuesday, June 14, 2011
  • 30. Track it like source code... % git log commit d640a8c6b370134d7043991894107d806595cc35 Author: jtimberman <joshua@opscode.com> Import nagios version 1.0.0 commit c40c818498710e78cf73c7f71e722e971fa574e7 Author: jtimberman <joshua@opscode.com> installation and usage instruction docs commit 99d0efb024314de17888f6b359c14414fda7bb91 Author: jtimberman <joshua@opscode.com> Import haproxy version 1.0.1 commit c89d0975ad3f4b152426df219fee0bfb8eafb7e4 Author: jtimberman <joshua@opscode.com> add mediawiki cookbook commit 89c0545cc03b9be26f1db246c9ba4ce9d58a6700 Author: jtimberman <joshua@opscode.com> multiple environments in data bag for mediawiki Tuesday, June 14, 2011
  • 31. LIVE DEMO!!! git clone git://github.com/opscode/velocity2011-chef-repo Tuesday, June 14, 2011 We thought we’d start with the live demo early on, since last year we were interrupted by a fire alarm.
  • 32. Live Demo • Behind the scenes we’re building a new infrastructure • Five nodes • Database master • Two App servers • Load Balanced • Monitored git clone git://github.com/opscode/velocity2011-chef-repo http://www.flickr.com/photos/takomabibelot/3787425422 Tuesday, June 14, 2011 During this workshop, we will build a cloud infrastructure before your very eyes (if we have multiple displays to show that while the slides are up.)
  • 33. How did we get here? git clone git://github.com/opscode/velocity2011-chef-repo Tuesday, June 14, 2011 How did we get to the point where we can build a multi-tiered, monitored infrastructure?
  • 34. Getting Started • Opscode Hosted Chef • Authentication Credentials • Workstation Installation • Source Code Repository git clone git://github.com/opscode/velocity2011-chef-repo Tuesday, June 14, 2011 We signed up for Opscode Hosted Chef, downloaded our authentication credentials (RSA private keys), installed Chef on our workstation and set up a source code repository.
  • 35. Getting Started: Opscode Hosted Chef • Sign up for Opscode Hosted Chef • https://community.opscode.com/users/new • Sign into Management Console • https://manage.opscode.com • Create an Organization git clone git://github.com/opscode/velocity2011-chef-repo Tuesday, June 14, 2011 The workshop installation instructions describe how to go about the process.
  • 36. Getting Started: Authentication Credentials • Download User Private Key • Download Organization Validation Private Key • Retrieve Cloud Credentials git clone git://github.com/opscode/velocity2011-chef-repo Tuesday, June 14, 2011 The signup process will provide instructions on how to retrieve your user private key and organization validation private key. The examples in the chef repository will use Amazon EC2. You’ll need the cloud credentials.
  • 37. Getting Started: Workstation Installation • Ruby (1.9.2 recommended) • RubyGems 1.3.7+ • Chef • Git git clone git://github.com/opscode/velocity2011-chef-repo Tuesday, June 14, 2011 Ruby 1.9.2 is recommended. It is higher performance, Chef works well with it and it comes with a reasonable, stable version of RubyGems, version 1.3.7. Those that received the installation instructions will note that we’re currently recommending RVM for workstation setup. This is not a recommendation for managed nodes. We’re working diligently on a full-stack installer for Chef, its in testing and will be done soon.
  • 38. Getting Started: Source Code Repository • Chef Repository for Velocity 2011 • git://github.com/opscode/velocity2011-chef-repo • Upload to Opscode Hosted Chef server • roles • data bags • cookbooks • environments git clone git://github.com/opscode/velocity2011-chef-repo Tuesday, June 14, 2011 The repository has a README-velocity.md file that describes how to Upload the Repository to the Opscode Hosted Chef server.
  • 39. Working in the Repository export ORGNAME="your_organization_name" export OPSCODE_USER="your_opscode_username" export AWS_ACCESS_KEY_ID="amazon aws access key id" export AWS_SECRET_ACCESS_KEY="amazon aws secret access key" export RACKSPACE_API_KEY="rackspace cloud api key" export RACKSPACE_API_USERNAME="rackspace cloud api username" % cd velocity2011-chef-repo % cat .chef/knife.rb % knife ec2 server list % knife rackspace server list % knife client list git clone git://github.com/opscode/velocity2011-chef-repo Tuesday, June 14, 2011 Export these variables with your cloud credentials. The README in the repository contains these instructions too.
  • 40. knife ec2 server create OR! knife rackspace server create git clone git://github.com/opscode/velocity2011-chef-repo Tuesday, June 14, 2011 With all that, we can run the series of knife ec2 server create commands. Nothing more than this to get fully automated infrastructure launched. The file README-velocity.md contains all the commands needed to get started with launching infrastructure for yourself.
  • 41. Anatomy of a Chef Run % knife ec2 server create -G default -I ami-7000f019 -f m1.small -S velocity-2011-aws -i ~/.ssh/velocity-2011-aws.pem -x ubuntu -E production -r 'role[base],role[mediawiki_database_master]' Tuesday, June 14, 2011 What happens when we run the knife command?
  • 42. Anatomy of a Chef Run: EC2 Create % knife ec2 server create -G default -I ami-7000f019 -f m1.small -S velocity-2011-aws -i ~/.ssh/velocity-2011-aws.pem -x ubuntu -E production -r 'role[base],role[mediawiki_database_master]' Instance ID: i-8157d9ef Flavor: m1.small Image: ami-7000f019 Availability Zone: us-east-1a Security Groups: default SSH Key: velocity-2011-aws Waiting for server............................... Public DNS Name: ec2-50-17-117-98.compute-1.amazonaws.com Public IP Address: 50.17.117.98 Private DNS Name: ip-10-245-87-117.ec2.internal Private IP Address: 10.245.87.117 Waiting for sshd....done Bootstrapping Chef on ec2-50-17-117-98.compute-1.amazonaws.com Tuesday, June 14, 2011 The knife ec2 server create command makes a call to the Amazon EC2 API through fog[0] and waits for SSH. There’s a lot here to type, so you can copy/paste out of the README-velocity.md. [0]: http://rubygems.org/gems/fog
  • 43. Anatomy of a Chef Run: Bootstrap Successfully installed mixlib-authentication-1.1.4 Successfully installed mime-types-1.16 Successfully installed rest-client-1.6.3 Successfully installed bunny-0.6.0 Successfully installed json-1.5.1 Successfully installed polyglot-0.3.1 Successfully installed treetop-1.4.9 Successfully installed net-ssh-2.1.4 Successfully installed net-ssh-gateway-1.1.0 Successfully installed net-ssh-multi-1.0.1 Successfully installed erubis-2.7.0 Successfully installed moneta-0.6.0 Successfully installed highline-1.6.2 Successfully installed uuidtools-2.1.2 Successfully installed chef-0.10.0 15 gems installed Tuesday, June 14, 2011 After the system is available in EC2 and SSH is up, the “bootstrap” process takes over. Chef is installed.
  • 44. Anatomy of a Chef Run: Validation ( cat <<'EOP' <%= validation_key %> EOP ) > /tmp/validation.pem awk NF /tmp/validation.pem > /etc/chef/validation.pem rm /tmp/validation.pem Tuesday, June 14, 2011 The bootstrap will write out the validation certificate from the local workstation to the target system.
  • 45. Anatomy of a Chef Run: Configuration ( cat <<'EOP' <%= config_content %> EOP ) > /etc/chef/client.rb Tuesday, June 14, 2011 The chef client configuration file is written based on values from the local system. The bootstrap is done from a template you can customize, so you can change the content in the EOP to whatever client.rb you want.
  • 46. /etc/chef/client.rb log_level :info log_location STDOUT chef_server_url "https://api.opscode.com/organizations/velocitydemo" validation_client_name "velocitydemo-validator" node_name "i-138c137d" Tuesday, June 14, 2011 For example, this is all it takes to configure the Chef Client on the new system.
  • 47. Anatomy of a Chef Run: Run List ( cat <<'EOP' <%= { "run_list" => @run_list }.to_json %> EOP ) > /etc/chef/first-boot.json Tuesday, June 14, 2011
  • 48. Anatomy of a Chef Run: chef-client chef-client -j /etc/chef/first-boot.json # run with debug output for full detail: chef-client -j /etc/chef/first-boot.json -l debug Tuesday, June 14, 2011 Normally we just run chef-client with info level log output. To get more detail, I ran it with debug. The -l debug option is available any time you want more detailed output from Chef.
  • 49. Anatomy of a Chef Run: Ohai! INFO: *** Chef 0.10.0 *** DEBUG: Loading plugin os DEBUG: Loading plugin kernel DEBUG: Loading plugin ruby DEBUG: Loading plugin languages DEBUG: Loading plugin hostname DEBUG: Loading plugin linux::hostname ... DEBUG: Loading plugin ec2 DEBUG: has_ec2_mac? == true DEBUG: can_metadata_connect? == true DEBUG: looks_like_ec2? == true DEBUG: Loading plugin rackspace ... DEBUG: Loading plugin cloud Tuesday, June 14, 2011 Chef runs ohai, the system profiling and data gathering tool. Ohai automatically detects a number of attributes about the system it is running on, including the kernel, operating system/platform, hostname and more.
  • 50. Run Ohai • Run `ohai | less` on your system. • Marvel at the amount of data it returns. Tuesday, June 14, 2011 You can run `ohai` on your local system with Chef installed to see what Chef discovers about it.
  • 51. Anatomy of a Chef Run: Authenticate INFO: Client key /etc/chef/client.pem is not present - registering DEBUG: Signing the request as velocitydemo-validator DEBUG: Sending HTTP Request via POST to api.opscode.com:443/ organizations/velocitydemo/clients DEBUG: Registration response: {"uri"=>"https:// api.opscode.com/organizations/velocitydemo/clients/ i-8157d9ef", "private_key"=>"SNIP!"} Tuesday, June 14, 2011 If /etc/chef/client.pem is not present, the validation client is used to register a new client automatically. The response comes back with the private key, which is written to /etc/chef/client.pem. All subsequent API requests to the server will use the newly created client, and the /etc/chef/validation.pem file can be deleted (we have chef- client::delete_validation for this). Yes, the client’s private key is displayed. Be mindful of this when pasting debug output. * http://tickets.opscode.com/browse/CHEF-2238
  • 52. Anatomy of a Chef Run: Build Node DEBUG: Building node object for i-8157d9ef DEBUG: Signing the request as i-8157d9ef DEBUG: Sending HTTP Request via GET to api.opscode.com:443/ organizations/velocitydemo/nodes/i-8157d9ef INFO: HTTP Request Returned 404 Not Found: Cannot load node i-8157d9ef DEBUG: Signing the request as i-8157d9ef DEBUG: Sending HTTP Request via POST to api.opscode.com:443/ organizations/velocitydemo/nodes DEBUG: Extracting run list from JSON attributes provided on command line INFO: Setting the run_list to ["role[base]", "role [mediawiki_database_master]"] from JSON DEBUG: Applying attributes from json file DEBUG: Platform is ubuntu version 10.04 Tuesday, June 14, 2011 We have 3 important pieces of information about building the node object at this point. First, the instance ID is used as the node name. This is automatically set up as the default node name by knife ec2 server create. Second, the JSON file passed into chef-client determines the run list of the node. Finally, during the ohai data gathering, it determined that the platform of the system is Ubuntu 10.04. This is important for how our resources will be configured by the underlying providers.
  • 53. Anatomy of a Chef Run: Sync Cookbooks INFO: Run List is [role[base], role [mediawiki_database_master]] INFO: Run List expands to [apt, zsh, users::sysadmins, sudo, git, build-essential, database::master] INFO: Starting Chef Run for i-8157d9ef DEBUG: Synchronizing cookbooks INFO: Loading cookbooks [apt, aws, build-essential, database, git, mysql, openssl, runit, sudo, users, xfs, zsh] Tuesday, June 14, 2011 Once the run list is determined, it is expanded to find all the recipes that will be applied. The names of the recipes indicate which cookbooks are required, and those cookbooks are downloaded. Cookbooks are like packages, so sometimes they depend on another which may not show up in the run list. Dependencies can be declared in cookbook metadata, similar to packaging system metadata for packages.
  • 54. Anatomy of a Chef Run: Load Cookbooks • Chef loads cookbook components after they are downloaded. • Libraries • Providers • Resources • Attributes • Definitions • Recipes Tuesday, June 14, 2011 Once all the cookbooks have been downloaded, Chef will load the Ruby components of the cookbook. This is done in the order above.
  • 55. Anatomy of a Chef Run: Load Recipes DEBUG: Loading Recipe zsh via include_recipe DEBUG: Found recipe default in cookbook zsh DEBUG: Loading Recipe users::sysadmins via include_recipe DEBUG: Found recipe sysadmins in cookbook users DEBUG: Sending HTTP Request via GET to api.opscode.com:443/ organizations/velocitydemo/search/users Tuesday, June 14, 2011 When recipes are loaded, the Ruby code they contain is evaluated. This is where things like search will hit the server API. We’ll see more of this later on. Chef is building what we call the “resource collection”, an ordered list of all the resources that should be configured on the node.
  • 56. Order Matters Tuesday, June 14, 2011 The order of the run list and the order of resources in recipes is important, because it matters how your systems are configured. A half configured system is a broken system, and a system configured out of order may be a broken system. Chef’s implicit ordering makes it easy to reason about the way systems are built, so you can identify and troubleshoot this easier.
  • 57. Anatomy of a Chef Run: Convergence user u['id'] do uid u['uid'] gid u['gid'] shell u['shell'] comment u['comment'] supports :manage_home => true home home_dir end directory "#{home_dir}/.ssh" do owner u['id'] group u['gid'] || u['id'] mode "0700" end template "#{home_dir}/.ssh/authorized_keys" do source "authorized_keys.erb" owner u['id'] group u['gid'] || u['id'] mode "0600" variables :ssh_keys => u['ssh_keys'] end Tuesday, June 14, 2011 For example, our users::sysadmins recipe creates some resources for each user it finds from the aforementioned search. These resources are added to the resource collection in the specified order. This is repeated for every user.
  • 58. Anatomy of a Chef Run: Convergence INFO: Processing user[velocity] action create (users::sysadmins line 41) INFO: Processing directory[/home/velocity/.ssh] action create (users::sysadmins line 51) INFO: Processing template[/home/velocity/.ssh/ authorized_keys] action create (users::sysadmins line 57) Tuesday, June 14, 2011 Convergence is the phase when the resources in the resource collection are configured. Providers take the appropriate action. Users are created, packages are installed, services are started and so on.
  • 59. Anatomy of a Chef Run: Save Node DEBUG: Saving the current state of node i-8157d9ef DEBUG: Signing the request as i-8157d9ef DEBUG: Sending HTTP Request via PUT to api.opscode.com:443/ organizations/velocitydemo/nodes/i-8157d9ef Tuesday, June 14, 2011 At the end of a run, the state of the node is saved, including all the attributes that were applied to the node from: * ohai * roles * cookbooks * environment This data is also indexed by the server for search.
  • 60. Anatomy of a Chef Run: Report Handlers INFO: Running report handlers INFO: Report handlers complete ... OR ... ERROR: Running exception handlers FATAL: Saving node information to /var/chef/cache/failed- run-data.json ERROR: Exception handlers complete FATAL: Stacktrace dumped to /var/chef/cache/chef- stacktrace.out FATAL: Some unhandled Ruby exception message here. Tuesday, June 14, 2011 At the end of the Chef run, report and exception handlers are executed. Report handlers are executed on a successful run. Exception handlers are executed on an unsuccessful run. * stack trace data and state of the failed run are also saved to files on the filesystem, and reported.
  • 61. I can haz cloud? http://www.flickr.com/photos/felixmorgner/4347750467/ Tuesday, June 14, 2011
  • 62. Configured systems are Nodes. http://www.flickr.com/photos/peterrosbjerg/3913766224/ Tuesday, June 14, 2011 Once a node is saved on the server, it is considered a managed system. In Chef, nodes do all the heavy lifting. All the above happens on the node, the server just handles API requests and serves data/cookbooks.
  • 63. knife node show % knife node show i-cda03aa3 Node Name: i-cda03aa3 Environment: production FQDN: ip-10-112-85-253.ec2.internal IP: 10.112.85.253 Run List: role[base], role[monitoring] Roles: monitoring, base Recipes apt, zsh, users::sysadmins, sudo, git, build- essential, nagios::client, nagios::server Platform: ubuntu 10.04 % knife node show i-cda03aa3 -m # non-automatic attributes % knife node show i-cda03aa3 -l # all attributes % knife node show i-cda03aa3 -Fj # JSON output Tuesday, June 14, 2011 We can show the nodes we have configured!
  • 64. Data Driven Tuesday, June 14, 2011 The deployment is data driven. Besides the data that came from the roles which we’re about to see, we also have arbitrary data about our infrastructure, namely the application we’re deploying and the users we’re creating. We didn’t have to write or modify any code to get a fully functional infrastructure.
  • 65. Writing Data Driven Cookbooks • Focus on primitives. • Apply the desired system state / behavior. • Don’t hardcode data. • Attributes • Data bags • Search Tuesday, June 14, 2011
  • 66. Data Driven Deployment data_bags ├── apps │   └── mediawiki.json └── users ├── nagiosadmin.json └── velocity.json Tuesday, June 14, 2011 We encapsulate all the information about our application, including environment-specific details. We also have two users we’re creating.
  • 67. Each Instance Has a Role roles ├── base.rb ├── mediawiki.rb Two app servers! ├── mediawiki_database_master.rb ├── mediawiki_load_balancer.rb └── monitoring.rb Tuesday, June 14, 2011
  • 68. All Your Base... Tuesday, June 14, 2011
  • 69. Base Role % knife role show base chef_type: role default_attributes: {} description: Base role applied to all nodes. env_run_lists: {} json_class: Chef::Role name: base override_attributes: authorization: sudo: passwordless: true users: ["ubuntu"] nagios: server_role: monitoring run_list: recipe[apt], recipe[zsh], recipe [users::sysadmins], recipe[sudo], recipe[git], recipe[build- essential] Tuesday, June 14, 2011 The base role is going to apply some settings that are common across the entire infrastructure. For example, apt ensures apt caches are updated, zsh installs the Z shell in case any users want it. Users::sysadmins creates all the system administrator users. Sudo sets up sudo permissions. Git ensures that our favorite version control system is installed. Build essential ensures that we can build our application, RubyGem native extensions, or other tools that should be installed by compilation.
  • 70. Packages vs Source Lean into it. Tuesday, June 14, 2011 The base role installs build-essential. You may opt to only have packages. Build your infrastructure the way you want :). We’re not going to have a holy war of packages vs source. Come to DevOpsDays Mountain View for a panel discussion on this topic.
  • 71. Nagios Server Tuesday, June 14, 2011 Every well built infrastructure needs monitoring. We’ve set up Nagios for our monitoring system. We could also add another tool such as munin to the mix if we wanted - there’s a munin cookbook that is data driven too.
  • 72. Nagios Server % knife role show monitoring chef_type: role default_attributes: nagios: server_auth_method: htauth description: Monitoring Server env_run_lists: {} json_class: Chef::Role name: monitoring override_attributes: {} run_list: recipe[nagios::server] Tuesday, June 14, 2011 We’ve modified the default behavior of the cookbook to enable htauth authentication.
  • 74. Load Balancer % knife role show mediawiki_load_balancer chef_type: role default_attributes: {} description: mediawiki load balancer env_run_lists: {} json_class: Chef::Role name: mediawiki_load_balancer override_attributes: haproxy: app_server_role: mediawiki run_list: recipe[haproxy::app_lb] Tuesday, June 14, 2011 We’re using haproxy, and we’ll search for a specific application to load balance. The recipe is written to search for the mediawiki role to find systems that should be pool members.
  • 75. MediaWiki App Servers (two) Tuesday, June 14, 2011 We actually have just the one system, we’ll add another one shortly :).
  • 76. MediaWiki App Servers % knife role show mediawiki chef_type: role default_attributes: {} description: mediawiki front end application server. env_run_lists: {} json_class: Chef::Role name: mediawiki override_attributes: {} run_list: recipe[mysql::client], recipe [application], recipe[mediawiki::status] Tuesday, June 14, 2011 The main thing in this role is the application recipe. The recipe will read in data from the data bag (in a predefined format) to determine what kind of application to deploy, the repository where it lives, details on where to put it, what roles to search for to find the database, and many more customizable properties. We launched two of these to have something to load balance :).
  • 77. Application Data Bag Item { "id": "mediawiki", "server_roles": [ "mediawiki" ], "type": { "mediawiki": [ "php", "mod_php_apache2" ] }, "database_master_role": [ "mediawiki_database_master" ], "repository": "git://github.com/mediawiki/mediawiki-trunk- phase3.git", "revision": { "production": "master", "staging": "master" }, ... Tuesday, June 14, 2011
  • 78. Database Master Tuesday, June 14, 2011 Every database backed application needs a master database. For this simple example we haven’t done any complex setup of master/slave replication, but the recipes are built such that this would be relatively easy to add.
  • 79. Database Master % knife role show mediawiki_database_master default_attributes: {} description: database master for the mediawiki application. env_run_lists: {} json_class: Chef::Role name: mediawiki_database_master override_attributes: {} run_list: recipe[database::master] Tuesday, June 14, 2011 The database master recipe will read the application information from the data bag and use it to create the database so the application can store its data.
  • 80. Cookbooks are easy to share. Tuesday, June 14, 2011 Chef is designed such that cookbooks are easy to share. Data is easy to separate from logic in recipes by using Attributes and Chef’s rich data discovery and look up features such as data bags.
  • 81. Data Driven Cookbooks • application & database • nagios • users http://www.flickr.com/photos/41176169@N00/2643328666/ Tuesday, June 14, 2011 Through data bag modification, role settings and Chef’s search feature, these cookbooks are data driven. No code was modified. You didn’t have to understand Ruby (though we think its a good idea :)), and you can deploy an infrastructure quickly and easily.
  • 82. Open Source Cookbooks knife cookbook site install nagios knife cookbook site install git knife cookbook site install application knife cookbook site install database knife cookbook site install haproxy knife cookbook site install sudo knife cookbook site install users knife cookbook site install zsh Tuesday, June 14, 2011 The cookbooks directory contains all the cookbooks we need. These do all kinds of things we didn’t have to write. These cookbooks all came from community.opscode.com
  • 83. Application-specific Cookbooks knife cookbook create mediawiki $EDITOR cookbooks/mediawiki/recipes/db_bootstrap.rb Tuesday, June 14, 2011 Your application probably doesn’t have a specific cookbook already shared by the community. We create our mediawiki cookbook for application specific purposes.
  • 84. mediawiki::db_bootstrap app = data_bag_item("apps", "mediawiki") dbm = search(:node, "role:mediawiki_database_master") db = app['databases'][node.chef_environment] execute "db_bootstrap" do command <<-EOH /usr/bin/mysql -u #{db['username']} -p#{db['password']} -h #{dbm['fqdn']} #{db['database']} < #{Chef::Config[:file_cache_path]}/schema.sql" EOH action :run end Tuesday, June 14, 2011 We retrieve some data up front. Then we use it to configure a resource.
  • 85. Systems Integration through Discovery. http://www.flickr.com/photos/c0t0s0d0/2425404674/ Tuesday, June 14, 2011 The systems we manage are running their own services to fullfill their purpose in the infrastructure. Each of those services is network accessible, and by expressing our systems through rich metadata, we can discover the systems that fullfill each role through searching the chef server.
  • 86. Search for Nodes with Knife % knife search node role:mediawiki_database_master 1 items found Node Name: i-8157d9ef Environment: production FQDN: ip-10-245-87-117.ec2.internal IP: 10.245.87.117 Run List: role[base], role[mediawiki_database_master] Roles: mediawiki_database_master, base Recipes apt, zsh, users::sysadmins, sudo, git, build- essential, database::master Platform: ubuntu 10.04 Tuesday, June 14, 2011
  • 87. Search for Nodes in Recipes results = search (:node, "role:mediawiki_database_master") template "/srv/mediawiki/shared/LocalSettings.php" do source "LocalSettings.erb" mode "644" variables( :path => "/srv/mediawiki/current", :host => results[0]['fqdn'] ) end Tuesday, June 14, 2011 You no longer need to track which system has an IP that should be applied as the database master. We can just use its fqdn from a search.
  • 88. Managing Infrastructure: Knife SSH % knife ssh 'role:mediawiki_database_master' 'sudo chef- client' -a ec2.public_hostname -x ubuntu ec2-50-17-117-98 INFO: *** Chef 0.10.0 *** ec2-50-17-117-98 INFO: Run List is [role[base], role [mediawiki_database_master]] ec2-50-17-117-98 INFO: Run List expands to [apt, zsh, users::sysadmins, sudo, git, build-essential, database::master] ec2-50-17-117-98 INFO: Starting Chef Run for i-8157d9ef ec2-50-17-117-98 INFO: Loading cookbooks [apt, aws, build- essential, database, git, mysql, openssl, runit, sudo, users, xfs, zsh] ec2-50-17-117-98 INFO: Chef Run complete in 9.471502 seconds ec2-50-17-117-98 INFO: Running report handlers ec2-50-17-117-98 INFO: Report handlers complete Tuesday, June 14, 2011
  • 89. What port is haproxy admin again? % knife ssh role:mediawiki_load_balancer -a ec2.public_hostname 'netstat -an | grep LISTEN' tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:22002 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:5666 0.0.0.0:* LISTEN tcp6 0 0 :::22 :::* LISTEN Tuesday, June 14, 2011 Oh that’s right. I always forget how many 2’s and 0’s.
  • 90. Managing Nodes through an API knife node run list add NODE "recipe[mediawiki::api_update]" knife exec -E 'nodes.transform("role:mediawiki") {|n| n.run_list << "recipe[mediawiki::api_update]"}' knife ssh 'role:mediawiki' -x velocity 'sudo chef-client' -a cloud.public_hostname Tuesday, June 14, 2011 We can programmatically add a recipe to the run list of all our nodes through the server API.
  • 91. Manage Infrastructure: Knife SSH • “SSH In a For Loop” is bad right? • Parallel command execution. • SSH is industry standard. • Use sudo NOPASSWD. Tuesday, June 14, 2011 “Best practice” suggests that ssh in a for loop is bad, because the prevailing idea is we’re doing “one-off” changes. We’re actually working toward parallel command execution. Kick off a chef-client run on a set of nodes, or gather some kind of command output. SSH is an industry standard that everyone understands and knows how to set up. A security best practice is to use sudo with NOPASSWD, which is e.g. how the Ubuntu AMIs are set up by Canonical.
  • 92. Wrap-up • Infrastructure as Code • Getting Started with Chef • Anatomy of a Chef Run • Data Driven Shareable Cookbooks • Managing Cloud Infrastructure http://www.flickr.com/photos/villes/358790270/ Tuesday, June 14, 2011 We’ve covered a lot of topics today! I’m sure you have questions...
  • 93. FAQ: Chef vs [Other Tool] Tuesday, June 14, 2011
  • 94. http://www.flickr.com/photos/gesika22/4458155541/ Tuesday, June 14, 2011 We can have that conversation over a pint :).
  • 95. FAQ: How do you test recipes? Tuesday, June 14, 2011
  • 96. FAQ: Testing • You launch cloud instances and watch them converge. • You use Vagrant with a Chef Provisioner Tuesday, June 14, 2011 We test recipes by running chef-client. Chef environments prevent recipe errors from affecting production. Or, you buy Stephen Nelson-Smith’s book!
  • 97. FAQ: Testing • You buy Stephen Nelson-Smith’s book! Tuesday, June 14, 2011
  • 98. FAQ: How does Chef scale? Tuesday, June 14, 2011
  • 99. FAQ: Scale • The Chef Server is a publishing system. • Nodes do the heavy lifting. • Chef scales like a service-oriented web application. • Opscode Hosted Chef was designed and built for massive scale. http://www.flickr.com/photos/amagill/61205408/ Tuesday, June 14, 2011
  • 100. Questions? • http://opscode.com • http://wiki.opscode.com • @opscode, #opschef • irc.freenode.net, #chef, #chef-hacking • http://lists.opscode.com • We’re in the exhibit hall this week. • We’ll be at DevOpsDays Mountain View. http://www.flickr.com/photos/oberazzi/318947873/ Tuesday, June 14, 2011
  • 101. Thanks! http://opscode.com @opscode #opschef Tuesday, June 14, 2011