0
Deploying Fully Automated, Highly AvailableDjango Application Stacks with Chef       schisamo@opscode.com          www.ops...
Who am I?
Who am I?•   Seth Chisamore    Senior Technical Consultant/Evangelist    Opscode, Inc.
Who am I?•   Seth Chisamore    Senior Technical Consultant/Evangelist    Opscode, Inc.•   Project lead for many Opscode sp...
Who are you? http://www.flickr.com/photos/timyates/2854357446/sizes/l/
Who are you?•   System administrators?                              http://www.flickr.com/photos/timyates/2854357446/sizes...
Who are you?•   System administrators?•   Developers?                              http://www.flickr.com/photos/timyates/2...
Who are you?•   System administrators?•   Developers?•   “Business” People?                              http://www.flickr...
What are we talking      about?     http://www.flickr.com/photos/peterkaminski/2174679908/
Agendahttp://www.flickr.com/photos/koalazymonkey/3590953001/
Agenda•   How’s and Why’s                      http://www.flickr.com/photos/koalazymonkey/3590953001/
Agenda•   How’s and Why’s•   Chef 101                      http://www.flickr.com/photos/koalazymonkey/3590953001/
Agenda•   How’s and Why’s•   Chef 101•   Deploying n-tier Django app stacks    with Chef                                  ...
Agenda•   How’s and Why’s•   Chef 101•   Deploying n-tier Django app stacks    with Chef•   Leveraging the same Chef code ...
Infrastructure as Code
A technical domainrevolving aroundbuilding andmanaginginfrastructureprogrammatically
Enable the reconstruction   of the business fromnothing but a source coderepository, an application  data backup, and bare...
ConfigurationManagement
ConfigurationManagement
System Integration     http://www.flickr.com/photos/opalsson/3773629074/
The Chef Framework   With thanks (and apologies) to Stephen Nelson-Smith
The Chef Framework
The Chef Framework•   Idempotent
The Chef Framework•   Idempotent•   Library & Primitives
The Chef Framework•   Idempotent•   Library & Primitives•   Flexibility
The Chef Framework•   Idempotent•   Library & Primitives•   Flexibility•   Reasonability
The Chef Framework•   Idempotent•   Library & Primitives•   Flexibility•   Reasonability•   TIMTOWTDI
Multiple applications of an operation do not  change the result
We start with APIs, you     supply data
Defaults are sane, but  easily changed
There is more than one     way to do it
The Chef Tool(s) With thanks (and apologies) to Stephen Nelson-Smith
The Chef Tool(s)
The Chef Tool(s)•   ohai
The Chef Tool(s)•   ohai•   chef-client
The Chef Tool(s)•   ohai•   chef-client•   knife
The Chef Tool(s)•   ohai•   chef-client•   knife•   shef
The Chef APIWith thanks (and apologies) to Stephen Nelson-Smith
The Chef API
The Chef API•   Fat Client/Thin Server
The Chef API•   Fat Client/Thin Server•   RESTful API w/ JSON
The Chef API•   Fat Client/Thin Server•   RESTful API w/ JSON•   Search Service
The Chef API•   Fat Client/Thin Server•   RESTful API w/ JSON•   Search Service•   Derivative Services (mash-up your    in...
The Chef Community   With thanks (and apologies) to Stephen Nelson-Smith
The Chef Community
The Chef Community•   Apache License, Version 2.0
The Chef Community•   Apache License, Version 2.0•   360+ Individual contributors
The Chef Community•   Apache License, Version 2.0•   360+ Individual contributors•   70+ Corporate contributors
The Chef Community•   Apache License, Version 2.0•   360+ Individual contributors•   70+ Corporate contributors    •   Del...
The Chef Community•   Apache License, Version 2.0•   360+ Individual contributors•   70+ Corporate contributors    •   Del...
The Chef Community•   Apache License, Version 2.0•   360+ Individual contributors•   70+ Corporate contributors    •   Del...
Chef client runs on your        systems
Clients talk to a Chef       server
Clients authenticate with        RSA keys
Each system youconfigure is a managed         node
Nodes have attributes{  "kernel": {    "machine": "x86_64",    "name": "Darwin",    "os": "Darwin",    "version": "Darwin ...
Nodes are Searchableknife search node ‘platform:mac_os_x’ search(:node, ‘platform:mac_os_x’)
Chef Enables Infrastructure as Code                       package "haproxy" do                        action :install     ...
Chef managesResources on Nodes
Chef Resourcespackage "haproxy" do action :installendtemplate "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner...
Chef Resources                   package "haproxy" do                    action :install                   end•   Have a t...
Chef Resources                   package "haproxy" do                    action :install                   end•   Have a t...
Chef Resources                       package "haproxy" do                        action :install                       end...
Chef Resources                                      package "haproxy" do                                       action :ins...
Chef Resources                                      package "haproxy" do                                       action :ins...
Resources take action through Providers
Chef Providers•   Multiple providers per resource type.•   For example the `package` resource    has providers for: Apt, Y...
Recipes are collections     of Resources
Chef Recipes                                    package "haproxy" do                                     action :install  ...
Chef Recipes•   Recipes can include other   include_recipe                                include_recipe                  ...
Chef Recipes%w{ python python-dev }.each do |pkg| package pkg do  action :install endendpool_members = search("node", "rol...
Chef Recipes                          %w{ python python-dev }.each do |pkg|                           package pkg do      ...
Chef Recipes                            %w{ python python-dev }.each do |pkg|                             package pkg do  ...
Cookbooks arepackages for recipes
Cookbook Metadata•   Cookbooks are like packages.•   Cookbooks have dependencies.
Cookbooks are easy to share.   community.opscode.com
Chef Rolesname "django_cms"description "django cms app server"run_list(  "recipe[mysql::client]",  "recipe[application]")n...
Chef Roles                            name "django_cms"                            description "django cms app server"    ...
Chef Roles                             name "django_cms"                             description "django cms app server"  ...
Chef Roles                                 name "django_cms"                                 description "django cms app s...
Data bags store arbitrary data
Data Bags% knife data bag show apps django_cmsdatabase_master_role: ["django_cms_database_master"]databases:  production: ...
Track it like source code...% git logcommit d640a8c6b370134d7043991894107d806595cc35Author: jtimberman <joshua@opscode.com...
Ponies ,(green )Unicorns and    n-tier oh my!
N-Tier FTW•   Provision
N-Tier FTW              Load BalancerApp Server                     App Server                                            ...
N-Tier FTW              Load Balancer                                            •   ProvisionApp Server                  ...
Chef Enables Systems Integration
Chef Enables Systems Integration•   Chef searches allow nodes to be aware of    each other and self configure
Chef Enables Systems Integration•   Chef searches allow nodes to be aware of    each other and self configure•   A shared ...
Chef Enables Systems Integration•   Chef searches allow nodes to be aware of    each other and self configure•   A shared ...
Chef Enables Systems Integration•   Chef searches allow nodes to be aware of    each other and self configure•   A shared ...
Database Master
Database Master% knife role show django_cms_database_masterdefault_attributes: {}description:       db master for the djan...
Database Master
Database Master•   Install and configure MySQL
Database Master•   Install and configure MySQL•   Create the application database
Database Master•   Install and configure MySQL•   Create the application database•   Grant the application user access
Django App Servers      (1..n)
Django App Servers% knife role show django_cmschef_type:         roledefault_attributes: {}description:       django_cms f...
Django App Servers
Django App Servers•   Deploy the Code
Django App Servers•   Deploy the Code    •   checkout the code with git
Django App Servers•   Deploy the Code    •   checkout the code with git    •   install required packages (os and python)
Django App Servers•   Deploy the Code    •   checkout the code with git    •   install required packages (os and python)  ...
Django App Servers•   Deploy the Code    •   checkout the code with git    •   install required packages (os and python)  ...
Django App Servers•   Deploy the Code    •   checkout the code with git    •   install required packages (os and python)  ...
Django App Servers•   Deploy the Code    •   checkout the code with git    •   install required packages (os and python)  ...
Search for Database Masterresults = search(:node, "role:django_cms_database_master")template "/srv/django_cms/shared/local...
Load Balancer
Load Balancer% knife role show django_cms_load_balancerchef_type:         roledefault_attributes: {}description:       Dja...
Load Balancer•   The balancing pool should grow as more    application servers are brought online
Search for App Serverspool_members = search(:node, "role:django_cms")template "/etc/haproxy/haproxy.cfg" do source "haprox...
One set of codez.... ...to rule them all
Same Code Different Configurations
Same Code Different Configurations•   We can deploy the same app stack on:
Same Code Different Configurations•   We can deploy the same app stack on:    •   a single node or a fully redundant      ...
Same Code Different Configurations•   We can deploy the same app stack on:    •   a single node or a fully redundant      ...
Same Code Different Configurations•   We can deploy the same app stack on:    •   a single node or a fully redundant      ...
Staging/QA - Single Node# launch stack in your data center% knife bootstrap 123.123.3.3   -r ‘role[django_cms_database_mas...
Production - Multi-Machine Cluster% knife ec2 server create   -r ‘role[django_cms_database_master]’% knife ec2 server crea...
Developer Workstation% cat VagrantfileVagrant::Config.run do |config| config.vm.box = base_box config.vm.provision :chef_server...
BONUS: deploy [insert   your favorite webframework here] stacks
Application Cookbook•   Supports the following web stacks:    •   Django    •   Java Webapps    •   Php    •   Rails    • ...
Further Resources & Closing•   www.opscode.com•   wiki.opscode.com•   @opscode, #opschef•   irc.freenode.net, #chef, #chef...
Questions? http://www.flickr.com/photos/oberazzi/318947873/
Thanks!   schisamo@opscode.com      www.opscode.com
SELF 2011: Deploying Django Application Stacks with Chef
Upcoming SlideShare
Loading in...5
×

SELF 2011: Deploying Django Application Stacks with Chef

3,816

Published on

Published in: Technology
0 Comments
7 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,816
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
83
Comments
0
Likes
7
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Managing infrastructure with Chef.\n
  • How&amp;#x2019;s and why&amp;#x2019;s of managing infrastructure with Chef.\nA quick overview of Chef\nWe&amp;#x2019;ll talk about the steps involved for automatically building our n-tier Django stack\nWe&amp;#x2019;ll discuss how your Chef code is reusable across environments\n\n
  • How&amp;#x2019;s and why&amp;#x2019;s of managing infrastructure with Chef.\nA quick overview of Chef\nWe&amp;#x2019;ll talk about the steps involved for automatically building our n-tier Django stack\nWe&amp;#x2019;ll discuss how your Chef code is reusable across environments\n\n
  • How&amp;#x2019;s and why&amp;#x2019;s of managing infrastructure with Chef.\nA quick overview of Chef\nWe&amp;#x2019;ll talk about the steps involved for automatically building our n-tier Django stack\nWe&amp;#x2019;ll discuss how your Chef code is reusable across environments\n\n
  • How&amp;#x2019;s and why&amp;#x2019;s of managing infrastructure with Chef.\nA quick overview of Chef\nWe&amp;#x2019;ll talk about the steps involved for automatically building our n-tier Django stack\nWe&amp;#x2019;ll discuss how your Chef code is reusable across environments\n\n
  • Why not just use some shell scripts?\n
  • \n
  • This is the dream. When everyone&amp;#x2019;s 2nd girlfriend &amp;#x2018;nagios&amp;#x2019; wakes you up in the middle of the night you are ready for the worst.\n
  • Keep track of all the steps required to take bare metal systems to doing their job in the infrastructure.\n\nIt is all about the policy. This can be lo-fi (ie a wiki page and the &amp;#x2018;meat&amp;#x2019; cloud) or fully automated with something like Chef.\n\nUsually this is the context of a single node.\n
  • Taking all the systems that have been configured to do their job, and make them work together to actually run the infrastructure.\n\nWRT Chef, we talk about Fully Automated Infrastructure.\n\nThis brings all the single nodes together.\n
  • Introducing Chef.\n\nMaybe you&amp;#x2019;ve already met!\n
  • Chef provides a framework for fully automating infrastructure, and has some important design principles.\n
  • Chef makes it easy to reason about your infrastructure, at scale. The declarative Ruby configuration language is easy to read, and the predictable ordering makes it easy to understand what&amp;#x2019;s going on.\n\nChef is flexible, and designed to allow you to build infrastructure. &amp;#x201C;Tim Toady&amp;#x201D;\n
  • Chef makes it easy to reason about your infrastructure, at scale. The declarative Ruby configuration language is easy to read, and the predictable ordering makes it easy to understand what&amp;#x2019;s going on.\n\nChef is flexible, and designed to allow you to build infrastructure. &amp;#x201C;Tim Toady&amp;#x201D;\n
  • Chef makes it easy to reason about your infrastructure, at scale. The declarative Ruby configuration language is easy to read, and the predictable ordering makes it easy to understand what&amp;#x2019;s going on.\n\nChef is flexible, and designed to allow you to build infrastructure. &amp;#x201C;Tim Toady&amp;#x201D;\n
  • Chef makes it easy to reason about your infrastructure, at scale. The declarative Ruby configuration language is easy to read, and the predictable ordering makes it easy to understand what&amp;#x2019;s going on.\n\nChef is flexible, and designed to allow you to build infrastructure. &amp;#x201C;Tim Toady&amp;#x201D;\n
  • Chef makes it easy to reason about your infrastructure, at scale. The declarative Ruby configuration language is easy to read, and the predictable ordering makes it easy to understand what&amp;#x2019;s going on.\n\nChef is flexible, and designed to allow you to build infrastructure. &amp;#x201C;Tim Toady&amp;#x201D;\n
  • Idempotent: Consistency, known system state, easy to build\nChef takes idempotent actions to put the system in the desired state.\n
  • Chef makes it easy to reason about your infrastructure.\n\nWe build an API, with primitives and libraries, and let you model your data. We don&amp;#x2019;t try to discover because we might make an incorrect assumption about the system.\n\nYou need an API to manage your infrastructure. The standard tools on Unix and Windows already use APIs to the system. Use an API to manage your infrastructure.\n
  • Chef comes from the perspective that the tool should do pretty much what you expect, hence sane defaults. To get there we had express some opinions. Those opinions can be changed easily, whether that is modifying the configuration of the chef-client, or how Apache is set up.\n
  • Chef is a flexible framework. You can make it do pretty much anything you want. Just like Perl doesn&amp;#x2019;t tell the programmer how to program, Chef doesn&amp;#x2019;t tell the system administrator how to manage the system. You can extend it, mold it and modify it to do what you need.\n\nBeing written in Ruby, using Ruby for the main language was very purposeful. The tool shouldn&amp;#x2019;t get in your way.\n
  • \n
  • -Chef is a library for building and managing infrastructure, and as such it happens to have some tools we wrote that help with that.\n-Ohai gathers data about your nodes.\n-Chef client runs on your nodes to configure them.\n-Knife is the CLI tool used to access the API.\n-Shef is an interactive console debugger.\n
  • -Chef is a library for building and managing infrastructure, and as such it happens to have some tools we wrote that help with that.\n-Ohai gathers data about your nodes.\n-Chef client runs on your nodes to configure them.\n-Knife is the CLI tool used to access the API.\n-Shef is an interactive console debugger.\n
  • -Chef is a library for building and managing infrastructure, and as such it happens to have some tools we wrote that help with that.\n-Ohai gathers data about your nodes.\n-Chef client runs on your nodes to configure them.\n-Knife is the CLI tool used to access the API.\n-Shef is an interactive console debugger.\n
  • -Chef is a library for building and managing infrastructure, and as such it happens to have some tools we wrote that help with that.\n-Ohai gathers data about your nodes.\n-Chef client runs on your nodes to configure them.\n-Knife is the CLI tool used to access the API.\n-Shef is an interactive console debugger.\n
  • \n
  • -thin server...fat clients =&gt; all execution happens down on the nodes...server is dumb REST/file server\n-easy to scale horizontally (like a stateless web app)\n
  • -thin server...fat clients =&gt; all execution happens down on the nodes...server is dumb REST/file server\n-easy to scale horizontally (like a stateless web app)\n
  • -thin server...fat clients =&gt; all execution happens down on the nodes...server is dumb REST/file server\n-easy to scale horizontally (like a stateless web app)\n
  • -thin server...fat clients =&gt; all execution happens down on the nodes...server is dumb REST/file server\n-easy to scale horizontally (like a stateless web app)\n
  • \n
  • Community is important.\n\nhttp://apache.org/licenses/LICENSE-2.0.html\nhttp://www.opscode.com/blog/2009/08/11/why-we-chose-the-apache-license/\nhttp://wiki.opscode.com/display/chef/How+to+Contribute\nhttp://wiki.opscode.com/display/chef/Approved+Contributors\n\n
  • Community is important.\n\nhttp://apache.org/licenses/LICENSE-2.0.html\nhttp://www.opscode.com/blog/2009/08/11/why-we-chose-the-apache-license/\nhttp://wiki.opscode.com/display/chef/How+to+Contribute\nhttp://wiki.opscode.com/display/chef/Approved+Contributors\n\n
  • Community is important.\n\nhttp://apache.org/licenses/LICENSE-2.0.html\nhttp://www.opscode.com/blog/2009/08/11/why-we-chose-the-apache-license/\nhttp://wiki.opscode.com/display/chef/How+to+Contribute\nhttp://wiki.opscode.com/display/chef/Approved+Contributors\n\n
  • Community is important.\n\nhttp://apache.org/licenses/LICENSE-2.0.html\nhttp://www.opscode.com/blog/2009/08/11/why-we-chose-the-apache-license/\nhttp://wiki.opscode.com/display/chef/How+to+Contribute\nhttp://wiki.opscode.com/display/chef/Approved+Contributors\n\n
  • Community is important.\n\nhttp://apache.org/licenses/LICENSE-2.0.html\nhttp://www.opscode.com/blog/2009/08/11/why-we-chose-the-apache-license/\nhttp://wiki.opscode.com/display/chef/How+to+Contribute\nhttp://wiki.opscode.com/display/chef/Approved+Contributors\n\n
  • Community is important.\n\nhttp://apache.org/licenses/LICENSE-2.0.html\nhttp://www.opscode.com/blog/2009/08/11/why-we-chose-the-apache-license/\nhttp://wiki.opscode.com/display/chef/How+to+Contribute\nhttp://wiki.opscode.com/display/chef/Approved+Contributors\n\n
  • \n
  • \n
  • Communication with the server is secure. Clients hold the private key, the server has the public key. The server is not a centralized store of authentication information, pushing the responsibility to the nodes.\n
  • \n
  • \n
  • \n
  • Manage system configuration as resources.\nPut resources together in recipes.\nDistribute recipes with cookbooks.\nAssign recipes to systems through roles.\nStore arbitrary data in data bags.\nTrack it all like source code.\n
  • Resources are an abstraction we feed data into. When you write recipes in Chef, you create resources of things you want to configure.\n
  • Resources declare a description of the state a part of the node should be in.\n
  • Resources declare a description of the state a part of the node should be in.\n
  • Resources declare a description of the state a part of the node should be in.\n
  • Resources declare a description of the state a part of the node should be in.\n
  • Resources declare a description of the state a part of the node should be in.\n
  • Resources declare a description of the state a part of the node should be in.\n
  • Resources declare a description of the state a part of the node should be in.\n
  • Resources declare a description of the state a part of the node should be in.\n
  • Resources declare a description of the state a part of the node should be in.\n
  • Resources declare a description of the state a part of the node should be in.\n
  • Resources declare a description of the state a part of the node should be in.\n
  • Resources declare a description of the state a part of the node should be in.\n
  • Resources declare a description of the state a part of the node should be in.\n
  • Resources declare a description of the state a part of the node should be in.\n
  • Resources declare a description of the state a part of the node should be in.\n
  • Resources declare a description of the state a part of the node should be in.\n
  • Resources declare a description of the state a part of the node should be in.\n
  • Resources declare a description of the state a part of the node should be in.\n
  • Resources declare a description of the state a part of the node should be in.\n
  • Resources declare a description of the state a part of the node should be in.\n
  • Resources declare a description of the state a part of the node should be in.\n
  • Resources declare a description of the state a part of the node should be in.\n
  • Resources declare a description of the state a part of the node should be in.\n
  • Resources declare a description of the state a part of the node should be in.\n
  • Resources declare a description of the state a part of the node should be in.\n
  • Resources declare a description of the state a part of the node should be in.\n
  • The abstraction over the commands or API calls that will configure the resource to be in the state you have defined.\n
  • Providers know how to actually configure the resources to be in the declared state\n\n
  • \n
  • \n
  • Just like recipes themselves are processed in order, the recipes included are processed in order, so when you include a recipe, all its resources are added to the resource collection, then Chef continues to the next.\n\n\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • -the cpan, PyPI or rubygems.org of the cookbbook world!\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Data bags are containers of grouped information.\n\nUsers, application information, network info, cabinet/rack locations. Describe components of your infrastructure with data, and use that data to configure systems.\n\nProcess data bags in recipes to configure systems dynamically.\n
  • \n
  • \n
  • In our example we are deploy Django CMS but this would apply to any Django application.\n
  • Provisioning is the first step. We need some computers on the internet. They&amp;#x2019;re going to be load balancers, webservers, database servers. Launch those with a cloud API. Every cloud does this.\n
  • You need to do something and you manage the configuration. Install packages, create users, etc. You can use pre-built images for this, but then just for a simple infrastructure (w/o monitoring/trending, middleware, etc) you have four discrete images to maintain for updates both for the OS and app updates.\n
  • This is the last mile. The systems are configured, but also need to be configured to talk to each other.\n
  • \n
  • \n
  • \n
  • \n
  • Every database backed application needs a master database. For this simple example we haven&amp;#x2019;t done any complex setup of master/slave replication, but the recipes are built such that this would be relatively easy to add.\n
  • The database master recipe will read the application information from the data bag and use it to create the database so the application can store its data.\n
  • all done fully automated\n
  • all done fully automated\n
  • all done fully automated\n
  • We start with one but scale horizontally as needed. We can even scale up and down based on demands.\n
  • The main thing in this role is the application recipe. \n\nThe recipe will read in data from the data bag (in a predefined format) to determine what kind of application to deploy, the repository where it lives, details on where to put it, what roles to search for to find the database, and many more customizable properties.\n\nWe launched two of these to have something to load balance :).\n
  • -acquire the code (remote file or scm)...create required config files (database.yml, local_settings.py etc.)\n-install the server software...create an application config for server\n
  • -acquire the code (remote file or scm)...create required config files (database.yml, local_settings.py etc.)\n-install the server software...create an application config for server\n
  • -acquire the code (remote file or scm)...create required config files (database.yml, local_settings.py etc.)\n-install the server software...create an application config for server\n
  • -acquire the code (remote file or scm)...create required config files (database.yml, local_settings.py etc.)\n-install the server software...create an application config for server\n
  • -acquire the code (remote file or scm)...create required config files (database.yml, local_settings.py etc.)\n-install the server software...create an application config for server\n
  • -acquire the code (remote file or scm)...create required config files (database.yml, local_settings.py etc.)\n-install the server software...create an application config for server\n
  • -acquire the code (remote file or scm)...create required config files (database.yml, local_settings.py etc.)\n-install the server software...create an application config for server\n
  • You no longer need to track which system has an IP that should be applied as the database master. We can just use its fqdn from a search.\n
  • \n
  • We&amp;#x2019;re using haproxy, and we&amp;#x2019;ll search for a specific application to load balance. The recipe is written to search for the django_cms role to find systems that should be pool members.\n
  • \n
  • -This search occurs everytime Chef runs on the load balancer.\n-If you add more app servers to the mix, the load balancer will *automatically* become aware!\n
  • A single set of Chef cookbooks, roles and databags allows you to deploy your application stack in numerous ways.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • https://github.com/opscode/cookbooks/tree/master/application\nhttp://www.opscode.com/blog/category/quickstarts/\n
  • Before we go to questions, here&amp;#x2019;s some resources to be aware of.\n
  • \n
  • \n
  • Transcript of "SELF 2011: Deploying Django Application Stacks with Chef"

    1. 1. Deploying Fully Automated, Highly AvailableDjango Application Stacks with Chef schisamo@opscode.com www.opscode.com
    2. 2. Who am I?
    3. 3. Who am I?• Seth Chisamore Senior Technical Consultant/Evangelist Opscode, Inc.
    4. 4. Who am I?• Seth Chisamore Senior Technical Consultant/Evangelist Opscode, Inc.• Project lead for many Opscode sponsored open source projects: • Opscode Cookbooks • Knife cloud provisioning plugins (ec2, rackspace & openstack) • Knife Windows plugin (knife-windows)
    5. 5. Who are you? http://www.flickr.com/photos/timyates/2854357446/sizes/l/
    6. 6. Who are you?• System administrators? http://www.flickr.com/photos/timyates/2854357446/sizes/l/
    7. 7. Who are you?• System administrators?• Developers? http://www.flickr.com/photos/timyates/2854357446/sizes/l/
    8. 8. Who are you?• System administrators?• Developers?• “Business” People? http://www.flickr.com/photos/timyates/2854357446/sizes/l/
    9. 9. What are we talking about? http://www.flickr.com/photos/peterkaminski/2174679908/
    10. 10. Agendahttp://www.flickr.com/photos/koalazymonkey/3590953001/
    11. 11. Agenda• How’s and Why’s http://www.flickr.com/photos/koalazymonkey/3590953001/
    12. 12. Agenda• How’s and Why’s• Chef 101 http://www.flickr.com/photos/koalazymonkey/3590953001/
    13. 13. Agenda• How’s and Why’s• Chef 101• Deploying n-tier Django app stacks with Chef http://www.flickr.com/photos/koalazymonkey/3590953001/
    14. 14. Agenda• How’s and Why’s• Chef 101• Deploying n-tier Django app stacks with Chef• Leveraging the same Chef code to deploy different configurations of our stack http://www.flickr.com/photos/koalazymonkey/3590953001/
    15. 15. Infrastructure as Code
    16. 16. A technical domainrevolving aroundbuilding andmanaginginfrastructureprogrammatically
    17. 17. Enable the reconstruction of the business fromnothing but a source coderepository, an application data backup, and bare metal resources.
    18. 18. ConfigurationManagement
    19. 19. ConfigurationManagement
    20. 20. System Integration http://www.flickr.com/photos/opalsson/3773629074/
    21. 21. The Chef Framework With thanks (and apologies) to Stephen Nelson-Smith
    22. 22. The Chef Framework
    23. 23. The Chef Framework• Idempotent
    24. 24. The Chef Framework• Idempotent• Library & Primitives
    25. 25. The Chef Framework• Idempotent• Library & Primitives• Flexibility
    26. 26. The Chef Framework• Idempotent• Library & Primitives• Flexibility• Reasonability
    27. 27. The Chef Framework• Idempotent• Library & Primitives• Flexibility• Reasonability• TIMTOWTDI
    28. 28. Multiple applications of an operation do not change the result
    29. 29. We start with APIs, you supply data
    30. 30. Defaults are sane, but easily changed
    31. 31. There is more than one way to do it
    32. 32. The Chef Tool(s) With thanks (and apologies) to Stephen Nelson-Smith
    33. 33. The Chef Tool(s)
    34. 34. The Chef Tool(s)• ohai
    35. 35. The Chef Tool(s)• ohai• chef-client
    36. 36. The Chef Tool(s)• ohai• chef-client• knife
    37. 37. The Chef Tool(s)• ohai• chef-client• knife• shef
    38. 38. The Chef APIWith thanks (and apologies) to Stephen Nelson-Smith
    39. 39. The Chef API
    40. 40. The Chef API• Fat Client/Thin Server
    41. 41. The Chef API• Fat Client/Thin Server• RESTful API w/ JSON
    42. 42. The Chef API• Fat Client/Thin Server• RESTful API w/ JSON• Search Service
    43. 43. The Chef API• Fat Client/Thin Server• RESTful API w/ JSON• Search Service• Derivative Services (mash-up your infrastructure)
    44. 44. The Chef Community With thanks (and apologies) to Stephen Nelson-Smith
    45. 45. The Chef Community
    46. 46. The Chef Community• Apache License, Version 2.0
    47. 47. The Chef Community• Apache License, Version 2.0• 360+ Individual contributors
    48. 48. The Chef Community• Apache License, Version 2.0• 360+ Individual contributors• 70+ Corporate contributors
    49. 49. The Chef Community• Apache License, Version 2.0• 360+ Individual contributors• 70+ Corporate contributors • Dell, Rackspace,VMware, RightScale, Heroku, and more
    50. 50. The Chef Community• Apache License, Version 2.0• 360+ Individual contributors• 70+ Corporate contributors • Dell, Rackspace,VMware, RightScale, Heroku, and more• 240+ cookbooks
    51. 51. The Chef Community• Apache License, Version 2.0• 360+ Individual contributors• 70+ Corporate contributors • Dell, Rackspace,VMware, RightScale, Heroku, and more• 240+ cookbooks• http://community.opscode.com
    52. 52. Chef client runs on your systems
    53. 53. Clients talk to a Chef server
    54. 54. Clients authenticate with RSA keys
    55. 55. Each system youconfigure is a managed node
    56. 56. Nodes have attributes{ "kernel": { "machine": "x86_64", "name": "Darwin", "os": "Darwin", "version": "Darwin Kernel Version 10.4.0: Fri Apr 23 18:28:53 PDT 2010;root:xnu-1504.7.4~1/RELEASE_I386", "release": "10.4.0" }, "platform_version": "10.6.4", "platform": "mac_os_x", "platform_build": "10F569", "domain": "local", "os": "darwin", "current_user": "schisamo", "ohai_time": 1278602661.60043, "os_version": "10.4.0", "uptime": "18 days 17 hours 49 minutes 18 seconds", "ipaddress": "10.13.37.116", "hostname": "helsinki", "fqdn": "helsinki.local", "uptime_seconds": 1619358}
    57. 57. Nodes are Searchableknife search node ‘platform:mac_os_x’ search(:node, ‘platform:mac_os_x’)
    58. 58. Chef Enables Infrastructure as Code package "haproxy" do action :install end• Resources template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb"• Recipes owner "root" group "root"• Cookbooks mode 0644• notifies :restart, "service[haproxy]" Roles end• Data Bags service "haproxy" do• Source Code supports :restart => true action [:enable, :start] end
    59. 59. Chef managesResources on Nodes
    60. 60. Chef Resourcespackage "haproxy" do action :installendtemplate "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "root" group "root" mode 0644 notifies :restart, "service[haproxy]"endservice "haproxy" do supports :restart => true action [:enable, :start]end
    61. 61. Chef Resources package "haproxy" do action :install end• Have a type. template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "root" group "root" mode 0644 notifies :restart, "service[haproxy]" end service "haproxy" do supports :restart => true action [:enable, :start] end
    62. 62. Chef Resources package "haproxy" do action :install end• Have a type. template "/etc/haproxy/haproxy.cfg" do• source "haproxy.cfg.erb" Have a name. owner "root" group "root" mode 0644 notifies :restart, "service[haproxy]" end service "haproxy" do supports :restart => true action [:enable, :start] end
    63. 63. Chef Resources package "haproxy" do action :install end• Have a type. template "/etc/haproxy/haproxy.cfg" do• source "haproxy.cfg.erb" Have a name. owner "root"• Have parameters. group "root" mode 0644 notifies :restart, "service[haproxy]" end service "haproxy" do supports :restart => true action [:enable, :start] end
    64. 64. Chef Resources package "haproxy" do action :install end• Have a type. template "/etc/haproxy/haproxy.cfg" do• source "haproxy.cfg.erb" Have a name. owner "root"• Have parameters. group "root" mode 0644• Take action to put the resource notifies :restart, "service[haproxy]" end in the declared state. service "haproxy" do supports :restart => true action [:enable, :start] end
    65. 65. Chef Resources package "haproxy" do action :install end• Have a type. template "/etc/haproxy/haproxy.cfg" do• source "haproxy.cfg.erb" Have a name. owner "root"• Have parameters. group "root" mode 0644• Take action to put the resource notifies :restart, "service[haproxy]" end in the declared state.• Can send notifications to other service "haproxy" do supports :restart => true resources. action [:enable, :start] end
    66. 66. Resources take action through Providers
    67. 67. Chef Providers• Multiple providers per resource type.• For example the `package` resource has providers for: Apt, Yum, Rubygems, Portage, Macports, FreeBSD Ports, etc.• If you are developer think of it as an interface and an implementation.
    68. 68. Recipes are collections of Resources
    69. 69. Chef Recipes package "haproxy" do action :install end template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb"• Recipes are evaluated for owner "root" resources in the order they group "root" mode 0644 appear. notifies :restart, "service[haproxy]"• Each resource object is added end to the Resource Collection. service "haproxy" do supports :restart => true action [:enable, :start] end
    70. 70. Chef Recipes• Recipes can include other include_recipe include_recipe "apache2" "apache2::mod_rewrite" recipes. include_recipe "apache2::mod_deflate"• Included recipes are include_recipe include_recipe "apache2::mod_headers" "apache2::mod_php5" processed in order.
    71. 71. Chef Recipes%w{ python python-dev }.each do |pkg| package pkg do action :install endendpool_members = search("node", "role:django_cms")template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members notifies :restart, "service[haproxy]"end
    72. 72. Chef Recipes %w{ python python-dev }.each do |pkg| package pkg do action :install end end• Extend recipes with pool_members = search("node", "role:django_cms") Ruby. template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members notifies :restart, "service[haproxy]" end
    73. 73. Chef Recipes %w{ python python-dev }.each do |pkg| package pkg do action :install end end• Extend recipes with pool_members = search("node", "role:django_cms") Ruby.• Dynamic configuration template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" through search. owner "root" group "root" mode 0644 variables :pool_members => pool_members notifies :restart, "service[haproxy]" end
    74. 74. Cookbooks arepackages for recipes
    75. 75. Cookbook Metadata• Cookbooks are like packages.• Cookbooks have dependencies.
    76. 76. Cookbooks are easy to share. community.opscode.com
    77. 77. Chef Rolesname "django_cms"description "django cms app server"run_list( "recipe[mysql::client]", "recipe[application]")name "django_cms_load_balancer"description "Django CMS load balancer"run_list( "recipe[haproxy::app_lb]")override_attributes( "haproxy" => { "app_server_role" => "django_cms" })
    78. 78. Chef Roles name "django_cms" description "django cms app server" run_list( "recipe[mysql::client]", "recipe[application]" )• Roles describe nodes. name "django_cms_load_balancer" description "Django CMS load balancer" run_list( "recipe[haproxy::app_lb]" ) override_attributes( "haproxy" => { "app_server_role" => "django_cms" } )
    79. 79. Chef Roles name "django_cms" description "django cms app server" run_list( "recipe[mysql::client]", "recipe[application]" )• Roles describe nodes. name "django_cms_load_balancer"• Roles have a run list. description "Django CMS load balancer" run_list( "recipe[haproxy::app_lb]" ) override_attributes( "haproxy" => { "app_server_role" => "django_cms" } )
    80. 80. Chef Roles name "django_cms" description "django cms app server" run_list( "recipe[mysql::client]", "recipe[application]" )• Roles describe nodes. name "django_cms_load_balancer"• Roles have a run list. description "Django CMS load balancer" run_list(• Roles can have attributes. ) "recipe[haproxy::app_lb]" override_attributes( "haproxy" => { "app_server_role" => "django_cms" } )
    81. 81. Data bags store arbitrary data
    82. 82. Data Bags% knife data bag show apps django_cmsdatabase_master_role: ["django_cms_database_master"]databases: production: adapter: mysql database: django_cms_production encoding: utf8 password: awesome_password port: 3306 username: django_cmsdeploy_to: /srv/django_cmsgroup: nogroupid: django_cmslocal_settings_file: local_settings.pymysql_root_password: production: mysql_rootowner: nobodypackages: curl:pips: django: 1.2.5repository: https://github.com/opscode/django-quick-start-cms.gitrevision: production: masterserver_roles: ["django_cms"]type: django_cms: django gunicorn
    83. 83. Track it like source code...% git logcommit d640a8c6b370134d7043991894107d806595cc35Author: jtimberman <joshua@opscode.com> Import nagios version 1.0.0commit c40c818498710e78cf73c7f71e722e971fa574e7Author: jtimberman <joshua@opscode.com> installation and usage instruction docscommit 99d0efb024314de17888f6b359c14414fda7bb91Author: jtimberman <joshua@opscode.com> Import haproxy version 1.0.1commit c89d0975ad3f4b152426df219fee0bfb8eafb7e4Author: jtimberman <joshua@opscode.com> add mediawiki cookbookcommit 89c0545cc03b9be26f1db246c9ba4ce9d58a6700Author: jtimberman <joshua@opscode.com> multiple environments in data bag for mediawiki
    84. 84. Ponies ,(green )Unicorns and n-tier oh my!
    85. 85. N-Tier FTW• Provision
    86. 86. N-Tier FTW Load BalancerApp Server App Server • Provision • Configure Database Master
    87. 87. N-Tier FTW Load Balancer • ProvisionApp Server App Server • Configure • Integrate Database Master
    88. 88. Chef Enables Systems Integration
    89. 89. Chef Enables Systems Integration• Chef searches allow nodes to be aware of each other and self configure
    90. 90. Chef Enables Systems Integration• Chef searches allow nodes to be aware of each other and self configure• A shared application data bag allows:
    91. 91. Chef Enables Systems Integration• Chef searches allow nodes to be aware of each other and self configure• A shared application data bag allows: • many app servers to checkout the same application code during initial bootstrapping
    92. 92. Chef Enables Systems Integration• Chef searches allow nodes to be aware of each other and self configure• A shared application data bag allows: • many app servers to checkout the same application code during initial bootstrapping • app server and database servers to initialize off of the same database config
    93. 93. Database Master
    94. 94. Database Master% knife role show django_cms_database_masterdefault_attributes: {}description: db master for the django_cms app.env_run_lists: {}json_class: Chef::Rolename: django_cms_database_masteroverride_attributes: {}run_list: recipe[database::master]
    95. 95. Database Master
    96. 96. Database Master• Install and configure MySQL
    97. 97. Database Master• Install and configure MySQL• Create the application database
    98. 98. Database Master• Install and configure MySQL• Create the application database• Grant the application user access
    99. 99. Django App Servers (1..n)
    100. 100. Django App Servers% knife role show django_cmschef_type: roledefault_attributes: {}description: django_cms front end application server.env_run_lists: {}json_class: Chef::Rolename: django_cmsoverride_attributes: {}run_list: recipe[mysql::client], recipe[application]
    101. 101. Django App Servers
    102. 102. Django App Servers• Deploy the Code
    103. 103. Django App Servers• Deploy the Code • checkout the code with git
    104. 104. Django App Servers• Deploy the Code • checkout the code with git • install required packages (os and python)
    105. 105. Django App Servers• Deploy the Code • checkout the code with git • install required packages (os and python) • create environment specific config files
    106. 106. Django App Servers• Deploy the Code • checkout the code with git • install required packages (os and python) • create environment specific config files• Serve the Code
    107. 107. Django App Servers• Deploy the Code • checkout the code with git • install required packages (os and python) • create environment specific config files• Serve the Code • install Gunicorn (or apache2/mod_wsgi)
    108. 108. Django App Servers• Deploy the Code • checkout the code with git • install required packages (os and python) • create environment specific config files• Serve the Code • install Gunicorn (or apache2/mod_wsgi) • create an app specific config file (vhost)
    109. 109. Search for Database Masterresults = search(:node, "role:django_cms_database_master")template "/srv/django_cms/shared/local_settings.py" do source "settings.py.erb" mode "644" variables( :host => results[0][fqdn] )end
    110. 110. Load Balancer
    111. 111. Load Balancer% knife role show django_cms_load_balancerchef_type: roledefault_attributes: {}description: Django CMS load balancerenv_run_lists: {}json_class: Chef::Rolename: django_cms_load_balanceroverride_attributes: haproxy: app_server_role: django_cmsrun_list: recipe[haproxy::app_lb]
    112. 112. Load Balancer• The balancing pool should grow as more application servers are brought online
    113. 113. Search for App Serverspool_members = search(:node, "role:django_cms")template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members notifies :restart, "service[haproxy]"end
    114. 114. One set of codez.... ...to rule them all
    115. 115. Same Code Different Configurations
    116. 116. Same Code Different Configurations• We can deploy the same app stack on:
    117. 117. Same Code Different Configurations• We can deploy the same app stack on: • a single node or a fully redundant cluster of nodes.
    118. 118. Same Code Different Configurations• We can deploy the same app stack on: • a single node or a fully redundant cluster of nodes. • physical hardware or any number of public/private cloud providers.
    119. 119. Same Code Different Configurations• We can deploy the same app stack on: • a single node or a fully redundant cluster of nodes. • physical hardware or any number of public/private cloud providers. • a developer workstation fully virtualized with Vagrant + VirtualBox
    120. 120. Staging/QA - Single Node# launch stack in your data center% knife bootstrap 123.123.3.3 -r ‘role[django_cms_database_master]’, ‘role[django_cms]’, ‘role[django_cms_load_balancer]’# launch stack in EC2% knife ec2 server create -r ‘role[django_cms_database_master]’, ‘role[django_cms]’, ‘role[django_cms_load_balancer]’# launch stack in Rackspace% knife rackspace server create -r ‘role[django_cms_database_master]’, ‘role[django_cms]’, ‘role[django_cms_load_balancer]’
    121. 121. Production - Multi-Machine Cluster% knife ec2 server create -r ‘role[django_cms_database_master]’% knife ec2 server create -r ‘role[django_cms]’% knife ec2 server create -r ‘role[django_cms]’% knife ec2 server create -r ‘role[django_cms_load_balancer]’
    122. 122. Developer Workstation% cat VagrantfileVagrant::Config.run do |config| config.vm.box = base_box config.vm.provision :chef_server do |chef| chef.chef_server_url = "https://api.opscode.com/organizations/foo" chef.validation_key_path = "~/.chef/private-pems/foo-validator.pem" chef.validation_client_name = "#{ENV[ORGNAME]}-validator" chef.provisioning_path = "/etc/chef" chef.run_list = [ role[django_cms_database_master], role[django_cms], role[django_cms_load_balancer] ] endend% vagrant up
    123. 123. BONUS: deploy [insert your favorite webframework here] stacks
    124. 124. Application Cookbook• Supports the following web stacks: • Django • Java Webapps • Php • Rails • Node.js (coming soon)• Opscode contributed quick starts walk you through example deployments
    125. 125. Further Resources & Closing• www.opscode.com• wiki.opscode.com• @opscode, #opschef• irc.freenode.net, #chef, #chef-hacking• lists.opscode.com
    126. 126. Questions? http://www.flickr.com/photos/oberazzi/318947873/
    127. 127. Thanks! schisamo@opscode.com www.opscode.com
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×