SELF 2011: Deploying Django Application Stacks with Chef

Deploying Fully Automated, Highly AvailableDjango Application Stacks with Chef schisamo@opscode.com www.opscode.com

Who am I?

Who am I?• Seth Chisamore Senior Technical Consultant/Evangelist Opscode, Inc.

Who am I?• Seth Chisamore Senior Technical Consultant/Evangelist Opscode, Inc.• Project lead for many Opscode sponsored open source projects: • Opscode Cookbooks • Knife cloud provisioning plugins (ec2, rackspace & openstack) • Knife Windows plugin (knife-windows)

Who are you? http://www.flickr.com/photos/timyates/2854357446/sizes/l/

Who are you?• System administrators? http://www.flickr.com/photos/timyates/2854357446/sizes/l/

Who are you?• System administrators?• Developers? http://www.flickr.com/photos/timyates/2854357446/sizes/l/

Who are you?• System administrators?• Developers?• “Business” People? http://www.flickr.com/photos/timyates/2854357446/sizes/l/

What are we talking about? http://www.flickr.com/photos/peterkaminski/2174679908/

Agendahttp://www.flickr.com/photos/koalazymonkey/3590953001/

Agenda• How’s and Why’s http://www.flickr.com/photos/koalazymonkey/3590953001/

Agenda• How’s and Why’s• Chef 101 http://www.flickr.com/photos/koalazymonkey/3590953001/

Agenda• How’s and Why’s• Chef 101• Deploying n-tier Django app stacks with Chef http://www.flickr.com/photos/koalazymonkey/3590953001/

Agenda• How’s and Why’s• Chef 101• Deploying n-tier Django app stacks with Chef• Leveraging the same Chef code to deploy different configurations of our stack http://www.flickr.com/photos/koalazymonkey/3590953001/

Infrastructure as Code

A technical domainrevolving aroundbuilding andmanaginginfrastructureprogrammatically

Enable the reconstruction of the business fromnothing but a source coderepository, an application data backup, and bare metal resources.

ConfigurationManagement

System Integration http://www.ﬂickr.com/photos/opalsson/3773629074/

The Chef Framework With thanks (and apologies) to Stephen Nelson-Smith

The Chef Framework

The Chef Framework• Idempotent

The Chef Framework• Idempotent• Library & Primitives

The Chef Framework• Idempotent• Library & Primitives• Flexibility

The Chef Framework• Idempotent• Library & Primitives• Flexibility• Reasonability

The Chef Framework• Idempotent• Library & Primitives• Flexibility• Reasonability• TIMTOWTDI

Multiple applications of an operation do not change the result

We start with APIs, you supply data

Defaults are sane, but easily changed

There is more than one way to do it

The Chef Tool(s) With thanks (and apologies) to Stephen Nelson-Smith

The Chef Tool(s)

The Chef Tool(s)• ohai

The Chef Tool(s)• ohai• chef-client

The Chef Tool(s)• ohai• chef-client• knife

The Chef Tool(s)• ohai• chef-client• knife• shef

The Chef APIWith thanks (and apologies) to Stephen Nelson-Smith

The Chef API

The Chef API• Fat Client/Thin Server

The Chef API• Fat Client/Thin Server• RESTful API w/ JSON

The Chef API• Fat Client/Thin Server• RESTful API w/ JSON• Search Service

The Chef API• Fat Client/Thin Server• RESTful API w/ JSON• Search Service• Derivative Services (mash-up your infrastructure)

The Chef Community With thanks (and apologies) to Stephen Nelson-Smith

The Chef Community

The Chef Community• Apache License, Version 2.0

The Chef Community• Apache License, Version 2.0• 360+ Individual contributors

The Chef Community• Apache License, Version 2.0• 360+ Individual contributors• 70+ Corporate contributors

The Chef Community• Apache License, Version 2.0• 360+ Individual contributors• 70+ Corporate contributors • Dell, Rackspace,VMware, RightScale, Heroku, and more

The Chef Community• Apache License, Version 2.0• 360+ Individual contributors• 70+ Corporate contributors • Dell, Rackspace,VMware, RightScale, Heroku, and more• 240+ cookbooks

The Chef Community• Apache License, Version 2.0• 360+ Individual contributors• 70+ Corporate contributors • Dell, Rackspace,VMware, RightScale, Heroku, and more• 240+ cookbooks• http://community.opscode.com

Chef client runs on your systems

Clients talk to a Chef server

Clients authenticate with RSA keys

Each system youconfigure is a managed node

Nodes have attributes{ "kernel": { "machine": "x86_64", "name": "Darwin", "os": "Darwin", "version": "Darwin Kernel Version 10.4.0: Fri Apr 23 18:28:53 PDT 2010;root:xnu-1504.7.4~1/RELEASE_I386", "release": "10.4.0" }, "platform_version": "10.6.4", "platform": "mac_os_x", "platform_build": "10F569", "domain": "local", "os": "darwin", "current_user": "schisamo", "ohai_time": 1278602661.60043, "os_version": "10.4.0", "uptime": "18 days 17 hours 49 minutes 18 seconds", "ipaddress": "10.13.37.116", "hostname": "helsinki", "fqdn": "helsinki.local", "uptime_seconds": 1619358}

Nodes are Searchableknife search node ‘platform:mac_os_x’ search(:node, ‘platform:mac_os_x’)

Chef Enables Infrastructure as Code package "haproxy" do action :install end• Resources template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb"• Recipes owner "root" group "root"• Cookbooks mode 0644• notiﬁes :restart, "service[haproxy]" Roles end• Data Bags service "haproxy" do• Source Code supports :restart => true action [:enable, :start] end

Chef managesResources on Nodes

Chef Resourcespackage "haproxy" do action :installendtemplate "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "root" group "root" mode 0644 notiﬁes :restart, "service[haproxy]"endservice "haproxy" do supports :restart => true action [:enable, :start]end

Chef Resources package "haproxy" do action :install end• Have a type. template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "root" group "root" mode 0644 notiﬁes :restart, "service[haproxy]" end service "haproxy" do supports :restart => true action [:enable, :start] end

Chef Resources package "haproxy" do action :install end• Have a type. template "/etc/haproxy/haproxy.cfg" do• source "haproxy.cfg.erb" Have a name. owner "root" group "root" mode 0644 notiﬁes :restart, "service[haproxy]" end service "haproxy" do supports :restart => true action [:enable, :start] end

Chef Resources package "haproxy" do action :install end• Have a type. template "/etc/haproxy/haproxy.cfg" do• source "haproxy.cfg.erb" Have a name. owner "root"• Have parameters. group "root" mode 0644 notiﬁes :restart, "service[haproxy]" end service "haproxy" do supports :restart => true action [:enable, :start] end

Chef Resources package "haproxy" do action :install end• Have a type. template "/etc/haproxy/haproxy.cfg" do• source "haproxy.cfg.erb" Have a name. owner "root"• Have parameters. group "root" mode 0644• Take action to put the resource notiﬁes :restart, "service[haproxy]" end in the declared state. service "haproxy" do supports :restart => true action [:enable, :start] end

Chef Resources package "haproxy" do action :install end• Have a type. template "/etc/haproxy/haproxy.cfg" do• source "haproxy.cfg.erb" Have a name. owner "root"• Have parameters. group "root" mode 0644• Take action to put the resource notiﬁes :restart, "service[haproxy]" end in the declared state.• Can send notifications to other service "haproxy" do supports :restart => true resources. action [:enable, :start] end

Resources take action through Providers

Chef Providers• Multiple providers per resource type.• For example the `package` resource has providers for: Apt, Yum, Rubygems, Portage, Macports, FreeBSD Ports, etc.• If you are developer think of it as an interface and an implementation.

Recipes are collections of Resources

Chef Recipes package "haproxy" do action :install end template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb"• Recipes are evaluated for owner "root" resources in the order they group "root" mode 0644 appear. notiﬁes :restart, "service[haproxy]"• Each resource object is added end to the Resource Collection. service "haproxy" do supports :restart => true action [:enable, :start] end

Chef Recipes• Recipes can include other include_recipe include_recipe "apache2" "apache2::mod_rewrite" recipes. include_recipe "apache2::mod_deﬂate"• Included recipes are include_recipe include_recipe "apache2::mod_headers" "apache2::mod_php5" processed in order.

Chef Recipes%w{ python python-dev }.each do |pkg| package pkg do action :install endendpool_members = search("node", "role:django_cms")template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members notiﬁes :restart, "service[haproxy]"end

Chef Recipes %w{ python python-dev }.each do |pkg| package pkg do action :install end end• Extend recipes with pool_members = search("node", "role:django_cms") Ruby. template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members notiﬁes :restart, "service[haproxy]" end

Chef Recipes %w{ python python-dev }.each do |pkg| package pkg do action :install end end• Extend recipes with pool_members = search("node", "role:django_cms") Ruby.• Dynamic configuration template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" through search. owner "root" group "root" mode 0644 variables :pool_members => pool_members notiﬁes :restart, "service[haproxy]" end

Cookbooks arepackages for recipes

Cookbook Metadata• Cookbooks are like packages.• Cookbooks have dependencies.

Cookbooks are easy to share. community.opscode.com

Chef Rolesname "django_cms"description "django cms app server"run_list( "recipe[mysql::client]", "recipe[application]")name "django_cms_load_balancer"description "Django CMS load balancer"run_list( "recipe[haproxy::app_lb]")override_attributes( "haproxy" => { "app_server_role" => "django_cms" })

Chef Roles name "django_cms" description "django cms app server" run_list( "recipe[mysql::client]", "recipe[application]" )• Roles describe nodes. name "django_cms_load_balancer" description "Django CMS load balancer" run_list( "recipe[haproxy::app_lb]" ) override_attributes( "haproxy" => { "app_server_role" => "django_cms" } )

Chef Roles name "django_cms" description "django cms app server" run_list( "recipe[mysql::client]", "recipe[application]" )• Roles describe nodes. name "django_cms_load_balancer"• Roles have a run list. description "Django CMS load balancer" run_list( "recipe[haproxy::app_lb]" ) override_attributes( "haproxy" => { "app_server_role" => "django_cms" } )

Chef Roles name "django_cms" description "django cms app server" run_list( "recipe[mysql::client]", "recipe[application]" )• Roles describe nodes. name "django_cms_load_balancer"• Roles have a run list. description "Django CMS load balancer" run_list(• Roles can have attributes. ) "recipe[haproxy::app_lb]" override_attributes( "haproxy" => { "app_server_role" => "django_cms" } )

Data bags store arbitrary data

Data Bags% knife data bag show apps django_cmsdatabase_master_role: ["django_cms_database_master"]databases: production: adapter: mysql database: django_cms_production encoding: utf8 password: awesome_password port: 3306 username: django_cmsdeploy_to: /srv/django_cmsgroup: nogroupid: django_cmslocal_settings_ﬁle: local_settings.pymysql_root_password: production: mysql_rootowner: nobodypackages: curl:pips: django: 1.2.5repository: https://github.com/opscode/django-quick-start-cms.gitrevision: production: masterserver_roles: ["django_cms"]type: django_cms: django gunicorn

Track it like source code...% git logcommit d640a8c6b370134d7043991894107d806595cc35Author: jtimberman <joshua@opscode.com> Import nagios version 1.0.0commit c40c818498710e78cf73c7f71e722e971fa574e7Author: jtimberman <joshua@opscode.com> installation and usage instruction docscommit 99d0efb024314de17888f6b359c14414fda7bb91Author: jtimberman <joshua@opscode.com> Import haproxy version 1.0.1commit c89d0975ad3f4b152426df219fee0bfb8eafb7e4Author: jtimberman <joshua@opscode.com> add mediawiki cookbookcommit 89c0545cc03b9be26f1db246c9ba4ce9d58a6700Author: jtimberman <joshua@opscode.com> multiple environments in data bag for mediawiki

Ponies ,(green )Unicorns and n-tier oh my!

N-Tier FTW• Provision

N-Tier FTW Load BalancerApp Server App Server • Provision • Configure Database Master

N-Tier FTW Load Balancer • ProvisionApp Server App Server • Configure • Integrate Database Master

Chef Enables Systems Integration

Chef Enables Systems Integration• Chef searches allow nodes to be aware of each other and self configure

Chef Enables Systems Integration• Chef searches allow nodes to be aware of each other and self configure• A shared application data bag allows:

Chef Enables Systems Integration• Chef searches allow nodes to be aware of each other and self configure• A shared application data bag allows: • many app servers to checkout the same application code during initial bootstrapping

Chef Enables Systems Integration• Chef searches allow nodes to be aware of each other and self configure• A shared application data bag allows: • many app servers to checkout the same application code during initial bootstrapping • app server and database servers to initialize off of the same database config

Database Master

Database Master% knife role show django_cms_database_masterdefault_attributes: {}description: db master for the django_cms app.env_run_lists: {}json_class: Chef::Rolename: django_cms_database_masteroverride_attributes: {}run_list: recipe[database::master]

Database Master

Database Master• Install and configure MySQL

Database Master• Install and configure MySQL• Create the application database

Database Master• Install and configure MySQL• Create the application database• Grant the application user access

Django App Servers (1..n)

Django App Servers% knife role show django_cmschef_type: roledefault_attributes: {}description: django_cms front end application server.env_run_lists: {}json_class: Chef::Rolename: django_cmsoverride_attributes: {}run_list: recipe[mysql::client], recipe[application]

Django App Servers

Django App Servers• Deploy the Code

Django App Servers• Deploy the Code • checkout the code with git

Django App Servers• Deploy the Code • checkout the code with git • install required packages (os and python)

Django App Servers• Deploy the Code • checkout the code with git • install required packages (os and python) • create environment specific config files

Django App Servers• Deploy the Code • checkout the code with git • install required packages (os and python) • create environment specific config files• Serve the Code

Django App Servers• Deploy the Code • checkout the code with git • install required packages (os and python) • create environment specific config files• Serve the Code • install Gunicorn (or apache2/mod_wsgi)

Django App Servers• Deploy the Code • checkout the code with git • install required packages (os and python) • create environment specific config files• Serve the Code • install Gunicorn (or apache2/mod_wsgi) • create an app specific config file (vhost)

Search for Database Masterresults = search(:node, "role:django_cms_database_master")template "/srv/django_cms/shared/local_settings.py" do source "settings.py.erb" mode "644" variables( :host => results[0][fqdn] )end

Load Balancer

Load Balancer% knife role show django_cms_load_balancerchef_type: roledefault_attributes: {}description: Django CMS load balancerenv_run_lists: {}json_class: Chef::Rolename: django_cms_load_balanceroverride_attributes: haproxy: app_server_role: django_cmsrun_list: recipe[haproxy::app_lb]

Load Balancer• The balancing pool should grow as more application servers are brought online

Search for App Serverspool_members = search(:node, "role:django_cms")template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members notiﬁes :restart, "service[haproxy]"end

One set of codez.... ...to rule them all

Same Code Different Configurations

Same Code Different Configurations• We can deploy the same app stack on:

Same Code Different Configurations• We can deploy the same app stack on: • a single node or a fully redundant cluster of nodes.

Same Code Different Configurations• We can deploy the same app stack on: • a single node or a fully redundant cluster of nodes. • physical hardware or any number of public/private cloud providers.

Same Code Different Configurations• We can deploy the same app stack on: • a single node or a fully redundant cluster of nodes. • physical hardware or any number of public/private cloud providers. • a developer workstation fully virtualized with Vagrant + VirtualBox

Staging/QA - Single Node# launch stack in your data center% knife bootstrap 123.123.3.3 -r ‘role[django_cms_database_master]’, ‘role[django_cms]’, ‘role[django_cms_load_balancer]’# launch stack in EC2% knife ec2 server create -r ‘role[django_cms_database_master]’, ‘role[django_cms]’, ‘role[django_cms_load_balancer]’# launch stack in Rackspace% knife rackspace server create -r ‘role[django_cms_database_master]’, ‘role[django_cms]’, ‘role[django_cms_load_balancer]’

Production - Multi-Machine Cluster% knife ec2 server create -r ‘role[django_cms_database_master]’% knife ec2 server create -r ‘role[django_cms]’% knife ec2 server create -r ‘role[django_cms]’% knife ec2 server create -r ‘role[django_cms_load_balancer]’

Developer Workstation% cat VagrantfileVagrant::Config.run do |config| config.vm.box = base_box config.vm.provision :chef_server do |chef| chef.chef_server_url = "https://api.opscode.com/organizations/foo" chef.validation_key_path = "~/.chef/private-pems/foo-validator.pem" chef.validation_client_name = "#{ENV[ORGNAME]}-validator" chef.provisioning_path = "/etc/chef" chef.run_list = [ role[django_cms_database_master], role[django_cms], role[django_cms_load_balancer] ] endend% vagrant up

BONUS: deploy [insert your favorite webframework here] stacks

Application Cookbook• Supports the following web stacks: • Django • Java Webapps • Php • Rails • Node.js (coming soon)• Opscode contributed quick starts walk you through example deployments

Further Resources & Closing• www.opscode.com• wiki.opscode.com• @opscode, #opschef• irc.freenode.net, #chef, #chef-hacking• lists.opscode.com

Questions? http://www.flickr.com/photos/oberazzi/318947873/

Thanks! schisamo@opscode.com www.opscode.com

SELF 2011: Deploying Django Application Stacks with Chef

by Opscode, Inc.

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 2

Statistics

SELF 2011: Deploying Django Application Stacks with Chef Presentation Transcript