Real world Django deployment using Chef
Upcoming SlideShare
Loading in...5

Real world Django deployment using Chef






Total Views
Views on SlideShare
Embed Views



7 Embeds 576 559 4 4
http://localhost 4 3 1 1


Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

Real world Django deployment using Chef Real world Django deployment using Chef Presentation Transcript

  • Real-world Django Deployment with Chef DjangoCon 2011 Noah Kantrowitz @kantrn
  • Who am I?• Django hacker• Ruby user• Developer• Sys admin
  • Who are you?• System administrators?• Designers?• Developers?• “Business” People?
  • What are we talking about?
  • Agenda• How’s and Why’s• Getting Started with Chef• Python-specific Tools• Case Study: Packaginator
  • Infrastructure as Code
  • A technical domainrevolving aroundbuilding andmanaginginfrastructureprogrammatically
  • Enable the reconstruction of the business fromnothing but a source coderepository, an application data backup, and bare metal resources.
  • ConfigurationManagement
  • System Integration
  • n-Tier Infrastructure Load BalancerApp Server { { App Server • • • Provision Configure Integrate Database Master
  • The Chef Framework With thanks (and apologies) to Stephen Nelson-Smith
  • The Chef Framework• Reasonability• Flexibility• Library & Primitives
  • The Chef Tool(s) With thanks (and apologies) to Stephen Nelson-Smith
  • The Chef Tool(s)• ohai• chef-client• chef-server• knife
  • The Chef APIWith thanks (and apologies) to Stephen Nelson-Smith
  • The Chef API• Client/Server• RESTful API w/ JSON• Search Service• Derivative Services
  • The Chef Community With thanks (and apologies) to Stephen Nelson-Smith
  • The Chef Community• Apache License, Version 2.0• 400+ Individual contributors• 90+ Corporate contributors • Dell, Rackspace,VMware, RightScale, Heroku, and more• 240+ cookbooks•
  • Chef Enables Infrastructure as Code package "haproxy" do action :install end template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb"• Resources owner "root" group "root"• Recipes mode 0644• notifies :restart, "service[haproxy]" Roles end• Source Code service "haproxy" do supports :restart => true action [:enable, :start] end
  • Chef Resources package "haproxy" do action :install end• Have a type. template "/etc/haproxy/haproxy.cfg" do• source "haproxy.cfg.erb" Have a name. owner "root"• Have parameters. group "root" mode 0644• Take action to put the resource notifies :restart, "service[haproxy]" end in the declared state.• Can send notifications to other service "haproxy" do supports :restart => true resources. action [:enable, :start] end
  • Resources take action through Providers
  • Chef Providerspackage “haproxy” { yum install haproxy apt-get install haproxy pacman sync haproxy pkg_add -r haproxy
  • Common Resourcespackage "apache2"template "/etc/apache2/httpd.conf"cookbook_file "/etc/apache2/key.pem"user "www-data"execute "/etc/init.d/apache2 restart"
  • Idempotence execute "createdb myapp" do not_if "psql -c list | grep myapp" end• Convergence execute "createdb myapp" do• Guard clauses only_if do db.query("SELECT ...").first == 0 end end
  • Recipes are collections of Resources
  • Chef Recipes package "haproxy" do action :install end template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb"• Recipes are evaluated for owner "root" resources in the order they group "root" mode 0644 appear. notifies :restart, "service[haproxy]"• Each resource object is added end to the Resource Collection. service "haproxy" do supports :restart => true action [:enable, :start] end
  • Chef Recipes• Recipes can include other include_recipe include_recipe "apache2" "apache2::mod_rewrite" recipes. include_recipe "apache2::mod_deflate"• Included recipes are include_recipe include_recipe "apache2::mod_headers" "apache2::mod_php5" processed in order.
  • Chef Recipes pool_members = search("node", "role:mediawiki") %w{ template "/etc/haproxy/haproxy.cfg" do• Extend recipes with template php5 php5-dev php5-cgi }.each do |pkg| "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" source "haproxy.cfg.erb" Ruby. owner "root" package pkg do owner "root" group "root"• action :install group "root" Dynamic configuration end mode 0644 mode 0644 notifies :restart, "service[haproxy]" through search. variables :pool_members => pool_members end end:restart, "service[haproxy]" notifies end
  • Chef Roles name "mediawiki" description "mediawiki app server" run_list( "recipe[mysql::client]", "recipe[application]", "recipe[mediawiki::status]" )• Roles describe nodes. name "mediawiki_load_balancer"• Roles have a run list. description "mediawiki load balancer" run_list(• Roles can have attributes. ) "recipe[haproxy::app_lb]" override_attributes( "haproxy" => { "app_server_role" => "mediawiki" } )
  • Track it like source code...% git logcommit d640a8c6b370134d7043991894107d806595cc35Author: jtimberman <> Import nagios version 1.0.0commit c40c818498710e78cf73c7f71e722e971fa574e7Author: jtimberman <> installation and usage instruction docscommit 99d0efb024314de17888f6b359c14414fda7bb91Author: jtimberman <> Import haproxy version 1.0.1commit c89d0975ad3f4b152426df219fee0bfb8eafb7e4Author: jtimberman <> add mediawiki cookbookcommit 89c0545cc03b9be26f1db246c9ba4ce9d58a6700Author: jtimberman <> multiple environments in data bag for mediawiki
  • Other Chef Terms• Cookbooks are collections of recipes• Environments are pegged cookbook versions• Servers are Nodes• Data bags are JSON blobs
  • Python Cookbook• python::package, python::source — Install Python• python::pip, python::virtualenv — Make it dance• python::default — All of the above!• python_pip, python_virtualenv
  • Installing a packagepython_virtualenv "/srv/myapp/env" do action :createendpython_pip "django" do action :upgrade version "1.3" virtualenv "/srv/myapp/env"end
  • Gunicorn Cookbook• gunicorn::default• gunicorn_config
  • Supervisor Cookbook• supervisor::default • Debian-style for now• supervisor_service
  • Packaginator
  • Role: base name "base" description "Base role applied to all nodes." run_list(• Install users "recipe[users::sysadmins]", "recipe[sudo]",• "recipe[apt]", Configure sudo ) "recipe[build-essential]"• override_attributes( apt-get update :authorization => { :sudo => {• Install gcc } :users => ["ubuntu"], :passwordless => true } )
  • Recipe: packaginator application "packaginator" do path "/srv/packaginator" owner "nobody" group "nogroup" repository "" revision "master" migrate true packages ["libpq-dev", "git-core", "mercurial"] django do packages ["redis"] requirements "requirements/mkii.txt" settings_template "" debug true database do database "packaginator"• engine "postgresql_psycopg2" The good stuff username "packaginator" password "awesome_password" end database_master_role "packaginator_database_master"• collectstatic "build_static --noinput" end General parameters gunicorn do only_if { node[roles].include? packaginator_application_server }• app_module :django port 8080 Sub-resources end celery do only_if { node[roles].include? packaginator_application_server } config "" django true celerybeat true celerycam true broker do transport "redis" end end nginx_load_balancer do only_if { node[roles].include? packaginator_load_balancer } application_port 8080 static_files "/site_media" => "site_media" end end
  • Application Resource application "packaginator" do• Who are we? path "/srv/packaginator"• Where are we going? owner "nobody" group "nogroup"
  • Application Resourcedirectory "/srv/packaginator"directory "/srv/packaginator/shared"deploy_revision "packaginator" do deploy_to "/srv/packaginator" owner "nobody" group "nogroup"
  • Application Resource repository ""• Source code revision "master"• Migration status migrate true packages ["libpq-dev",• Base packages "git-core","mercurial"]
  • Django Resource• Python packages• pip requirements django do packages ["redis"]• Settings file requirements "requirements/mkii.txt" settings_template ""• Debug mode debug true collectstatic "build_static --noinput"• Static files
  • Django Resourcepython_pip "redis"execute "pip -E ... -r ..."template "shared/"execute " build_static"
  • Folder Layout• /srv/myapp • /srv/myapp/shared • /srv/myapp/shared/releases • /srv/myapp/shared/releases/4faedb2ee... • /srv/myapp/shared/releases/9457be295... • ... • /srv/myapp/current
  • Folder Layout• /srv/myapp • ...9457be295/ -> ../shared/ • /srv/myapp/current -> .../releases/9457be295...
  • Django Resource database do database "packaginator"• Database engine "postgresql_psycopg2" username "packaginator"• Where? password "awesome_password" end database_master_role "packaginator_database_master"
  • Django Resource if node["roles"].include? ...• Check local first node else• Cloud IP! search(:node, "roles:...").first end
  • Gunicorn Resource gunicorn do only_if do• Only on frontends node["roles"].include? ...• Use Django mode end app_module :django• Internal port port 8080 end
  • Gunicorn Resource• Write out config gunicorn_config ...• Symlink it in supervisor_service ...• Load service
  • Celery Resource celery do only_if ... config ""• Where? django true• Django-mode celerybeat true• Enable all three celerycam true broker do• Broker and results transport "redis" end end
  • Nginx Resource nginx_load_balancer do only_if ... application_port 8080• Internal port again static_files({• Static map "/site_media" => "site_media" • URL => path }) end
  • Noah Kantrowitz @kantrn