Real world Django deployment using Chef

Real-world Django Deployment with Chef DjangoCon 2011 Noah Kantrowitz noah@opscode.com @kantrn

Who am I?• Django hacker• Ruby user• Developer• Sys admin

Who are you?• System administrators?• Designers?• Developers?• “Business” People? http://www.ﬂickr.com/photos/timyates/2854357446/sizes/l/

What are we talking about? http://www.ﬂickr.com/photos/peterkaminski/2174679908/

Agenda• How’s and Why’s• Getting Started with Chef• Python-specific Tools• Case Study: Packaginator http://www.ﬂickr.com/photos/koalazymonkey/3590953001/

Infrastructure as Code

A technical domainrevolving aroundbuilding andmanaginginfrastructureprogrammatically

Enable the reconstruction of the business fromnothing but a source coderepository, an application data backup, and bare metal resources.

ConfigurationManagement

System Integration http://www.ﬂickr.com/photos/opalsson/3773629074/

n-Tier Infrastructure Load BalancerApp Server { { App Server • • • Provision Configure Integrate Database Master

The Chef Framework With thanks (and apologies) to Stephen Nelson-Smith

The Chef Framework• Reasonability• Flexibility• Library & Primitives

The Chef Tool(s) With thanks (and apologies) to Stephen Nelson-Smith

The Chef Tool(s)• ohai• chef-client• chef-server• knife

The Chef APIWith thanks (and apologies) to Stephen Nelson-Smith

The Chef API• Client/Server• RESTful API w/ JSON• Search Service• Derivative Services

The Chef Community With thanks (and apologies) to Stephen Nelson-Smith

The Chef Community• Apache License, Version 2.0• 400+ Individual contributors• 90+ Corporate contributors • Dell, Rackspace,VMware, RightScale, Heroku, and more• 240+ cookbooks• http://community.opscode.com/

Chef Enables Infrastructure as Code package "haproxy" do action :install end template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb"• Resources owner "root" group "root"• Recipes mode 0644• notifies :restart, "service[haproxy]" Roles end• Source Code service "haproxy" do supports :restart => true action [:enable, :start] end

Chef Resources package "haproxy" do action :install end• Have a type. template "/etc/haproxy/haproxy.cfg" do• source "haproxy.cfg.erb" Have a name. owner "root"• Have parameters. group "root" mode 0644• Take action to put the resource notifies :restart, "service[haproxy]" end in the declared state.• Can send notifications to other service "haproxy" do supports :restart => true resources. action [:enable, :start] end

Resources take action through Providers

Chef Providerspackage “haproxy” { yum install haproxy apt-get install haproxy pacman sync haproxy pkg_add -r haproxy

Common Resourcespackage "apache2"template "/etc/apache2/httpd.conf"cookbook_file "/etc/apache2/key.pem"user "www-data"execute "/etc/init.d/apache2 restart"

Idempotence execute "createdb myapp" do not_if "psql -c list | grep myapp" end• Convergence execute "createdb myapp" do• Guard clauses only_if do db.query("SELECT ...").first == 0 end end

Recipes are collections of Resources

Chef Recipes package "haproxy" do action :install end template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb"• Recipes are evaluated for owner "root" resources in the order they group "root" mode 0644 appear. notifies :restart, "service[haproxy]"• Each resource object is added end to the Resource Collection. service "haproxy" do supports :restart => true action [:enable, :start] end

Chef Recipes• Recipes can include other include_recipe include_recipe "apache2" "apache2::mod_rewrite" recipes. include_recipe "apache2::mod_deflate"• Included recipes are include_recipe include_recipe "apache2::mod_headers" "apache2::mod_php5" processed in order.

Chef Recipes pool_members = search("node", "role:mediawiki") %w{ template "/etc/haproxy/haproxy.cfg" do• Extend recipes with template php5 php5-dev php5-cgi }.each do |pkg| "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" source "haproxy.cfg.erb" Ruby. owner "root" package pkg do owner "root" group "root"• action :install group "root" Dynamic configuration end mode 0644 mode 0644 notifies :restart, "service[haproxy]" through search. variables :pool_members => pool_members end end:restart, "service[haproxy]" notifies end

Chef Roles name "mediawiki" description "mediawiki app server" run_list( "recipe[mysql::client]", "recipe[application]", "recipe[mediawiki::status]" )• Roles describe nodes. name "mediawiki_load_balancer"• Roles have a run list. description "mediawiki load balancer" run_list(• Roles can have attributes. ) "recipe[haproxy::app_lb]" override_attributes( "haproxy" => { "app_server_role" => "mediawiki" } )

Track it like source code...% git logcommit d640a8c6b370134d7043991894107d806595cc35Author: jtimberman <joshua@opscode.com> Import nagios version 1.0.0commit c40c818498710e78cf73c7f71e722e971fa574e7Author: jtimberman <joshua@opscode.com> installation and usage instruction docscommit 99d0efb024314de17888f6b359c14414fda7bb91Author: jtimberman <joshua@opscode.com> Import haproxy version 1.0.1commit c89d0975ad3f4b152426df219fee0bfb8eafb7e4Author: jtimberman <joshua@opscode.com> add mediawiki cookbookcommit 89c0545cc03b9be26f1db246c9ba4ce9d58a6700Author: jtimberman <joshua@opscode.com> multiple environments in data bag for mediawiki

Other Chef Terms• Cookbooks are collections of recipes• Environments are pegged cookbook versions• Servers are Nodes• Data bags are JSON blobs

Python Cookbook• python::package, python::source — Install Python• python::pip, python::virtualenv — Make it dance• python::default — All of the above!• python_pip, python_virtualenv

Installing a packagepython_virtualenv "/srv/myapp/env" do action :createendpython_pip "django" do action :upgrade version "1.3" virtualenv "/srv/myapp/env"end

Gunicorn Cookbook• gunicorn::default• gunicorn_config

Supervisor Cookbook• supervisor::default • Debian-style for now• supervisor_service

Packaginator

Role: base name "base" description "Base role applied to all nodes." run_list(• Install users "recipe[users::sysadmins]", "recipe[sudo]",• "recipe[apt]", Configure sudo ) "recipe[build-essential]"• override_attributes( apt-get update :authorization => { :sudo => {• Install gcc } :users => ["ubuntu"], :passwordless => true } )

Recipe: packaginator application "packaginator" do path "/srv/packaginator" owner "nobody" group "nogroup" repository "https://github.com/coderanger/packaginator.git" revision "master" migrate true packages ["libpq-dev", "git-core", "mercurial"] django do packages ["redis"] requirements "requirements/mkii.txt" settings_template "settings.py.erb" debug true database do database "packaginator"• engine "postgresql_psycopg2" The good stuff username "packaginator" password "awesome_password" end database_master_role "packaginator_database_master"• collectstatic "build_static --noinput" end General parameters gunicorn do only_if { node[roles].include? packaginator_application_server }• app_module :django port 8080 Sub-resources end celery do only_if { node[roles].include? packaginator_application_server } config "celery_settings.py" django true celerybeat true celerycam true broker do transport "redis" end end nginx_load_balancer do only_if { node[roles].include? packaginator_load_balancer } application_port 8080 static_files "/site_media" => "site_media" end end

Application Resource application "packaginator" do• Who are we? path "/srv/packaginator"• Where are we going? owner "nobody" group "nogroup"

Application Resourcedirectory "/srv/packaginator"directory "/srv/packaginator/shared"deploy_revision "packaginator" do deploy_to "/srv/packaginator" owner "nobody" group "nogroup"

Application Resource repository "https://github.com/..."• Source code revision "master"• Migration status migrate true packages ["libpq-dev",• Base packages "git-core","mercurial"]

Django Resource• Python packages• pip requirements django do packages ["redis"]• Settings file requirements "requirements/mkii.txt" settings_template "settings.py.erb"• Debug mode debug true collectstatic "build_static --noinput"• Static files

Django Resourcepython_pip "redis"execute "pip -E ... -r ..."template "shared/local_settings.py"execute "manage.py build_static"

Folder Layout• /srv/myapp • /srv/myapp/shared • /srv/myapp/shared/releases • /srv/myapp/shared/releases/4faedb2ee... • /srv/myapp/shared/releases/9457be295... • ... • /srv/myapp/current

Folder Layout• /srv/myapp • ...9457be295/local_settings.py -> ../shared/local_settings.py • /srv/myapp/current -> .../releases/9457be295...

Django Resource database do database "packaginator"• Database engine "postgresql_psycopg2" username "packaginator"• Where? password "awesome_password" end database_master_role "packaginator_database_master"

Django Resource if node["roles"].include? ...• Check local first node else• Cloud IP! search(:node, "roles:...").first end

Gunicorn Resource gunicorn do only_if do• Only on frontends node["roles"].include? ...• Use Django mode end app_module :django• Internal port port 8080 end

Gunicorn Resource• Write out config gunicorn_config ...• Symlink it in supervisor_service ...• Load service

Celery Resource celery do only_if ... config "celery_settings.py"• Where? django true• Django-mode celerybeat true• Enable all three celerycam true broker do• Broker and results transport "redis" end end

Nginx Resource nginx_load_balancer do only_if ... application_port 8080• Internal port again static_files({• Static map "/site_media" => "site_media" • URL => path }) end

https://github.com/coderanger/djangocon2011

Noah Kantrowitz @kantrn noah@opscode.com

http://lanyrd.com	487
https://twitter.com	4
http://twitter.com	4
http://paper.li	3
http://leapf.org	1
https://si0.twimg.com	1

Real world Django deployment using Chef

by coderanger

Accessibility

Categories

Upload Details

Usage Rights

6 Embeds 500

Statistics

Real world Django deployment using Chef Presentation Transcript