Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Real world Django deployment using Chef


Published on

Published in: Technology, Self Improvement
  • Be the first to comment

Real world Django deployment using Chef

  1. 1. Real-world Django Deployment with Chef DjangoCon 2011 Noah Kantrowitz @kantrn
  2. 2. Who am I?• Django hacker• Ruby user• Developer• Sys admin
  3. 3. Who are you?• System administrators?• Designers?• Developers?• “Business” People?
  4. 4. What are we talking about?
  5. 5. Agenda• How’s and Why’s• Getting Started with Chef• Python-specific Tools• Case Study: Packaginator
  6. 6. Infrastructure as Code
  7. 7. A technical domainrevolving aroundbuilding andmanaginginfrastructureprogrammatically
  8. 8. Enable the reconstruction of the business fromnothing but a source coderepository, an application data backup, and bare metal resources.
  9. 9. ConfigurationManagement
  10. 10. System Integration
  11. 11. n-Tier Infrastructure Load BalancerApp Server { { App Server • • • Provision Configure Integrate Database Master
  12. 12. The Chef Framework With thanks (and apologies) to Stephen Nelson-Smith
  13. 13. The Chef Framework• Reasonability• Flexibility• Library & Primitives
  14. 14. The Chef Tool(s) With thanks (and apologies) to Stephen Nelson-Smith
  15. 15. The Chef Tool(s)• ohai• chef-client• chef-server• knife
  16. 16. The Chef APIWith thanks (and apologies) to Stephen Nelson-Smith
  17. 17. The Chef API• Client/Server• RESTful API w/ JSON• Search Service• Derivative Services
  18. 18. The Chef Community With thanks (and apologies) to Stephen Nelson-Smith
  19. 19. The Chef Community• Apache License, Version 2.0• 400+ Individual contributors• 90+ Corporate contributors • Dell, Rackspace,VMware, RightScale, Heroku, and more• 240+ cookbooks•
  20. 20. Chef Enables Infrastructure as Code package "haproxy" do action :install end template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb"• Resources owner "root" group "root"• Recipes mode 0644• notifies :restart, "service[haproxy]" Roles end• Source Code service "haproxy" do supports :restart => true action [:enable, :start] end
  21. 21. Chef Resources package "haproxy" do action :install end• Have a type. template "/etc/haproxy/haproxy.cfg" do• source "haproxy.cfg.erb" Have a name. owner "root"• Have parameters. group "root" mode 0644• Take action to put the resource notifies :restart, "service[haproxy]" end in the declared state.• Can send notifications to other service "haproxy" do supports :restart => true resources. action [:enable, :start] end
  22. 22. Resources take action through Providers
  23. 23. Chef Providerspackage “haproxy” { yum install haproxy apt-get install haproxy pacman sync haproxy pkg_add -r haproxy
  24. 24. Common Resourcespackage "apache2"template "/etc/apache2/httpd.conf"cookbook_file "/etc/apache2/key.pem"user "www-data"execute "/etc/init.d/apache2 restart"
  25. 25. Idempotence execute "createdb myapp" do not_if "psql -c list | grep myapp" end• Convergence execute "createdb myapp" do• Guard clauses only_if do db.query("SELECT ...").first == 0 end end
  26. 26. Recipes are collections of Resources
  27. 27. Chef Recipes package "haproxy" do action :install end template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb"• Recipes are evaluated for owner "root" resources in the order they group "root" mode 0644 appear. notifies :restart, "service[haproxy]"• Each resource object is added end to the Resource Collection. service "haproxy" do supports :restart => true action [:enable, :start] end
  28. 28. Chef Recipes• Recipes can include other include_recipe include_recipe "apache2" "apache2::mod_rewrite" recipes. include_recipe "apache2::mod_deflate"• Included recipes are include_recipe include_recipe "apache2::mod_headers" "apache2::mod_php5" processed in order.
  29. 29. Chef Recipes pool_members = search("node", "role:mediawiki") %w{ template "/etc/haproxy/haproxy.cfg" do• Extend recipes with template php5 php5-dev php5-cgi }.each do |pkg| "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" source "haproxy.cfg.erb" Ruby. owner "root" package pkg do owner "root" group "root"• action :install group "root" Dynamic configuration end mode 0644 mode 0644 notifies :restart, "service[haproxy]" through search. variables :pool_members => pool_members end end:restart, "service[haproxy]" notifies end
  30. 30. Chef Roles name "mediawiki" description "mediawiki app server" run_list( "recipe[mysql::client]", "recipe[application]", "recipe[mediawiki::status]" )• Roles describe nodes. name "mediawiki_load_balancer"• Roles have a run list. description "mediawiki load balancer" run_list(• Roles can have attributes. ) "recipe[haproxy::app_lb]" override_attributes( "haproxy" => { "app_server_role" => "mediawiki" } )
  31. 31. Track it like source code...% git logcommit d640a8c6b370134d7043991894107d806595cc35Author: jtimberman <> Import nagios version 1.0.0commit c40c818498710e78cf73c7f71e722e971fa574e7Author: jtimberman <> installation and usage instruction docscommit 99d0efb024314de17888f6b359c14414fda7bb91Author: jtimberman <> Import haproxy version 1.0.1commit c89d0975ad3f4b152426df219fee0bfb8eafb7e4Author: jtimberman <> add mediawiki cookbookcommit 89c0545cc03b9be26f1db246c9ba4ce9d58a6700Author: jtimberman <> multiple environments in data bag for mediawiki
  32. 32. Other Chef Terms• Cookbooks are collections of recipes• Environments are pegged cookbook versions• Servers are Nodes• Data bags are JSON blobs
  33. 33. Python Cookbook• python::package, python::source — Install Python• python::pip, python::virtualenv — Make it dance• python::default — All of the above!• python_pip, python_virtualenv
  34. 34. Installing a packagepython_virtualenv "/srv/myapp/env" do action :createendpython_pip "django" do action :upgrade version "1.3" virtualenv "/srv/myapp/env"end
  35. 35. Gunicorn Cookbook• gunicorn::default• gunicorn_config
  36. 36. Supervisor Cookbook• supervisor::default • Debian-style for now• supervisor_service
  37. 37. Packaginator
  38. 38. Role: base name "base" description "Base role applied to all nodes." run_list(• Install users "recipe[users::sysadmins]", "recipe[sudo]",• "recipe[apt]", Configure sudo ) "recipe[build-essential]"• override_attributes( apt-get update :authorization => { :sudo => {• Install gcc } :users => ["ubuntu"], :passwordless => true } )
  39. 39. Recipe: packaginator application "packaginator" do path "/srv/packaginator" owner "nobody" group "nogroup" repository "" revision "master" migrate true packages ["libpq-dev", "git-core", "mercurial"] django do packages ["redis"] requirements "requirements/mkii.txt" settings_template "" debug true database do database "packaginator"• engine "postgresql_psycopg2" The good stuff username "packaginator" password "awesome_password" end database_master_role "packaginator_database_master"• collectstatic "build_static --noinput" end General parameters gunicorn do only_if { node[roles].include? packaginator_application_server }• app_module :django port 8080 Sub-resources end celery do only_if { node[roles].include? packaginator_application_server } config "" django true celerybeat true celerycam true broker do transport "redis" end end nginx_load_balancer do only_if { node[roles].include? packaginator_load_balancer } application_port 8080 static_files "/site_media" => "site_media" end end
  40. 40. Application Resource application "packaginator" do• Who are we? path "/srv/packaginator"• Where are we going? owner "nobody" group "nogroup"
  41. 41. Application Resourcedirectory "/srv/packaginator"directory "/srv/packaginator/shared"deploy_revision "packaginator" do deploy_to "/srv/packaginator" owner "nobody" group "nogroup"
  42. 42. Application Resource repository ""• Source code revision "master"• Migration status migrate true packages ["libpq-dev",• Base packages "git-core","mercurial"]
  43. 43. Django Resource• Python packages• pip requirements django do packages ["redis"]• Settings file requirements "requirements/mkii.txt" settings_template ""• Debug mode debug true collectstatic "build_static --noinput"• Static files
  44. 44. Django Resourcepython_pip "redis"execute "pip -E ... -r ..."template "shared/"execute " build_static"
  45. 45. Folder Layout• /srv/myapp • /srv/myapp/shared • /srv/myapp/shared/releases • /srv/myapp/shared/releases/4faedb2ee... • /srv/myapp/shared/releases/9457be295... • ... • /srv/myapp/current
  46. 46. Folder Layout• /srv/myapp • ...9457be295/ -> ../shared/ • /srv/myapp/current -> .../releases/9457be295...
  47. 47. Django Resource database do database "packaginator"• Database engine "postgresql_psycopg2" username "packaginator"• Where? password "awesome_password" end database_master_role "packaginator_database_master"
  48. 48. Django Resource if node["roles"].include? ...• Check local first node else• Cloud IP! search(:node, "roles:...").first end
  49. 49. Gunicorn Resource gunicorn do only_if do• Only on frontends node["roles"].include? ...• Use Django mode end app_module :django• Internal port port 8080 end
  50. 50. Gunicorn Resource• Write out config gunicorn_config ...• Symlink it in supervisor_service ...• Load service
  51. 51. Celery Resource celery do only_if ... config ""• Where? django true• Django-mode celerybeat true• Enable all three celerycam true broker do• Broker and results transport "redis" end end
  52. 52. Nginx Resource nginx_load_balancer do only_if ... application_port 8080• Internal port again static_files({• Static map "/site_media" => "site_media" • URL => path }) end
  53. 53.
  54. 54. Noah Kantrowitz @kantrn