Successfully reported this slideshow.
Your SlideShare is downloading. ×

Real world Django deployment using Chef

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Upcoming SlideShare
Picconf12
Picconf12
Loading in …3
×

Check these out next

1 of 55 Ad
Advertisement

More Related Content

Slideshows for you (19)

Similar to Real world Django deployment using Chef (20)

Advertisement

Recently uploaded (20)

Real world Django deployment using Chef

  1. 1. Real-world Django Deployment with Chef DjangoCon 2011 Noah Kantrowitz noah@opscode.com @kantrn
  2. 2. Who am I? • Django hacker • Ruby user • Developer • Sys admin
  3. 3. Who are you? • System administrators? • Designers? • Developers? • “Business” People? http://www.flickr.com/photos/timyates/2854357446/sizes/l/
  4. 4. What are we talking about? http://www.flickr.com/photos/peterkaminski/2174679908/
  5. 5. Agenda • How’s and Why’s • Getting Started with Chef • Python-specific Tools • Case Study: Packaginator http://www.flickr.com/photos/koalazymonkey/3590953001/
  6. 6. Infrastructure as Code
  7. 7. A technical domain revolving around building and managing infrastructure programmatically
  8. 8. Enable the reconstruction of the business from nothing but a source code repository, an application data backup, and bare metal resources.
  9. 9. Configuration Management
  10. 10. System Integration http://www.flickr.com/photos/opalsson/3773629074/
  11. 11. n-Tier Infrastructure Load Balancer App Server { { App Server • • • Provision Configure Integrate Database Master
  12. 12. The Chef Framework With thanks (and apologies) to Stephen Nelson-Smith
  13. 13. The Chef Framework • Reasonability • Flexibility • Library & Primitives
  14. 14. The Chef Tool(s) With thanks (and apologies) to Stephen Nelson-Smith
  15. 15. The Chef Tool(s) • ohai • chef-client • chef-server • knife
  16. 16. The Chef API With thanks (and apologies) to Stephen Nelson-Smith
  17. 17. The Chef API • Client/Server • RESTful API w/ JSON • Search Service • Derivative Services
  18. 18. The Chef Community With thanks (and apologies) to Stephen Nelson-Smith
  19. 19. The Chef Community • Apache License, Version 2.0 • 400+ Individual contributors • 90+ Corporate contributors • Dell, Rackspace,VMware, RightScale, Heroku, and more • 240+ cookbooks • http://community.opscode.com/
  20. 20. Chef Enables Infrastructure as Code package "haproxy" do action :install end template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" • Resources owner "root" group "root" • Recipes mode 0644 • notifies :restart, "service[haproxy]" Roles end • Source Code service "haproxy" do supports :restart => true action [:enable, :start] end
  21. 21. Chef Resources package "haproxy" do action :install end • Have a type. template "/etc/haproxy/haproxy.cfg" do • source "haproxy.cfg.erb" Have a name. owner "root" • Have parameters. group "root" mode 0644 • Take action to put the resource notifies :restart, "service[haproxy]" end in the declared state. • Can send notifications to other service "haproxy" do supports :restart => true resources. action [:enable, :start] end
  22. 22. Resources take action through Providers
  23. 23. Chef Providers package “haproxy” { yum install haproxy apt-get install haproxy pacman sync haproxy pkg_add -r haproxy
  24. 24. Common Resources package "apache2" template "/etc/apache2/httpd.conf" cookbook_file "/etc/apache2/key.pem" user "www-data" execute "/etc/init.d/apache2 restart"
  25. 25. Idempotence execute "createdb myapp" do not_if "psql -c list | grep myapp" end • Convergence execute "createdb myapp" do • Guard clauses only_if do db.query("SELECT ...").first == 0 end end
  26. 26. Recipes are collections of Resources
  27. 27. Chef Recipes package "haproxy" do action :install end template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" • Recipes are evaluated for owner "root" resources in the order they group "root" mode 0644 appear. notifies :restart, "service[haproxy]" • Each resource object is added end to the Resource Collection. service "haproxy" do supports :restart => true action [:enable, :start] end
  28. 28. Chef Recipes • Recipes can include other include_recipe include_recipe "apache2" "apache2::mod_rewrite" recipes. include_recipe "apache2::mod_deflate" • Included recipes are include_recipe include_recipe "apache2::mod_headers" "apache2::mod_php5" processed in order.
  29. 29. Chef Recipes pool_members = search("node", "role:mediawiki") %w{ template "/etc/haproxy/haproxy.cfg" do • Extend recipes with template php5 php5-dev php5-cgi }.each do |pkg| "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" source "haproxy.cfg.erb" Ruby. owner "root" package pkg do owner "root" group "root" • action :install group "root" Dynamic configuration end mode 0644 mode 0644 notifies :restart, "service[haproxy]" through search. variables :pool_members => pool_members end end:restart, "service[haproxy]" notifies end
  30. 30. Chef Roles name "mediawiki" description "mediawiki app server" run_list( "recipe[mysql::client]", "recipe[application]", "recipe[mediawiki::status]" ) • Roles describe nodes. name "mediawiki_load_balancer" • Roles have a run list. description "mediawiki load balancer" run_list( • Roles can have attributes. ) "recipe[haproxy::app_lb]" override_attributes( "haproxy" => { "app_server_role" => "mediawiki" } )
  31. 31. Track it like source code... % git log commit d640a8c6b370134d7043991894107d806595cc35 Author: jtimberman <joshua@opscode.com> Import nagios version 1.0.0 commit c40c818498710e78cf73c7f71e722e971fa574e7 Author: jtimberman <joshua@opscode.com> installation and usage instruction docs commit 99d0efb024314de17888f6b359c14414fda7bb91 Author: jtimberman <joshua@opscode.com> Import haproxy version 1.0.1 commit c89d0975ad3f4b152426df219fee0bfb8eafb7e4 Author: jtimberman <joshua@opscode.com> add mediawiki cookbook commit 89c0545cc03b9be26f1db246c9ba4ce9d58a6700 Author: jtimberman <joshua@opscode.com> multiple environments in data bag for mediawiki
  32. 32. Other Chef Terms • Cookbooks are collections of recipes • Environments are pegged cookbook versions • Servers are Nodes • Data bags are JSON blobs
  33. 33. Python Cookbook • python::package, python::source — Install Python • python::pip, python::virtualenv — Make it dance • python::default — All of the above! • python_pip, python_virtualenv
  34. 34. Installing a package python_virtualenv "/srv/myapp/env" do action :create end python_pip "django" do action :upgrade version "1.3" virtualenv "/srv/myapp/env" end
  35. 35. Gunicorn Cookbook • gunicorn::default • gunicorn_config
  36. 36. Supervisor Cookbook • supervisor::default • Debian-style for now • supervisor_service
  37. 37. Packaginator
  38. 38. Role: base name "base" description "Base role applied to all nodes." run_list( • Install users "recipe[users::sysadmins]", "recipe[sudo]", • "recipe[apt]", Configure sudo ) "recipe[build-essential]" • override_attributes( apt-get update :authorization => { :sudo => { • Install gcc } :users => ["ubuntu"], :passwordless => true } )
  39. 39. Recipe: packaginator application "packaginator" do path "/srv/packaginator" owner "nobody" group "nogroup" repository "https://github.com/coderanger/packaginator.git" revision "master" migrate true packages ["libpq-dev", "git-core", "mercurial"] django do packages ["redis"] requirements "requirements/mkii.txt" settings_template "settings.py.erb" debug true database do database "packaginator" • engine "postgresql_psycopg2" The good stuff username "packaginator" password "awesome_password" end database_master_role "packaginator_database_master" • collectstatic "build_static --noinput" end General parameters gunicorn do only_if { node['roles'].include? 'packaginator_application_server' } • app_module :django port 8080 Sub-resources end celery do only_if { node['roles'].include? 'packaginator_application_server' } config "celery_settings.py" django true celerybeat true celerycam true broker do transport "redis" end end nginx_load_balancer do only_if { node['roles'].include? 'packaginator_load_balancer' } application_port 8080 static_files "/site_media" => "site_media" end end
  40. 40. Application Resource application "packaginator" do • Who are we? path "/srv/packaginator" • Where are we going? owner "nobody" group "nogroup"
  41. 41. Application Resource directory "/srv/packaginator" directory "/srv/packaginator/shared" deploy_revision "packaginator" do deploy_to "/srv/packaginator" owner "nobody" group "nogroup"
  42. 42. Application Resource repository "https://github.com/..." • Source code revision "master" • Migration status migrate true packages ["libpq-dev", • Base packages "git-core","mercurial"]
  43. 43. Django Resource • Python packages • pip requirements django do packages ["redis"] • Settings file requirements "requirements/mkii.txt" settings_template "settings.py.erb" • Debug mode debug true collectstatic "build_static --noinput" • Static files
  44. 44. Django Resource python_pip "redis" execute "pip -E ... -r ..." template "shared/local_settings.py" execute "manage.py build_static"
  45. 45. Folder Layout • /srv/myapp • /srv/myapp/shared • /srv/myapp/shared/releases • /srv/myapp/shared/releases/4faedb2ee... • /srv/myapp/shared/releases/9457be295... • ... • /srv/myapp/current
  46. 46. Folder Layout • /srv/myapp • ...9457be295/local_settings.py -> ../shared/local_settings.py • /srv/myapp/current -> .../releases/9457be295...
  47. 47. Django Resource database do database "packaginator" • Database engine "postgresql_psycopg2" username "packaginator" • Where? password "awesome_password" end database_master_role "packaginator_database_master"
  48. 48. Django Resource if node["roles"].include? ... • Check local first node else • Cloud IP! search(:node, "roles:...").first end
  49. 49. Gunicorn Resource gunicorn do only_if do • Only on frontends node["roles"].include? ... • Use Django mode end app_module :django • Internal port port 8080 end
  50. 50. Gunicorn Resource • Write out config gunicorn_config ... • Symlink it in supervisor_service ... • Load service
  51. 51. Celery Resource celery do only_if ... config "celery_settings.py" • Where? django true • Django-mode celerybeat true • Enable all three celerycam true broker do • Broker and results transport "redis" end end
  52. 52. Nginx Resource nginx_load_balancer do only_if ... application_port 8080 • Internal port again static_files({ • Static map "/site_media" => "site_media" • URL => path }) end
  53. 53. https://github.com/coderanger/djangocon2011
  54. 54. Noah Kantrowitz @kantrn noah@opscode.com

×