This document discusses deploying fully automated, highly available Django application stacks using Chef. Chef is an infrastructure automation tool that allows users to define infrastructure as code. It enables deploying and managing multi-tier Django application stacks across servers through recipes, which are collections of resources that define what software and configuration should exist on nodes. Chef recipes can be extended with Ruby code and configured dynamically based on node attributes and data from searches. Cookbooks package recipes for reuse across environments. Roles are used to describe nodes and assign recipes and attributes. Data bags store arbitrary configuration data.

1. Deploying Fully Automated, Highly Available
Django Application Stacks with Chef
schisamo@opscode.com
www.opscode.com

2. Who am I?

3. Who am I?
• Seth Chisamore
Senior Technical Consultant/Evangelist
Opscode, Inc.

4. Who am I?
• Seth Chisamore
Senior Technical Consultant/Evangelist
Opscode, Inc.
• Project lead for many Opscode sponsored
open source projects:
• Opscode Cookbooks
• Knife cloud provisioning plugins (ec2,
rackspace & openstack)
• Knife Windows plugin (knife-windows)

5. Who are you?
http://www.flickr.com/photos/timyates/2854357446/sizes/l/

6. Who are you?
• System administrators?
http://www.flickr.com/photos/timyates/2854357446/sizes/l/

7. Who are you?
• System administrators?
• Developers?
http://www.flickr.com/photos/timyates/2854357446/sizes/l/

8. Who are you?
• System administrators?
• Developers?
• “Business” People?
http://www.flickr.com/photos/timyates/2854357446/sizes/l/

9. What are we talking
about?
http://www.flickr.com/photos/peterkaminski/2174679908/

10. Agenda
http://www.flickr.com/photos/koalazymonkey/3590953001/

11. Agenda
• How’s and Why’s
http://www.flickr.com/photos/koalazymonkey/3590953001/

12. Agenda
• How’s and Why’s
• Chef 101
http://www.flickr.com/photos/koalazymonkey/3590953001/

13. Agenda
• How’s and Why’s
• Chef 101
• Deploying n-tier Django app stacks
with Chef
http://www.flickr.com/photos/koalazymonkey/3590953001/

14. Agenda
• How’s and Why’s
• Chef 101
• Deploying n-tier Django app stacks
with Chef
• Leveraging the same Chef code to
deploy different configurations of
our stack
http://www.flickr.com/photos/koalazymonkey/3590953001/

15. Infrastructure as Code

16. A technical domain
revolving around
building and
managing
infrastructure
programmatically

17. Enable the reconstruction
of the business from
nothing but a source code
repository, an application
data backup, and bare
metal resources.

18. Configuration
Management

19. Configuration
Management

20. System Integration
http://www.flickr.com/photos/opalsson/3773629074/

22. The Chef Framework
With thanks (and apologies) to Stephen Nelson-Smith

23. The Chef Framework

24. The Chef Framework
• Idempotent

25. The Chef Framework
• Idempotent
• Library & Primitives

26. The Chef Framework
• Idempotent
• Library & Primitives
• Flexibility

27. The Chef Framework
• Idempotent
• Library & Primitives
• Flexibility
• Reasonability

28. The Chef Framework
• Idempotent
• Library & Primitives
• Flexibility
• Reasonability
• TIMTOWTDI

29. Multiple applications of
an operation do not
change the result

30. We start with APIs, you
supply data

31. Defaults are sane, but
easily changed

32. There is more than one
way to do it

33. The Chef Tool(s)
With thanks (and apologies) to Stephen Nelson-Smith

34. The Chef Tool(s)

35. The Chef Tool(s)
• ohai

36. The Chef Tool(s)
• ohai
• chef-client

37. The Chef Tool(s)
• ohai
• chef-client
• knife

38. The Chef Tool(s)
• ohai
• chef-client
• knife
• shef

39. The Chef API
With thanks (and apologies) to Stephen Nelson-Smith

40. The Chef API

41. The Chef API
• Fat Client/Thin Server

42. The Chef API
• Fat Client/Thin Server
• RESTful API w/ JSON

43. The Chef API
• Fat Client/Thin Server
• RESTful API w/ JSON
• Search Service

44. The Chef API
• Fat Client/Thin Server
• RESTful API w/ JSON
• Search Service
• Derivative Services (mash-up your
infrastructure)

45. The Chef Community
With thanks (and apologies) to Stephen Nelson-Smith

46. The Chef Community

47. The Chef Community
• Apache License, Version 2.0

48. The Chef Community
• Apache License, Version 2.0
• 360+ Individual contributors

49. The Chef Community
• Apache License, Version 2.0
• 360+ Individual contributors
• 70+ Corporate contributors

50. The Chef Community
• Apache License, Version 2.0
• 360+ Individual contributors
• 70+ Corporate contributors
• Dell, Rackspace,VMware, RightScale,
Heroku, and more

51. The Chef Community
• Apache License, Version 2.0
• 360+ Individual contributors
• 70+ Corporate contributors
• Dell, Rackspace,VMware, RightScale,
Heroku, and more
• 240+ cookbooks

52. The Chef Community
• Apache License, Version 2.0
• 360+ Individual contributors
• 70+ Corporate contributors
• Dell, Rackspace,VMware, RightScale,
Heroku, and more
• 240+ cookbooks
• http://community.opscode.com

53. Chef client runs on your
systems

54. Clients talk to a Chef
server

55. Clients authenticate with
RSA keys

56. Each system you
configure is a managed
node

57. Nodes have attributes
{
"kernel": {
"machine": "x86_64",
"name": "Darwin",
"os": "Darwin",
"version": "Darwin Kernel Version 10.4.0: Fri Apr 23 18:28:53 PDT 2010;
root:xnu-1504.7.4~1/RELEASE_I386",
"release": "10.4.0"
},
"platform_version": "10.6.4",
"platform": "mac_os_x",
"platform_build": "10F569",
"domain": "local",
"os": "darwin",
"current_user": "schisamo",
"ohai_time": 1278602661.60043,
"os_version": "10.4.0",
"uptime": "18 days 17 hours 49 minutes 18 seconds",
"ipaddress": "10.13.37.116",
"hostname": "helsinki",
"fqdn": "helsinki.local",
"uptime_seconds": 1619358
}

58. Nodes are Searchable
knife search node ‘platform:mac_os_x’
search(:node, ‘platform:mac_os_x’)

59. Chef Enables Infrastructure as Code
package "haproxy" do
action :install
end
• Resources template "/etc/haproxy/haproxy.cfg" do
source "haproxy.cfg.erb"
• Recipes owner "root"
group "root"
• Cookbooks mode 0644
•
notifies :restart, "service[haproxy]"
Roles end
• Data Bags service "haproxy" do
• Source Code supports :restart => true
action [:enable, :start]
end

60. Chef manages
Resources on Nodes

61. Chef Resources
package "haproxy" do
action :install
end
template "/etc/haproxy/haproxy.cfg" do
source "haproxy.cfg.erb"
owner "root"
group "root"
mode 0644
notifies :restart, "service[haproxy]"
end
service "haproxy" do
supports :restart => true
action [:enable, :start]
end

62. Chef Resources
package "haproxy" do
action :install
end
• Have a type. template "/etc/haproxy/haproxy.cfg" do
source "haproxy.cfg.erb"
owner "root"
group "root"
mode 0644
notifies :restart, "service[haproxy]"
end
service "haproxy" do
supports :restart => true
action [:enable, :start]
end

63. Chef Resources
package "haproxy" do
action :install
end
• Have a type. template "/etc/haproxy/haproxy.cfg" do
•
source "haproxy.cfg.erb"
Have a name. owner "root"
group "root"
mode 0644
notifies :restart, "service[haproxy]"
end
service "haproxy" do
supports :restart => true
action [:enable, :start]
end

64. Chef Resources
package "haproxy" do
action :install
end
• Have a type. template "/etc/haproxy/haproxy.cfg" do
•
source "haproxy.cfg.erb"
Have a name. owner "root"
• Have parameters. group "root"
mode 0644
notifies :restart, "service[haproxy]"
end
service "haproxy" do
supports :restart => true
action [:enable, :start]
end

65. Chef Resources
package "haproxy" do
action :install
end
• Have a type. template "/etc/haproxy/haproxy.cfg" do
•
source "haproxy.cfg.erb"
Have a name. owner "root"
• Have parameters. group "root"
mode 0644
• Take action to put the resource notifies :restart, "service[haproxy]"
end
in the declared state.
service "haproxy" do
supports :restart => true
action [:enable, :start]
end

66. Chef Resources
package "haproxy" do
action :install
end
• Have a type. template "/etc/haproxy/haproxy.cfg" do
•
source "haproxy.cfg.erb"
Have a name. owner "root"
• Have parameters. group "root"
mode 0644
• Take action to put the resource notifies :restart, "service[haproxy]"
end
in the declared state.
• Can send notifications to other service "haproxy" do
supports :restart => true
resources. action [:enable, :start]
end

67. Resources take action
through Providers

68. Chef Providers
• Multiple providers per resource type.
• For example the `package` resource
has providers for: Apt, Yum, Rubygems,
Portage, Macports, FreeBSD Ports, etc.
• If you are developer think of it as an
interface and an implementation.

69. Recipes are collections
of Resources

70. Chef Recipes
package "haproxy" do
action :install
end
template "/etc/haproxy/haproxy.cfg" do
source "haproxy.cfg.erb"
• Recipes are evaluated for owner "root"
resources in the order they group "root"
mode 0644
appear. notifies :restart, "service[haproxy]"
• Each resource object is added end
to the Resource Collection. service "haproxy" do
supports :restart => true
action [:enable, :start]
end

71. Chef Recipes
• Recipes can include other include_recipe
include_recipe
"apache2"
"apache2::mod_rewrite"
recipes. include_recipe "apache2::mod_deflate"
• Included recipes are
include_recipe
include_recipe
"apache2::mod_headers"
"apache2::mod_php5"
processed in order.

72. Chef Recipes
%w{ python python-dev }.each do |pkg|
package pkg do
action :install
end
end
pool_members = search("node", "role:django_cms")
template "/etc/haproxy/haproxy.cfg" do
source "haproxy.cfg.erb"
owner "root"
group "root"
mode 0644
variables :pool_members => pool_members
notifies :restart, "service[haproxy]"
end

73. Chef Recipes
%w{ python python-dev }.each do |pkg|
package pkg do
action :install
end
end
• Extend recipes with
pool_members = search("node", "role:django_cms")
Ruby.
template "/etc/haproxy/haproxy.cfg" do
source "haproxy.cfg.erb"
owner "root"
group "root"
mode 0644
variables :pool_members => pool_members
notifies :restart, "service[haproxy]"
end

74. Chef Recipes
%w{ python python-dev }.each do |pkg|
package pkg do
action :install
end
end
• Extend recipes with
pool_members = search("node", "role:django_cms")
Ruby.
• Dynamic configuration template "/etc/haproxy/haproxy.cfg" do
source "haproxy.cfg.erb"
through search. owner "root"
group "root"
mode 0644
variables :pool_members => pool_members
notifies :restart, "service[haproxy]"
end

75. Cookbooks are
packages for recipes

76. Cookbook Metadata
• Cookbooks are like packages.
• Cookbooks have dependencies.

77. Cookbooks are easy to share.
community.opscode.com

78. Chef Roles
name "django_cms"
description "django cms app server"
run_list(
"recipe[mysql::client]",
"recipe[application]"
)
name "django_cms_load_balancer"
description "Django CMS load balancer"
run_list(
"recipe[haproxy::app_lb]"
)
override_attributes(
"haproxy" => {
"app_server_role" => "django_cms"
}
)

79. Chef Roles
name "django_cms"
description "django cms app server"
run_list(
"recipe[mysql::client]",
"recipe[application]"
)
• Roles describe nodes. name "django_cms_load_balancer"
description "Django CMS load balancer"
run_list(
"recipe[haproxy::app_lb]"
)
override_attributes(
"haproxy" => {
"app_server_role" => "django_cms"
}
)

80. Chef Roles
name "django_cms"
description "django cms app server"
run_list(
"recipe[mysql::client]",
"recipe[application]"
)
• Roles describe nodes. name "django_cms_load_balancer"
• Roles have a run list.
description "Django CMS load balancer"
run_list(
"recipe[haproxy::app_lb]"
)
override_attributes(
"haproxy" => {
"app_server_role" => "django_cms"
}
)

81. Chef Roles
name "django_cms"
description "django cms app server"
run_list(
"recipe[mysql::client]",
"recipe[application]"
)
• Roles describe nodes. name "django_cms_load_balancer"
• Roles have a run list.
description "Django CMS load balancer"
run_list(
• Roles can have attributes. )
"recipe[haproxy::app_lb]"
override_attributes(
"haproxy" => {
"app_server_role" => "django_cms"
}
)

82. Data bags store
arbitrary data

83. Data Bags
% knife data bag show apps django_cms
database_master_role: ["django_cms_database_master"]
databases:
production:
adapter: mysql
database: django_cms_production
encoding: utf8
password: awesome_password
port: 3306
username: django_cms
deploy_to: /srv/django_cms
group: nogroup
id: django_cms
local_settings_file: local_settings.py
mysql_root_password:
production: mysql_root
owner: nobody
packages:
curl:
pips:
django: 1.2.5
repository: https://github.com/opscode/django-quick-start-cms.git
revision:
production: master
server_roles: ["django_cms"]
type:
django_cms:
django
gunicorn

84. Track it like source code...
% git log
commit d640a8c6b370134d7043991894107d806595cc35
Author: jtimberman <joshua@opscode.com>
Import nagios version 1.0.0
commit c40c818498710e78cf73c7f71e722e971fa574e7
Author: jtimberman <joshua@opscode.com>
installation and usage instruction docs
commit 99d0efb024314de17888f6b359c14414fda7bb91
Author: jtimberman <joshua@opscode.com>
Import haproxy version 1.0.1
commit c89d0975ad3f4b152426df219fee0bfb8eafb7e4
Author: jtimberman <joshua@opscode.com>
add mediawiki cookbook
commit 89c0545cc03b9be26f1db246c9ba4ce9d58a6700
Author: jtimberman <joshua@opscode.com>
multiple environments in data bag for mediawiki

85. Ponies ,
(green )Unicorns and
n-tier oh my!

86. N-Tier FTW
• Provision

87. N-Tier FTW
Load Balancer
App Server App Server
• Provision
• Configure
Database Master

88. N-Tier FTW
Load Balancer
• Provision
App Server App Server
• Configure
• Integrate
Database Master

89. Chef Enables Systems Integration

90. Chef Enables Systems Integration
• Chef searches allow nodes to be aware of
each other and self configure

91. Chef Enables Systems Integration
• Chef searches allow nodes to be aware of
each other and self configure
• A shared application data bag allows:

92. Chef Enables Systems Integration
• Chef searches allow nodes to be aware of
each other and self configure
• A shared application data bag allows:
• many app servers to checkout the same
application code during initial
bootstrapping

93. Chef Enables Systems Integration
• Chef searches allow nodes to be aware of
each other and self configure
• A shared application data bag allows:
• many app servers to checkout the same
application code during initial
bootstrapping
• app server and database servers to
initialize off of the same database config

94. Database Master

95. Database Master
% knife role show django_cms_database_master
default_attributes: {}
description: db master for the django_cms app.
env_run_lists: {}
json_class: Chef::Role
name: django_cms_database_master
override_attributes: {}
run_list: recipe[database::master]

96. Database Master

97. Database Master
• Install and configure MySQL

98. Database Master
• Install and configure MySQL
• Create the application database

99. Database Master
• Install and configure MySQL
• Create the application database
• Grant the application user access

100. Django App Servers
(1..n)

101. Django App Servers
% knife role show django_cms
chef_type: role
default_attributes: {}
description: django_cms front end application server.
env_run_lists: {}
json_class: Chef::Role
name: django_cms
override_attributes: {}
run_list: recipe[mysql::client], recipe[application]

102. Django App Servers

103. Django App Servers
• Deploy the Code

104. Django App Servers
• Deploy the Code
• checkout the code with git

105. Django App Servers
• Deploy the Code
• checkout the code with git
• install required packages (os and python)

106. Django App Servers
• Deploy the Code
• checkout the code with git
• install required packages (os and python)
• create environment specific config files

107. Django App Servers
• Deploy the Code
• checkout the code with git
• install required packages (os and python)
• create environment specific config files
• Serve the Code

108. Django App Servers
• Deploy the Code
• checkout the code with git
• install required packages (os and python)
• create environment specific config files
• Serve the Code
• install Gunicorn (or apache2/mod_wsgi)

109. Django App Servers
• Deploy the Code
• checkout the code with git
• install required packages (os and python)
• create environment specific config files
• Serve the Code
• install Gunicorn (or apache2/mod_wsgi)
• create an app specific config file (vhost)

110. Search for Database Master
results = search(:node, "role:django_cms_database_master")
template "/srv/django_cms/shared/local_settings.py" do
source "settings.py.erb"
mode "644"
variables(
:host => results[0]['fqdn']
)
end

111. Load Balancer

112. Load Balancer
% knife role show django_cms_load_balancer
chef_type: role
default_attributes: {}
description: Django CMS load balancer
env_run_lists: {}
json_class: Chef::Role
name: django_cms_load_balancer
override_attributes:
haproxy:
app_server_role: django_cms
run_list: recipe[haproxy::app_lb]

113. Load Balancer
• The balancing pool should grow as more
application servers are brought online

114. Search for App Servers
pool_members = search(:node, "role:django_cms")
template "/etc/haproxy/haproxy.cfg" do
source "haproxy-app_lb.cfg.erb"
owner "root"
group "root"
mode 0644
variables :pool_members => pool_members
notifies :restart, "service[haproxy]"
end

115. One set of codez....
...to rule them all

116. Same Code Different Configurations

117. Same Code Different Configurations
• We can deploy the same app stack on:

118. Same Code Different Configurations
• We can deploy the same app stack on:
• a single node or a fully redundant
cluster of nodes.

119. Same Code Different Configurations
• We can deploy the same app stack on:
• a single node or a fully redundant
cluster of nodes.
• physical hardware or any number of
public/private cloud providers.

120. Same Code Different Configurations
• We can deploy the same app stack on:
• a single node or a fully redundant
cluster of nodes.
• physical hardware or any number of
public/private cloud providers.
• a developer workstation fully virtualized
with Vagrant + VirtualBox

121. Staging/QA - Single Node
# launch stack in your data center
% knife bootstrap 123.123.3.3
-r ‘role[django_cms_database_master]’,
‘role[django_cms]’,
‘role[django_cms_load_balancer]’
# launch stack in EC2
% knife ec2 server create
-r ‘role[django_cms_database_master]’,
‘role[django_cms]’,
‘role[django_cms_load_balancer]’
# launch stack in Rackspace
% knife rackspace server create
-r ‘role[django_cms_database_master]’,
‘role[django_cms]’,
‘role[django_cms_load_balancer]’

122. Production - Multi-Machine Cluster
% knife ec2 server create
-r ‘role[django_cms_database_master]’
% knife ec2 server create
-r ‘role[django_cms]’
% knife ec2 server create
-r ‘role[django_cms]’
% knife ec2 server create
-r ‘role[django_cms_load_balancer]’

123. Developer Workstation
% cat Vagrantfile
Vagrant::Config.run do |config|
config.vm.box = base_box
config.vm.provision :chef_server do |chef|
chef.chef_server_url = "https://api.opscode.com/organizations/foo"
chef.validation_key_path = "~/.chef/private-pems/foo-validator.pem"
chef.validation_client_name = "#{ENV['ORGNAME']}-validator"
chef.provisioning_path = "/etc/chef"
chef.run_list = [
'role[django_cms_database_master]',
'role[django_cms]',
'role[django_cms_load_balancer]'
]
end
end
% vagrant up

124. BONUS: deploy [insert
your favorite web
framework here] stacks

125. Application Cookbook
• Supports the following web stacks:
• Django
• Java Webapps
• Php
• Rails
• Node.js (coming soon)
• Opscode contributed quick starts walk you
through example deployments

126. Further Resources & Closing
• www.opscode.com
• wiki.opscode.com
• @opscode, #opschef
• irc.freenode.net, #chef, #chef-hacking
• lists.opscode.com

127. Questions?
http://www.flickr.com/photos/oberazzi/318947873/

128. Thanks!
schisamo@opscode.com
www.opscode.com