The document discusses the concept of software-defined infrastructure and its evolution, highlighting the challenges of managing complex systems and the role of Chef as a solution. Chef enables infrastructure as code, allowing organizations to programmatically provision and manage their resources through configurations stored in version control. It emphasizes the importance of monitoring, modularity, and community contributions in achieving effective infrastructure management.

2. Chef: Software Defined
Infrastructure Today & Tomorrow
Matt Ray
TXLF
June 14, 2014

3. Introductions
• Matt Ray
• Director of Partner
Integration at Chef
• matt@getchef.com
• mattray GitHub|IRC|Twitter

4. http://www.flickr.com/photos/michaelheiss/3090102907/
Complexity
4

5. Items of Manipulation (Resources)
• Networking
• Files
• Directories
• Symlinks
• Mounts
• Registry Keys
• Powershell Scripts
• Users
• Groups
• Packages
• Services
• Filesystems
• and so much more
5

6. Application
A tale of growth...
6

7. Application
Application Database
Add a database
7

8. Application
App Databases
Make database redundant
8

9. App Servers
App Databases
Application server redundancy
9

10. App LB
App Servers
App Databases
Add a load balancer
10

11. App LBs
App Servers
App Databases
Webscale!
11

12. App LBs
App Servers
App DB Cache
App DBs
Now we need a caching layer
12

13. App LBs
App Servers
App DB Cache
App DBs
Infrastructure has a Topology
13

14. Round Robin DNS
App Servers
App DB Cache
App DBs
Floating IP?
Your Infrastructure is a Snowflake
14

15. App LBs
App Servers
< Shiny!
DB slaves
Cache
DB Cache
DBs
Complexity Increases Quickly
Are we monitoring??
15

16. App LBs
App Servers
DB slaves
Cache
DB Cache
DBs
...and change happens!
16

17. App LBs
App Servers
DB slaves
Cache
DB Cache
DBs
...and change happens!
Add a Central Log Host
Central Log Host
17

18. App LBs
App Servers
DB slaves
Cache
DB Cache
DBs
...and change happens!
Add a Central Log Host!
!
Update syslog.conf on
all Nodes
Central Log Host
18

19. Chef Solves This Problem
• But you already
guessed that, didn’t
you?
19

20. Chef is Infrastructure as Code
• Programmatically
provision and
configure components
http://www.flickr.com/photos/louisb/4555295187/
20

21. Chef is Infrastructure as Code
• Treat like any other
code base
http://www.flickr.com/photos/louisb/4555295187/
21

22. Chef is Infrastructure as Code
• Reconstruct business
from code repository,
data backups, and
compute resources
http://www.flickr.com/photos/louisb/4555295187/
22

23. Chef is Infrastructure as Code
• Programmatically
provision and
configure components
• Treat like any other
code base
• Reconstruct business
from code repository,
data backup, and
compute resourceshttp://www.flickr.com/photos/louisb/4555295187/
23

24. Configuration Code
• Chef ensures each Node complies with the policy
• Policy is determined by the configurations in each
Node’s run list
• Reduce management complexity through abstraction
• Store the configuration of your infrastructure in
version control
24

25. Declarative Interface to Resources
• You define the policy in your Chef configuration
• Your policy states what state each resource should
be in, but not how to get there
• Chef-client will pull the policy from the Chef Server
and enforce the policy on the Node
25

26. How does it work?
http://i3.kym-cdn.com/photos/images/original/000/046/123/magnets.jpg

27. Managing Complexity
• Organizations
• Environments
• Roles
• Nodes
• Recipes
• Cookbooks
• Search
27

28. Their Infrastructure
Organizations
My Infrastructure Your Infrastructure
28

29. Environments
Development Staging Production
29

30. Environments Define Policy
• Environments may include data attributes necessary
for configuring your infrastructure, e.g.
• The URL of your payment service’s API
• The location of your package repository
• The version of the Chef configuration files that
should be used
30

31. Roles
Load Balancers
Application Servers
DB Cache
Database
31

32. Roles Define Policy
• Roles may include an ordered list of Chef
configuration files that should be applied
• This list is called a Run List
• Order is always important in the Run List
• Roles may include data attributes necessary for
configuring your infrastructure, for example:
• The port that the application server listens on
• A list of applications that should be deployed
32

33. Nodes
33

34. Node
• Each Node will
• Belong to one Organization
• Belong to one Environment
• Have zero or more Roles
34

35. Nodes Adhere to Policy
• The chef-client application runs on each node, which
• Gathers the current system configuration of the
node
• Downloads the desired system configuration
policies from the Chef server for that node
• Configures the node such that it adheres to those
policies
35

36. Resources
• A Resource represents a piece of the system and its
desired state
• A package that should be installed
• A service that should be running
• A file that should be generated
• A cron job that should be configured
• A user that should be managed
• and more
36

37. Resources in Recipes
• Resources are the fundamental building blocks of
Chef configuration
• Resources are gathered into Recipes
• Recipes ensure the system is in the desired state
37

38. Recipes
• Configuration files that describe resources and their
desired state
• Recipes can:
• Install and configure software components
• Manage files
• Deploy applications
• Execute other recipes
• and more
38

39. package "apache2"
template "/etc/apache2/apache2.conf" do!
source "apache2.conf.erb"!
owner "root"!
group "root"!
mode "0644"!
variables(:allow_override => "All")!
notifies :reload, "service[apache2]"!
end
service "apache2" do!
action [:enable,:start]!
supports :reload => true!
end
Example Recipe

40. Cookbooks
• Recipes are stored in
Cookbooks
• Cookbooks contain recipes,
templates, files, custom
resources, etc
• Code re-use and modularity
http://www.flickr.com/photos/shutterhacks/4474421855/
40

41. Run List
Node
Enterprise
Chef
chef-client
What policy should I follow?
41

42. Run List
Node
Enterprise
Chef
chef-client
What policy should I follow?
"recipe[ntp::client]"
"recipe[users]"
"role[webserver]"
42

43. Run List
Enterprise
Chef
chef-client
What policy should I follow?
"recipe[ntp::client]"
"recipe[users]"
"role[webserver]"
43

44. Run List Specifies Policy
• The Run List is an ordered collection of policies that
the Node should follow
• Chef-client obtains the Run List from the Chef
Server
• Chef-client ensures the Node complies with the
policy in the Run List
44

45. Search
• Search for nodes with Roles
• Find Topology Data
!
• IP addresses
• Hostnames
• FQDNs
http://www.flickr.com/photos/kathycsus/268677262545

46. Search for Nodes
pool_members = search("node","role:webserver")!
!
template "/etc/haproxy/haproxy.cfg" do!
source "haproxy-app_lb.cfg.erb"!
owner "root"!
group "root"!
mode 0644!
variables :pool_members => pool_members.uniq!
notifies :restart, "service[haproxy]"!
end
46

47. Search for Nodes
pool_members = search("node","role:webserver")!
!
template "/etc/haproxy/haproxy.cfg" do!
source "haproxy-app_lb.cfg.erb"!
owner "root"!
group "root"!
mode 0644!
variables :pool_members => pool_members.uniq!
notifies :restart, "service[haproxy]"!
end
47

48. Pass results into Templates
# Set up application listeners here.!
listen application 0.0.0.0:80!
balance roundrobin!
<% @pool_members.each do |member| -%>!
server <%= member[:hostname] %> <%= member[:ipaddress] %>:>
weight 1 maxconn 1 check!
<% end -%>!
<% if node["haproxy"]["enable_admin"] -%>!
listen admin 0.0.0.0:22002!
mode http!
stats uri /!
<% end -%>
48

49. Pass results into Templates
# Set up application listeners here.!
listen application 0.0.0.0:80!
balance roundrobin!
<% @pool_members.each do |member| -%>!
server <%= member[:hostname] %> <%= member[:ipaddress] %>:>
weight 1 maxconn 1 check!
<% end -%>!
<% if node["haproxy"]["enable_admin"] -%>!
listen admin 0.0.0.0:22002!
mode http!
stats uri /!
<% end -%>
49

50. # Set up application listeners here.!
listen application 0.0.0.0:80!
balance roundrobin!
<% @pool_members.each do |member| -%>!
server <%= member[:hostname] %> <%= member[:ipaddress] %>:>
weight 1 maxconn 1 check!
<% end -%>!
<% if node["haproxy"]["enable_admin"] -%>!
listen admin 0.0.0.0:22002!
mode http!
stats uri /!
<% end -%>
Pass results into Templates
50

51. Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite
So when this...
51

52. Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite
...becomes this
52

53. Memcache
Postgres Slaves
Postgres Master
NagiosGraphite
!
!
!
!
!
Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite
...this can happen automatically
53

54. NagiosGraphite
!
!
!
!
!
NagiosGraphite
Memcache
Postgres Slaves
• Load balancer config
• Nagios host ping
• Nagios host ssh
• Nagios host HTTP
• Nagios host app health
• Graphite CPU
• Graphite Memory
• Graphite Disk
• Graphite SNMP
• Memcache firewall
• Postgres firewall
Postgres authZ config
• 12+ resource changes for 1 node addition
Count the Resources
Jboss App
54

55. Which Operating Systems?
• Many supported
platforms and
architectures
• Relatively easy to port
• Omnibus-Chef
• AIX, Arch, Fedora,
Gentoo, OmniOS,
OpenBSD, Rasbian,
SmartOS and more
55

56. The Chef Community
• Apache License, Version 2.0
• Thousands of Individual and Corporate contributors.
• Thousands of cookbooks available from the
community
• http://community.opscode.com

57. The Chef API and Server
• HTTPS, RESTful API w/ JSON, RSA key auth
• Infrastructure data store such as node data
• Environments
• Search Service
• Data bags
• SSH and Push jobs
http://www.flickr.com/photos/core-materials/4419853626/sizes/o/in/photostream/

58. knife

59. knife - with the Chef Server
• knife node
• create / edit / delete
• list
• knife cookbook ...
• knife role ...
• knife environment ...
59

60. knife - with clouds
• knife azure
• knife cloudstack
• knife do
• knife ec2
• knife gce
• knife hp
• knife openstack
• knife rackspace
• knife vcloud
• knife vsphere
• ...and many others
60

61. Virtualization and Containers?
• Docker
• Hyper-V
• KVM
• LXC
• OpenVZ
• Vagrant
• Virtualbox
• VMware
• Xen
• Zones
61

62. Test Kitchen
• Integration testing for your
infrastructure code
• Tests your cookbooks on all
the supported platforms with
“real” machines
62

63. ™
Desktop, Virtualization & Cloud
• Vagrant
• VMware
• CloudStack
• Eucalyptus
• OpenStack
• bare metal
• AWS
• Rackspace
• HP
• Google
• Azure
• many others

64. • AWS
• Rackspace
• HP
• Google
• Azure
• many others
Desktop, Virtualization & Cloud
• Vagrant
• VMware
• CloudStack
• Eucalyptus
• OpenStack
• bare metal
™

65. How about Storage?
• Ceph
• EMC
• Gluster
• NetApp
• Nexenta
• and more
65

66. What about Networking?
• Routers and switches
• Load balancers
• Firewalls
• SDN
66

68. Chef Metal
• Chef recipes for deploying infrastructure
• Libraries for repeatably creating machines and
deployments with Chef primitives
• Bootstrappers for many infrastructure types
• Provisioner nodes, remote command execution
68

69. Chef Metal: Providers
• Cloud
• Digital Ocean, EC2, Fog, OpenStack
• Virtualization
• Vagrant (VirtualBox, Fusion), vSphere
• Containers
• Docker & LXC
• SSH
• PXE in progress
69

70. Chef Metal: Example Recipe
machine 'mario' do!
recipe 'mydb'!
tag 'mydb_master'!
end!
num_webservers = 1!
1.upto(num_webservers) do |i|!
machine "luigi#{i}" do!
recipe 'mywebapp'!
end!
end
70

71. What does this all mean?
•Every infrastructure is a unique
snowflake
•Infrastructure as Code brings
transparency and traceability
•Test your deployments at every stage
•Use the same infrastructure code for
wherever you want to deploy
™

72. Austin, Texas
• Lots of Chef users in Austin
• Austin Chef Meetup
• Wednesday June 18, Maudies
Triangle 8:30-10:30am
• Austin DevOps Meetup
• www.meetup.com/austin-devops/
• Agile Austin DevOps
• Cloud, Docker, OpenStack, etc..
72

73. Thanks!
Matt Ray
matt@getchef.com
@mattray