Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

TXLF: Chef- Software Defined Infrastructure Today & Tomorrow

1,293 views

Published on

The open source configuration management and automation framework Chef is used to configure, deploy and manage infrastructure of every sort. In addition to managing Linux, Windows and many other operating systems; Chef may be used to manage network hardware and storage systems. This session will provide an overview of the concepts and capabilities of Chef and discuss upcoming projects and how they fit into the Chef ecosystem.

Published in: Technology, Self Improvement
  • Be the first to comment

TXLF: Chef- Software Defined Infrastructure Today & Tomorrow

  1. 1. Chef: Software Defined Infrastructure Today & Tomorrow Matt Ray TXLF June 14, 2014
  2. 2. Introductions • Matt Ray • Director of Partner Integration at Chef • matt@getchef.com • mattray GitHub|IRC|Twitter
  3. 3. http://www.flickr.com/photos/michaelheiss/3090102907/ Complexity 4
  4. 4. Items of Manipulation (Resources) • Networking • Files • Directories • Symlinks • Mounts • Registry Keys • Powershell Scripts • Users • Groups • Packages • Services • Filesystems • and so much more 5
  5. 5. Application A tale of growth... 6
  6. 6. Application Application Database Add a database 7
  7. 7. Application App Databases Make database redundant 8
  8. 8. App Servers App Databases Application server redundancy 9
  9. 9. App LB App Servers App Databases Add a load balancer 10
  10. 10. App LBs App Servers App Databases Webscale! 11
  11. 11. App LBs App Servers App DB Cache App DBs Now we need a caching layer 12
  12. 12. App LBs App Servers App DB Cache App DBs Infrastructure has a Topology 13
  13. 13. Round Robin DNS App Servers App DB Cache App DBs Floating IP? Your Infrastructure is a Snowflake 14
  14. 14. App LBs App Servers < Shiny! DB slaves Cache DB Cache DBs Complexity Increases Quickly Are we monitoring?? 15
  15. 15. App LBs App Servers DB slaves Cache DB Cache DBs ...and change happens! 16
  16. 16. App LBs App Servers DB slaves Cache DB Cache DBs ...and change happens! Add a Central Log Host Central Log Host 17
  17. 17. App LBs App Servers DB slaves Cache DB Cache DBs ...and change happens! Add a Central Log Host! ! Update syslog.conf on all Nodes Central Log Host 18
  18. 18. Chef Solves This Problem • But you already guessed that, didn’t you? 19
  19. 19. Chef is Infrastructure as Code • Programmatically provision and configure components http://www.flickr.com/photos/louisb/4555295187/ 20
  20. 20. Chef is Infrastructure as Code • Treat like any other code base http://www.flickr.com/photos/louisb/4555295187/ 21
  21. 21. Chef is Infrastructure as Code • Reconstruct business from code repository, data backups, and compute resources http://www.flickr.com/photos/louisb/4555295187/ 22
  22. 22. Chef is Infrastructure as Code • Programmatically provision and configure components • Treat like any other code base • Reconstruct business from code repository, data backup, and compute resourceshttp://www.flickr.com/photos/louisb/4555295187/ 23
  23. 23. Configuration Code • Chef ensures each Node complies with the policy • Policy is determined by the configurations in each Node’s run list • Reduce management complexity through abstraction • Store the configuration of your infrastructure in version control 24
  24. 24. Declarative Interface to Resources • You define the policy in your Chef configuration • Your policy states what state each resource should be in, but not how to get there • Chef-client will pull the policy from the Chef Server and enforce the policy on the Node 25
  25. 25. How does it work? http://i3.kym-cdn.com/photos/images/original/000/046/123/magnets.jpg
  26. 26. Managing Complexity • Organizations • Environments • Roles • Nodes • Recipes • Cookbooks • Search 27
  27. 27. Their Infrastructure Organizations My Infrastructure Your Infrastructure 28
  28. 28. Environments Development Staging Production 29
  29. 29. Environments Define Policy • Environments may include data attributes necessary for configuring your infrastructure, e.g. • The URL of your payment service’s API • The location of your package repository • The version of the Chef configuration files that should be used 30
  30. 30. Roles Load Balancers Application Servers DB Cache Database 31
  31. 31. Roles Define Policy • Roles may include an ordered list of Chef configuration files that should be applied • This list is called a Run List • Order is always important in the Run List • Roles may include data attributes necessary for configuring your infrastructure, for example: • The port that the application server listens on • A list of applications that should be deployed 32
  32. 32. Nodes 33
  33. 33. Node • Each Node will • Belong to one Organization • Belong to one Environment • Have zero or more Roles 34
  34. 34. Nodes Adhere to Policy • The chef-client application runs on each node, which • Gathers the current system configuration of the node • Downloads the desired system configuration policies from the Chef server for that node • Configures the node such that it adheres to those policies 35
  35. 35. Resources • A Resource represents a piece of the system and its desired state • A package that should be installed • A service that should be running • A file that should be generated • A cron job that should be configured • A user that should be managed • and more 36
  36. 36. Resources in Recipes • Resources are the fundamental building blocks of Chef configuration • Resources are gathered into Recipes • Recipes ensure the system is in the desired state 37
  37. 37. Recipes • Configuration files that describe resources and their desired state • Recipes can: • Install and configure software components • Manage files • Deploy applications • Execute other recipes • and more 38
  38. 38. package "apache2" template "/etc/apache2/apache2.conf" do! source "apache2.conf.erb"! owner "root"! group "root"! mode "0644"! variables(:allow_override => "All")! notifies :reload, "service[apache2]"! end service "apache2" do! action [:enable,:start]! supports :reload => true! end Example Recipe
  39. 39. Cookbooks • Recipes are stored in Cookbooks • Cookbooks contain recipes, templates, files, custom resources, etc • Code re-use and modularity http://www.flickr.com/photos/shutterhacks/4474421855/ 40
  40. 40. Run List Node Enterprise Chef chef-client What policy should I follow? 41
  41. 41. Run List Node Enterprise Chef chef-client What policy should I follow? "recipe[ntp::client]" "recipe[users]" "role[webserver]" 42
  42. 42. Run List Enterprise Chef chef-client What policy should I follow? "recipe[ntp::client]" "recipe[users]" "role[webserver]" 43
  43. 43. Run List Specifies Policy • The Run List is an ordered collection of policies that the Node should follow • Chef-client obtains the Run List from the Chef Server • Chef-client ensures the Node complies with the policy in the Run List 44
  44. 44. Search • Search for nodes with Roles • Find Topology Data ! • IP addresses • Hostnames • FQDNs http://www.flickr.com/photos/kathycsus/268677262545
  45. 45. Search for Nodes pool_members = search("node","role:webserver")! ! template "/etc/haproxy/haproxy.cfg" do! source "haproxy-app_lb.cfg.erb"! owner "root"! group "root"! mode 0644! variables :pool_members => pool_members.uniq! notifies :restart, "service[haproxy]"! end 46
  46. 46. Search for Nodes pool_members = search("node","role:webserver")! ! template "/etc/haproxy/haproxy.cfg" do! source "haproxy-app_lb.cfg.erb"! owner "root"! group "root"! mode 0644! variables :pool_members => pool_members.uniq! notifies :restart, "service[haproxy]"! end 47
  47. 47. Pass results into Templates # Set up application listeners here.! listen application 0.0.0.0:80! balance roundrobin! <% @pool_members.each do |member| -%>! server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check! <% end -%>! <% if node["haproxy"]["enable_admin"] -%>! listen admin 0.0.0.0:22002! mode http! stats uri /! <% end -%> 48
  48. 48. Pass results into Templates # Set up application listeners here.! listen application 0.0.0.0:80! balance roundrobin! <% @pool_members.each do |member| -%>! server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check! <% end -%>! <% if node["haproxy"]["enable_admin"] -%>! listen admin 0.0.0.0:22002! mode http! stats uri /! <% end -%> 49
  49. 49. # Set up application listeners here.! listen application 0.0.0.0:80! balance roundrobin! <% @pool_members.each do |member| -%>! server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check! <% end -%>! <% if node["haproxy"]["enable_admin"] -%>! listen admin 0.0.0.0:22002! mode http! stats uri /! <% end -%> Pass results into Templates 50
  50. 50. Jboss App Memcache Postgres Slaves Postgres Master NagiosGraphite So when this... 51
  51. 51. Jboss App Memcache Postgres Slaves Postgres Master NagiosGraphite ...becomes this 52
  52. 52. Memcache Postgres Slaves Postgres Master NagiosGraphite ! ! ! ! ! Jboss App Memcache Postgres Slaves Postgres Master NagiosGraphite ...this can happen automatically 53
  53. 53. NagiosGraphite ! ! ! ! ! NagiosGraphite Memcache Postgres Slaves • Load balancer config • Nagios host ping • Nagios host ssh • Nagios host HTTP • Nagios host app health • Graphite CPU • Graphite Memory • Graphite Disk • Graphite SNMP • Memcache firewall • Postgres firewall Postgres authZ config • 12+ resource changes for 1 node addition Count the Resources Jboss App 54
  54. 54. Which Operating Systems? • Many supported platforms and architectures • Relatively easy to port • Omnibus-Chef • AIX, Arch, Fedora, Gentoo, OmniOS, OpenBSD, Rasbian, SmartOS and more 55
  55. 55. The Chef Community • Apache License, Version 2.0 • Thousands of Individual and Corporate contributors. • Thousands of cookbooks available from the community • http://community.opscode.com
  56. 56. The Chef API and Server • HTTPS, RESTful API w/ JSON, RSA key auth • Infrastructure data store such as node data • Environments • Search Service • Data bags • SSH and Push jobs http://www.flickr.com/photos/core-materials/4419853626/sizes/o/in/photostream/
  57. 57. knife
  58. 58. knife - with the Chef Server • knife node • create / edit / delete • list • knife cookbook ... • knife role ... • knife environment ... 59
  59. 59. knife - with clouds • knife azure • knife cloudstack • knife do • knife ec2 • knife gce • knife hp • knife openstack • knife rackspace • knife vcloud • knife vsphere • ...and many others 60
  60. 60. Virtualization and Containers? • Docker • Hyper-V • KVM • LXC • OpenVZ • Vagrant • Virtualbox • VMware • Xen • Zones 61
  61. 61. Test Kitchen • Integration testing for your infrastructure code • Tests your cookbooks on all the supported platforms with “real” machines 62
  62. 62. ™ Desktop, Virtualization & Cloud • Vagrant • VMware • CloudStack • Eucalyptus • OpenStack • bare metal • AWS • Rackspace • HP • Google • Azure • many others
  63. 63. • AWS • Rackspace • HP • Google • Azure • many others Desktop, Virtualization & Cloud • Vagrant • VMware • CloudStack • Eucalyptus • OpenStack • bare metal ™
  64. 64. How about Storage? • Ceph • EMC • Gluster • NetApp • Nexenta • and more 65
  65. 65. What about Networking? • Routers and switches • Load balancers • Firewalls • SDN 66
  66. 66. Chef Metal • Chef recipes for deploying infrastructure • Libraries for repeatably creating machines and deployments with Chef primitives • Bootstrappers for many infrastructure types • Provisioner nodes, remote command execution 68
  67. 67. Chef Metal: Providers • Cloud • Digital Ocean, EC2, Fog, OpenStack • Virtualization • Vagrant (VirtualBox, Fusion), vSphere • Containers • Docker & LXC • SSH • PXE in progress 69
  68. 68. Chef Metal: Example Recipe machine 'mario' do! recipe 'mydb'! tag 'mydb_master'! end! num_webservers = 1! 1.upto(num_webservers) do |i|! machine "luigi#{i}" do! recipe 'mywebapp'! end! end 70
  69. 69. What does this all mean? •Every infrastructure is a unique snowflake •Infrastructure as Code brings transparency and traceability •Test your deployments at every stage •Use the same infrastructure code for wherever you want to deploy ™
  70. 70. Austin, Texas • Lots of Chef users in Austin • Austin Chef Meetup • Wednesday June 18, Maudies Triangle 8:30-10:30am • Austin DevOps Meetup • www.meetup.com/austin-devops/ • Agile Austin DevOps • Cloud, Docker, OpenStack, etc.. 72
  71. 71. Thanks! Matt Ray matt@getchef.com @mattray

×