Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Prepping the Kitchen - Chef Conceptsand Fundamentals    someara@opscode.com      www.opscode.com
Overview• Infrastructure as code• Configuration Management    Strategies•   Chef
Infrastructure as code
Infrastructure"It is common to think in terms of individual machines      rather than view an entire infrastructure as a  ...
.... as code!•   Programmatically provision and    configure•   Treat like any other code base•   Reconstruct operations f...
Considerations                   •     Infrastructure changes over time                   •     Entropy                   ...
Methodology              http://www.flickr.com/photos/drachmann/327122302/
Configuration Management       Strategies
Manual                                    Configuration                                •     Labor intensive               ...
Scripting•   Typically very brittle•   Throw away, one off scripts•   grep sed awk perl•   curl | bash                    ...
File                                                  Distribution                                                  •   NF...
This used to be          awesomefor i in `cat servers.txt` ; do scp ntp.conf root@$i:/etc/ntpd.conf ; donefor i in `cat se...
Declarative                                                        Syntax                                                 ...
Declarative SyntaxDeclarative Tools•   LCFG•   CFEngine•   BCFG2•   Puppet• Chef
Declarative Syntaxpackage "ntp" do action :install                                     Idempotence                        ...
Declarative Syntaxwhile true do  package "ntp" do   action :install                                       Idempotence     ...
Declarative Syntax                                                  Convergence                                           ...
Declarative Syntax                                     Convergenceservice "ntpd" do action [:enable,:start] ignore_failure...
Declarative Syntax# echo “boom” > /etc/ntp.conf ; chef-client                                          Convergence$ grep s...
Config Generation•   Often made by hand (still!?)•   Stop that.•   Generate them based on    database content• Infrastructu...
See NodeApplication
See NodesApplicationApplication Database
See Nodes GrowApplicationApp Databases
See Nodes GrowApp ServersApp Databases
See Nodes GrowApp LB         App ServersApp Databases
See Nodes Grow   App LBs                   App Servers   App Databases
See Nodes Grow  App LBs                 App Servers  App DB Cache  App DBs
Stitched together with configs            App LBs                           App Servers            App DB Cache           ...
Stitched together with configs              App LBs                            App Servers             App DB Cache       ...
Complexity increases quickly           App LBs                 Cache                      App ServersNoSQL            DB C...
Complexity increases very quickly           DC2DC1                       DC3
Generate configs•   Centralized generation•   Version control!•   Distribute with packages, Chef,    git, whatever.        ...
Generate configs•   Local generation directly on nodes•   Reduces management complexity•   No need to distribute•   Version...
Chef
All That Stuff•   Declarative interface to resources•   Database of nodes and their roles•   Grab remote configs•   Genera...
and more!•   Data Driven Infrastructure•   Use APIs to obtain data    •   chef-server, SQL, anything.•   Feed resources pa...
Architecture•   Code Repository•   Chef Server•   Chef Clients•   Data Bags•   Recipes and Cookbooks•   Roles and Run List...
Code Repository     •   Version control     •   Development         workflows     •   Sharing is Caring
Chef Server                 Server                   Server                chef-server                 Server             ...
Chef Clients                                              Server                                                Server    ...
Chef Clients                                                Server                                                  Server...
Run Lists Server   Serverchef-server Server   Server                                              Ohai!   API             ...
Run Lists Server   Serverchef-server Server   Server                                                          Ohai!       ...
Run Lists Server   Serverchef-server Server   Server                                                                      ...
Roles         Server           Server        chef-server         Server           Server                                  ...
Roles Server   Serverchef-server Server   Server                                                       chef-client        ...
Roles Server   Serverchef-server Server   Server                                ntp                                       ...
Bootstrapping   nodes•   Get chef-client installed•   Write run list to a file•   “Press go”                              ...
Bootstrapping nodes                                                     •   knife ec2 server create -r                    ...
Bootstrapping nodes                                  {                                   "kernel": {                      ...
Bootstrapping nodes                                                     •   Run list is requested                         ...
Cookbooks    and Recipes•   Cookbooks contain recipes•   And everything they need to    work•   Templates, files, custom  ...
Cookbooks                                  $ tree -a cookbooks/haproxy/                                         README.md ...
Recipes                               package "haproxy" do                                action :install                 ...
Resources
Resourcespackage "apache2" do version "2.2.11-2ubuntu2.6" action :installendtemplate "/etc/apache2/apache2.conf" do source...
Resources                  package "apache2" do•                   version "2.2.11-2ubuntu2.6"    Have a type    action :i...
Resources                  package "apache2" do•                   version "2.2.11-2ubuntu2.6"    Have a type    action :i...
Resources                      package "apache2" do•                       version "2.2.11-2ubuntu2.6"    Have a type     ...
Resources                                      package "apache2" do•                                       version "2.2.11...
Searchinghttp://www.flickr.com/photos/fotos_medem/3399096196/
Searching                                                       •   All object in Chef server are                         ...
Searching                                                       •   All object in Chef server are                         ...
Searching                                                       •   All object in Chef server are                         ...
Searching                                                       •   All object in Chef server are                         ...
Systems Integrationknife search node role:webserver webservers = search("node", "role:webserver”)
Pass results into Templatespool_members = search("node","role:webserver”)template "/etc/haproxy/haproxy.cfg" do source "ha...
Pass results into Templatespool_members = search("node","role:webserver”)template "/etc/haproxy/haproxy.cfg" do source "ha...
Pass results into Templates# Set up application listeners here.listen application 0.0.0.0:80  balance roundrobin  <% @pool...
Change•   Various ways•   Add or remove a node to the    infrastructure•   Run chef-client
Run chef-client$ grep servers /etc/haproxy/haproxy.cfgservers node2.mylan 10.9.8.10servers node3.mylan 10.9.8.11$ knife ec...
Change Inputs   •     Edit recipes   •     Edit run lists   •     chef-clienthttp://www.flickr.com/photos/dhutchman/128541...
Out of slides!http://www.flickr.com/photos/calonyr11/2630312566/
Questions?    sales@opscode.com     www.opscode.com
Preppingthekitchen 1.0.3
Upcoming SlideShare
Loading in …5
×

Preppingthekitchen 1.0.3

1,359 views

Published on

Prepping the Kitchen : Chef Concepts and Fundamentals sides used at Surge 2011

Published in: Technology
  • Be the first to comment

Preppingthekitchen 1.0.3

  1. 1. Prepping the Kitchen - Chef Conceptsand Fundamentals someara@opscode.com www.opscode.com
  2. 2. Overview• Infrastructure as code• Configuration Management Strategies• Chef
  3. 3. Infrastructure as code
  4. 4. Infrastructure"It is common to think in terms of individual machines rather than view an entire infrastructure as a combined whole"“A good infrastructure, whether departmental,divisional, or enterprise-wide, is a single loosely-coupled virtual machine, with hundreds orthousands of hard drives and CPUs.” -- Bootstrapping an Infrastructure USENIX LISA ’98 http://www.infrastructures.org/papers/bootstrap/bootstrap.html
  5. 5. .... as code!• Programmatically provision and configure• Treat like any other code base• Reconstruct operations from code repository, data backup, and bare metal resources. http://www.flickr.com/photos/louisb/4555295187/
  6. 6. Considerations • Infrastructure changes over time • Entropy • Changing business requirementshttp://www.flickr.com/photos/seatbelt67/502255276/
  7. 7. Methodology http://www.flickr.com/photos/drachmann/327122302/
  8. 8. Configuration Management Strategies
  9. 9. Manual Configuration • Labor intensive • Error prone • Hard to reproduce • Unsustainablehttp://www.flickr.com/photos/pureimaginations/4805330106/
  10. 10. Scripting• Typically very brittle• Throw away, one off scripts• grep sed awk perl• curl | bash http://www.flickr.com/photos/40389360@N00/2428706650/
  11. 11. File Distribution • NFS mounts • rdist • scp-on-a-for-loop • rsync on cronhttp://www.flickr.com/photos/walkadog/4317655660
  12. 12. This used to be awesomefor i in `cat servers.txt` ; do scp ntp.conf root@$i:/etc/ntpd.conf ; donefor i in `cat servers.txt` ; do ssh root@$i /etc/init.d/ntpdrestart ; donefor i in `cat servers.txt` ; do ssh root@$i chkconfig ntpdon ; done• ^ does not scale http://www.flickr.com/photos/alexerde/3479006495
  13. 13. Declarative Syntax • Define policy • Say what, not how • Abstract interface to resources • Enables some interesting behaviorhttp://www.flickr.com/photos/bixentro/2591838509/
  14. 14. Declarative SyntaxDeclarative Tools• LCFG• CFEngine• BCFG2• Puppet• Chef
  15. 15. Declarative Syntaxpackage "ntp" do action :install Idempotence •endcookbook_file "/etc/ntp.conf" do You’ll hear this a lot source "ntp.conf" owner "root" • Property of declarative group "root" interface mode 0644 action :create notifies :restart, “service[ntpd]” • Eliminates brittleness ofend scriptingservice "ntpd" do action [:enable,:start] • Identity function: f(x)=xend
  16. 16. Declarative Syntaxwhile true do package "ntp" do action :install Idempotence • end cookbook_file "/etc/ntp.conf" do You’ll hear this a lot source "ntp.conf" owner "root" • Property of declarative group "root" interface mode 0644 action :create notifies :restart, “service[ntpd]” • Eliminates brittleness of end scripting service "ntpd" do action [:enable,:start] • Identity function: f(x)=x end • Safe to repeatend
  17. 17. Declarative Syntax Convergence • Agents “converge” a system to desired state • Repetition inches closer to desired state • It eventually gets there • SCIENCE!http://www.flickr.com/photos/tolomea/4852616645/
  18. 18. Declarative Syntax Convergenceservice "ntpd" do action [:enable,:start] ignore_failure trueendcookbook_file "/etc/ntp.conf" do • Agents “converge” a system to source "ntp.conf" desired state owner "root" group "root" mode 0644 • Repetition inches closer to action :create desired state • notifies :restart, “service[ntpd]” ignore_failure true It eventually gets there •end SCIENCE!package "ntp" do action :install ignore_failure trueend
  19. 19. Declarative Syntax# echo “boom” > /etc/ntp.conf ; chef-client Convergence$ grep server /etc/ntp.conf | head -n 1us.pool.ntp.org$ ps -e | grep ntp • Fights entropy, unauthorized 1799 ? 00:00:00 ntpd changes, and gingivitis# /etc/init.d/ntpd stop ; chef-client • Update function inputs to deal with changing requirementsps -e | grep ntp 1822 ? 00:00:00 ntpd
  20. 20. Config Generation• Often made by hand (still!?)• Stop that.• Generate them based on database content• Infrastructures evolve http://www.flickr.com/photos/jabella/4753170413/
  21. 21. See NodeApplication
  22. 22. See NodesApplicationApplication Database
  23. 23. See Nodes GrowApplicationApp Databases
  24. 24. See Nodes GrowApp ServersApp Databases
  25. 25. See Nodes GrowApp LB App ServersApp Databases
  26. 26. See Nodes Grow App LBs App Servers App Databases
  27. 27. See Nodes Grow App LBs App Servers App DB Cache App DBs
  28. 28. Stitched together with configs App LBs App Servers App DB Cache App DBs
  29. 29. Stitched together with configs App LBs App Servers App DB Cache Floating IP? App DBs
  30. 30. Complexity increases quickly App LBs Cache App ServersNoSQL DB Cache DB slaves DBs
  31. 31. Complexity increases very quickly DC2DC1 DC3
  32. 32. Generate configs• Centralized generation• Version control!• Distribute with packages, Chef, git, whatever. http://www.flickr.com/photos/ssoosay/5126146763/
  33. 33. Generate configs• Local generation directly on nodes• Reduces management complexity• No need to distribute• Version control the programs instead http://www.flickr.com/photos/ssoosay/5126146763/
  34. 34. Chef
  35. 35. All That Stuff• Declarative interface to resources• Database of nodes and their roles• Grab remote configs• Generate configs locally
  36. 36. and more!• Data Driven Infrastructure• Use APIs to obtain data • chef-server, SQL, anything.• Feed resources parameters • IPs, FQDNs, memory sizes, • Templates, package, firewall rules
  37. 37. Architecture• Code Repository• Chef Server• Chef Clients• Data Bags• Recipes and Cookbooks• Roles and Run Lists http://www.flickr.com/photos/boedker/3871267007
  38. 38. Code Repository • Version control • Development workflows • Sharing is Caring
  39. 39. Chef Server Server Server chef-server Server Server • Upload from laptop with knife RESTful API Cookbook CookbookCookbook Data Bag Knife Knife Role Knife
  40. 40. Chef Clients Server Server chef-server Server Server Knife• Clients are API users• Read RESTful API Knife• Write• Search chef-client chef-client chef-client chef-client chef-client
  41. 41. Chef Clients Server Server someara.pub chef-server Server Server jtimberman.pub node5.fqdn.pub Knife someara.pem• Clients are API users• Public keys on server RESTful API Knife jtimberman.pem• Private keys local to machines chef-client chef-client chef-client chef-client chef-client node5.fqdn.pem
  42. 42. Run Lists Server Serverchef-server Server Server Ohai! API chef-client Give me recipe[ntp::client] ntp node client.rb
  43. 43. Run Lists Server Serverchef-server Server Server Ohai! chef-client API Give me “ntp::client”, ntp “openssh::server” openssh node client.rb server.rb
  44. 44. Run Lists Server Serverchef-server Server Server Ohai! chef-client Give me API “recipe[ntp::client]”, ntp “recipe[openssh::server]”, “recipe[apache]”, openssh node “recipe[php]” client.rb apache server.rb php default.rb default.rb
  45. 45. Roles Server Server chef-server Server Server Role Recipe API Role Role Recipe Role Recipe RecipeKnife Recipe Recipe Recipe
  46. 46. Roles Server Serverchef-server Server Server chef-client Ohai! API Give me ntp “role[base]”, “role[webserver]” openssh node client.rb apache server.rb php default.rb default.rb
  47. 47. Roles Server Serverchef-server Server Server ntp openssh chef-client API client.rb apache php server.rb “role[webserver]” default.rb ntp default.rb node openssh chef-client client.rb mysql server.rb server.rb “role[database]” node
  48. 48. Bootstrapping nodes• Get chef-client installed• Write run list to a file• “Press go” http://www.flickr.com/photos/liftarn/1447521121/
  49. 49. Bootstrapping nodes • knife ec2 server create -r ‘role[webserver]’ • knife bootstrap 10.9.8.7 -r ‘role[webserver]’ • Cobblerhttp://www.flickr.com/photos/hakonjarl/4010080214/
  50. 50. Bootstrapping nodes { "kernel": { "machine": "x86_64", "name": "Darwin", "os": "Darwin", "version": "Darwin Kernel Version 10.4.0: Fri Apr 23 18:28:53 PDT 2010;• root:xnu-1504.7.4~1/RELEASE_I386", Ohai generates a JSON }, "release": "10.4.0" attributes list "platform_version": "10.6.4",• "platform": "mac_os_x", Run list and attributes are "platform_build": "10F569", "domain": "local", combined into a Node object "os": "darwin", "current_user": "mray",• "ohai_time": 1278602661.60043, Can be viewed and "os_version": "10.4.0", "uptime": "18 days 17 hours 49 minutes 18 seconds", searched through API "ipaddress": "10.13.37.116", "hostname": "morbo", "fqdn": "morbomorbo.local", "uptime_seconds": 1619358 }
  51. 51. Bootstrapping nodes • Run list is requested • Cookbooks downloaded • Recipes executed • Node saved to chef-serverhttp://www.flickr.com/photos/architopher/457885721
  52. 52. Cookbooks and Recipes• Cookbooks contain recipes• And everything they need to work• Templates, files, custom resources, etc http://www.flickr.com/photos/shutterhacks/4474421855/
  53. 53. Cookbooks $ tree -a cookbooks/haproxy/ README.md attributes    default.rb• Cookbooks contain recipes metadata.rb• And everything they need to recipes work    app_lb.rb    default.rb• Templates, files, custom templates resources, etc default haproxy-app_lb.cfg.erb haproxy-default.erb haproxy.cfg.erb
  54. 54. Recipes package "haproxy" do action :install end template "/etc/default/haproxy" do source "haproxy-default.erb"• Recipes contain lists of owner "root" group "root" resources mode 0644 notifies :restart, "service[haproxy]" end service "haproxy" do action [:enable, :start] end
  55. 55. Resources
  56. 56. Resourcespackage "apache2" do version "2.2.11-2ubuntu2.6" action :installendtemplate "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "root" group "root" mode 0644 action :createend
  57. 57. Resources package "apache2" do• version "2.2.11-2ubuntu2.6" Have a type action :install end template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "root" group "root" mode 0644 action :create end
  58. 58. Resources package "apache2" do• version "2.2.11-2ubuntu2.6" Have a type action :install• end Have a name template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "root" group "root" mode 0644 action :create end
  59. 59. Resources package "apache2" do• version "2.2.11-2ubuntu2.6" Have a type action :install• end Have a name• template "/etc/apache2/apache2.conf" do Have parameters source "apache2.conf.erb" owner "root" group "root" mode 0644 action :create end
  60. 60. Resources package "apache2" do• version "2.2.11-2ubuntu2.6" Have a type action :install• end Have a name• template "/etc/apache2/apache2.conf" do Have parameters source "apache2.conf.erb" owner "root"• Take action to put the resource group "root" mode 0644 in the declared state action :create end
  61. 61. Searchinghttp://www.flickr.com/photos/fotos_medem/3399096196/
  62. 62. Searching • All object in Chef server are indexed by Solrhttp://www.flickr.com/photos/fotos_medem/3399096196/
  63. 63. Searching • All object in Chef server are indexed by Solr • Can search through the APIhttp://www.flickr.com/photos/fotos_medem/3399096196/
  64. 64. Searching • All object in Chef server are indexed by Solr • Can search through the API • From knife and in recipeshttp://www.flickr.com/photos/fotos_medem/3399096196/
  65. 65. Searching • All object in Chef server are indexed by Solr • Can search through the API • From knife and in recipes • Returns an array of JSON Node objectshttp://www.flickr.com/photos/fotos_medem/3399096196/
  66. 66. Systems Integrationknife search node role:webserver webservers = search("node", "role:webserver”)
  67. 67. Pass results into Templatespool_members = search("node","role:webserver”)template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]"end
  68. 68. Pass results into Templatespool_members = search("node","role:webserver”)template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]"end
  69. 69. Pass results into Templates# Set up application listeners here.listen application 0.0.0.0:80 balance roundrobin <% @pool_members.each do |member| -%> server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1check <% end -%><% if node["haproxy"]["enable_admin"] -%>listen admin 0.0.0.0:22002 mode http stats uri /<% end -%>
  70. 70. Change• Various ways• Add or remove a node to the infrastructure• Run chef-client
  71. 71. Run chef-client$ grep servers /etc/haproxy/haproxy.cfgservers node2.mylan 10.9.8.10servers node3.mylan 10.9.8.11$ knife ec2 server create -r ‘webserver’$ knife ec2 server create -r ‘webserver’$ knife ssh ‘role:webserver’ chef-client$ grep servers /etc/haproxy/haproxy.cfgservers node2.mylan 10.9.8.10servers node3.mylan 10.9.8.11servers node4.mylan 10.9.8.12servers node5.mylan 10.9.8.13
  72. 72. Change Inputs • Edit recipes • Edit run lists • chef-clienthttp://www.flickr.com/photos/dhutchman/128541987
  73. 73. Out of slides!http://www.flickr.com/photos/calonyr11/2630312566/
  74. 74. Questions? sales@opscode.com www.opscode.com

×