Preppingthekitchen 1.0.3

1,251 views
1,175 views

Published on

Prepping the Kitchen : Chef Concepts and Fundamentals sides used at Surge 2011

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,251
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
18
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • Specific, technical meaning.\nThink of an infrastructure holistically \nGraphic from Infrastructures.org\n.... 13 years ago\n\n
  • We live in the future\nWe have the technology\nlet’s write us some infrastructure!\num... okay so how do we do that\n\n
  • Let’s think about this...\n\n
  • \nWhat do we code?\n\n
  • \n
  • Lovingly hand crafted systems\n\n
  • curl that into your pipe and bash it\n
  • \n
  • (still is)\n
  • Run locally on all nodes\nPull from server\nResource primitives are packages, files, directories, symlinks, mounts, routes, etc.\n
  • \n
  • Don’t talk about resources and providers yet. Save that for later.\nStress repeatability of individual operations\n
  • Don’t talk about resources and providers yet. Save that for later.\nStress repeatability.\n
  • Chef would actually fail here\n
  • 2 runs to converge onto state\nfail, succeed, succeed\nbroken state\nsucceed, succeed, noop\n3rd run: noop, noop, noop\n\nORDER MATTERS BITCHES\n
  • time + declarations == convergence\n^ don’t say that\n
  • This goes for configuration files as well as registries or database settings\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Snowflakes.\nYour application is unique, and so is your infrastructure.\nThey evolve symbiotically.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • SSL encryption and stuff\n
  • \n
  • \n
  • executed in order!\n
  • executed in order!\n
  • executed in order!\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Chef Resources are declarative.\n
  • Chef Resources are declarative.\n
  • Chef Resources are declarative.\n
  • Chef Resources are declarative.\n
  • Chef Resources are declarative.\n
  • Chef Resources are declarative.\n
  • Chef Resources are declarative.\n
  • Chef Resources are declarative.\n
  • Chef Resources are declarative.\n
  • Chef Resources are declarative.\n
  • Chef Resources are declarative.\n
  • Chef Resources are declarative.\n
  • Chef Resources are declarative.\n
  • Chef Resources are declarative.\n
  • Chef Resources are declarative.\n
  • Chef Resources are declarative.\n
  • Chef Resources are declarative.\n
  • Chef Resources are declarative.\n
  • Chef Resources are declarative.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • (still is)\n
  • \n
  • (still is)\n
  • \n
  • \n
  • Preppingthekitchen 1.0.3

    1. 1. Prepping the Kitchen - Chef Conceptsand Fundamentals someara@opscode.com www.opscode.com
    2. 2. Overview• Infrastructure as code• Configuration Management Strategies• Chef
    3. 3. Infrastructure as code
    4. 4. Infrastructure"It is common to think in terms of individual machines rather than view an entire infrastructure as a combined whole"“A good infrastructure, whether departmental,divisional, or enterprise-wide, is a single loosely-coupled virtual machine, with hundreds orthousands of hard drives and CPUs.” -- Bootstrapping an Infrastructure USENIX LISA ’98 http://www.infrastructures.org/papers/bootstrap/bootstrap.html
    5. 5. .... as code!• Programmatically provision and configure• Treat like any other code base• Reconstruct operations from code repository, data backup, and bare metal resources. http://www.flickr.com/photos/louisb/4555295187/
    6. 6. Considerations • Infrastructure changes over time • Entropy • Changing business requirementshttp://www.flickr.com/photos/seatbelt67/502255276/
    7. 7. Methodology http://www.flickr.com/photos/drachmann/327122302/
    8. 8. Configuration Management Strategies
    9. 9. Manual Configuration • Labor intensive • Error prone • Hard to reproduce • Unsustainablehttp://www.flickr.com/photos/pureimaginations/4805330106/
    10. 10. Scripting• Typically very brittle• Throw away, one off scripts• grep sed awk perl• curl | bash http://www.flickr.com/photos/40389360@N00/2428706650/
    11. 11. File Distribution • NFS mounts • rdist • scp-on-a-for-loop • rsync on cronhttp://www.flickr.com/photos/walkadog/4317655660
    12. 12. This used to be awesomefor i in `cat servers.txt` ; do scp ntp.conf root@$i:/etc/ntpd.conf ; donefor i in `cat servers.txt` ; do ssh root@$i /etc/init.d/ntpdrestart ; donefor i in `cat servers.txt` ; do ssh root@$i chkconfig ntpdon ; done• ^ does not scale http://www.flickr.com/photos/alexerde/3479006495
    13. 13. Declarative Syntax • Define policy • Say what, not how • Abstract interface to resources • Enables some interesting behaviorhttp://www.flickr.com/photos/bixentro/2591838509/
    14. 14. Declarative SyntaxDeclarative Tools• LCFG• CFEngine• BCFG2• Puppet• Chef
    15. 15. Declarative Syntaxpackage "ntp" do action :install Idempotence •endcookbook_file "/etc/ntp.conf" do You’ll hear this a lot source "ntp.conf" owner "root" • Property of declarative group "root" interface mode 0644 action :create notifies :restart, “service[ntpd]” • Eliminates brittleness ofend scriptingservice "ntpd" do action [:enable,:start] • Identity function: f(x)=xend
    16. 16. Declarative Syntaxwhile true do package "ntp" do action :install Idempotence • end cookbook_file "/etc/ntp.conf" do You’ll hear this a lot source "ntp.conf" owner "root" • Property of declarative group "root" interface mode 0644 action :create notifies :restart, “service[ntpd]” • Eliminates brittleness of end scripting service "ntpd" do action [:enable,:start] • Identity function: f(x)=x end • Safe to repeatend
    17. 17. Declarative Syntax Convergence • Agents “converge” a system to desired state • Repetition inches closer to desired state • It eventually gets there • SCIENCE!http://www.flickr.com/photos/tolomea/4852616645/
    18. 18. Declarative Syntax Convergenceservice "ntpd" do action [:enable,:start] ignore_failure trueendcookbook_file "/etc/ntp.conf" do • Agents “converge” a system to source "ntp.conf" desired state owner "root" group "root" mode 0644 • Repetition inches closer to action :create desired state • notifies :restart, “service[ntpd]” ignore_failure true It eventually gets there •end SCIENCE!package "ntp" do action :install ignore_failure trueend
    19. 19. Declarative Syntax# echo “boom” > /etc/ntp.conf ; chef-client Convergence$ grep server /etc/ntp.conf | head -n 1us.pool.ntp.org$ ps -e | grep ntp • Fights entropy, unauthorized 1799 ? 00:00:00 ntpd changes, and gingivitis# /etc/init.d/ntpd stop ; chef-client • Update function inputs to deal with changing requirementsps -e | grep ntp 1822 ? 00:00:00 ntpd
    20. 20. Config Generation• Often made by hand (still!?)• Stop that.• Generate them based on database content• Infrastructures evolve http://www.flickr.com/photos/jabella/4753170413/
    21. 21. See NodeApplication
    22. 22. See NodesApplicationApplication Database
    23. 23. See Nodes GrowApplicationApp Databases
    24. 24. See Nodes GrowApp ServersApp Databases
    25. 25. See Nodes GrowApp LB App ServersApp Databases
    26. 26. See Nodes Grow App LBs App Servers App Databases
    27. 27. See Nodes Grow App LBs App Servers App DB Cache App DBs
    28. 28. Stitched together with configs App LBs App Servers App DB Cache App DBs
    29. 29. Stitched together with configs App LBs App Servers App DB Cache Floating IP? App DBs
    30. 30. Complexity increases quickly App LBs Cache App ServersNoSQL DB Cache DB slaves DBs
    31. 31. Complexity increases very quickly DC2DC1 DC3
    32. 32. Generate configs• Centralized generation• Version control!• Distribute with packages, Chef, git, whatever. http://www.flickr.com/photos/ssoosay/5126146763/
    33. 33. Generate configs• Local generation directly on nodes• Reduces management complexity• No need to distribute• Version control the programs instead http://www.flickr.com/photos/ssoosay/5126146763/
    34. 34. Chef
    35. 35. All That Stuff• Declarative interface to resources• Database of nodes and their roles• Grab remote configs• Generate configs locally
    36. 36. and more!• Data Driven Infrastructure• Use APIs to obtain data • chef-server, SQL, anything.• Feed resources parameters • IPs, FQDNs, memory sizes, • Templates, package, firewall rules
    37. 37. Architecture• Code Repository• Chef Server• Chef Clients• Data Bags• Recipes and Cookbooks• Roles and Run Lists http://www.flickr.com/photos/boedker/3871267007
    38. 38. Code Repository • Version control • Development workflows • Sharing is Caring
    39. 39. Chef Server Server Server chef-server Server Server • Upload from laptop with knife RESTful API Cookbook CookbookCookbook Data Bag Knife Knife Role Knife
    40. 40. Chef Clients Server Server chef-server Server Server Knife• Clients are API users• Read RESTful API Knife• Write• Search chef-client chef-client chef-client chef-client chef-client
    41. 41. Chef Clients Server Server someara.pub chef-server Server Server jtimberman.pub node5.fqdn.pub Knife someara.pem• Clients are API users• Public keys on server RESTful API Knife jtimberman.pem• Private keys local to machines chef-client chef-client chef-client chef-client chef-client node5.fqdn.pem
    42. 42. Run Lists Server Serverchef-server Server Server Ohai! API chef-client Give me recipe[ntp::client] ntp node client.rb
    43. 43. Run Lists Server Serverchef-server Server Server Ohai! chef-client API Give me “ntp::client”, ntp “openssh::server” openssh node client.rb server.rb
    44. 44. Run Lists Server Serverchef-server Server Server Ohai! chef-client Give me API “recipe[ntp::client]”, ntp “recipe[openssh::server]”, “recipe[apache]”, openssh node “recipe[php]” client.rb apache server.rb php default.rb default.rb
    45. 45. Roles Server Server chef-server Server Server Role Recipe API Role Role Recipe Role Recipe RecipeKnife Recipe Recipe Recipe
    46. 46. Roles Server Serverchef-server Server Server chef-client Ohai! API Give me ntp “role[base]”, “role[webserver]” openssh node client.rb apache server.rb php default.rb default.rb
    47. 47. Roles Server Serverchef-server Server Server ntp openssh chef-client API client.rb apache php server.rb “role[webserver]” default.rb ntp default.rb node openssh chef-client client.rb mysql server.rb server.rb “role[database]” node
    48. 48. Bootstrapping nodes• Get chef-client installed• Write run list to a file• “Press go” http://www.flickr.com/photos/liftarn/1447521121/
    49. 49. Bootstrapping nodes • knife ec2 server create -r ‘role[webserver]’ • knife bootstrap 10.9.8.7 -r ‘role[webserver]’ • Cobblerhttp://www.flickr.com/photos/hakonjarl/4010080214/
    50. 50. Bootstrapping nodes { "kernel": { "machine": "x86_64", "name": "Darwin", "os": "Darwin", "version": "Darwin Kernel Version 10.4.0: Fri Apr 23 18:28:53 PDT 2010;• root:xnu-1504.7.4~1/RELEASE_I386", Ohai generates a JSON }, "release": "10.4.0" attributes list "platform_version": "10.6.4",• "platform": "mac_os_x", Run list and attributes are "platform_build": "10F569", "domain": "local", combined into a Node object "os": "darwin", "current_user": "mray",• "ohai_time": 1278602661.60043, Can be viewed and "os_version": "10.4.0", "uptime": "18 days 17 hours 49 minutes 18 seconds", searched through API "ipaddress": "10.13.37.116", "hostname": "morbo", "fqdn": "morbomorbo.local", "uptime_seconds": 1619358 }
    51. 51. Bootstrapping nodes • Run list is requested • Cookbooks downloaded • Recipes executed • Node saved to chef-serverhttp://www.flickr.com/photos/architopher/457885721
    52. 52. Cookbooks and Recipes• Cookbooks contain recipes• And everything they need to work• Templates, files, custom resources, etc http://www.flickr.com/photos/shutterhacks/4474421855/
    53. 53. Cookbooks $ tree -a cookbooks/haproxy/ README.md attributes    default.rb• Cookbooks contain recipes metadata.rb• And everything they need to recipes work    app_lb.rb    default.rb• Templates, files, custom templates resources, etc default haproxy-app_lb.cfg.erb haproxy-default.erb haproxy.cfg.erb
    54. 54. Recipes package "haproxy" do action :install end template "/etc/default/haproxy" do source "haproxy-default.erb"• Recipes contain lists of owner "root" group "root" resources mode 0644 notifies :restart, "service[haproxy]" end service "haproxy" do action [:enable, :start] end
    55. 55. Resources
    56. 56. Resourcespackage "apache2" do version "2.2.11-2ubuntu2.6" action :installendtemplate "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "root" group "root" mode 0644 action :createend
    57. 57. Resources package "apache2" do• version "2.2.11-2ubuntu2.6" Have a type action :install end template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "root" group "root" mode 0644 action :create end
    58. 58. Resources package "apache2" do• version "2.2.11-2ubuntu2.6" Have a type action :install• end Have a name template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "root" group "root" mode 0644 action :create end
    59. 59. Resources package "apache2" do• version "2.2.11-2ubuntu2.6" Have a type action :install• end Have a name• template "/etc/apache2/apache2.conf" do Have parameters source "apache2.conf.erb" owner "root" group "root" mode 0644 action :create end
    60. 60. Resources package "apache2" do• version "2.2.11-2ubuntu2.6" Have a type action :install• end Have a name• template "/etc/apache2/apache2.conf" do Have parameters source "apache2.conf.erb" owner "root"• Take action to put the resource group "root" mode 0644 in the declared state action :create end
    61. 61. Searchinghttp://www.flickr.com/photos/fotos_medem/3399096196/
    62. 62. Searching • All object in Chef server are indexed by Solrhttp://www.flickr.com/photos/fotos_medem/3399096196/
    63. 63. Searching • All object in Chef server are indexed by Solr • Can search through the APIhttp://www.flickr.com/photos/fotos_medem/3399096196/
    64. 64. Searching • All object in Chef server are indexed by Solr • Can search through the API • From knife and in recipeshttp://www.flickr.com/photos/fotos_medem/3399096196/
    65. 65. Searching • All object in Chef server are indexed by Solr • Can search through the API • From knife and in recipes • Returns an array of JSON Node objectshttp://www.flickr.com/photos/fotos_medem/3399096196/
    66. 66. Systems Integrationknife search node role:webserver webservers = search("node", "role:webserver”)
    67. 67. Pass results into Templatespool_members = search("node","role:webserver”)template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]"end
    68. 68. Pass results into Templatespool_members = search("node","role:webserver”)template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]"end
    69. 69. Pass results into Templates# Set up application listeners here.listen application 0.0.0.0:80 balance roundrobin <% @pool_members.each do |member| -%> server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1check <% end -%><% if node["haproxy"]["enable_admin"] -%>listen admin 0.0.0.0:22002 mode http stats uri /<% end -%>
    70. 70. Change• Various ways• Add or remove a node to the infrastructure• Run chef-client
    71. 71. Run chef-client$ grep servers /etc/haproxy/haproxy.cfgservers node2.mylan 10.9.8.10servers node3.mylan 10.9.8.11$ knife ec2 server create -r ‘webserver’$ knife ec2 server create -r ‘webserver’$ knife ssh ‘role:webserver’ chef-client$ grep servers /etc/haproxy/haproxy.cfgservers node2.mylan 10.9.8.10servers node3.mylan 10.9.8.11servers node4.mylan 10.9.8.12servers node5.mylan 10.9.8.13
    72. 72. Change Inputs • Edit recipes • Edit run lists • chef-clienthttp://www.flickr.com/photos/dhutchman/128541987
    73. 73. Out of slides!http://www.flickr.com/photos/calonyr11/2630312566/
    74. 74. Questions? sales@opscode.com www.opscode.com

    ×