Introduction to Chef


Published on

A presentation I gave at the 2013 Salishan High Speed Computing conference. With 2 hours' notice :)

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Introduction to Chef

  1. 1. An Extemporaneous IntroductionTo ChefKevin A. SmithDirector of Server Engineering
  2. 2. Who am I?• Director of Server Engineering @ Opscode• Software developer for 17 years• 7 years with Erlang• Alumni of DCRI, SAS, Red Hat, Basho• Erlang In Practice @ PragProg
  3. 3. Agenda• Infrastructure as Code• Configuration Management• Chef 101• Chef in Large Environments
  4. 4. Infrastructure as Code
  5. 5. Building andmanaginginfrastructureprogrammaticallyInfrastructure as Code
  6. 6. Enable the reconstructionof the business fromnothing but a source coderepository, an applicationdata backup, and baremetal resources.Infrastructure as Code
  7. 7. ConfigurationManagement
  8. 8. The Old Way
  9. 9. Manual Configuration• Labor intensive• Error prone• Hard to reproduce
  10. 10. Scripting• Very brittle• Throw away, one off scripts• grep sed awk perl• curl | bash
  11. 11. File Distribution• NFS mounts• rdist• scp-on-a-for-loop• rsync on cron
  12. 12. This does not scale!for i in `cat servers.txt` ; do scp ntp.conf root@$i:/etc/ntpd.conf ; donefor i in `cat servers.txt` ; do ssh root@$i /etc/init.d/ntpdrestart ; donefor i in `cat servers.txt` ; do ssh root@$i chkconfig ntpdon ; done
  13. 13. See nodes grow.LoadBalancerApplicationServerDatabaseApplicationServer
  14. 14. LoadBalancerApplicationServerDatabaseApplicationServerLoadBalancerDatabaseGrow, nodes. Grow!
  15. 15. Datacenter #1LoadBalancerAppServerDatabaseAppServerLoadBalancerDatabaseDatacenter #2LoadBalancerAppServerDatabaseAppServerLoadBalancerDatabaseInternetThere are a lot of nodes!
  16. 16. A New Way
  17. 17. Declarative Configuration• Define policy• Say what, not how• Abstract interface to resources
  18. 18. Idempotence• Property of adeclarative interface• f(x) = x• Eliminates brittleness• Safe to run over andoverpackage "ntp" doaction :installendtemplate "/etc/ntp.conf" dosource "ntp.conf.erb"owner "root"group "root"mode 0644notifies :restart, "service[ntpd]"endservice "ntpd" doaction [:enable,:start]end
  19. 19. Convergence• Running an agent “converges”a system onto desired state• Fights entropy andunauthorized changes• Update function inputs to dealwith changing requirements$ echo “boom” > /etc/ntp.conf$ chef-client$ grep server /etc/ntp.conf | head -n$ ps -e | grep ntp1799 ? 00:00:00 ntpd$ /etc/init.d/ntpd stop$ chef-clientps -e | grep ntp1822 ? 00:00:00 ntpd
  20. 20. Chef 101
  21. 21. The chef-client runs on yoursystems.
  22. 22. Clients talk to a Chef server.
  23. 23. Client server conversationsare protected with SSL andRSA signatures.
  24. 24. Each system running Chef iscalled a Managed Node.
  25. 25. Chef APIServerRDBMSSearchEngineAssetStoreManaged NodeChefClientSystem Architecture
  26. 26. Nodes have attributes{"kernel": {"machine": "x86_64","name": "Darwin","os": "Darwin","version": "Darwin Kernel Version 10.4.0: Fri Apr 23 18:28:53 PDT2010; root:xnu-1504.7.4~1/RELEASE_I386","release": "10.4.0"},"platform_version": "10.6.4","platform": "mac_os_x","platform_build": "10F569","domain": "local","os": "darwin","current_user": "mray","ohai_time": 1278602661.60043,"os_version": "10.4.0","uptime": "18 days 17 hours 49 minutes 18 seconds","ipaddress": "","hostname": "morbo","fqdn": "morbomorbo.local","uptime_seconds": 1619358}PlatformKernelHostname, etc.
  27. 27. Node attributes aresearchable.$ knife search node ‘platform:mac_os_x’search(:node, ‘platform:mac_os_x’)
  28. 28. Nodes have “to do” lists.
  29. 29. Nodes have a Run List% knife node show -r{"run_list": ["role[base]","role[hadoop-worker]"]}
  30. 30. Nodes can have Roles.
  31. 31. Aspirational Roles• webserver• database_master• monitoring• hadoop-worker
  32. 32. Roles have Attributesand a run list.
  33. 33. Rolesname "hadoop-worker"description "Hadoop cluster member”run_list("role[base]","recipe[java]","recipe[hadoop]",“recipe[hadoop-config]")default_attributes("hadoop-config" => {"config_path" => “/etc/hadoop”})
  34. 34. chef-client configuresresources on managed nodes.cookbook_filetemplateservicepackagedeploygithttp_requestlinkruby_blocklogbashexecuteremote_fileuser
  35. 35. Chef Resources• Have a type.• Have a name.• Have parameters.• Take action to put the resourcein the declared state.• Can send notifications to otherresources.package "apache2" doaction :installendtemplate "/etc/apache2/apache2.conf" dosource "apache2.conf.erb"owner "root"group "root"mode 0644notifies :restart, "service[apache2]"endservice "apache2" dosupports :restart => trueaction [:enable, :start]end
  36. 36. package “hadoop”{yum install hadoopapt-get install hadooppacman sync hadooppkg_add -r hadoopChef Providers
  37. 37. Recipes are collectionsof resources.
  38. 38. Chef Recipes• Resources are evaluated in theorder they appear.package "haproxy" doaction :installendtemplate "/etc/haproxy/haproxy.cfg" dosource "haproxy.cfg.erb"owner "root"group "root"mode 0644notifies :restart, "service[haproxy]"endservice "haproxy" dosupports :restart => trueaction [:enable, :start]end
  39. 39. Chef Recipes• Recipes can include otherrecipes.• Included recipes are alsoevaluated in order.include_recipe "apache2"include_recipe "apache2::mod_rewrite"include_recipe "apache2::mod_deflate"include_recipe "apache2::mod_headers"include_recipe "apache2::mod_php5"
  40. 40. Chef Recipes• Extend recipes withRuby.%w{ php5 php5-dev php5-cgi }.each do |pkg|package pkg doaction :installendend
  41. 41. • Dynamic configurationthrough search.pool_members = search("node", "role:app_server")template "/etc/haproxy/haproxy.cfg" dosource "haproxy.cfg.erb"owner "root"group "root"mode 0644variables :pool_members => pool_membersnotifies :restart, "service[haproxy]"endChef Recipes
  42. 42. Cookbooks are packages forrecipes and related files.
  43. 43. Cookbook Metadatamaintainer "Opscode, Inc."maintainer_email ""license "Apache 2.0"description "Installs/Configures tomcat"long_description, "0.10.3"%w{ java jpackage }.each do |cb|depends cbend%w{ debian ubuntu centos redhat fedora }.each do |os|supports osendrecipe "tomcat::default", "Installs and configures Tomcat"
  44. 44. Cookbooks are Source Code% git logcommit d640a8c6b370134d7043991894107d806595cc35Author: jtimberman <>Import nagios version 1.0.0commit c40c818498710e78cf73c7f71e722e971fa574e7Author: jtimberman <>installation and usage instruction docscommit 99d0efb024314de17888f6b359c14414fda7bb91Author: jtimberman <>Import haproxy version 1.0.1commit c89d0975ad3f4b152426df219fee0bfb8eafb7e4Author: jtimberman <>add mediawiki cookbookcommit 89c0545cc03b9be26f1db246c9ba4ce9d58a6700Author: jtimberman <>multiple environments in data bag for mediawiki
  45. 45. OSS & Community Oriented• Apache 2.0 License• Wiki, mailing lists, shared cookbook repos•• Healthy ecosystem• 20k+ users• Hundreds of contributors• Community tooling: Food Critic,Test Kitchen, Berkshelf
  46. 46. Chef In“Large” Environments
  47. 47. New Server• Ground up rewrite Ruby/C Erlang• Order of magnitude more scalable• 2k nodes 20k+ nodes per server**Depending on specific work load
  48. 48. High Scalability Users• Facebook• Cycle Computing•
  49. 49. Push Execution• Converge infrastructure on demand• Real-timey view of managed infrastructure• Reduces change latency• 4k nodes now, 10k soon
  50. 50. Network Automation• Network provisioning and configuration• VLANs, QoS, etc.• Partnered w/Arista on PoC (Fall 2012)• More coming soon!
  51. 51. ThankYou