Oscon2011 tutorial
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Oscon2011 tutorial

  • 2,753 views
Uploaded on

Presentation for our 3 hour tutorial at OSCON 2011....

Presentation for our 3 hour tutorial at OSCON 2011.

http://www.oscon.com/oscon2011/public/schedule/detail/19882

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
2,753
On Slideshare
2,707
From Embeds
46
Number of Embeds
1

Actions

Shares
Downloads
44
Comments
0
Likes
0

Embeds 46

http://lanyrd.com 46

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. OSCON 2011: Getting Started with Chef Joshua Timberman joshua@opscode.com, @jtimberman Aaron Peterson aaron@opscode.com, @metaxis http://opscode.comMonday, July 25, 2011
  • 2. Meta Information • OSCON tutorials are recorded • Rate the tutorial and comment • http://bit.ly/chef-oscon2011 • Twitter: • #oscon • @opscode, #opschef • @jtimberman, @metaxis • Slides and Code will be postedMonday, July 25, 2011
  • 3. Who are we? • Joshua Timberman • Aaron PetersonMonday, July 25, 2011
  • 4. Who are you? • System administrators? • Developers? • “Business” People? http://www.flickr.com/photos/timyates/2854357446/sizes/l/Monday, July 25, 2011
  • 5. Agenda • Tutorial Logistics • Hows and whys • Getting Started • Anatomy of a Chef Run • Hands on configuring a node • Common patterns & best practices • Question/Answer http://www.flickr.com/photos/koalazymonkey/3590953001/Monday, July 25, 2011
  • 6. What are we talking about here? http://www.flickr.com/photos/peterkaminski/2174679908/Monday, July 25, 2011
  • 7. Managing Infrastructure is Hard • Sysadmins: • Setup production machines • Manage deployed application(s)Monday, July 25, 2011
  • 8. System administrators... • Install packages • Configure running services • OS settings • User management • Monitoring and trending integrationMonday, July 25, 2011
  • 9. Managing Infrastructure is Hard • Developers: • Setup local machine • Deploy application for testingMonday, July 25, 2011
  • 10. Developers... • Developers want self-service • Full application stack • Abstract the detailsMonday, July 25, 2011
  • 11. Automation is GoodMonday, July 25, 2011
  • 12. Automation is Good • Operable • Reasonable • Flexible • RepeatableMonday, July 25, 2011
  • 13. Monday, July 25, 2011
  • 14. The Chef Framework • Reasonability • Flexibility • Library & Primitives • TIMTOWTDIMonday, July 25, 2011
  • 15. The Chef Tool(s) • ohai - information gathering • chef-client - configuration agent • knife - command-line API tool • shef - console debuggerMonday, July 25, 2011
  • 16. The Chef API • RSA key authentication w/ Signed Headers • RESTful API w/ JSON • Search Service • Derivative ServicesMonday, July 25, 2011
  • 17. The Chef Community • Apache License, Version 2.0 • 360+ Individual contributors • 70+ Corporate contributors • Dell, Rackspace,VMware, RightScale, Basho Technologies, and more • http://community.opscode.com • 260+ cookbooksMonday, July 25, 2011
  • 18. Getting Started with Chef git clone git://github.com/opscode/oscon2011-chef-repoMonday, July 25, 2011
  • 19. Required Software • SSH/SCP • Git • Build toolchain (gcc and friends) • Ruby (1.8.7 or 1.9.2) • RubyGems (1.3.7+) • Chef (0.10.0+) git clone git://github.com/opscode/oscon2011-chef-repoMonday, July 25, 2011
  • 20. Why Opscode Hosted Chef? • Limited time for tutorial • Free up to 5 nodes • Chef Server API • Open Source Chef Server git clone git://github.com/opscode/oscon2011-chef-repoMonday, July 25, 2011
  • 21. Source Code Repository • Chef Repository for OSCON 2011 • git clone git://github.com/opscode/oscon2011-chef-repoMonday, July 25, 2011
  • 22. Files from Opscode Hosted Chef Signup • Knife configuration • .chef/knife.rb • User certificate • .chef/USER.pem • Validation certificate • .chef/ORGNAME-validator.pem git clone git://github.com/opscode/oscon2011-chef-repoMonday, July 25, 2011
  • 23. Verify Access % knife client list oscon2011-validator % knife node from file dummy.example.com.json Updated Node dummy.example.com! % knife node list dummy.example.com % knife node show dummy.example.com Node Name: dummy.example.com Environment: _default FQDN: dummy.example.com IP: 10.1.1.1 Run List: Roles: Recipes Platform: centos 5.5Monday, July 25, 2011
  • 24. Virtual Machine Setup • Setup outside scope of this tutorial • Linux Virtual Machine or Cloud Instance • SSH access as root or user w/ sudoMonday, July 25, 2011
  • 25. A quick tour of ChefMonday, July 25, 2011
  • 26. Chef runs on your systemsMonday, July 25, 2011
  • 27. API Clients authenticate to the Chef ServerMonday, July 25, 2011
  • 28. Each system running Chef is a managed NodeMonday, July 25, 2011
  • 29. Nodes have attributes and a list of things to runMonday, July 25, 2011
  • 30. Roles are a description of what a node should beMonday, July 25, 2011
  • 31. Chef configures Resources on your systemsMonday, July 25, 2011
  • 32. Recipes are lists of resourcesMonday, July 25, 2011
  • 33. Cookbooks are packages for Recipes and related filesMonday, July 25, 2011
  • 34. Let’s manage some infrastructure...Monday, July 25, 2011
  • 35. Managing Infrastructure • Write or download cookbooks • Create a role that uses the cookbooks • Deploy cookbooks and role to Chef Server • Apply the role to a node • Run Chef on the nodeMonday, July 25, 2011
  • 36. Anatomy of a Chef RunMonday, July 25, 2011
  • 37. Profile the Node with OhaiMonday, July 25, 2011
  • 38. Run Ohai • Run `ohai | less` on your system. • Marvel at the amount of data it returns.Monday, July 25, 2011
  • 39. AuthenticateMonday, July 25, 2011
  • 40. Retrieve Node from Chef ServerMonday, July 25, 2011
  • 41. Sync Cookbooks from Chef ServerMonday, July 25, 2011
  • 42. Load CookbooksMonday, July 25, 2011
  • 43. Load RecipesMonday, July 25, 2011
  • 44. ConvergeMonday, July 25, 2011
  • 45. Save Node to Chef ServerMonday, July 25, 2011
  • 46. Break Time • Questions from 1st half • Hands on in 2nd half http://www.flickr.com/photos/refractedmoments/65794219/Monday, July 25, 2011
  • 47. Questions? http://www.flickr.com/photos/oberazzi/318947873/Monday, July 25, 2011
  • 48. Reasoning about InfrastructureMonday, July 25, 2011
  • 49. Reasoning about Infrastructure • Break down complexity into components you can think about. • Think about commonality and differences between systems and applications. • Capture these in roles.Monday, July 25, 2011
  • 50. Reasoning about Infrastructure • For a given application, think about requirements to fulfill its job. • Think about how to meet the requirements.Monday, July 25, 2011
  • 51. Concrete use case • Stand in for common patterns • Things we want on all systems in the infrastructure. • User management • Essential network service (NTP)Monday, July 25, 2011
  • 52. Upload Chef Repository % knife role from file base.rb % knife cookbook upload -a % knife data bag create users % knife data bag from file users luke.json % knife data bag from file users leia.jsonMonday, July 25, 2011
  • 53. Configure a node • Invoke action from the local workstation to happen on a remote machine over SSH. • Virtual Machine IP address • SSH key or password for root/privileged (sudo) user • Optional: Use a cloud computing provider (See README.md)Monday, July 25, 2011
  • 54. Knife Bootstrap knife bootstrap FQDN (options) -d DISTRO Target a specific distro (default ubuntu) -i IDENTITY_FILE SSH identity file for authentication -r RUN_LIST Run list for the node -P PASSWORD The ssh password -x USERNAME The ssh username (default root) --sudo Execute bootstrap with sudo % knife bootstrap --help % knife help bootstrap # full man page!Monday, July 25, 2011
  • 55. Bootstrap Cloud Instances • Knife works with Cloud providers through plugins • Knife Cloud plugins use Fog • Cloud instances are launched via their API then provisioned with bootstrap • Additional RubyGems • knife-ec2, knife-rackspace, etc • Additional Knife ConfigurationMonday, July 25, 2011
  • 56. Configure a node # Append -Ppassword or -i ~/.ssh/ssh-private-key-for-you to ssh # Ubuntu: knife bootstrap $IPADDRESS -r role[base] knife bootstrap $IPADDRESS -r role[base] -x ubuntu --sudo # Debian 6: knife bootstrap $IPADDRESS -r role[base] -x root knife bootstrap $IPADDRESS -r role[base] -x username --sudo # CentOS 5.x: knife bootstrap $IPADDRESS -r role[base] -d centos5-gems knife bootstrap $IPADDRESS -r role[base] -d centos5-gems -x username --sudo # Scientific Linux 6.x: knife bootstrap $IPADDRESS -r role[base] -d scientific6-gems knife bootstrap $IPADDRESS -r role[base] -d scientific6-gems -x username --sudo # Example (Ubuntu 10.04): knife bootstrap 172.16.156.130 -r role[base] -x jtimberman --sudo -Poscon2011Monday, July 25, 2011
  • 57. What happened on the node?Monday, July 25, 2011
  • 58. recipe[ntp] INFO: Processing package[ntp] action install (ntp::default line 27) INFO: package[ntp] installed version 1:4.2.4p8+dfsg-1ubuntu2.1 INFO: Processing package[ntp] action install (ntp::default line 27) INFO: package[ntp] installed version 1:4.2.4p8+dfsg-1ubuntu2.1 INFO: Processing template[/etc/ntp.conf] action create (ntp::default line 31) INFO: template[/etc/ntp.conf] backed up to /var/chef/backup/etc/ ntp.conf.chef-20110717131907 INFO: template[/etc/ntp.conf] mode changed to 644 INFO: template[/etc/ntp.conf] updated content INFO: Processing service[ntp] action enable (ntp::default line 39) INFO: Processing service[ntp] action start (ntp::default line 39) [ ... end of run (delayed) ... ] INFO: template[/etc/ntp.conf] sending restart action to service[ntp] (delayed) INFO: Processing service[ntp] action restart (ntp::default line 39) INFO: service[ntp] restartedMonday, July 25, 2011
  • 59. SSH to the Node and inspect % ssh 172.16.156.130 % dpkg -l ntp ii ntp 1:4.2.4p8+dfsg Network Time Protocol daemon and % grep server /etc/ntp.conf server 0.pool.ntp.org server 1.pool.ntp.org % /etc/init.d/ntp status * NTP server is running % ls /etc/rc2.d/*ntp /etc/rc2.d/S23ntpMonday, July 25, 2011
  • 60. recipe[users::sysadmin] INFO: Processing user[luke] action create (users::sysadmins line 41) INFO: user[luke] created INFO: Processing directory[/home/luke/.ssh] action create (users::sysadmins line 51) INFO: directory[/home/luke/.ssh] created directory /home/luke/.ssh INFO: directory[/home/luke/.ssh] owner changed to 2001 INFO: directory[/home/luke/.ssh] group changed to 2001 INFO: directory[/home/luke/.ssh] mode changed to 700 INFO: Processing template[/home/luke/.ssh/authorized_keys] action create (users::sysadmins line 57) INFO: template[/home/luke/.ssh/authorized_keys] owner changed to 2001 INFO: template[/home/luke/.ssh/authorized_keys] owner changed to 2001 INFO: template[/home/luke/.ssh/authorized_keys] updated content INFO: Processing user[leia] action create (users::sysadmins line 41) INFO: user[leia] created INFO: Processing directory[/home/leia/.ssh] action create (users::sysadmins line 51) INFO: directory[/home/leia/.ssh] created directory /home/leia/.ssh INFO: directory[/home/leia/.ssh] owner changed to 2002 INFO: directory[/home/leia/.ssh] group changed to 2002 INFO: directory[/home/leia/.ssh] mode changed to 700 INFO: Processing template[/home/leia/.ssh/authorized_keys] action create (users::sysadmins line 57) INFO: template[/home/leia/.ssh/authorized_keys] owner changed to 2002 INFO: template[/home/leia/.ssh/authorized_keys] owner changed to 2002 INFO: template[/home/leia/.ssh/authorized_keys] updated content INFO: Processing group[sysadmin] action create (users::sysadmins line 66) INFO: group[sysadmin] createdMonday, July 25, 2011
  • 61. recipe[users::sysadmins] % ssh 172.16.156.130 % getent passwd luke leia luke:x:2001:2001:Force is strong with this one:/home/luke:/bin/bash leia:x:2002:2002:There is another:/home/leia:/bin/bash # ls ~{luke,leia}/.ssh /home/luke/.ssh: authorized_keys /home/leia/.ssh: authorized_keysMonday, July 25, 2011
  • 62. recipe[sudo] INFO: Processing package[sudo] action upgrade (sudo::default line 20) INFO: Processing template[/etc/sudoers] action create (sudo::default line 24) INFO: template[/etc/sudoers] backed up to /var/chef/backup/ etc/sudoers.chef-20110717131908 INFO: template[/etc/sudoers] mode changed to 440 INFO: template[/etc/sudoers] updated contentMonday, July 25, 2011
  • 63. recipe[sudo] # grep ALL /etc/sudoers root ALL=(ALL) ALL %sysadmin ALL=(ALL) ALLMonday, July 25, 2011
  • 64. What happened on the Chef Server?Monday, July 25, 2011
  • 65. Chef Repository on Chef Server % knife role list base % knife cookbook list ntp 1.0.0 sudo 1.0.0 users 1.0.0 % knife data bag list users % knife data bag show users leia lukeMonday, July 25, 2011
  • 66. Base Role % knife role show base chef_type: role default_attributes: {} description: Base role applied to all systems env_run_lists: {} json_class: Chef::Role name: base override_attributes: {} run_list: recipe[ntp], recipe[users::sysadmins], recipe[sudo]Monday, July 25, 2011
  • 67. NTP Cookbook package "ntp" do action :install end template "/etc/ntp.conf" do source "ntp.conf.erb" owner "root" group "root" mode 0644 notifies :restart, "service[ntp]" end service "ntp" do action [:enable, :start] endMonday, July 25, 2011
  • 68. NTP configuration (template) template "/etc/ntp.conf" do source "ntp.conf.erb" owner "root" group "root" mode 0644 notifies :restart, "service[ntp]" end Template source: <% node[:ntp][:servers].each do |ntpserver| -%> server <%= ntpserver %> <% end -%> <% end -%> Cookbook Attributes: default[:ntp][:servers] = ["0.pool.ntp.org", "1.pool.ntp.org"]Monday, July 25, 2011
  • 69. NTP service management template "/etc/ntp.conf" do # ... notifies :restart, "service[ntp]" end service "ntp" do action [:enable, :start] endMonday, July 25, 2011
  • 70. Sysadmin users data bag items % cat data_bags/users/luke.json { "id": "luke", "ssh_keys": "ssh-rsa For example purposes only", "groups": "sysadmin", "uid": 2001, "shell": "/bin/bash", "comment": "Force is strong with this one" } % cat data_bags/users/leia.json { "id": "leia", "ssh_keys": "ssh-rsa For example purposes only", "groups": "sysadmin", "uid": 2002, "shell": "/bin/bash", "comment": "There is another" }Monday, July 25, 2011
  • 71. users::sysadmins recipe search(:users, groups:sysadmin) do |u| user u[id] do uid u[uid] gid u[id] shell u[shell] comment u[comment] supports :manage_home => true home "/home/#{u[uid]}" end directory "#{home_dir}/.ssh" do owner u[id] group u[id] mode "0700" end template "#{home_dir}/.ssh/authorized_keys" do source "authorized_keys.erb" owner u[id] group u[id] mode "0600" variables :ssh_keys => u[ssh_keys] end endMonday, July 25, 2011
  • 72. Sudo cookbook package "sudo" do action :upgrade end template "/etc/sudoers" do source "sudoers.erb" mode 0440 owner "root" group "root" variables( :sudoers_groups => node[authorization][sudo][groups], :sudoers_users => node[authorization][sudo][users], :passwordless => node[authorization][sudo][passwordless] ) endMonday, July 25, 2011
  • 73. Sudoers template Template source: root ALL=(ALL) ALL %sysadmin ALL=(ALL) <%= "NOPASSWD:" if @passwordless %>ALL Cookbook attributes: default[authorization][sudo][passwordless] = false Rendered content: root ALL=(ALL) ALL %sysadmin ALL=(ALL) ALLMonday, July 25, 2011
  • 74. Nodes % knife node list dummy.example.com ubuntu1004test.example.com % knife node show ubuntu1004test.example.com Node Name: ubuntu1004test.example.com Environment: _default FQDN: ubuntu1004test.example.com IP: 172.16.156.130 Run List: role[base] Roles: base Recipes ntp, users::sysadmins, sudo Platform: ubuntu 10.04 % knife node show --help % knife help nodeMonday, July 25, 2011
  • 75. Searching the Server # Search nodes: % knife search node "role:base" % knife search node "platform:ubuntu" % knife search node "platform:centos" # Search roles: % knife search role "run_list:recipe[users*" # Search data bags (bag name is the index): % knife search users "groups:sysadmin" % knife search users "shell:*bash"Monday, July 25, 2011
  • 76. Common Patterns and Best PracticesMonday, July 25, 2011
  • 77. Common Patterns • Install a package • Update a configuration file • Restart a serviceMonday, July 25, 2011
  • 78. Common Patterns • Search for nodes with a particular role • Search for data bag items • Make decisions or render templates based on search results.Monday, July 25, 2011
  • 79. Search example in a recipe pool_members = search("node", "role:webserver") template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]" endMonday, July 25, 2011
  • 80. Common Patterns • Ask questions about the infrastructure. • Target a subset of servers and take action. • Search with Roles • Search with Node Attributes • Parallel execution of commands.Monday, July 25, 2011
  • 81. Operational Use Case % knife ssh platform:ubuntu vmstat xwing.example.com procs -----------memory---------- ... xwing.example.com r b swpd free buff cache ... xwing.example.com 0 0 0 684804 461656 6052916 ... tiefighter.example.com procs -----------memory---------- ... tiefighter.example.com r b swpd free buff cache ... tiefighter.example.com 0 0 0 169020 708844 6120008 ...Monday, July 25, 2011
  • 82. Best Practices: Cookbooks • Publicly shared cookbooks: • http://community.opscode.com • Create your own • knife cookbook create foo • $EDITOR cookbooks/foo/recipes/default.rbMonday, July 25, 2011
  • 83. Getting Community Cookbooks # Install apache2 cookbook from site in Git chef-repo % knife cookbook site install apache2 # Download and install apache2 cookbook in non-Git chef-repo % knife cookbook site download apache2 % tar -zxf apache2-VERSION.tar.gz -C cookbooksMonday, July 25, 2011
  • 84. Best Practices: Cookbooks • Cookbook for each service • Recipe for each component or deployment of the service • Set sane defaults in attributes files • Modify attributes through roles for specific usage requirementsMonday, July 25, 2011
  • 85. Best Practices: Roles • Roles are descriptions • webserver • database_master • load_balancer • Set role-specific attributes when necessary • listen ports, deploy locations, etcMonday, July 25, 2011
  • 86. Best Practices: Nodes • Use “Just Enough OS” • Use fully updated systems • Kickstart, AMI, etc • Ensure system clock is synchronized • Be ready to deploy from scratchMonday, July 25, 2011
  • 87. Managing Resources • Chef’s primary purpose is managing resources on nodes. • Think in terms of resources vs commands • Chef comes with 28 kinds of resources • You can create your own resources in cookbooksMonday, July 25, 2011
  • 88. Thinking in terms of resources • package vs yum install • service vs chkconfig • template vs echo ‘coolstuff’ >> /etc/config • or sed ‘s/badstuff/coolstuff/’... • mode, owner and group parameters vs chmod/chown • http://wiki.opscode.com/display/chef/ResourcesMonday, July 25, 2011
  • 89. FAQ: Chef vs [Other Tool]Monday, July 25, 2011
  • 90. http://www.flickr.com/photos/gesika22/4458155541/Monday, July 25, 2011
  • 91. FAQ: How do you test recipes?Monday, July 25, 2011
  • 92. FAQ: Testing • You launch cloud instances and watch them converge. • You use Vagrant with a Chef ProvisionerMonday, July 25, 2011
  • 93. FAQ: Testing • You buy Stephen Nelson-Smith’s book!Monday, July 25, 2011
  • 94. FAQ: How does Chef scale?Monday, July 25, 2011
  • 95. FAQ: Scale • The Chef Server is a publishing system. • Nodes do the heavy lifting. • Chef scales like a service-oriented web application. • Opscode Hosted Chef was designed and built for massive scale. http://www.flickr.com/photos/amagill/61205408/Monday, July 25, 2011
  • 96. Questions? • http://bit.ly/chef-oscon2011 • http://opscode.com • @opscode, #opschef • irc.freenode.net, #chef, #chef-hacking • http://lists.opscode.com http://www.flickr.com/photos/oberazzi/318947873/Monday, July 25, 2011
  • 97. Thanks! http://opscode.com @opscode #opschefMonday, July 25, 2011