vBACD - Introduction to Opscode Chef - 2/29


Published on

The shift to cloud computing means that organizations are undergoing a major shift as they develop scale-out infrastructure that can respond to apace of business change faster than ever before. Opscode Chef® is an open-source systems integration framework build specifically for
automating the cloud by making it easy to deploy and scale servers and applications throughout your infrastructure. Join us for this session
containing an introduction to Chef including:

An Overview of Chef
The Chef Architecture
Cookbook Components
System Integration
Live demo launching a Java Stack on Amazon EC2, Rackspace, Ubuntu, and

[Presented as part of the Open Source Build a Cloud program on 2/29/2012 - http://cloudstack.org/about-cloudstack/cloudstack-events.html?categoryid=6]

Published in: Technology
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Contratulations! You have yourself some clooooud. But now what?
  • But then what? 5 minutes later, you can have an entire rack of servers at your disposal. But until you do one important thing, all they ’re doing is sitting around eating electricity and costing you money.
  • Introducing Chef. Hopefully you ’ve already met! Today we're going to talk about what Chef is and what it's good for.
  • APIs are awesome. They ’re what make the Cloud the Cloud. You can provision resources by simply flinging the right combination of packets at the appropriate DNS address.
  • And Chef can help with that. Knife is our command line tool for talking to APIs And we have plugins for all sorts of cloud providers, both public and private. This lets you provision a server, install the chef agent on it and configure it as a database, webserver, tomcat stack or whatever from a single command.
  • Let's walk through the evolution of your infrastructure. Things are going well, you've just started a new project and your new application has come online.
  • As you get your feet under you and get a feel for what you're doing, you move your database to another machine to help handle the overloaded box.
  • Turns out, the database was the bottleneck, so you add another.
  • Demand continues to grow, so you add another application server.
  • You're going to need a load balancer for that of course, so everyone can use the same IP.
  • And things are really taking off now, 2 load balancers, 5 application servers and a pair of databases. We're growing fast!
  • Caching, time to add some
  • This Infrastructure has a Topology. All the nodes are talking to each other and need to know about their individual interests. Maybe you don ’t want to do it that way.
  • How should I know. It ’s your application. Your application is unique, and so is your infrastructure. They evolve organically.
  • And as they evolve, things continue to change as you switch out components and scale
  • And success breeds success, we're going nuts now.
  • And a basic fact about Infrastructure -- it EVOLVES.
  • Currently, the most widely used configuration management strategy is Cloning and Snapshotting. THIS DOES NOT WORK (and you know it.)
  • OK, it's a JBoss stack on PostgreSQL with Nagios monitoring.
  • Policy change time!!! SSH on port 22 is a security liability (OK, maybe not, but stick with the story)
  • First we'll update the sshd_config on 6 golden images
  • We'll have to replace the instances that are there
  • 12 new boxes, be careful not to break anything. We only have 30 minutes
  • IP addresses all changed, since we're in the cloud right? Oh wait, Bob screwed up.
  • Tracking all these changes by hand breaks down fast. Mistakes get made and things get overlooked.
  • Keep track of all the steps required to take bare metal systems to doing their job in the infrastructure. It is all about the policy. Taking all the systems that have been configured to do their job, and make them work together to actually run the infrastructure.
  • How do we do this? WRT Chef, we talk about Fully Automated Infrastructure. Chef provides a framework for fully automating infrastructure, and has some important design principles. Chef makes it easy to reason about your infrastructure at scale and the predictable ordering makes it easy to understand what ’s going on. The declarative Ruby configuration language is easy to read, easy to share and flexible enough to do powerful things. Chef gives you the tools you need to manage large scale infrastructure in a coherent, logical fashion that can be picked up by the next person doing your job.
  • In Chef a Node is an Abstraction of a server. With the chef server, node state data is persisted between runs. The edge node does all the heavy lifting. Resources are the things on Nodes that we manage. ... a collection of Resources that can span nodes and networks. Resources are simple things that you deal with every day as a systems administrator or developer. Resources include files, directories, mounts, routes, users, groups, packages installations, source code deployments, configuration files, and “stuff” in general.
  • All this is arranged in a very specific way, to it acts in concert to provide ...
  • a service. That ’s it. An Application Infrastructure provides a view of all it’s component nodes and their attributes, as well as information that needs to be shared among resources.
  • When dealing with Chef, need to literally “think outside the box”, by shifting your thinking about configuration away from a single system, to that of an Application Infrastructure. The concept of an Infrastructure is an abstract one with a specific technical meaning. When we talk about Infrastructure, we mean..
  • Chef gives you declarative interfaces into the Resources on those Nodes. Being declarative means that you say what you want to do, instead of how to do it. For example, you declare that package foobar-1.2.3 should be installed, or that the directory /var/log/foobar should exist. Chef pulls down policy from the chef-server, ensuring that a node down for maintenance will receive its policy update when it comes back online.
  • Because we use a 3GL for the recipe config files, we can use features of ruby like case statements and iterative loops. Sysadmins don ’t need to be afraid of Ruby, they’ve been dealing with sub-standard programming languages like configuration files for years. They ’re also not limited to what the language tells them they can do.
  • By using Ruby we can make calls to web services, in this case we're calling search against the Chef server for all the users stored there. We're going to iterate over them, create the users, their home directories and write out the authorized_keys file. As you need to do more complex and powerful things with your infrastructure, Chef's use of Ruby will allow you to harness whatever resources you need.
  • The nodes are going to execute their run lists to configure their Resources defined in your Cookbooks and Recipes. The chef-client maintains the state of your machines, and are also responsible for generating the configuration that is the topology of the infrastructure.
  • This is where the sauce is, and what enables systems integration. (back up to previous slide) When provisioning on Clouds, you typically don ’t get to do up front IP address planning So how do you point a web server to its database? You search for it.
  • Chef is hackable! Permissive Apache2 license, vibrant community of awesome folks. More than 360 individual contributors, over 70 corporate contributors. Community is very important to us. That's why we're here.
  • vBACD - Introduction to Opscode Chef - 2/29

    1. 1. Automating Cloud Deployments with Chef [email_address] www.opscode.com Twitter/IRC/GitHub: mattray
    2. 2. <ul><li>U has a cloud </li></ul><ul><li>Now what? </li></ul>http://www.flickr.com/photos/ian_munroe/4758240536/ Congratulations!!!
    3. 3. But then what? http://www.flickr.com/photos/doctorow/2698336843
    4. 4. You need to configure them
    5. 6. APIs are awesome! <ul><li>You can provision compute resources in seconds </li></ul><ul><li>You can provision storage resources in seconds </li></ul><ul><li>That’ s cool. </li></ul>http://www.flickr.com/photos/jdhancock/3634246981/
    6. 7. Chef can help with that <ul><li>knife ec2 server create </li></ul><ul><li>knife rackspace server create </li></ul><ul><li>knife terremark server create </li></ul><ul><li>knife voxel server create </li></ul><ul><li>knife cloudstack server create </li></ul><ul><li>knife kvm server create </li></ul><ul><li>knife vsphere server create </li></ul><ul><li>knife eucalyptus server create </li></ul><ul><li>knife openstack server create </li></ul>http://www.flickr.com/photos/kyz/3122499444/
    7. 8. See Node Application Server
    8. 9. See Nodes Application Server Application Database
    9. 10. See Nodes Grow Application Server Application Databases
    10. 11. Application Servers Application Databases See Nodes Grow
    11. 12. Application Servers Application Databases Load Balancer See Nodes Grow
    12. 13. See Nodes Grow Application Servers Application Databases Load Balancers
    13. 14. See Nodes Grow Application Servers Application Database Cache Load Balancers Application Databases
    14. 15. Tied together with Config Application Servers Application Database Cache Load Balancers Application Databases
    15. 16. Infrastructure is a Snowflake Application Servers Application Database Cache Load Balancers Floating IP? Application Databases
    16. 17. Evolving Complexity Load Balancers Application Servers NoSQL Database Slaves Application Cache Database Cache Database
    17. 18. Complexity Grows Quickly DC1 DC3 DC2
    18. 19. http://www.flickr.com/photos/ 16339684 @N00/2681435235/ And it Continues to Evolve
    19. 20. Golden Images are not the answer <ul><li>Gold is heavy </li></ul><ul><li>Hard to transport </li></ul><ul><li>Hard to mold </li></ul><ul><li>Easy to lose configuration detail </li></ul>http://www.flickr.com/photos/garysoup/2977173063/
    20. 21. Typical Boring Infrastructure Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite
    21. 22. <ul><li>Move SSH off port 22 </li></ul><ul><li>Lets put it on 2022 </li></ul>New Compliance Mandate Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite
    22. 23. <ul><li>edit /etc/ssh/sshd_config </li></ul>6 Golden Image Updates Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite 1 2 3 4 5 6
    23. 24. <ul><li>Delete, launch </li></ul><ul><li>Repeat </li></ul><ul><li>Typically manually </li></ul>8 12 Instance Replacements Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite 1 2 3 4 5 6 7 9 10 11 12
    24. 25. <ul><li>Don't break anything! </li></ul>In a Maintenance Window 5 Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite 1 2 4 5 6 7 8 9 10 11 12 3
    25. 26. <ul><li>Invalid Configs </li></ul><ul><li>Bob just got fired :( </li></ul>With Different IP Addresses? Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite
    26. 27. http://www.flickr.com/photos/francoforeshock/5716969942/ Configuration Desperation
    27. 28. Configuration Management and Automated Systems Integration is the Answer http://www.flickr.com/photos/philliecasablanca/3354734116/
    28. 29. Chef Solves This Problem <ul><li>But you already guessed that, didn ’t you? </li></ul>
    29. 30. Collections of Resources <ul><li>Networking </li></ul><ul><li>Files </li></ul><ul><li>Directories </li></ul><ul><li>Symlinks </li></ul><ul><li>Mounts </li></ul><ul><li>Routes </li></ul><ul><li>Users </li></ul><ul><li>Groups </li></ul><ul><li>Tasks </li></ul><ul><li>Packages </li></ul><ul><li>Software </li></ul><ul><li>Services </li></ul><ul><li>Configurations </li></ul><ul><li>Other Stuff </li></ul>http://www.flickr.com/photos/stevekeys/3123167585/
    30. 31. http://www.flickr.com/photos/glowjangles/4081048126/ Acting in Concert
    31. 32. http://www.flickr.com/photos/28309157@N08/3743455858/ To Provide a Service
    32. 33. Chef is Infrastructure as Code http://www.flickr.com/photos/louisb/4555295187/ <ul><li>Programmatically provision and configure </li></ul><ul><li>Treat like any other code base </li></ul><ul><li>Reconstruct business from code repository, data backup, and bare metal resources. </li></ul>
    33. 34. Declarative Interface to Resources <ul><li>Define policy </li></ul><ul><li>Say what, not how </li></ul><ul><li>Pull not Push </li></ul>http://www.flickr.com/photos/bixentro/2591838509/
    34. 35. That looks like this extra_packages = case node['platform'] when &quot;ubuntu&quot;,&quot;debian&quot; %w{ ruby1.8 ruby1.8-dev rdoc1.8 ri1.8 libopenssl-ruby } end extra_packages.each do |pkg| package pkg do action :install end end
    35. 36. Or this search(:users, '*:*') do |u| user u['id'] do uid u['uid'] shell u['shell'] home &quot;/home/#{u['id']}&quot; end directory &quot;#{home_dir}/.ssh&quot; do owner u['id'] group u['gid'] mode &quot;0700&quot; end template &quot;#{home_dir}/.ssh/authorized_keys&quot; do source &quot;authorized_keys.erb&quot; owner u['id'] group u['id'] mode &quot;0600&quot; variables :ssh_keys => u['ssh_keys'] end end
    36. 37. Recipes and Cookbooks <ul><li>Recipes are collections of Resources </li></ul><ul><li>Cookbooks contain recipes, templates, files, custom resources, etc </li></ul><ul><li>Code re-use and modularity </li></ul><ul><li>Hundreds already on Community.opscode.com </li></ul>http://www.flickr.com/photos/shutterhacks/4474421855/
    37. 38. <ul><li>Chef-Client generates configurations directly on nodes from their run list </li></ul><ul><li>Reduce management complexity through abstraction </li></ul><ul><li>Store the configuration of your programs in version control </li></ul>http://www.flickr.com/photos/ssoosay/5126146763/ Nodes
    38. 39. Upload your infrastructure knife cookbook upload chef-client knife cookbook upload java knife cookbook upload jpackage knife cookbook upload ntp knife cookbook upload sudo knife cookbook upload tomcat knife cookbook upload users knife cookbook upload sample knife role from file base.rb knife role from file tc.rb knife role from file sample.rb knife data bag create users knife data bag from file users mray.json
    39. 40. Build it somewhere #EC2 knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-a7a97dce -f m1.small -d omnibus -r 'role[base],role[tc],role[sample] ’ #Rackspace knife rackspace server create --image 110 --flavor 2 -i ~/.ssh/mray.pem -d omnibus -r 'role[base],role[tc],role[sample] ’ #CloudStack knife cs server create -S &quot;small instance&quot; -T &quot;CentOS 5.5(64-bit) no GUI (KVM)&quot; -i ~/.ssh/mray.pem -d omnibus -r 'role[base],role[tc],role[sample] ’ #Ubuntu Linux VM knife bootstrap test.lab -i ~/.ssh/mray.pem -x ubuntu --sudo -d omnibus -r 'role[base],role[tc],role[sample]'
    40. 41. Tomcat stack deployed ec2-107-21-179-169.compute-1.amazonaws.com [Thu, 23 Feb 2012 03:16:27 +0000] INFO: Chef Run complete in 125.548799554 seconds ec2-107-21-179-169.compute-1.amazonaws.com [Thu, 23 Feb 2012 03:16:27 +0000] INFO: Running report handlers ec2-107-21-179-169.compute-1.amazonaws.com [Thu, 23 Feb 2012 03:16:27 +0000] INFO: Report handlers complete Instance ID: i-ee18148b Flavor: m1.small Image: ami-0c6ebd65 Region: us-east-1 Availability Zone: us-east-1b Security Groups: default SSH Key: mray Root Device Type: instance-store Public DNS Name: ec2-107-21-179-169.compute-1.amazonaws.com Public IP Address: Private DNS Name: ip-10-120-255-91.ec2.internal Private IP Address: Environment: _default Run List: role[base], role[tc], role[sample]
    41. 42. Tomcat stack deployed
    42. 43. Update the sample cookbook $ knife cookbook upload sample Uploading sample [0.0.2] upload complete
    43. 44. Update the nodes $ knife ssh &quot;role:base&quot; &quot;sudo chef-client&quot; -i ~/.ssh/mray.pem -x ubuntu -a cloud.public_ipv4 [Thu, 23 Feb 2012 03:30:22 +0000] INFO: *** Chef 0.10.8 *** [Thu, 23 Feb 2012 03:30:30 +0000] INFO: Run List is [role[base], role[tc], role[sample]] [Thu, 23 Feb 2012 03:30:30 +0000] INFO: Run List expands to [chef-client, ntp, sudo, users::sysadmins, java, tomcat, sample] [Thu, 23 Feb 2012 03:30:30 +0000] INFO: Starting Chef Run for i-ee18148b [Thu, 23 Feb 2012 03:30:30 +0000] INFO: Running start handlers [Thu, 23 Feb 2012 03:30:30 +0000] INFO: Start handlers complete. [Thu, 23 Feb 2012 03:30:36 +0000] INFO: Loading cookbooks [chef-client, java, jpackage, ntp, sample, sudo, tomcat, users] [Thu, 23 Feb 2012 03:30:36 +0000] INFO: Storing updated cookbooks/sample/recipes/default.rb in the cache. <SNIP> [Thu, 23 Feb 2012 03:30:37 +0000] INFO: template[/var/lib/tomcat6/webapps/sample/index.html] updated content [Thu, 23 Feb 2012 03:30:37 +0000] INFO: template[/var/lib/tomcat6/webapps/sample/index.html] sending restart action to service[tomcat] (immediate) [Thu, 23 Feb 2012 03:30:37 +0000] INFO: Processing service[tomcat] action restart (tomcat::default line 42) [Thu, 23 Feb 2012 03:30:45 +0000] INFO: service[tomcat] restarted [Thu, 23 Feb 2012 03:30:45 +0000] INFO: Chef Run complete in 15.170097638 seconds [Thu, 23 Feb 2012 03:30:45 +0000] INFO: Running report handlers [Thu, 23 Feb 2012 03:30:45 +0000] INFO: Report handlers complete
    44. 45. Tomcat stack updated
    45. 46. http://www.flickr.com/photos/kathycsus/2686772625 <ul><li>IP addresses </li></ul><ul><li>Hostnames </li></ul><ul><li>FQDNs </li></ul><ul><li>Search for nodes with Roles </li></ul><ul><li>Find configuration data </li></ul>Search
    46. 47. pool_members = search(&quot;node&quot;,&quot;role:webserver ”) template &quot;/etc/haproxy/haproxy.cfg&quot; do source &quot;haproxy-app_lb.cfg.erb ” owner &quot;root&quot; group &quot;root ” mode 0644 variables :pool_members => pool_members.uniq notifies :restart, &quot;service[haproxy] ” end Pass results into Templates
    47. 48. # Set up application listeners here. listen application balance roundrobin <% @pool_members.each do |memb| -%> server <%= memb[:hostname] %> <%= memb[:ipaddress] %>:> weight 1 maxconn 1 check <% end -%> <% if node[&quot;haproxy&quot;][&quot;enable_admin&quot;] -%> listen admin mode http stats uri / <% end -%> Pass results into Templates
    48. 49. Jboss App Memcache Postgres Slaves Postgres Master So when this Nagios Graphite
    49. 50. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite Becomes this
    50. 51. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite Updates can be automatic
    51. 52. Nagios Graphite Count the resources Jboss App Memcache Postgres Slaves <ul><li>Load balancer config </li></ul><ul><li>Nagios host ping </li></ul><ul><li>Nagios host ssh </li></ul><ul><li>Nagios host HTTP </li></ul><ul><li>Nagios host app health </li></ul><ul><li>Graphite CPU </li></ul><ul><li>Graphite Memory </li></ul><ul><li>Graphite Disk </li></ul><ul><li>Graphite SNMP </li></ul><ul><li>Memcache firewall </li></ul><ul><li>Postgres firewall </li></ul><ul><li>Postgres authZ config </li></ul>12+ resource changes for 1 node addition
    52. 53. http://www.flickr.com/photos/evelynishere/2798236471/ CLONING CANNOT COPE WITH THIS <ul><li>Chef can. </li></ul>
    53. 54. Build anything <ul><li>Simple internal applications </li></ul><ul><li>Complex external applications </li></ul><ul><li>Workstations </li></ul><ul><li>Hadoop clusters </li></ul><ul><li>IaaS infrastructure </li></ul><ul><li>PaaS infrastructure </li></ul><ul><li>SaaS applications </li></ul><ul><li>Storage systems </li></ul><ul><li>You name it </li></ul>http://www.flickr.com/photos/hyku/245010680/
    54. 55. And manage it simply http://www.flickr.com/photos/helico/404640681/ <ul><li>Automatically reconfigure everything </li></ul><ul><li>Linux, Windows, Unixes, BSDs </li></ul><ul><li>Load balancers </li></ul><ul><li>Metrics collection systems </li></ul><ul><li>Monitoring systems </li></ul><ul><li>Cloud migrations become trivial </li></ul>
    55. 56. The Chef Community <ul><li>Apache License, Version 2.0 </li></ul><ul><li>550+ Individual contributors </li></ul><ul><li>100+ Corporate contributors </li></ul><ul><ul><li>Dell, Rackspace,VMware, RightScale, Heroku, and many more </li></ul></ul><ul><li>400+ Community cookbooks </li></ul><ul><li>http://community.opscode.com </li></ul>
    56. 57. Questions? http://www.flickr.com/photos/mrchippy/443960682/ Questions?
    57. 58. Thanks! [email_address] www.opscode.com Twitter/IRC/GitHub: mattray
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.