• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Novalug 07142012
 

Novalug 07142012

on

  • 1,209 views

Intro to Chef at Novalug, July 14, 2012

Intro to Chef at Novalug, July 14, 2012

Statistics

Views

Total Views
1,209
Views on SlideShare
1,207
Embed Views
2

Actions

Likes
1
Downloads
6
Comments
0

2 Embeds 2

http://www.linkedin.com 1
https://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • \n
  • \n
  • Your business is about applications - through one lens, the entire business is a single, very complicated application\n
  • Those applications are supported by Infrastructure - literally all the stuff that’s needed to run the application.\n\nWhen dealing with Chef, need to literally “think outside the box”, by shifting your thinking about configuration away from a single system, to that of an Appliction Infrastructure. The concept of an Infrastructure is an abstract one with a specific technical meaning. When we talk about Infrastructure, we mean.. \n
  • ... a collection of Resources that can span nodes and networks. Resources are simple things that you deal with every day as a systems administrator or developer. Resources include files, directories, mounts, routes, users, groups, packages installations, source code deployments, configuration files, and “stuff” in general.\n\n
  • All this is arranged in a very specific way, to it acts in concert to provide ... \n
  • a service. That’s it. An Application Infrastructure provides a view of all it’s component nodes and their attributes, as well as information that needs to be shared among resources.\n
  • And a basic fact about Infrastructure -- it EVOLVES.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • an Infrastructure has a Topology. Maybe you don’t want to do it that way.\n
  • \nHow should I know. It’s your application.\n\nYour application is unique, and so is your infrastructure.\nThey evolve symbiotically.\n\nReference earlier, when we said you bring the knowledge about your systems\n\n\n
  • \n
  • \n
  • The answer to all the complexity is configuration management - you need to ‘manage’ the ‘configuration’ of all that stuff\n
  • Currently, the most widely used configuration management strategy is Cloning and Snapshotting.\nTHIS DOES NOT WORK (and you know it.)\n\n
  • OK\n
  • Policy change time!!!\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • How do we do this?\nWe let you write \n
  • PROGRAMS\n\nThese programs maintain the state of your machines, and are also responsible for generating the configuration that is the topology of the infrastructure.\n\n\n
  • Chef gives you declarative interfaces into resources.\n\nBeing declarative means that you say what you want to do, instead of how to do it.\nFor example, \nyou declare that package foobar-1.2.3 should be installed, or that the directory /var/log/foobar should exist.\n\nChef pulls down policy from the chef-server, ensuring that a node down for maintenance will receive its policy update when it comes back online.\n\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • executed in order!\n
  • executed in order!\n
  • executed in order!\n
  • \n
  • \n
  • executed in order!\n
  • \n
  • \n
  • This is where the sauce is, and what enables systems integration.\n(back up to previous slide)\nWhen provisioning on Clouds, you typically don’t get to do up front IP address planning\nSo how do you point a web server to its database?\nYou search for it.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Before we go further, let's get all the terminology out on the table.\n\nThese slides can simply be shown and read quickly, they're for awareness, and will be explained in further detail throughout the workshop.\n
  • When we're configuring systems with Chef, we run the client program, "chef-client" on the individual nodes. They're responsible for keeping themselves configured. We can run chef-client from cron, as a service, with runit or daemontools, by hand... as often or as rarely as we want to, it’s up to you!\n
  • The chef-client program connects to the configured Chef Server, whether it’s hosted chef, private chef, or open source chef\n
  • API Clients are the thing that authenticate to the Server using RSA keypairs. Chef-client uses an API client on behalf of the system it is running as, \n
  • A node is a thing that Chef is managing. It can be an Ec2 instance, a Virtual Machine, a Rackmounted server in a data center.\n
  • Attributes are merely data about the node. IPs, kernel modules, memory, platform and version. Ohai gave them to us, and we can add our own based on what’s needed in our applications.\n
  • Run lists are arrays, so they're ordered. They can contain roles or recipes.\n
  • \n
  • Resources are the abstracted declarative interface to a particular thing that should be configured on the node. Packages, services, templates, etc.\n\nThey describe WHAT, not HOW.\n
  • Providers are the underlying implementation that take care of the HOW for WHAT was declared in the resource.\n\nThey are intended to be idempotent, that is, they do not take action if they do not need to. "multiple application of the same thing do not change the output."\n\n(Not mathematical reference of idempotent functions)\n
  • Within Chef, the providers do all the heavy lifting for you. Package resources can have apt or yum or gem or other providers.\n
  • \n
  • Because there weren't enough package management systems out there...\n\nCookbooks contain recipes and other supporting code for configuring a particular thing, like apache, mysql, nginx, or trac\n
  • Knife is a plugin-based system that includes a number of plugins that work with the Chef server, and it can be extended for other purposes as we'll see!\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

Novalug 07142012 Novalug 07142012 Presentation Transcript

  • Overview of Chef Mandi Walls novalug July 14, 2012
  • whoami• Mandi Walls• mandi@opscode.com• @lnxchk
  • Applicationshttp://www.flickr.com/photos/steffenz/337700069/ http://www.flickr.com/photos/kky/704056791/
  • Infrastructur http://www.flickr.com/photos/sbh/462754460/
  • Collection of•Nodes •Routes•Networking •Users•Files •Groups•Directories •Tasks•Symlinks •Packages•Mounts •Software •Services •Configurations •Stuff http://www.flickr.com/photos/stevekeys/3123167585/
  • Acting in concert http://www.flickr.com/photos/glowjangles/4081048126/
  • To provide a Service http://www.flickr.com/photos/28309157@N08/3743455858/
  • And it evolves http://www.flickr.com/photos/16339684@N00/2681435235/
  • See NodeApplication
  • See NodesApplicationApplication Database
  • See Nodes GrowApplicationApp Databases
  • See Nodes GrowApp ServersApp Databases
  • See Nodes GrowApp LB App ServersApp Databases
  • See Nodes Grow App LBs App Servers App Databases
  • See Nodes Grow App LBs App Servers App DB Cache App DBs
  • Stitched together with configs App LBs App Servers App DB Cache App DBs
  • Your Infrastructure is a snow flake App LB App Servers App DB Cache Floating IP? App DBs
  • Complexity increases quickly App LBs Cache App ServersNoSQL DB Cache DB slaves DBs
  • Complexity increases very quickly DC2DC1 DC3
  • Configuration Management http://www.flickr.com/photos/philliecasablanca/3354734116/
  • Golden Images are not the answer•Gold is heavy•Hard to transport•Hard to mold•Easy to lose configuration detail http://www.flickr.com/photos/garysoup/2977173063/
  • Typical Boring InfrastructureGraphite Nagios Jboss App Memcache Postgres Slaves Postgres Master
  • New Compliance Mandate Graphite Nagios Jboss App Memcache• Move SSH off port 22• Lets put it on 2022 Postgres Slaves Postgres Master
  • 6 Golden Image Updates Graphite 1 2 Nagios 3 Jboss App 4 Memcache 5 Postgres Slaves• edit /etc/ssh/sshd_config 6 Postgres Master
  • 12 Instance Replacements Graphite 1 2 Nagios 3 4 5 6 7 Jboss App• Delete, launch 8 9 Memcache• Repeat• Typically manually 10 11 Postgres Slaves 12 Postgres Master
  • Done in Maintenance WindowsGraphite 1 2 Nagios 3 4 5 6 7 Jboss App 8 9 Memcache 5 10 11 Postgres Slaves 12 Postgres Master
  • Different IP Addresses? Graphite Nagios Jboss App Memcache Postgres Slaves• Invalid configs! Postgres Master
  • Configuration Desperation http://www.flickr.com/photos/francoforeshock/5716969942/
  • Chef Solves This Problem Infrastructure as Code!
  • Programs! • Generate configurations directly on nodes • Reduce management complexity • Version control the programshttp://www.flickr.com/photos/ssoosay/5126146763/
  • Declarative Interface to Resources• Define policy• Say what, not how• Pull not Pushhttp://www.flickr.com/photos/bixentro/2591838509/
  • Chef is Infrastructure as • Programmatically provision and configure • Treat like any other code base • Reconstruct business from code repository, data backup, and bare metal resources. http://www.flickr.com/photos/louisb/4555295187/
  • That looks like thispackage "ntp" do action :installend template "/etc/ntpd.conf" do source "ntpd.conf.erb" owner "root" group "root" mode 0644 action :create variables(:time_server => “time.example.com”) notifies :restart, “service[ntpd]” end service "ntpd" do action [:enable,:start] end
  • Or thispackage "net-snmp" do action :installend template "/etc/snmpd.conf" do source "snmpd.conf.erb" owner "root" group "root" mode 0644 action :create variables(:community_string => “not_public”) notifies :restart, “service[snmpd]” end service "snmpd" do action [:enable,:start] end
  • "hostname": "server-1", "fqdn": "server-1.example.com", "domain": "example.com", "memory": { "swap": { "cached": "0kB", Ohai! "network": { "total": "4128760kB", "block_device": { "interfaces": { "free": "4128760kB" "ram0": { "eth0": { }, "size": "32768", "type": "eth", "total": "2055676kB", "removable": "0" "free": "1646524kB", }, "number": "0", "buffers": "35032kB", "ram1": { "encapsulation": "Ethernet", "cached": "210276kB", "size": "32768", "addresses": { "active": "125336kB", "removable": "0" "inactive": "142884kB", }, "00:0C:29:43:26:C5": { "dirty": "8kB", "ram2": { "family": "lladdr" "writeback": "0kB", "size": "32768", }, "anon_pages": "22976kB", "removable": "0" "mapped": "8416kB", }, "192.168.177.138": { "slab": "121512kB", "family": "inet", "slab_reclaimable": "41148kB", "broadcast": "192.168.177.255", "slab_unreclaim": "80364kB", "netmask": "255.255.255.0" "page_tables": "1784kB", }, "nfs_unstable": "0kB", "bounce": "0kB", "fe80::20c:29ff:fe43:26c5": { "commit_limit": "5156596kB", "family": "inet6", "committed_as": "74980kB", "prefixlen": "64", "vmalloc_total": "34359738367kB", "vmalloc_used": "274512kB", "scope": "Link" "vmalloc_chunk": "34359449936kB" } }, },
  • Decide what to declareexecute "load sysctl" do command "/sbin/sysctl -p" action :nothingendbytes = node[‘memory’][‘total’].split("kB")[0].to_i * 1024 / 3,pages = node[‘memory’][‘total’].split("kB")[0].to_i * 1024 / 3 / 2048# adjust shared memory and semaphorestemplate "/etc/sysctl.conf" do source "sysctl.conf.erb" variables( :shmmax_in_bytes => bytes, :shmall_in_pages => pages ) notifies :run, "execute[load sysctl]", :immediatelyend
  • Recipes and Cookbooks• Recipes are collections of Resources• Cookbooks contain recipes, templates, files, custom resources, etc• Code re-use and modularity http://www.flickr.com/photos/shutterhacks/4474421855/
  • Run Lists Server Serverchef-server Server Server API chef-client recipe[ntp::client] ntp node client.rb
  • Run Lists Server Serverchef-server Server Server chef-client API “ntp::client”, “openssh::server” ntp openssh node client.rb server.rb
  • Run Lists Server Serverchef-server Server Server chef-client “recipe[ntp::client]”, API “recipe[openssh::server]”, ntp “recipe[apache]”, “recipe[php]” openssh node client.rb apache server.rb php default.rb default.rb
  • Roles name "webserver" description "webserver server" run_list [ "role[base]", "recipe[nginx::server]" ]name "base"description "base"run_list [ "recipe[selinux::disabled]", "recipe[etchosts]", "recipe[yum::epel]", "recipe[debugtools]"]
  • Roles Server Server chef-server Server Server Role Recipe API Role Role Recipe Role Recipe RecipeKnife Recipe Recipe Recipe
  • Run Lists Server Serverchef-server Server Server chef-client “recipe[ntp::client]”, API “recipe[openssh::server]”, ntp “recipe[apache]”, “recipe[php]” openssh node client.rb apache server.rb php default.rb default.rb
  • Roles Server Serverchef-server Server Server API chef-client “role[base]”, ntp “role[webserver]” openssh node client.rb apache server.rb php default.rb default.rb
  • Roles Server Serverchef-server Server Server ntp openssh chef-client API client.rb apache php server.rb “role[webserver]” default.rb ntp default.rb node openssh chef-client client.rb mysql server.rb server.rb “role[database]” node
  • Search• Search for nodes with Roles• Find configuration data• IP addresses• Hostnames• FQDNs http://www.flickr.com/photos/kathycsus/2686772625
  • Search for nodespool_members = search("node","role:webserver”)template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]"end
  • Pass results into Templates# Set up application listeners here.listen application 0.0.0.0:80 balance roundrobin <% @pool_members.each do |member| -%> server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check <% end -%><% if node["haproxy"]["enable_admin"] -%>listen admin 0.0.0.0:22002 mode http stats uri /<% end -%>
  • munin::server examplenode.set[:munin][:server] = truemunin_clients = search(:node, "munin_client:true")cookbook_file "/etc/cron.d/munin" do source "munin-cron" mode "0644" owner "root" group "root"endtemplate "/etc/munin/munin.conf" do source "munin.conf.erb" mode 0644 variables(:munin_clients => munin_clients)end
  • munin::client examplenode.set[:munin][:client] = truemunin_servers = search(:node, "munin_server:true")unless munin_servers.empty? package "munin-node" do action :install end template "/etc/munin/munin-node.conf" do source "munin-node.conf.erb" mode 0644 variables :munin_servers => munin_servers notifies :restart, "service[munin-node]" end service "munin-node" do supports :restart => true action [ :enable, :start ] endend
  • So whenGraphite Nagios Jboss App Memcache Postgres Slaves Postgres Master
  • Becomes thisGraphite Nagios Jboss App Memcache Postgres Slaves Postgres Master
  • This can happen automaticallyGraphite Nagios Jboss App Memcache Postgres Slaves Postgres Master
  • Count the resources • Load balancer config Graphite Nagios • Nagios host ping • Nagios host ssh Jboss App • Nagios host HTTP • Nagios host app health Memcache • Graphite CPU • Graphite Memory Postgres Slaves • Graphite Disk • Graphite SNMP • Memcache firewall• 12+ resource changes for 1 node addition • Postgres firewall
  • Tools in Your Kitchen
  • chef-client runs on your systems
  • chef-client talks to a Chef Server
  • API Clients authenticate with RSA keys The server has the public key
  • Configured or managedsystems are called Nodes
  • Nodes have data called attributesAttributes are indexed for search on the Chef Server
  • Nodes have a Run List The list of roles or recipes to apply in order
  • Roles have a Run List The list of roles or recipes to apply in order
  • Chef manages Resources on the Node
  • Resources take idempotent action through Providers
  • Providers know how toperform the actions to configure a Resource
  • Recipes are lists of resources Resources are applied in the order theyre written in recipes!
  • Cookbooks are packages for Recipes
  • Knife is the command-line tool for Chef.
  • Try It Out
  • How to Get Chef• Hosted Chef is a SaaS product hosted by Opscode• http://manage.opscode.com• You can create an account and add up to five nodes for free to try out chef• Our new installer makes installing Chef on nodes super easy! • http://www.opscode.com/chef/install • Provides a full stack, don’t worry about Ruby version issues
  • Further resources: Cookbooks and Plugins• Useful cookbooks • Application cookbooks: • DNS: djbdns, pdns, • application, database dnsimple, dynect, route53 • python, java, php, ruby • Monitoring: nagios, munin, • Plugins zenoss, zabbix • Cloud: knife-ec2, knife- • Package repos: yum, apt, rackspace, knife-openstack, freebsd knife-hp • Security: ossec, snort, • Windows: knife-windows cis_benchmark • http://wiki.opscode.com/ • Logging: rsyslog, syslog-ng, display/chef/Community logstash, logwatch +Plugins
  • Further Resources• http://opscode.com/• http://community.opscode.com/• http://wiki.opscode.com/• http://lists.opscode.com• http://github.com/opscode and http://github.com/opscode-cookbooks• http://youtube.com/user/Opscode• http://shop.oreilly.com/product/0636920025146.do • (Chef: The Definitive Guide, coming soon!)
  • Supported Platformsvia http://www.opscode.com/chef/install/ • Ubuntu (10.04, 10.10, 11.04, 11.10, 12.04) • Debian (5.0, 6.0) • RHEL & CentOS (5.x, 6.x) • Fedora 10+ • Mac OS X (10.4, 10.5, 10.6) • Windows 7 and Server 2003 R2, 2008 R2• Others via the chef ruby gem
  • Thanks, Novalug!!• mandi walls• mandi@opscode.com• @lnxchk