Novalug 07142012


Published on

Intro to Chef at Novalug, July 14, 2012

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • Your business is about applications - through one lens, the entire business is a single, very complicated application\n
  • Those applications are supported by Infrastructure - literally all the stuff that’s needed to run the application.\n\nWhen dealing with Chef, need to literally “think outside the box”, by shifting your thinking about configuration away from a single system, to that of an Appliction Infrastructure. The concept of an Infrastructure is an abstract one with a specific technical meaning. When we talk about Infrastructure, we mean.. \n
  • ... a collection of Resources that can span nodes and networks. Resources are simple things that you deal with every day as a systems administrator or developer. Resources include files, directories, mounts, routes, users, groups, packages installations, source code deployments, configuration files, and “stuff” in general.\n\n
  • All this is arranged in a very specific way, to it acts in concert to provide ... \n
  • a service. That’s it. An Application Infrastructure provides a view of all it’s component nodes and their attributes, as well as information that needs to be shared among resources.\n
  • And a basic fact about Infrastructure -- it EVOLVES.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • an Infrastructure has a Topology. Maybe you don’t want to do it that way.\n
  • \nHow should I know. It’s your application.\n\nYour application is unique, and so is your infrastructure.\nThey evolve symbiotically.\n\nReference earlier, when we said you bring the knowledge about your systems\n\n\n
  • \n
  • \n
  • The answer to all the complexity is configuration management - you need to ‘manage’ the ‘configuration’ of all that stuff\n
  • Currently, the most widely used configuration management strategy is Cloning and Snapshotting.\nTHIS DOES NOT WORK (and you know it.)\n\n
  • OK\n
  • Policy change time!!!\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • How do we do this?\nWe let you write \n
  • PROGRAMS\n\nThese programs maintain the state of your machines, and are also responsible for generating the configuration that is the topology of the infrastructure.\n\n\n
  • Chef gives you declarative interfaces into resources.\n\nBeing declarative means that you say what you want to do, instead of how to do it.\nFor example, \nyou declare that package foobar-1.2.3 should be installed, or that the directory /var/log/foobar should exist.\n\nChef pulls down policy from the chef-server, ensuring that a node down for maintenance will receive its policy update when it comes back online.\n\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • executed in order!\n
  • executed in order!\n
  • executed in order!\n
  • \n
  • \n
  • executed in order!\n
  • \n
  • \n
  • This is where the sauce is, and what enables systems integration.\n(back up to previous slide)\nWhen provisioning on Clouds, you typically don’t get to do up front IP address planning\nSo how do you point a web server to its database?\nYou search for it.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Before we go further, let's get all the terminology out on the table.\n\nThese slides can simply be shown and read quickly, they're for awareness, and will be explained in further detail throughout the workshop.\n
  • When we're configuring systems with Chef, we run the client program, "chef-client" on the individual nodes. They're responsible for keeping themselves configured. We can run chef-client from cron, as a service, with runit or daemontools, by hand... as often or as rarely as we want to, it’s up to you!\n
  • The chef-client program connects to the configured Chef Server, whether it’s hosted chef, private chef, or open source chef\n
  • API Clients are the thing that authenticate to the Server using RSA keypairs. Chef-client uses an API client on behalf of the system it is running as, \n
  • A node is a thing that Chef is managing. It can be an Ec2 instance, a Virtual Machine, a Rackmounted server in a data center.\n
  • Attributes are merely data about the node. IPs, kernel modules, memory, platform and version. Ohai gave them to us, and we can add our own based on what’s needed in our applications.\n
  • Run lists are arrays, so they're ordered. They can contain roles or recipes.\n
  • \n
  • Resources are the abstracted declarative interface to a particular thing that should be configured on the node. Packages, services, templates, etc.\n\nThey describe WHAT, not HOW.\n
  • Providers are the underlying implementation that take care of the HOW for WHAT was declared in the resource.\n\nThey are intended to be idempotent, that is, they do not take action if they do not need to. "multiple application of the same thing do not change the output."\n\n(Not mathematical reference of idempotent functions)\n
  • Within Chef, the providers do all the heavy lifting for you. Package resources can have apt or yum or gem or other providers.\n
  • \n
  • Because there weren't enough package management systems out there...\n\nCookbooks contain recipes and other supporting code for configuring a particular thing, like apache, mysql, nginx, or trac\n
  • Knife is a plugin-based system that includes a number of plugins that work with the Chef server, and it can be extended for other purposes as we'll see!\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Novalug 07142012

    1. 1. Overview of Chef Mandi Walls novalug July 14, 2012
    2. 2. whoami• Mandi Walls•• @lnxchk
    3. 3. Applications
    4. 4. Infrastructur
    5. 5. Collection of•Nodes •Routes•Networking •Users•Files •Groups•Directories •Tasks•Symlinks •Packages•Mounts •Software •Services •Configurations •Stuff
    6. 6. Acting in concert
    7. 7. To provide a Service
    8. 8. And it evolves
    9. 9. See NodeApplication
    10. 10. See NodesApplicationApplication Database
    11. 11. See Nodes GrowApplicationApp Databases
    12. 12. See Nodes GrowApp ServersApp Databases
    13. 13. See Nodes GrowApp LB App ServersApp Databases
    14. 14. See Nodes Grow App LBs App Servers App Databases
    15. 15. See Nodes Grow App LBs App Servers App DB Cache App DBs
    16. 16. Stitched together with configs App LBs App Servers App DB Cache App DBs
    17. 17. Your Infrastructure is a snow flake App LB App Servers App DB Cache Floating IP? App DBs
    18. 18. Complexity increases quickly App LBs Cache App ServersNoSQL DB Cache DB slaves DBs
    19. 19. Complexity increases very quickly DC2DC1 DC3
    20. 20. Configuration Management
    21. 21. Golden Images are not the answer•Gold is heavy•Hard to transport•Hard to mold•Easy to lose configuration detail
    22. 22. Typical Boring InfrastructureGraphite Nagios Jboss App Memcache Postgres Slaves Postgres Master
    23. 23. New Compliance Mandate Graphite Nagios Jboss App Memcache• Move SSH off port 22• Lets put it on 2022 Postgres Slaves Postgres Master
    24. 24. 6 Golden Image Updates Graphite 1 2 Nagios 3 Jboss App 4 Memcache 5 Postgres Slaves• edit /etc/ssh/sshd_config 6 Postgres Master
    25. 25. 12 Instance Replacements Graphite 1 2 Nagios 3 4 5 6 7 Jboss App• Delete, launch 8 9 Memcache• Repeat• Typically manually 10 11 Postgres Slaves 12 Postgres Master
    26. 26. Done in Maintenance WindowsGraphite 1 2 Nagios 3 4 5 6 7 Jboss App 8 9 Memcache 5 10 11 Postgres Slaves 12 Postgres Master
    27. 27. Different IP Addresses? Graphite Nagios Jboss App Memcache Postgres Slaves• Invalid configs! Postgres Master
    28. 28. Configuration Desperation
    29. 29. Chef Solves This Problem Infrastructure as Code!
    30. 30. Programs! • Generate configurations directly on nodes • Reduce management complexity • Version control the programs
    31. 31. Declarative Interface to Resources• Define policy• Say what, not how• Pull not Push
    32. 32. Chef is Infrastructure as • Programmatically provision and configure • Treat like any other code base • Reconstruct business from code repository, data backup, and bare metal resources.
    33. 33. That looks like thispackage "ntp" do action :installend template "/etc/ntpd.conf" do source "ntpd.conf.erb" owner "root" group "root" mode 0644 action :create variables(:time_server => “”) notifies :restart, “service[ntpd]” end service "ntpd" do action [:enable,:start] end
    34. 34. Or thispackage "net-snmp" do action :installend template "/etc/snmpd.conf" do source "snmpd.conf.erb" owner "root" group "root" mode 0644 action :create variables(:community_string => “not_public”) notifies :restart, “service[snmpd]” end service "snmpd" do action [:enable,:start] end
    35. 35. "hostname": "server-1", "fqdn": "", "domain": "", "memory": { "swap": { "cached": "0kB", Ohai! "network": { "total": "4128760kB", "block_device": { "interfaces": { "free": "4128760kB" "ram0": { "eth0": { }, "size": "32768", "type": "eth", "total": "2055676kB", "removable": "0" "free": "1646524kB", }, "number": "0", "buffers": "35032kB", "ram1": { "encapsulation": "Ethernet", "cached": "210276kB", "size": "32768", "addresses": { "active": "125336kB", "removable": "0" "inactive": "142884kB", }, "00:0C:29:43:26:C5": { "dirty": "8kB", "ram2": { "family": "lladdr" "writeback": "0kB", "size": "32768", }, "anon_pages": "22976kB", "removable": "0" "mapped": "8416kB", }, "": { "slab": "121512kB", "family": "inet", "slab_reclaimable": "41148kB", "broadcast": "", "slab_unreclaim": "80364kB", "netmask": "" "page_tables": "1784kB", }, "nfs_unstable": "0kB", "bounce": "0kB", "fe80::20c:29ff:fe43:26c5": { "commit_limit": "5156596kB", "family": "inet6", "committed_as": "74980kB", "prefixlen": "64", "vmalloc_total": "34359738367kB", "vmalloc_used": "274512kB", "scope": "Link" "vmalloc_chunk": "34359449936kB" } }, },
    36. 36. Decide what to declareexecute "load sysctl" do command "/sbin/sysctl -p" action :nothingendbytes = node[‘memory’][‘total’].split("kB")[0].to_i * 1024 / 3,pages = node[‘memory’][‘total’].split("kB")[0].to_i * 1024 / 3 / 2048# adjust shared memory and semaphorestemplate "/etc/sysctl.conf" do source "sysctl.conf.erb" variables( :shmmax_in_bytes => bytes, :shmall_in_pages => pages ) notifies :run, "execute[load sysctl]", :immediatelyend
    37. 37. Recipes and Cookbooks• Recipes are collections of Resources• Cookbooks contain recipes, templates, files, custom resources, etc• Code re-use and modularity
    38. 38. Run Lists Server Serverchef-server Server Server API chef-client recipe[ntp::client] ntp node client.rb
    39. 39. Run Lists Server Serverchef-server Server Server chef-client API “ntp::client”, “openssh::server” ntp openssh node client.rb server.rb
    40. 40. Run Lists Server Serverchef-server Server Server chef-client “recipe[ntp::client]”, API “recipe[openssh::server]”, ntp “recipe[apache]”, “recipe[php]” openssh node client.rb apache server.rb php default.rb default.rb
    41. 41. Roles name "webserver" description "webserver server" run_list [ "role[base]", "recipe[nginx::server]" ]name "base"description "base"run_list [ "recipe[selinux::disabled]", "recipe[etchosts]", "recipe[yum::epel]", "recipe[debugtools]"]
    42. 42. Roles Server Server chef-server Server Server Role Recipe API Role Role Recipe Role Recipe RecipeKnife Recipe Recipe Recipe
    43. 43. Run Lists Server Serverchef-server Server Server chef-client “recipe[ntp::client]”, API “recipe[openssh::server]”, ntp “recipe[apache]”, “recipe[php]” openssh node client.rb apache server.rb php default.rb default.rb
    44. 44. Roles Server Serverchef-server Server Server API chef-client “role[base]”, ntp “role[webserver]” openssh node client.rb apache server.rb php default.rb default.rb
    45. 45. Roles Server Serverchef-server Server Server ntp openssh chef-client API client.rb apache php server.rb “role[webserver]” default.rb ntp default.rb node openssh chef-client client.rb mysql server.rb server.rb “role[database]” node
    46. 46. Search• Search for nodes with Roles• Find configuration data• IP addresses• Hostnames• FQDNs
    47. 47. Search for nodespool_members = search("node","role:webserver”)template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]"end
    48. 48. Pass results into Templates# Set up application listeners here.listen application balance roundrobin <% @pool_members.each do |member| -%> server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check <% end -%><% if node["haproxy"]["enable_admin"] -%>listen admin mode http stats uri /<% end -%>
    49. 49. munin::server examplenode.set[:munin][:server] = truemunin_clients = search(:node, "munin_client:true")cookbook_file "/etc/cron.d/munin" do source "munin-cron" mode "0644" owner "root" group "root"endtemplate "/etc/munin/munin.conf" do source "munin.conf.erb" mode 0644 variables(:munin_clients => munin_clients)end
    50. 50. munin::client examplenode.set[:munin][:client] = truemunin_servers = search(:node, "munin_server:true")unless munin_servers.empty? package "munin-node" do action :install end template "/etc/munin/munin-node.conf" do source "munin-node.conf.erb" mode 0644 variables :munin_servers => munin_servers notifies :restart, "service[munin-node]" end service "munin-node" do supports :restart => true action [ :enable, :start ] endend
    51. 51. So whenGraphite Nagios Jboss App Memcache Postgres Slaves Postgres Master
    52. 52. Becomes thisGraphite Nagios Jboss App Memcache Postgres Slaves Postgres Master
    53. 53. This can happen automaticallyGraphite Nagios Jboss App Memcache Postgres Slaves Postgres Master
    54. 54. Count the resources • Load balancer config Graphite Nagios • Nagios host ping • Nagios host ssh Jboss App • Nagios host HTTP • Nagios host app health Memcache • Graphite CPU • Graphite Memory Postgres Slaves • Graphite Disk • Graphite SNMP • Memcache firewall• 12+ resource changes for 1 node addition • Postgres firewall
    55. 55. Tools in Your Kitchen
    56. 56. chef-client runs on your systems
    57. 57. chef-client talks to a Chef Server
    58. 58. API Clients authenticate with RSA keys The server has the public key
    59. 59. Configured or managedsystems are called Nodes
    60. 60. Nodes have data called attributesAttributes are indexed for search on the Chef Server
    61. 61. Nodes have a Run List The list of roles or recipes to apply in order
    62. 62. Roles have a Run List The list of roles or recipes to apply in order
    63. 63. Chef manages Resources on the Node
    64. 64. Resources take idempotent action through Providers
    65. 65. Providers know how toperform the actions to configure a Resource
    66. 66. Recipes are lists of resources Resources are applied in the order theyre written in recipes!
    67. 67. Cookbooks are packages for Recipes
    68. 68. Knife is the command-line tool for Chef.
    69. 69. Try It Out
    70. 70. How to Get Chef• Hosted Chef is a SaaS product hosted by Opscode•• You can create an account and add up to five nodes for free to try out chef• Our new installer makes installing Chef on nodes super easy! • • Provides a full stack, don’t worry about Ruby version issues
    71. 71. Further resources: Cookbooks and Plugins• Useful cookbooks • Application cookbooks: • DNS: djbdns, pdns, • application, database dnsimple, dynect, route53 • python, java, php, ruby • Monitoring: nagios, munin, • Plugins zenoss, zabbix • Cloud: knife-ec2, knife- • Package repos: yum, apt, rackspace, knife-openstack, freebsd knife-hp • Security: ossec, snort, • Windows: knife-windows cis_benchmark • • Logging: rsyslog, syslog-ng, display/chef/Community logstash, logwatch +Plugins
    72. 72. Further Resources••••• and•• • (Chef: The Definitive Guide, coming soon!)
    73. 73. Supported Platformsvia • Ubuntu (10.04, 10.10, 11.04, 11.10, 12.04) • Debian (5.0, 6.0) • RHEL & CentOS (5.x, 6.x) • Fedora 10+ • Mac OS X (10.4, 10.5, 10.6) • Windows 7 and Server 2003 R2, 2008 R2• Others via the chef ruby gem
    74. 74. Thanks, Novalug!!• mandi walls•• @lnxchk