Intro to Chef
Upcoming SlideShare
Loading in...5
×
 

Intro to Chef

on

  • 1,786 views

 

Statistics

Views

Total Views
1,786
Slideshare-icon Views on SlideShare
1,781
Embed Views
5

Actions

Likes
2
Downloads
59
Comments
0

2 Embeds 5

https://twitter.com 4
http://www.hanrss.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Intro to Chef Intro to Chef Presentation Transcript

    • Introduzione a Chef Giacomo Bagnoli Develer Workshops - 12 Settembre 2012@gbagnoli Introduzione a Chef 12/09/2012 1 / 50
    • # whoami• System Administrator• Operations Engineer• Python guy (having an affair with Ruby) @gbagnoli Introduzione a Chef 12/09/2012 2 / 50
    • Outline1. Introduction2. Chef Overview3. Example @gbagnoli Introduzione a Chef 12/09/2012 3 / 50
    • Introduction@gbagnoli Introduzione a Chef 12/09/2012 4 / 50
    • Infrastructure as codeBuilding and managing infrastructure programmatically @gbagnoli Introduzione a Chef 12/09/2012 5 / 50
    • Infrastructure as code Building and managing infrastructure programmaticallyEnable the reconstruction of the business from: @gbagnoli Introduzione a Chef 12/09/2012 5 / 50
    • Infrastructure as code Building and managing infrastructure programmaticallyEnable the reconstruction of the business from: • computing resources @gbagnoli Introduzione a Chef 12/09/2012 5 / 50
    • Infrastructure as code Building and managing infrastructure programmaticallyEnable the reconstruction of the business from: • computing resources • a source code repository @gbagnoli Introduzione a Chef 12/09/2012 5 / 50
    • Infrastructure as code Building and managing infrastructure programmaticallyEnable the reconstruction of the business from: • computing resources • a source code repository • data backups @gbagnoli Introduzione a Chef 12/09/2012 5 / 50
    • Infrastructure as code (2)source: Chef wikiProvisioning Get new computing resourcesConfiguration Management Keeps track of all steps required to take bare metal resources to doing their jobSystem Integration Takes all configured systems and make them work together. @gbagnoli Introduzione a Chef 12/09/2012 6 / 50
    • What is Chef@gbagnoli Introduzione a Chef 12/09/2012 7 / 50
    • What is Chef• A library/framework for configuration management @gbagnoli Introduzione a Chef 12/09/2012 7 / 50
    • What is Chef• A library/framework for configuration management• A configuration management system @gbagnoli Introduzione a Chef 12/09/2012 7 / 50
    • What is Chef• A library/framework for configuration management• A configuration management system• A system integration platform @gbagnoli Introduzione a Chef 12/09/2012 7 / 50
    • What is Chef• A library/framework for configuration management• A configuration management system• A system integration platform• An API for the infrastructure @gbagnoli Introduzione a Chef 12/09/2012 7 / 50
    • What is Chef• A library/framework for configuration management• A configuration management system• A system integration platform• An API for the infrastructure• Open Source! (Apache License, version 2.0) @gbagnoli Introduzione a Chef 12/09/2012 7 / 50
    • Opscode Chef@gbagnoli Introduzione a Chef 12/09/2012 8 / 50
    • Chef Principles@gbagnoli Introduzione a Chef 12/09/2012 9 / 50
    • Chef PrinciplesIdempotent Describes states @gbagnoli Introduzione a Chef 12/09/2012 9 / 50
    • Chef PrinciplesIdempotent Describes states Extensible Uses ruby as the DSL @gbagnoli Introduzione a Chef 12/09/2012 9 / 50
    • Chef PrinciplesIdempotent Describes states Extensible Uses ruby as the DSL Order It matters @gbagnoli Introduzione a Chef 12/09/2012 9 / 50
    • Chef Principles Idempotent Describes states Extensible Uses ruby as the DSL Order It mattersClient-Server Thick Clients, thin Server @gbagnoli Introduzione a Chef 12/09/2012 9 / 50
    • Chef Principles Idempotent Describes states Extensible Uses ruby as the DSL Order It mattersClient-Server Thick Clients, thin ServerVarious Flavours: @gbagnoli Introduzione a Chef 12/09/2012 9 / 50
    • Chef Principles Idempotent Describes states Extensible Uses ruby as the DSL Order It mattersClient-Server Thick Clients, thin ServerVarious Flavours: • Hosted on Opscode platform (chef-hosted) @gbagnoli Introduzione a Chef 12/09/2012 9 / 50
    • Chef Principles Idempotent Describes states Extensible Uses ruby as the DSL Order It mattersClient-Server Thick Clients, thin ServerVarious Flavours: • Hosted on Opscode platform (chef-hosted) • Open source Chef Server @gbagnoli Introduzione a Chef 12/09/2012 9 / 50
    • Chef Principles Idempotent Describes states Extensible Uses ruby as the DSL Order It mattersClient-Server Thick Clients, thin ServerVarious Flavours: • Hosted on Opscode platform (chef-hosted) • Open source Chef Server • Serverless - chef-solo @gbagnoli Introduzione a Chef 12/09/2012 9 / 50
    • Chef Principles Idempotent Describes states Extensible Uses ruby as the DSL Order It mattersClient-Server Thick Clients, thin ServerVarious Flavours: • Hosted on Opscode platform (chef-hosted) • Open source Chef Server • Serverless - chef-solo • Private Chef (opscode-supported behind-the-firewall installation) @gbagnoli Introduzione a Chef 12/09/2012 9 / 50
    • Chef API• A RESTful service with JSON responses• RSA key authentication with Signed Headers• Search Service• Derivative (easy to integrate with other tools)• . . . i.e pychef :-) @gbagnoli Introduzione a Chef 12/09/2012 10 / 50
    • Chef API • A RESTful service with JSON responses • RSA key authentication with Signed Headers • Search Service • Derivative (easy to integrate with other tools) • . . . i.e pychef :-)chef-client connects to the server consuming the API.CLI management tool knife and the webUI use the API too. @gbagnoli Introduzione a Chef 12/09/2012 10 / 50
    • Chef Architecture Solr CouchDB Indexer RabbitMQ Chef API Server Chef WebUI Client Client Client Knife Knife@gbagnoli Introduzione a Chef 12/09/2012 11 / 50
    • API ClientIn chef, an API client provides the identity used to authenticaterequests to the API server. @gbagnoli Introduzione a Chef 12/09/2012 12 / 50
    • API ClientIn chef, an API client provides the identity used to authenticaterequests to the API server.The public half of the public/private of a key pair is stored in the db onthe server, while the private part is local to the client. @gbagnoli Introduzione a Chef 12/09/2012 12 / 50
    • API ClientIn chef, an API client provides the identity used to authenticaterequests to the API server.The public half of the public/private of a key pair is stored in the db onthe server, while the private part is local to the client.Each request to the API contains a request signature in the HTTPheaders.The request signature is computed by the hash of the request content andencrypted with the client private key, so it’s possible to verify the identityof the user/machine making the request. @gbagnoli Introduzione a Chef 12/09/2012 12 / 50
    • NodesA Node is a host that runs the chef-client. • Has attributes • Has a run list • Has 0+ roles • Belongs to an environmentIn the common case, 1 host ⇔ 1 node ⇔ 1 client @gbagnoli Introduzione a Chef 12/09/2012 13 / 50
    • run list"run_list": { "role[python_hosting]", "recipe[postgresql::client]", "recipe[chishop]"} @gbagnoli Introduzione a Chef 12/09/2012 14 / 50
    • Roles• Have attributes• Have a run list• Declared in JSON or . . .• Declared with the ruby DSL (automatically compiled to JSON) @gbagnoli Introduzione a Chef 12/09/2012 15 / 50
    • Roles • Have attributes • Have a run list • Declared in JSON or . . . • Declared with the ruby DSL (automatically compiled to JSON)If 1+ roles are in the node run list, the node run list is expanded @gbagnoli Introduzione a Chef 12/09/2012 15 / 50
    • Roles (2)An example role (in ruby):name "python_hosting"description "Python App hosting"default_attributes( "nginx" => { "default_site_enabled" => false })run_list( "recipe[python::virtualenv]", "recipe[uwsgi]", "recipe[nginx]") @gbagnoli Introduzione a Chef 12/09/2012 16 / 50
    • AttributesStore node data (i.e. ip address, hostname, fqdn, database host address,etc.)There are four types of attributes (in order of precedence, lowest tohighest): • default • normal • override • automatic @gbagnoli Introduzione a Chef 12/09/2012 17 / 50
    • AttributesStore node data (i.e. ip address, hostname, fqdn, database host address,etc.)There are four types of attributes (in order of precedence, lowest tohighest): • default • normal • override • automaticAttributes can be set in: • cookbooks • environments • roles • nodes @gbagnoli Introduzione a Chef 12/09/2012 17 / 50
    • Attributes (2)So, in the end, as attributes are deep-merged, the following precedenceapplies: @gbagnoli Introduzione a Chef 12/09/2012 18 / 50
    • Attributes (2)So, in the end, as attributes are deep-merged, the following precedenceapplies: • default attributes applied in an cookbook • default attributes applied in an environment • default attributes applied in a role • default attributes applied on a node directly in a recipe • normal attributes applied in a cookbook • normal attributes applied on a node directly in a recipe • override attributes applied in an cookbook • override attributes applied in an environment • override attributes applied in a role • override attributes applied on a node directly in a recipe • automatic attributes generated by Ohai @gbagnoli Introduzione a Chef 12/09/2012 18 / 50
    • Attributes (3)Automatic, ovverride and default are reset at the beginning of every run.Normal attributes persist between runs. @gbagnoli Introduzione a Chef 12/09/2012 19 / 50
    • Attributes (3)Automatic, ovverride and default are reset at the beginning of every run.Normal attributes persist between runs.Attributes are searchable: search(:node, ’platform:ubuntu’)or knife search node "platform:ubuntu" @gbagnoli Introduzione a Chef 12/09/2012 19 / 50
    • Attributes (4)Summary: @gbagnoli Introduzione a Chef 12/09/2012 20 / 50
    • Attributes (4)Summary: • (sane) defaults in cookbooks @gbagnoli Introduzione a Chef 12/09/2012 20 / 50
    • Attributes (4)Summary: • (sane) defaults in cookbooks • . . . overridden in roles @gbagnoli Introduzione a Chef 12/09/2012 20 / 50
    • Attributes (4)Summary: • (sane) defaults in cookbooks • . . . overridden in roles • . . . and node-specific data as normal attributes on the node. @gbagnoli Introduzione a Chef 12/09/2012 20 / 50
    • Attributes (4)Summary: • (sane) defaults in cookbooks • . . . overridden in roles • . . . and node-specific data as normal attributes on the node. • override and node.set can be used to force values @gbagnoli Introduzione a Chef 12/09/2012 20 / 50
    • ResourcesChef manages resources on a node. @gbagnoli Introduzione a Chef 12/09/2012 21 / 50
    • ResourcesChef manages resources on a node.Resources are specified in recipes, recipes stored in cookbooks. @gbagnoli Introduzione a Chef 12/09/2012 21 / 50
    • ResourcesChef manages resources on a node.Resources are specified in recipes, recipes stored in cookbooks.The expanded run list specifies all the recipes (and thus the resources) tomanage on a given node. @gbagnoli Introduzione a Chef 12/09/2012 21 / 50
    • Resources (2)A resource • has a type package "tar" do • has a name version "1.16.1-1" action :install • has parameters end • takes actions @gbagnoli Introduzione a Chef 12/09/2012 22 / 50
    • Resources (2)A resource • has a type package "tar" do • has a name version "1.16.1-1" action :install • has parameters end • takes actionsActions are taken using providers, providers are chosen based on the nodeplatform.(i.e. the package resource installs packages using apt on debian/ubuntuand using yum on centos/RHEL)! @gbagnoli Introduzione a Chef 12/09/2012 22 / 50
    • RecipesRecipes evaluate resources in the order they appearpackage "pdns-recursor" do action :installendtemplate "#{node[:pdns][:confd]}/recursor.conf" do source "recursor.cfg.erb" owner "root" group "root" mode 0644 notifies :restart, "service[pdns-recursor]"endservice "pdns-recursor" do action [:enable, :start]end @gbagnoli Introduzione a Chef 12/09/2012 23 / 50
    • Recipes (2)Recipes can include other resources, and are just ruby code include_recipe "apache2" ... %w{config logs files}.each do |dir| directory "#{node[:myrecipe][:base_dir]}/#{dir}" do recursive true owner "myuser" group "mygroup" mode 02775 end end @gbagnoli Introduzione a Chef 12/09/2012 24 / 50
    • CookbooksCookbooks are (shareable) packages for recipes.Cookbooks for chef ⇔ gems for ruby cookbooks / python / | - - attributes | ‘-- default . rb | - - files | ‘-- default | - - LICENSE | - - metadata . rb | - - providers | | - - pip . rb | ‘-- virtualenv . rb | - - README . md | - - recipes | | - - default . rb | | - - package . rb | | - - pip . rb | | - - source . rb | ‘-- virtualenv . rb | - - resources | | - - pip . rb | ‘-- virtualenv . rb ‘-- templates ‘-- default ‘-- p r o f i l e _ v i r t u a l e n v w r a p p e r . sh . erb @gbagnoli Introduzione a Chef 12/09/2012 25 / 50
    • Cookbooks (2)Currently 133 cookbooks available on opscode-cookbooks GitHub org.too many to list them all!https://github.com/opscode-cookbooksMore cookbooks on the community site athttp://community.opscode.com/cookbooks @gbagnoli Introduzione a Chef 12/09/2012 26 / 50
    • Metadatamaintainer "Opscode, Inc."maintainer_email "cookbooks@opscode.com"license "Apache 2.0"description "Installs and configures mysql for client or server"long_description IO.read(File.join(File.dirname(__FILE__), ’README.md’))version "1.2.5"recipe "mysql", "Includes the client recipe to configure a client"recipe "mysql::client", "Installs packages required for mysql clients using run_action magic"recipe "mysql::server", "Installs packages required for mysql servers w/o manual intervention"recipe "mysql::server_ec2", "Performs EC2-specific mountpoint manipulation"%w{ debian ubuntu centos suse fedora redhat scientific amazon }.each do |os| supports osend @gbagnoli Introduzione a Chef 12/09/2012 27 / 50
    • EnvironmentsEnvironments can be used to manage different environments (production,test, etc) in a single Chef setup.Roles can have different run list on different environments @gbagnoli Introduzione a Chef 12/09/2012 28 / 50
    • EnvironmentsEnvironments can be used to manage different environments (production,test, etc) in a single Chef setup.Roles can have different run list on different environmentsname "production"description "The production environment"cookbook_versions( "mysql" => "= 1.2.5", # use version 1.2.5 only "apache2" => "~> 1.1" # anything 1.1.0 < x < 1.2.0)# default attributes for this environmentattributes( "apache2" => { "listen_ports" => ["80", "443"] }) @gbagnoli Introduzione a Chef 12/09/2012 28 / 50
    • EnvironmentsEnvironments can be used to manage different environments (production,test, etc) in a single Chef setup.Roles can have different run list on different environmentsname "production"description "The production environment"cookbook_versions( "mysql" => "= 1.2.5", # use version 1.2.5 only "apache2" => "~> 1.1" # anything 1.1.0 < x < 1.2.0)# default attributes for this environmentattributes( "apache2" => { "listen_ports" => ["80", "443"] })As with role, the ruby DSL gets compiled to JSON when uploading to server @gbagnoli Introduzione a Chef 12/09/2012 28 / 50
    • Environments (2)$ knife environment listproduction$ knife environment show production -F json{ "name": "production", "description": "The production environment", "cookbook_versions": { "mysql": "= 1.2.5", "apache2": "~> 1.1" }, "json_class": "Chef::Environment", "chef_type": "environment", "default_attributes": { "apache2": { "listen_ports": [ "80", "443" ] } }, "override_attributes": { }} @gbagnoli Introduzione a Chef 12/09/2012 29 / 50
    • Environments (3)Cookbooks can be frozen, so that following uploads with the same versionwill fail. $ # -E automatically sets a requirement for the specified environment $ knife cookbook upload redis -- freeze -E production Uploading redis ... upload complete $ knife cookbook show redis 1.0.2 | grep " frozen " frozen ?: true $ knife cookbook upload redis Uploading redis ... ERROR : Version 1.0.2 of cookbook redis is frozen . Use -- force to override . ERROR : Failed to upload 1 cookbook . $ knife environment show production | grep redis users : = 1.0.2 @gbagnoli Introduzione a Chef 12/09/2012 30 / 50
    • Data BagsData bags provide an arbitrary store of globally available JSON data. @gbagnoli Introduzione a Chef 12/09/2012 31 / 50
    • Data BagsData bags provide an arbitrary store of globally available JSON data.Data bags can be encrypted (but then cannot be searched, except for id) @gbagnoli Introduzione a Chef 12/09/2012 31 / 50
    • Data Bags (Encrypted)(warn: fake data ahead) $ knife data bag show accounts gbagnoli comment : tenoh > dieliSh ’ i7eexeijeiSh ^ u9phaeGhuu4chaa *= email : A h r 8 i s 3 a h C h o h m 6 a e n e i c ( aef " a h1e eree Voh hie 6Up = group : Waix8Pa # iniy # oh6eem$eij = groups : xa i7on g7a ihi u1n eH & ah3ier3Goh } rae7nik$einaeb = id : gbagnoli shadow : yi e@ja h0v e$g 2Ae Gh } ido6koobuew | aebeenaequeRo ( xaiYei8eizi + f 7 ohqu < i@enequ & oh7ef - ahdae8dia [ c h a h 7 e e 4 y i e $ N 4 E e B i c h e e 5 e i r o h2JaGhae ^ k 6a e ph o hj ah s h6 Ae j a ^ cheew } o ) i 0 w o 5 i e s i s h 3 d i g h i e w o h tohoh0eegho7eik = shell : aeshi2ohy , ai6ai h2Ahquu = ssh_keys : [.. cut ..] uid : thahvo2IGhoh3osho8Ees /a= username : poh5WiuZ2Er : it ! ee1ahf { u = @gbagnoli Introduzione a Chef 12/09/2012 32 / 50
    • Data Bags (Decrypted)$ knife data bag show accounts gbagnoli -- secret - file ~/. chef / enc_db_secretcomment : Giacomo Bagnoliemail : g . bagnoli@asidev . comgroup : wheelgroups : [ asidev , users ]id : gbagnoliorganization : asidevshadow : $6$ [... cut ...]shell : / bin / bashssh_keys : [" ssh - rsa [... cut ...] Giacomo Bagnoli "]uid : 3000username : g . bagnoli @gbagnoli Introduzione a Chef 12/09/2012 33 / 50
    • Anatomy of a Chef Run@gbagnoli Introduzione a Chef 12/09/2012 34 / 50
    • Anatomy of a Chef Run• chef-client starts @gbagnoli Introduzione a Chef 12/09/2012 34 / 50
    • Anatomy of a Chef Run• chef-client starts• Builds node (runs ohai, perform deep-merge of attrs) @gbagnoli Introduzione a Chef 12/09/2012 34 / 50
    • Anatomy of a Chef Run• chef-client starts• Builds node (runs ohai, perform deep-merge of attrs)• chef-client registers with the server @gbagnoli Introduzione a Chef 12/09/2012 34 / 50
    • Anatomy of a Chef Run• chef-client starts• Builds node (runs ohai, perform deep-merge of attrs)• chef-client registers with the server• Cookbook sync @gbagnoli Introduzione a Chef 12/09/2012 34 / 50
    • Anatomy of a Chef Run• chef-client starts• Builds node (runs ohai, perform deep-merge of attrs)• chef-client registers with the server• Cookbook sync• Compiles resource collection, loading: @gbagnoli Introduzione a Chef 12/09/2012 34 / 50
    • Anatomy of a Chef Run• chef-client starts• Builds node (runs ohai, perform deep-merge of attrs)• chef-client registers with the server• Cookbook sync• Compiles resource collection, loading: • libraries @gbagnoli Introduzione a Chef 12/09/2012 34 / 50
    • Anatomy of a Chef Run• chef-client starts• Builds node (runs ohai, perform deep-merge of attrs)• chef-client registers with the server• Cookbook sync• Compiles resource collection, loading: • libraries • resources / definitions @gbagnoli Introduzione a Chef 12/09/2012 34 / 50
    • Anatomy of a Chef Run• chef-client starts• Builds node (runs ohai, perform deep-merge of attrs)• chef-client registers with the server• Cookbook sync• Compiles resource collection, loading: • libraries • resources / definitions • attributes @gbagnoli Introduzione a Chef 12/09/2012 34 / 50
    • Anatomy of a Chef Run• chef-client starts• Builds node (runs ohai, perform deep-merge of attrs)• chef-client registers with the server• Cookbook sync• Compiles resource collection, loading: • libraries • resources / definitions • attributes • recipes @gbagnoli Introduzione a Chef 12/09/2012 34 / 50
    • Anatomy of a Chef Run• chef-client starts• Builds node (runs ohai, perform deep-merge of attrs)• chef-client registers with the server• Cookbook sync• Compiles resource collection, loading: • libraries • resources / definitions • attributes • recipes• Executes - Configure Node @gbagnoli Introduzione a Chef 12/09/2012 34 / 50
    • Anatomy of a Chef Run• chef-client starts• Builds node (runs ohai, perform deep-merge of attrs)• chef-client registers with the server• Cookbook sync• Compiles resource collection, loading: • libraries • resources / definitions • attributes • recipes• Executes - Configure Node • Converge: each resource is mapped to a provider and which takes action on it @gbagnoli Introduzione a Chef 12/09/2012 34 / 50
    • Anatomy of a Chef Run• chef-client starts• Builds node (runs ohai, perform deep-merge of attrs)• chef-client registers with the server• Cookbook sync• Compiles resource collection, loading: • libraries • resources / definitions • attributes • recipes• Executes - Configure Node • Converge: each resource is mapped to a provider and which takes action on it • Saves Node @gbagnoli Introduzione a Chef 12/09/2012 34 / 50
    • Anatomy of a Chef Run • chef-client starts • Builds node (runs ohai, perform deep-merge of attrs) • chef-client registers with the server • Cookbook sync • Compiles resource collection, loading: • libraries • resources / definitions • attributes • recipes • Executes - Configure Node • Converge: each resource is mapped to a provider and which takes action on it • Saves Node • Runs notification HandlersOn errors, exception handlers are run. @gbagnoli Introduzione a Chef 12/09/2012 34 / 50
    • Chef development workflow@gbagnoli Introduzione a Chef 12/09/2012 35 / 50
    • Chef development workflow• Write cookbooks/recipe @gbagnoli Introduzione a Chef 12/09/2012 35 / 50
    • Chef development workflow• Write cookbooks/recipe• Upload the modified cookbook to the chef server @gbagnoli Introduzione a Chef 12/09/2012 35 / 50
    • Chef development workflow• Write cookbooks/recipe• Upload the modified cookbook to the chef server• Add the cookbook to a run list (in a node or in a role) @gbagnoli Introduzione a Chef 12/09/2012 35 / 50
    • Chef development workflow• Write cookbooks/recipe• Upload the modified cookbook to the chef server• Add the cookbook to a run list (in a node or in a role)• Wait for chef-client to run on nodes @gbagnoli Introduzione a Chef 12/09/2012 35 / 50
    • Chef development workflow• Write cookbooks/recipe• Upload the modified cookbook to the chef server• Add the cookbook to a run list (in a node or in a role)• Wait for chef-client to run on nodes• Commit changes in git @gbagnoli Introduzione a Chef 12/09/2012 35 / 50
    • SearchFull-text query engine based on Apache Solr.Searches can be performed from knife and in recipes.Almost any object is indexed by the chef server, like roles, nodes, apiclients and environments. $ knife search node " recipes : apache2 " 7 items found ... $ knife search node " recipes : apache2 AND chef_environment : production " 5 items found ... $ knife search node " roles : lxc_guest " 9 items found ... $ knife search client " admin : true " 4 items found ... $ knife search role " name : lxc *" 2 items found ... @gbagnoli Introduzione a Chef 12/09/2012 36 / 50
    • BootstrappingBootstrapping is installing chef on new nodes . . . using chef.First, create the node: $ knife node create mynewnode . example . com # .. fires up $EDITOR # .. set run_list / attributes / etc / environment @gbagnoli Introduzione a Chef 12/09/2012 37 / 50
    • BootstrappingBootstrapping is installing chef on new nodes . . . using chef.First, create the node: $ knife node create mynewnode . example . com # .. fires up $EDITOR # .. set run_list / attributes / etc / environmentAssuming that the new node is a bare ubuntu install, bootstrap the node $ knife boostrap -N mynewnode . example . com -d ubuntu $NODE_IP -- sudo -V -x ubuntu @gbagnoli Introduzione a Chef 12/09/2012 37 / 50
    • BootstrappingBootstrapping is installing chef on new nodes . . . using chef.First, create the node: $ knife node create mynewnode . example . com # .. fires up $EDITOR # .. set run_list / attributes / etc / environmentAssuming that the new node is a bare ubuntu install, bootstrap the node $ knife boostrap -N mynewnode . example . com -d ubuntu $NODE_IP -- sudo -V -x ubuntuOr, combine with provisioning (i.e. Amazon AWS) knife ec2 server create -I ami - db595faf -- flavor t1 . micro -- region eu - west -1 -G default -x ubuntu -N newnode . example . com -d ubuntu -Z eu - west -1 aChef Omnibus bootstrap template @gbagnoli Introduzione a Chef 12/09/2012 37 / 50
    • Monitoring: CheckMK and Chef @gbagnoli Introduzione a Chef 12/09/2012 38 / 50
    • MonitoringWe use CheckMK to configure Icinga for monitoring our infrastructure. @gbagnoli Introduzione a Chef 12/09/2012 39 / 50
    • MonitoringWe use CheckMK to configure Icinga for monitoring our infrastructure.CheckMK is a general purpose nagios plugin to retrive data from hosts. @gbagnoli Introduzione a Chef 12/09/2012 39 / 50
    • MonitoringWe use CheckMK to configure Icinga for monitoring our infrastructure.CheckMK is a general purpose nagios plugin to retrive data from hosts. 1. One active check per host per check interval (calling check mk as a plugin). @gbagnoli Introduzione a Chef 12/09/2012 39 / 50
    • MonitoringWe use CheckMK to configure Icinga for monitoring our infrastructure.CheckMK is a general purpose nagios plugin to retrive data from hosts. 1. One active check per host per check interval (calling check mk as a plugin). 2. The connection is done via TCP to the check mk agent on the target host @gbagnoli Introduzione a Chef 12/09/2012 39 / 50
    • MonitoringWe use CheckMK to configure Icinga for monitoring our infrastructure.CheckMK is a general purpose nagios plugin to retrive data from hosts. 1. One active check per host per check interval (calling check mk as a plugin). 2. The connection is done via TCP to the check mk agent on the target host (All host data is sent back at once as ASCII text.) @gbagnoli Introduzione a Chef 12/09/2012 39 / 50
    • MonitoringWe use CheckMK to configure Icinga for monitoring our infrastructure.CheckMK is a general purpose nagios plugin to retrive data from hosts. 1. One active check per host per check interval (calling check mk as a plugin). 2. The connection is done via TCP to the check mk agent on the target host (All host data is sent back at once as ASCII text.) 3. check mk extracts performance data. @gbagnoli Introduzione a Chef 12/09/2012 39 / 50
    • MonitoringWe use CheckMK to configure Icinga for monitoring our infrastructure.CheckMK is a general purpose nagios plugin to retrive data from hosts. 1. One active check per host per check interval (calling check mk as a plugin). 2. The connection is done via TCP to the check mk agent on the target host (All host data is sent back at once as ASCII text.) 3. check mk extracts performance data. 4. check mk checks warn/crit thresholds and submits results to Icinga as passive checks. @gbagnoli Introduzione a Chef 12/09/2012 39 / 50
    • Monitoring - NodesOn nodes, the check mk::agent recipe, included in all nodes via the baserole, installs the check mk agent. @gbagnoli Introduzione a Chef 12/09/2012 40 / 50
    • Monitoring - NodesOn nodes, the check mk::agent recipe, included in all nodes via the baserole, installs the check mk agent.It also sets up xinetd and the firewall so that connections to the agentare allowed only from the monitoring host(s). @gbagnoli Introduzione a Chef 12/09/2012 40 / 50
    • Monitoring - NodesOn nodes, the check mk::agent recipe, included in all nodes via the baserole, installs the check mk agent.It also sets up xinetd and the firewall so that connections to the agentare allowed only from the monitoring host(s).Monitoring hosts are specified as attributes in the base role. @gbagnoli Introduzione a Chef 12/09/2012 40 / 50
    • Monitoring - Nodes (2)(almost) Every cookbook pushes a MRPE or check mk plugin check tothe node, so the check mk agent returns data for all configured services.i.e. , in the mysql:: server recipemrpe_check "mysql" do script "check_mysql" variables( :passwd => node["mysql"]["server_root_password"], :checks => checks, :tunables => node["mysql"]["tunable"] )end @gbagnoli Introduzione a Chef 12/09/2012 41 / 50
    • Monitoring - Nodes (2)(almost) Every cookbook pushes a MRPE or check mk plugin check tothe node, so the check mk agent returns data for all configured services.i.e. , in the mysql:: server recipemrpe_check "mysql" do script "check_mysql" variables( :passwd => node["mysql"]["server_root_password"], :checks => checks, :tunables => node["mysql"]["tunable"] )end> telnet mysql-server.example.com 6556[...]<<<mrpe>>>(check_mysql) mysql_idx 0 OK - index usage 53.60% | index_usage=53.60%;0:;0:(check_mysql) mysql_running 0 OK - 0 long running processes | long_running_procs=0;10;20(check_mysql) mysql_threads 0 OK - 18 client connection threads | threads_connected=18;80;95[...] @gbagnoli Introduzione a Chef 12/09/2012 41 / 50
    • Monitoring - ServerOn the server, the check mk:: server recipe installs and configures icinga,nsca, nagvis, pnp4nagios smokeping and check mk. @gbagnoli Introduzione a Chef 12/09/2012 42 / 50
    • Monitoring - ServerOn the server, the check mk:: server recipe installs and configures icinga,nsca, nagvis, pnp4nagios smokeping and check mk.The check mk configuration file (which is used by check mk to configureactive and passive checks in icinga) is managed as a template by therecipe. The recipe use the search API to discover nodes using aconfigurable query. @gbagnoli Introduzione a Chef 12/09/2012 42 / 50
    • Monitoring - ServerOn the server, the check mk:: server recipe installs and configures icinga,nsca, nagvis, pnp4nagios smokeping and check mk.The check mk configuration file (which is used by check mk to configureactive and passive checks in icinga) is managed as a template by therecipe. The recipe use the search API to discover nodes using aconfigurable query.nodes = search(:node, node["check_mk"]["search_query"])default query: default [”check mk”][”search query”] = ”chef environment:production” @gbagnoli Introduzione a Chef 12/09/2012 42 / 50
    • Monitoring - Server (2)So, when a node is promoted to production, as soon as chef runs on themonitoring server, @gbagnoli Introduzione a Chef 12/09/2012 43 / 50
    • Monitoring - Server (2)So, when a node is promoted to production, as soon as chef runs on themonitoring server, • It adds the node to the check mk conf file (the template is the same, but data has changed) @gbagnoli Introduzione a Chef 12/09/2012 43 / 50
    • Monitoring - Server (2)So, when a node is promoted to production, as soon as chef runs on themonitoring server, • It adds the node to the check mk conf file (the template is the same, but data has changed) • . . . the modified templates notifies the check mk write conf resource, which is queued @gbagnoli Introduzione a Chef 12/09/2012 43 / 50
    • Monitoring - Server (2)So, when a node is promoted to production, as soon as chef runs on themonitoring server, • It adds the node to the check mk conf file (the template is the same, but data has changed) • . . . the modified templates notifies the check mk write conf resource, which is queuedtemplate "#{node[’check_mk’][’conf_dir’]}/main.mk" do source "check_mk_main.mk.erb" variables( :nodes => nodes, :hostgroups => hostgroups, :params => node[’check_mk’][’params’] ) mode 0644 owner "root" group "root" notifies :run, "execute[check_mk_write_conf]"end @gbagnoli Introduzione a Chef 12/09/2012 43 / 50
    • Monitoring - Server (3)The check mk write conf resource regenerates icinga conf @gbagnoli Introduzione a Chef 12/09/2012 44 / 50
    • Monitoring - Server (3)The check mk write conf resource regenerates icinga confexecute "check_mk_write_conf" do command "#{node[’check_mk’][’prefix’]}/bin/check_mk -O" action :nothingend @gbagnoli Introduzione a Chef 12/09/2012 44 / 50
    • Monitoring - Server (3)The check mk write conf resource regenerates icinga confexecute "check_mk_write_conf" do command "#{node[’check_mk’][’prefix’]}/bin/check_mk -O" action :nothingendThen the recipe scan nodes for services (using check mk inventory)This is done only the first time (no reinventory)nodes.each do |n| if not n[’tags’] or not n[’tags’].include? "noagent" check_mk_inventory n[’fqdn’] endendcheck mk inventory is an LWRP defined in the check mk cookbook @gbagnoli Introduzione a Chef 12/09/2012 44 / 50
    • Monitoring - Server (4)That way new nodes are automatically added to the monitoring when theyshow up as result of the search query.Since cookbooks push and configure check mk plugins, everything getsmonitored andall the monitoring logic is in the cookbook itself. @gbagnoli Introduzione a Chef 12/09/2012 45 / 50
    • Monitoring Chef ClientsOnce you start relying on chef, you want to know if chef-client iscorrectly running on nodes and if/when it fails. @gbagnoli Introduzione a Chef 12/09/2012 46 / 50
    • Monitoring Chef ClientsOnce you start relying on chef, you want to know if chef-client iscorrectly running on nodes and if/when it fails.Instead on relying on the active model of check mk, we use a completelypassive approach. We use NSCA for this. @gbagnoli Introduzione a Chef 12/09/2012 46 / 50
    • Monitoring Chef ClientsOnce you start relying on chef, you want to know if chef-client iscorrectly running on nodes and if/when it fails.Instead on relying on the active model of check mk, we use a completelypassive approach. We use NSCA for this.When chef-client runs on the node, it pushes a report/exceptionhandler called NSCAHandler.This handler runs at the end of a chef-client run, and submits theresult to the NSCA server on the monitoring host, so that: @gbagnoli Introduzione a Chef 12/09/2012 46 / 50
    • Monitoring Chef ClientsOnce you start relying on chef, you want to know if chef-client iscorrectly running on nodes and if/when it fails.Instead on relying on the active model of check mk, we use a completelypassive approach. We use NSCA for this.When chef-client runs on the node, it pushes a report/exceptionhandler called NSCAHandler.This handler runs at the end of a chef-client run, and submits theresult to the NSCA server on the monitoring host, so that: • check is in critical state if chef run failed. • check is in warning/critical state if time elapsed is above thresholds • check is in warning/critical state if the number of modified resources is above thresholds @gbagnoli Introduzione a Chef 12/09/2012 46 / 50
    • Monitoring Chef Clients (2)Passive checks are created on the server for every node which runs thechef-clientThese checks have a freshness threshold of 1 day, so that if no dataarrives from the node the check will enter the UNKOWN state, meaning thechef-client is not running on that node. @gbagnoli Introduzione a Chef 12/09/2012 47 / 50
    • Monitoring - Final WordsThe check mk:: server recipe also configures: • host parents • host groups • service groups • notification periods for services • contact groups and administrators (data is in the accounts data bag) • uses smokeping to perfom hosts checks (instead of using check ping) • . . . and integrates smokeping web ui with check mk multisite. • host icon for the status map :-)Behavior can be changed by setting nodes tag with knife. i.e, this changesthe notification period for a host.knife tag create myhost.example.com workhours @gbagnoli Introduzione a Chef 12/09/2012 48 / 50
    • Questions?Bagnoli Giacomog.bagnoli@asidev.comtwitter.com/@gbagnoligithub.com/gbagnolibitbucket.org/gbagnoligplus.to/gbagnoli @gbagnoli Introduzione a Chef 12/09/2012 49 / 50
    • Thank you!@gbagnoli Introduzione a Chef 12/09/2012 50 / 50