Achieving Infrastructure Portability with Chef

Achieving Infrastructure Portabilitywith Chef CloudOpen 2012 Matt Ray matt@opscode.com IRC/Twitter/GitHub: mattray www.opscode.com

Data Centers...

To the Cloud!

Why the Cloud?• Instant infrastructure• Unlimited capacity• Autoscaling• No commitment• Immediate replacement

Cloud Differentiation• Each cloud defines themselves against Amazon• Entry into the market is easier• Not a lot of price competition• Feature parity is growing

The Dark Side of the Cloud

Why not the Cloud?• Reliability• Performance• Security• Price

Data Gravity

Know our escape planfor every infrastructure provider

Why Chef?

See NodeApplication Server

See NodesApplication ServerApplication Database

See Nodes GrowApplication ServerApplication Databases

See Nodes GrowApplication ServersApplication Databases

See Nodes GrowLoad BalancerApplication ServersApplication Databases

See Nodes GrowLoad Balancers Application ServersApplication Databases

See Nodes GrowLoad Balancers Application ServersApplication Database CacheApplication Databases

Tied together with ConfigLoad Balancers Application ServersApplication Database CacheApplication Databases

Infrastructure is a SnowflakeLoad Balancers Application ServersApplication Database CacheFloating IP?Application Databases

Evolving Complexity Load Balancers ApplicationCache Application ServersNoSQL Database Cache Database Slaves Database

Complexity Grows Quickly DC2DC1 DC3

And it Continues to EvolveThats great and all, but tell me about Chef! http://www.ﬂickr.com/photos/16339684@N00/2681435235/

Chef is Infrastructure as Code • Programmatically provision and configure • Treat like any other code base • Reconstruct business from code repository, data backup, and bare metal resources. http://www.ﬂickr.com/photos/louisb/4555295187/

Nodes • Chef-Client generates configurations directly on nodes from their run list • Reduce management complexity through abstraction • Store the configuration of your programs in version controlhttp://www.flickr.com/photos/ssoosay/5126146763/

Collections of Resources• Networking • Routes • Users• Files • Groups• Directories • Tasks• Symlinks • Packages• Mounts • Software • Services • Configurations • Other Stuff http://www.ﬂickr.com/photos/stevekeys/3123167585/

Declarative Interface to Resources• Define policy• Say what, not how• Pull not Pushhttp://www.ﬂickr.com/photos/bixentro/2591838509/

Ruby!extra_packages = case node[platform] when "ubuntu","debian" %w{ ruby1.8 ruby1.8-dev rdoc1.8 ri1.8 libopenssl-ruby } endextra_packages.each do |pkg| package pkg do action :install endend

Recipes and Cookbooks• Recipes are collections of Resources• Cookbooks contain recipes, templates, files, custom resources, etc• Code re-use and modularity• Hundreds already on Community.opscode.com http://www.flickr.com/photos/shutterhacks/4474421855/

Search• Search for nodes with Roles• Find configuration data• IP addresses• Hostnames• FQDNs http://www.ﬂickr.com/photos/kathycsus/2686772625

Pass Results to Templatespool_members = search("node","role:webserver”)template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]"end

Pass Results to Templates# Set up application listeners here.listen application 0.0.0.0:80 balance roundrobin <% @pool_members.each do |member| -%> server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check <% end -%><% if node["haproxy"]["enable_admin"] -%>listen admin 0.0.0.0:22002 mode http stats uri /<% end -%>

So when thisGraphite Nagios Jboss App Memcache Postgres Slaves Postgres Master

Becomes thisGraphite Nagios Jboss App Memcache Postgres Slaves Postgres Master

Updates can be automaticGraphite Nagios Jboss App Memcache Postgres Slaves Postgres Master

Count the resources • Load balancer config Graphite Nagios • Nagios host ping • Nagios host ssh Jboss App • Nagios host HTTP • Nagios host app health Memcache • Graphite CPU • Graphite Memory Postgres Slaves • Graphite Disk • Graphite SNMP • Memcache firewall• 12+ resource changes for 1 node addition • Postgres firewall Postgres authZ config

Build anything • Simple internal applications • Complex external applications • Workstations • Hadoop clusters • IaaS infrastructure • PaaS infrastructure • SaaS applications • Storage systems • You name ithttp://www.ﬂickr.com/photos/hyku/245010680/

And manage it simply• Automatically reconfigure everything• Linux, Windows, Unixes, BSDs• Load balancers• Metrics collection systems• Monitoring systems• Cloud migrations become trivial http://www.ﬂickr.com/photos/helico/404640681/

knife with the Chef Server• knife node • create/delete/edit • list• knife cookbook ...• knife role ...• knife environment ...

knife bootstrap knife bootstrap SERVER -r role[webserver] -i ~/.ssh/id_rsa• SSH to the machine given existing credentials• Install the Chef Client• Register with the Chef Server• Run the initial Run List• Now managed with Chef!

knife ec2$ knife ec2Available ec2 subcommands: (for details, knife SUB-COMMAND --help)** EC2 COMMANDS **knife ec2 flavor list (options)knife ec2 instance data (options)knife ec2 server create (options)knife ec2 server delete SERVER [SERVER] (options)knife ec2 server list (options)$ knife ec2 server create -S keypair -i ~/.ssh/id_rsa -x ubuntu-I ami-4721882e -f m1.small -r role[webserver]

knife openstack$ knife openstackAvailable openstack subcommands: (for details, knife SUB-COMMAND --help)** OPENSTACK COMMANDS **knife openstack flavor list (options)knife openstack image list (options)knife openstack server create (options)knife openstack server delete SERVER [SERVER] (options)knife openstack server list (options)$ knife openstack server create -S keypair -i ~/.ssh/id_rsa-x ubuntu -I 1231 -f standard.small -r role[webserver]

Chef for Infrastructure Portability• knife ec2 • knife cloudstack• knife rackspace • knife openstack• knife hp • knife vsphere• knife google • ... and many• knife azure others

The Chef Community• Apache License, Version 2.0• 850+ Individual contributors• 150+ Corporate contributors • HP, Dell, Rackspace, VMware, Joyent, Calxeda, Heroku, SUSE and many more• 550+ cookbooks• http://community.opscode.com

Desktop, Virtualization, Private & Public Clouds• Vagrant • AWS• VMware • Rackspace• CloudStack • HP• Eucalyptus • Google• OpenStack • Azure• bare metal • many others

Tale of the Tape• Artur Bergman, CEO at Fastly • Its All About Speed • http://youtu.be/qRnTejOMbZU• Jason Stowe, CEO at Cycle Computing • CycleCloud + Chef = 50,000-core Utility Supercomputer for Science • http://youtu.be/cEaQB6e7G0Q

Chef and Abstractions• Resources and Providers• Cookbooks (may) normalize deployment• Knife treats APIs the same• Chef strives to not be opinionated

Environments• Lock down versions of cookbooks • stable releases vs. development• Enforce attributes for deployment • Ports, addresses, etc.• Different run lists based on environment • Debugging enabled?

Environments• Use the same infrastructure code for wherever you deploy• Development, QA, Pre-Production, Prod• Role-based Access Controls to restrict the promotion of deployment code

TL;DL• Every infrastructure is a unique snowflake• Understand the costs associated with the features of your platform(s) of choice.• Chef enables Infrastructure Portability• "Data Gravity" is the primary concern

Thanks! Matt Ray matt@opscode.com IRC/Twitter/GitHub: mattray www.opscode.com

Achieving Infrastructure Portability with Chef

by Matt Ray on Aug 30, 2012

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 18

Statistics