Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

SCALE 10x Build a Cloud Day


Published on

Matt Ray's Introduction to Chef talk from the 2012 Southern California Linux Expo's Build a Cloud Day.

Published in: Technology

SCALE 10x Build a Cloud Day

  1. 1. Introduction to ChefSCALE Build a Cloud Day @mattray
  2. 2. Congratulations!!! • U has a cloud • Now what?
  3. 3. APIs are awesome!• You can provision compute resources in seconds• You can provision storage resources in seconds• That’s cool.
  4. 4. Chef can help with that• knife ec2 server • knife cloudstack create server create• knife rackspace • knife kvm server server create create• knife terremark • knife vsphere server create server create• knife voxel • knife eucalyptus server create server create• knife gandi • knife openstack server create server create
  5. 5. But then what?
  6. 6. You need to configure them
  7. 7. See NodeApplication Server
  8. 8. See NodesApplication ServerApplication Database
  9. 9. See Nodes GrowApplication ServerApplication Databases
  10. 10. See Nodes GrowApplication ServersApplication Databases
  11. 11. See Nodes GrowLoad BalancerApplication ServersApplication Databases
  12. 12. See Nodes GrowLoad Balancers Application ServersApplication Databases
  13. 13. See Nodes GrowLoad Balancers Application ServersApplication Database CacheApplication Databases
  14. 14. Tied together with Config Load Balancers Application Servers Application Database Cache Application Databases
  15. 15. Infrastructure is a Snowflake Load Balancers Application Servers Application Database Cache Floating IP? Application Databases
  16. 16. Evolving Complexity Load Balancers ApplicationCache Application ServersNoSQL Database Cache Database Slaves Database
  17. 17. Complexity Grows Quickly DC2DC1 DC3
  18. 18. And it Continues to Evolve
  19. 19. Golden Images are not the answer• Gold is heavy• Hard to transport• Hard to mold• Easy to lose configuration detail
  20. 20. Typical Boring InfrastructureGraphite Nagios Jboss App Memcache Postgres Slaves Postgres Master
  21. 21. New Compliance Mandate Graphite Nagios Jboss App• Move SSH off port 22• Lets put it on 2022 Memcache Postgres Slaves Postgres Master
  22. 22. 6 Golden Image Updates Graphite 1 2 Nagios 3 Jboss App• edit /etc/ssh/sshd_config 4 Memcache 5 Postgres Slaves 6 Postgres Master
  23. 23. 12 Instance Replacements Graphite 1 2 Nagios 3 4 5 6 7 Jboss App• Delete, launch 8• Repeat 9 Memcache• Typically manually 10 11 Postgres Slaves 12 Postgres Master
  24. 24. In a Maintenance Window Graphite 1 2 Nagios 3 4 5 6 7 Jboss App• Dont break anything! 8 9 Memcache 5 10 11 Postgres Slaves 12 Postgres Master
  25. 25. With Different IP Addresses? Graphite Nagios Jboss App• Invalid Configs• Bob just got fired :( Memcache Postgres Slaves Postgres Master
  26. 26. Configuration Desperation
  27. 27. Configuration Management andAutomated Systems Integration is the Answer
  28. 28. Chef Solves This Problem • But you already guessed that, didn’t you?
  29. 29. Collections of Resources• Networking • Routes • Users• Files • Groups• Directories • Tasks• Symlinks • Packages• Mounts • Software • Services • Configurations • Other Stuff
  30. 30. Acting in Concert
  31. 31. To Provide a Service
  32. 32. Chef is Infrastructure as Code • Programmatically provision and configure • Treat like any other code base • Reconstruct business from code repository, data backup, and bare metal resources.
  33. 33. Declarative Interface to Resources• Define policy• Say what, not how• Pull not Push
  34. 34. That looks like thisextra_packages = case node[platform] when "ubuntu","debian" %w{ ruby1.8 ruby1.8-dev rdoc1.8 ri1.8 libopenssl-ruby } endextra_packages.each do |pkg| package pkg do action :install endend
  35. 35. Or thissearch(:users, *:*) do |u| user u[id] do uid u[uid] shell u[shell] home "/home/#{u[id]}" end directory "#{home_dir}/.ssh" do owner u[id] group u[gid] mode "0700" end template "#{home_dir}/.ssh/authorized_keys" do source "authorized_keys.erb" owner u[id] group u[id] mode "0600" variables :ssh_keys => u[ssh_keys] endend
  36. 36. Recipes and Cookbooks• Recipes are collections of Resources• Cookbooks contain recipes, templates, files, custom resources, etc• Code re-use and modularity• Hundreds already on
  37. 37. Nodes • Chef-Client generates configurations directly on nodes • Reduce management complexity through abstraction • Store the configuration of your programs in version control
  38. 38. Upload your infrastructureknife cookbook upload aptknife cookbook upload chef-clientknife cookbook upload javaknife cookbook upload jpackageknife cookbook upload ntpknife cookbook upload sudoknife cookbook upload tomcatknife cookbook upload usersknife cookbook upload sampleknife role from file base-cloud.rbknife role from file tc.rbknife role from file sample.rbknife data bag create usersknife data bag from file users mray.json
  39. 39. Build it somewhere#EC2knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-a7a97dce-f m1.small -d omnibus -r role[base-cloud],role[tc],role[sample]#Rackspaceknife rackspace server create --image 110 --flavor 2 -i ~/.ssh/mray.pem -d omnibus -rrole[base-cloud],role[tc],role[sample]#CloudStackknife cs server create -S "small instance" -T "CentOS 5.5(64-bit) no GUI (KVM)" -i~/.ssh/mray.pem -d omnibus -r role[base-cloud],role[tc],role[sample]#Ubuntu Linuxknife bootstrap test.lab -r role[webserver] -i ~/.ssh/mray.pem -x ubuntu --sudo -domnibus -r role[base-cloud],role[tc],role[sample]
  40. 40. Search• Search for nodes with Roles• Find configuration data• IP addresses• Hostnames• FQDNs
  41. 41. Pass results into Templatespool_members = search("node","role:webserver”)template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]"end
  42. 42. Pass results into Templates# Set up application listeners here.listen application balance roundrobin <% @pool_members.each do |member| -%> server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check <% end -%><% if node["haproxy"]["enable_admin"] -%>listen admin mode http stats uri /<% end -%>
  43. 43. So when thisGraphite Nagios Jboss App Memcache Postgres Slaves Postgres Master
  44. 44. Becomes thisGraphite Nagios Jboss App Memcache Postgres Slaves Postgres Master
  45. 45. Updates can be automaticGraphite Nagios Jboss App Memcache Postgres Slaves Postgres Master
  46. 46. Count the resources • Load balancer config Graphite Nagios • Nagios host ping • Nagios host ssh Jboss App • Nagios host HTTP • Nagios host app health Memcache • Graphite CPU • Graphite Memory Postgres Slaves • Graphite Disk • Graphite SNMP • Memcache firewall• 12+ resource changes for 1 node addition • Postgres firewall Postgres authZ config
  48. 48. Build anything • Simple internal applications • Complex external applications • Workstations • Hadoop clusters • IaaS infrastructure • PaaS infrastructure • SaaS applications • Storage systems • You name it
  49. 49. And manage it simply• Automatically reconfigure everything• Linux, Windows, Unixes, BSDs• Load balancers• Metrics collection systems• Monitoring systems• Cloud migrations become trivial
  50. 50. The Chef Community• Apache License, Version 2.0• 550+ Individual contributors• 100+ Corporate contributors • Dell, Rackspace,VMware, RightScale, Heroku, and many more• Nearly 400 cookbooks•
  51. 51. Questions?
  52. 52. Thanks!