Automated Deploymentof OpenStack with Chef    Texas Linux Fest      April 2, 2011                         1
Introductions            Matt Ray            Senior Technical Evangelist            matt@opscode.com            @mattray  ...
What is OpenStack?                     3
Founders operate atmassive scale                 NASA                         4
OpenStack: The Mission           "To produce the ubiquitous Open           Source cloud computing platform           that ...
OpenStack Founding Principles          Apache 2.0 license (OSI), open development process          Open design process, 2x...
Community with Broad Support                               7
Software to provision virtual machines on                    standard hardware at massive scaleOpenStack Computecreating o...
OpenStack Compute Key Features                                             ReST-based API       Asynchronous eventually co...
User ManagerCloud Controller: Global state ofsystem, talks to LDAP, OpenStackObject Storage, and node/storageworkers throu...
Hardware Requirements           OpenStack is designed to run on industry           standard hardware, with flexible config...
Why is OpenStack important?         Open eliminates vendor lock-in         Working together, we all go faster         Free...
What is Chef?                13
Chef enables Infrastructure as Code           Manage configuration as idempotent           Resources.           Put them to...
At a High Level           Library for configuration management           Configuration management system           Systems i...
Fully automated Infrastructure                  16
Principles             Idempotent             Data-driven             Sane defaults             Hackability             TM...
Open Source and Community          Apache 2 licensed          Large and active community          Over 300 individual cont...
19
How does it Work?                    20
How does it Work?    Miracles!                    21
How does it Work?    Miracles!   (no really)                    22
Chef Client runs on  your System                      23
Chef Client runs on  your System       ohai!                      24
Clients talk to the   Chef Server                      25
The Opscode Platformis a hosted Chef Server                          26
We call each system you configure a        Node                      27
Nodes have Attributes{  "kernel": {     "machine": "x86_64",     "name": "Darwin",                                       K...
Nodes have a Run List  What Roles and Recipes    to Apply in Order                           29
Nodes have Roleswebserver, database, monitoring, etc.                                        30
Roles have a Run List  What Roles and Recipes    to Apply in Order                           31
name "webserver"description "Systems that serve HTTP traffic"run_list(  "role[base]",  "recipe[apache2]",  "recipe[apache2...
name "webserver"description "Systems that serve HTTP traffic"run_list(  "role[base]",                                 Can ...
Chef managesResources on Nodes                     33
Resources   Declare a description of the state a part of the node should be in                                            ...
Resources                               package "apache2" do                                 version "2.2.11-2ubuntu2.6"  ...
Resources ‣ Have a type                 package "apache2" do                                 version "2.2.11-2ubuntu2.6"  ...
Resources ‣ Have a type                 package "apache2" do                                 version "2.2.11-2ubuntu2.6"  ...
Resources ‣ Have a type                 package "apache2" do                                 version "2.2.11-2ubuntu2.6"  ...
Resources ‣ Have a type                  package "apache2" do                                  version "2.2.11-2ubuntu2.6"...
Resources take action  through Providers                        35
Recipes are lists of   Resources                       36
Recipes                                           1                           package "apache2" do                        ...
Order Matters                38
Recipes are just Ruby!extra_packages = case node[:platform]  when "ubuntu","debian"    %w{      ruby1.8      ruby1.8-dev  ...
Cookbooks arepackages for Recipes                       40
Cookbooks            Distributable, shareable            comunity.opscode.com            Infrastructure as Code           ...
Cookbooks            Recipes            Files            Templates            Attributes            Metadata              ...
Data bags store arbitrary data                  43
A user data bag item...% knife data bag show users mray{  "comment": "Matt Ray",  "groups": "sysadmin",  "ssh_keys": "ssh-...
Environments manageversioned infrastructure                           45
Command-line API  utility, Knife       http://www.flickr.com/photos/myklroventine/3474391066/     Copyright © 2011 Opscode,...
Search                         $ knife search node platform:ubuntu‣ CLI or in Ruby                         search(:node, ‘...
48
HOW TO: Turn Racks ofStandard Hardware Into a  Cloud with OpenStack                           49
What Works Today?                    50
Compute (Nova)          Single machine installation         ‣ Role: nova-single-machine         ‣ MySQL, RabbitMQ         ...
Role: nova-single-machinename "nova-single-machine-install"description "Installs everything required to run Nova on a sing...
Role: nova-multi-controllername "nova-multi-controller"description "Installs requirements to run the Controller node in aN...
Role: nova-multi-computename "nova-multi-compute"description "Installs requirements to run a Compute node in a Novacluster...
What does this look like?                            55
Crowbar‣ Codename for the OpenStack installer from Dell ‣ Dell is releasing this under the  Apache 2 license ‣ Extension o...
Crowbar - What does it Do?‣ Crowbar is a PXE state machine‣ starts with bare metal hardware‣ manages and configures BIOS an...
OpenStack Installation‣ Cookbooks uploaded   $                       $                           knife cookbook upload -a ...
AMIsname "nova-ami-urls"description "Feed in a list URLs for AMIs to download"default_attributes(  "nova" => {    "images"...
Assign the Roles$ knife node run_list add crushinator.localdomain "role[nova-ami-urls]"{  "run_list": [    "role[nova-ami-...
chef-clientmray@ubuntu1010:~$ sudo chef-client[Fri, 25 Feb 2011 11:52:59 -0800] INFO:   Starting Chef Run (Version0.9.12)....
The Moment of Truthnova@$ nova-manage service listnova@$ euca-describe-imagesnova@$ euca-run-instances ami-h8wh0j17 -k myk...
How Did We Get Here?                       63
Forked from Anso Labs’ Cookbooks        Bootstrapped by Opscode        Chef Solo/Vagrant installs for Developers        ht...
Who’s involvedso far?                 65
What’s Next?               66
Nova needed enhancements         Pluggable/Modular Roles         ‣ Database         ‣ ObjectStore         ‣ Network       ...
68
Dashboard            69
Knife        ‣ http://github.com/opscode/knife-openstack        ‣ Nova has same API as Amazon        ‣ Fog supports OpenSt...
Object Storage (Swift)           ‣ Recipes originated from Anso Labs’ repository           ‣ Will be managed with Chef and...
Image Registry (Glance)          ‣ Recipes originated from Anso Labs’ repository          ‣ Will be managed with Chef and ...
Scaling changes howwe deploy OpenStack!                       73
Deployment Scenarios         ‣ Single machine is special case of multi-install         ‣ Controller + Compute nodes is a k...
Cactus, Diablo, ...            Development continues...            Branches for each stable release            Design Summ...
Rackspace Cloud Builders          Commercial support and Training for          OpenStack          ‣ Opscode          ‣ Del...
Get Involved!    https://github.com/mattray/openstack-cookbooks/tree/bexar    http://lists.openstack.org    http://lists.o...
Upcoming SlideShare
Loading in …5
×

TXLF: Automated Deployment of OpenStack with Chef

4,473 views
4,313 views

Published on

Automated Deployment of OpenStack with Chef talk given April 2, 2011 at the Texas Linux Fest.

Published in: Technology
1 Comment
6 Likes
Statistics
Notes
No Downloads
Views
Total views
4,473
On SlideShare
0
From Embeds
0
Number of Embeds
83
Actions
Shares
0
Downloads
242
Comments
1
Likes
6
Embeds 0
No embeds

No notes for slide

TXLF: Automated Deployment of OpenStack with Chef

  1. 1. Automated Deploymentof OpenStack with Chef Texas Linux Fest April 2, 2011 1
  2. 2. Introductions Matt Ray Senior Technical Evangelist matt@opscode.com @mattray GitHub:mattray 2
  3. 3. What is OpenStack? 3
  4. 4. Founders operate atmassive scale NASA 4
  5. 5. OpenStack: The Mission "To produce the ubiquitous Open Source cloud computing platform that will meet the needs of public and private cloud providers regardless of size, by being simple to implement and massively scalable." 5
  6. 6. OpenStack Founding Principles Apache 2.0 license (OSI), open development process Open design process, 2x year public Design Summits Publicly available open source code repositories Open community processes documented and transparent Commitment to drive and adopt open standards Modular design for deployment flexibility via APIs 6
  7. 7. Community with Broad Support 7
  8. 8. Software to provision virtual machines on standard hardware at massive scaleOpenStack Computecreating open source software to build public and private clouds Software to reliably store billions of objects distributed across standard hardware OpenStackObject Storage 8
  9. 9. OpenStack Compute Key Features ReST-based API Asynchronous eventually consistent communication Horizontally and massively scalable Hypervisor agnostic: support for Xen ,XenServer, Hyper-V, KVM, UML and ESX is coming Hardware agnostic: standard hardware, RAID not required 9
  10. 10. User ManagerCloud Controller: Global state ofsystem, talks to LDAP, OpenStackObject Storage, and node/storageworkers through a queue ATAoE / iSCSIAPI: Receives HTTP requests,converts commands to/from APIformat, and sends requests to cloudcontroller Host Machines: workers that spawn instances Glance: HTTP + OpenStack ObjectOpenStack Compute Storage for server images 10
  11. 11. Hardware Requirements OpenStack is designed to run on industry standard hardware, with flexible configurations Compute x86 Server (Hardware Virt. recommended) Storage flexible (Local, SAN, NAS) Object Storage x86 Server (other architectures possible) Do not deploy with RAID (can use controller for cache) 11
  12. 12. Why is OpenStack important? Open eliminates vendor lock-in Working together, we all go faster Freedom to federate, or move between clouds 12
  13. 13. What is Chef? 13
  14. 14. Chef enables Infrastructure as Code Manage configuration as idempotent Resources. Put them together in Recipes. Track it like Source Code. Configure your servers. 14
  15. 15. At a High Level Library for configuration management Configuration management system Systems integration platform API for your entire Infrastructure 15
  16. 16. Fully automated Infrastructure 16
  17. 17. Principles Idempotent Data-driven Sane defaults Hackability TMTOWTDI 17
  18. 18. Open Source and Community Apache 2 licensed Large and active community Over 300 individual contributors (60+ corporate) Community is Important! 18
  19. 19. 19
  20. 20. How does it Work? 20
  21. 21. How does it Work? Miracles! 21
  22. 22. How does it Work? Miracles! (no really) 22
  23. 23. Chef Client runs on your System 23
  24. 24. Chef Client runs on your System ohai! 24
  25. 25. Clients talk to the Chef Server 25
  26. 26. The Opscode Platformis a hosted Chef Server 26
  27. 27. We call each system you configure a Node 27
  28. 28. Nodes have Attributes{ "kernel": { "machine": "x86_64", "name": "Darwin", Kernel info! "os": "Darwin", "version": "Darwin Kernel Version 10.4.0: Fri Apr 23 18:28:53 PDT 2010;root:xnu-1504.7.4~1/RELEASE_I386", }, "release": "10.4.0" Platform info! "platform_version": "10.6.4", "platform": "mac_os_x", "platform_build": "10F569", "domain": "local", "os": "darwin", "current_user": "mray", "ohai_time": 1278602661.60043, "os_version": "10.4.0", Hostname and IP! "uptime": "18 days 17 hours 49 minutes 18 seconds", "ipaddress": "10.13.37.116", "hostname": "morbo", "fqdn": "morbomorbo.local", "uptime_seconds": 1619358} 28
  29. 29. Nodes have a Run List What Roles and Recipes to Apply in Order 29
  30. 30. Nodes have Roleswebserver, database, monitoring, etc. 30
  31. 31. Roles have a Run List What Roles and Recipes to Apply in Order 31
  32. 32. name "webserver"description "Systems that serve HTTP traffic"run_list( "role[base]", "recipe[apache2]", "recipe[apache2::mod_ssl]")default_attributes( "apache" => { "listen_ports" => [ "80", "443" ] })override_attributes( "apache" => { "max_children" => "50" }) 32 32
  33. 33. name "webserver"description "Systems that serve HTTP traffic"run_list( "role[base]", Can include "recipe[apache2]", other roles! "recipe[apache2::mod_ssl]")default_attributes( "apache" => { "listen_ports" => [ "80", "443" ] })override_attributes( "apache" => { "max_children" => "50" }) 32 32
  34. 34. Chef managesResources on Nodes 33
  35. 35. Resources Declare a description of the state a part of the node should be in 34
  36. 36. Resources package "apache2" do version "2.2.11-2ubuntu2.6" action :install end template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "root" group "root" mode 0644 action :create end Declare a description of the state a part of the node should be in 34
  37. 37. Resources ‣ Have a type package "apache2" do version "2.2.11-2ubuntu2.6" action :install end template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "root" group "root" mode 0644 action :create end Declare a description of the state a part of the node should be in 34
  38. 38. Resources ‣ Have a type package "apache2" do version "2.2.11-2ubuntu2.6" action :install ‣ Have a name end template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "root" group "root" mode 0644 action :create end Declare a description of the state a part of the node should be in 34
  39. 39. Resources ‣ Have a type package "apache2" do version "2.2.11-2ubuntu2.6" action :install ‣ Have a name end template "/etc/apache2/apache2.conf" do ‣ Have parameters source "apache2.conf.erb" owner "root" group "root" mode 0644 action :create end Declare a description of the state a part of the node should be in 34
  40. 40. Resources ‣ Have a type package "apache2" do version "2.2.11-2ubuntu2.6" action :install ‣ Have a name end template "/etc/apache2/apache2.conf" do ‣ Have parameters source "apache2.conf.erb" owner "root" ‣ Take action to put the group "root" mode 0644 resource in the action :create declared state end Declare a description of the state a part of the node should be in 34
  41. 41. Resources take action through Providers 35
  42. 42. Recipes are lists of Resources 36
  43. 43. Recipes 1 package "apache2" do version "2.2.11-2ubuntu2.6" action :install end Evaluate and apply template "/etc/apache2/apache2.conf" do Resources in the order source "apache2.conf.erb" owner "root" they appear group "root" mode 0644 action :create2 end 37
  44. 44. Order Matters 38
  45. 45. Recipes are just Ruby!extra_packages = case node[:platform] when "ubuntu","debian" %w{ ruby1.8 ruby1.8-dev rdoc1.8 ri1.8 libopenssl-ruby } endextra_packages.each do |pkg| package pkg do action :install endend 39
  46. 46. Cookbooks arepackages for Recipes 40
  47. 47. Cookbooks Distributable, shareable comunity.opscode.com Infrastructure as Code Versioned Hundreds 41
  48. 48. Cookbooks Recipes Files Templates Attributes Metadata 42
  49. 49. Data bags store arbitrary data 43
  50. 50. A user data bag item...% knife data bag show users mray{ "comment": "Matt Ray", "groups": "sysadmin", "ssh_keys": "ssh-rsa SUPERSEKRATS mray@morbo", "files": { ".bashrc": { "mode": "0644", "source": "dot-bashrc" }, ".emacs": { "mode": "0644", "source": "dot-emacs" } }, "id": "mray", "uid": 7004, "shell": "/usr/bin/bash" } 44
  51. 51. Environments manageversioned infrastructure 45
  52. 52. Command-line API utility, Knife http://www.flickr.com/photos/myklroventine/3474391066/ Copyright © 2011 Opscode, Inc - All Rights Reserved 46 46
  53. 53. Search $ knife search node platform:ubuntu‣ CLI or in Ruby search(:node, ‘platform:centos’)‣ Nodes are searchable $ knife search role max_children:50‣ Roles are searchable search(:role, ‘max_children:50’)‣ Recipes are $ knife search node ‘role:webserver’ searchable search(:node, ‘role:webserver’)‣ Data bags are $ knife users ‘shell:/bin/bash’ searchable search (:users, ‘group:sysadmins’) 47
  54. 54. 48
  55. 55. HOW TO: Turn Racks ofStandard Hardware Into a Cloud with OpenStack 49
  56. 56. What Works Today? 50
  57. 57. Compute (Nova) Single machine installation ‣ Role: nova-single-machine ‣ MySQL, RabbitMQ ‣ Nova-(api|scheduler|network|objectstore|compute) Multi-machine ‣ Role: nova-multi-controller (1) ‣ Role: nova-multi-compute (N) 51
  58. 58. Role: nova-single-machinename "nova-single-machine-install"description "Installs everything required to run Nova on a singlemachine"run_list( "role[nova-multi-controller]", "role[nova-multi-compute]" ) 52
  59. 59. Role: nova-multi-controllername "nova-multi-controller"description "Installs requirements to run the Controller node in aNova cluster"run_list( "role[nova-support-server]", "role[nova-head]", "role[nova-cloud-controller]", "role[nova-super-user-setup]" ) 53
  60. 60. Role: nova-multi-computename "nova-multi-compute"description "Installs requirements to run a Compute node in a Novacluster"run_list( "recipe[nova::compute]") 54
  61. 61. What does this look like? 55
  62. 62. Crowbar‣ Codename for the OpenStack installer from Dell ‣ Dell is releasing this under the Apache 2 license ‣ Extension of the Chef server ‣ Jointly developed by Dell, Rackspace and Opscode 56
  63. 63. Crowbar - What does it Do?‣ Crowbar is a PXE state machine‣ starts with bare metal hardware‣ manages and configures BIOS and network settings‣ network boot and installation‣ nodes are configured with Chef‣ deploys OpenStack, could be used for anything 57
  64. 64. OpenStack Installation‣ Cookbooks uploaded $ $ knife cookbook upload -a knife cookbook list $ rake roles‣ Roles uploaded $ knife role list $ knife node list‣ Nodes ready 58
  65. 65. AMIsname "nova-ami-urls"description "Feed in a list URLs for AMIs to download"default_attributes( "nova" => { "images" =>["http://192.168.11.7/ubuntu1010-UEC-localuser-image.tar.gz”] } )$ knife role from file roles/nova-ami-urls.rb‣ Use an existing AMI‣ Update URL to your own 59
  66. 66. Assign the Roles$ knife node run_list add crushinator.localdomain "role[nova-ami-urls]"{ "run_list": [ "role[nova-ami-urls]" ]}$ knife node run_list add crushinator.localdomain "role[nova-single-machine-install]"{ "run_list": [ "role[nova-ami-urls]" "role[nova-single-machine-install]", ]} 60
  67. 67. chef-clientmray@ubuntu1010:~$ sudo chef-client[Fri, 25 Feb 2011 11:52:59 -0800] INFO: Starting Chef Run (Version0.9.12)...[Fri, 25 Feb 2011 11:56:05 -0800] INFO: Chef Run complete in5.911955 seconds[Fri, 25 Feb 2011 11:56:05 -0800] INFO: cleaning the checksum cache[Fri, 25 Feb 2011 11:56:05 -0800] INFO: Running report handlers[Fri, 25 Feb 2011 11:56:05 -0800] INFO: Report handlers complete 61
  68. 68. The Moment of Truthnova@$ nova-manage service listnova@$ euca-describe-imagesnova@$ euca-run-instances ami-h8wh0j17 -k mykey -t m1.tinynova@$ euca-describe-instancesnova@$ ssh -i mykey.priv ubuntu@10.0.0.2Linux i-00000001 2.6.35-24-virtual #42-Ubuntu SMP Thu Mar 3005:15:26 UTC 2011 x86_64 GNU/LinuxUbuntu 10.10Welcome to Ubuntu!<SNIP>See "man sudo_root" for details.ubuntu@i-00000001:~$ 62
  69. 69. How Did We Get Here? 63
  70. 70. Forked from Anso Labs’ Cookbooks Bootstrapped by Opscode Chef Solo/Vagrant installs for Developers http://github.com/ansolabs/openstack-cookbooks 64
  71. 71. Who’s involvedso far? 65
  72. 72. What’s Next? 66
  73. 73. Nova needed enhancements Pluggable/Modular Roles ‣ Database ‣ ObjectStore ‣ Network ‣ Virtualization Swift and Glance integration 67
  74. 74. 68
  75. 75. Dashboard 69
  76. 76. Knife ‣ http://github.com/opscode/knife-openstack ‣ Nova has same API as Amazon ‣ Fog supports OpenStack already ‣ knife openstack server create ‘role [base]’ -i ami-a403f6xd -f m1.micro - A “OpenStack instance” 70
  77. 77. Object Storage (Swift) ‣ Recipes originated from Anso Labs’ repository ‣ Will be managed with Chef and Crowbar ‣ Included in the ‘bexar’ branch ‣ Untested so far (Cactus will tackle) 71
  78. 78. Image Registry (Glance) ‣ Recipes originated from Anso Labs’ repository ‣ Will be managed with Chef and Crowbar ‣ Included in the ‘bexar’ branch ‣ Untested so far (Cactus!) 72
  79. 79. Scaling changes howwe deploy OpenStack! 73
  80. 80. Deployment Scenarios ‣ Single machine is special case of multi-install ‣ Controller + Compute nodes is a known quantity for small installations ‣ Nova + Swift + Glance in large installations ‣ Services separated and HA configurations supported ‣ Documentation and Chef Roles will be the solution 74
  81. 81. Cactus, Diablo, ... Development continues... Branches for each stable release Design Summit later this month Design Summit in the Fall 75
  82. 82. Rackspace Cloud Builders Commercial support and Training for OpenStack ‣ Opscode ‣ Dell ‣ Equinix ‣ Cloudscaling ‣ Citrix 76
  83. 83. Get Involved! https://github.com/mattray/openstack-cookbooks/tree/bexar http://lists.openstack.org http://lists.opscode.com #chef on irc.freenode.net #openstack on irc.freenode.net matt@opscode.com jordan@openstack.com 77

×