Chef for OpenStack
Matt Ray & JJ Asghar
OSCON
July 24, 2014
Introductions
• Matt Ray
• Director of Partner
Engineering
• matt@getchef.com
• mattray GitHub|IRC|Twitter
Introductions
• JJ Asghar
• Sr. Partner Engineer
• jj@getchef.com
• jjasghar GitHub and Twitter
• j^2 on IRC
Chef
•Open Source configuration management
and systems automation framework
•Infrastructure as Code, written in Ruby
•Abstractions of Resources on the
systems
•Client/server model over REST APIs
•Huge, vibrant community of contributors
™
OpenStack Controls Compute, Storage & Networking
Architecture
Architecture
Overview & Current Status of
Chef for OpenStack
Chef for OpenStack: Project
• Community around the automated deployment and
management of OpenStack
• Reduce fragmentation and increase collaboration
• Deploying OpenStack is not "Secret Sauce"
• Project, not a 'Product'
• Apache 2 License
Community
• #openstack-chef on irc.freenode.net
• groups.google.com/group/opscode-chef-openstack
• Google Hangout weekly status meetings (Monday
11am EST)
• @chefopenstack
• Stackalytics (stackforge->chef-group)
Who's Involved?
• AT&T
• Bluebox
• Chef
• Dell
• DreamHost
• HP
• IBM
• Korea Telecom
• Rackspace
• SUSE
• and many more
Chef Requirements
• Chef 11
• Ruby 1.9.x
• Foodcritic, Rubocop, ChefSpec for
testing
• attribute-driven by Environments
• platform logic in attributes
• currently packages-only installation
StackForge: Cookbooks
• "Official" OpenStack repositories
•github.com/stackforge/cookbook-openstack-*
•gated by review.openstack.org
• OpenStack services for Grizzly, Havana and Icehouse cookbooks
•block-storage, common, compute, dashboard, data-processing,
database, identity, image, network, object-storage, orchestration,
telemetry
• Operational support cookbooks
•ceph, client, integration-testing, ops-database, ops-messaging
Launchpad
• Blueprints
•https://blueprints.launchpad.net/
openstack-chef
• Bugs
•https://bugs.launchpad.net/
openstack-chef
StackForge: Deployment
• Chef repository for deploying Grizzly, Havana, or Icehouse
•example Environments and Roles
•example "All-in-One" Vagrant deployment
•github.com/stackforge/openstack-chef-repo
• Gated by review.openstack.org
•More single and multi-node testing coming
Reference Implementation
• Deployment examples in documentation
•All-in-One Compute
•Single Controller + N Compute
•more coming
• Will provide example HA configurations
• Operations outside of scope of core repository
•logging, monitoring, provisioning
docs.getchef.com/
openstack.html
Documentation
• docs.getchef.com/openstack.html
•Architecture
•Deployment Prerequisites
•Installation
•Development
•Cookbooks and Repositories
•Example Deployments
• github.com/opscode/chef-docs
•Creative Commons, no CLA required
Example Deployments
• Vagrant "All-in-One" for development/testing
•nova-network or Neutron
•Ubuntu 12.04 or CentOS 6.5
• Developer lab deployment "1+N"
•Single controller, N compute boxes
•5 boxes, consumer-grade hardware
StackForge: Grizzly Status
• branch ‘stable/grizzly’
• Operating Systems: Ubuntu 12.04, SLES 11 SP2
• Databases: MySQL, SQLite (testing)
• Messaging: RabbitMQ
• Compute: KVM, LXC, Qemu
• Network: Nova + Quantum (Open vSwitch)
• Block Storage: LVM
• Object Storage: Swift
• Dashboard: Apache or Nginx
StackForge: Havana Status
• branch ‘stable/havana’
• Operating Systems: Ubuntu 12.04, SLES 11 SP3, RHEL 6
• Databases: DB2, MySQL, SQLite (testing)
• Messaging: Qpid, RabbitMQ
• Compute: KVM, LXC, QEMU
• Network: Nova, Neutron (Open vSwitch)
• Block Storage: LVM, NetApp, others
• Object Storage: Swift
• Dashboard: Apache or Nginx
StackForge: Icehouse Status
• branch ‘master’
• Operating Systems: Ubuntu 12.04/14.04, SLES 11 SP3, RHEL 6
• Databases: DB2, MySQL Postgres, SQLite (testing)
• Messaging: Qpid, RabbitMQ
• Compute: KVM, LXC, QEMU
• Network: Nova, Neutron (Open vSwitch)
• Block Storage: LVM, NetApp, others
• Object Storage: Swift
• Dashboard: Apache or Nginx
StackForge: Roadmap
• branching for Juno (tomorrow?)
•'master' move to ‘stable/icehouse’
•Havana available in ‘stable/havana'
•Grizzly available in ‘stable/grizzly'
• More contributors!
StackForge: Roadmap
• Operating Systems: Debian
• Compute: Baremetal, Docker, ESX,
Hyper-V, Xen
• Network: NSX, OpenDaylight
• Block Storage: Ceph
• Object Storage: Ceph
• Source builds via Omnibus
OpenStack Ops Mid-Cycle Meetup
• Operators mini summit
• August 25-26, San Antonio TX
• http://www.eventbrite.com/e/
openstack-ops-mid-cycle-meetup-
tickets-12149171499
Chef Community Summit
• Developer/Community unconference
• October 2 & 3 in Seattle, WA
OpenStack Summit
knife-openstack
Available openstack subcommands: (for details, knife SUB-COMMAND --help)!
!
** OPENSTACK COMMANDS **!
knife openstack flavor list (options)!
knife openstack group list (options)!
knife openstack image list (options)!
knife openstack network list (options)!
knife openstack server create (options)!
knife openstack server delete INSTANCEID [INSTANCEID] (options)!
knife openstack server list (options)!
knife openstack server show (options)
$ knife openstack
Name ID Virtual CPUs RAM Disk !
m1.large 4 4 8192 MB 80 GB !
m1.medium 3 2 4096 MB 40 GB !
m1.small 2 1 2048 MB 20 GB !
m1.tiny 1 1 512 MB 10 GB !
m1.xlarge 5 8 16384 MB 160 GB
$ knife openstack flavor list
Name Protocol From To CIDR Description!
default tcp 22 22 0.0.0.0/0 default!
default icmp -1 -1 0.0.0.0/0 default!
haproxy tcp 22002 22002 0.0.0.0/0 22022!
$ knife openstack group list
Name ID Snapshot!
centos-6.5 68555833-8497-4d14-88ca-c9062e25f14b no !
coreos 83d37ea5-d9ae-44cd-9110-d4d39ad997ce no !
fedora-19 9add7e14-25e3-41d8-963a-ca744d081f2e no !
fedora-20 acb6eba5-226a-4ed5-8db6-33a6fd8cf20d no !
freebsd-10.0 0e270df7-1a02-4e91-9fc3-6f5311c58193 no !
ubuntu-12.04 ce268db5-ceda-4a90-93c8-3b987ac3705f no !
ubuntu-14.04 4a4f85bf-f164-4e54-83d8-8b2e7d0712b2 no !
Windows Server 2012 64e7cba7-7a50-443f-8fa6-a065406e0b04 no !
$ knife openstack image list
Name ID Tenant Shared!
external 06dc9d5a-f55a-410d-a7fd-4c7cb34ad927 5da25cc3853f4c54850898f9614c20bb true !
internal ba0fdd03-72b5-41eb-bb67-fef437fd6cb4 5da25cc3853f4c54850898f9614c20bb true
$ knife openstack network list
Name Instance ID Public IP Private IP Flavor Image Keypair State!
os-45539345723309377 08f2d9f7-eeb0-45e7-8562-63aed8f096cc 50.56.12.229 2 737969f8-6091-4896-ba9c-f3cf63bd25c5 rs-demo active!
os-9924426691020416 43c6bbf5-b397-4986-8aec-392d955ce5b1 50.56.12.232 2 737969f8-6091-4896-ba9c-f3cf63bd25c5 rs-demo active!
os-553425714287088 c1b9e3df-e566-4378-8a52-ed998b516608 50.56.12.230 2 737969f8-6091-4896-ba9c-f3cf63bd25c5 rs-demo active!
os-07459550287500682 f3edc5da-ef99-4acb-a141-d957e09809e3 50.56.12.231 2 737969f8-6091-4896-ba9c-f3cf63bd25c5 rs-demo active
$ knife openstack server list
$ knife openstack server create -a -f 2 -I 737969f8-6091-4896-ba9c-f3cf63bd25c5 -S rs-demo -i ~/.ssh/rs-demo.pem -x ubuntu -r "role[base]"!
Instance Name: os-45539345723309377!
Instance ID: 08f2d9f7-eeb0-45e7-8562-63aed8f096cc!
!
Waiting for server.........!
Flavor: 2!
Image: 737969f8-6091-4896-ba9c-f3cf63bd25c5!
SSH Identity File: /Users/mray/.ssh/rs-demo.pem!
SSH Keypair: rs-demo!
Public IP Address: 10.241.0.12!
Floating IP Address: 50.56.12.229!
Waiting for sshd.....done!
Bootstrapping Chef on 50.56.12.229!
Instance Name: os-45539345723309377!
Instance ID: 08f2d9f7-eeb0-45e7-8562-63aed8f096cc!
Flavor: 2!
Image: 737969f8-6091-4896-ba9c-f3cf63bd25c5!
SSH Keypair: rs-demo!
Public IP Address: 50.56.12.229!
Environment: _default!
Run List: role[base]!
knife openstack server create
knife openstack server create
knife openstack Compatibility
• Uses the OpenStack API
• Diablo, Essex, Folsom, Grizzly, Havana, Icehouse, trunk
• BlueBox
• Cloudscaling
• Crowbar
• DreamHost
• IBM
• MetaCloud
• Nebula
• Piston
• Rackspace Private Cloud
knife openstack Resources
• knife openstack --help
• docs.getchef.com/
plugin_knife_openstack.html
• github.com/opscode/knife-openstack
• github.com/opscode/knife-openstack/
issues
knife openstack Roadmap
• Continuous Integration for Chef-
supported knife plugins soon
•Testing against multiple
deployments
• 1.0.0: common knife-cloud base class
• 1.1.0: Network enhancements
• 1.2.0 guid cleanup, other niceties
Test Kitchen
Test Kitchen
• Integration tool for developing and testing
infrastructure code and software on isolated target
platforms
• Integration test platform for your cookbooks on all the
supported platforms with virtual machines
• https://github.com/test-kitchen/kitchen-openstack
driver:!
name: openstack!
openstack_username: [YOUR OPENSTACK USERNAME]!
openstack_api_key: [YOUR OPENSTACK API KEY]!
openstack_auth_url: [YOUR OPENSTACK AUTH URL]!
require_chef_omnibus: latest!
image_ref: [SERVER IMAGE ID]!
flavor_ref: [SERVER FLAVOR ID]
Test Kitchen: kitchen.yml
Chef Metal
• Chef recipes for deploying
infrastructure
• Libraries for repeatably creating
machines and deployments with
Chef primitives
• Bootstrappers for many
infrastructure types
Chef Metal: Providers
• Cloud
• Digital Ocean, EC2, Fog, OpenStack
• Virtualization
• Vagrant (VirtualBox, Fusion), VSphere
• Containers
• Docker & LXC
• SSH
• PXE in progress
Chef Metal: Example Recipe
Thanks!
Any questions?
Matt Ray
matt@getchef.com
@mattray

Chef & OpenStack: OSCON 2014

  • 2.
    Chef for OpenStack MattRay & JJ Asghar OSCON July 24, 2014
  • 3.
    Introductions • Matt Ray •Director of Partner Engineering • matt@getchef.com • mattray GitHub|IRC|Twitter
  • 4.
    Introductions • JJ Asghar •Sr. Partner Engineer • jj@getchef.com • jjasghar GitHub and Twitter • j^2 on IRC
  • 5.
    Chef •Open Source configurationmanagement and systems automation framework •Infrastructure as Code, written in Ruby •Abstractions of Resources on the systems •Client/server model over REST APIs •Huge, vibrant community of contributors ™
  • 6.
    OpenStack Controls Compute,Storage & Networking
  • 7.
  • 8.
  • 9.
    Overview & CurrentStatus of Chef for OpenStack
  • 10.
    Chef for OpenStack:Project • Community around the automated deployment and management of OpenStack • Reduce fragmentation and increase collaboration • Deploying OpenStack is not "Secret Sauce" • Project, not a 'Product' • Apache 2 License
  • 11.
    Community • #openstack-chef onirc.freenode.net • groups.google.com/group/opscode-chef-openstack • Google Hangout weekly status meetings (Monday 11am EST) • @chefopenstack • Stackalytics (stackforge->chef-group)
  • 12.
    Who's Involved? • AT&T •Bluebox • Chef • Dell • DreamHost • HP • IBM • Korea Telecom • Rackspace • SUSE • and many more
  • 14.
    Chef Requirements • Chef11 • Ruby 1.9.x • Foodcritic, Rubocop, ChefSpec for testing • attribute-driven by Environments • platform logic in attributes • currently packages-only installation
  • 15.
    StackForge: Cookbooks • "Official"OpenStack repositories •github.com/stackforge/cookbook-openstack-* •gated by review.openstack.org • OpenStack services for Grizzly, Havana and Icehouse cookbooks •block-storage, common, compute, dashboard, data-processing, database, identity, image, network, object-storage, orchestration, telemetry • Operational support cookbooks •ceph, client, integration-testing, ops-database, ops-messaging
  • 16.
  • 17.
    StackForge: Deployment • Chefrepository for deploying Grizzly, Havana, or Icehouse •example Environments and Roles •example "All-in-One" Vagrant deployment •github.com/stackforge/openstack-chef-repo • Gated by review.openstack.org •More single and multi-node testing coming
  • 18.
    Reference Implementation • Deploymentexamples in documentation •All-in-One Compute •Single Controller + N Compute •more coming • Will provide example HA configurations • Operations outside of scope of core repository •logging, monitoring, provisioning
  • 19.
  • 20.
    Documentation • docs.getchef.com/openstack.html •Architecture •Deployment Prerequisites •Installation •Development •Cookbooksand Repositories •Example Deployments • github.com/opscode/chef-docs •Creative Commons, no CLA required
  • 21.
    Example Deployments • Vagrant"All-in-One" for development/testing •nova-network or Neutron •Ubuntu 12.04 or CentOS 6.5 • Developer lab deployment "1+N" •Single controller, N compute boxes •5 boxes, consumer-grade hardware
  • 22.
    StackForge: Grizzly Status •branch ‘stable/grizzly’ • Operating Systems: Ubuntu 12.04, SLES 11 SP2 • Databases: MySQL, SQLite (testing) • Messaging: RabbitMQ • Compute: KVM, LXC, Qemu • Network: Nova + Quantum (Open vSwitch) • Block Storage: LVM • Object Storage: Swift • Dashboard: Apache or Nginx
  • 23.
    StackForge: Havana Status •branch ‘stable/havana’ • Operating Systems: Ubuntu 12.04, SLES 11 SP3, RHEL 6 • Databases: DB2, MySQL, SQLite (testing) • Messaging: Qpid, RabbitMQ • Compute: KVM, LXC, QEMU • Network: Nova, Neutron (Open vSwitch) • Block Storage: LVM, NetApp, others • Object Storage: Swift • Dashboard: Apache or Nginx
  • 24.
    StackForge: Icehouse Status •branch ‘master’ • Operating Systems: Ubuntu 12.04/14.04, SLES 11 SP3, RHEL 6 • Databases: DB2, MySQL Postgres, SQLite (testing) • Messaging: Qpid, RabbitMQ • Compute: KVM, LXC, QEMU • Network: Nova, Neutron (Open vSwitch) • Block Storage: LVM, NetApp, others • Object Storage: Swift • Dashboard: Apache or Nginx
  • 25.
    StackForge: Roadmap • branchingfor Juno (tomorrow?) •'master' move to ‘stable/icehouse’ •Havana available in ‘stable/havana' •Grizzly available in ‘stable/grizzly' • More contributors!
  • 26.
    StackForge: Roadmap • OperatingSystems: Debian • Compute: Baremetal, Docker, ESX, Hyper-V, Xen • Network: NSX, OpenDaylight • Block Storage: Ceph • Object Storage: Ceph • Source builds via Omnibus
  • 27.
    OpenStack Ops Mid-CycleMeetup • Operators mini summit • August 25-26, San Antonio TX • http://www.eventbrite.com/e/ openstack-ops-mid-cycle-meetup- tickets-12149171499
  • 28.
    Chef Community Summit •Developer/Community unconference • October 2 & 3 in Seattle, WA
  • 29.
  • 30.
  • 31.
    Available openstack subcommands:(for details, knife SUB-COMMAND --help)! ! ** OPENSTACK COMMANDS **! knife openstack flavor list (options)! knife openstack group list (options)! knife openstack image list (options)! knife openstack network list (options)! knife openstack server create (options)! knife openstack server delete INSTANCEID [INSTANCEID] (options)! knife openstack server list (options)! knife openstack server show (options) $ knife openstack
  • 32.
    Name ID VirtualCPUs RAM Disk ! m1.large 4 4 8192 MB 80 GB ! m1.medium 3 2 4096 MB 40 GB ! m1.small 2 1 2048 MB 20 GB ! m1.tiny 1 1 512 MB 10 GB ! m1.xlarge 5 8 16384 MB 160 GB $ knife openstack flavor list
  • 33.
    Name Protocol FromTo CIDR Description! default tcp 22 22 0.0.0.0/0 default! default icmp -1 -1 0.0.0.0/0 default! haproxy tcp 22002 22002 0.0.0.0/0 22022! $ knife openstack group list
  • 34.
    Name ID Snapshot! centos-6.568555833-8497-4d14-88ca-c9062e25f14b no ! coreos 83d37ea5-d9ae-44cd-9110-d4d39ad997ce no ! fedora-19 9add7e14-25e3-41d8-963a-ca744d081f2e no ! fedora-20 acb6eba5-226a-4ed5-8db6-33a6fd8cf20d no ! freebsd-10.0 0e270df7-1a02-4e91-9fc3-6f5311c58193 no ! ubuntu-12.04 ce268db5-ceda-4a90-93c8-3b987ac3705f no ! ubuntu-14.04 4a4f85bf-f164-4e54-83d8-8b2e7d0712b2 no ! Windows Server 2012 64e7cba7-7a50-443f-8fa6-a065406e0b04 no ! $ knife openstack image list
  • 35.
    Name ID TenantShared! external 06dc9d5a-f55a-410d-a7fd-4c7cb34ad927 5da25cc3853f4c54850898f9614c20bb true ! internal ba0fdd03-72b5-41eb-bb67-fef437fd6cb4 5da25cc3853f4c54850898f9614c20bb true $ knife openstack network list
  • 36.
    Name Instance IDPublic IP Private IP Flavor Image Keypair State! os-45539345723309377 08f2d9f7-eeb0-45e7-8562-63aed8f096cc 50.56.12.229 2 737969f8-6091-4896-ba9c-f3cf63bd25c5 rs-demo active! os-9924426691020416 43c6bbf5-b397-4986-8aec-392d955ce5b1 50.56.12.232 2 737969f8-6091-4896-ba9c-f3cf63bd25c5 rs-demo active! os-553425714287088 c1b9e3df-e566-4378-8a52-ed998b516608 50.56.12.230 2 737969f8-6091-4896-ba9c-f3cf63bd25c5 rs-demo active! os-07459550287500682 f3edc5da-ef99-4acb-a141-d957e09809e3 50.56.12.231 2 737969f8-6091-4896-ba9c-f3cf63bd25c5 rs-demo active $ knife openstack server list
  • 37.
    $ knife openstackserver create -a -f 2 -I 737969f8-6091-4896-ba9c-f3cf63bd25c5 -S rs-demo -i ~/.ssh/rs-demo.pem -x ubuntu -r "role[base]"! Instance Name: os-45539345723309377! Instance ID: 08f2d9f7-eeb0-45e7-8562-63aed8f096cc! ! Waiting for server.........! Flavor: 2! Image: 737969f8-6091-4896-ba9c-f3cf63bd25c5! SSH Identity File: /Users/mray/.ssh/rs-demo.pem! SSH Keypair: rs-demo! Public IP Address: 10.241.0.12! Floating IP Address: 50.56.12.229! Waiting for sshd.....done! Bootstrapping Chef on 50.56.12.229! Instance Name: os-45539345723309377! Instance ID: 08f2d9f7-eeb0-45e7-8562-63aed8f096cc! Flavor: 2! Image: 737969f8-6091-4896-ba9c-f3cf63bd25c5! SSH Keypair: rs-demo! Public IP Address: 50.56.12.229! Environment: _default! Run List: role[base]! knife openstack server create
  • 38.
  • 39.
    knife openstack Compatibility •Uses the OpenStack API • Diablo, Essex, Folsom, Grizzly, Havana, Icehouse, trunk • BlueBox • Cloudscaling • Crowbar • DreamHost • IBM • MetaCloud • Nebula • Piston • Rackspace Private Cloud
  • 40.
    knife openstack Resources •knife openstack --help • docs.getchef.com/ plugin_knife_openstack.html • github.com/opscode/knife-openstack • github.com/opscode/knife-openstack/ issues
  • 41.
    knife openstack Roadmap •Continuous Integration for Chef- supported knife plugins soon •Testing against multiple deployments • 1.0.0: common knife-cloud base class • 1.1.0: Network enhancements • 1.2.0 guid cleanup, other niceties
  • 42.
  • 43.
    Test Kitchen • Integrationtool for developing and testing infrastructure code and software on isolated target platforms • Integration test platform for your cookbooks on all the supported platforms with virtual machines • https://github.com/test-kitchen/kitchen-openstack
  • 44.
    driver:! name: openstack! openstack_username: [YOUROPENSTACK USERNAME]! openstack_api_key: [YOUR OPENSTACK API KEY]! openstack_auth_url: [YOUR OPENSTACK AUTH URL]! require_chef_omnibus: latest! image_ref: [SERVER IMAGE ID]! flavor_ref: [SERVER FLAVOR ID] Test Kitchen: kitchen.yml
  • 46.
    Chef Metal • Chefrecipes for deploying infrastructure • Libraries for repeatably creating machines and deployments with Chef primitives • Bootstrappers for many infrastructure types
  • 47.
    Chef Metal: Providers •Cloud • Digital Ocean, EC2, Fog, OpenStack • Virtualization • Vagrant (VirtualBox, Fusion), VSphere • Containers • Docker & LXC • SSH • PXE in progress
  • 48.
  • 49.