Chef 11 Preview/Chef for OpenStack

                           Boston 1/22/2013

                               Matt Ray
                          matt@opscode.com
                              @mattray


Tuesday, January 22, 13
What's New Since Chef 0.10?



                  •       Windows support              •   why-run

                  •       Librarian                    •   output formatters

                  •       Spiceweasel                  •   Test Kitchen

                  •       Foodcritic                   •   Solaris support

                  •       Cookbooks in separate        •   Chef Developer Summit(2!)
                          repositories
                                                       •   Berkshelf
                  •       Private Chef
                                                       •   ChefConf
                  •       Food Fight podcast
                                                       •   Fauxhai
                  •       knife plugins for
                                                       •   docs.opscode.com
                          openstack, hp, azure, gce

                  •       0.10 renumbered to Chef 10   •   Hangouts for reviews

                  •       full-stack client            •   nyan-cat output formatter

Tuesday, January 22, 13

Vaguely chronological, so much good stuff in the Chef Community
What is NOT in Chef 11




                  •       CouchDB

                  •       Ruby-based chef-server-api

                  •       depsolver, gecode, treetop

                  •       Merb

                  •       OpenID support in Web UI

                  •       Migration tooling (coming soon)




                                     http://trainweb.org/carl/Pullman150/IMG_3380.jpg
Tuesday, January 22, 13
What’s NEW in Chef 11




                  •       PostgreSQL

                  •       erchef

                  •       nginx

                  •       Rails

                  •       bookshelf

                  •       omnibus-chef server

                  •       chef-apply

                  •       partial search

                  •       users with key pairs

                  •       partials in templates

                  •       knife-essentials

Tuesday, January 22, 13
Chef Server Architecture


                                                          nginx




                                                 erchef           bookshelf    chef-webui




                                 RabbitMQ




                                 chef-expander




                                               solr
                                                                  filesystem
                                            (lucene)              cookbook    PostgreSQL
                                                                     store


                                            filesystem
                                              search
                                               index

Tuesday, January 22, 13

Postgres has replaced Couch (who knew SQL was so good?)
Erlang-based erchef has replaced Ruby-based chef-server-api
chef-webui has been upgraded to Rails 3 from Merb
Bookshelf is our S3-compatibilish storage service
erchef Architecture

                                                                           erchef




                                                                          chef_wm




                                  chef_objects        chef_authn      chef_index               chef_db




                                                 bookshelf         solr             RabbitMQ
                                                                                                     PostgreSQL



Tuesday, January 22, 13

Erlang applications, each on GitHub
CPU Usage on Chef Server




Tuesday, January 22, 13

These are old graphs from last year. Code is actually much faster and more efficient now that we've had a year of tuning.
This is with erchef and Ruby code on the same box
3 CPU vm graph is aggregate
CouchDB Uptime




Tuesday, January 22, 13

the smaller spikes were when we chron'd restarts
Database CPU


                          CouchDB                                                        MySQL




Tuesday, January 22, 13

Postgres is about the same performance, but we've found we're able to focus optimizations on a single rdbms and put time that
we would otherwise have to spend on ensuring both backend dbs work properly into other areas of the code for fixes, enhances,
features.
Database Memory

                          CouchDB    MySQL




Tuesday, January 22, 13

Nice and flat
Database Load Average


                          CouchDB                     MySQL




Tuesday, January 22, 13

Licensing is much better for us with Postgres
API Average Latency




Tuesday, January 22, 13

And things have gotten better from here with Private Chef and Open Source Chef. Not everything has been ported to Opscode
Hosted Chef yet. Work has started to complete the migration of OHC to the OPC code base, things with OHC will get far better.
omnibus-chef server




                  •       full-stack for the Chef server, everything you need

                  •       installs to /opt/chef-server

                  •       RPMs, DEBs for now

                  •       build your own packages

                  •       chef-server-ctl

                          •   status

                          •   start/stop

                          •   tail




                                                                    http://apod.nasa.gov/apod/astropix.html
Tuesday, January 22, 13

Much of the same tooling OPC
chef-apply




                  •       run a single recipe file

                          •   without modifying the node's run_list

                  •       included with Chef gem, /usr/bin/chef-apply

                  •       chef-apply /path/to/recipe_file

                  •       chef-apply "content of a recipe file"




                                               http://www.flickr.com/photos/albill/sets/72157628046395000/
Tuesday, January 22, 13

http://tickets.opscode.com/browse/CHEF-3571
partial search


      partial_search(:node,	
  'role:web',
      	
  	
  	
  :keys	
  =>	
  {	
  'name'	
  =>	
  [	
  'name'	
  ],
      	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  'ip'	
  	
  	
  =>	
  [	
  'ipaddress'	
  ],
      	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  'kernel_version'	
  =>	
  [	
  'kernel',	
  'version'	
  ]
      	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  }
      ).each	
  do	
  |result|
      	
  	
  puts	
  result['name']
      	
  	
  puts	
  result['ip']
      	
  	
  puts	
  result['kernel_version']
      end


         •      instead of entire node, just the pieces you want

         •      massive reduction in bandwidth and memory

         •      http://community.opscode.com/cookbooks/partial_search

                •         to use with Chef 10 and Hosted Chef today

                                                                      http://www.flickr.com/photos/albill/sets/72157628046395000/
Tuesday, January 22, 13
users with keypairs




                  •       users can have key pairs, just like clients

                  •       knife actions as a user instead of a client

                  •       post a public key when you create a user/client




                                                          http://www.flickr.com/photos/albill/sets/72157628046395000/
Tuesday, January 22, 13

For users in Chef 11, the key point is:
Before: users were just a concept of the webui. If you wanted a knife setup, you needed a client.
After: users can have key pairs just like clients. So you can do knife actions as your user.
partials in templates


                          <?xml version='1.0' encoding='UTF-8'?>

                          <server xmlns="urn:jboss:domain:1.2">
                              <extensions>
                                    <% if @infinispan -%>
                                  <extension module="org.jboss.as.clustering.infinispan"/>
                                  <% end -%>
                                  <extension module="org.jboss.as.web"/>
                                    <% if @webservices -%>
                                  <extension module="org.jboss.as.webservices"/>
                                  <% end -%>
                                  <extension module="org.jboss.as.weld"/>
                              </extensions>
                               <% if @infinispan
                                          include_template "infinispan.xml.erb"
                               end -%>
                               <% if @webservices
                                          include_template "webservices.xml.erb"
                               end -%>
                             <socket-binding-group name="standard-sockets" >
                              ......
                             </socket-binding-group>
                          </server>


                           •    Thanks to Andrea Campi!
                                                          http://www.flickr.com/photos/modern_fred/2095565021/
Tuesday, January 22, 13
knife-essentials




           •       Unified commands that work on everything

           •       https://github.com/jkeiser/knife-essentials

           •       knife download [pattern1 pattern2 ...]

                  •       knife download roles data_bags cookbooks/emacs

           •       knife diff cookbooks/*apache*

           •       knife show *base*

           •       knife upload apache*

           •       knife list data_bags/users

           •       knife deps roles/base.json



                                      http://photography.nationalgeographic.com/photography/photo-of-the-day/
Tuesday, January 22, 13

knife    download
knife    diff
knife    show
knife    list
knife    upload
Chef 11 Breaking Changes




      •      http://wiki.opscode.com/display/chef/Breaking
             +Changes+in+Chef+11

      •      chef-shell (formerly shef)

      •      no implicits node attributes (no more node['blah'] = 'foo')

      •      attribute files may access role and environment attributes

      •      delayed notifications run after failed converge

      •      encrypted data bag item format change

      •      chef-client lock so safe from simultaneous runs




                                           http://www.flickr.com/photos/modern_fred/2096352938/
Tuesday, January 22, 13

More changes on the wiki page
Chef 11 Server Preview




  •      http://wiki.opscode.com/display/chef/Chef+11+Server
         +Preview

  •      Nightly Builds

  •      Centos 5 & 6

  •      Ubuntu 10.04, 11.04 and 12.04

  •      Everything is on GitHub

  •      More testing means better releases!

  •      Client is in beta: http://lists.opscode.com/sympa/arc/
         chef-dev/2013-01/msg00028.html



                                               http://www.brandonbird.com/lazysunday.html
Tuesday, January 22, 13

Coming soon!
Overview & Status of
                          Chef for OpenStack


Tuesday, January 22, 13
Chef for OpenStack: Who




Tuesday, January 22, 13

These companies are currently involved to some extent
Chef for OpenStack: Why




                  •       Community for the automated deployment
                          and management of OpenStack
                  •       Reduce fragmentation and encourage
                          collaboration
                  •       Deploying OpenStack is not "secret sauce"
                  •       Project not a product
                  •       Apache 2 license


Tuesday, January 22, 13
Chef for OpenStack: What



                  •       Chef Repository for Deploying OpenStack
                  •       Documentation for Chef for OpenStack
                  •       Cookbooks
                          •   Keystone

                          •   Glance

                          •   Nova

                          •   Horizon

                          •   Swift

                          •   Quantum

                          •   Cinder

                  •       knife-openstack
Tuesday, January 22, 13
Chef for OpenStack: Where



                  •       groups.google.com/group/opscode-chef-
                          openstack
                  •       #openstack-chef on irc.freenode.net
                  •       github.com/opscode/openstack-chef-repo
                  •       github.com/mattray/openstack-chef-docs
                  •       github.com/opscode-cookbooks/
                          •   keystone, glance, nova, horizon,
                              swift,quantum,cinder

                  •       github.com/opscode/knife-openstack
                  •       @chefopenstack
Tuesday, January 22, 13
Chef for OpenStack: When (Today)




                  •       Chef repo for Essex (2012.1.1 tag)
                  •       Operating Systems (Ubuntu 12.04)
                  •       Hypervisors (KVM, LXC)
                  •       Databases (MySQL)
                  •       FlatDHCP & VLAN networking
                          •   manual floating IPs documented
                  •       Test Kitchen integration


Tuesday, January 22, 13
Chef for OpenStack: When (Tomorrow)




                  •       Documentation (docs.opscode.com)
                  •       Folsom is under active development
                  •       AT&T, DreamHost and Rackspace have
                          active branches
                  •       Move to openstack-common
                  •       Cinder (lvm, Netapp)
                  •       Quantum (Nicira with Open vSwitch)


Tuesday, January 22, 13
Chef for OpenStack When: (Roadmap)




                  •       Documentation (docs.opscode.com)
                  •       Continuous Integration testing
                  •       Grizzly & trunk(!?)
                  •       Cinder (Ceph)
                  •       Quantum (Midokura)
                  •       Hypervisors (Hyper-V, bare metal)
                  •       Databases (PostgreSQL)
                  •       Operating Systems (RHEL, Debian, SUSE)
                  •
Tuesday, January 22, 13
                          HA Configurations
Chef for OpenStack: How




                          github.com/mattray/openstack-chef-docs
                                         moving to
                               github.com/opscode/chef-docs




Tuesday, January 22, 13

Working on a permanent URL
HTML, PDF, Epub for your iPad
Questions?

                           Boston 1/22/2013

                               Matt Ray
                          matt@opscode.com
                              @mattray


Tuesday, January 22, 13

ChefConf 2013 April 2013 "OPSCODE-MEETUP"
Metarepo and CI work

Chef 11 Preview/Chef for OpenStack

  • 1.
    Chef 11 Preview/Cheffor OpenStack Boston 1/22/2013 Matt Ray matt@opscode.com @mattray Tuesday, January 22, 13
  • 2.
    What's New SinceChef 0.10? • Windows support • why-run • Librarian • output formatters • Spiceweasel • Test Kitchen • Foodcritic • Solaris support • Cookbooks in separate • Chef Developer Summit(2!) repositories • Berkshelf • Private Chef • ChefConf • Food Fight podcast • Fauxhai • knife plugins for • docs.opscode.com openstack, hp, azure, gce • 0.10 renumbered to Chef 10 • Hangouts for reviews • full-stack client • nyan-cat output formatter Tuesday, January 22, 13 Vaguely chronological, so much good stuff in the Chef Community
  • 3.
    What is NOTin Chef 11 • CouchDB • Ruby-based chef-server-api • depsolver, gecode, treetop • Merb • OpenID support in Web UI • Migration tooling (coming soon) http://trainweb.org/carl/Pullman150/IMG_3380.jpg Tuesday, January 22, 13
  • 4.
    What’s NEW inChef 11 • PostgreSQL • erchef • nginx • Rails • bookshelf • omnibus-chef server • chef-apply • partial search • users with key pairs • partials in templates • knife-essentials Tuesday, January 22, 13
  • 5.
    Chef Server Architecture nginx erchef bookshelf chef-webui RabbitMQ chef-expander solr filesystem (lucene) cookbook PostgreSQL store filesystem search index Tuesday, January 22, 13 Postgres has replaced Couch (who knew SQL was so good?) Erlang-based erchef has replaced Ruby-based chef-server-api chef-webui has been upgraded to Rails 3 from Merb Bookshelf is our S3-compatibilish storage service
  • 6.
    erchef Architecture erchef chef_wm chef_objects chef_authn chef_index chef_db bookshelf solr RabbitMQ PostgreSQL Tuesday, January 22, 13 Erlang applications, each on GitHub
  • 7.
    CPU Usage onChef Server Tuesday, January 22, 13 These are old graphs from last year. Code is actually much faster and more efficient now that we've had a year of tuning. This is with erchef and Ruby code on the same box 3 CPU vm graph is aggregate
  • 8.
    CouchDB Uptime Tuesday, January22, 13 the smaller spikes were when we chron'd restarts
  • 9.
    Database CPU CouchDB MySQL Tuesday, January 22, 13 Postgres is about the same performance, but we've found we're able to focus optimizations on a single rdbms and put time that we would otherwise have to spend on ensuring both backend dbs work properly into other areas of the code for fixes, enhances, features.
  • 10.
    Database Memory CouchDB MySQL Tuesday, January 22, 13 Nice and flat
  • 11.
    Database Load Average CouchDB MySQL Tuesday, January 22, 13 Licensing is much better for us with Postgres
  • 12.
    API Average Latency Tuesday,January 22, 13 And things have gotten better from here with Private Chef and Open Source Chef. Not everything has been ported to Opscode Hosted Chef yet. Work has started to complete the migration of OHC to the OPC code base, things with OHC will get far better.
  • 13.
    omnibus-chef server • full-stack for the Chef server, everything you need • installs to /opt/chef-server • RPMs, DEBs for now • build your own packages • chef-server-ctl • status • start/stop • tail http://apod.nasa.gov/apod/astropix.html Tuesday, January 22, 13 Much of the same tooling OPC
  • 14.
    chef-apply • run a single recipe file • without modifying the node's run_list • included with Chef gem, /usr/bin/chef-apply • chef-apply /path/to/recipe_file • chef-apply "content of a recipe file" http://www.flickr.com/photos/albill/sets/72157628046395000/ Tuesday, January 22, 13 http://tickets.opscode.com/browse/CHEF-3571
  • 15.
    partial search partial_search(:node,  'role:web',      :keys  =>  {  'name'  =>  [  'name'  ],                            'ip'      =>  [  'ipaddress'  ],                            'kernel_version'  =>  [  'kernel',  'version'  ]                        } ).each  do  |result|    puts  result['name']    puts  result['ip']    puts  result['kernel_version'] end • instead of entire node, just the pieces you want • massive reduction in bandwidth and memory • http://community.opscode.com/cookbooks/partial_search • to use with Chef 10 and Hosted Chef today http://www.flickr.com/photos/albill/sets/72157628046395000/ Tuesday, January 22, 13
  • 16.
    users with keypairs • users can have key pairs, just like clients • knife actions as a user instead of a client • post a public key when you create a user/client http://www.flickr.com/photos/albill/sets/72157628046395000/ Tuesday, January 22, 13 For users in Chef 11, the key point is: Before: users were just a concept of the webui. If you wanted a knife setup, you needed a client. After: users can have key pairs just like clients. So you can do knife actions as your user.
  • 17.
    partials in templates <?xml version='1.0' encoding='UTF-8'?> <server xmlns="urn:jboss:domain:1.2"> <extensions> <% if @infinispan -%> <extension module="org.jboss.as.clustering.infinispan"/> <% end -%> <extension module="org.jboss.as.web"/> <% if @webservices -%> <extension module="org.jboss.as.webservices"/> <% end -%> <extension module="org.jboss.as.weld"/> </extensions> <% if @infinispan include_template "infinispan.xml.erb" end -%> <% if @webservices include_template "webservices.xml.erb" end -%> <socket-binding-group name="standard-sockets" > ...... </socket-binding-group> </server> • Thanks to Andrea Campi! http://www.flickr.com/photos/modern_fred/2095565021/ Tuesday, January 22, 13
  • 18.
    knife-essentials • Unified commands that work on everything • https://github.com/jkeiser/knife-essentials • knife download [pattern1 pattern2 ...] • knife download roles data_bags cookbooks/emacs • knife diff cookbooks/*apache* • knife show *base* • knife upload apache* • knife list data_bags/users • knife deps roles/base.json http://photography.nationalgeographic.com/photography/photo-of-the-day/ Tuesday, January 22, 13 knife download knife diff knife show knife list knife upload
  • 19.
    Chef 11 BreakingChanges • http://wiki.opscode.com/display/chef/Breaking +Changes+in+Chef+11 • chef-shell (formerly shef) • no implicits node attributes (no more node['blah'] = 'foo') • attribute files may access role and environment attributes • delayed notifications run after failed converge • encrypted data bag item format change • chef-client lock so safe from simultaneous runs http://www.flickr.com/photos/modern_fred/2096352938/ Tuesday, January 22, 13 More changes on the wiki page
  • 20.
    Chef 11 ServerPreview • http://wiki.opscode.com/display/chef/Chef+11+Server +Preview • Nightly Builds • Centos 5 & 6 • Ubuntu 10.04, 11.04 and 12.04 • Everything is on GitHub • More testing means better releases! • Client is in beta: http://lists.opscode.com/sympa/arc/ chef-dev/2013-01/msg00028.html http://www.brandonbird.com/lazysunday.html Tuesday, January 22, 13 Coming soon!
  • 21.
    Overview & Statusof Chef for OpenStack Tuesday, January 22, 13
  • 22.
    Chef for OpenStack:Who Tuesday, January 22, 13 These companies are currently involved to some extent
  • 23.
    Chef for OpenStack:Why • Community for the automated deployment and management of OpenStack • Reduce fragmentation and encourage collaboration • Deploying OpenStack is not "secret sauce" • Project not a product • Apache 2 license Tuesday, January 22, 13
  • 24.
    Chef for OpenStack:What • Chef Repository for Deploying OpenStack • Documentation for Chef for OpenStack • Cookbooks • Keystone • Glance • Nova • Horizon • Swift • Quantum • Cinder • knife-openstack Tuesday, January 22, 13
  • 25.
    Chef for OpenStack:Where • groups.google.com/group/opscode-chef- openstack • #openstack-chef on irc.freenode.net • github.com/opscode/openstack-chef-repo • github.com/mattray/openstack-chef-docs • github.com/opscode-cookbooks/ • keystone, glance, nova, horizon, swift,quantum,cinder • github.com/opscode/knife-openstack • @chefopenstack Tuesday, January 22, 13
  • 26.
    Chef for OpenStack:When (Today) • Chef repo for Essex (2012.1.1 tag) • Operating Systems (Ubuntu 12.04) • Hypervisors (KVM, LXC) • Databases (MySQL) • FlatDHCP & VLAN networking • manual floating IPs documented • Test Kitchen integration Tuesday, January 22, 13
  • 27.
    Chef for OpenStack:When (Tomorrow) • Documentation (docs.opscode.com) • Folsom is under active development • AT&T, DreamHost and Rackspace have active branches • Move to openstack-common • Cinder (lvm, Netapp) • Quantum (Nicira with Open vSwitch) Tuesday, January 22, 13
  • 28.
    Chef for OpenStackWhen: (Roadmap) • Documentation (docs.opscode.com) • Continuous Integration testing • Grizzly & trunk(!?) • Cinder (Ceph) • Quantum (Midokura) • Hypervisors (Hyper-V, bare metal) • Databases (PostgreSQL) • Operating Systems (RHEL, Debian, SUSE) • Tuesday, January 22, 13 HA Configurations
  • 29.
    Chef for OpenStack:How github.com/mattray/openstack-chef-docs moving to github.com/opscode/chef-docs Tuesday, January 22, 13 Working on a permanent URL HTML, PDF, Epub for your iPad
  • 30.
    Questions? Boston 1/22/2013 Matt Ray matt@opscode.com @mattray Tuesday, January 22, 13 ChefConf 2013 April 2013 "OPSCODE-MEETUP" Metarepo and CI work