Rick	  Copeland	  @rick446	  Arborian	  Consulting,	  LLC	  
  Infrastructure	  as	  Code	    Resources	  &	  Providers	    Cookbooks,	  Recipes,	  Clients,	  and	  Nodes	  
CouchDB	                                                     Solr	                                                        ...
Build,	  register,	  and	  authenticate	  the	  node	                     Synchronize	  cookbooks	  Build	  resource	  col...
CouchDB	                                                            Solr	                                                 ...
  Chef	  assumes	  a	  bootstrapped	  node	  exists	    Chef	  doesn’t	  keep	  release	  notes	    Code	  and	  infras...
MonQ	           MongoDB	                                                                        Solr	  Indexer	           ...
  Reduce	  #	  of	  processes	  &	  technologies	    Don’t	  know	  Ruby	  well	    Keep	  private	  keys	  out	  of	  ...
/clients	         /nodes	     /data	       /environments	     /roles	       /sandboxes	  /cookbooks	         /search	  
  Mostly	  JSON	  	  almost	  BSON	     References	  to	  Ruby	  files	  stored	  separately	  {	  	  	  "name":	  "allu...
MongoDB	  Validator	  role = collection(     chef.role, doc_session,     Field(_id, S.ObjectId()),     Field(account_id, S...
Models	  know	                                  where	  they	  live	  class Role(object):      def url(self):         retu...
class RoleSchema(JSONModelSchema):!     model_class=CM.role!     chef_type=role’!     json_class=Chef::Role’!     exclude_...
  /foo/bar/baz	  	  root.foo.bar.baz	    Lots	  of	  decorators	       Validation	  (params/body	  	  **kwargs)	     ...
Returns	  JSON	  class RolesController(RESTController):      @expose(template_engine=json)      def _get(self):…          ...
class RoleController(RESTController):     @expose(template_engine=json)     def _get(self):         …                     ...
class RoleController(RESTController):!                                                AttributeError	  	  	        def __...
class RoleController(RESTController):!      …     @expose(template_engine=json,             acl=[CACE.admin(True), ACE.any...
    Don’t	  trust	  the	  docs	         Don’t	  trust	  the	  docs	          ▪  Don’t	  trust	  the	  docs	      Use	  ...
  Better	  test	  coverage	    Search	  support	  (SOLR	  /	  ElasticSearch)	    More	  testing	  with	  real-­‐world	 ...
Rick	  Copeland	  @rick446	  Arborian	  Consulting,	  LLC	  
  http://openmymind.net/2011/10/28/ CouchDB-­‐And-­‐MongoDB-­‐Performance/	      MongoDB	  is	  14x	  faster	    http:/...
Upcoming SlideShare
Loading in …5

Chef on Python and MongoDB


Published on

DevOps is the new rage among system administrators, applying agile software development techniques to infrastructure configuration management. In the center of the DevOps movement is the open-source Chef tool, implemented in Ruby atop CouchDB. Unsatisfied with the performance of the open-source and/or hosted Chef server and needing better integration with our Python web application, we set out to build a new implementation in Python atop MongoDB. This talk will give you an overview of Chef, reasons for doing a new implementation, and lots of code examples of how we made it all work together to get a chef server that screams.

Published in: Technology, Self Improvement
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Keep your infrastructure definitions in source controlResources = users, packages, services,files, etc. Providers = How you build a resource on a particular platformCookbook = Collection of resources, providers, templates, libraries, and filesRecipe = Ruby script defining the configuration of some resourcesClient = anyone talking to the chef serverNode = a machine instance managed by chef
  • “url” field Opscode stores everything on S3, I store it all in GridFS
  • Note the ‘loads’  there IS valid JSON that is not a valid BSON doc for a collection. Dotted key names, for example.
  • Chef on Python and MongoDB

    1. 1. Rick  Copeland  @rick446  Arborian  Consulting,  LLC  
    2. 2.   Infrastructure  as  Code    Resources  &  Providers    Cookbooks,  Recipes,  Clients,  and  Nodes  
    3. 3. CouchDB   Solr   Solr  Indexer  Chef  Server     RabbitMQ   RabbitMQ   RabbitMQ   (API)   HTTP  REST  API  Chef  Server   knife   chef-­‐client   (Web  UI)  
    4. 4. Build,  register,  and  authenticate  the  node   Synchronize  cookbooks  Build  resource  collection  (run  the  recipes  in  order)   Configure  node  (“converge”)   Run  notification  handlers  
    5. 5. CouchDB   Solr   Ruby   Solr  Indexer   Chef  Server     RabbitMQ   RabbitMQ   RabbitMQ   (API)   Ruby  Ruby   HTTP  REST  API   Chef  Server   knife   chef-­‐client   (Web  UI)  
    6. 6.   Chef  assumes  a  bootstrapped  node  exists    Chef  doesn’t  keep  release  notes    Code  and  infrastructure  are  versioned   differently    Solution:  Web  app  to  manage  deployments  &   generate  release  notes  
    7. 7. MonQ   MongoDB   Solr  Indexer   Python   Chef  Server     Solr   (API  +  web)   Ruby  HTTP  REST  API   knife   chef-­‐client  
    8. 8.   Reduce  #  of  processes  &  technologies    Don’t  know  Ruby  well    Keep  private  keys  out  of  the  system    Integrate  with  existing  authentication    Performance  
    9. 9. /clients   /nodes   /data   /environments   /roles   /sandboxes  /cookbooks   /search  
    10. 10.   Mostly  JSON    almost  BSON     References  to  Ruby  files  stored  separately  {      "name":  "allura-­‐0.0.1”,…      "json_class":  "Chef::CookbookVersion",   Ruby  files  stored  on  S3      "attributes":  [          {              "name":  "default.rb",              "url":  "https://s3.amazonaws.com/opscode-­‐platform-­‐production-­‐data/…                                  "checksum":  "749a3a328d6c47a32d134b336183981f",              "path":  "attributes/default.rb",              "specificity":  "default”…  
    11. 11. MongoDB  Validator  role = collection( chef.role, doc_session, Field(_id, S.ObjectId()), Field(account_id, S.ObjectId(if_missing=None)), Shorthand  with   Field(name, str), Python  Types   Field(description, str), Field(default_attributes, str), Field(override_attributes, str), Field(run_list, [ str ] ), Field(env_run_lists, { str: [ str ]}), Index(account_id, name, unique=True)) Embedded   Index  Definitions   Documents  
    12. 12. Models  know   where  they  live  class Role(object): def url(self): return request.relative_url( config.chef_api_root + /roles/ + self.name) Models  can  be   def __json__(self): return dict( turned  into  dict  (to   chef_type=role, be  JSONified)   json_class=Chef::Role,! … default_attributes=loads(self.default_attributes),! …) Models  can  be   def update(self, d): self.name = d[name] updated  from  dict   … self.default_attributes = dumps(d[default_attributes]) self.override_attributes = dumps(d[override_attributes])! …
    13. 13. class RoleSchema(JSONModelSchema):! model_class=CM.role! chef_type=role’! json_class=Chef::Role’! exclude_fields=[_id, account_id]
    14. 14.   /foo/bar/baz    root.foo.bar.baz    Lots  of  decorators     Validation  (params/body    **kwargs)     Authorization     Rendering    HTTP  method  lookup     GET  /foo/bar    root.foo.bar._get()  
    15. 15. Returns  JSON  class RolesController(RESTController): @expose(template_engine=json) def _get(self):… Only  admins  can   access   @expose(template_engine=json, acl=[CACE.admin(True), ACE.any(False)], schema=cv.RoleSchema) Convert  and   def _post(self, **kwargs):… Validate  POST   def __getattr__(self, name): return RoleController(name) Continue  dotted   lookup  
    16. 16. class RoleController(RESTController): @expose(template_engine=json) def _get(self): … PUT  looks  just  like  a  POST   @expose(template_engine=json, acl=[CACE.admin(True), ACE.any(False)], schema=cv.RoleSchema) def _put(self, name, **kwargs): … @expose(template_engine=json, acl=[CACE.admin(True), ACE.any(False)]) def _delete(self): …  
    17. 17. class RoleController(RESTController):! AttributeError       def __init__(self, name): HTTP  404  Not  Found   self._role = CM.Role.query.get( account_id=c.account._id, name=name) if self._role is None: raise AttributeError, name! @expose(template_engine=json) def _get(self): return self._role! Auto-­‐JSONify  
    18. 18. class RoleController(RESTController):! … @expose(template_engine=json, acl=[CACE.admin(True), ACE.any(False)], Update  model   schema=cv.RoleSchema) from  kwargs   def _put(self, name, **kwargs): assert name == self._role.name self._role.update(kwargs) return self._role! @expose(template_engine=json, acl=[CACE.admin(True), ACE.any(False)]) def _delete(self): self._role.delete() return self._role
    19. 19.   Don’t  trust  the  docs     Don’t  trust  the  docs   ▪  Don’t  trust  the  docs    Use  fat  models    Framework  support  for  REST  &  JSON    You’re  gonna  have  to  learn  some  Ruby  anyway    JSON  !=  BSON  
    20. 20.   Better  test  coverage    Search  support  (SOLR  /  ElasticSearch)    More  testing  with  real-­‐world  deployments    Finalize  integration  with  deployment   manager  
    21. 21. Rick  Copeland  @rick446  Arborian  Consulting,  LLC  
    22. 22.   http://openmymind.net/2011/10/28/ CouchDB-­‐And-­‐MongoDB-­‐Performance/     MongoDB  is  14x  faster    http://www.snailinaturtleneck.com/blog/ 2009/06/29/couchdb-­‐vs-­‐mongodb-­‐ benchmark/