Rick Copeland @rick446Arborian Consulting, LLC
   Infrastructure as Code   Resources & Providers   Cookbooks, Recipes, Clients, and Nodes
CouchDB                                Solr                                     SolrIndexerChef Server                    ...
Build, register, and authenticate the node             Synchronize cookbooksBuild resource collection (run the recipes in ...
CouchDB                                  Solr                                       Ruby                                  ...
   Chef assumes a bootstrapped node exists   Chef doesn’t keep release notes   Code and infrastructure are versioned   ...
MonQ     MongoDB                                           SolrIndexer                                  Python    Chef Ser...
   Reduce # of processes & technologies   Don’t know Ruby well   Keep private keys out of the system   Integrate with ...
/clients       /nodes  /data      /environments  /roles      /sandboxes/cookbooks      /search
   Mostly JSON  almost BSON       References to Ruby files stored separately{    "name": "allura-0.0.1”,…    "json_clas...
MongoDBValidatorrole = collection(chef.role, doc_session,Field(_id, S.ObjectId()),Field(account_id, S.ObjectId(if_missing=...
classRole(ModelBase):                                              Models know                                            ...
classRoleSchema(JSONModelSchema):model_class=CM.rolechef_type=role’json_class=Chef::Role’exclude_fields=[_id, account_id]
   Find resource     /foo/bar/bazRoot()[‘foo’][‘bar’][‘baz’]   (Resource, method)  view     @view_config(context=Res...
classResourceCollection(object):__name__ = None                                   Check auth__model__ = Nonekey_property=n...
@view_config(context=Roles, renderer=json,          Resources can berequest_method=GET’, permission=read)                 ...
   Don’t trust the docs     Don’t trust the docs      ▪ Don’t trust the docs   Use fat models   Framework support for ...
   Port from homegrown framework to Pyramid   Better test coverage   Search support (SOLR / ElasticSearch)   More test...
Rick Copeland @rick446Arborian Consulting, LLC
Rick Copeland @rick446Arborian Consulting, LLC
Upcoming SlideShare
Loading in …5
×

Chef on MongoDB and Pyramid

2,652 views

Published on

DevOps is the new rage among system administrators, applying agile software development techniques to infrastructure configuration management. In the center of the DevOps movement is the open-source Chef tool, implemented in Ruby atop CouchDB. Unsatisfied with the performance of the open-source and/or hosted Chef server and needing better integration with our Python web application, we set out to build a new implementation in Python atop MongoDB. This talk will give you an overview of Chef, reasons for doing a new implementation, and lots of code examples of how we made it all work together to get a chef server that screams.

This talk is updated with the latest version of MongoPyChef, ported to run on Pyramid and open sourced at https://github.com/rick446/MongoPyChef

Published in: Technology, Self Improvement
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,652
On SlideShare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
23
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Keep your infrastructure definitions in source controlResources = users, packages, services,files, etc. Providers = How you build a resource on a particular platformCookbook = Collection of resources, providers, templates, libraries, and filesRecipe = Ruby script defining the configuration of some resourcesClient = anyone talking to the chef serverNode = a machine instance managed by chef
  • “url” field Opscode stores everything on S3, I store it all in GridFS
  • Note the ‘loads’  there IS valid JSON that is not a valid BSON doc for a collection. Dotted key names, for example.
  • Chef on MongoDB and Pyramid

    1. 1. Rick Copeland @rick446Arborian Consulting, LLC
    2. 2.  Infrastructure as Code Resources & Providers Cookbooks, Recipes, Clients, and Nodes
    3. 3. CouchDB Solr SolrIndexerChef Server RabbitMQ RabbitMQ RabbitMQ (API) HTTP REST APIChef Server knife chef-client (Web UI)
    4. 4. Build, register, and authenticate the node Synchronize cookbooksBuild resource collection (run the recipes in order) Configure node (“converge”) Run notification handlers
    5. 5. CouchDB Solr Ruby SolrIndexer Chef Server RabbitMQ RabbitMQ RabbitMQ (API) RubyRuby HTTP REST API Chef Server knife chef-client (Web UI)
    6. 6.  Chef assumes a bootstrapped node exists Chef doesn’t keep release notes Code and infrastructure are versioned differently Solution: Web app to manage deployments & generate release notes
    7. 7. MonQ MongoDB SolrIndexer Python Chef Server (API + web) Solr RubyHTTP REST API knife chef-client
    8. 8.  Reduce # of processes & technologies Don’t know Ruby well Keep private keys out of the system Integrate with existing authentication Performance
    9. 9. /clients /nodes /data /environments /roles /sandboxes/cookbooks /search
    10. 10.  Mostly JSON  almost BSON  References to Ruby files stored separately{ "name": "allura-0.0.1”,… "json_class": "Chef::CookbookVersion", Ruby files stored on S3 "attributes": [ { "name": "default.rb", "url": "https://s3.amazonaws.com/opscode-platform-production-data/… "checksum": "749a3a328d6c47a32d134b336183981f", "path": "attributes/default.rb", "specificity": "default”…
    11. 11. MongoDBValidatorrole = collection(chef.role, doc_session,Field(_id, S.ObjectId()),Field(account_id, S.ObjectId(if_missing=None)), Shorthand withField(name, str), Python TypesField(description, str),Field(default_attributes, str),Field(override_attributes, str),Field(run_list, [ str ] ),Field(env_run_lists, { str: [ str ]}),Index(account_id, name, unique=True)) Embedded Index Definitions Documents
    12. 12. classRole(ModelBase): Models know where they livedef__name__(self):returnself.name Models can bedef__json__(self): turned into dict (toreturndict( be JSONified)chef_type=role,json_class=Chef::Role, …default_attributes=loads(self.default_attributes), Models can be …) updated from dictdefupdate(self, d):self.name = d[name] …self.default_attributes = dumps(d[default_attributes])self.override_attributes = dumps(d[override_attributes]) …
    13. 13. classRoleSchema(JSONModelSchema):model_class=CM.rolechef_type=role’json_class=Chef::Role’exclude_fields=[_id, account_id]
    14. 14.  Find resource  /foo/bar/bazRoot()[‘foo’][‘bar’][‘baz’] (Resource, method)  view  @view_config(context=Resource, request_method=‘GET’, …) Authorization  @view_config(…, permission=‘read’, …) Rendering  @view_config(…, renderer=‘json’)
    15. 15. classResourceCollection(object):__name__ = None Check auth__model__ = Nonekey_property=name’ defallow_access(self, client, permission):return permission == readorclient.admin Traverse to find sub-objectsdef__getitem__(self, name):obj = self.account.get_object(self.__model__, **{self.key_property: name })ifobjisNone: raiseexc.HTTPNotFound()obj.__parent__ = selfreturnobj
    16. 16. @view_config(context=Roles, renderer=json, Resources can berequest_method=GET’, permission=read) locateddeflist_roles(context, request):returndict((n.name, request.resource_url(n))fornincontext.find()) Convert and@view_config(context=Roles, renderer=json, validate JSONrequest_method=POST’,permission=create) inputdefcreate_role(context, request):n = context.new_object() value = V.RoleSchema().to_python(request.json, None)n.update(value) …
    17. 17.  Don’t trust the docs  Don’t trust the docs ▪ Don’t trust the docs Use fat models Framework support for REST & JSON You’re gonna have to learn some Ruby anyway JSON != BSON
    18. 18.  Port from homegrown framework to Pyramid Better test coverage Search support (SOLR / ElasticSearch) More testing with real-world deployments Finalize integration with deployment manager
    19. 19. Rick Copeland @rick446Arborian Consulting, LLC
    20. 20. Rick Copeland @rick446Arborian Consulting, LLC

    ×