Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building a Better Mousetrap for the Cloud


Published on

Published in: Economy & Finance, Business
  • Be the first to comment

Building a Better Mousetrap for the Cloud

  1. 1. Rube Goldberg Architecture Building a Better Mousetrap for the Cloud Ezra Zygmuntowicz
  2. 2. 3 Pillars of a good Cloud Infrastructure Automation Command & Control Scalable State Storage
  3. 3. Automation: Chef Command & Control: Nanite Scalable State Storage: Redis
  4. 4. Go big or go home...
  5. 5. Automation: Chef • Idempotent configuration management • Embed-able and flexible • Heavy lifting behind Solo/Flex • No more “every server is a unique snowflake” • Badass
  6. 6. Basic Concepts: Resources Recipes Providers
  7. 7. Resources: apt_package bash package cron perl csh portage_package directory python execute remote_directory file remote_file gem_package route group ruby http_request script link service mount template user
  8. 8. Recipes:
  9. 9. Providers:
  10. 10. Converging • Recipes are loaded in specified order • Resources are compiled into objects and stored in a ResourceCollection • ResourceCollection is iterated and the right Provider for each Resource is invoked • The Providers runs the specified action on each Resource
  11. 11. Chef Solo • chef-solo -r • Downloads a tarball of recipes and runs them • Dead simple to get started with
  12. 12. Chef Server • chef-client • Communicates with the chef-server to get recipes and JSON data • Allows for searching across other nodes attributes • Uses open-id for node authentication
  13. 13. Chef Server
  14. 14. State Storage: Redis persistent memcached on steroids
  15. 15. Redis Features • Asynchronous Persist to disk • Horizontally scalable • Values can have types • STRING’s, LIST’s and SET’s • Atomic Operations (push, pop, incr, decr, set intersection)
  16. 16. Let’s see a demo
  17. 17. Nanite
  18. 18. Built around RabbitMQ • Written in erlang, cluster-able, highly scalable, fast as hell. • AMQP protocol provides many nice features • Transient, Persistent and Transactional semantics
  19. 19. <3 RabbitMQ
  20. 20. Nanite agents consist of multiple Actors
  21. 21. Nanite agents advertise their services and status Feeds#crawl advertises: /feeds/crawl Load average is advertised as default status
  22. 22. Nanite Mappers Track nanites and their advertised services and status Can do dispatch based on a number of factors Run inside your Merb or Rails app or as a separate service State of all nanites is replicated across all mappers in memory *or* stored in Redis
  23. 23. Multiple Dispatch Styles
  24. 24. Least loaded dispatch and the fitness function
  25. 25. Agents ping the mapper exchange every @ping_time seconds. Mappers track the state of all nanites and remove them from mapping if they haven’t reported in within a timeout
  26. 26. Nanite gives us: • Presence, we know when nanites are ready for requests or not. • Self assembly, nanites can come and go and can run anywhere with zero configuration in the mappers. • Dispatch based on load or any fitness function that suits your app • Easily take advantage of cloud
  27. 27. Nanite Demo Please download and run this shell script:
  28. 28. Questions?