Your SlideShare is downloading. ×
0
+Programmatically Managing Python Workloads AcrossMultiple CloudsChayim I. Kirshen (chayim@gnupower.net / @chayimk)
+    Traditional Development                       Developers write code, OPS supports                        code       ...
+    Whither DevOps            Traditional Operations                      DevOps       Supports operational objectives  ...
+    To Collaborate We Must    Understand…       How can Operations formalize and track system change?           Not jus...
+    Fit the Middle            Software                                  Operations           Development                 ...
+    Technology Stack       Django       libCloud       Celery       Paramiko       Puppet
+    Exploring the Django-ized    Components
+    Where This Went               Create nodes across EC2 (us-west-1, us-                west-2, us-east-1), and GoGrid ...
+    Settings Loading is Painful       settings.py gets to be too big       Easier to reduce testing impact by segregati...
+# helpers.pydef settings_loader(module_base, CONFIG_DIR):  if CONFIG_DIR not in sys.path:     sys.path.append(CONFIG_DIR)...
+    Configuring Components                                    libCloud_settings.py    root = os.path.abspath(os.path.dirn...
+    Unfogging libCloud                  NodeImage                  NodeSize         Driver               create_node   .d...
+    EC2Driver = get_driver(Provider.EC2)    self.driver = EC2Driver(EC2_ACCESS_ID, EC2_SECRET_KEY)    image = NodeImage(i...
+    Node Pattern       Connect to node           Touch /etc/publicip           Set hostname           Puppetmaster Ad...
+    Working with Vegetables       Django integrated                   All celeryable tasks go in           python mana...
+       Celerized object is always returned           result = <foo>.delay(arg, arg, arg)           result.ready == Tru...
+    Parallelization Problems       SSH           Fabric == AWESOMESAUCE!           Fabric == Singleton            Pa...
+    To Collaborate we understood!       How did Operations formalize and track system change?           Revision contro...
Upcoming SlideShare
Loading in...5
×

Programmatically Managing Python Workloads Across Multiple Clouds

726

Published on

I delivered this talk at PyCon Canada 2012. The focus: understanding the role of DevOps, how it differs from the traditional model dev+operational deployment model. There's also some lightweight detail on implementing this using Python.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
726
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Ops feels like developers sling things over the fenceDev feels that Ops doesn’t support themOPS feels that dev writes it, but that OPS supports forever
  • Devops solves operation issues through developmentDevops embraces change – but really works on codifying that change
  • As developers, all the value is in testing as close to the customer as possible
  • Small django application allowing people to deploy environmentsWhy django? I like the ORM – I LOVE the ORM
  • So what did I learn?
  • Assume you’re already django happy
  • Issues with broker versus responses
  • Paramiko -&gt; wait for commands with read and outLogger.debug(outputs)
  • As developers, all the value is in testing as close to the customer as possible
  • Transcript of "Programmatically Managing Python Workloads Across Multiple Clouds"

    1. 1. +Programmatically Managing Python Workloads AcrossMultiple CloudsChayim I. Kirshen (chayim@gnupower.net / @chayimk)
    2. 2. + Traditional Development  Developers write code, OPS supports code  OPS Teams eschew risk  Systems that change are unknown quantities  All changes live forever
    3. 3. + Whither DevOps Traditional Operations DevOps  Supports operational objectives  Drives operational objectives  Service focused  Customer focused  Frequent manual intervention  Managing Operations through  IT Mindset - Scripter, Automation and Development SysAdmin, Paranoid Futurist  IT Developer - Coder, SysAdmin, Coach, Paranoid  Strives for consistency accepts Prepared Futurist relativity  Ensures consistency through code
    4. 4. + To Collaborate We Must Understand…  How can Operations formalize and track system change?  Not just changes in code, but changes in infrastructure  How can Operations embrace change, pro-actively support organizational needs, but reduce risk?  How can Development work in simulated production?  How can we encourage collaboration?
    5. 5. + Fit the Middle Software Operations Development DevOps
    6. 6. + Technology Stack  Django  libCloud  Celery  Paramiko  Puppet
    7. 7. + Exploring the Django-ized Components
    8. 8. + Where This Went  Create nodes across EC2 (us-west-1, us- west-2, us-east-1), and GoGrid  Parallelized workload creation/destruction  Configured all automatically with Puppet
    9. 9. + Settings Loading is Painful  settings.py gets to be too big  Easier to reduce testing impact by segregating services  Separate the concern # settings.py thismodule = sys.modules[__name__] helpers.settings_loader(thismodule, CONFIG_DIR)
    10. 10. +# helpers.pydef settings_loader(module_base, CONFIG_DIR): if CONFIG_DIR not in sys.path: sys.path.append(CONFIG_DIR) settings_files = os.listdir(CONFIG_DIR) for setting in settings_files: if setting[-3:] != ".py": continue # import the module module = __import__(setting[:-3]) for key in dir(module): # hidden variables should never be imported, theyre either # internals for our imported module - or not worth importing if key[:2] == "__": continue # reflect in, and ensure the setting were loading is a string # only those can be dynamicaly loaded. setattr(module_base, key, getattr(module, key))
    11. 11. + Configuring Components libCloud_settings.py root = os.path.abspath(os.path.dirname(__file__) AWS_ACCESS_KEY = ”<xxx>” AWS_SECRET_KEY = ”<yyy>” SSH_USERNAME = "ubuntu” SSH_KEY = os.path.join(root, “yourkey.pem”) celery_settings.py  BROKER_URL = mongodb://localhost:27017/celeries  BACKEND_URL = mongodb://localhost:27017/celery_results
    12. 12. + Unfogging libCloud NodeImage NodeSize Driver create_node .delay
    13. 13. + EC2Driver = get_driver(Provider.EC2) self.driver = EC2Driver(EC2_ACCESS_ID, EC2_SECRET_KEY) image = NodeImage(id=“ami-87712ac2”, name="", driver="") size = NodeSize(id=“m1.small”, name="", ram=None, disk=None, bandwidth=None, price=None, driver="") self.driver.create_node(name=name, image=image, size=size, ex_keyname = sshkeyname, ex_securitygroup = securitygroup) nodes = self.driver.list_nodes()
    14. 14. + Node Pattern  Connect to node  Touch /etc/publicip  Set hostname  Puppetmaster Address in /etc/hosts  Register with the puppet master  puppet agent --waitforcert 60 –-verbose –server=puppet.master  Sign cert on the puppet master  Puppetize the client
    15. 15. + Working with Vegetables  Django integrated  All celeryable tasks go in  python manage.py migrate tasks.py for your django celery || python manage.py project syncdb  Decorate functions with  python manage.py celery @task worker –loglevel=info  Call existing function with your code with  Standalone <name>.delay(arg, arg, arg)  celery -A tasks worker --loglevel=info -B –E
    16. 16. +  Celerized object is always returned  result = <foo>.delay(arg, arg, arg)  result.ready == True|false  For blocking  Add all cerery results to a list  Iterate looking for completion
    17. 17. + Parallelization Problems  SSH  Fabric == AWESOMESAUCE!  Fabric == Singleton   Paramiko!  Logging  Prefixing log lines with proposed host name
    18. 18. + To Collaborate we understood!  How did Operations formalize and track system change?  Revision control, code, comment  How did Operations embrace change, pro-actively support organizational needs, but reduce risk?  Deploy infrastructure through automation, without impacting production  How did Development work in simulated production?  Simulating the setup, with a different cloud provider  How did we encourage collaboration?  Developers and OPS write puppet code  Develops and OPS sit together  Developers and OPS learn from each other
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×