Create a simple Service Offering with Python & Solaris<br />Python with Solaris Zones, Dtrace & OCCI<br />MUCOSUG meeting,...
Create a simple Service Offering with Python & Solaris<br />Warning:Clouds ahead<br />Python with Solaris Zones, Dtrace & ...
Intention: present why Python and Solaris are a perfect fit…<br />2011 (c) Thijs MetschThis work is licensed under a Creat...
This is just a showcase to demonstrate all necessary ingredients...<br />
Why Python, OCCI and Solaris?<br />
Why Python, OCCI and Solaris?<br />Containers<br />Robustness<br />ZFS<br />SMF<br />DTrace<br />Performance<br />
Why Python, OCCI and Solaris?<br />Innovation<br />RESTful<br />Python Powered<br />Standard<br />Cloud Protocol<br />Cont...
Why Python, OCCI and Solaris?<br />Innovation<br />RESTful<br />Readability<br />Simplicity<br />Python Powered<br />Maint...
Why Python, OCCI and Solaris?<br />Innovation<br />RESTful<br />Readability<br />Simplicity<br />Python Powered<br />Maint...
Source: http://xkcd.com/353/<br />
Source: lesher.ws/choose_python.pdf<br />
Cutting edge,Cool & Robust Technology<br />
General Flow<br />
Big Picture*<br />WSGIApp<br />Monitoring<br />ServiceOffering<br />Deployment<br />Developer<br />Manage<br />Access<br /...
Big Picture*<br />pyOCCI<br />WSGIApp<br />Monitoring<br />OCCI/Mercurial<br />ServiceOffering<br />Deployment<br />Develo...
Step 1 – Service definition<br />
A simple WSGI app<br />fromwsgiref.simple_serverimportmake_server<br />defapplication(env,response):<br />response_body='H...
Step 2 – Service deployment<br />
<ul><li>Python allows rapid prototyping and development
During the creation of OCCI we updated implementations periodically
Complete implementation of the Specification ~1000 LOC
Testframe written in Python</li></ul>„The Open Cloud Computing Interface (OCCI) is a RESTfulProtocol and API for all kinds...
<ul><li>Python allows rapid prototyping and development
During the creation of OCCI we updated implementations periodically
Complete implementation of the Specification ~1000 LOC
Testframe written in Python</li></ul>„The Open Cloud Computing Interface (OCCI) is a RESTfulProtocol and API for all kinds...
Step 3 – Backen Magic<br />
Python is ideal to create a (Scalable/Cloud) Service<br />-Very easy to integrate with other services (e.g. For Authentica...
Python is ideal to create a (Scalable/Cloud) Service<br />-Very easy to integrate with other services (e.g. For Authentica...
Integration with LDAP, CouchDB, AMQP<br />importldap<br />server=ldap.initialize('ldap://localhost:8080')<br />server.sear...
Now let‘s handle Resource Pools & Zones (OS Integration) with Python<br />
Wrapping zone.h with SWIG...<br />$ make all      <br />swig -Wall -python pyzone.i<br />gcc -I/usr/include/python2.6 -c p...
Step 4 – Using the service<br />
Batteries included (for the Client) ...<br />importhttplib<br />connection=httplib.HTTPConnection('localhost:8080')<br />c...
Step 5 – Monitoring<br />
You can user Dtrace to monitor the ZFS, Zones, Network etc.<br />
You can also create stuff like this...<br />Source: http://wiki.joyent.com/display/node/Using+Cloud+Analytics<br />
Or even trace Python itself...<br />
Check if Python probes are available<br />$ pfexec dtrace -ln 'python*:::'   ID   PROVIDER            MODULE              ...
Check if Python probes are available<br />$ pfexec dtrace -ln 'python*:::'   ID   PROVIDER            MODULE              ...
Count calls<br />#pragma D option quietpython*:::function-entry/(self->name = copyinstr(arg0)) != NULL && self->name >= "/...
Count calls<br />#pragma D option quietpython*:::function-entry/(self->name = copyinstr(arg0)) != NULL && self->name >= "/...
Trace the flow...<br />#pragma D option quietpython*:::function-entry/(self->name = copyinstr(arg0)) != NULL && self->name...
Upcoming SlideShare
Loading in …5
×

Simple Service Offering with Python, Solaris, OCCI and DTrace

2,866 views
2,829 views

Published on

Slideset describing a fictitious Cloud service like GAE or no.de but with python to demo the power of Python, OCCI and Solaris!

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,866
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Simple Service Offering with Python, Solaris, OCCI and DTrace

  1. 1. Create a simple Service Offering with Python & Solaris<br />Python with Solaris Zones, Dtrace & OCCI<br />MUCOSUG meeting, June 16th, 2011<br />
  2. 2. Create a simple Service Offering with Python & Solaris<br />Warning:Clouds ahead<br />Python with Solaris Zones, Dtrace & OCCI<br />MUCOSUG meeting, June 16th, 2011<br />
  3. 3. Intention: present why Python and Solaris are a perfect fit…<br />2011 (c) Thijs MetschThis work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.<br />
  4. 4. This is just a showcase to demonstrate all necessary ingredients...<br />
  5. 5. Why Python, OCCI and Solaris?<br />
  6. 6. Why Python, OCCI and Solaris?<br />Containers<br />Robustness<br />ZFS<br />SMF<br />DTrace<br />Performance<br />
  7. 7. Why Python, OCCI and Solaris?<br />Innovation<br />RESTful<br />Python Powered<br />Standard<br />Cloud Protocol<br />Containers<br />Robustness<br />ZFS<br />SMF<br />DTrace<br />Performance<br />
  8. 8. Why Python, OCCI and Solaris?<br />Innovation<br />RESTful<br />Readability<br />Simplicity<br />Python Powered<br />Maintainability<br />BDFL<br />Standard<br />Cloud Protocol<br />Development Speed<br />Containers<br />Robustness<br />ZFS<br />SMF<br />DTrace<br />Performance<br />
  9. 9. Why Python, OCCI and Solaris?<br />Innovation<br />RESTful<br />Readability<br />Simplicity<br />Python Powered<br />Maintainability<br />BDFL<br />Standard<br />Cloud Protocol<br />Development Speed<br />Fastest way to create a service and start earning money!<br />Containers<br />Robustness<br />ZFS<br />SMF<br />DTrace<br />Performance<br />
  10. 10. Source: http://xkcd.com/353/<br />
  11. 11. Source: lesher.ws/choose_python.pdf<br />
  12. 12. Cutting edge,Cool & Robust Technology<br />
  13. 13. General Flow<br />
  14. 14. Big Picture*<br />WSGIApp<br />Monitoring<br />ServiceOffering<br />Deployment<br />Developer<br />Manage<br />Access<br />Enduser<br />Solaris Zonewith WSGI App<br />Enduser<br />Solaris Zonewith WSGI App<br />Enduser<br />Solaris Zonewith WSGI App<br />...<br />...(Scaling)<br />Monitoring (Analytics)<br />Solaris Box<br />Internet<br />* Everybody needs one<br />
  15. 15. Big Picture*<br />pyOCCI<br />WSGIApp<br />Monitoring<br />OCCI/Mercurial<br />ServiceOffering<br />Deployment<br />Developer<br />Manage<br />Python<br />Access<br />Enduser<br />Solaris Zonewith WSGI App<br />Enduser<br />Solaris Zonewith WSGI App<br />Enduser<br />Solaris Zonewith WSGI App<br />...<br />...(Scaling)<br />Monitoring (Analytics)<br />DTrace<br />Solaris Box<br />Website<br />Internet<br />* Everybody needs one<br />
  16. 16. Step 1 – Service definition<br />
  17. 17. A simple WSGI app<br />fromwsgiref.simple_serverimportmake_server<br />defapplication(env,response):<br />response_body='Hello World'<br />status='200 OK'<br />response_headers=[('Content-Type','text/plain'),<br />('Content-Length',str(len(response_body)))<br />]<br />response(status,response_headers)<br />return[response_body]<br />httpd=make_server('localhost',4711,application)<br />httpd.handle_request()<br />
  18. 18. Step 2 – Service deployment<br />
  19. 19. <ul><li>Python allows rapid prototyping and development
  20. 20. During the creation of OCCI we updated implementations periodically
  21. 21. Complete implementation of the Specification ~1000 LOC
  22. 22. Testframe written in Python</li></ul>„The Open Cloud Computing Interface (OCCI) is a RESTfulProtocol and API for all kinds of Management tasks. OCCI was originally initiated to create a remote management API for IaaS model based Services. It has since evolved into a flexible API with a strong focus on integration, portability, interoperability and innovation while still offering a high degree of extensibility. The current release of the Open Cloud Computing Interface is suitable to serve many other models in addition to IaaS, including e.g. PaaS and SaaS.” – http://www.occi-wg.org<br />
  23. 23. <ul><li>Python allows rapid prototyping and development
  24. 24. During the creation of OCCI we updated implementations periodically
  25. 25. Complete implementation of the Specification ~1000 LOC
  26. 26. Testframe written in Python</li></ul>„The Open Cloud Computing Interface (OCCI) is a RESTfulProtocol and API for all kinds of Management tasks. OCCI was originally initiated to create a remote management API for IaaS model based Services. It has since evolved into a flexible API with a strong focus on integration, portability, interoperability and innovation while still offering a high degree of extensibility. The current release of the Open Cloud Computing Interface is suitable to serve many other models in addition to IaaS, including e.g. PaaS and SaaS.” – http://www.occi-wg.org<br />Python based OCCI implementation: pyOCCI<br />
  27. 27. Step 3 – Backen Magic<br />
  28. 28. Python is ideal to create a (Scalable/Cloud) Service<br />-Very easy to integrate with other services (e.g. For Authentication, DBs, Messaging, SCM...)<br />
  29. 29. Python is ideal to create a (Scalable/Cloud) Service<br />-Very easy to integrate with other services (e.g. For Authentication, DBs, Messaging, SCM...)<br />For Data transfer: e.g. Uploading the WSGI through mercurial<br />
  30. 30. Integration with LDAP, CouchDB, AMQP<br />importldap<br />server=ldap.initialize('ldap://localhost:8080')<br />server.search_s('dc=example,dc=com',ldap.SCOPE_SUBTREE,'(cn=foo*)',['cn','mail'])<br />importcouchdb<br />couch=couchdb.Server('http://localhost:5984/')<br />db=couch['services']<br />doc={'user':'foo'}<br />db.save(doc)<br />importpika<br />connection=pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))<br />channel=connection.channel()<br />channel.queue_declare(queue='hello')<br />channel.basic_publish(exchange='',routing_key='hello',body='Hello World!')<br />connection.close()<br />
  31. 31. Now let‘s handle Resource Pools & Zones (OS Integration) with Python<br />
  32. 32. Wrapping zone.h with SWIG...<br />$ make all <br />swig -Wall -python pyzone.i<br />gcc -I/usr/include/python2.6 -c pyzone_wrap.c<br />ld -shared pyzone_wrap.o -o _pyzone.so <br />$ cat test.py <br />import pyzone<br />zones = pyzone.zone_list()<br />print len(zones) # equal to: pyzone.get_nr_of_act_zones()<br />print zones<br />$ python test.py<br />2<br />['global', 'foo']<br />I can release this if there is interest…<br />
  33. 33. Step 4 – Using the service<br />
  34. 34. Batteries included (for the Client) ...<br />importhttplib<br />connection=httplib.HTTPConnection('localhost:8080')<br />connection.request('GET','/')<br />r1=connection.getresponse()<br />printr1.status,r1.reason<br />importjson<br />printjson.dumps({'msg':'Hello World'},sort_keys=True,indent=2)<br />Python also offers many GUI frameworks – also build-in (and therefore available on all platforms/clients) is Tk (Easy to use with pytkgen*)...<br />* Creates Tk based GUIs from JSON definition files.<br />
  35. 35. Step 5 – Monitoring<br />
  36. 36. You can user Dtrace to monitor the ZFS, Zones, Network etc.<br />
  37. 37. You can also create stuff like this...<br />Source: http://wiki.joyent.com/display/node/Using+Cloud+Analytics<br />
  38. 38. Or even trace Python itself...<br />
  39. 39. Check if Python probes are available<br />$ pfexec dtrace -ln 'python*:::'   ID   PROVIDER            MODULE                          FUNCTION NAME 9398  python861 libpython2.6.so.1.0                PyEval_EvalFrameEx function-entry 9399  python861 libpython2.6.so.1.0                      dtrace_entry function-entry 9400  python861 libpython2.6.so.1.0                PyEval_EvalFrameEx function-return 9401  python861 libpython2.6.so.1.0                     dtrace_return function-return<br />
  40. 40. Check if Python probes are available<br />$ pfexec dtrace -ln 'python*:::'   ID   PROVIDER            MODULE                          FUNCTION NAME 9398  python861 libpython2.6.so.1.0                PyEval_EvalFrameEx function-entry 9399  python861 libpython2.6.so.1.0                      dtrace_entry function-entry 9400  python861 libpython2.6.so.1.0                PyEval_EvalFrameEx function-return 9401  python861 libpython2.6.so.1.0                     dtrace_return function-return<br />If you need more you could tinker around with ceval.c in the Python sources<br />
  41. 41. Count calls<br />#pragma D option quietpython*:::function-entry/(self->name = copyinstr(arg0)) != NULL && self->name >= "/export/home/tmetsch/data/workspace/pyssf/pyocci/" /{    @[copyinstr(arg0)] = count();    self->ts=timestamp;} <br />$ pfexec dtrace -s ./test2.d   /export/home/tmetsch/data/workspace/pyssf/pyocci/examples/vm_skeleton.py                9<br />
  42. 42. Count calls<br />#pragma D option quietpython*:::function-entry/(self->name = copyinstr(arg0)) != NULL && self->name >= "/export/home/tmetsch/data/workspace/pyssf/pyocci/" /{    @[copyinstr(arg0)] = count();    self->ts=timestamp;} <br />Predicate<br />$ pfexec dtrace -s ./test2.d   /export/home/tmetsch/data/workspace/pyssf/pyocci/examples/vm_skeleton.py                9<br />
  43. 43. Trace the flow...<br />#pragma D option quietpython*:::function-entry/(self->name = copyinstr(arg0)) != NULL && self->name == "/export/home/tmetsch/data/workspace/pyssf/pyocci/core.py" /{    printf("step into: %s %s n", copyinstr(arg1), copyinstr(arg0));    self->traced = 1}python*:::function-return/self->traced/{    printf("back from: %s n", copyinstr(arg1));    self->traced = 0}<br />
  44. 44. Trace the flow...(2)<br />$ pfexec dtrace -s ./test4.d -c '/usr/bin/python ./pyocci/examples/vms_main.py' step into: <module> ./pyocci/examples/vms_main.py back from: <module> step into: Login ./pyocci/examples/vms_main.py back from: Login step into: MyService ./pyocci/examples/vms_main.py back from: MyService step into: __init__ ./pyocci/examples/vms_main.py back from: _load_ui_modules step into: start ./pyocci/examples/vms_main.py back from: __init__ <br />
  45. 45. Monitoring the OCCI service<br />$ pfexec dtrace -s ./test2.d -c '/usr/bin/python ./pyocci/examples/vms_main.py' Creating the virtual machineStarting virtual machine with id/compute/c768773a-2b6e-ce34-f08a-be3fd3a3670c  ./pyocci/examples/vms_main.py                                     5  Login                                                          2193  MyService                                                      3538<br />...<br />
  46. 46. Python + Solaris = Perfect Platform*<br />Cool Technology<br />* http://www.nohuddleoffense.de/2011/02/21/my-software-development-environment-for-python/<br />
  47. 47. Also for Development BTW (See Blog post)<br />Python + Solaris = Perfect Platform*<br />Cool Technology<br />* http://www.nohuddleoffense.de/2011/02/21/my-software-development-environment-for-python/<br />
  48. 48. References<br /><ul><li>Technologies
  49. 49. http://pyssf.sf.net
  50. 50. http://www.openindiana.org
  51. 51. http://wsgi.org/wsgi
  52. 52. http://www.occi-wg.org
  53. 53. https://github.com/tmetsch/pytkgen
  54. 54. http://mercurial.selenic.com/
  55. 55. Python
  56. 56. http://www.python.org
  57. 57. http://diveintopython3.org
  58. 58. Solaris
  59. 59. http://www.opensolaris.org
  60. 60. http://www.illumos.org
  61. 61. http://www.solarisinternals.com</li></ul>Source: amazon.de<br />
  62. 62. Thank you!<br />(c) 2011 - Thijs Metschtmetsch at opensolaris dot org<br /> <br />http://www.twitter.com/befreax<br />http://www.tmetsch.org<br />

×