Simple Service Offering with Python, Solaris, OCCI and DTrace

Create a simple Service Offering with Python & Solaris
Python with Solaris Zones, Dtrace & OCCI
MUCOSUG meeting, June 16th, 2011

Create a simple Service Offering with Python & Solaris
Warning:Clouds ahead
Python with Solaris Zones, Dtrace & OCCI
MUCOSUG meeting, June 16th, 2011

Intention: present why Python and Solaris are a perfect fit…
2011 (c) Thijs MetschThis work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

This is just a showcase to demonstrate all necessary ingredients...

Why Python, OCCI and Solaris?

Containers
Robustness
ZFS
SMF
DTrace
Performance

Innovation
RESTful
Python Powered
Standard
Cloud Protocol
Containers
Robustness
ZFS
SMF
DTrace
Performance

Innovation
RESTful
Readability
Simplicity
Python Powered
Maintainability
BDFL
Standard
Cloud Protocol
Development Speed
Containers
Robustness
ZFS
SMF
DTrace
Performance

Innovation
RESTful
Readability
Simplicity
Python Powered
Maintainability
BDFL
Standard
Cloud Protocol
Development Speed
Fastest way to create a service and start earning money!
Containers
Robustness
ZFS
SMF
DTrace
Performance

Source: http://xkcd.com/353/

Source: lesher.ws/choose_python.pdf

Cutting edge,Cool & Robust Technology

General Flow

Big Picture*
WSGIApp
Monitoring
ServiceOffering
Deployment
Developer
Manage
Access
Enduser
Solaris Zonewith WSGI App
Enduser
Enduser
...
...(Scaling)
Monitoring (Analytics)
Solaris Box
Internet
* Everybody needs one

Big Picture*
pyOCCI
WSGIApp
Monitoring
OCCI/Mercurial
ServiceOffering
Deployment
Developer
Manage
Python
Access
Enduser
Enduser
Enduser
...
...(Scaling)
Monitoring (Analytics)
DTrace
Solaris Box
Website
Internet
* Everybody needs one

Step 1 – Service definition

A simple WSGI app
fromwsgiref.simple_serverimportmake_server
defapplication(env,response):
response_body='Hello World'
status='200 OK'
response_headers=[('Content-Type','text/plain'),
('Content-Length',str(len(response_body)))
]
response(status,response_headers)
return[response_body]
httpd=make_server('localhost',4711,application)
httpd.handle_request()

Step 2 – Service deployment

Python allows rapid prototyping and development
During the creation of OCCI we updated implementations periodically
Complete implementation of the Specification ~1000 LOC
Testframe written in Python
„The Open Cloud Computing Interface (OCCI) is a RESTfulProtocol and API for all kinds of Management tasks. OCCI was originally initiated to create a remote management API for IaaS model based Services. It has since evolved into a flexible API with a strong focus on integration, portability, interoperability and innovation while still offering a high degree of extensibility. The current release of the Open Cloud Computing Interface is suitable to serve many other models in addition to IaaS, including e.g. PaaS and SaaS.” – http://www.occi-wg.org

Python allows rapid prototyping and development
During the creation of OCCI we updated implementations periodically
Complete implementation of the Specification ~1000 LOC
Testframe written in Python
„The Open Cloud Computing Interface (OCCI) is a RESTfulProtocol and API for all kinds of Management tasks. OCCI was originally initiated to create a remote management API for IaaS model based Services. It has since evolved into a flexible API with a strong focus on integration, portability, interoperability and innovation while still offering a high degree of extensibility. The current release of the Open Cloud Computing Interface is suitable to serve many other models in addition to IaaS, including e.g. PaaS and SaaS.” – http://www.occi-wg.org
Python based OCCI implementation: pyOCCI

Step 3 – Backen Magic

Python is ideal to create a (Scalable/Cloud) Service
-Very easy to integrate with other services (e.g. For Authentication, DBs, Messaging, SCM...)

Python is ideal to create a (Scalable/Cloud) Service
-Very easy to integrate with other services (e.g. For Authentication, DBs, Messaging, SCM...)
For Data transfer: e.g. Uploading the WSGI through mercurial

Integration with LDAP, CouchDB, AMQP
importldap
server=ldap.initialize('ldap://localhost:8080')
server.search_s('dc=example,dc=com',ldap.SCOPE_SUBTREE,'(cn=foo*)',['cn','mail'])
importcouchdb
couch=couchdb.Server('http://localhost:5984/')
db=couch['services']
doc={'user':'foo'}
db.save(doc)
importpika
connection=pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
channel=connection.channel()
channel.queue_declare(queue='hello')
channel.basic_publish(exchange='',routing_key='hello',body='Hello World!')
connection.close()

Now let‘s handle Resource Pools & Zones (OS Integration) with Python

Wrapping zone.h with SWIG...
$ make all
swig -Wall -python pyzone.i
gcc -I/usr/include/python2.6 -c pyzone_wrap.c
ld -shared pyzone_wrap.o -o _pyzone.so
$ cat test.py
import pyzone
zones = pyzone.zone_list()
print len(zones) # equal to: pyzone.get_nr_of_act_zones()
print zones
$ python test.py
2
['global', 'foo']
I can release this if there is interest…

Step 4 – Using the service

Batteries included (for the Client) ...
importhttplib
connection=httplib.HTTPConnection('localhost:8080')
connection.request('GET','/')
r1=connection.getresponse()
printr1.status,r1.reason
importjson
printjson.dumps({'msg':'Hello World'},sort_keys=True,indent=2)
Python also offers many GUI frameworks – also build-in (and therefore available on all platforms/clients) is Tk (Easy to use with pytkgen*)...
* Creates Tk based GUIs from JSON definition files.

Step 5 – Monitoring

You can user Dtrace to monitor the ZFS, Zones, Network etc.

You can also create stuff like this...
Source: http://wiki.joyent.com/display/node/Using+Cloud+Analytics

Or even trace Python itself...

Check if Python probes are available
$ pfexec dtrace -ln 'python*:::' ID PROVIDER MODULE FUNCTION NAME 9398 python861 libpython2.6.so.1.0 PyEval_EvalFrameEx function-entry 9399 python861 libpython2.6.so.1.0 dtrace_entry function-entry 9400 python861 libpython2.6.so.1.0 PyEval_EvalFrameEx function-return 9401 python861 libpython2.6.so.1.0 dtrace_return function-return

Check if Python probes are available
$ pfexec dtrace -ln 'python*:::' ID PROVIDER MODULE FUNCTION NAME 9398 python861 libpython2.6.so.1.0 PyEval_EvalFrameEx function-entry 9399 python861 libpython2.6.so.1.0 dtrace_entry function-entry 9400 python861 libpython2.6.so.1.0 PyEval_EvalFrameEx function-return 9401 python861 libpython2.6.so.1.0 dtrace_return function-return
If you need more you could tinker around with ceval.c in the Python sources

Count calls
#pragma D option quietpython*:::function-entry/(self->name = copyinstr(arg0)) != NULL && self->name >= "/export/home/tmetsch/data/workspace/pyssf/pyocci/" /{ @[copyinstr(arg0)] = count(); self->ts=timestamp;}
$ pfexec dtrace -s ./test2.d /export/home/tmetsch/data/workspace/pyssf/pyocci/examples/vm_skeleton.py 9

Count calls
#pragma D option quietpython*:::function-entry/(self->name = copyinstr(arg0)) != NULL && self->name >= "/export/home/tmetsch/data/workspace/pyssf/pyocci/" /{ @[copyinstr(arg0)] = count(); self->ts=timestamp;}
Predicate
$ pfexec dtrace -s ./test2.d /export/home/tmetsch/data/workspace/pyssf/pyocci/examples/vm_skeleton.py 9

Trace the flow...
#pragma D option quietpython*:::function-entry/(self->name = copyinstr(arg0)) != NULL && self->name == "/export/home/tmetsch/data/workspace/pyssf/pyocci/core.py" /{ printf("step into: %s %s n", copyinstr(arg1), copyinstr(arg0)); self->traced = 1}python*:::function-return/self->traced/{ printf("back from: %s n", copyinstr(arg1)); self->traced = 0}

Trace the flow...(2)
$ pfexec dtrace -s ./test4.d -c '/usr/bin/python ./pyocci/examples/vms_main.py' step into: <module> ./pyocci/examples/vms_main.py back from: <module> step into: Login ./pyocci/examples/vms_main.py back from: Login step into: MyService ./pyocci/examples/vms_main.py back from: MyService step into: __init__ ./pyocci/examples/vms_main.py back from: _load_ui_modules step into: start ./pyocci/examples/vms_main.py back from: __init__

Monitoring the OCCI service
$ pfexec dtrace -s ./test2.d -c '/usr/bin/python ./pyocci/examples/vms_main.py' Creating the virtual machineStarting virtual machine with id/compute/c768773a-2b6e-ce34-f08a-be3fd3a3670c ./pyocci/examples/vms_main.py 5 Login 2193 MyService 3538
...

Python + Solaris = Perfect Platform*
Cool Technology
* http://www.nohuddleoffense.de/2011/02/21/my-software-development-environment-for-python/

Also for Development BTW (See Blog post)
Python + Solaris = Perfect Platform*
Cool Technology
* http://www.nohuddleoffense.de/2011/02/21/my-software-development-environment-for-python/

References
Technologies
http://pyssf.sf.net
http://www.openindiana.org
http://wsgi.org/wsgi
http://www.occi-wg.org
https://github.com/tmetsch/pytkgen
http://mercurial.selenic.com/
Python
http://www.python.org
http://diveintopython3.org
Solaris
http://www.opensolaris.org
http://www.illumos.org
http://www.solarisinternals.com
Source: amazon.de

Thank you!
(c) 2011 - Thijs Metschtmetsch at opensolaris dot org

http://www.twitter.com/befreax
http://www.tmetsch.org

http://www.nohuddleoffense.de	519
url_unknown	13
https://twitter.com	3

Simple Service Offering with Python, Solaris, OCCI and DTrace

by befreax on Jun 16, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

3 Embeds 535

Statistics

Simple Service Offering with Python, Solaris, OCCI and DTrace — Presentation Transcript