• Save
Simple Service Offering with Python, Solaris, OCCI and DTrace
Upcoming SlideShare
Loading in...5
×
 

Simple Service Offering with Python, Solaris, OCCI and DTrace

on

  • 2,765 views

Slideset describing a fictitious Cloud service like GAE or no.de but with python to demo the power of Python, OCCI and Solaris!

Slideset describing a fictitious Cloud service like GAE or no.de but with python to demo the power of Python, OCCI and Solaris!

Statistics

Views

Total Views
2,765
Views on SlideShare
1,859
Embed Views
906

Actions

Likes
1
Downloads
0
Comments
0

3 Embeds 906

http://www.nohuddleoffense.de 890
url_unknown 13
https://twitter.com 3

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

CC Attribution-NonCommercial-NoDerivs LicenseCC Attribution-NonCommercial-NoDerivs LicenseCC Attribution-NonCommercial-NoDerivs License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Simple Service Offering with Python, Solaris, OCCI and DTrace Simple Service Offering with Python, Solaris, OCCI and DTrace Presentation Transcript

  • Create a simple Service Offering with Python & Solaris
    Python with Solaris Zones, Dtrace & OCCI
    MUCOSUG meeting, June 16th, 2011
  • Create a simple Service Offering with Python & Solaris
    Warning:Clouds ahead
    Python with Solaris Zones, Dtrace & OCCI
    MUCOSUG meeting, June 16th, 2011
  • Intention: present why Python and Solaris are a perfect fit…
    2011 (c) Thijs MetschThis work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.
  • This is just a showcase to demonstrate all necessary ingredients...
  • Why Python, OCCI and Solaris?
  • Why Python, OCCI and Solaris?
    Containers
    Robustness
    ZFS
    SMF
    DTrace
    Performance
  • Why Python, OCCI and Solaris?
    Innovation
    RESTful
    Python Powered
    Standard
    Cloud Protocol
    Containers
    Robustness
    ZFS
    SMF
    DTrace
    Performance
  • Why Python, OCCI and Solaris?
    Innovation
    RESTful
    Readability
    Simplicity
    Python Powered
    Maintainability
    BDFL
    Standard
    Cloud Protocol
    Development Speed
    Containers
    Robustness
    ZFS
    SMF
    DTrace
    Performance
  • Why Python, OCCI and Solaris?
    Innovation
    RESTful
    Readability
    Simplicity
    Python Powered
    Maintainability
    BDFL
    Standard
    Cloud Protocol
    Development Speed
    Fastest way to create a service and start earning money!
    Containers
    Robustness
    ZFS
    SMF
    DTrace
    Performance
  • Source: http://xkcd.com/353/
  • Source: lesher.ws/choose_python.pdf
  • Cutting edge,Cool & Robust Technology
  • General Flow
  • Big Picture*
    WSGIApp
    Monitoring
    ServiceOffering
    Deployment
    Developer
    Manage
    Access
    Enduser
    Solaris Zonewith WSGI App
    Enduser
    Solaris Zonewith WSGI App
    Enduser
    Solaris Zonewith WSGI App
    ...
    ...(Scaling)
    Monitoring (Analytics)
    Solaris Box
    Internet
    * Everybody needs one
  • Big Picture*
    pyOCCI
    WSGIApp
    Monitoring
    OCCI/Mercurial
    ServiceOffering
    Deployment
    Developer
    Manage
    Python
    Access
    Enduser
    Solaris Zonewith WSGI App
    Enduser
    Solaris Zonewith WSGI App
    Enduser
    Solaris Zonewith WSGI App
    ...
    ...(Scaling)
    Monitoring (Analytics)
    DTrace
    Solaris Box
    Website
    Internet
    * Everybody needs one
  • Step 1 – Service definition
  • A simple WSGI app
    fromwsgiref.simple_serverimportmake_server
    defapplication(env,response):
    response_body='Hello World'
    status='200 OK'
    response_headers=[('Content-Type','text/plain'),
    ('Content-Length',str(len(response_body)))
    ]
    response(status,response_headers)
    return[response_body]
    httpd=make_server('localhost',4711,application)
    httpd.handle_request()
  • Step 2 – Service deployment
    • Python allows rapid prototyping and development
    • During the creation of OCCI we updated implementations periodically
    • Complete implementation of the Specification ~1000 LOC
    • Testframe written in Python
    „The Open Cloud Computing Interface (OCCI) is a RESTfulProtocol and API for all kinds of Management tasks. OCCI was originally initiated to create a remote management API for IaaS model based Services. It has since evolved into a flexible API with a strong focus on integration, portability, interoperability and innovation while still offering a high degree of extensibility. The current release of the Open Cloud Computing Interface is suitable to serve many other models in addition to IaaS, including e.g. PaaS and SaaS.” – http://www.occi-wg.org
    • Python allows rapid prototyping and development
    • During the creation of OCCI we updated implementations periodically
    • Complete implementation of the Specification ~1000 LOC
    • Testframe written in Python
    „The Open Cloud Computing Interface (OCCI) is a RESTfulProtocol and API for all kinds of Management tasks. OCCI was originally initiated to create a remote management API for IaaS model based Services. It has since evolved into a flexible API with a strong focus on integration, portability, interoperability and innovation while still offering a high degree of extensibility. The current release of the Open Cloud Computing Interface is suitable to serve many other models in addition to IaaS, including e.g. PaaS and SaaS.” – http://www.occi-wg.org
    Python based OCCI implementation: pyOCCI
  • Step 3 – Backen Magic
  • Python is ideal to create a (Scalable/Cloud) Service
    -Very easy to integrate with other services (e.g. For Authentication, DBs, Messaging, SCM...)
  • Python is ideal to create a (Scalable/Cloud) Service
    -Very easy to integrate with other services (e.g. For Authentication, DBs, Messaging, SCM...)
    For Data transfer: e.g. Uploading the WSGI through mercurial
  • Integration with LDAP, CouchDB, AMQP
    importldap
    server=ldap.initialize('ldap://localhost:8080')
    server.search_s('dc=example,dc=com',ldap.SCOPE_SUBTREE,'(cn=foo*)',['cn','mail'])
    importcouchdb
    couch=couchdb.Server('http://localhost:5984/')
    db=couch['services']
    doc={'user':'foo'}
    db.save(doc)
    importpika
    connection=pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
    channel=connection.channel()
    channel.queue_declare(queue='hello')
    channel.basic_publish(exchange='',routing_key='hello',body='Hello World!')
    connection.close()
  • Now let‘s handle Resource Pools & Zones (OS Integration) with Python
  • Wrapping zone.h with SWIG...
    $ make all
    swig -Wall -python pyzone.i
    gcc -I/usr/include/python2.6 -c pyzone_wrap.c
    ld -shared pyzone_wrap.o -o _pyzone.so
    $ cat test.py
    import pyzone
    zones = pyzone.zone_list()
    print len(zones) # equal to: pyzone.get_nr_of_act_zones()
    print zones
    $ python test.py
    2
    ['global', 'foo']
    I can release this if there is interest…
  • Step 4 – Using the service
  • Batteries included (for the Client) ...
    importhttplib
    connection=httplib.HTTPConnection('localhost:8080')
    connection.request('GET','/')
    r1=connection.getresponse()
    printr1.status,r1.reason
    importjson
    printjson.dumps({'msg':'Hello World'},sort_keys=True,indent=2)
    Python also offers many GUI frameworks – also build-in (and therefore available on all platforms/clients) is Tk (Easy to use with pytkgen*)...
    * Creates Tk based GUIs from JSON definition files.
  • Step 5 – Monitoring
  • You can user Dtrace to monitor the ZFS, Zones, Network etc.
  • You can also create stuff like this...
    Source: http://wiki.joyent.com/display/node/Using+Cloud+Analytics
  • Or even trace Python itself...
  • Check if Python probes are available
    $ pfexec dtrace -ln 'python*:::'   ID   PROVIDER            MODULE                          FUNCTION NAME 9398  python861 libpython2.6.so.1.0                PyEval_EvalFrameEx function-entry 9399  python861 libpython2.6.so.1.0                      dtrace_entry function-entry 9400  python861 libpython2.6.so.1.0                PyEval_EvalFrameEx function-return 9401  python861 libpython2.6.so.1.0                     dtrace_return function-return
  • Check if Python probes are available
    $ pfexec dtrace -ln 'python*:::'   ID   PROVIDER            MODULE                          FUNCTION NAME 9398  python861 libpython2.6.so.1.0                PyEval_EvalFrameEx function-entry 9399  python861 libpython2.6.so.1.0                      dtrace_entry function-entry 9400  python861 libpython2.6.so.1.0                PyEval_EvalFrameEx function-return 9401  python861 libpython2.6.so.1.0                     dtrace_return function-return
    If you need more you could tinker around with ceval.c in the Python sources
  • Count calls
    #pragma D option quietpython*:::function-entry/(self->name = copyinstr(arg0)) != NULL && self->name >= "/export/home/tmetsch/data/workspace/pyssf/pyocci/" /{    @[copyinstr(arg0)] = count();    self->ts=timestamp;}
    $ pfexec dtrace -s ./test2.d   /export/home/tmetsch/data/workspace/pyssf/pyocci/examples/vm_skeleton.py                9
  • Count calls
    #pragma D option quietpython*:::function-entry/(self->name = copyinstr(arg0)) != NULL && self->name >= "/export/home/tmetsch/data/workspace/pyssf/pyocci/" /{    @[copyinstr(arg0)] = count();    self->ts=timestamp;}
    Predicate
    $ pfexec dtrace -s ./test2.d   /export/home/tmetsch/data/workspace/pyssf/pyocci/examples/vm_skeleton.py                9
  • Trace the flow...
    #pragma D option quietpython*:::function-entry/(self->name = copyinstr(arg0)) != NULL && self->name == "/export/home/tmetsch/data/workspace/pyssf/pyocci/core.py" /{    printf("step into: %s %s n", copyinstr(arg1), copyinstr(arg0));    self->traced = 1}python*:::function-return/self->traced/{    printf("back from: %s n", copyinstr(arg1));    self->traced = 0}
  • Trace the flow...(2)
    $ pfexec dtrace -s ./test4.d -c '/usr/bin/python ./pyocci/examples/vms_main.py' step into: <module> ./pyocci/examples/vms_main.py back from: <module> step into: Login ./pyocci/examples/vms_main.py back from: Login step into: MyService ./pyocci/examples/vms_main.py back from: MyService step into: __init__ ./pyocci/examples/vms_main.py back from: _load_ui_modules step into: start ./pyocci/examples/vms_main.py back from: __init__
  • Monitoring the OCCI service
    $ pfexec dtrace -s ./test2.d -c '/usr/bin/python ./pyocci/examples/vms_main.py' Creating the virtual machineStarting virtual machine with id/compute/c768773a-2b6e-ce34-f08a-be3fd3a3670c  ./pyocci/examples/vms_main.py                                     5  Login                                                          2193  MyService                                                      3538
    ...
  • Python + Solaris = Perfect Platform*
    Cool Technology
    * http://www.nohuddleoffense.de/2011/02/21/my-software-development-environment-for-python/
  • Also for Development BTW (See Blog post)
    Python + Solaris = Perfect Platform*
    Cool Technology
    * http://www.nohuddleoffense.de/2011/02/21/my-software-development-environment-for-python/
  • References
    • Technologies
    • http://pyssf.sf.net
    • http://www.openindiana.org
    • http://wsgi.org/wsgi
    • http://www.occi-wg.org
    • https://github.com/tmetsch/pytkgen
    • http://mercurial.selenic.com/
    • Python
    • http://www.python.org
    • http://diveintopython3.org
    • Solaris
    • http://www.opensolaris.org
    • http://www.illumos.org
    • http://www.solarisinternals.com
    Source: amazon.de
  • Thank you!
    (c) 2011 - Thijs Metschtmetsch at opensolaris dot org
     
    http://www.twitter.com/befreax
    http://www.tmetsch.org