From SQLAlchemy to Ming with TurboGears2

From SQLAlchemy to Ming with TurboGears2 Alessandro Molina alessandro.molina@axant.it @__amol__

Simple, Stupid, Serviceable● We wanted to offer to development team members the opportunity to take a first look at MongoDB.● We wanted to do this by letting them write an actual web app using Python.● We wanted a real use case: “expenses and incomings” tracking (serviceable)● We wanted everyone to understand how it is expected to work (stupid)● Must be blazing fast to implement, we dont have a lot of time (simple)

Involved Technologies ● Python → http://www.python.org ● TurboGears2 → http://www.turbogears.org ● SQLAlchemy → http://www.sqlalchemy.org ● Ming → http://merciless.sourceforge.net/index.html ● Genshi → http://genshi.edgewall.org ● ToscaWidgets → http://toscawidgets.org ● Sprox → http://www.sprox.org

Python Technologies● Why? Needed a WebApp ready in 5 minutes while being possible to quickly play with the code● Python: Our team is manly python developers based● TurboGears2: Our team is used to TG. Object Dispatch is really comfortable and the framework is incredibly fast to start with while still making possible to quickly get under the hood.● Genshi: clean and plain xhtml, even designers can understand it, also it came by default● ToscaWidgets: Can do most of the HTML for us● Sprox: Can do most of the CRUD for us

Quickstarting Our Project (tg2dev)Quasar-2:tuts amol$ paster quickstart mt_sqla● Project Structure Enter package name [mt_sqla]: paster quickstart mt_sqla (tg2dev)Quasar-2:tuts amol$ Enter package name [mt_sqla]: Would you prefer mako templates? (yes/[no]): Do you need prefer mako templates? (yes/[no]): Would you authentication and authorization in this project? ([yes]/no): with Controllers Selected need implied templates: authorization in this project? ([yes]/no): Do you and authentication and Selected and implied templates: tg.devtools#turbogears2 TurboGears 2. Standard Quickstart Template tg.devtools#turbogears2 TurboGears 2. Standard Quickstart Template for Index Page (tg2dev)Quasar-2:mt-sqla amol$ python setup.py develop (tg2dev)Quasar-2:mt-sqla amol$ python setup.py develop paster srunning develop paster srunning develop Installed /Users/amol/wrk/tuts/mt-sqla Installed /Users/amol/wrk/tuts/mt-sqla● Models for Users, (tg2dev)Quasar-2:mt-sqla amol$ paster setup-app development.ini (tg2dev)Quasar-2:mt-sqla amol$ paster setup-app development.ini Running setup_config() from mt_sqla.websetup Running setup_config() from mt_sqla.websetup Creating tables Groups and Creating tables (tg2dev)Quasar-2:mt-sqla amol$ paster serve development.ini (tg2dev)Quasar-2:mt-sqla amol$ paster serve development.ini Starting server in PID 23939. Permission serving on server in PID 23939. Starting http://127.0.0.1:8080 serving on http://127.0.0.1:8080● Authentication (tg2dev)Quasar-2:tuts amol$ paster quickstart –ming mt_ming Enter package name [mt_ming]: paster quickstart –ming mt_ming (tg2dev)Quasar-2:tuts amol$ Enter package name [mt_ming]: and Authorization Would you prefer mako templates? (yes/[no]): Do you need prefer mako templates? (yes/[no]): Would you authentication and authorization in this project? ([yes]/no): Selected need implied templates: authorization in this project? ([yes]/no): Do you and authentication and Selected and implied templates: tg.devtools#turbogears2 TurboGears 2. Standard Quickstart Template tg.devtools#turbogears2 TurboGears 2. Standard Quickstart Template● Both on SQLA (tg2dev)Quasar-2:mt-ming amol$ python setup.py develop (tg2dev)Quasar-2:mt-ming amol$ python setup.py develop paster srunning develop paster srunning develop Installed /Users/amol/wrk/tuts/mt-ming and Ming Installed /Users/amol/wrk/tuts/mt-ming (tg2dev)Quasar-2:mt-ming amol$ paster setup-app development.ini (tg2dev)Quasar-2:mt-ming amol$ paster setup-app development.ini Running setup_app() from mt_ming.websetup Running setup_app() from mt_ming.websetup (tg2dev)Quasar-2:mt-ming amol$ paster serve development.ini (tg2dev)Quasar-2:mt-ming amol$ paster serve development.ini Starting server in PID 23939. serving on server in PID 23939. Starting http://127.0.0.1:8080 serving on http://127.0.0.1:8080

Overview Browser Genshi ToscaWidgets TurboGears Controller Code Sprox (TGAdmin) Ming SQLAlchemy

SQLAlchemy● Supports pratically everything: PostgresSQL, MySQL, SQLite, Firebird, Oracle, MSSQL, Sybase, DB2, Informix, SAPDB, MSAccess● Declarative ORM layer with access to a powerful and flexible query builder● Unlike many tools, it never enforces schemas or relies on naming conventions of any kind.● Unit Of Work system organizes pending insert/update/delete operations into queues and flushes them all in one batch (Patterns of Enterprise Application Architecture - Martin Fowler)

Python MongoDB Ecosystem ● PyMongo: Driver for MongoDB, everything is a dictionary ● MongoEngine: ORM, query language is familiar to Django users ● Ming: ORM, query language is familiar to SQLAlchemy users ● MongoKit: ORM, written to be as simple and light as possibleMongoKit Mingclass BlogPost(Document): class WikiComment(MappedClass): structure = { class __mongometa__: title:unicode, session = session body:unicode, name = wiki_comment author:unicode, date_creation:datetime.datetime, _id = FieldProperty(schema.ObjectId) rank:int text=FieldProperty(str, if_missing=) } page_id = ForeignIdProperty(WikiPage) page=RelationProperty(WikiPage)tutorial.BlogPost.find({body: {$exists: True}}) WikiComment.query.find({text:{$exists:True}})MongoEngine PyMongoclass User(Document): post = {"author": "Mike", email = StringField(required=True) ... "text": "My first blog post!", first_name = StringField(max_length=50) ... "tags": ["mongodb", "python"], last_name = StringField(max_length=50) ... "date": datetime.datetime.utcnow()}Users.objects(age__lte=18) db.posts.find({"author": "Mike"})

Ming● SQLAlchemy inspired, easy to catch up if you are used to SQLAlchemy● Declarative ORM layer, use ORM through objects and get direct access to mongo through collections● Integrated Migrations (deprecated attributes and even on-demand lazy migration code)● Unit of Work, we tried it, we loved it, we wanted it even on MongoDB● Mongo In Memory, complete implementation of MongoDB in memory for unit testing

Some Little Magic (TGAdmin)● It did CRUD for us.● Works both with SQLAlchemy and Ming● Comes for free with TurboGears2● Sprox based, deeply customizable

Storing our Datafrom sqlalchemy import Table, Column from ming import schema as s from sqlalchemy import Table, Columnfrom sqlalchemy.types import Unicode, Integer, from ming import schema as s from ming.orm import FieldProperty, MapperFloat, sqlalchemy.types import Unicode, Integer, from DateTime from ming.orm import FieldProperty, Mapper from ming.orm.declarative import MappedClass Float, DateTimefrom datetime import datetime from ming.orm.declarative import MappedClass from session import DBSession from datetime import datetime from session import DBSession from datetime import datetime from datetime import datetimeclass MoneyTransfer(DeclarativeBase): class MoneyTransfer(MappedClass): class MoneyTransfer(DeclarativeBase): __tablename__ = money_transfers class MoneyTransfer(MappedClass): class __mongometa__: __tablename__ = money_transfers class __mongometa__: session = DBSession session = DBSession name = money_transfers name = money_transfers uid = Column(Integer, autoincrement=True, _id = FieldProperty(s.ObjectId) uid = Column(Integer,primary_key=True) autoincrement=True, _id = FieldProperty(s.ObjectId) date = FieldProperty(s.DateTime, primary_key=True) date = Column(DateTime, default=datetime.now) date = FieldProperty(s.DateTime, if_missing=datetime.now) sign = =Column(Integer, default=1) date Column(DateTime, default=datetime.now) if_missing=datetime.now) sign = FieldProperty(s.Int, if_missing=1) sign = Column(Integer, default=1) description = Column(Unicode(255)) sign = FieldProperty(s.Int, if_missing=1) description = FieldProperty(s.String) description = Column(Unicode(255)) value = Column(Float, default=0) description = FieldProperty(s.String) value = FieldProperty(s.Float, if_missing=0) value = Column(Float, default=0) value = FieldProperty(s.Float, if_missing=0)

Retrieving and Displaying Datafrom tw.forms import DataGrid from tw.forms import DataGridfrom tw.forms.datagrid DataGrid from tw.forms import import Column from tw.forms.datagrid DataGrid from tw.forms import import Column from tw.forms.datagrid import Column from tw.forms.datagrid import Columntransactions_grid = DataGrid(fields=[ transactions_grid = DataGrid(fields=[ transactions_grid date), Column(Date, = DataGrid(fields=[ transactions_grid date), Column(Date, = DataGrid(fields=[ Column(Date, date), Column(Description, description), Column(Date, date), Column(Description, description), Column(Description, description), Column(Value, lambda row:row.value*row.sign)]) Column(Description, description), Column(Value, lambda row:row.value*row.sign)]) Column(Value, lambda row:row.value*row.sign)]) Column(Value, lambda row:row.value*row.sign)])class RootController(BaseController): class RootController(BaseController): class RootController(BaseController): @expose(mt_sqla.templates.index) class RootController(BaseController): @expose(mt_ming.templates.index) @expose(mt_sqla.templates.index) def index(self): @expose(mt_ming.templates.index) def index(self): def index(self): DBSession.query(MoneyTransfer).all() transactions = def index(self): MoneyTransfer.query.find().all() transactions = transactions = DBSession.query(MoneyTransfer).all() total = sum((t.value*t.sign for t in transactions)) transactions = MoneyTransfer.query.find().all() total = sum((t.value*t.sign for t in transactions)) total = sum((t.value*t.sign for t in transactions)) return dict(page=index, total = sum((t.value*t.sign for t in transactions)) return dict(page=index, return dict(page=index, transactions=transactions, return dict(page=index, transactions=transactions, transactions=transactions, transactions_grid=transactions_grid, transactions=transactions, transactions_grid=transactions_grid, transactions_grid=transactions_grid, total=total) transactions_grid=transactions_grid, total=total) total=total) total=total) <div style="width:650px; margin-top:20px;"> <div style="width:650px; margin-top:20px;"> <a href="${tg.url(/admin/moneytransfers/new)}">Add New Transaction</a> <a href="${tg.url(/admin/moneytransfers/new)}">Add New Transaction</a> ${transactions_grid(transactions)} ${transactions_grid(transactions)} <div style="text-align:right;"><strong>AVAILABLE:</strong> ${total}</div> <div style="text-align:right;"><strong>AVAILABLE:</strong> ${total}</div> </div> </div>

What changed● Template remained the same. Thanks god we have an ORM.● Had to change the model.● Had to change just 1 line of controller code to retrieve the data. Ming and SQLA are similar enough● Did we really need to change that line? We just want to get a list of objects...

Sprox, abstraction over abstraction ● ORMProvider, provides an abstraction over the ORM ● ORMProviderSelector, automatically detects the provider to use from a model. ● Mix those together and you have a db independent layer. ● Provider.query(self, entity, limit=None, offset=0, limit_fields=None, order_by=None, desc=False) → get all objects of a collection ● Provider.get_obj(self, entity, params) → get an object ● Provider.update(self, entity, params) → update an object ● Provider.create(self, entity, params) → create a new objectSQLAlchemytransactions = DBSession.query(MoneyTransfer).all() transactions = DBSession.query(MoneyTransfer).all() Sprox count, transactions = provider.query(MoneyTransfer)Ming count, transactions = provider.query(MoneyTransfer)transactions = MoneyTransfer.query.find().all() transactions = MoneyTransfer.query.find().all()

Experiments Experience● Starting with a really simple use case made people get comfortable with MongoDB and feel confident enough to start learning more complex features.● The idea of being possible to use sprox to abstract over the db, making possible to switch back anytime, created a “safety net” idea in people.● When people feel safe they start experimenting a lot more and learn by themselves.

From SQLAlchemy to Ming with TurboGears2

by amol on Oct 02, 2011

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 1

Statistics