Making Django and NoSQL Play Nice

Making Django and NoSQL Play Nice Alex Gaynor Berlin

NoSQL Any database that doesn’t speak SQL Usually non-relational databases e.g. Cassandra, Redis, MongoDB

2 Part Talk 50% 50% Current Internals Coming Changes

What does playing nice mean? from mongoengine import connection def my_view(request): objects = connection.do_something() BAD

# settings.py DATABASES = { "default": { "ENGINE": "django_mongo", } } # models.py from django.db import models class MyPerfectlyNormalModel(models.Model): name = models.CharField(max_length=12) GOOD

Why do we care? Admin Forms Serializers Model validation API Generators Metadata Makes my brain hurt less

Into the rabbit hole we go!

Lay of the land Models Managers QuerySets Queries Compilers Backends

Models from django.db import models class Category(models.Model): name = models.CharField(max_length=100) slug = models.SlugField() parent = models.ForeignKey("self", null=True)

Managers Category.objects

QuerySets Category.objects.get_query_set()

Queries Category.objects.get_query_set().query

Compilers qs = Category.objects.get_query_set() qs.query.get_compiler(qs.db)

QuerySets The whole damned thing

QuerySet This is the top layer of query state. From here on out it’s like an onion. Not backend speciﬁc. _db _result_cache _iter query

Query Right now this holds all state for a query. It’s semi-backend speciﬁc. Right now there’s one, and it’s speciﬁc to SQL backends. Computes most JOINS, aggregates, etc. Translates Q objects into Where objects.

The Query Problem It’s something of a lossy translation. Translating ﬁlter(), values(), and other calls into internal datastructures is lossless with respect to the database being used, but not with respect to other databases. If you’ve got all SQL databases you’re ﬁne, but if you mix in a non-relational DB you’ve got problems. More on this later.

SQLCompiler Takes a Query and a connection and turns it into SQL (and executes it). This also does some computation of joins (for select_related). This is the totally backend speciﬁc part (only part that knows about the actual connection and database). The rest of the chain just *assumes* a SQL db.

django.db.backends.* This is where backends live. Not super exciting. A bunch of ﬂags and methods to control very small parts of SQL creation. Also introspection, creation, and shell.

You call methods on a QuerySet Which calls methods on a Query Which mutates some datastructures You evaluate a QuerySet Which asks it’s Query for a Compiler Which generates some SQL Which calls some methods on the backend Which gets a cursor and evaluates it

The Problem

Query thinks in terms of SQL It chooses between join types it generates table aliases it splits ﬁlters between HAVING and WHERE and probably some other stuff

Why is this an issue How do I ask a MongoDBCompiler to compile a LEFT OUTER JOIN vs. an INNER JOIN? Or a HAVING vs a WHERE? These concepts don’t map cleanly, so the translation is lossy across backends

Design Decisions Not everything is a technical problem

Do we emulate JOINs? Category.objects.filter(parent__parent__name="Tech")

Do we maintain secondary indices? Category.objects.filter(name="Tech")

Different databases have different features True of SQL databases, but more so for non-relational databases. No lingua franca like SQL is.

A solution Or something close enough...

Make Query do less Instead of generating two trees of WHERE and HAVING, generate a single tree of ﬁlters. Don’t generate JOINs at all. Push that all down to the compiler.

Make SQLCompiler do more Generate all JOINs Split filter tree into HAVING vs WHERE Can generate more efficient JOINs with global knowledge. Probably makes it easier to do fix some other ORM bugs.

Plan of action Change the ORM up Build MongoDB prototype backend ??? Proﬁt

http://alexgaynor.net/ Slides will be up there

http://www.slideshare.net	999
http://blog.chedushi.com	15
http://kiss-emacs.blogspot.com	10
http://facebook.slideshare.com	4
http://m.andric.us	2
http://www.iweb34.com	2
http://kiss-emacs.blogspot.ca	2
http://www.taaza.com	1
http://webcache.googleusercontent.com	1
http://kiss-emacs.blogspot.de	1

Making Django and NoSQL Play Nice

by Alex Gaynor on May 24, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

10 Embeds 1,037

Statistics

Making Django and NoSQL Play Nice — Presentation Transcript