Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Pro ORM
Alex Gaynor
@alex_gaynor




               Django NYC
What is this talk?
  ORM Architecture   Practical Things




        50%          50%
ORM Architecture
QuerySet-Refactor
• A big refactor of the ORM internals
• Happened a bit before Django 1.0

   Old                      New
Multi DB
• GSOC project this past summer
• Added multiple database support to
  Django

• Ripped up a lot of internals, and then
  ...
from django.db import models

class MyModel(models.Model):
    pass



                               MyModel.objects




...
class Manager(object):
    def get_query_set(self):
       return QuerySet(self.model)

    def get(self, *args, **kwargs)...
QuerySet
• Backend agnostic
• Basically it handles turning the public
  API into calls to methods on Query

• Surprisingly...
Query
• This part is backend specific, SQL vs.
  GAE here, not Postgres vs. Oracle
• It carries all the info around, ordering,
  ...
self.model = model
self.alias_refcount = {}
self.alias_map = {}
self.table_map = {}
self.join_map = {}
self.rev_join_map =...
self.order_by = []
self.low_mark, self.high_mark = 0, None
self.distinct = False
self.select_related = False
self.related_...
I told you it was a lot of
          stuff
The Major Players
   Attributes            Methods
   self.where       self.get_compiler()
  self.having          self.clo...
Dude, where’s my
         SQL?
There are no relevant images for SQL, so
you get to settle for me trying to be funny
SQLCompiler

• Takes Querys and turns ‘em into SQL.
• Also executes SQL.
• This handles the backend specific
  stuff.

• We...
31 Flavors!

  SQLCompiler        SQLInsertCompiler

SQLDeleteCompiler     SQLDateCompiler

SQLUpdateCompiler   SQLAggrega...
django.db.backends.*
• Interoperate with    • Running the cmd line
  database drivers       shell

• Handle most of the   ...
Putting this Stuff to
     Good Use
aka the part you might use at your day
                 job
Custom Aggregates
• Django comes with a few aggregates
                     django.db.models.aggregates
                                Avg
...
Aggregates Come in Two
        Parts
• One part is a data carrier, it knows
  what field you’re aggregating over.
• The oth...
Part I
from django.db.models import Aggregate


class MyAggregate(Aggregate):
    def add_to_query(self, query, alias, col...
Part II
from django.db.models.sql.aggregates import 
    Aggregate as SQLAggregate


class SQLMyAggregate(SQLAggregate):
 ...
Automatic Caching
Single Object Caching
• Basically we want to automatically
  cache any get() lookups on unique
  fields (primary key, slugs...
from django.core.cache import cache
from django.db.models.query import QuerySet


class CachingQuerySet(QuerySet):
    def...
from django.db.models.signals import pre_save, pre_delete


class CachingManager(QuerySet):
    use_for_related_fields = T...
def invalide_cache(instance, sender, **kwargs):
    cache_key = "%s:%s:%s" % (
        instance._meta.app_label,
        i...
Questions?
Extra
Custom Field Review
Methods

                          Converts value from
   to_python
                        serialized form to Python

   ...
Django Pro ORM
Django Pro ORM
Django Pro ORM
Django Pro ORM
Upcoming SlideShare
Loading in …5
×

Django Pro ORM

6,437 views

Published on

Published in: Technology
  • Be the first to comment

Django Pro ORM

  1. 1. Pro ORM Alex Gaynor @alex_gaynor Django NYC
  2. 2. What is this talk? ORM Architecture Practical Things 50% 50%
  3. 3. ORM Architecture
  4. 4. QuerySet-Refactor
  5. 5. • A big refactor of the ORM internals • Happened a bit before Django 1.0 Old New
  6. 6. Multi DB
  7. 7. • GSOC project this past summer • Added multiple database support to Django • Ripped up a lot of internals, and then put them back together Old New
  8. 8. from django.db import models class MyModel(models.Model): pass MyModel.objects Managers
  9. 9. class Manager(object): def get_query_set(self): return QuerySet(self.model) def get(self, *args, **kwargs): return self.get_query_set().get(*args, **kwargs) def filter(self, *args, **kwargs): return self.get_query_set().filter(*args, **kwargs) # ETC... get_query_set(), your gateway to the rabbit
  10. 10. QuerySet • Backend agnostic • Basically it handles turning the public API into calls to methods on Query • Surprisingly little meat for a 1400 line file • Also, a few subclasses for values(), values_list(), and dates()
  11. 11. Query
  12. 12. • This part is backend specific, SQL vs. GAE here, not Postgres vs. Oracle • It carries all the info around, ordering, joins, where, aggregates, etc... • It used to generate SQL, not anymore (more on this later)
  13. 13. self.model = model self.alias_refcount = {} self.alias_map = {} self.table_map = {} self.join_map = {} self.rev_join_map = {} self.quote_cache = {} self.default_cols = True self.default_ordering = True self.standard_ordering = True self.ordering_aliases = [] self.select_fields = [] self.related_select_fields = [] self.dupe_avoidance = {} self.used_aliases = set() self.filter_is_sticky = False self.included_inherited_models = {} self.select = [] self.tables = [] self.where = where() self.where_class = where self.group_by = None self.having = where()
  14. 14. self.order_by = [] self.low_mark, self.high_mark = 0, None self.distinct = False self.select_related = False self.related_select_cols = [] self.aggregates = SortedDict() self.aggregate_select_mask = None self._aggregate_select_cache = None self.max_depth = 5 self.extra = SortedDict() self.extra_select_mask = None self._extra_select_cache = None self.extra_tables = () self.extra_order_by = () self.deferred_loading = (set(), True)
  15. 15. I told you it was a lot of stuff
  16. 16. The Major Players Attributes Methods self.where self.get_compiler() self.having self.clone() self.aggregates self.join() self.alias_map self.add_filter() self.select_fields self.add_ordering()
  17. 17. Dude, where’s my SQL? There are no relevant images for SQL, so you get to settle for me trying to be funny
  18. 18. SQLCompiler • Takes Querys and turns ‘em into SQL. • Also executes SQL. • This handles the backend specific stuff. • We ship 2 sets of them. 1 for Oracle, and one for pretty much everything else.
  19. 19. 31 Flavors! SQLCompiler SQLInsertCompiler SQLDeleteCompiler SQLDateCompiler SQLUpdateCompiler SQLAggregateCompiler
  20. 20. django.db.backends.* • Interoperate with • Running the cmd line database drivers shell • Handle most of the • django.contrib.gis.db. nuances of the backends various databases’ SQL dialects • DDL for creation • Introspection
  21. 21. Putting this Stuff to Good Use aka the part you might use at your day job
  22. 22. Custom Aggregates
  23. 23. • Django comes with a few aggregates django.db.models.aggregates Avg • But databases have other Count aggregates Max • Some of them Min even let you StdDev create your own! Sum Variance
  24. 24. Aggregates Come in Two Parts • One part is a data carrier, it knows what field you’re aggregating over. • The other part turns that into SQL. • Starting to sound familiar?
  25. 25. Part I from django.db.models import Aggregate class MyAggregate(Aggregate): def add_to_query(self, query, alias, col, course, is_summary): aggregate = SQLMyAggregate(col, source=source, is_summary=is_summary, **extra) query.aggregates[alias] = aggregate
  26. 26. Part II from django.db.models.sql.aggregates import Aggregate as SQLAggregate class SQLMyAggregate(SQLAggregate): sql_template = "%(function)(cast(%(field)s as numeric) / 100)" sql_function = "MYFUNCTION" is_ordinal = True
  27. 27. Automatic Caching
  28. 28. Single Object Caching • Basically we want to automatically cache any get() lookups on unique fields (primary key, slugs, etc.) • These types of queries seem to pop up a lot (think every single foreign key traversal) • Also automatically do invalidation for us.
  29. 29. from django.core.cache import cache from django.db.models.query import QuerySet class CachingQuerySet(QuerySet): def get(self, *args, **kwargs): sup = lambda: super(CachingQuerySet, self). get(*args, **kwargs) if len(args) != 1 or not kwargs or self.query.where: return sup() key, value = kwargs.iteritems().next() if key.endswith("__exact"): key = key[:-len("__exact")] if key not in ["pk", self.model._meta.pk.name]: return sup() cache_key = "%s:%s:%s" % ( self.model._meta.app_label, self.model._meta.object_name, value ) obj = cache.get(cache_key) if obj is not None: return obj obj = sup() cache.set(cache_key, obj) return obj
  30. 30. from django.db.models.signals import pre_save, pre_delete class CachingManager(QuerySet): use_for_related_fields = True def get_query_set(self): return CachingQuerySet(self.model) def contribute_to_class(self, *args, **kwargs): super(CachingManager, self). contribute_to_class(*args, **kwargs) pre_save.connect(invalidate_cache, self.model) pre_delete.connect(invalidate_cache, self.model)
  31. 31. def invalide_cache(instance, sender, **kwargs): cache_key = "%s:%s:%s" % ( instance._meta.app_label, instance._meta.object_name, instance.pk ) cache.delete(cache_key) Aaand, Done!
  32. 32. Questions?
  33. 33. Extra Custom Field Review
  34. 34. Methods Converts value from to_python serialized form to Python validate Performs validation db_type Returns the database type Performs DB agnostic get_prep_value coercion and validation DB specific coercion/ get_db_prep_value validation formfield Provides a form field

×