0
Pro ORM
Alex Gaynor
@alex_gaynor




               Django NYC
What is this talk?
  ORM Architecture   Practical Things




        50%          50%
ORM Architecture
QuerySet-Refactor
• A big refactor of the ORM internals
• Happened a bit before Django 1.0

   Old                      New
Multi DB
• GSOC project this past summer
• Added multiple database support to
  Django

• Ripped up a lot of internals, and then
  ...
from django.db import models

class MyModel(models.Model):
    pass



                               MyModel.objects




...
class Manager(object):
    def get_query_set(self):
       return QuerySet(self.model)

    def get(self, *args, **kwargs)...
QuerySet
• Backend agnostic
• Basically it handles turning the public
  API into calls to methods on Query

• Surprisingly...
Query
• This part is backend specific, SQL vs.
  GAE here, not Postgres vs. Oracle
• It carries all the info around, ordering,
  ...
self.model = model
self.alias_refcount = {}
self.alias_map = {}
self.table_map = {}
self.join_map = {}
self.rev_join_map =...
self.order_by = []
self.low_mark, self.high_mark = 0, None
self.distinct = False
self.select_related = False
self.related_...
I told you it was a lot of
          stuff
The Major Players
   Attributes            Methods
   self.where       self.get_compiler()
  self.having          self.clo...
Dude, where’s my
         SQL?
There are no relevant images for SQL, so
you get to settle for me trying to be funny
SQLCompiler

• Takes Querys and turns ‘em into SQL.
• Also executes SQL.
• This handles the backend specific
  stuff.

• We...
31 Flavors!

  SQLCompiler        SQLInsertCompiler

SQLDeleteCompiler     SQLDateCompiler

SQLUpdateCompiler   SQLAggrega...
django.db.backends.*
• Interoperate with    • Running the cmd line
  database drivers       shell

• Handle most of the   ...
Putting this Stuff to
     Good Use
aka the part you might use at your day
                 job
Custom Aggregates
• Django comes with a few aggregates
                     django.db.models.aggregates
                                Avg
...
Aggregates Come in Two
        Parts
• One part is a data carrier, it knows
  what field you’re aggregating over.
• The oth...
Part I
from django.db.models import Aggregate


class MyAggregate(Aggregate):
    def add_to_query(self, query, alias, col...
Part II
from django.db.models.sql.aggregates import 
    Aggregate as SQLAggregate


class SQLMyAggregate(SQLAggregate):
 ...
Automatic Caching
Single Object Caching
• Basically we want to automatically
  cache any get() lookups on unique
  fields (primary key, slugs...
from django.core.cache import cache
from django.db.models.query import QuerySet


class CachingQuerySet(QuerySet):
    def...
from django.db.models.signals import pre_save, pre_delete


class CachingManager(QuerySet):
    use_for_related_fields = T...
def invalide_cache(instance, sender, **kwargs):
    cache_key = "%s:%s:%s" % (
        instance._meta.app_label,
        i...
Questions?
Extra
Custom Field Review
Methods

                          Converts value from
   to_python
                        serialized form to Python

   ...
Django Pro ORM
Django Pro ORM
Django Pro ORM
Django Pro ORM
Upcoming SlideShare
Loading in...5
×

Django Pro ORM

5,498

Published on

Published in: Technology
0 Comments
21 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
5,498
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
186
Comments
0
Likes
21
Embeds 0
No embeds

No notes for slide






































  • Transcript of "Django Pro ORM"

    1. 1. Pro ORM Alex Gaynor @alex_gaynor Django NYC
    2. 2. What is this talk? ORM Architecture Practical Things 50% 50%
    3. 3. ORM Architecture
    4. 4. QuerySet-Refactor
    5. 5. • A big refactor of the ORM internals • Happened a bit before Django 1.0 Old New
    6. 6. Multi DB
    7. 7. • GSOC project this past summer • Added multiple database support to Django • Ripped up a lot of internals, and then put them back together Old New
    8. 8. from django.db import models class MyModel(models.Model): pass MyModel.objects Managers
    9. 9. class Manager(object): def get_query_set(self): return QuerySet(self.model) def get(self, *args, **kwargs): return self.get_query_set().get(*args, **kwargs) def filter(self, *args, **kwargs): return self.get_query_set().filter(*args, **kwargs) # ETC... get_query_set(), your gateway to the rabbit
    10. 10. QuerySet • Backend agnostic • Basically it handles turning the public API into calls to methods on Query • Surprisingly little meat for a 1400 line file • Also, a few subclasses for values(), values_list(), and dates()
    11. 11. Query
    12. 12. • This part is backend specific, SQL vs. GAE here, not Postgres vs. Oracle • It carries all the info around, ordering, joins, where, aggregates, etc... • It used to generate SQL, not anymore (more on this later)
    13. 13. self.model = model self.alias_refcount = {} self.alias_map = {} self.table_map = {} self.join_map = {} self.rev_join_map = {} self.quote_cache = {} self.default_cols = True self.default_ordering = True self.standard_ordering = True self.ordering_aliases = [] self.select_fields = [] self.related_select_fields = [] self.dupe_avoidance = {} self.used_aliases = set() self.filter_is_sticky = False self.included_inherited_models = {} self.select = [] self.tables = [] self.where = where() self.where_class = where self.group_by = None self.having = where()
    14. 14. self.order_by = [] self.low_mark, self.high_mark = 0, None self.distinct = False self.select_related = False self.related_select_cols = [] self.aggregates = SortedDict() self.aggregate_select_mask = None self._aggregate_select_cache = None self.max_depth = 5 self.extra = SortedDict() self.extra_select_mask = None self._extra_select_cache = None self.extra_tables = () self.extra_order_by = () self.deferred_loading = (set(), True)
    15. 15. I told you it was a lot of stuff
    16. 16. The Major Players Attributes Methods self.where self.get_compiler() self.having self.clone() self.aggregates self.join() self.alias_map self.add_filter() self.select_fields self.add_ordering()
    17. 17. Dude, where’s my SQL? There are no relevant images for SQL, so you get to settle for me trying to be funny
    18. 18. SQLCompiler • Takes Querys and turns ‘em into SQL. • Also executes SQL. • This handles the backend specific stuff. • We ship 2 sets of them. 1 for Oracle, and one for pretty much everything else.
    19. 19. 31 Flavors! SQLCompiler SQLInsertCompiler SQLDeleteCompiler SQLDateCompiler SQLUpdateCompiler SQLAggregateCompiler
    20. 20. django.db.backends.* • Interoperate with • Running the cmd line database drivers shell • Handle most of the • django.contrib.gis.db. nuances of the backends various databases’ SQL dialects • DDL for creation • Introspection
    21. 21. Putting this Stuff to Good Use aka the part you might use at your day job
    22. 22. Custom Aggregates
    23. 23. • Django comes with a few aggregates django.db.models.aggregates Avg • But databases have other Count aggregates Max • Some of them Min even let you StdDev create your own! Sum Variance
    24. 24. Aggregates Come in Two Parts • One part is a data carrier, it knows what field you’re aggregating over. • The other part turns that into SQL. • Starting to sound familiar?
    25. 25. Part I from django.db.models import Aggregate class MyAggregate(Aggregate): def add_to_query(self, query, alias, col, course, is_summary): aggregate = SQLMyAggregate(col, source=source, is_summary=is_summary, **extra) query.aggregates[alias] = aggregate
    26. 26. Part II from django.db.models.sql.aggregates import Aggregate as SQLAggregate class SQLMyAggregate(SQLAggregate): sql_template = "%(function)(cast(%(field)s as numeric) / 100)" sql_function = "MYFUNCTION" is_ordinal = True
    27. 27. Automatic Caching
    28. 28. Single Object Caching • Basically we want to automatically cache any get() lookups on unique fields (primary key, slugs, etc.) • These types of queries seem to pop up a lot (think every single foreign key traversal) • Also automatically do invalidation for us.
    29. 29. from django.core.cache import cache from django.db.models.query import QuerySet class CachingQuerySet(QuerySet): def get(self, *args, **kwargs): sup = lambda: super(CachingQuerySet, self). get(*args, **kwargs) if len(args) != 1 or not kwargs or self.query.where: return sup() key, value = kwargs.iteritems().next() if key.endswith("__exact"): key = key[:-len("__exact")] if key not in ["pk", self.model._meta.pk.name]: return sup() cache_key = "%s:%s:%s" % ( self.model._meta.app_label, self.model._meta.object_name, value ) obj = cache.get(cache_key) if obj is not None: return obj obj = sup() cache.set(cache_key, obj) return obj
    30. 30. from django.db.models.signals import pre_save, pre_delete class CachingManager(QuerySet): use_for_related_fields = True def get_query_set(self): return CachingQuerySet(self.model) def contribute_to_class(self, *args, **kwargs): super(CachingManager, self). contribute_to_class(*args, **kwargs) pre_save.connect(invalidate_cache, self.model) pre_delete.connect(invalidate_cache, self.model)
    31. 31. def invalide_cache(instance, sender, **kwargs): cache_key = "%s:%s:%s" % ( instance._meta.app_label, instance._meta.object_name, instance.pk ) cache.delete(cache_key) Aaand, Done!
    32. 32. Questions?
    33. 33. Extra Custom Field Review
    34. 34. Methods Converts value from to_python serialized form to Python validate Performs validation db_type Returns the database type Performs DB agnostic get_prep_value coercion and validation DB specific coercion/ get_db_prep_value validation formfield Provides a form field
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×