Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Django Pro ORM

6,693 views

Published on

Published in: Technology
  • Login to see the comments

Django Pro ORM

  1. 1. Pro ORM Alex Gaynor @alex_gaynor Django NYC
  2. 2. What is this talk? ORM Architecture Practical Things 50% 50%
  3. 3. ORM Architecture
  4. 4. QuerySet-Refactor
  5. 5. • A big refactor of the ORM internals • Happened a bit before Django 1.0 Old New
  6. 6. Multi DB
  7. 7. • GSOC project this past summer • Added multiple database support to Django • Ripped up a lot of internals, and then put them back together Old New
  8. 8. from django.db import models class MyModel(models.Model): pass MyModel.objects Managers
  9. 9. class Manager(object): def get_query_set(self): return QuerySet(self.model) def get(self, *args, **kwargs): return self.get_query_set().get(*args, **kwargs) def filter(self, *args, **kwargs): return self.get_query_set().filter(*args, **kwargs) # ETC... get_query_set(), your gateway to the rabbit
  10. 10. QuerySet • Backend agnostic • Basically it handles turning the public API into calls to methods on Query • Surprisingly little meat for a 1400 line file • Also, a few subclasses for values(), values_list(), and dates()
  11. 11. Query
  12. 12. • This part is backend specific, SQL vs. GAE here, not Postgres vs. Oracle • It carries all the info around, ordering, joins, where, aggregates, etc... • It used to generate SQL, not anymore (more on this later)
  13. 13. self.model = model self.alias_refcount = {} self.alias_map = {} self.table_map = {} self.join_map = {} self.rev_join_map = {} self.quote_cache = {} self.default_cols = True self.default_ordering = True self.standard_ordering = True self.ordering_aliases = [] self.select_fields = [] self.related_select_fields = [] self.dupe_avoidance = {} self.used_aliases = set() self.filter_is_sticky = False self.included_inherited_models = {} self.select = [] self.tables = [] self.where = where() self.where_class = where self.group_by = None self.having = where()
  14. 14. self.order_by = [] self.low_mark, self.high_mark = 0, None self.distinct = False self.select_related = False self.related_select_cols = [] self.aggregates = SortedDict() self.aggregate_select_mask = None self._aggregate_select_cache = None self.max_depth = 5 self.extra = SortedDict() self.extra_select_mask = None self._extra_select_cache = None self.extra_tables = () self.extra_order_by = () self.deferred_loading = (set(), True)
  15. 15. I told you it was a lot of stuff
  16. 16. The Major Players Attributes Methods self.where self.get_compiler() self.having self.clone() self.aggregates self.join() self.alias_map self.add_filter() self.select_fields self.add_ordering()
  17. 17. Dude, where’s my SQL? There are no relevant images for SQL, so you get to settle for me trying to be funny
  18. 18. SQLCompiler • Takes Querys and turns ‘em into SQL. • Also executes SQL. • This handles the backend specific stuff. • We ship 2 sets of them. 1 for Oracle, and one for pretty much everything else.
  19. 19. 31 Flavors! SQLCompiler SQLInsertCompiler SQLDeleteCompiler SQLDateCompiler SQLUpdateCompiler SQLAggregateCompiler
  20. 20. django.db.backends.* • Interoperate with • Running the cmd line database drivers shell • Handle most of the • django.contrib.gis.db. nuances of the backends various databases’ SQL dialects • DDL for creation • Introspection
  21. 21. Putting this Stuff to Good Use aka the part you might use at your day job
  22. 22. Custom Aggregates
  23. 23. • Django comes with a few aggregates django.db.models.aggregates Avg • But databases have other Count aggregates Max • Some of them Min even let you StdDev create your own! Sum Variance
  24. 24. Aggregates Come in Two Parts • One part is a data carrier, it knows what field you’re aggregating over. • The other part turns that into SQL. • Starting to sound familiar?
  25. 25. Part I from django.db.models import Aggregate class MyAggregate(Aggregate): def add_to_query(self, query, alias, col, course, is_summary): aggregate = SQLMyAggregate(col, source=source, is_summary=is_summary, **extra) query.aggregates[alias] = aggregate
  26. 26. Part II from django.db.models.sql.aggregates import Aggregate as SQLAggregate class SQLMyAggregate(SQLAggregate): sql_template = "%(function)(cast(%(field)s as numeric) / 100)" sql_function = "MYFUNCTION" is_ordinal = True
  27. 27. Automatic Caching
  28. 28. Single Object Caching • Basically we want to automatically cache any get() lookups on unique fields (primary key, slugs, etc.) • These types of queries seem to pop up a lot (think every single foreign key traversal) • Also automatically do invalidation for us.
  29. 29. from django.core.cache import cache from django.db.models.query import QuerySet class CachingQuerySet(QuerySet): def get(self, *args, **kwargs): sup = lambda: super(CachingQuerySet, self). get(*args, **kwargs) if len(args) != 1 or not kwargs or self.query.where: return sup() key, value = kwargs.iteritems().next() if key.endswith("__exact"): key = key[:-len("__exact")] if key not in ["pk", self.model._meta.pk.name]: return sup() cache_key = "%s:%s:%s" % ( self.model._meta.app_label, self.model._meta.object_name, value ) obj = cache.get(cache_key) if obj is not None: return obj obj = sup() cache.set(cache_key, obj) return obj
  30. 30. from django.db.models.signals import pre_save, pre_delete class CachingManager(QuerySet): use_for_related_fields = True def get_query_set(self): return CachingQuerySet(self.model) def contribute_to_class(self, *args, **kwargs): super(CachingManager, self). contribute_to_class(*args, **kwargs) pre_save.connect(invalidate_cache, self.model) pre_delete.connect(invalidate_cache, self.model)
  31. 31. def invalide_cache(instance, sender, **kwargs): cache_key = "%s:%s:%s" % ( instance._meta.app_label, instance._meta.object_name, instance.pk ) cache.delete(cache_key) Aaand, Done!
  32. 32. Questions?
  33. 33. Extra Custom Field Review
  34. 34. Methods Converts value from to_python serialized form to Python validate Performs validation db_type Returns the database type Performs DB agnostic get_prep_value coercion and validation DB specific coercion/ get_db_prep_value validation formfield Provides a form field

×