Pulling more out of Django application Supreet Sethi Email < [email_address] This talk is possible because of kindness of some clients, who let me use their machinery for my utterly wierd experiments

Supreet Sethi

Email < [email_address]

This talk is possible because of kindness of some clients, who let me use their machinery for my utterly wierd experiments

Perspective on databases Strictly structured data Sophisticated relation description language Depends heavily on locality of data and indexing

Strictly structured data

Sophisticated relation description language

Depends heavily on locality of data and indexing

Perspective on databases 2 ORM see database as storage ORMs are less sophisticated in mapping relations 1-1 mapping is sometimes not possible Atomicity of operations cannot of insured

ORM see database as storage

ORMs are less sophisticated in mapping relations

1-1 mapping is sometimes not possible

Atomicity of operations cannot of insured

Development cycle with databases Migraine Designing schema Programming business logic Adding fixtures

With Django Code on diet Model Templates View Model serves as one point place for data schema and business logic Url patterns

Understanding Django application stack Server Database Layer User Interface Business Logic Common Storage

Problem with the stack Tightly coupled

General overview of django scaling Using javascript to load as much dynamic data making static content in pages cacheable (AJAX) Compress applications CSS and javascripts (django-compress) Optimize pagination Connection pooling Memory Caching

Using javascript to load as much dynamic data making static content in pages cacheable (AJAX)

Compress applications CSS and javascripts (django-compress)

Optimize pagination

Connection pooling

Memory Caching

Database scaling for django application Schema design and modeling Debug queries Data specific changes in the model Database architecture for scaling

Schema design and modeling

Debug queries

Data specific changes in the model

Database architecture for scaling

Schema design and modeling Using appropriate field type CharField for small text DateField for date and time Text, Integer, Slug, Email, Url, Image, File Using hinting whereever possible or required Define primary_key for primary key field Map relations well Use choice modifier BooleanField Unique modifier

Using appropriate field type

CharField for small text

DateField for date and time

Text, Integer, Slug, Email, Url, Image, File

Using hinting whereever possible or required

Define primary_key for primary key field

Map relations well

Use choice modifier

BooleanField

Unique modifier

Class Pizza(models.Model): pizza_type = models.CharField(choices=(('ch','chicken'), ('pr':'pepperoni'),) crust_type = models.CharField(choices=(('hr','hard'), ('fil', 'filled'),) topping_type = models.CharField(choices=(('fr', 'farm fresh veggies', ('hu','hunk of meat'),) date_of_order = models.DateTimeField(primary_key=true) Class Pizza(models.Model): pizza_type = models.CharField() crust_type = models.CharField() topping_type = models.CharField() date_of_order = models.CharField() Wrong Right

Debug the queries http://github.com/robhudson/django-debug-toolbar/tree/master

Data specific changes in the model db_index modifier on fields Be sure to use them carefully Increases the write costs on the field Helps in seeking data faster Unique constraint with unique modifier Recheck relations between tables

db_index modifier on fields

Be sure to use them carefully

Increases the write costs on the field

Helps in seeking data faster

Unique constraint with unique modifier

Recheck relations between tables

Database architecture for scaling (1) Loadbalancing Application is unaware of the scaling logic Database can be loaded using django admin Normalization of related tables is not required Its scalable linearly Orders 2007 Load balancer Application Server 3 Orders 2007 Orders 2007 Orders 2007 Application Server 1 Application Server 2

Application is unaware of the scaling logic

Database can be loaded using django admin

Normalization of related tables is not required

Its scalable linearly

Database architecture for scaling (2) Partitioning Transparent to django application Database cannot be loaded using django admin Normalization of related tables does not change Its scalable linearly upto database limits Pizza orders database Order 2007 Orders 2009 Orders 2008 Application Layer

Transparent to django application

Database cannot be loaded using django admin

Normalization of related tables does not change

Its scalable linearly upto database limits

Database architecture for scaling (3) Sharding Application should be aware of the sharding logic Database cannot be loaded using django admin Normalization of related tables is required Its scalable linearly Orders 2007 Application Layer Sharding logic Orders 2007 Orders 2007 Orders 2007

Application should be aware of the sharding logic

Database cannot be loaded using django admin

Normalization of related tables is required

Its scalable linearly

Example of sharding http://www.eflorenzano.com/blog/tag/database-sharding/

Where not to use django Where atomicity of operation is required Where data needs to be updated on tables and pulled using a view Complex schemas

Where atomicity of operation is required

Where data needs to be updated on tables and pulled using a view

Complex schemas