Django Database Optimization


Published on

Talk delivered at 2009

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Django Database Optimization

  1. 1. Pulling more out of Django application <ul><li>Supreet Sethi </li></ul><ul><li>Email < [email_address] </li></ul><ul><li>This talk is possible because of kindness of some clients, who let me use their machinery for my utterly wierd experiments </li></ul>
  2. 2. Perspective on databases <ul><li>Strictly structured data </li></ul><ul><li>Sophisticated relation description language </li></ul><ul><li>Depends heavily on locality of data and indexing </li></ul>
  3. 3. Perspective on databases 2 <ul><li>ORM see database as storage </li></ul><ul><li>ORMs are less sophisticated in mapping relations </li></ul><ul><li>1-1 mapping is sometimes not possible </li></ul><ul><li>Atomicity of operations cannot of insured </li></ul>
  4. 4. Development cycle with databases Migraine Designing schema Programming business logic Adding fixtures
  5. 5. With Django Code on diet Model Templates View Model serves as one point place for data schema and business logic Url patterns
  6. 6. Understanding Django application stack Server Database Layer User Interface Business Logic Common Storage
  7. 7. Problem with the stack Tightly coupled
  8. 8. General overview of django scaling <ul><li>Using javascript to load as much dynamic data making static content in pages cacheable (AJAX) </li></ul><ul><li>Compress applications CSS and javascripts (django-compress) </li></ul><ul><li>Optimize pagination </li></ul><ul><li>Connection pooling </li></ul><ul><li>Memory Caching </li></ul>
  9. 9. Database scaling for django application <ul><li>Schema design and modeling </li></ul><ul><li>Debug queries </li></ul><ul><li>Data specific changes in the model </li></ul><ul><li>Database architecture for scaling </li></ul>
  10. 10. Schema design and modeling <ul><li>Using appropriate field type </li></ul><ul><ul><li>CharField for small text </li></ul></ul><ul><ul><li>DateField for date and time </li></ul></ul><ul><ul><li>Text, Integer, Slug, Email, Url, Image, File </li></ul></ul><ul><li>Using hinting whereever possible or required </li></ul><ul><ul><li>Define primary_key for primary key field </li></ul></ul><ul><ul><li>Map relations well </li></ul></ul><ul><ul><li>Use choice modifier </li></ul></ul><ul><ul><li>BooleanField </li></ul></ul><ul><ul><li>Unique modifier </li></ul></ul>
  11. 11. Class Pizza(models.Model): pizza_type = models.CharField(choices=(('ch','chicken'), ('pr':'pepperoni'),) crust_type = models.CharField(choices=(('hr','hard'), ('fil', 'filled'),) topping_type = models.CharField(choices=(('fr', 'farm fresh veggies', ('hu','hunk of meat'),) date_of_order = models.DateTimeField(primary_key=true) Class Pizza(models.Model): pizza_type = models.CharField() crust_type = models.CharField() topping_type = models.CharField() date_of_order = models.CharField() Wrong Right
  12. 12. Debug the queries
  13. 13. Data specific changes in the model <ul><li>db_index modifier on fields </li></ul><ul><ul><li>Be sure to use them carefully </li></ul></ul><ul><ul><li>Increases the write costs on the field </li></ul></ul><ul><ul><li>Helps in seeking data faster </li></ul></ul><ul><li>Unique constraint with unique modifier </li></ul><ul><li>Recheck relations between tables </li></ul>
  14. 14. Database architecture for scaling (1) Loadbalancing <ul><li>Application is unaware of the scaling logic </li></ul><ul><li>Database can be loaded using django admin </li></ul><ul><li>Normalization of related tables is not required </li></ul><ul><li>Its scalable linearly </li></ul>Orders 2007 Load balancer Application Server 3 Orders 2007 Orders 2007 Orders 2007 Application Server 1 Application Server 2
  15. 15. Database architecture for scaling (2) Partitioning <ul><li>Transparent to django application </li></ul><ul><li>Database cannot be loaded using django admin </li></ul><ul><li>Normalization of related tables does not change </li></ul><ul><li>Its scalable linearly upto database limits </li></ul>Pizza orders database Order 2007 Orders 2009 Orders 2008 Application Layer
  16. 16. Database architecture for scaling (3) Sharding <ul><li>Application should be aware of the sharding logic </li></ul><ul><li>Database cannot be loaded using django admin </li></ul><ul><li>Normalization of related tables is required </li></ul><ul><li>Its scalable linearly </li></ul>Orders 2007 Application Layer Sharding logic Orders 2007 Orders 2007 Orders 2007
  17. 17. Example of sharding
  18. 18. Where not to use django <ul><li>Where atomicity of operation is required </li></ul><ul><li>Where data needs to be updated on tables and pulled using a view </li></ul><ul><li>Complex schemas </li></ul>