Django and working with large database tables

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Django and working with
large database tables
Django Stockholm Meetup Group
March 30, 2017

About me
Ilian Iliev
Platform Engineer at Lifesum
ilian@ilian.io
www.ilian.io

The setup
2.5GHz i7, 16GB Ram, MacBook Pro
Django 1.10
MySQL 5.7.14
PostgreSQL 9.5.4

The Models
class Tag(models.Model):
name = models.CharField(max_length=255)
class User(models.Model):
name = models.CharFi...

The Change
class Message(models.Model):
sender = models.ForeignKey(User, related_name='sent_messages')
receiver = models.F...

The weird migration
ALTER TABLE `big_tables_message_tags` DROP FOREIGN KEY
`big_tables_message_tags_tag_id_5eb6034e_fk_big...

MySQL
Rows ~ 2.7M
Size ~ 88MB
message_id index size ~ 48MB
tags_id index size ~ 61MB
Migration time ~ 41 sec

The weird migration
ALTER TABLE "big_tables_message_tags" DROP CONSTRAINT
"big_tables_message_tags_tag_id_5eb6034e_fk_big_...

PostgreSQL
Rows ~ 2.8M
Size ~ 83MB
message_id index size ~ 77MB
tags_id index size ~ 119MB
Migration time ~ 3.2 sec

Modify the migration that created the field and add the change there
* It is a know issue https://code.djangoproject.com/t...

class MessagesTags(models.Model):
message = models.ForeignKey(Message)
tag = models.ForeignKey(Tag)
added_by = models.Fore...

MySQL: 31 sec
PostgreSQL: 5.3 sec
Timing

MySQL INPLACE
ALTER TABLE `big_tables_message_tags` ADD COLUMN `added_by_id`
integer NULL, ALGORITHM INPLACE, LOCK NONE;
A...

Running this on prod
Running in on prod resulted in the API crashing
Non locking query but still too heavy for the DB
Auro...

Alternative
class MessagesTagsExtend(models.Model):
STATUS_PENDING_REVIEW = 0
STATUS_APPROVED = 10
DEFAULT_STATUS = STATUS...

Alternative
class MessagesTags(models.Model):
...
@property
def status(self):
try:
return self.messagestagsextend.status
e...

Iterating on big tables
for x in MessagesTags.objects.all():
print x
+ Single SQL query
- Loads everything in memory

Iterating on big tables
for x in MessagesTags.objects.iterator():
print x
+ Single SQL query
+ Loads pieces of the result ...

Questions?

You are reading a preview.

Activate your 30 day free trial to continue reading.

Share SlideShare

Facebook
Twitter
LinkedIn

Embed

Size (px)

Show related SlideShares at end

WordPress Shortcode

Link

LinkedIn
Facebook
Twitter
Email

Replay

Top clipped slide

1 of 19 Ad

1 of 19 Ad