The document explores scaling multi-tenant applications using Django ORM and PostgreSQL, defining multi-tenancy and offering three solutions: one database per tenant, one schema per tenant, and shared tables architecture. It emphasizes the benefits and drawbacks of each approach, particularly focusing on the implementation of shared tables with Django-Multitenant for efficient tenant isolation. Additionally, it highlights the advantages of using PostgreSQL and the Citus extension for distributed architecture to enhance scalability.

1. Scaling Multi-Tenant Applications
Using the Django ORM & Postgres
Louise Grandjonc - Pycaribbean 2019
@louisemeta

2. About me
Software Engineer at Citus Data
Postgres enthusiast
@louisemeta and @citusdata on twitter
www.louisemeta.com
louise@citusdata.com
@louisemeta

3. Today’s agenda
1. What do we mean by “multi-tenancy”?
2. Three ways to scale a multi-tenant app
3. Shared tables in your Django apps
4. Postgres, citus and shared tables
@louisemeta

4. What do we mean by
“multi-tenancy”
@louisemeta

5. What do we mean by “multi-tenancy”
- Multiple customers (tenants)
- Each with their own data
- SaaS
- Example: shopify, salesforce
@louisemeta

6. What do we mean by “multi-tenancy”
A very realistic example
Owner examples:
- Hogwarts
- Ministry of Magic
- Harry Potter
- Post office
@louisemeta

7. What do we mean by “multi-tenancy”
The problem of scaling multi tenant apps
@louisemeta

8. 3 ways to scale
a multi-tenant app
@louisemeta

9. Solution 1:
One database per tenant
@louisemeta

10. - Organized collection of interrelated data
- Don’t share resources:
- Username and password
- Connections
- Memory
@louisemeta
One database per tenant

11. @louisemeta
One database per tenant

12. 1. Changing the settings
DATABASES = {
'tenant-{id1}': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'hogwarts',
'USER': 'louise',
'PASSWORD': ‘abc',
'HOST': '…',
'PORT': '5432'
},
'tenant-{id2}': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': ‘ministry',
'USER': 'louise',
'PASSWORD': 'abc',
'HOST': '…',
'PORT': '5432'
},
…
}
Warnings !
- You need to have each tenant in
the settings.
- When you have a new customer,
you need to create a database
and change the settings.
One database per tenant
@louisemeta

13. 2. Handling migrations
python manage.py migrate —database=tenant_id1;
For each tenant, when you have a new migration
One database per tenant
@louisemeta

14. Changes needed to handle it with Django ORM
3. Creating your own database router
class ExampleDatabaseRouter(object):
"""
Determines on which tenant database to read/write
"""
def db_for_read(self, model, **hints):
“”"Returns the name of the right database depending on the query"""
return ‘tenant_idx’
def db_for_write(self, model, **hints):
“”"Returns the name of the right database depending on the query"""
return ‘tenant_idx’
def allow_relation(self, obj1, obj2, **hints):
"""Determine if relationship is allowed between two objects.
The two objects have to be on the same database ;)”””
pass
One database per tenant
@louisemeta

15. PROS
- Start quickly
- Isolate customer (tenant) data
- Compliance is a bit easier
- If one customer is queried a lot,
performance degrade will be low
- Time for DBA/developer to manage
- Hard to handle with ORMs
- Maintain consistency
(ex: create index across all databases)
- Longer running migrations
- Performance degrades as # customers (tenants)
goes up
CONS
One database per tenant
@louisemeta

16. Solution 2:
One schema per tenant
@louisemeta

17. - Logical namespaces to hold a set of tables
- Share resources:
- Username and password
- Connections
- Memory
One schema per tenant
@louisemeta

18. One schema per tenant
@louisemeta

19. PROS
- Better resource utilization vs.
one database per tenant
- Start quickly
- Logical isolation
- Hard to manage (ex: add column across
all schemas)
- Longer running migrations
- Performance degrades as # customers
(tenants) goes up
CONS
One schema per tenant
@louisemeta

20. Solution 3:
Shared tables architecture
@louisemeta

21. Shared tables architecture
@louisemeta

22. Shared tables architecture
@louisemeta

23. PROS
- Easy maintenance
- Faster running migrations
- Best resource utilization
- Faster performance
- Scales to 1k-100k tenants
- Application code to guarantee isolation
- Make sure ORM calls are always scoped to
a single tenant
CONS
Shared tables architecture
@louisemeta

24. 3 ways to scale multi-tenant apps
@louisemeta

25. Shared tables in your Django apps
@louisemeta

26. Main problems to solve
- Make sure ORM calls are always scoped to a
single tenant
- Include the tenant column to joins
@louisemeta

27. django-multitenant
Automates all ORM calls
to be scoped to a single tenant
@louisemeta

28. django-multitenant
Owl.objects.filter(name=‘Hedwige’)
<=>
SELECT * from app_owl where name=‘Hedwige’
Owl.objects.filter(id=1)
<=>
SELECT * from app_owl
WHERE name=‘Hedwige’
AND owner_id = <tenant_id>

29. django-multitenant
Letter.objects.filter(id=1).select_related(‘deliverer_id’)
<=>
SELECT * from app_letter
INNER JOIN app_owl ON (app_owl.id=app_letter.deliverer_id)
WHERE app_letter.id=1
Letter.objects.filter(id=1).select_related(‘deliverer_id’)
<=>
SELECT * from app_letter
INNER JOIN app_owl ON (app_owl.id=app_letter.deliverer_id
AND app_owl.owner_id=app_letter.owner_id)
WHERE app_letter.id=1 AND app_owl.owner_id = <tenant_id>
@louisemeta

30. django-multitenant
3 steps
1. Change models to use TenantMixin and TenantManagerMixin
2.Change ForeignKey to TenantForeignKey
3.Define tenant scoping: set_current_tenant(t)
@louisemeta

31. django-multitenant
3 steps
Models before using django-multitenant
class Owner(models.Model):
type = models.CharField(max_length=10) # add choice
name = models.CharField(max_length=255)
class Owl(models.Model):
name = models.CharField(max_length=255)
owner = models.ForeignKey(Owner)
feather_color = models.CharField(max_length=255)
favorite_food = models.CharField(max_length=255)
class Letters(models.Model):
content = models.TextField()
deliverer = models.ForeignKey(Owl)
@louisemeta

32. django-multitenant
3 steps
Models with django-multitenant
class TenantManager(TenantManagerMixin, models.Manager):
pass
class Owner(TenantModelMixin, models.Model):
type = models.CharField(max_length=10) # add choice
name = models.CharField(max_length=255)
tenant_id = ‘id’
objects = TenantManager()
class Owl(TenantModelMixin, models.Model):
name = models.CharField(max_length=255)
owner = TenantForeignKey(Owner)
feather_color = models.CharField(max_length=255)
favorite_food = models.CharField(max_length=255)
tenant_id = ‘owner_id’
objects = TenantManager()
class Letters(TenantModelMixin, models.Model):
content = models.TextField()
deliverer = models.ForeignKey(Owl)
owner = TenantForeignKey(Owner)
tenant_id = ‘owner_id’
objects = TenantManager()
@louisemeta

33. django-multitenant
3 steps
set_current_tenant(t)
- Specifies which tenant the APIs should be scoped to
- Set at authentication logic via middleware
- Set explicitly at top of function (ex. view, external tasks/jobs)
@louisemeta

34. django-multitenant
3 steps
set_current_tenant(t) in Middleware
class TenantMiddleware:
def __init__(self, get_response):
self.get_response = get_response
# One-time configuration and initialization.
def __call__(self, request):
#Assuming your app has a function to get the tenant associated for a user
current_tenant = get_tenant_for_user(request.user)
set_current_tenant(current_tenant)
response = self.get_response(request)
return response
@louisemeta

35. django-multitenant
Benefits of django-multitenant
- Drop-in implementation of shared tables architecture
- Guarantees isolation
- Ready to scale with distributed Postgres (Citus)
@louisemeta

36. Postgres, citus
and shared tables
@louisemeta

37. Why Postgres
- Open source
- Constraints
- Rich SQL support
- Extensions
- PostGIS / Geospatial
- HLL
- TopN
- Citus
- Foreign data wrappers
- Fun indexes (GIN, GiST, BRIN…)
- CTEs
- Window functions
- Full text search
- Datatypes
- JSONB
@louisemeta

38. Why citus
- Citus is an open source extension for postgreSQL
- Implements a distributed architecture for postgres
- Allows you to scale out CPU, memory, etc.
- Compatible with modern postgres (up to 11)
@louisemeta

39. Distributed Postgres with citus
@louisemeta

40. Distributed Postgres with citus
Foreign key colocation
@louisemeta

41. - Full SQL support for queries on a single set of co-located
shards
- Multi-statement transaction support for modifications on a
single set of co-located shards
- Foreign keys
- …
Distributed Postgres with citus
Foreign key colocation
@louisemeta

42. Scope your queries !
Distributed Postgres with citus
@louisemeta

43. Why citus
@louisemeta

44. Scale out Django!
github.com/citusdata/django-multitenant
louise@citusdata.com
citusdata.com/newsletter
@louisemeta. @citusdata