Django Performance Recipes

Django
Performance
Recipes
September 2015
Jon Atkinson
Technical Director
FARM Digital

Me
• I am not a very good programmer.
• I’m quite a good problem solver.
• I have made a lot of mistakes.
• I prefer valuable solutions.

My Context
• “Fast-moving agency environment”.
• We often participate in launches; demand is
spiky.
• Complex software.

Shared Context
• Trafﬁc is unpredictable.
• We probably host in the cloud.
• The Django ecosystem.

Performance Matters
Speed is all about perception, but:
0-100ms Instant!
100-300ms Small delay.
300-1000ms Something is happening.
1000ms+ Likely task switch.
10000ms+ Abandoned task.

Rule of thumb
50% of users will abandon a task after
3 seconds.

Environment
A web server, talking to:
A managed python process, talking to:
A cache, and a database.

Request Cycle
Web Server
ORM
Middleware
Views + Templates
Web Server
Tools

ORM
• Optimising your database is a separate
presentation entirely.
• There are no ‘slow’ databases any more.
• In a read-heavy environment, caching Querysets
is a huge advantage.

ORM
django-cachalot
Caches your Django ORM queries and
automatically invalidates them.

ORM
$ pip install django-cachalot
INSTALLED_APPS += (‘cachalot’)
settings.CACHALOT_ENABLED = True

ORM
from django.conf import settings
from django.test.utils import override_settings
with override_settings(CACHALOT_ENABLED=False):
# SQL queries are not cached in this block
@override_settings(CACHALOT_CACHE=‘second_cache')
def your_function():
# What’s in this function uses another cache
# Globally disables SQL caching until you set it back to True
settings.CACHALOT_ENABLED = False

ORM
• A few other data access tips:
• Is your database doing DNS lookups?
• Do you have a connection timeout set? The
default is 0, and setup/teardown costs time.
settings.DATABASES[‘…’][‘CONN_MAX_AGE’] = 600

Middleware
Middleware is dumb.

Middleware
• Think hard. Milliseconds add up.
• Middleware (and context processors!) often
becomes a dumping ground for common
features.

Middleware
Middleware is helpful.

Middleware
django.middleware.http.ConditionalGetMiddleware
Optimises GET requests from modern browsers
django.middleware.http.GZipMiddleware
Compresses responses. But be aware of BREACH!

Middleware
$ pip install django-cprofile-middleware
“Once you've installed it, log in as a user who has
staff privileges and add ?prof to any URL to see the
proﬁler's stats.”
eg. http://localhost:8000/foo/?prof.

Middleware
7986 function calls (7850 primitive calls) in 1.725 CPU seconds
Ordered by: internal time, call count
List reduced from 392 to 20 due to restriction <20>
ncalls tottime percall cumtime percall filename:lineno(function)
2 1.570 0.785 1.570 0.785 /…/django/db/backends/__init__.py:36(_commit)
15 0.043 0.003 0.043 0.003 /…/linecache.py:68(updatecache)
1 0.020 0.020 0.027 0.027 /…/django/contrib/auth/models.py:1(<module>)
12 0.014 0.001 0.030 0.002 /…/django/utils/importlib.py:18(import_module)
1013 0.010 0.000 0.010 0.000 /…/posixpath.py:56(join)

Views & Templates
• View performance problems are usually obvious:
• Avoid nested loops (especially when generating
QuerySets!)
• Cache where you can.
• Always return at the earliest possible moment.

Views & Templates
• Templates are more interesting.
• It’s easy to duplicate ORM calls already made in
the view.
• It’s easy to traverse relationships in the template
language.
• Templates are loaded from disk by default.

Views & Templates
class Person(models.Model):
def friends(self):
# Hit the database here for something complex…
return friends
# View:
if person.friends():
# Do something here…
# Template:
{% for friend in person.friends %}

Views & Templates
from django.utils.functional import
cached_property
@cached_property
def friends(self):
# Hit the database here for something complex…
return friends

Views & Templates
• Caching template fragments is very powerful.
• Sometimes you need to do something expensive in a
template, but:
{% load cache %}
{% cache 500 sidebar %}
… do something expensive here …
{% endcache %}

Views & Templates
• Where do your templates actually live?
• Cloud disk performance can be erratic.

Views & Templates
# Default setting.
TEMPLATE_LOADERS = (
'django.template.loaders.filesystem.Loader',
'django.template.loaders.app_directories.Loader',
)

Views & Templates
TEMPLATE_LOADERS = (
('django.template.loaders.cached.Loader', (
'django.template.loaders.filesystem.Loader',
'django.template.loaders.app_directories.Loader',
)),
)

nginx
This is my “one weird tip”…
… with a trade-off.

uwsgi_cache_path /tmp/nginx levels=1:2 keys_zone=my_zone:10m;
server {
listen 80;
server_name example.com;
…
uwsgi_cache_use_stale error timeout invalid_header http_500;
uwsgi_cache_valid 10m;
location / {
include uwsgi_params;
uwsgi_pass unix:///tmp/example.sock;
uwsgi_cache cache;
uwsgi_cache_key $scheme:$host$request_uri:$request_method;
uwsgi_cache_bypass $http_pragma $http_authorization;
uwsgi_no_cache $http_pragma $http_authorization;
}

Tools
• The tool ecosystem is richer now than ever
before.
$ pip install django-debug-toolbar
$ pip install dogslow
• Measure twice, cut once.
• Remember, Schrödinger’s web app.

import newrelic.agent
from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()
# This needs to be called after we bootstrapped the application
# otherwise the settings wouldn't be configured
from django.conf import settings # noqa
if hasattr(settings, 'NEWRELIC_CONFIG'):
newrelic.agent.initialize(settings.NEWRELIC_CONFIG
getattr(settings, 'NEWRELIC_ENVIRONMENT', None))
application = newrelic.agent.WSGIApplicationWrapper(application)

Meta 1
Performance can be personal.

Meta 3
Performance is of huge value.
10x programmers probably don’t exist.
Keep your eyes up.
Concentrate on where the value lies.

Thank You
Questions?
Resources & Credits: http://blog.wearefarm.com

Django Performance Recipes

More Related Content

What's hot

Similar to Django Performance Recipes

Recently uploaded

Django Performance Recipes