I've written a lot of Django code in my career, some slow, and some fast; my aim is to share a few of the techniques you can use to go from 300ms response times to 30ms. We'll travel up and down the stack, looking to identify, monitor and solve performance issues, while dodging those which just aren't worth solving. Full video is available at: https://skillsmatter.com/skillscasts/6628-django-performance-recipes

1. Django
Performance
Recipes
September 2015
Jon Atkinson
Technical Director
FARM Digital

2. Me
• I am not a very good programmer.
• I’m quite a good problem solver.
• I have made a lot of mistakes.
• I prefer valuable solutions.

3. My Context
• “Fast-moving agency environment”.
• We often participate in launches; demand is
spiky.
• Complex software.

4. Shared Context
• Traffic is unpredictable.
• We probably host in the cloud.
• The Django ecosystem.

5. Performance Matters
Speed is all about perception, but:
0-100ms Instant!
100-300ms Small delay.
300-1000ms Something is happening.
1000ms+ Likely task switch.
10000ms+ Abandoned task.

6. Rule of thumb
50% of users will abandon a task after
3 seconds.

7. Environment
A web server, talking to:
A managed python process, talking to:
A cache, and a database.

8. Request Cycle
Web Server
ORM
Middleware
Views + Templates
Web Server
Tools

9. !

10. ORM
• Optimising your database is a separate
presentation entirely.
• There are no ‘slow’ databases any more.
• In a read-heavy environment, caching Querysets
is a huge advantage.

11. ORM
django-cachalot
Caches your Django ORM queries and
automatically invalidates them.

12. ORM
$ pip install django-cachalot
INSTALLED_APPS += (‘cachalot’)
settings.CACHALOT_ENABLED = True

13. ORM
from django.conf import settings
from django.test.utils import override_settings
with override_settings(CACHALOT_ENABLED=False):
# SQL queries are not cached in this block
@override_settings(CACHALOT_CACHE=‘second_cache')
def your_function():
# What’s in this function uses another cache
# Globally disables SQL caching until you set it back to True
settings.CACHALOT_ENABLED = False

14. ORM
• A few other data access tips:
• Is your database doing DNS lookups?
• Do you have a connection timeout set? The
default is 0, and setup/teardown costs time.
settings.DATABASES[‘…’][‘CONN_MAX_AGE’] = 600

15. Middleware
Middleware is dumb.

16. Middleware

17. Middleware
• Think hard. Milliseconds add up.
• Middleware (and context processors!) often
becomes a dumping ground for common
features.

18. Middleware
Middleware is helpful.

19. Middleware
django.middleware.http.ConditionalGetMiddleware
Optimises GET requests from modern browsers
django.middleware.http.GZipMiddleware
Compresses responses. But be aware of BREACH!

20. Middleware
$ pip install django-cprofile-middleware
“Once you've installed it, log in as a user who has
staff privileges and add ?prof to any URL to see the
profiler's stats.”
eg. http://localhost:8000/foo/?prof.

21. Middleware
7986 function calls (7850 primitive calls) in 1.725 CPU seconds
Ordered by: internal time, call count
List reduced from 392 to 20 due to restriction <20>
ncalls tottime percall cumtime percall filename:lineno(function)
2 1.570 0.785 1.570 0.785 /…/django/db/backends/__init__.py:36(_commit)
15 0.043 0.003 0.043 0.003 /…/linecache.py:68(updatecache)
1 0.020 0.020 0.027 0.027 /…/django/contrib/auth/models.py:1(<module>)
12 0.014 0.001 0.030 0.002 /…/django/utils/importlib.py:18(import_module)
1013 0.010 0.000 0.010 0.000 /…/posixpath.py:56(join)

22. Views & Templates
• View performance problems are usually obvious:
• Avoid nested loops (especially when generating
QuerySets!)
• Cache where you can.
• Always return at the earliest possible moment.

23. Views & Templates
• Templates are more interesting.
• It’s easy to duplicate ORM calls already made in
the view.
• It’s easy to traverse relationships in the template
language.
• Templates are loaded from disk by default.

24. Views & Templates
class Person(models.Model):
def friends(self):
# Hit the database here for something complex…
return friends
# View:
if person.friends():
# Do something here…
# Template:
{% for friend in person.friends %}

25. Views & Templates
from django.utils.functional import
cached_property
@cached_property
def friends(self):
# Hit the database here for something complex…
return friends

26. Views & Templates
• Caching template fragments is very powerful.
• Sometimes you need to do something expensive in a
template, but:
{% load cache %}
{% cache 500 sidebar %}
… do something expensive here …
{% endcache %}

27. Views & Templates
• Where do your templates actually live?
• Cloud disk performance can be erratic.

28. Views & Templates
# Default setting.
TEMPLATE_LOADERS = (
'django.template.loaders.filesystem.Loader',
'django.template.loaders.app_directories.Loader',
)

29. Views & Templates
TEMPLATE_LOADERS = (
('django.template.loaders.cached.Loader', (
'django.template.loaders.filesystem.Loader',
'django.template.loaders.app_directories.Loader',
)),
)

30. nginx
This is my “one weird tip”…
… with a trade-off.

31. uwsgi_cache_path /tmp/nginx levels=1:2 keys_zone=my_zone:10m;
server {
listen 80;
server_name example.com;
…
uwsgi_cache_use_stale error timeout invalid_header http_500;
uwsgi_cache_valid 10m;
location / {
include uwsgi_params;
uwsgi_pass unix:///tmp/example.sock;
uwsgi_cache cache;
uwsgi_cache_key $scheme:$host$request_uri:$request_method;
uwsgi_cache_bypass $http_pragma $http_authorization;
uwsgi_no_cache $http_pragma $http_authorization;
}

32. nginx

33. Tools
• The tool ecosystem is richer now than ever
before.
$ pip install django-debug-toolbar
$ pip install dogslow
• Measure twice, cut once.
• Remember, Schrödinger’s web app.

34. Tools

35. Tools

36. import newrelic.agent
from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()
# This needs to be called after we bootstrapped the application
# otherwise the settings wouldn't be configured
from django.conf import settings # noqa
if hasattr(settings, 'NEWRELIC_CONFIG'):
newrelic.agent.initialize(settings.NEWRELIC_CONFIG
getattr(settings, 'NEWRELIC_ENVIRONMENT', None))
application = newrelic.agent.WSGIApplicationWrapper(application)

37. Meta 1
Performance can be personal.

38. Meta 2

39. Meta 2

40. Meta 3
Performance is of huge value.
10x programmers probably don’t exist.
Keep your eyes up.
Concentrate on where the value lies.

41. Thank You
Questions?
Resources & Credits: http://blog.wearefarm.com