Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Django
Performance
Recipes
September 2015
Jon Atkinson
Technical Director
FARM Digital
Me
• I am not a very good programmer.
• I’m quite a good problem solver.
• I have made a lot of mistakes.
• I prefer valua...
My Context
• “Fast-moving agency environment”.
• We often participate in launches; demand is
spiky.
• Complex software.
Shared Context
• Traffic is unpredictable.
• We probably host in the cloud.
• The Django ecosystem.
Performance Matters
Speed is all about perception, but:
0-100ms Instant!
100-300ms Small delay.
300-1000ms Something is ha...
Rule of thumb
50% of users will abandon a task after
3 seconds.
Environment
A web server, talking to:
A managed python process, talking to:
A cache, and a database.
Request Cycle
Web Server
ORM
Middleware
Views + Templates
Web Server
Tools
!
ORM
• Optimising your database is a separate
presentation entirely.
• There are no ‘slow’ databases any more.
• In a read-...
ORM
django-cachalot
Caches your Django ORM queries and
automatically invalidates them.
ORM
$ pip install django-cachalot
INSTALLED_APPS += (‘cachalot’)
settings.CACHALOT_ENABLED = True
ORM
from django.conf import settings
from django.test.utils import override_settings
with override_settings(CACHALOT_ENABL...
ORM
• A few other data access tips:
• Is your database doing DNS lookups?
• Do you have a connection timeout set? The
defa...
Middleware
Middleware is dumb.
Middleware
Middleware
• Think hard. Milliseconds add up.
• Middleware (and context processors!) often
becomes a dumping ground for co...
Middleware
Middleware is helpful.
Middleware
django.middleware.http.ConditionalGetMiddleware
Optimises GET requests from modern browsers
django.middleware.h...
Middleware
$ pip install django-cprofile-middleware
“Once you've installed it, log in as a user who has
staff privileges a...
Middleware
7986 function calls (7850 primitive calls) in 1.725 CPU seconds
Ordered by: internal time, call count
List redu...
Views & Templates
• View performance problems are usually obvious:
• Avoid nested loops (especially when generating
QueryS...
Views & Templates
• Templates are more interesting.
• It’s easy to duplicate ORM calls already made in
the view.
• It’s ea...
Views & Templates
class Person(models.Model):
def friends(self):
# Hit the database here for something complex…
return fri...
Views & Templates
from django.utils.functional import
cached_property
@cached_property
def friends(self):
# Hit the databa...
Views & Templates
• Caching template fragments is very powerful.
• Sometimes you need to do something expensive in a
templ...
Views & Templates
• Where do your templates actually live?
• Cloud disk performance can be erratic.
Views & Templates
# Default setting.
TEMPLATE_LOADERS = (
'django.template.loaders.filesystem.Loader',
'django.template.lo...
Views & Templates
TEMPLATE_LOADERS = (
('django.template.loaders.cached.Loader', (
'django.template.loaders.filesystem.Loa...
nginx
This is my “one weird tip”…
… with a trade-off.
uwsgi_cache_path /tmp/nginx levels=1:2 keys_zone=my_zone:10m;
server {
listen 80;
server_name example.com;
…
uwsgi_cache_u...
nginx
Tools
• The tool ecosystem is richer now than ever
before.
$ pip install django-debug-toolbar
$ pip install dogslow
• Meas...
Tools
Tools
import newrelic.agent
from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()
# This needs ...
Meta 1
Performance can be personal.
Meta 2
Meta 2
Meta 3
Performance is of huge value.
10x programmers probably don’t exist.
Keep your eyes up.
Concentrate on where the val...
Thank You
Questions?
Resources & Credits: http://blog.wearefarm.com
Upcoming SlideShare
Loading in …5
×

Django Performance Recipes

1,551 views

Published on

I've written a lot of Django code in my career, some slow, and some fast; my aim is to share a few of the techniques you can use to go from 300ms response times to 30ms. We'll travel up and down the stack, looking to identify, monitor and solve performance issues, while dodging those which just aren't worth solving.

Full video is available at: https://skillsmatter.com/skillscasts/6628-django-performance-recipes

Published in: Technology
  • Be the first to comment

Django Performance Recipes

  1. 1. Django Performance Recipes September 2015 Jon Atkinson Technical Director FARM Digital
  2. 2. Me • I am not a very good programmer. • I’m quite a good problem solver. • I have made a lot of mistakes. • I prefer valuable solutions.
  3. 3. My Context • “Fast-moving agency environment”. • We often participate in launches; demand is spiky. • Complex software.
  4. 4. Shared Context • Traffic is unpredictable. • We probably host in the cloud. • The Django ecosystem.
  5. 5. Performance Matters Speed is all about perception, but: 0-100ms Instant! 100-300ms Small delay. 300-1000ms Something is happening. 1000ms+ Likely task switch. 10000ms+ Abandoned task.
  6. 6. Rule of thumb 50% of users will abandon a task after 3 seconds.
  7. 7. Environment A web server, talking to: A managed python process, talking to: A cache, and a database.
  8. 8. Request Cycle Web Server ORM Middleware Views + Templates Web Server Tools
  9. 9. !
  10. 10. ORM • Optimising your database is a separate presentation entirely. • There are no ‘slow’ databases any more. • In a read-heavy environment, caching Querysets is a huge advantage.
  11. 11. ORM django-cachalot Caches your Django ORM queries and automatically invalidates them.
  12. 12. ORM $ pip install django-cachalot INSTALLED_APPS += (‘cachalot’) settings.CACHALOT_ENABLED = True
  13. 13. ORM from django.conf import settings from django.test.utils import override_settings with override_settings(CACHALOT_ENABLED=False): # SQL queries are not cached in this block @override_settings(CACHALOT_CACHE=‘second_cache') def your_function(): # What’s in this function uses another cache # Globally disables SQL caching until you set it back to True settings.CACHALOT_ENABLED = False
  14. 14. ORM • A few other data access tips: • Is your database doing DNS lookups? • Do you have a connection timeout set? The default is 0, and setup/teardown costs time. settings.DATABASES[‘…’][‘CONN_MAX_AGE’] = 600
  15. 15. Middleware Middleware is dumb.
  16. 16. Middleware
  17. 17. Middleware • Think hard. Milliseconds add up. • Middleware (and context processors!) often becomes a dumping ground for common features.
  18. 18. Middleware Middleware is helpful.
  19. 19. Middleware django.middleware.http.ConditionalGetMiddleware Optimises GET requests from modern browsers django.middleware.http.GZipMiddleware Compresses responses. But be aware of BREACH!
  20. 20. Middleware $ pip install django-cprofile-middleware “Once you've installed it, log in as a user who has staff privileges and add ?prof to any URL to see the profiler's stats.” eg. http://localhost:8000/foo/?prof.
  21. 21. Middleware 7986 function calls (7850 primitive calls) in 1.725 CPU seconds Ordered by: internal time, call count List reduced from 392 to 20 due to restriction <20> ncalls tottime percall cumtime percall filename:lineno(function) 2 1.570 0.785 1.570 0.785 /…/django/db/backends/__init__.py:36(_commit) 15 0.043 0.003 0.043 0.003 /…/linecache.py:68(updatecache) 1 0.020 0.020 0.027 0.027 /…/django/contrib/auth/models.py:1(<module>) 12 0.014 0.001 0.030 0.002 /…/django/utils/importlib.py:18(import_module) 1013 0.010 0.000 0.010 0.000 /…/posixpath.py:56(join)
  22. 22. Views & Templates • View performance problems are usually obvious: • Avoid nested loops (especially when generating QuerySets!) • Cache where you can. • Always return at the earliest possible moment.
  23. 23. Views & Templates • Templates are more interesting. • It’s easy to duplicate ORM calls already made in the view. • It’s easy to traverse relationships in the template language. • Templates are loaded from disk by default.
  24. 24. Views & Templates class Person(models.Model): def friends(self): # Hit the database here for something complex… return friends # View: if person.friends(): # Do something here… # Template: {% for friend in person.friends %}
  25. 25. Views & Templates from django.utils.functional import cached_property @cached_property def friends(self): # Hit the database here for something complex… return friends
  26. 26. Views & Templates • Caching template fragments is very powerful. • Sometimes you need to do something expensive in a template, but: {% load cache %} {% cache 500 sidebar %} … do something expensive here … {% endcache %}
  27. 27. Views & Templates • Where do your templates actually live? • Cloud disk performance can be erratic.
  28. 28. Views & Templates # Default setting. TEMPLATE_LOADERS = ( 'django.template.loaders.filesystem.Loader', 'django.template.loaders.app_directories.Loader', )
  29. 29. Views & Templates TEMPLATE_LOADERS = ( ('django.template.loaders.cached.Loader', ( 'django.template.loaders.filesystem.Loader', 'django.template.loaders.app_directories.Loader', )), )
  30. 30. nginx This is my “one weird tip”… … with a trade-off.
  31. 31. uwsgi_cache_path /tmp/nginx levels=1:2 keys_zone=my_zone:10m; server { listen 80; server_name example.com; … uwsgi_cache_use_stale error timeout invalid_header http_500; uwsgi_cache_valid 10m; location / { include uwsgi_params; uwsgi_pass unix:///tmp/example.sock; uwsgi_cache cache; uwsgi_cache_key $scheme:$host$request_uri:$request_method; uwsgi_cache_bypass $http_pragma $http_authorization; uwsgi_no_cache $http_pragma $http_authorization; }
  32. 32. nginx
  33. 33. Tools • The tool ecosystem is richer now than ever before. $ pip install django-debug-toolbar $ pip install dogslow • Measure twice, cut once. • Remember, Schrödinger’s web app.
  34. 34. Tools
  35. 35. Tools
  36. 36. import newrelic.agent from django.core.wsgi import get_wsgi_application application = get_wsgi_application() # This needs to be called after we bootstrapped the application # otherwise the settings wouldn't be configured from django.conf import settings # noqa if hasattr(settings, 'NEWRELIC_CONFIG'): newrelic.agent.initialize(settings.NEWRELIC_CONFIG getattr(settings, 'NEWRELIC_ENVIRONMENT', None)) application = newrelic.agent.WSGIApplicationWrapper(application)
  37. 37. Meta 1 Performance can be personal.
  38. 38. Meta 2
  39. 39. Meta 2
  40. 40. Meta 3 Performance is of huge value. 10x programmers probably don’t exist. Keep your eyes up. Concentrate on where the value lies.
  41. 41. Thank You Questions? Resources & Credits: http://blog.wearefarm.com

×