This document discusses optimizing Django performance. It begins by describing the server, Apache, database, and Django configurations. It then discusses optimizing various aspects of the Django project like the ORM, database queries, template rendering, caching, and DjangoCMS-specific optimizations. Key recommendations include using select_related, prefetch_related, batch database operations to avoid queries in loops, optimizing templates, reducing cache misses, and handling {% page_url %} tags efficiently in DjangoCMS projects. The document provides examples of inefficient code and more optimized alternatives.

1. Django performance
unchained
Artur Barseghyan
Job Ganzevoort
Goldmund, Wyldebeast & Wunderliebe
http://www.goldmund-wyldebeast-wunderliebe.nl/

2. Introduction

4. Restructuring
iSeries
Webserver

5. Architecture

6. Bottlenecks
● Server
● Apache
● Database
● Django
● Sessions
● AJAX API
● Search
● iSeries hammering

7. Server specs
● Virtualized environment
● Single VPS, horizontal scaling possible
● Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
● 24 cores
● 96GB RAM
● RHEL7

8. Apache 2.4
● Prefork MPM has a high overhead
● Event MPM scales really well
<IfModule event.c>
StartServers 10
MinSpareThreads 160
MaxSpareThreads 800
ThreadLimit 80
ThreadsPerChild 80
ServerLimit 20
MaxRequestWorkers 1280
MaxRequestsPerChild 0
</IfModule>

9. Database
● PostgresQL
● Tuning:
max_connections 100 1000
shared_buffers 32MB 8GB
work_mem 1MB
64MB
maintenance_work_mem 16MB 256MB
effective_cache_size 128MB 2GB
log_min_duration_statement -1 500
log_checkpoints off on
log_connections off on
log_disconnections off on
log_line_prefix '' '%d %s
%m: '
log_lock_waits off on

10. Django
● Running under Gunicorn
● 32 workers

11. Sessions
● In most cases, we don’t need sessions
● When we do, memcached

12. AJAX API
● Search

13. AJAX API

14. AJAX API
● Campings
● Facet counts

15. AJAX API
● Rendered results

16. Search
● Static parameters
● Volatile availability
● Set operations
● Facet counting
● Why not elasticsearch

17. iSeries API

18. Varnish

19. Halfway Summary
● Full-stack approach:
● Tuning of server, database, apache
● Varnish reverse proxy
● API desing
● Smart search filtering/faceting with set operations
● Caching of iSeries calls (Python)

20. Django optimisations
What costs time (and what can we optimise)?
● ORM
● Database queries
● Abusive cache usage
● Template rendering

21. But how to measure performance?

22. django-debug-toolbar
https://pypi.python.org/pypi/django-debug-toolbar

23. Surprised?

24. Just joking

25. django-debug-toolbar-template-profiler
https://pypi.python.org/pypi/django-debug-toolbar-template-profiler

26. Before we move on
django-debug-toolbar is an excellent tool, but page rendering timings lie.

27. Let’s start with imaginary app...

28. Imaginary app (page 1)
class Publisher(models.Model):
name = models.CharField(max_length=30)
address = models.CharField(max_length=50)
city = models.CharField(max_length=60)
state_province = models.CharField(max_length=30)
country = models.CharField(max_length=50)
website = models.URLField()
class Author(models.Model):
salutation = models.CharField(max_length=10)
name = models.CharField(max_length=200)
email = models.EmailField()
headshot = models.ImageField(upload_to='authors', null=True, blank=True)

29. Imaginary app (page 2)
class Book(models.Model):
title = models.CharField(max_length=100)
authors = models.ManyToManyField('books.Author', related_name='books')
publisher = models.ForeignKey(Publisher, related_name='books')
publication_date = models.DateField()
isbn = models.CharField(max_length=100, unique=True)
price = models.DecimalField(max_digits=10, decimal_places=2)
pages = models.PositiveIntegerField(default=200)
stock_count = models.PositiveIntegerField(default=30)

30. Imaginary app (page 3)
class Order(models.Model):
owner = models.ForeignKey(settings.AUTH_USER_MODEL)
lines = models.ManyToManyField("books.OrderLine", blank=True)
created = models.DateField(auto_now_add=True)
updated = models.DateField(auto_now=True)
class OrderLine(models.Model):
book = models.ForeignKey('books.Book', related_name='order_lines')

31. ORM & Database queries
Tips:
● Make use of `select_related` and `prefetch_related` for fetching related
objects at once.
● Make use of `only` and `defer` (typical use case: listing views), but use
them with caution.
● Use `annotate` and `aggregate` for calculations on database level.
● When fetching IDs of related item (if you only need ID), use
{field_name}_id instead of {field_name}.id, since the first one does not
cause table JOIN.
● Consider using `values` or `values_list` for listing views, when possible.

32. Use case
Listing of books (about 2000 in total) with the following information:
● Book title
● Names of all authors of the book separated by comma
● Publisher name
● Number of pages
● Price

33. Bad example
books = Book.objects.all()

34. Results
4023 queries took 730 ms, page rendered in 23932 ms

35. Let’s improve
Adding:
● select_related
● prefetch_related
● only

36. Good example
books = Book.objects.all()
.select_related('publisher')
.prefetch_related('authors')
.only('id', 'title', 'pages', 'price',
'publisher__id', 'publisher__name',
'authors__id', 'authors__name')

37. Results
2 queries took 12 ms, page rendered in 2104 ms

38. Let’s improve more
Adding:
● values
● annotate

39. Even better example
books = Book.objects.all()
.values('id', 'title', 'pages', 'price',
'publisher__id', 'publisher__name')
.annotate(
authors__name=GroupConcat('authors__name', separator=', ')
)
.distinct()

40. Results
1 query took 13 ms, page rendered in 568 ms

41. But what is GroupConcat?
But what is GroupConcat?

42. Custom aggregation functions
from django.db.models import Aggregate
class GroupConcat(Aggregate):
function = 'group_concat'
@property
def template(self):
separator = self.extra.get('separator')
if separator:
return '%(function)s(%(field)s, "%(separator)s")'
else:
return '%(function)s(%(field)s)'

43. Surprised?

44. Avoid database hits in the loop!
Operate in batch as much as possible!

45. Use case
Create 50 authors

46. Bad example
for __i in range(1, 50):
author = Author.objects.create(
salutation='Something %s' % uuid.uuid4(),
name='Some name %s' % uuid.uuid4(),
email='name%s@example.com' % uuid.uuid4()
)

47. Results
98 queries took 40 ms, page rendered in 652 ms

48. Let’s improve
Adding:
● bulk_create

49. Good example
authors_list = []
for __i in range(1, 50):
author = Author(
salutation='Something %s' % uuid.uuid4(),
name='Some name %s' % uuid.uuid4(),
email='name%s@example.com' % uuid.uuid4()
)
authors_list.append(author)
Author.objects.bulk_create(authors_list)

50. Results
2 queries took 1.73 ms, page rendered in 60 ms

51. If you use django-debug-toolbar already...
...you may say

52. I know it, django-debug-toolbar is great...
...but what about non-HTML (JSON, partial HTML) views?

53. django-debug-toolbar-force
https://pypi.python.org/pypi/django-debug-toolbar-force
MIDDLEWARE_CLASSES += (
'debug_toolbar.middleware.DebugToolbarMiddleware',
'debug_toolbar_force.middleware.ForceDebugToolbarMiddleware',
)
And add ?debug-toolbar to any non-HTML (JSON, AJAX) view URL

54. Surprised?

55. Hey, enough with these faces!

56. Fine. Let’s move on...

57. One important thing!
And you’ll hear it many times
in this presentation...

58. Avoid querying in the loop!

59. Given the following model...
class Item(models.Model):
text = models.TextField()
page_id = models.IntegerField()
...where page_id is the correspondent ID
of the CMS page

60. But WHY???

61. Because it happens and you have to deal with it!

62. Surprised?

63. Bad example
items = Item.objects.all()
for item in items:
page = Page.objects.get(id=item.page_id)
# ... do something else with item

64. Results
101 database queries

65. Improved example
# Freeze the queryset
items = Item.objects.all()[:]
# Collect all page ids
page_ids = [item.page_id for item in items]
pages = Page.objects.filter(id__in=page_ids)
pages_dict = {page.id: page for page in pages}
for item in items:
page = pages_dict.get(item.page_id)
# ... do something else with item

66. Results
No additional database queries and no missed cache hits.

67. Unnecessary cache hits
Tips:
● Analyze cache usage with django-debug-toolbar.
● Use template fragment caching to minimize the number of cache queries.
● Identify heavy parts of your templates with TemplateProfilerPanel.
Optimise them first and after that, if they are still heavy, cache them.
● Try to avoid repetitive missed cache queries.
● The well known {% page_url %} and {% page_attribute %} tags of
DjangoCMS may produce a lot of missed cache queries.
● Document cache usages. Explain in the code why do you cache that
certain part. Keep track of all cache usages in a separate section of your
developer documentation.

68. Optimise templates
Tips:
● Clean up your templates. Remove all unused import statements.
● Use cached template loading on production.
● Avoid database queries done in the loop. Especially {% page_url %} and
{% page_attribute %} template tags of DjangoCMS.

69. A couple of things
● Try to reduce the number of database queries
● Use EXPLAIN when things are still slow.
● Add indexes when necessary.
● Compare the "before" and "after". Do it often.
● Test page load speed with caching on and off.
● Test the first page load and the second page load. Analyze the difference.
● Try to reduce the number of missed cache queries.

70. Tips
● Try to get rid of context processors and middleware that hit the database.
● Be careful with templatetags that query the database/cache. Especially,
when they are done in a loop.

71. DjangoCMS specific tricks

72. DjangoCMS is fine...
...but all your non-CMS views would be affected with
additional checks and queries, even if they are totally
irrelevant.

73. Scary?

74. Hold on, there’s a way to fix it!
# Some sort of a fake page container, to trick django-cms.
PageData = namedtuple('PageData', ['pk'])
class MyView(View):
def get(self, request):
# This is to trick the django-cms middleware, so that we don't
# fetch page from the request (not needed here).
request._current_page_cache = PageData("_")
context = self.get_context(request)
return render_to_response(
self.template_name),
context,
request
)

75. Now you non-CMS views are clean

76. Surprised?

77. Dealing with {% page_url %}
Avoid:
● Use of {% page_url %} or {% page_attribute %} tags on IDs. Pass
complete objects only.
● Apply tricks to the querysets to fetch the page objects efficiently.
● If you list a lot of DjangoCMS Page objects (with URLs) on a page, don’t
use {% page_url %} at all. Instead, use customized template tag which fits
your needs better.

78. Use case
Sitemap page with about 200 CMS pages.

79. Bad example
pages = Page.objects.all()
{% for page in pages %}
{% page_url page.id %}
{% endfor %}

80. Results
2000 database queries and 3000 missed cached hits

81. Improved example (page 1)
pages = Page.objects.all()[]
page_ids = [page.id for page in pages]
titles = Title.objects
.filter(language=get_language())
.filter(page__id__in=page_ids)
.select_related('page', 'page__publisher_public', 'page__site')
.only('id', 'title', 'slug', 'path', 'page__id', 'page__reverse_id',
'page__publisher_is_draft',
'page__publisher_public__publisher_is_draft',
'page__publication_date', 'page__publication_end_date',
'page__site', 'page__is_home', 'page__revision_id')[:]

82. Improved example (page 2)
pages_dict = {}
for _title in titles:
if not hasattr(_title.page, 'title_cache'):
_title.page.title_cache = {}
_title.page.title_cache[get_language()] = _title
pages_dict[_title.page.id] = _title.page
_pages = []
for page in pages:
_pages.append(page_dict.get(page.id, page))
pages = _pages

83. Improved example (page 3)
class AddonsPageUrlNoCache(PageUrl):
"""PageUrl made that doesn't hit the cache.
You should not be using this everywhere, however if you know what you're
doing, it can save you a lot of page rendering time.
"""
name = 'addons_page_url_no_cache'

84. Improved example (page 4)
def get_value(self, context, page_lookup, lang, site):
site_id = get_site_id(site)
request = context.get('request', False)
if not request:
return ''
if lang is None:
lang = get_language_from_request(request)
page = _get_page_by_untyped_arg(page_lookup, request, site_id)
if page:
url = page.get_absolute_url(language=lang)
if url:
return url
return ''

85. Improved example (page 5)
register.tag(AddonsPageUrlNoCache)
{% for page in pages %}
{% addons_page_url_no_cache page %}
{% endfor %}

86. Results
No additional database queries and no missed cache hits.

87. Surprised?

88. Use get_render_queryset on CMS plugins
Use get_render_queryset class method to fetch objects efficiently.

89. Use case
Write a CMS plugin which has 2 foreign key relations.

90. Bad example
class TextPictureLink(AbstractText):
title = models.TextField(_("Title"), null=True, blank=True)
image = FilerImageField(null=True, blank=True)
page_link = PageField(null=True, blank=True)
class GenericContentPlugin(TextPlugin):
module = _('Blocks')
render_template = 'path/to/generic_text_plugin.html'
name = _('Generic Plugin')
model = models.TextPictureLink

91. Results
3 database hits for a single plugin. How many plugins do you have on a page?

92. Good example
class GenericContentPlugin(TextPlugin):
# ... original code
@classmethod
def get_render_queryset(cls):
"""Tweak the queryset."""
return cls.model._default_manager
.all()
.select_related('image', 'page_link')

93. Results
1 database hit for a single plugin.

94. Load testing

95. Apache JMeter
Highlights:
● Load/stress testing tool for analyzing/measuring the performance.
● Focused on web applications.
● Plugin based architecture.
● Supports variable parametrisation, assertions (response validation), per-
thread cookies, configuration variables.
● Comes with a large variety of reports (aggregations, graphs).
● Can replay access logs, including Apache2, Nginx, Tomcat and more.
● You can save your tests and repeat them later.

96. We measured “before”...

97. We measured “after”...

98. And “after” was a way better!

99. We’re happy

100. Linkodrome
1 / n
This presentation on slideshare
https://www.slideshare.net/barseghyanartur/pygrunn-2017-django-performance-unchained-slides
django-debug-toolbar-template-profiler https://pypi.python.org/pypi/django-debug-toolbar-template-
profiler
django-debug-toolbar-force
https://pypi.python.org/pypi/django-debug-toolbar-force
Apache JMeter
http://jmeter.apache.org/
Custom aggregation functions
https://docs.djangoproject.com/en/1.8/ref/models/expressions/#creating-your-own-aggregate-functions

101. Linkodrome (page 2)
1 / n
My GitHub account
https://github.com/barseghyanartur
django-debug-toolbar
https://pypi.python.org/pypi/django-debug-toolbar

102. Questions?

103. Thank you!
Artur Barseghyan
Job Ganzevoort
Goldmund, Wyldebeast & Wunderliebe
http://www.goldmund-wyldebeast-wunderliebe.nl/