Anatomy of a large Django site

Anatomy of a large Django siteAndy McKay Mozilla mozilla

Vancouver mozilla

PythonZope and Plone...now at Mozilla mozilla

Using Django http://www.djangoproject.comCredit: http://www.ﬂickr.com/photos/abiavati/3110357974/ mozilla

1. About the site2. Performance3. Localisation4. Reuse mozilla

1. About the site mozilla

mozilla

All code is open:https://github.com/jbalogh/zamboni mozilla

All* bugs are open: https://bugzilla.mozilla.org mozilla

Convert from CakePHP (remora)to Django (zamboni) mozilla

Credit: http://www.ﬂickr.com/photos/improbcat/4177702580/ mozilla

Changing one URL at a time from CakePHP to Django General trend to move away from PHP and do more Python and Django mozilla

How large? 250k+ addons150 mn views month500 mn API hits day mozilla

Lines of code PHP 40kPython 18k mozilla

Lines of code PHP 40k Python 18kUnit tests 15k mozilla

running both php and python side by side. a few issues on thatNo pages go out until they are faster mozilla

3 zeus load balancers 24 django (and php) 1 mysql + 4 slaves 3 memcached 3 sphinx 1 rabbitmq + 2 celeryd 1 redis master + 1 slaveCredit: http://www.ﬂickr.com/photos/tbridge/15300843/ mozilla

2. Performance mozilla

As usual, database bottleneck mozilla

Cache machine http://bit.ly/cache-machineCredit: http://www.ﬂickr.com/photos/mwichary/4063534688/ mozilla

from django.db import modelsimport caching.baseclass Addon(caching.base.CachingMixin, models.Model): ... status = models.IntegerField() objects = caching.base.CachingManager() available as a mixin need to addin the custom manager mozilla

>>> Addon.objects.filter(status=public)>>> len(connection.queries) 13 mozilla

>>> Addon.objects.filter(status=public)>>> len(connection.queries) 13>>> Addon.objects.filter(status=public)>>> len(connection.queries) 13 mozilla

Invalidation mozilla

md5(‘select... a’) [addon 3615] mozilla

md5(‘select... a’) [addon 3615] addon 3615 md5(‘select... a’) mozilla

md5(‘select... a’) [addon 3615]md5(‘select... b’) [addon 3615, addon 1685] md5(‘select... a’) addon 3615 md5(‘select... b’) mozilla

Memcachedrules = cache.get(3615)rules.add(select...)cache.set(3615, rules) mozilla

Redisredis.SADD(3615, ‘select...’) mozilla

Home page 20+ addons400+ sql queries mozilla

add-on version version version filesstandard answer indjango is select-related files files mozilla

django: select_related()http://bit.ly/select-related mozilla

simonwTransformerhttp://bit.ly/queryset-transform mozilla

@staticmethoddef transformer(addons): addon_dict = dict((a.id, a) for a in addons) vs = filter(None, (a.current_version_id for a in addons) versions = list(Version.objects.filter(id__in=vs)) for version in versions: addon_dict[version.addon_id].current_version = version slightly outdated example mozilla

big SQL statements... :( 14313 characterHome page 20+ addons~14 sql queries mozilla

Update Called on startupabout:conﬁg extensions.update.url mozilla

Incoming 8,000 req/secUncached 1,600 req/sec Im used to Plone in the bad old days mozilla

Incoming 8,000 req/secUncached 1,600 req/sec PHP 550 req/sec Im used to Plone in the bad old days mozilla

v1 Plain DjangoPHP Python mozilla

v2 Min. SQL queriesPHP Python mozilla

oh god mozilla

v3 Django and raw SQL max-requests 200, actually we hit 210PHP Python mozilla

v4 WSGI no DjangoPHP Python mozilla

v5 Pooling, optimised queries Thats 700 req/secPHP Python which translates into mozilla

Reducing the SQL queries... doesn’t always help mozilla

mySQL query cache is fast mozilla

Celery http://celeryproject.orgCredit: http://www.ﬂickr.com/photos/chiotsrun/3843988392/ mozilla

Push things async email image processing add-on validation specifically fixing data changes bet ween php and python mozilla

from celeryutils import task@taskdef update_tag(tag, **kw): tag.update_stat() mozilla

from celeryutils import task@task(rate_limit=60/h)def update_tag(tag, **kw): tag.update_stat() mozilla

from tasks import update_tagupdate_tag.delay(tag) mozilla

Measurement mozilla

Timing Middleware http://bit.ly/timing-wareCredit: http://www.ﬂickr.com/photos/wwarby/3297205226/ mozilla

mozilla

3. Localization mozilla

show site in arabic?40+ languages including rtl mozilla

content translated and templatesDatabase strings mozilla

class Addon(caching.base.CachingMixin, models.Model): ... name = models.ForeignKey(Translation) mozilla

addon.name = nameaddon.save() mozilla

addon.name = nameaddon.save()addon.name = {fr: la nomme}addon.save() mozilla

Templates mozilla

Django{% blocktrans with app=request.APP %} Add-ons for {{ app }} {% endblocktrans %} mozilla

Jinja2 http://jinja.pocoo.org/{{ _(Add-ons for {0})|f(request.APP) }} mozilla

Python Unicode hellUnicodeDecodeError: ascii codec cantdecode byte 0xd0 in position 16: ordinal not in range(128) mozilla

4. Reuse mozilla

Bleach http://pypi.python.org/pypi/Credit: http://www.ﬂickr.com/photos/maisonbisson/3350954463/ mozilla

>>> bleach.clean(an<script>evil()</script>example)an <script>evil()</script> example mozilla

>>> bleach.linkify(an http://ex.com url)an <a href="http://ex.com"rel="nofollow">http://ex.com</a> url mozilla

Javascript tests mozilla

django-qunithttp://bit.ly/django-qunit kumar mozilla

test(English, function() { z.refreshL10n(en-us); equals($(textarea:visible, this.sandbox).text().trim(), Firebug integrates with Firefox to put + a wealth of development tools...);}); mozilla

test(Japanese, function() { z.refreshL10n(ja); equals($(textarea:visible, this.sandbox).text().trim(), Firebug Web + Firefox + );}); mozilla

So we use hudson for CI, but haven’t got the automated tests in yet Hoping to do this via jstestnetUse HudsonJenkins http://bit.ly/jstestnet mozilla

pep 8 py flakes MacCabe Flake8 http://bit.ly/ﬂake8Credit: http://www.ﬂickr.com/photos/nebarnix/357779131/ mozilla

~/sandboxes/zamboni(632719) $ flake8 apps/editors/tasks.pyapps/editors/tasks.py:1: datetime imported but unusedapps/editors/tasks.py:3: stat imported but unused mozilla

Playdoh http://bit.ly/mozilla-playdoh fred wenzelCredit: http://www.ﬂickr.com/photos/ahmee/97960570/ mozilla

Basis for new projects mozilla

Celery supportJinja2 supportSimple migrationsBy default: SHA-512 password hashing X-Frame-Options: DENY secure and httponly ﬂags on cookies fred wenzel mozilla

Take inspiration from...but not the best for you for example jinja2 which makes integration with lots of django addons possible, but a bit harder mozilla

Questions? @andymckay andym@mozilla.comandym on irc.freenode.net, irc.mozilla.org mozilla

http://127.0.0.1	20
http://a0.twimg.com	1
https://twimg0-a.akamaihd.net	1

Anatomy of a large Django site

by ConFoo

Accessibility

Categories

Upload Details

Usage Rights

3 Embeds 22

Statistics

Anatomy of a large Django site Presentation Transcript