• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Anatomy of a large Django site

Anatomy of a large Django site






Total Views
Views on SlideShare
Embed Views



4 Embeds 23 20
http://a0.twimg.com 1
https://twimg0-a.akamaihd.net 1
https://twitter.com 1



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.


11 of 1 previous next

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    Anatomy of a large Django site Anatomy of a large Django site Presentation Transcript

    • Anatomy of a large Django siteAndy McKay Mozilla mozilla
    • Vancouver mozilla
    • PythonZope and Plone...now at Mozilla mozilla
    • Using Django http://www.djangoproject.comCredit: http://www.flickr.com/photos/abiavati/3110357974/ mozilla
    • 1. About the site2. Performance3. Localisation4. Reuse mozilla
    • 1. About the site mozilla
    • mozilla
    • All code is open:https://github.com/jbalogh/zamboni mozilla
    • All* bugs are open: https://bugzilla.mozilla.org mozilla
    • Convert from CakePHP (remora)to Django (zamboni) mozilla
    • Credit: http://www.flickr.com/photos/improbcat/4177702580/ mozilla
    • Changing one URL at a time from CakePHP to Django General trend to move away from PHP and do more Python and Django mozilla
    • How large? 250k+ addons150 mn views month500 mn API hits day mozilla
    • Lines of code PHP 40kPython 18k mozilla
    • Lines of code PHP 40k Python 18kUnit tests 15k mozilla
    • running both php and python side by side. a few issues on thatNo pages go out until they are faster mozilla
    • 3 zeus load balancers 24 django (and php) 1 mysql + 4 slaves 3 memcached 3 sphinx 1 rabbitmq + 2 celeryd 1 redis master + 1 slaveCredit: http://www.flickr.com/photos/tbridge/15300843/ mozilla
    • 2. Performance mozilla
    • As usual, database bottleneck mozilla
    • Cache machine http://bit.ly/cache-machineCredit: http://www.flickr.com/photos/mwichary/4063534688/ mozilla
    • from django.db import modelsimport caching.baseclass Addon(caching.base.CachingMixin, models.Model): ... status = models.IntegerField() objects = caching.base.CachingManager() available as a mixin need to addin the custom manager mozilla
    • >>> Addon.objects.filter(status=public)>>> len(connection.queries) 13 mozilla
    • >>> Addon.objects.filter(status=public)>>> len(connection.queries) 13>>> Addon.objects.filter(status=public)>>> len(connection.queries) 13 mozilla
    • Invalidation mozilla
    • md5(‘select... a’) [addon 3615] mozilla
    • md5(‘select... a’) [addon 3615] addon 3615 md5(‘select... a’) mozilla
    • md5(‘select... a’) [addon 3615]md5(‘select... b’) [addon 3615, addon 1685] md5(‘select... a’) addon 3615 md5(‘select... b’) mozilla
    • Memcachedrules = cache.get(3615)rules.add(select...)cache.set(3615, rules) mozilla
    • Redisredis.SADD(3615, ‘select...’) mozilla
    • Home page 20+ addons400+ sql queries mozilla
    • add-on version version version filesstandard answer indjango is select-related files files mozilla
    • django: select_related()http://bit.ly/select-related mozilla
    • simonwTransformerhttp://bit.ly/queryset-transform mozilla
    • @staticmethoddef transformer(addons): addon_dict = dict((a.id, a) for a in addons) vs = filter(None, (a.current_version_id for a in addons) versions = list(Version.objects.filter(id__in=vs)) for version in versions: addon_dict[version.addon_id].current_version = version slightly outdated example mozilla
    • big SQL statements... :( 14313 characterHome page 20+ addons~14 sql queries mozilla
    • Update Called on startupabout:config extensions.update.url mozilla
    • Incoming 8,000 req/secUncached 1,600 req/sec Im used to Plone in the bad old days mozilla
    • Incoming 8,000 req/secUncached 1,600 req/sec PHP 550 req/sec Im used to Plone in the bad old days mozilla
    • v1 Plain DjangoPHP Python mozilla
    • v2 Min. SQL queriesPHP Python mozilla
    • oh god mozilla
    • v3 Django and raw SQL max-requests 200, actually we hit 210PHP Python mozilla
    • v4 WSGI no DjangoPHP Python mozilla
    • v5 Pooling, optimised queries Thats 700 req/secPHP Python which translates into mozilla
    • Reducing the SQL queries... doesn’t always help mozilla
    • mySQL query cache is fast mozilla
    • Celery http://celeryproject.orgCredit: http://www.flickr.com/photos/chiotsrun/3843988392/ mozilla
    • Push things async email image processing add-on validation specifically fixing data changes bet ween php and python mozilla
    • from celeryutils import task@taskdef update_tag(tag, **kw): tag.update_stat() mozilla
    • from celeryutils import task@task(rate_limit=60/h)def update_tag(tag, **kw): tag.update_stat() mozilla
    • from tasks import update_tagupdate_tag.delay(tag) mozilla
    • Measurement mozilla
    • Timing Middleware http://bit.ly/timing-wareCredit: http://www.flickr.com/photos/wwarby/3297205226/ mozilla
    • mozilla
    • 3. Localization mozilla
    • show site in arabic?40+ languages including rtl mozilla
    • content translated and templatesDatabase strings mozilla
    • class Addon(caching.base.CachingMixin, models.Model): ... name = models.ForeignKey(Translation) mozilla
    • addon.name = nameaddon.save() mozilla
    • addon.name = nameaddon.save()addon.name = {fr: la nomme}addon.save() mozilla
    • Templates mozilla
    • Django{% blocktrans with app=request.APP %} Add-ons for {{ app }} {% endblocktrans %} mozilla
    • Jinja2 http://jinja.pocoo.org/{{ _(Add-ons for {0})|f(request.APP) }} mozilla
    • Python Unicode hellUnicodeDecodeError: ascii codec cantdecode byte 0xd0 in position 16: ordinal not in range(128) mozilla
    • 4. Reuse mozilla
    • Bleach http://pypi.python.org/pypi/Credit: http://www.flickr.com/photos/maisonbisson/3350954463/ mozilla
    • >>> bleach.clean(an<script>evil()</script>example)an &lt;script&gt;evil()&lt;/script&gt; example mozilla
    • >>> bleach.linkify(an http://ex.com url)an <a href="http://ex.com"rel="nofollow">http://ex.com</a> url mozilla
    • Javascript tests mozilla
    • django-qunithttp://bit.ly/django-qunit kumar mozilla
    • test(English, function() {    z.refreshL10n(en-us);    equals($(textarea:visible, this.sandbox).text().trim(),           Firebug integrates with Firefox to put + a wealth of development tools...);}); mozilla
    • test(Japanese, function() {    z.refreshL10n(ja);    equals($(textarea:visible, this.sandbox).text().trim(),           Firebug Web + Firefox +           );}); mozilla
    • So we use hudson for CI, but haven’t got the automated tests in yet Hoping to do this via jstestnetUse HudsonJenkins http://bit.ly/jstestnet mozilla
    • pep 8 py flakes MacCabe Flake8 http://bit.ly/flake8Credit: http://www.flickr.com/photos/nebarnix/357779131/ mozilla
    • ~/sandboxes/zamboni(632719) $ flake8 apps/editors/tasks.pyapps/editors/tasks.py:1: datetime imported but unusedapps/editors/tasks.py:3: stat imported but unused mozilla
    • Playdoh http://bit.ly/mozilla-playdoh fred wenzelCredit: http://www.flickr.com/photos/ahmee/97960570/ mozilla
    • Basis for new projects mozilla
    • Celery supportJinja2 supportSimple migrationsBy default: SHA-512 password hashing X-Frame-Options: DENY secure and httponly flags on cookies fred wenzel mozilla
    • Take inspiration from...but not the best for you for example jinja2 which makes integration with lots of django addons possible, but a bit harder mozilla
    • Questions? @andymckay andym@mozilla.comandym on irc.freenode.net, irc.mozilla.org mozilla