Anatomy of a large Django site
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Anatomy of a large Django site

on

  • 11,634 views

 

Statistics

Views

Total Views
11,634
Views on SlideShare
11,611
Embed Views
23

Actions

Likes
15
Downloads
69
Comments
1

4 Embeds 23

http://127.0.0.1 20
http://a0.twimg.com 1
https://twimg0-a.akamaihd.net 1
https://twitter.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Anatomy of a large Django site Presentation Transcript

  • 1. Anatomy of a large Django siteAndy McKay Mozilla mozilla
  • 2. Vancouver mozilla
  • 3. PythonZope and Plone...now at Mozilla mozilla
  • 4. Using Django http://www.djangoproject.comCredit: http://www.flickr.com/photos/abiavati/3110357974/ mozilla
  • 5. 1. About the site2. Performance3. Localisation4. Reuse mozilla
  • 6. 1. About the site mozilla
  • 7. mozilla
  • 8. All code is open:https://github.com/jbalogh/zamboni mozilla
  • 9. All* bugs are open: https://bugzilla.mozilla.org mozilla
  • 10. Convert from CakePHP (remora)to Django (zamboni) mozilla
  • 11. Credit: http://www.flickr.com/photos/improbcat/4177702580/ mozilla
  • 12. Changing one URL at a time from CakePHP to Django General trend to move away from PHP and do more Python and Django mozilla
  • 13. How large? 250k+ addons150 mn views month500 mn API hits day mozilla
  • 14. Lines of code PHP 40kPython 18k mozilla
  • 15. Lines of code PHP 40k Python 18kUnit tests 15k mozilla
  • 16. running both php and python side by side. a few issues on thatNo pages go out until they are faster mozilla
  • 17. 3 zeus load balancers 24 django (and php) 1 mysql + 4 slaves 3 memcached 3 sphinx 1 rabbitmq + 2 celeryd 1 redis master + 1 slaveCredit: http://www.flickr.com/photos/tbridge/15300843/ mozilla
  • 18. 2. Performance mozilla
  • 19. As usual, database bottleneck mozilla
  • 20. Cache machine http://bit.ly/cache-machineCredit: http://www.flickr.com/photos/mwichary/4063534688/ mozilla
  • 21. from django.db import modelsimport caching.baseclass Addon(caching.base.CachingMixin, models.Model): ... status = models.IntegerField() objects = caching.base.CachingManager() available as a mixin need to addin the custom manager mozilla
  • 22. >>> Addon.objects.filter(status=public)>>> len(connection.queries) 13 mozilla
  • 23. >>> Addon.objects.filter(status=public)>>> len(connection.queries) 13>>> Addon.objects.filter(status=public)>>> len(connection.queries) 13 mozilla
  • 24. Invalidation mozilla
  • 25. md5(‘select... a’) [addon 3615] mozilla
  • 26. md5(‘select... a’) [addon 3615] addon 3615 md5(‘select... a’) mozilla
  • 27. md5(‘select... a’) [addon 3615]md5(‘select... b’) [addon 3615, addon 1685] md5(‘select... a’) addon 3615 md5(‘select... b’) mozilla
  • 28. Memcachedrules = cache.get(3615)rules.add(select...)cache.set(3615, rules) mozilla
  • 29. Redisredis.SADD(3615, ‘select...’) mozilla
  • 30. Home page 20+ addons400+ sql queries mozilla
  • 31. add-on version version version filesstandard answer indjango is select-related files files mozilla
  • 32. django: select_related()http://bit.ly/select-related mozilla
  • 33. simonwTransformerhttp://bit.ly/queryset-transform mozilla
  • 34. @staticmethoddef transformer(addons): addon_dict = dict((a.id, a) for a in addons) vs = filter(None, (a.current_version_id for a in addons) versions = list(Version.objects.filter(id__in=vs)) for version in versions: addon_dict[version.addon_id].current_version = version slightly outdated example mozilla
  • 35. big SQL statements... :( 14313 characterHome page 20+ addons~14 sql queries mozilla
  • 36. Update Called on startupabout:config extensions.update.url mozilla
  • 37. Incoming 8,000 req/secUncached 1,600 req/sec Im used to Plone in the bad old days mozilla
  • 38. Incoming 8,000 req/secUncached 1,600 req/sec PHP 550 req/sec Im used to Plone in the bad old days mozilla
  • 39. v1 Plain DjangoPHP Python mozilla
  • 40. v2 Min. SQL queriesPHP Python mozilla
  • 41. oh god mozilla
  • 42. v3 Django and raw SQL max-requests 200, actually we hit 210PHP Python mozilla
  • 43. v4 WSGI no DjangoPHP Python mozilla
  • 44. v5 Pooling, optimised queries Thats 700 req/secPHP Python which translates into mozilla
  • 45. Reducing the SQL queries... doesn’t always help mozilla
  • 46. mySQL query cache is fast mozilla
  • 47. Celery http://celeryproject.orgCredit: http://www.flickr.com/photos/chiotsrun/3843988392/ mozilla
  • 48. Push things async email image processing add-on validation specifically fixing data changes bet ween php and python mozilla
  • 49. from celeryutils import task@taskdef update_tag(tag, **kw): tag.update_stat() mozilla
  • 50. from celeryutils import task@task(rate_limit=60/h)def update_tag(tag, **kw): tag.update_stat() mozilla
  • 51. from tasks import update_tagupdate_tag.delay(tag) mozilla
  • 52. Measurement mozilla
  • 53. Timing Middleware http://bit.ly/timing-wareCredit: http://www.flickr.com/photos/wwarby/3297205226/ mozilla
  • 54. mozilla
  • 55. 3. Localization mozilla
  • 56. show site in arabic?40+ languages including rtl mozilla
  • 57. content translated and templatesDatabase strings mozilla
  • 58. class Addon(caching.base.CachingMixin, models.Model): ... name = models.ForeignKey(Translation) mozilla
  • 59. addon.name = nameaddon.save() mozilla
  • 60. addon.name = nameaddon.save()addon.name = {fr: la nomme}addon.save() mozilla
  • 61. Templates mozilla
  • 62. Django{% blocktrans with app=request.APP %} Add-ons for {{ app }} {% endblocktrans %} mozilla
  • 63. Jinja2 http://jinja.pocoo.org/{{ _(Add-ons for {0})|f(request.APP) }} mozilla
  • 64. Python Unicode hellUnicodeDecodeError: ascii codec cantdecode byte 0xd0 in position 16: ordinal not in range(128) mozilla
  • 65. 4. Reuse mozilla
  • 66. Bleach http://pypi.python.org/pypi/Credit: http://www.flickr.com/photos/maisonbisson/3350954463/ mozilla
  • 67. >>> bleach.clean(an<script>evil()</script>example)an &lt;script&gt;evil()&lt;/script&gt; example mozilla
  • 68. >>> bleach.linkify(an http://ex.com url)an <a href="http://ex.com"rel="nofollow">http://ex.com</a> url mozilla
  • 69. Javascript tests mozilla
  • 70. django-qunithttp://bit.ly/django-qunit kumar mozilla
  • 71. test(English, function() {    z.refreshL10n(en-us);    equals($(textarea:visible, this.sandbox).text().trim(),           Firebug integrates with Firefox to put + a wealth of development tools...);}); mozilla
  • 72. test(Japanese, function() {    z.refreshL10n(ja);    equals($(textarea:visible, this.sandbox).text().trim(),           Firebug Web + Firefox +           );}); mozilla
  • 73. So we use hudson for CI, but haven’t got the automated tests in yet Hoping to do this via jstestnetUse HudsonJenkins http://bit.ly/jstestnet mozilla
  • 74. pep 8 py flakes MacCabe Flake8 http://bit.ly/flake8Credit: http://www.flickr.com/photos/nebarnix/357779131/ mozilla
  • 75. ~/sandboxes/zamboni(632719) $ flake8 apps/editors/tasks.pyapps/editors/tasks.py:1: datetime imported but unusedapps/editors/tasks.py:3: stat imported but unused mozilla
  • 76. Playdoh http://bit.ly/mozilla-playdoh fred wenzelCredit: http://www.flickr.com/photos/ahmee/97960570/ mozilla
  • 77. Basis for new projects mozilla
  • 78. Celery supportJinja2 supportSimple migrationsBy default: SHA-512 password hashing X-Frame-Options: DENY secure and httponly flags on cookies fred wenzel mozilla
  • 79. Take inspiration from...but not the best for you for example jinja2 which makes integration with lots of django addons possible, but a bit harder mozilla
  • 80. Questions? @andymckay andym@mozilla.comandym on irc.freenode.net, irc.mozilla.org mozilla