Anatomy of a large   Django siteAndy McKay Mozilla                     mozilla
Vancouver            mozilla
PythonZope and Plone...now at Mozilla                    mozilla
Using Django         http://www.djangoproject.comCredit: http://www.flickr.com/photos/abiavati/3110357974/                 ...
1. About the site2. Performance3. Localisation4. Reuse                    mozilla
1. About the site                    mozilla
mozilla
All code is open:https://github.com/jbalogh/zamboni                                     mozilla
All* bugs are open: https://bugzilla.mozilla.org                                mozilla
Convert from CakePHP (remora)to Django (zamboni)                      mozilla
Credit: http://www.flickr.com/photos/improbcat/4177702580/                                                            mozilla
Changing one URL at a time from CakePHP to Django            General trend to move            away from PHP and do        ...
How large?   250k+ addons150 mn views month500 mn API hits day                      mozilla
Lines of code   PHP 40kPython 18k                mozilla
Lines of code      PHP 40k  Python 18kUnit tests 15k                 mozilla
running both php and             python side by side. a             few issues on thatNo pages go out until   they are fas...
3 zeus load balancers                                 24 django (and php)                                 1 mysql + 4 slav...
2. Performance                 mozilla
As usual, database bottleneck                            mozilla
Cache machine   http://bit.ly/cache-machineCredit: http://www.flickr.com/photos/mwichary/4063534688/                       ...
from django.db import modelsimport caching.baseclass Addon(caching.base.CachingMixin,            models.Model):    ...    ...
>>> Addon.objects.filter(status=public)>>> len(connection.queries)    13                                          mozilla
>>> Addon.objects.filter(status=public)>>> len(connection.queries)    13>>> Addon.objects.filter(status=public)>>> len(con...
Invalidation               mozilla
md5(‘select... a’)   [addon 3615]                                    mozilla
md5(‘select... a’)      [addon 3615]    addon 3615       md5(‘select... a’)                                          mozilla
md5(‘select... a’)         [addon 3615]md5(‘select... b’)   [addon 3615, addon 1685]                        md5(‘select......
Memcachedrules = cache.get(3615)rules.add(select...)cache.set(3615, rules)                          mozilla
Redisredis.SADD(3615, ‘select...’)                           mozilla
Home page  20+ addons400+ sql queries                   mozilla
add-on                     version                              version                                     version   files...
django: select_related()http://bit.ly/select-related                               mozilla
simonwTransformerhttp://bit.ly/queryset-transform                                      mozilla
@staticmethoddef transformer(addons):    addon_dict = dict((a.id, a) for a in addons)    vs = filter(None, (a.current_vers...
big SQL statements... :(                  14313 characterHome page 20+ addons~14 sql queries                              ...
Update   Called on startupabout:config   extensions.update.url                                      mozilla
Incoming 8,000 req/secUncached 1,600 req/sec           Im used to Plone in the           bad old days                     ...
Incoming 8,000 req/secUncached 1,600 req/sec     PHP 550 req/sec           Im used to Plone in the           bad old days ...
v1                 Plain               DjangoPHP   Python                   mozilla
v2               Min. SQL                queriesPHP   Python                    mozilla
oh god         mozilla
v3               Django and                  raw SQL               max-requests 200,               actually we hit 210PHP ...
v4                   WSGI               no DjangoPHP   Python                     mozilla
v5                             Pooling,                           optimised                             queries           ...
Reducing the SQL queries...   doesn’t always help                          mozilla
mySQL query cache is fast                        mozilla
Celery                                                       http://celeryproject.orgCredit: http://www.flickr.com/photos/c...
Push things async       email  image processing  add-on validation                      specifically fixing data          ...
from celeryutils import task@taskdef update_tag(tag, **kw):    tag.update_stat()                             mozilla
from celeryutils import task@task(rate_limit=60/h)def update_tag(tag, **kw):    tag.update_stat()                         ...
from tasks import update_tagupdate_tag.delay(tag)                          mozilla
Measurement              mozilla
Timing Middleware                                                         http://bit.ly/timing-wareCredit: http://www.flick...
mozilla
3. Localization                  mozilla
show site in arabic?40+ languages including rtl                               mozilla
content translated          and          templatesDatabase strings                               mozilla
class Addon(caching.base.CachingMixin,            models.Model):    ...    name = models.ForeignKey(Translation)          ...
addon.name = nameaddon.save()                      mozilla
addon.name = nameaddon.save()addon.name = {fr: la nomme}addon.save()                                  mozilla
Templates            mozilla
Django{% blocktrans with app=request.APP %}        Add-ons for {{ app }}         {% endblocktrans %}                      ...
Jinja2          http://jinja.pocoo.org/{{ _(Add-ons for {0})|f(request.APP) }}                                     mozilla
Python Unicode hellUnicodeDecodeError: ascii codec cantdecode byte 0xd0 in position 16: ordinal            not in range(12...
4. Reuse           mozilla
Bleach http://pypi.python.org/pypi/Credit: http://www.flickr.com/photos/maisonbisson/3350954463/                           ...
>>> bleach.clean(an<script>evil()</script>example)an &lt;script&gt;evil()&lt;/script&gt; example                          ...
>>> bleach.linkify(an http://ex.com url)an <a href="http://ex.com"rel="nofollow">http://ex.com</a> url                    ...
Javascript tests                   mozilla
django-qunithttp://bit.ly/django-qunit                             kumar                                     mozilla
test(English, function() {    z.refreshL10n(en-us);    equals($(textarea:visible, this.sandbox).text().trim(),           F...
test(Japanese, function() {    z.refreshL10n(ja);    equals($(textarea:visible, this.sandbox).text().trim(),           Fir...
So we use hudson for CI,                but haven’t got the                automated tests in yet                Hoping to...
pep 8                                                          py flakes                                                  ...
~/sandboxes/zamboni(632719) $ flake8 apps/editors/tasks.pyapps/editors/tasks.py:1: datetime imported but unusedapps/editor...
Playdoh                                       http://bit.ly/mozilla-playdoh                                               ...
Basis for new projects                     mozilla
Celery supportJinja2 supportSimple migrationsBy default:		 SHA-512 password hashing		 X-Frame-Options: DENY		 secure and h...
Take inspiration from...but not the best for you                  for example jinja2 which                  makes integrat...
Questions?              @andymckay         andym@mozilla.comandym on irc.freenode.net, irc.mozilla.org                    ...
Upcoming SlideShare
Loading in...5
×

Anatomy of a large Django site

11,742

Published on

Published in: Technology
1 Comment
16 Likes
Statistics
Notes
No Downloads
Views
Total Views
11,742
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
84
Comments
1
Likes
16
Embeds 0
No embeds

No notes for slide

Anatomy of a large Django site

  1. 1. Anatomy of a large Django siteAndy McKay Mozilla mozilla
  2. 2. Vancouver mozilla
  3. 3. PythonZope and Plone...now at Mozilla mozilla
  4. 4. Using Django http://www.djangoproject.comCredit: http://www.flickr.com/photos/abiavati/3110357974/ mozilla
  5. 5. 1. About the site2. Performance3. Localisation4. Reuse mozilla
  6. 6. 1. About the site mozilla
  7. 7. mozilla
  8. 8. All code is open:https://github.com/jbalogh/zamboni mozilla
  9. 9. All* bugs are open: https://bugzilla.mozilla.org mozilla
  10. 10. Convert from CakePHP (remora)to Django (zamboni) mozilla
  11. 11. Credit: http://www.flickr.com/photos/improbcat/4177702580/ mozilla
  12. 12. Changing one URL at a time from CakePHP to Django General trend to move away from PHP and do more Python and Django mozilla
  13. 13. How large? 250k+ addons150 mn views month500 mn API hits day mozilla
  14. 14. Lines of code PHP 40kPython 18k mozilla
  15. 15. Lines of code PHP 40k Python 18kUnit tests 15k mozilla
  16. 16. running both php and python side by side. a few issues on thatNo pages go out until they are faster mozilla
  17. 17. 3 zeus load balancers 24 django (and php) 1 mysql + 4 slaves 3 memcached 3 sphinx 1 rabbitmq + 2 celeryd 1 redis master + 1 slaveCredit: http://www.flickr.com/photos/tbridge/15300843/ mozilla
  18. 18. 2. Performance mozilla
  19. 19. As usual, database bottleneck mozilla
  20. 20. Cache machine http://bit.ly/cache-machineCredit: http://www.flickr.com/photos/mwichary/4063534688/ mozilla
  21. 21. from django.db import modelsimport caching.baseclass Addon(caching.base.CachingMixin, models.Model): ... status = models.IntegerField() objects = caching.base.CachingManager() available as a mixin need to addin the custom manager mozilla
  22. 22. >>> Addon.objects.filter(status=public)>>> len(connection.queries) 13 mozilla
  23. 23. >>> Addon.objects.filter(status=public)>>> len(connection.queries) 13>>> Addon.objects.filter(status=public)>>> len(connection.queries) 13 mozilla
  24. 24. Invalidation mozilla
  25. 25. md5(‘select... a’) [addon 3615] mozilla
  26. 26. md5(‘select... a’) [addon 3615] addon 3615 md5(‘select... a’) mozilla
  27. 27. md5(‘select... a’) [addon 3615]md5(‘select... b’) [addon 3615, addon 1685] md5(‘select... a’) addon 3615 md5(‘select... b’) mozilla
  28. 28. Memcachedrules = cache.get(3615)rules.add(select...)cache.set(3615, rules) mozilla
  29. 29. Redisredis.SADD(3615, ‘select...’) mozilla
  30. 30. Home page 20+ addons400+ sql queries mozilla
  31. 31. add-on version version version filesstandard answer indjango is select-related files files mozilla
  32. 32. django: select_related()http://bit.ly/select-related mozilla
  33. 33. simonwTransformerhttp://bit.ly/queryset-transform mozilla
  34. 34. @staticmethoddef transformer(addons): addon_dict = dict((a.id, a) for a in addons) vs = filter(None, (a.current_version_id for a in addons) versions = list(Version.objects.filter(id__in=vs)) for version in versions: addon_dict[version.addon_id].current_version = version slightly outdated example mozilla
  35. 35. big SQL statements... :( 14313 characterHome page 20+ addons~14 sql queries mozilla
  36. 36. Update Called on startupabout:config extensions.update.url mozilla
  37. 37. Incoming 8,000 req/secUncached 1,600 req/sec Im used to Plone in the bad old days mozilla
  38. 38. Incoming 8,000 req/secUncached 1,600 req/sec PHP 550 req/sec Im used to Plone in the bad old days mozilla
  39. 39. v1 Plain DjangoPHP Python mozilla
  40. 40. v2 Min. SQL queriesPHP Python mozilla
  41. 41. oh god mozilla
  42. 42. v3 Django and raw SQL max-requests 200, actually we hit 210PHP Python mozilla
  43. 43. v4 WSGI no DjangoPHP Python mozilla
  44. 44. v5 Pooling, optimised queries Thats 700 req/secPHP Python which translates into mozilla
  45. 45. Reducing the SQL queries... doesn’t always help mozilla
  46. 46. mySQL query cache is fast mozilla
  47. 47. Celery http://celeryproject.orgCredit: http://www.flickr.com/photos/chiotsrun/3843988392/ mozilla
  48. 48. Push things async email image processing add-on validation specifically fixing data changes bet ween php and python mozilla
  49. 49. from celeryutils import task@taskdef update_tag(tag, **kw): tag.update_stat() mozilla
  50. 50. from celeryutils import task@task(rate_limit=60/h)def update_tag(tag, **kw): tag.update_stat() mozilla
  51. 51. from tasks import update_tagupdate_tag.delay(tag) mozilla
  52. 52. Measurement mozilla
  53. 53. Timing Middleware http://bit.ly/timing-wareCredit: http://www.flickr.com/photos/wwarby/3297205226/ mozilla
  54. 54. mozilla
  55. 55. 3. Localization mozilla
  56. 56. show site in arabic?40+ languages including rtl mozilla
  57. 57. content translated and templatesDatabase strings mozilla
  58. 58. class Addon(caching.base.CachingMixin, models.Model): ... name = models.ForeignKey(Translation) mozilla
  59. 59. addon.name = nameaddon.save() mozilla
  60. 60. addon.name = nameaddon.save()addon.name = {fr: la nomme}addon.save() mozilla
  61. 61. Templates mozilla
  62. 62. Django{% blocktrans with app=request.APP %} Add-ons for {{ app }} {% endblocktrans %} mozilla
  63. 63. Jinja2 http://jinja.pocoo.org/{{ _(Add-ons for {0})|f(request.APP) }} mozilla
  64. 64. Python Unicode hellUnicodeDecodeError: ascii codec cantdecode byte 0xd0 in position 16: ordinal not in range(128) mozilla
  65. 65. 4. Reuse mozilla
  66. 66. Bleach http://pypi.python.org/pypi/Credit: http://www.flickr.com/photos/maisonbisson/3350954463/ mozilla
  67. 67. >>> bleach.clean(an<script>evil()</script>example)an &lt;script&gt;evil()&lt;/script&gt; example mozilla
  68. 68. >>> bleach.linkify(an http://ex.com url)an <a href="http://ex.com"rel="nofollow">http://ex.com</a> url mozilla
  69. 69. Javascript tests mozilla
  70. 70. django-qunithttp://bit.ly/django-qunit kumar mozilla
  71. 71. test(English, function() {    z.refreshL10n(en-us);    equals($(textarea:visible, this.sandbox).text().trim(),           Firebug integrates with Firefox to put + a wealth of development tools...);}); mozilla
  72. 72. test(Japanese, function() {    z.refreshL10n(ja);    equals($(textarea:visible, this.sandbox).text().trim(),           Firebug Web + Firefox +           );}); mozilla
  73. 73. So we use hudson for CI, but haven’t got the automated tests in yet Hoping to do this via jstestnetUse HudsonJenkins http://bit.ly/jstestnet mozilla
  74. 74. pep 8 py flakes MacCabe Flake8 http://bit.ly/flake8Credit: http://www.flickr.com/photos/nebarnix/357779131/ mozilla
  75. 75. ~/sandboxes/zamboni(632719) $ flake8 apps/editors/tasks.pyapps/editors/tasks.py:1: datetime imported but unusedapps/editors/tasks.py:3: stat imported but unused mozilla
  76. 76. Playdoh http://bit.ly/mozilla-playdoh fred wenzelCredit: http://www.flickr.com/photos/ahmee/97960570/ mozilla
  77. 77. Basis for new projects mozilla
  78. 78. Celery supportJinja2 supportSimple migrationsBy default: SHA-512 password hashing X-Frame-Options: DENY secure and httponly flags on cookies fred wenzel mozilla
  79. 79. Take inspiration from...but not the best for you for example jinja2 which makes integration with lots of django addons possible, but a bit harder mozilla
  80. 80. Questions? @andymckay andym@mozilla.comandym on irc.freenode.net, irc.mozilla.org mozilla
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×