Djangocon

4,115 views
3,983 views

Published on

0 Comments
16 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
4,115
On SlideShare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
63
Comments
0
Likes
16
Embeds 0
No embeds

No notes for slide

  • Python since 2007
    Django since 2008
    Started at Mozilla in 2009

    http://www.flickr.com/photos/morgamic/4466526784/sizes/o/in/set-72157623711339916/

  • trusted place for hosting for Firefox add-ons
    sorting, searching, discovery
    the app store without the store part (at this point)

  • This talk is about the problems we’ve solved getting Django to power a large website









  • Pinning






  • what is the race condition?
  • memcached is not the right place to store sets
    use redis



  • cron/celery/signals
  • /extensions: 14 vs. 150+ queries
  • ideas from batch_select
    pick up any relations you’re going to use a lot: foreign keys, m2m, reverse relations
  • Besides denormalization and indexes, I’m out of ideas for making the database go faster.
    Now we’ll go into some other things we do differently from a vanilla Django project.




  • 700 tests run after every commit with Hudson
    RadicalTestSuiteRunner
    fixtures (hudson trend graph?)








  • 400 tests in remora, 700 in zamboni
    2,3,5x as fast


  • sharing code, 4 production site

  • python developers who love the web, js/frontend devs, operations, other groups


  • Djangocon

    1. 1. Rewriting addons.mozilla.org with Django Jeff Balogh github.com/jbalogh @jeffbalogh
    2. 2. Oh god how did I get here I am not good with PHP
    3. 3. addons.mozilla.org AMO github.com/jbalogh/zamboni
    4. 4. addons.mozilla.org • 165 million request per month
    5. 5. Hardware • 3 Zeus load balancers • 24 web servers • mod_wsgi: 16 procs, no threading • 1 MySQL master, 4 slaves • 3 memcached • 3 sphinx • 1 RabbitMQ, 2 celeryd • 1 Redis master, 1 slave
    6. 6. How we’re switching • One url at a time • PHP & Python run side by side
    7. 7. class Addon(models.Model): name = models.ForeignKey('Translation') description = models.ForeignKey('Translation') authors = models.ManyToManyField('User') # Denormalization current_version = models.ForeignKey('Version') class Translation(models.Model): locale = models.CharField() localized_string = models.CharField() class Version(models.Model): addon = models.ForeignKey(Addon) version = models.CharField()
    8. 8. class Addon(models.Model): name = models.ForeignKey('Translation') description = models.ForeignKey('Translation') authors = models.ManyToManyField('User') # Denormalization current_version = models.ForeignKey('Version') class Translation(models.Model): locale = models.CharField() localized_string = models.CharField() class Version(models.Model): addon = models.ForeignKey(Addon) version = models.CharField()
    9. 9. multidb • github.com/jbalogh/django-multidb- router • Too easy now (thanks Alex & Russel)
    10. 10. Object caching • github.com/jbalogh/django-cache- machine/ • automatically cache all queries
    11. 11. Invalidation
    12. 12. SELECT * from addons ORDER BY rating DESC LIMIT 10 [<Addon 1865: Adblock Plus>, …] md5(SELECT…rating…) [<Addon 1865>, …]
    13. 13. SELECT * from addons ORDER BY rating DESC LIMIT 10 [<Addon 1865: Adblock Plus>, …] md5(SELECT…rating…) [<Addon 1865>, …] flush:md5(<Addon 1865>) [md5(SELECT…rating…)]
    14. 14. SELECT * from addons WHERE id = 1865 <Addon 1865: Adblock Plus> md5(SELECT…rating…) [<Addon 1865>, …] flush:md5(<Addon 1865>) [md5(SELECT…rating…)] md5(SELECT…id=1865) <Addon 1865>
    15. 15. SELECT * from addons WHERE id = 1865 <Addon 1865: Adblock Plus> md5(SELECT…rating…) [<Addon 1865>, …] [md5(SELECT…rating…), flush:md5(<Addon 1865>) md5(SELECT…id=1865)] md5(SELECT…id=1865) <Addon 1865>
    16. 16. cache.set_many({ md5(SELECT…id=1865): None, md5(SELECT…rating…): None } md5(SELECT…rating…) [<Addon 1865>, …] [md5(SELECT…rating…), flush:md5(<Addon 1865>) md5(SELECT…id=1865)] md5(SELECT…id=1865) <Addon 1865>
    17. 17. Querysets are big
    18. 18. Querysets are big • cached_with(queryset, function) • {% cache obj %} • From 29r/s to 186r/s
    19. 19. Related objects • addon.current_version
    20. 20. Related objects • github.com/simonw/django-queryset- transform/ • select_related on steroids
    21. 21. Transforms Addon.objects.transform(Addon.transformer) @staticmethod def transformer(addons): addon_dict = dict((a.id, a) for a in addons) vs = filter(None, (a.current_version_id for a in addons)) versions = list(Version.objects.filter(id__in=vs)) for version in versions: addon_dict[version.addon_id].current_version = version
    22. 22. Templates • {% load %} • {% blocktrans with article.price as amount %} • {% ifnotequal %}
    23. 23. Use Jinja2 • github.com/jbalogh/jingo • Tags & filters are just functions • {{ _('Add-ons for {0}')|f(app) }}
    24. 24. Tests • github.com/jbalogh/django-nose • nose makes testing easier
    25. 25. UnicodeDecodeErro r • UnicodeEncodeError: 'ascii' codec can't encode characters in position 64-72: ordinal not in range(128)
    26. 26. UnicodeDecodeErro r • UnicodeEncodeError: 'ascii' codec can't encode characters in position 64-72: ordinal not in range(128)
    27. 27. Django issues • .filter() is slow • DELETE CASCADE • SELECT before UPDATE
    28. 28. Django issues • .filter() is slow • DELETE CASCADE • SELECT before UPDATE
    29. 29. Django issues • .filter() is slow • DELETE CASCADE • SELECT before UPDATE
    30. 30. Does Django scale? • Yes, with help • We don’t ship unless it’s faster than PHP • 44,000 lines of PHP vs. 12,500 lines of Python
    31. 31. Django @ Mozilla • “I've been a PHP developer for 8 years and a Python developer for 6 months, and I don't want to go back.”
    32. 32. http://www.flickr.com/photos/wayneandwax/4731112862/
    33. 33. Jobs Jobs Jobs • mzl.la/djangocon • bit.ly/djangocon • bit.ly uses a single database for all their links!
    34. 34. Thank You • github.com/jbalogh • jbalogh.me/djangocon.pdf (slides)
    35. 35. References github.com/jbalogh github.com/jbalogh/zamboni github.com/jbalogh/django-multidb-router github.com/jbalogh/django-cache-machine/ github.com/jbalogh/jingo github.com/jbalogh/django-nose github.com/simonw/django-queryset-transform/ mzl.la/djangocon github.com/mozilla jbalogh.me/djangocon.pdf (slides) jbalogh.me/djangocon.txt (this page)

    ×