Djangocon

  • 3,449 views
Uploaded on

 

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
3,449
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
62
Comments
0
Likes
16

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

  • Python since 2007
    Django since 2008
    Started at Mozilla in 2009

    http://www.flickr.com/photos/morgamic/4466526784/sizes/o/in/set-72157623711339916/

  • trusted place for hosting for Firefox add-ons
    sorting, searching, discovery
    the app store without the store part (at this point)

  • This talk is about the problems we’ve solved getting Django to power a large website









  • Pinning






  • what is the race condition?
  • memcached is not the right place to store sets
    use redis



  • cron/celery/signals
  • /extensions: 14 vs. 150+ queries
  • ideas from batch_select
    pick up any relations you’re going to use a lot: foreign keys, m2m, reverse relations
  • Besides denormalization and indexes, I’m out of ideas for making the database go faster.
    Now we’ll go into some other things we do differently from a vanilla Django project.




  • 700 tests run after every commit with Hudson
    RadicalTestSuiteRunner
    fixtures (hudson trend graph?)








  • 400 tests in remora, 700 in zamboni
    2,3,5x as fast


  • sharing code, 4 production site

  • python developers who love the web, js/frontend devs, operations, other groups


Transcript

  • 1. Rewriting addons.mozilla.org with Django Jeff Balogh github.com/jbalogh @jeffbalogh
  • 2. Oh god how did I get here I am not good with PHP
  • 3. addons.mozilla.org AMO github.com/jbalogh/zamboni
  • 4. addons.mozilla.org • 165 million request per month
  • 5. Hardware • 3 Zeus load balancers • 24 web servers • mod_wsgi: 16 procs, no threading • 1 MySQL master, 4 slaves • 3 memcached • 3 sphinx • 1 RabbitMQ, 2 celeryd • 1 Redis master, 1 slave
  • 6. How we’re switching • One url at a time • PHP & Python run side by side
  • 7. class Addon(models.Model): name = models.ForeignKey('Translation') description = models.ForeignKey('Translation') authors = models.ManyToManyField('User') # Denormalization current_version = models.ForeignKey('Version') class Translation(models.Model): locale = models.CharField() localized_string = models.CharField() class Version(models.Model): addon = models.ForeignKey(Addon) version = models.CharField()
  • 8. class Addon(models.Model): name = models.ForeignKey('Translation') description = models.ForeignKey('Translation') authors = models.ManyToManyField('User') # Denormalization current_version = models.ForeignKey('Version') class Translation(models.Model): locale = models.CharField() localized_string = models.CharField() class Version(models.Model): addon = models.ForeignKey(Addon) version = models.CharField()
  • 9. multidb • github.com/jbalogh/django-multidb- router • Too easy now (thanks Alex & Russel)
  • 10. Object caching • github.com/jbalogh/django-cache- machine/ • automatically cache all queries
  • 11. Invalidation
  • 12. SELECT * from addons ORDER BY rating DESC LIMIT 10 [<Addon 1865: Adblock Plus>, …] md5(SELECT…rating…) [<Addon 1865>, …]
  • 13. SELECT * from addons ORDER BY rating DESC LIMIT 10 [<Addon 1865: Adblock Plus>, …] md5(SELECT…rating…) [<Addon 1865>, …] flush:md5(<Addon 1865>) [md5(SELECT…rating…)]
  • 14. SELECT * from addons WHERE id = 1865 <Addon 1865: Adblock Plus> md5(SELECT…rating…) [<Addon 1865>, …] flush:md5(<Addon 1865>) [md5(SELECT…rating…)] md5(SELECT…id=1865) <Addon 1865>
  • 15. SELECT * from addons WHERE id = 1865 <Addon 1865: Adblock Plus> md5(SELECT…rating…) [<Addon 1865>, …] [md5(SELECT…rating…), flush:md5(<Addon 1865>) md5(SELECT…id=1865)] md5(SELECT…id=1865) <Addon 1865>
  • 16. cache.set_many({ md5(SELECT…id=1865): None, md5(SELECT…rating…): None } md5(SELECT…rating…) [<Addon 1865>, …] [md5(SELECT…rating…), flush:md5(<Addon 1865>) md5(SELECT…id=1865)] md5(SELECT…id=1865) <Addon 1865>
  • 17. Querysets are big
  • 18. Querysets are big • cached_with(queryset, function) • {% cache obj %} • From 29r/s to 186r/s
  • 19. Related objects • addon.current_version
  • 20. Related objects • github.com/simonw/django-queryset- transform/ • select_related on steroids
  • 21. Transforms Addon.objects.transform(Addon.transformer) @staticmethod def transformer(addons): addon_dict = dict((a.id, a) for a in addons) vs = filter(None, (a.current_version_id for a in addons)) versions = list(Version.objects.filter(id__in=vs)) for version in versions: addon_dict[version.addon_id].current_version = version
  • 22. Templates • {% load %} • {% blocktrans with article.price as amount %} • {% ifnotequal %}
  • 23. Use Jinja2 • github.com/jbalogh/jingo • Tags & filters are just functions • {{ _('Add-ons for {0}')|f(app) }}
  • 24. Tests • github.com/jbalogh/django-nose • nose makes testing easier
  • 25. UnicodeDecodeErro r • UnicodeEncodeError: 'ascii' codec can't encode characters in position 64-72: ordinal not in range(128)
  • 26. UnicodeDecodeErro r • UnicodeEncodeError: 'ascii' codec can't encode characters in position 64-72: ordinal not in range(128)
  • 27. Django issues • .filter() is slow • DELETE CASCADE • SELECT before UPDATE
  • 28. Django issues • .filter() is slow • DELETE CASCADE • SELECT before UPDATE
  • 29. Django issues • .filter() is slow • DELETE CASCADE • SELECT before UPDATE
  • 30. Does Django scale? • Yes, with help • We don’t ship unless it’s faster than PHP • 44,000 lines of PHP vs. 12,500 lines of Python
  • 31. Django @ Mozilla • “I've been a PHP developer for 8 years and a Python developer for 6 months, and I don't want to go back.”
  • 32. http://www.flickr.com/photos/wayneandwax/4731112862/
  • 33. Jobs Jobs Jobs • mzl.la/djangocon • bit.ly/djangocon • bit.ly uses a single database for all their links!
  • 34. Thank You • github.com/jbalogh • jbalogh.me/djangocon.pdf (slides)
  • 35. References github.com/jbalogh github.com/jbalogh/zamboni github.com/jbalogh/django-multidb-router github.com/jbalogh/django-cache-machine/ github.com/jbalogh/jingo github.com/jbalogh/django-nose github.com/simonw/django-queryset-transform/ mzl.la/djangocon github.com/mozilla jbalogh.me/djangocon.pdf (slides) jbalogh.me/djangocon.txt (this page)