Djangocon
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
3,763
On Slideshare
3,763
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
62
Comments
0
Likes
16

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • <br />
  • Python since 2007 <br /> Django since 2008 <br /> Started at Mozilla in 2009 <br /> <br /> http://www.flickr.com/photos/morgamic/4466526784/sizes/o/in/set-72157623711339916/ <br />
  • <br />
  • trusted place for hosting for Firefox add-ons <br /> sorting, searching, discovery <br /> the app store without the store part (at this point) <br />
  • <br />
  • This talk is about the problems we&#x2019;ve solved getting Django to power a large website <br /> <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • Pinning <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • what is the race condition? <br />
  • memcached is not the right place to store sets <br /> use redis <br />
  • <br />
  • <br />
  • <br />
  • cron/celery/signals <br />
  • /extensions: 14 vs. 150+ queries <br />
  • ideas from batch_select <br /> pick up any relations you&#x2019;re going to use a lot: foreign keys, m2m, reverse relations <br />
  • Besides denormalization and indexes, I&#x2019;m out of ideas for making the database go faster. <br /> Now we&#x2019;ll go into some other things we do differently from a vanilla Django project. <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • 700 tests run after every commit with Hudson <br /> RadicalTestSuiteRunner <br /> fixtures (hudson trend graph?) <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • 400 tests in remora, 700 in zamboni <br /> 2,3,5x as fast <br />
  • <br />
  • <br />
  • sharing code, 4 production site <br />
  • <br />
  • python developers who love the web, js/frontend devs, operations, other groups <br />
  • <br />
  • <br />

Transcript

  • 1. Rewriting addons.mozilla.org with Django Jeff Balogh github.com/jbalogh @jeffbalogh
  • 2. Oh god how did I get here I am not good with PHP
  • 3. addons.mozilla.org AMO github.com/jbalogh/zamboni
  • 4. addons.mozilla.org • 165 million request per month
  • 5. Hardware • 3 Zeus load balancers • 24 web servers • mod_wsgi: 16 procs, no threading • 1 MySQL master, 4 slaves • 3 memcached • 3 sphinx • 1 RabbitMQ, 2 celeryd • 1 Redis master, 1 slave
  • 6. How we’re switching • One url at a time • PHP & Python run side by side
  • 7. class Addon(models.Model): name = models.ForeignKey('Translation') description = models.ForeignKey('Translation') authors = models.ManyToManyField('User') # Denormalization current_version = models.ForeignKey('Version') class Translation(models.Model): locale = models.CharField() localized_string = models.CharField() class Version(models.Model): addon = models.ForeignKey(Addon) version = models.CharField()
  • 8. class Addon(models.Model): name = models.ForeignKey('Translation') description = models.ForeignKey('Translation') authors = models.ManyToManyField('User') # Denormalization current_version = models.ForeignKey('Version') class Translation(models.Model): locale = models.CharField() localized_string = models.CharField() class Version(models.Model): addon = models.ForeignKey(Addon) version = models.CharField()
  • 9. multidb • github.com/jbalogh/django-multidb- router • Too easy now (thanks Alex & Russel)
  • 10. Object caching • github.com/jbalogh/django-cache- machine/ • automatically cache all queries
  • 11. Invalidation
  • 12. SELECT * from addons ORDER BY rating DESC LIMIT 10 [<Addon 1865: Adblock Plus>, …] md5(SELECT…rating…) [<Addon 1865>, …]
  • 13. SELECT * from addons ORDER BY rating DESC LIMIT 10 [<Addon 1865: Adblock Plus>, …] md5(SELECT…rating…) [<Addon 1865>, …] flush:md5(<Addon 1865>) [md5(SELECT…rating…)]
  • 14. SELECT * from addons WHERE id = 1865 <Addon 1865: Adblock Plus> md5(SELECT…rating…) [<Addon 1865>, …] flush:md5(<Addon 1865>) [md5(SELECT…rating…)] md5(SELECT…id=1865) <Addon 1865>
  • 15. SELECT * from addons WHERE id = 1865 <Addon 1865: Adblock Plus> md5(SELECT…rating…) [<Addon 1865>, …] [md5(SELECT…rating…), flush:md5(<Addon 1865>) md5(SELECT…id=1865)] md5(SELECT…id=1865) <Addon 1865>
  • 16. cache.set_many({ md5(SELECT…id=1865): None, md5(SELECT…rating…): None } md5(SELECT…rating…) [<Addon 1865>, …] [md5(SELECT…rating…), flush:md5(<Addon 1865>) md5(SELECT…id=1865)] md5(SELECT…id=1865) <Addon 1865>
  • 17. Querysets are big
  • 18. Querysets are big • cached_with(queryset, function) • {% cache obj %} • From 29r/s to 186r/s
  • 19. Related objects • addon.current_version
  • 20. Related objects • github.com/simonw/django-queryset- transform/ • select_related on steroids
  • 21. Transforms Addon.objects.transform(Addon.transformer) @staticmethod def transformer(addons): addon_dict = dict((a.id, a) for a in addons) vs = filter(None, (a.current_version_id for a in addons)) versions = list(Version.objects.filter(id__in=vs)) for version in versions: addon_dict[version.addon_id].current_version = version
  • 22. Templates • {% load %} • {% blocktrans with article.price as amount %} • {% ifnotequal %}
  • 23. Use Jinja2 • github.com/jbalogh/jingo • Tags & filters are just functions • {{ _('Add-ons for {0}')|f(app) }}
  • 24. Tests • github.com/jbalogh/django-nose • nose makes testing easier
  • 25. UnicodeDecodeErro r • UnicodeEncodeError: 'ascii' codec can't encode characters in position 64-72: ordinal not in range(128)
  • 26. UnicodeDecodeErro r • UnicodeEncodeError: 'ascii' codec can't encode characters in position 64-72: ordinal not in range(128)
  • 27. Django issues • .filter() is slow • DELETE CASCADE • SELECT before UPDATE
  • 28. Django issues • .filter() is slow • DELETE CASCADE • SELECT before UPDATE
  • 29. Django issues • .filter() is slow • DELETE CASCADE • SELECT before UPDATE
  • 30. Does Django scale? • Yes, with help • We don’t ship unless it’s faster than PHP • 44,000 lines of PHP vs. 12,500 lines of Python
  • 31. Django @ Mozilla • “I've been a PHP developer for 8 years and a Python developer for 6 months, and I don't want to go back.”
  • 32. http://www.flickr.com/photos/wayneandwax/4731112862/
  • 33. Jobs Jobs Jobs • mzl.la/djangocon • bit.ly/djangocon • bit.ly uses a single database for all their links!
  • 34. Thank You • github.com/jbalogh • jbalogh.me/djangocon.pdf (slides)
  • 35. References github.com/jbalogh github.com/jbalogh/zamboni github.com/jbalogh/django-multidb-router github.com/jbalogh/django-cache-machine/ github.com/jbalogh/jingo github.com/jbalogh/django-nose github.com/simonw/django-queryset-transform/ mzl.la/djangocon github.com/mozilla jbalogh.me/djangocon.pdf (slides) jbalogh.me/djangocon.txt (this page)