Djangocon
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Djangocon

on

  • 3,687 views

 

Statistics

Views

Total Views
3,687
Views on SlideShare
3,687
Embed Views
0

Actions

Likes
16
Downloads
62
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • <br />
  • Python since 2007 <br /> Django since 2008 <br /> Started at Mozilla in 2009 <br /> <br /> http://www.flickr.com/photos/morgamic/4466526784/sizes/o/in/set-72157623711339916/ <br />
  • <br />
  • trusted place for hosting for Firefox add-ons <br /> sorting, searching, discovery <br /> the app store without the store part (at this point) <br />
  • <br />
  • This talk is about the problems we&#x2019;ve solved getting Django to power a large website <br /> <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • Pinning <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • what is the race condition? <br />
  • memcached is not the right place to store sets <br /> use redis <br />
  • <br />
  • <br />
  • <br />
  • cron/celery/signals <br />
  • /extensions: 14 vs. 150+ queries <br />
  • ideas from batch_select <br /> pick up any relations you&#x2019;re going to use a lot: foreign keys, m2m, reverse relations <br />
  • Besides denormalization and indexes, I&#x2019;m out of ideas for making the database go faster. <br /> Now we&#x2019;ll go into some other things we do differently from a vanilla Django project. <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • 700 tests run after every commit with Hudson <br /> RadicalTestSuiteRunner <br /> fixtures (hudson trend graph?) <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • 400 tests in remora, 700 in zamboni <br /> 2,3,5x as fast <br />
  • <br />
  • <br />
  • sharing code, 4 production site <br />
  • <br />
  • python developers who love the web, js/frontend devs, operations, other groups <br />
  • <br />
  • <br />

Djangocon Presentation Transcript

  • 1. Rewriting addons.mozilla.org with Django Jeff Balogh github.com/jbalogh @jeffbalogh
  • 2. Oh god how did I get here I am not good with PHP
  • 3. addons.mozilla.org AMO github.com/jbalogh/zamboni
  • 4. addons.mozilla.org • 165 million request per month
  • 5. Hardware • 3 Zeus load balancers • 24 web servers • mod_wsgi: 16 procs, no threading • 1 MySQL master, 4 slaves • 3 memcached • 3 sphinx • 1 RabbitMQ, 2 celeryd • 1 Redis master, 1 slave
  • 6. How we’re switching • One url at a time • PHP & Python run side by side
  • 7. class Addon(models.Model): name = models.ForeignKey('Translation') description = models.ForeignKey('Translation') authors = models.ManyToManyField('User') # Denormalization current_version = models.ForeignKey('Version') class Translation(models.Model): locale = models.CharField() localized_string = models.CharField() class Version(models.Model): addon = models.ForeignKey(Addon) version = models.CharField()
  • 8. class Addon(models.Model): name = models.ForeignKey('Translation') description = models.ForeignKey('Translation') authors = models.ManyToManyField('User') # Denormalization current_version = models.ForeignKey('Version') class Translation(models.Model): locale = models.CharField() localized_string = models.CharField() class Version(models.Model): addon = models.ForeignKey(Addon) version = models.CharField()
  • 9. multidb • github.com/jbalogh/django-multidb- router • Too easy now (thanks Alex & Russel)
  • 10. Object caching • github.com/jbalogh/django-cache- machine/ • automatically cache all queries
  • 11. Invalidation
  • 12. SELECT * from addons ORDER BY rating DESC LIMIT 10 [<Addon 1865: Adblock Plus>, …] md5(SELECT…rating…) [<Addon 1865>, …]
  • 13. SELECT * from addons ORDER BY rating DESC LIMIT 10 [<Addon 1865: Adblock Plus>, …] md5(SELECT…rating…) [<Addon 1865>, …] flush:md5(<Addon 1865>) [md5(SELECT…rating…)]
  • 14. SELECT * from addons WHERE id = 1865 <Addon 1865: Adblock Plus> md5(SELECT…rating…) [<Addon 1865>, …] flush:md5(<Addon 1865>) [md5(SELECT…rating…)] md5(SELECT…id=1865) <Addon 1865>
  • 15. SELECT * from addons WHERE id = 1865 <Addon 1865: Adblock Plus> md5(SELECT…rating…) [<Addon 1865>, …] [md5(SELECT…rating…), flush:md5(<Addon 1865>) md5(SELECT…id=1865)] md5(SELECT…id=1865) <Addon 1865>
  • 16. cache.set_many({ md5(SELECT…id=1865): None, md5(SELECT…rating…): None } md5(SELECT…rating…) [<Addon 1865>, …] [md5(SELECT…rating…), flush:md5(<Addon 1865>) md5(SELECT…id=1865)] md5(SELECT…id=1865) <Addon 1865>
  • 17. Querysets are big
  • 18. Querysets are big • cached_with(queryset, function) • {% cache obj %} • From 29r/s to 186r/s
  • 19. Related objects • addon.current_version
  • 20. Related objects • github.com/simonw/django-queryset- transform/ • select_related on steroids
  • 21. Transforms Addon.objects.transform(Addon.transformer) @staticmethod def transformer(addons): addon_dict = dict((a.id, a) for a in addons) vs = filter(None, (a.current_version_id for a in addons)) versions = list(Version.objects.filter(id__in=vs)) for version in versions: addon_dict[version.addon_id].current_version = version
  • 22. Templates • {% load %} • {% blocktrans with article.price as amount %} • {% ifnotequal %}
  • 23. Use Jinja2 • github.com/jbalogh/jingo • Tags & filters are just functions • {{ _('Add-ons for {0}')|f(app) }}
  • 24. Tests • github.com/jbalogh/django-nose • nose makes testing easier
  • 25. UnicodeDecodeErro r • UnicodeEncodeError: 'ascii' codec can't encode characters in position 64-72: ordinal not in range(128)
  • 26. UnicodeDecodeErro r • UnicodeEncodeError: 'ascii' codec can't encode characters in position 64-72: ordinal not in range(128)
  • 27. Django issues • .filter() is slow • DELETE CASCADE • SELECT before UPDATE
  • 28. Django issues • .filter() is slow • DELETE CASCADE • SELECT before UPDATE
  • 29. Django issues • .filter() is slow • DELETE CASCADE • SELECT before UPDATE
  • 30. Does Django scale? • Yes, with help • We don’t ship unless it’s faster than PHP • 44,000 lines of PHP vs. 12,500 lines of Python
  • 31. Django @ Mozilla • “I've been a PHP developer for 8 years and a Python developer for 6 months, and I don't want to go back.”
  • 32. http://www.flickr.com/photos/wayneandwax/4731112862/
  • 33. Jobs Jobs Jobs • mzl.la/djangocon • bit.ly/djangocon • bit.ly uses a single database for all their links!
  • 34. Thank You • github.com/jbalogh • jbalogh.me/djangocon.pdf (slides)
  • 35. References github.com/jbalogh github.com/jbalogh/zamboni github.com/jbalogh/django-multidb-router github.com/jbalogh/django-cache-machine/ github.com/jbalogh/jingo github.com/jbalogh/django-nose github.com/simonw/django-queryset-transform/ mzl.la/djangocon github.com/mozilla jbalogh.me/djangocon.pdf (slides) jbalogh.me/djangocon.txt (this page)