Your SlideShare is downloading. ×
0
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Djangocon
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Djangocon

3,586

Published on

0 Comments
16 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,586
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
63
Comments
0
Likes
16
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

  • Python since 2007
    Django since 2008
    Started at Mozilla in 2009

    http://www.flickr.com/photos/morgamic/4466526784/sizes/o/in/set-72157623711339916/

  • trusted place for hosting for Firefox add-ons
    sorting, searching, discovery
    the app store without the store part (at this point)

  • This talk is about the problems we’ve solved getting Django to power a large website









  • Pinning






  • what is the race condition?
  • memcached is not the right place to store sets
    use redis



  • cron/celery/signals
  • /extensions: 14 vs. 150+ queries
  • ideas from batch_select
    pick up any relations you’re going to use a lot: foreign keys, m2m, reverse relations
  • Besides denormalization and indexes, I’m out of ideas for making the database go faster.
    Now we’ll go into some other things we do differently from a vanilla Django project.




  • 700 tests run after every commit with Hudson
    RadicalTestSuiteRunner
    fixtures (hudson trend graph?)








  • 400 tests in remora, 700 in zamboni
    2,3,5x as fast


  • sharing code, 4 production site

  • python developers who love the web, js/frontend devs, operations, other groups


  • Transcript

    • 1. Rewriting addons.mozilla.org with Django Jeff Balogh github.com/jbalogh @jeffbalogh
    • 2. Oh god how did I get here I am not good with PHP
    • 3. addons.mozilla.org AMO github.com/jbalogh/zamboni
    • 4. addons.mozilla.org • 165 million request per month
    • 5. Hardware • 3 Zeus load balancers • 24 web servers • mod_wsgi: 16 procs, no threading • 1 MySQL master, 4 slaves • 3 memcached • 3 sphinx • 1 RabbitMQ, 2 celeryd • 1 Redis master, 1 slave
    • 6. How we’re switching • One url at a time • PHP & Python run side by side
    • 7. class Addon(models.Model): name = models.ForeignKey('Translation') description = models.ForeignKey('Translation') authors = models.ManyToManyField('User') # Denormalization current_version = models.ForeignKey('Version') class Translation(models.Model): locale = models.CharField() localized_string = models.CharField() class Version(models.Model): addon = models.ForeignKey(Addon) version = models.CharField()
    • 8. class Addon(models.Model): name = models.ForeignKey('Translation') description = models.ForeignKey('Translation') authors = models.ManyToManyField('User') # Denormalization current_version = models.ForeignKey('Version') class Translation(models.Model): locale = models.CharField() localized_string = models.CharField() class Version(models.Model): addon = models.ForeignKey(Addon) version = models.CharField()
    • 9. multidb • github.com/jbalogh/django-multidb- router • Too easy now (thanks Alex & Russel)
    • 10. Object caching • github.com/jbalogh/django-cache- machine/ • automatically cache all queries
    • 11. Invalidation
    • 12. SELECT * from addons ORDER BY rating DESC LIMIT 10 [<Addon 1865: Adblock Plus>, …] md5(SELECT…rating…) [<Addon 1865>, …]
    • 13. SELECT * from addons ORDER BY rating DESC LIMIT 10 [<Addon 1865: Adblock Plus>, …] md5(SELECT…rating…) [<Addon 1865>, …] flush:md5(<Addon 1865>) [md5(SELECT…rating…)]
    • 14. SELECT * from addons WHERE id = 1865 <Addon 1865: Adblock Plus> md5(SELECT…rating…) [<Addon 1865>, …] flush:md5(<Addon 1865>) [md5(SELECT…rating…)] md5(SELECT…id=1865) <Addon 1865>
    • 15. SELECT * from addons WHERE id = 1865 <Addon 1865: Adblock Plus> md5(SELECT…rating…) [<Addon 1865>, …] [md5(SELECT…rating…), flush:md5(<Addon 1865>) md5(SELECT…id=1865)] md5(SELECT…id=1865) <Addon 1865>
    • 16. cache.set_many({ md5(SELECT…id=1865): None, md5(SELECT…rating…): None } md5(SELECT…rating…) [<Addon 1865>, …] [md5(SELECT…rating…), flush:md5(<Addon 1865>) md5(SELECT…id=1865)] md5(SELECT…id=1865) <Addon 1865>
    • 17. Querysets are big
    • 18. Querysets are big • cached_with(queryset, function) • {% cache obj %} • From 29r/s to 186r/s
    • 19. Related objects • addon.current_version
    • 20. Related objects • github.com/simonw/django-queryset- transform/ • select_related on steroids
    • 21. Transforms Addon.objects.transform(Addon.transformer) @staticmethod def transformer(addons): addon_dict = dict((a.id, a) for a in addons) vs = filter(None, (a.current_version_id for a in addons)) versions = list(Version.objects.filter(id__in=vs)) for version in versions: addon_dict[version.addon_id].current_version = version
    • 22. Templates • {% load %} • {% blocktrans with article.price as amount %} • {% ifnotequal %}
    • 23. Use Jinja2 • github.com/jbalogh/jingo • Tags & filters are just functions • {{ _('Add-ons for {0}')|f(app) }}
    • 24. Tests • github.com/jbalogh/django-nose • nose makes testing easier
    • 25. UnicodeDecodeErro r • UnicodeEncodeError: 'ascii' codec can't encode characters in position 64-72: ordinal not in range(128)
    • 26. UnicodeDecodeErro r • UnicodeEncodeError: 'ascii' codec can't encode characters in position 64-72: ordinal not in range(128)
    • 27. Django issues • .filter() is slow • DELETE CASCADE • SELECT before UPDATE
    • 28. Django issues • .filter() is slow • DELETE CASCADE • SELECT before UPDATE
    • 29. Django issues • .filter() is slow • DELETE CASCADE • SELECT before UPDATE
    • 30. Does Django scale? • Yes, with help • We don’t ship unless it’s faster than PHP • 44,000 lines of PHP vs. 12,500 lines of Python
    • 31. Django @ Mozilla • “I've been a PHP developer for 8 years and a Python developer for 6 months, and I don't want to go back.”
    • 32. http://www.flickr.com/photos/wayneandwax/4731112862/
    • 33. Jobs Jobs Jobs • mzl.la/djangocon • bit.ly/djangocon • bit.ly uses a single database for all their links!
    • 34. Thank You • github.com/jbalogh • jbalogh.me/djangocon.pdf (slides)
    • 35. References github.com/jbalogh github.com/jbalogh/zamboni github.com/jbalogh/django-multidb-router github.com/jbalogh/django-cache-machine/ github.com/jbalogh/jingo github.com/jbalogh/django-nose github.com/simonw/django-queryset-transform/ mzl.la/djangocon github.com/mozilla jbalogh.me/djangocon.pdf (slides) jbalogh.me/djangocon.txt (this page)

    ×