Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

High Availability Django - Djangocon 2016

Demo repo:

Description:, with over 30 million monthly unique visitors, is one of the most heavily trafficked media sites built entirely on Django. In this talk I will discuss the lessons we have learned in deploying Django at scale.

Abstract: One year ago we completed a years-long project of migrating from a sprawling PHP codebase to a Python application built on Django. Our first attempt at a load-balanced Python stack had serious flaws, as we quickly learned. Since then we have completely remade our stack from the bottom up; we have built tools that improve our ability to monitor for performance and service degradation; and we have developed a deployment process that incorporates automated testing and that allows us to push out updates without incurring any downtime. I will discuss the mistakes we made, the steps we took to identify performance problems and server resource issues, what our current stack looks like, and how we achieved the holy grail of zero-downtime deploys.

  • Login to see the comments

High Availability Django - Djangocon 2016

  1. 1. High-Availability Django Frankie Dintino The Atlantic
  2. 2. Performance woes…
  3. 3. …and crises
  4. 4. Today
  5. 5. First steps • Monitoring and profiling: understand your bottlenecks • Tackle the “easy” stuff first: query optimization, caching • Don’t neglect front-end performance
  6. 6. Profiling • Hosted services: New Relic, Opbeat • django-debug-toolbar, django-silk • cProfile / pyprof2calltree / KCacheGrind for masochists • WSGI middleware
  7. 7. Monitoring • Downtime notifications: Alertra, New Relic, Opbeat, Chartbeat • Nagios and its many forks, Zabbix
  8. 8. Caching • Use a CDN • Proxy cache (Nginx, Squid, Varnish) • {% cache %}, @cached_property, CachedStaticFilesStorage, cached template loader • Page caching frameworks • ORM caching: a mixed bag
  9. 9. Query optimization • .prefetch_related(), .select_related() • prefetch_related_objects() for generic foreign keys • .values() and .values_list() where it counts, if possible • Database parameter tuning, judicious indexing
  10. 10. Throwing more hardware at the problem also works.
  11. 11. “Request Queuing?”
  12. 12. “Request Queuing?”
  13. 13. theatlantic :4000 citylab :5000 admin :6000 :80
  14. 14. upstream theatlantic { server apps1:4000; server apps2:4000; server apps3:4000; } upstream citylab { server apps1:5000; # ... }
  15. 15. upstream theatlantic_stage { server apps1:4500; server apps2:4500; server apps3:4500; } upstream citylab_stage { server apps1:5500; # ... }
  16. 16. /www/site/ stage/ live/ a/ b/
  17. 17. /www/site/ stage/ live/ a/ b/
  18. 18. Basic principles • “a” and “b” repositories are always “production- ready.” • Force as much Django code to load as possible before defining def application() in your WSGI file. • Before swapping the upstreams, “warm up” the workers with concurrent HTTP requests.
  19. 19. mod_wsgi vs. uWSGI
  20. 20. mod_wsgi • In general, comparable performance • Runs on Apache HTTP server • Mostly the work of a single developer • Documentation terminally out-of-date • Important configuration options missing or only discoverable in the release notes
  21. 21. uWSGI • Broad and active community • Very thorough documentation (overwhelming, in fact) • Highly configurable
  22. 22. @frankiedintino
  23. 23. High-Availability Django