Demo repo: https://github.com/hadjango/djangocon-2016-demo/ Description: TheAtlantic.com, with over 30 million monthly unique visitors, is one of the most heavily trafficked media sites built entirely on Django. In this talk I will discuss the lessons we have learned in deploying Django at scale. Abstract: One year ago we completed a years-long project of migrating theatlantic.com from a sprawling PHP codebase to a Python application built on Django. Our first attempt at a load-balanced Python stack had serious flaws, as we quickly learned. Since then we have completely remade our stack from the bottom up; we have built tools that improve our ability to monitor for performance and service degradation; and we have developed a deployment process that incorporates automated testing and that allows us to push out updates without incurring any downtime. I will discuss the mistakes we made, the steps we took to identify performance problems and server resource issues, what our current stack looks like, and how we achieved the holy grail of zero-downtime deploys.

1. High-Availability Django
Frankie Dintino
The Atlantic

4. Performance woes…

5. …and crises

6. Today

7. First steps
• Monitoring and profiling: understand your
bottlenecks
• Tackle the “easy” stuff first: query optimization,
caching
• Don’t neglect front-end performance

8. Profiling
• Hosted services: New Relic, Opbeat
• django-debug-toolbar, django-silk
• cProfile / pyprof2calltree / KCacheGrind for
masochists
• WSGI middleware

9. Monitoring
• Downtime notifications: Alertra, New Relic,
Opbeat, Chartbeat
• Nagios and its many forks, Zabbix

10. Caching
• Use a CDN
• Proxy cache (Nginx, Squid, Varnish)
• {% cache %}, @cached_property,
CachedStaticFilesStorage, cached template
loader
• Page caching frameworks
• ORM caching: a mixed bag

11. Query optimization
• .prefetch_related(), .select_related()
• prefetch_related_objects() for generic
foreign keys
• .values() and .values_list() where it
counts, if possible
• Database parameter tuning, judicious indexing

12. Throwing more hardware at the
problem also works.

13. “Request Queuing?”

14. “Request Queuing?”

15. theatlantic :4000
citylab :5000
admin :6000
:80

16. upstream theatlantic {
server apps1:4000;
server apps2:4000;
server apps3:4000;
}
upstream citylab {
server apps1:5000;
# ...
}

17. upstream theatlantic_stage {
server apps1:4500;
server apps2:4500;
server apps3:4500;
}
upstream citylab_stage {
server apps1:5500;
# ...
}

18. /www/site/
stage/
live/
a/
b/

19. /www/site/
stage/
live/
a/
b/

20. Basic principles
• “a” and “b” repositories are always “production-
ready.”
• Force as much Django code to load as possible
before defining def application() in your
WSGI file.
• Before swapping the upstreams, “warm up” the
workers with concurrent HTTP requests.

21. mod_wsgi vs. uWSGI

22. mod_wsgi
• In general, comparable performance
• Runs on Apache HTTP server
• Mostly the work of a single developer
• Documentation terminally out-of-date
• Important configuration options missing or only
discoverable in the release notes

24. uWSGI
• Broad and active community
• Very thorough documentation (overwhelming, in
fact)
• Highly configurable

31. github.com/hadjango
@frankiedintino
frankie@theatlantic.com

32. High-Availability Django