PyCon 2011 Scaling Disqus

Python at 400 500 million visitors DISQ US Jason Yan @ jasonyan David Cramer @ zeeg Got feedback? Use hashtag #sckrw

Agenda <ul><li>What is DISQUS ? </li></ul><ul><li>An Overview of the Infrastructure </li></ul><ul><li>Iterative Development and Deployment </li></ul><ul><li>Why We Love Python </li></ul>

What is DISQUS? We are a comment system with an emphasis on connecting communities http://disqus.com/about/ dis·cuss • dĭ-skŭs'

Embeddable Comments

A Brief History

Startup-ish <ul><li>Founded just about 4 years ago </li></ul><ul><li>16 employees, 8 engineers </li></ul><ul><li>Traffic increasing 15-20% a month </li></ul><ul><li>Flat organizational structure, every engineer is a product manager </li></ul><ul><li>Fast turnaround, new feature launches every week (sometimes daily) </li></ul>

Traffic March 2008 through March 2011

DjangoCon 2010 <ul><li>17,000 requests/second peak </li></ul><ul><li>450,000 websites </li></ul><ul><li>15 million profiles </li></ul><ul><li>75 million comments </li></ul><ul><li>250 million visitors </li></ul>

Six Months Later <ul><li>25,000 requests/second peak </li></ul><ul><li>700,000 websites </li></ul><ul><li>30 million profiles </li></ul><ul><li>170 million comments </li></ul><ul><li>500 million visitors </li></ul><ul><li>17,000 requests/second peak </li></ul><ul><li>450,000 websites </li></ul><ul><li>15 million profiles </li></ul><ul><li>75 million comments </li></ul><ul><li>250 million visitors </li></ul>

Six Months Later <ul><li>September 2010: 250 million uniques </li></ul><ul><li>March 2011: 500 million uniques </li></ul><ul><li>Handling over 2x the traffic </li></ul>

Six Months Later <ul><li>September 2010: ~100 servers </li></ul><ul><li>March 2011: ~100 servers </li></ul><ul><li>Scale diagonally </li></ul>

Scaling Diagonally <ul><li>We still rent hardware , so there is no “commodity hardware” </li></ul><ul><ul><li>Cheaper to upgrade </li></ul></ul><ul><li>Everything is redundant </li></ul><ul><li>Partition data where you need to, scale partitions vertically </li></ul><ul><li>Upgrade hardware (more RAM, more drives, more cores) </li></ul><ul><ul><li>Python apps tend to be CPU bound </li></ul></ul>

Infrastructure <ul><li>35% Web Servers (Apache + mod_wsgi) </li></ul><ul><li>15% Utility Servers (Python scripts, background workers) </li></ul><ul><li>20% Databases (PostgreSQL, Redis, Membase) </li></ul><ul><li>20% Load Balancing / High Availability (HAProxy + Heartbeat) </li></ul><ul><li>10% Caching servers (Memcached, Varnish) </li></ul><ul><li>Half of our servers run Python </li></ul>

Python Web Servers

Background Workers <ul><li>Lots of tasks that don’t need to be done in web application process: </li></ul><ul><ul><li>Crawling URLs </li></ul></ul><ul><ul><li>Updating avatars </li></ul></ul><ul><ul><li>Email notifications </li></ul></ul><ul><ul><li>Analytics </li></ul></ul><ul><ul><li>Counters </li></ul></ul>

Background Workers (cont’d) <ul><li>Most jobs are I/O bound </li></ul><ul><ul><li>Slow external calls </li></ul></ul><ul><ul><ul><li>Twitter is slow </li></ul></ul></ul><ul><ul><ul><li>Facebook is slow </li></ul></ul></ul><ul><li>Could parallelize with multiple processes, but... </li></ul>

Background Workers (cont’d) <ul><li>Waste of memory </li></ul><ul><li>Use non-blocking I/O </li></ul><ul><ul><li>Celery 2.2 adds support for gevent/eventlet </li></ul></ul>

Monitoring <ul><li>Application side: Graphite </li></ul><ul><ul><li>Real-time(ish) graphing </li></ul></ul><ul><ul><li>Django front-end, Python backend </li></ul></ul><ul><li>Etsy’s StatsD proxy to Graphite </li></ul><ul><ul><li>UDP (fire and forget) </li></ul></ul><ul><ul><li>Batches updates </li></ul></ul>

Monitoring <ul><li>Track application metrics </li></ul><ul><ul><li>Errors, exceptions </li></ul></ul><ul><ul><li>New comments, users, sites, etc. </li></ul></ul><ul><ul><li>Anything </li></ul></ul>

Monitoring <ul><li>Check out Etsy’s posts: </li></ul><ul><ul><li>Measure Anything, Measure Everything http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/ </li></ul></ul><ul><ul><li>Tracking Every Release http://codeascraft.etsy.com/2010/12/08/track-every-release/ </li></ul></ul>

What about the code?

Powered By Django

Which means... <ul><li>Largest Django-powered web application </li></ul><ul><li>We fork , and even sometimes monkey patch to make it scale to our needs </li></ul><ul><ul><li>Fortunately, we don’t have to do too much (Yay, Django!) </li></ul></ul><ul><ul><li>Unfortunately, we can’t use the whole of the Django internal components (and if we do, we do it in atypical ways) </li></ul></ul>

Iterative Development Release Early Release Often

Iterating Quickly <ul><li>Abstracting our application environment </li></ul><ul><ul><li>Less dependancies locally </li></ul></ul><ul><ul><li>Rely on CI for dependency coverage </li></ul></ul><ul><li>Heavy use of open source packages </li></ul><ul><ul><li>No NIH syndrome </li></ul></ul><ul><li>Deploy frequently , 3-7 times a day </li></ul><ul><li>Lots of branches, but master is “stable” </li></ul><ul><li>Realtime reporting on exceptions, metrics </li></ul><ul><li>Our test suite is the main blocker (slow) </li></ul>

Dealing with Deploys

Gargoyle Being users of our product , we actively use early versions of features before public release Deploy features to portions of a user base at a time to ensure smooth, measurable releases

The Deployment Problem <ul><li>Make some changes locally </li></ul><ul><li>Run a subset of the test suite </li></ul><ul><li>Push your commits </li></ul><ul><li>CI server begins running tests </li></ul><ul><li>.... </li></ul>

Waiting on the test suite...

Rinse and Repeat <ul><li>30 minutes later tests fail , start over </li></ul><ul><li>Finally, deploy to a subset of servers </li></ul><ul><ul><li>Open Sentry (our exception logger) </li></ul></ul><ul><ul><li>Monitor Graphite </li></ul></ul><ul><li>Deploy to 35 servers ( ~8 minutes ) </li></ul><ul><ul><li>Full rollback in < 30 seconds </li></ul></ul>

Wait, Sentry?

Testing

Testing Code <ul><li>Test suite takes around 25 minutes usually </li></ul><ul><li>“ Stuck” with Hudson (or Jenkins ) </li></ul><ul><ul><li>Most tightly integrated plugins are geared towards Java developers </li></ul></ul><ul><li>Which framework do we use? </li></ul><ul><ul><li>unittest(2), nose, doctests, LETTUCE? </li></ul></ul><ul><ul><li>We use unittest and nose </li></ul></ul><ul><li>Need to report code coverage , speed of tests , pylint (or pyflakes ) </li></ul>

We Love Python

Love-ish <ul><li>Many of us started with PHP or Rails </li></ul><ul><li>Clean syntax , clear standards </li></ul><ul><ul><li>All languages need PEP8.py and PyFlakes </li></ul></ul><ul><li>Interpreted , fast... enough </li></ul><ul><li>Very easy to learn </li></ul><ul><ul><li>We all started by learning Django first , then Python </li></ul></ul>

Haters Gonna Hate If you could choose one thing in Python to hate on...

Better package management

What can we do? <ul><li>Too many forks, too many frameworks </li></ul><ul><ul><li>We need less clones , and more combined effort </li></ul></ul><ul><li>Improving existing Python solutions </li></ul><ul><li>More Python solutions for existing products </li></ul>

Python Rocks!

Questions? DISQ US psst, we’re hiring [email_address]

References <ul><li>Sentry (our exception tracking tool) http://github.com/dcramer/django-sentry </li></ul><ul><li>Gargoyle (feature switches) https://github.com/disqus/gargoyle </li></ul><ul><li>Django DB Utils (collection of db helpers for Django) https://github.com/disqus/django-db-utils </li></ul><ul><li>Jenkins CI http://jenkins-ci.org/ </li></ul>code.disqus.com