Malcolm Box presented on scaling Django web applications to handle over 10,000 requests per second. He discussed challenges in caching, counting, and using Cassandra to handle a write-heavy workload from user interactions on live TV shows. Key aspects of their architecture included caching using Pylibmc, implementing sharded counters to count things rapidly, and using Cassandra as the core database scaled through caching and virtual nodes.

1. SCALING DJANGO FOR X FACTOR
MALCOLM BOX, DJUGL OCTOBER 2012

2. WHAT I’M TALKING ABOUT
Scaling Django to >10K request/s
Caching, Counting and Cassandra
Toolbox

3. ME
Malcolm Box, CTO & Co-Founder
@malcolmbox
malcolm@tellybug.com
http://tellybug.com

4. Making TV more
entertaining
Live interaction
Highly social
Unique content

5. WHO ARE YOU?
Technical?
Running Django?
Scale?

6. THE CHALLENGE

7. THE CHALLENGE
Millions of people watch the
shows we work with

8. THE CHALLENGE
Millions of people watch the
shows we work with
TV tells them to buzz/clap/
score....

9. THE CHALLENGE
Millions of people watch the
shows we work with
TV tells them to buzz/clap/
score....
A giant DDOS is launched
against our servers

10. HOW BIG?
Peak loads of 10,000 requests/s
Read/write mix
Write-heavy workload - lots of user interactions

11. HOW BIG?
10K REQUESTS/S IS
25,920,000,000
REQUESTS/MONTH

12. The Internet
ARCHITECTURE Static assets
HAProxy layer
Entirely cloud
based Web layer
Chef
Nodes come and Cache
go - frequently! Monitor
Cassandra Cluster
Automatic Task
deployment direct
RDS MySQL
Server
from Github via Amazon AWS eu-west-1
Logs, backups
Amazon S3
Chef

13. CACHING
Cache as speedup or Cache as mission-critical?
Use Django cache framework
Pylibmc - consistent hashing and server death patches
Problems as you scale up...

14. CACHE PROBLEMS
Cache miss behaviour value = cache.get(key)
if value is None:
try:
Thundering herds are bad lock = cache.add(lock_key(key))
if lock:
Key overload # Do something expensive
new_value = calculate_new_value()
cache.set(key, new_value)
Server overload return new_value
finally:
Dualcache - https:// if lock:
cache.delete(lock_key(key)
gist.github.com/953524
return value

15. COUNTING
Hard to count a few things very fast
And have real-time access to the latest result
Things we tried:
memcache
Cassandra counters
Final solution: Sharded counters

16. SHARDED COUNTERS
Implemented in about 350 lines of Python
To provide two basic operations!
incr()
get()
Uses a combination of two layers of memcache and
Cassandra to provide real-time, scalable counters

17. CASSANDRA
Core piece of our infrastructure
Highly write-scalable
Reads scaled from cache
Using Acunu Cassandra for virtual nodes
“Fake” Django ORM classes to make it feel more natural
But no automatic join support

18. TOOLBOX
Development
Django Extensions, Celery, Piston (heavily forked), iPython, pycassa
Tsung (load testing tool)
Deployment:
Fabric, Chef, Boto
Operations
Sentry, Gargoyle

19. THINGS THAT STILL SUCK
Monitoring

20. Q&A
AND YES, WE’RE HIRING SO IF YOU’RE INTERESTED IN BUILDING EXTREMELY LARGE
DJANGO SITES THEN GET IN TOUCH
MALCOLM@TELLYBUG.COM