How does the guardian website scale?
With millions of page views per month, we need to think about scaling to an extreme level. But being Agile we did it as we went.
Scaling the Guardian
Michael Brunton-Spall (@bruntonspall)
The Guardian - Some Figures
ABCe Audited (Dec 2009)
Unique Users - 36.9m per month, 1.8m per day
Page Impressions - 259m per month, 9.2m per day
Log file analysis
37m requests per day, 1.1bn requests per month - not
inlcuding images / static files
More reduction in Appserver load
Must handle customisation outside of cache
Memcached for pages is filter
Page customisation is a higher filter
Time based decache only
Decache only on direct page edit
Getting a Scaling Solution
The problem isn't technical
It's all about the process
Agile doesn't scale well!
Onsite customer doesn't care about scaling
Dedicated 10% team to look at "platform" issues
Still Agile, Customer is Operations Team & Architects
(backend and frontend)
Scaling small apps rapidly
On Thursday 15th 2010 there was a historic UK event - a
televised national debate.
Always sounds simple:
"Let people viewing the page vote at anytime whether they like
or dislike what the party leader is saying. Oh, and lets show it
with a real time graph"
Bad words here
The poll itself
Google App Engine
An inhouse, inplatform cache
The Naive Implementation
Poll.get().libdems += 1
Google App Engine has transaction locks, simultaneous
threads can't atomically increment a counter (duh)
If you wrap in a txn, all threads are serialised.
You just turned Googles massively parallel data center
into a very expensive file backed db
Our Implementation (Phase 1)
Sharded counters are the way to go
Follow the article at code.google.com/appengine on
Gives parallel counters
Some interesting notes
Average of around 100-120 req/s
Peaked at 400 req/s
Total of nearly 1,000,000 requests
Surprisingly little cheating
Only 2000 requests
Between 1 sec and 8 seconds!
Not enough shards
Our Implementation (2)
Increase shards by factor of 10?
Completely reduces transaction failures
Each request still takes 200ms
The cost is the datastore write
Replace datastore with memcache?
vote does memcache atomic
results get from memcache
cronjob 1/min reads from memcache and
writes to datastore
requests now take 20 ms