Scaling small apps

  • 436 views
Uploaded on

In March 2010, the UK had an election with a televised leadership debate. We built an application that scaled to 2.7 million hits in 90 minutes on top of AppEngine which we built in about 3 days. …

In March 2010, the UK had an election with a televised leadership debate. We built an application that scaled to 2.7 million hits in 90 minutes on top of AppEngine which we built in about 3 days. See the mistakes we made and what went well.

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
436
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
5
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • \n
  • Scalability on journalistic deadlines.\nOn Thursday 15th April 2010, there was an historic occasion, the UK’s first televised party leader debate for an election.\nAs a news organisation we wanted to not only report the news and information about the debate, but also get feedback from our readers as to what they thought of the debate.\n
  • Our coverage on the website looked something like this. This was the picture that we put on the main page to encourage people to view our live blog.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

Transcript

  • 1. Michael Brunton-Spall Developer Advocate guardian.co.uk @bruntonspall
  • 2. Scaling small apps on AppEngine
  • 3. Our coverage  
  • 4. Poll Charts• "Let people viewing the page vote at anytime whether they like or dislike what the party leader is saying.  Oh, and lets show it with a real time graph"• Bad words here • anytime • real-time
  • 5. The Design
  • 6. The reality
  • 7. Timeline• Thursday 8th April 2010 - Project Start• Thursday 15th April 2010 - Debate 1 - Go Live• Thursday 22th April 2010 - Debate 2• Thursday 29th April 2010 - Final Debate
  • 8. Countersget(self):    return self.val
  • 9. Countersincr(self, val):    self.val += val
  • 10. Locks are bad, Mkay!• transaction write locks• all threads are serialised
  • 11. Sharded Countersincr(self, val=1):    c = randomcounter c.val += val
  • 12. Sharded Countersget(self):    return sum(counters)
  • 13. Shards are better, Mkay! • writes lock only 1 shard at a time • n simultaneous writes
  • 14. On the first night
  • 15. The poll itself
  • 16. Requests Per Second
  • 17. In Detail
  • 18. Numbers• Average: 100-120 req/s• Max: 400 req/s• Total requests: 1,000,000 in 90 mins• Little cheating - 2000 requests
  • 19. Request Duration
  • 20. Request Duration• 1 to 8 seconds• Thread contention• Not enough shards?
  • 21. Not enough shards?• Increase number of shards? • by factor of 10? • No transaction failures • Each request still takes 200ms
  • 22. 200ms per incr?• Datastore write• requires disk access
  • 23. Memcached?• Change architecture • atomic increment/decrement • get from memcache • cronjob 1/min • memcache => datastore • votes now take 20 ms each
  • 24. The Flu
  • 25. The second night
  • 26. The Results?
  • 27. The Results?
  • 28. Some notes• Total of around 2,727,000 requests• Average of around 454 req/s• Peaked at 750 req/s 
  • 29. Requests per Second
  • 30. Request Duration
  • 31. The problems
  • 32. Errors / Second
  • 33. Google AppEngine• Request limit: • x requests per day • But also... • y requests per minute • approx 1/3 reqs refused
  • 34. Guardian• Serving up json via guardian domain • Heavier process than we thought • cronjob to upload to cdn 1/min
  • 35. Request Duration
  • 36. The solution!
  • 37. The third night
  • 38. The results
  • 39. Requests per second
  • 40. Some notes• Total of around 750,000 requests• Average of around 138 req/s• Peaked at around 230 req/s 
  • 41. £££ (or $$$)• Appengine Cost: $Negligable• plus cdn bandwidth• plus dev costs• etc...
  • 42. Any Questions? Michael Brunton-Spall @bruntonspallmichael.brunton-spall@guardian.co.uk