Scaling small apps
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Scaling small apps

  • 585 views
Uploaded on

In March 2010, the UK had an election with a televised leadership debate. We built an application that scaled to 2.7 million hits in 90 minutes on top of AppEngine which we built in about 3 days. ......

In March 2010, the UK had an election with a televised leadership debate. We built an application that scaled to 2.7 million hits in 90 minutes on top of AppEngine which we built in about 3 days. See the mistakes we made and what went well.

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
585
On Slideshare
573
From Embeds
12
Number of Embeds
2

Actions

Shares
Downloads
5
Comments
0
Likes
1

Embeds 12

http://lanyrd.com 9
http://coderwall.com 3

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • \n
  • Scalability on journalistic deadlines.\nOn Thursday 15th April 2010, there was an historic occasion, the UK’s first televised party leader debate for an election.\nAs a news organisation we wanted to not only report the news and information about the debate, but also get feedback from our readers as to what they thought of the debate.\n
  • Our coverage on the website looked something like this. This was the picture that we put on the main page to encourage people to view our live blog.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

Transcript

  • 1. Michael Brunton-Spall Developer Advocate guardian.co.uk @bruntonspall
  • 2. Scaling small apps on AppEngine
  • 3. Our coverage  
  • 4. Poll Charts• "Let people viewing the page vote at anytime whether they like or dislike what the party leader is saying.  Oh, and lets show it with a real time graph"• Bad words here • anytime • real-time
  • 5. The Design
  • 6. The reality
  • 7. Timeline• Thursday 8th April 2010 - Project Start• Thursday 15th April 2010 - Debate 1 - Go Live• Thursday 22th April 2010 - Debate 2• Thursday 29th April 2010 - Final Debate
  • 8. Countersget(self):    return self.val
  • 9. Countersincr(self, val):    self.val += val
  • 10. Locks are bad, Mkay!• transaction write locks• all threads are serialised
  • 11. Sharded Countersincr(self, val=1):    c = randomcounter c.val += val
  • 12. Sharded Countersget(self):    return sum(counters)
  • 13. Shards are better, Mkay! • writes lock only 1 shard at a time • n simultaneous writes
  • 14. On the first night
  • 15. The poll itself
  • 16. Requests Per Second
  • 17. In Detail
  • 18. Numbers• Average: 100-120 req/s• Max: 400 req/s• Total requests: 1,000,000 in 90 mins• Little cheating - 2000 requests
  • 19. Request Duration
  • 20. Request Duration• 1 to 8 seconds• Thread contention• Not enough shards?
  • 21. Not enough shards?• Increase number of shards? • by factor of 10? • No transaction failures • Each request still takes 200ms
  • 22. 200ms per incr?• Datastore write• requires disk access
  • 23. Memcached?• Change architecture • atomic increment/decrement • get from memcache • cronjob 1/min • memcache => datastore • votes now take 20 ms each
  • 24. The Flu
  • 25. The second night
  • 26. The Results?
  • 27. The Results?
  • 28. Some notes• Total of around 2,727,000 requests• Average of around 454 req/s• Peaked at 750 req/s 
  • 29. Requests per Second
  • 30. Request Duration
  • 31. The problems
  • 32. Errors / Second
  • 33. Google AppEngine• Request limit: • x requests per day • But also... • y requests per minute • approx 1/3 reqs refused
  • 34. Guardian• Serving up json via guardian domain • Heavier process than we thought • cronjob to upload to cdn 1/min
  • 35. Request Duration
  • 36. The solution!
  • 37. The third night
  • 38. The results
  • 39. Requests per second
  • 40. Some notes• Total of around 750,000 requests• Average of around 138 req/s• Peaked at around 230 req/s 
  • 41. £££ (or $$$)• Appengine Cost: $Negligable• plus cdn bandwidth• plus dev costs• etc...
  • 42. Any Questions? Michael Brunton-Spall @bruntonspallmichael.brunton-spall@guardian.co.uk