Your SlideShare is downloading. ×
0
Michael Brunton-Spall   Developer Advocate     guardian.co.uk     @bruntonspall
Scaling small apps on AppEngine
Our coverage      
Poll Charts• "Let people viewing the page vote at  anytime whether they like or dislike what  the party leader is saying. ...
The Design
The reality
Timeline• Thursday 8th April 2010 - Project Start• Thursday 15th April 2010 - Debate 1 - Go  Live• Thursday 22th April 201...
Countersget(self):    return self.val
Countersincr(self, val):    self.val += val
Locks are bad, Mkay!• transaction write locks• all threads are serialised
Sharded   Countersincr(self, val=1):    c = randomcounter   c.val += val
Sharded     Countersget(self):    return sum(counters)
Shards are better, Mkay! • writes lock only 1 shard at a time • n simultaneous writes
On the first night
The poll itself
Requests Per Second
In Detail
Numbers• Average: 100-120 req/s• Max: 400 req/s• Total requests: 1,000,000 in 90 mins• Little cheating - 2000 requests
Request Duration
Request Duration• 1 to 8 seconds• Thread contention• Not enough shards?
Not enough shards?• Increase number of shards? • by factor of 10?    • No transaction failures    • Each request still tak...
200ms per incr?• Datastore write• requires disk access
Memcached?• Change architecture • atomic increment/decrement • get from memcache • cronjob 1/min   • memcache => datastore...
The Flu
The second night
The Results?
The Results?
Some notes• Total of around 2,727,000 requests• Average of around 454 req/s• Peaked at 750 req/s 
Requests per Second
Request Duration
The problems
Errors / Second
Google AppEngine• Request limit: • x requests per day • But also... • y requests per minute   • approx 1/3 reqs refused
Guardian• Serving up json via guardian domain • Heavier process than we thought • cronjob to upload to cdn 1/min
Request Duration
The solution!
The third night
The results
Requests per second
Some notes• Total of around 750,000 requests• Average of around 138 req/s• Peaked at around 230 req/s 
£££ (or $$$)• Appengine Cost: $Negligable• plus cdn bandwidth• plus dev costs• etc...
Any Questions?        Michael Brunton-Spall           @bruntonspallmichael.brunton-spall@guardian.co.uk
Scaling small apps
Scaling small apps
Scaling small apps
Upcoming SlideShare
Loading in...5
×

Scaling small apps

486

Published on

In March 2010, the UK had an election with a televised leadership debate. We built an application that scaled to 2.7 million hits in 90 minutes on top of AppEngine which we built in about 3 days. See the mistakes we made and what went well.

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
486
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
6
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • \n
  • Scalability on journalistic deadlines.\nOn Thursday 15th April 2010, there was an historic occasion, the UK’s first televised party leader debate for an election.\nAs a news organisation we wanted to not only report the news and information about the debate, but also get feedback from our readers as to what they thought of the debate.\n
  • Our coverage on the website looked something like this. This was the picture that we put on the main page to encourage people to view our live blog.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Transcript of "Scaling small apps"

    1. 1. Michael Brunton-Spall Developer Advocate guardian.co.uk @bruntonspall
    2. 2. Scaling small apps on AppEngine
    3. 3. Our coverage  
    4. 4. Poll Charts• "Let people viewing the page vote at anytime whether they like or dislike what the party leader is saying.  Oh, and lets show it with a real time graph"• Bad words here • anytime • real-time
    5. 5. The Design
    6. 6. The reality
    7. 7. Timeline• Thursday 8th April 2010 - Project Start• Thursday 15th April 2010 - Debate 1 - Go Live• Thursday 22th April 2010 - Debate 2• Thursday 29th April 2010 - Final Debate
    8. 8. Countersget(self):    return self.val
    9. 9. Countersincr(self, val):    self.val += val
    10. 10. Locks are bad, Mkay!• transaction write locks• all threads are serialised
    11. 11. Sharded Countersincr(self, val=1):    c = randomcounter c.val += val
    12. 12. Sharded Countersget(self):    return sum(counters)
    13. 13. Shards are better, Mkay! • writes lock only 1 shard at a time • n simultaneous writes
    14. 14. On the first night
    15. 15. The poll itself
    16. 16. Requests Per Second
    17. 17. In Detail
    18. 18. Numbers• Average: 100-120 req/s• Max: 400 req/s• Total requests: 1,000,000 in 90 mins• Little cheating - 2000 requests
    19. 19. Request Duration
    20. 20. Request Duration• 1 to 8 seconds• Thread contention• Not enough shards?
    21. 21. Not enough shards?• Increase number of shards? • by factor of 10? • No transaction failures • Each request still takes 200ms
    22. 22. 200ms per incr?• Datastore write• requires disk access
    23. 23. Memcached?• Change architecture • atomic increment/decrement • get from memcache • cronjob 1/min • memcache => datastore • votes now take 20 ms each
    24. 24. The Flu
    25. 25. The second night
    26. 26. The Results?
    27. 27. The Results?
    28. 28. Some notes• Total of around 2,727,000 requests• Average of around 454 req/s• Peaked at 750 req/s 
    29. 29. Requests per Second
    30. 30. Request Duration
    31. 31. The problems
    32. 32. Errors / Second
    33. 33. Google AppEngine• Request limit: • x requests per day • But also... • y requests per minute • approx 1/3 reqs refused
    34. 34. Guardian• Serving up json via guardian domain • Heavier process than we thought • cronjob to upload to cdn 1/min
    35. 35. Request Duration
    36. 36. The solution!
    37. 37. The third night
    38. 38. The results
    39. 39. Requests per second
    40. 40. Some notes• Total of around 750,000 requests• Average of around 138 req/s• Peaked at around 230 req/s 
    41. 41. £££ (or $$$)• Appengine Cost: $Negligable• plus cdn bandwidth• plus dev costs• etc...
    42. 42. Any Questions? Michael Brunton-Spall @bruntonspallmichael.brunton-spall@guardian.co.uk
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×