• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Scaling small apps
 

Scaling small apps

on

  • 528 views

In March 2010, the UK had an election with a televised leadership debate. We built an application that scaled to 2.7 million hits in 90 minutes on top of AppEngine which we built in about 3 days. ...

In March 2010, the UK had an election with a televised leadership debate. We built an application that scaled to 2.7 million hits in 90 minutes on top of AppEngine which we built in about 3 days. See the mistakes we made and what went well.

Statistics

Views

Total Views
528
Views on SlideShare
516
Embed Views
12

Actions

Likes
1
Downloads
5
Comments
0

2 Embeds 12

http://lanyrd.com 9
http://coderwall.com 3

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • \n
  • Scalability on journalistic deadlines.\nOn Thursday 15th April 2010, there was an historic occasion, the UK’s first televised party leader debate for an election.\nAs a news organisation we wanted to not only report the news and information about the debate, but also get feedback from our readers as to what they thought of the debate.\n
  • Our coverage on the website looked something like this. This was the picture that we put on the main page to encourage people to view our live blog.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

Scaling small apps Scaling small apps Presentation Transcript

  • Michael Brunton-Spall Developer Advocate guardian.co.uk @bruntonspall
  • Scaling small apps on AppEngine
  • Our coverage  
  • Poll Charts• "Let people viewing the page vote at anytime whether they like or dislike what the party leader is saying.  Oh, and lets show it with a real time graph"• Bad words here • anytime • real-time
  • The Design
  • The reality
  • Timeline• Thursday 8th April 2010 - Project Start• Thursday 15th April 2010 - Debate 1 - Go Live• Thursday 22th April 2010 - Debate 2• Thursday 29th April 2010 - Final Debate
  • Countersget(self):    return self.val
  • Countersincr(self, val):    self.val += val
  • Locks are bad, Mkay!• transaction write locks• all threads are serialised
  • Sharded Countersincr(self, val=1):    c = randomcounter c.val += val
  • Sharded Countersget(self):    return sum(counters)
  • Shards are better, Mkay! • writes lock only 1 shard at a time • n simultaneous writes
  • On the first night
  • The poll itself
  • Requests Per Second
  • In Detail
  • Numbers• Average: 100-120 req/s• Max: 400 req/s• Total requests: 1,000,000 in 90 mins• Little cheating - 2000 requests
  • Request Duration
  • Request Duration• 1 to 8 seconds• Thread contention• Not enough shards?
  • Not enough shards?• Increase number of shards? • by factor of 10? • No transaction failures • Each request still takes 200ms
  • 200ms per incr?• Datastore write• requires disk access
  • Memcached?• Change architecture • atomic increment/decrement • get from memcache • cronjob 1/min • memcache => datastore • votes now take 20 ms each
  • The Flu
  • The second night
  • The Results?
  • The Results?
  • Some notes• Total of around 2,727,000 requests• Average of around 454 req/s• Peaked at 750 req/s 
  • Requests per Second
  • Request Duration
  • The problems
  • Errors / Second
  • Google AppEngine• Request limit: • x requests per day • But also... • y requests per minute • approx 1/3 reqs refused
  • Guardian• Serving up json via guardian domain • Heavier process than we thought • cronjob to upload to cdn 1/min
  • Request Duration
  • The solution!
  • The third night
  • The results
  • Requests per second
  • Some notes• Total of around 750,000 requests• Average of around 138 req/s• Peaked at around 230 req/s 
  • £££ (or $$$)• Appengine Cost: $Negligable• plus cdn bandwidth• plus dev costs• etc...
  • Any Questions? Michael Brunton-Spall @bruntonspallmichael.brunton-spall@guardian.co.uk