Your SlideShare is downloading. ×

Scaling the Britain's Got Talent Buzzer

2,161

Published on

How Live Talkback scaled the Britain's Got Talent buzzer to support 50,000 requests/second

How Live Talkback scaled the Britain's Got Talent buzzer to support 50,000 requests/second

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
2,161
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
15
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. 1 Powering the Britain’s Got Talent buzzer* *And Big Data Big Data Meetup, London 25/5/2011Thursday, 26 May 2011 1
  • 2. 2 What we doThursday, 26 May 2011 2
  • 3. 3 Me Malcolm Box, Co-founder & CTO boxm@livetalkback.com @malcolmboxThursday, 26 May 2011 3
  • 4. 4 The Buzzer BIG DATAThursday, 26 May 2011 4
  • 5. 5 The challenge 10 Million+ viewers Design goal of 50,000 requests/s, 10,000 buzzes/second Equivalent to 130 Billion requests/month But just on Saturday night And four weeks to buildThursday, 26 May 2011 5
  • 6. 6 The challenge Where does 130 Billion requests fit? Source: http://www.google.com/adplanner/static/top1000/#Thursday, 26 May 2011 6
  • 7. 7 Where we started.... app.livetalkback.com cdn.livetalkback.com Control plane ELB CloudFront Zabbix Webserver Webserver Django Django Ubuntu Ubuntu MySQL S3Thursday, 26 May 2011 7
  • 8. 8 Step 1: Testing Started with a platform with a previous peak of 100 requests/s No idea where it would break Tsung! http://tsung.erlang-projects.org/Thursday, 26 May 2011 8
  • 9. 9 Step 2: ELB Amazon Elastic Load Balancer “Infinite capacity” BUT very long impulse response and NO controls :( HAProxy to the rescue 5K requests/s per nodeThursday, 26 May 2011 9
  • 10. 10 Step 3: Avoid the DB MySQL was never going to be able to handle 10,000 writes/s, nor 50,000 reads “Hey, Django does memcached. Problem solved” Help, our memcached server I/O is maxed out :( Two-layer cache: https://gist.github.com/953524 Write-behind dataThursday, 26 May 2011 10
  • 11. 11 But we want analytics! Now 10K things to write to disk every second Logging? Database? This is starting to look like BIG DATAThursday, 26 May 2011 11
  • 12. 12 Step 4: BabyThursday, 26 May 2011 12
  • 13. 13 Step 5: Cassandra Deployed Cassandra cluster on EC2 to handle buzz records Tested to > 10K writes/s All good! “So how many users did we have last night?”Thursday, 26 May 2011 13
  • 14. 14 Where we ended... app.livetalkback.com cdn.livetalkback.com 10 Control plane HAProxy HAProxy CloudFront nodes Chef Webserver Webserver 100+ nodes Django Django Ubuntu Ubuntu Zabbix Memcached Cassandra Memcached Cassandra RDS Master S3Thursday, 26 May 2011 14
  • 15. 15 Scaling up - and down Configuring 100+ servers by hand each week would have been a pain Used to Chef to automate Also builds the test swarm http://wiki.opscode.com/display/ chef/HomeThursday, 26 May 2011 15
  • 16. 16 Now what? Still challenges with analytics & ad-hoc queries Looking at Brisk and Hadoop We’re sucking the Twitter firehose for Tellybug MySQL is coping so far, but only justThursday, 26 May 2011 16
  • 17. 17 Questions? boxm@livetalkback.com @malcolmboxThursday, 26 May 2011 17

×