Scaling the Britain's Got Talent Buzzer
Upcoming SlideShare
Loading in...5
×
 

Scaling the Britain's Got Talent Buzzer

on

  • 2,556 views

How Live Talkback scaled the Britain's Got Talent buzzer to support 50,000 requests/second

How Live Talkback scaled the Britain's Got Talent buzzer to support 50,000 requests/second

Statistics

Views

Total Views
2,556
Views on SlideShare
2,509
Embed Views
47

Actions

Likes
0
Downloads
14
Comments
0

5 Embeds 47

http://lanyrd.com 35
http://fasoulas.posterous.com 6
http://blog.fasoulas.com 4
http://posterous.com 1
http://attentionshard.wordpress.com 1

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Scaling the Britain's Got Talent Buzzer Scaling the Britain's Got Talent Buzzer Presentation Transcript

  • 1 Powering the Britain’s Got Talent buzzer* *And Big Data Big Data Meetup, London 25/5/2011Thursday, 26 May 2011 1
  • 2 What we doThursday, 26 May 2011 2
  • 3 Me Malcolm Box, Co-founder & CTO boxm@livetalkback.com @malcolmboxThursday, 26 May 2011 3
  • 4 The Buzzer BIG DATAThursday, 26 May 2011 4
  • 5 The challenge 10 Million+ viewers Design goal of 50,000 requests/s, 10,000 buzzes/second Equivalent to 130 Billion requests/month But just on Saturday night And four weeks to buildThursday, 26 May 2011 5
  • 6 The challenge Where does 130 Billion requests fit? Source: http://www.google.com/adplanner/static/top1000/#Thursday, 26 May 2011 6
  • 7 Where we started.... app.livetalkback.com cdn.livetalkback.com Control plane ELB CloudFront Zabbix Webserver Webserver Django Django Ubuntu Ubuntu MySQL S3Thursday, 26 May 2011 7
  • 8 Step 1: Testing Started with a platform with a previous peak of 100 requests/s No idea where it would break Tsung! http://tsung.erlang-projects.org/Thursday, 26 May 2011 8
  • 9 Step 2: ELB Amazon Elastic Load Balancer “Infinite capacity” BUT very long impulse response and NO controls :( HAProxy to the rescue 5K requests/s per nodeThursday, 26 May 2011 9
  • 10 Step 3: Avoid the DB MySQL was never going to be able to handle 10,000 writes/s, nor 50,000 reads “Hey, Django does memcached. Problem solved” Help, our memcached server I/O is maxed out :( Two-layer cache: https://gist.github.com/953524 Write-behind dataThursday, 26 May 2011 10
  • 11 But we want analytics! Now 10K things to write to disk every second Logging? Database? This is starting to look like BIG DATAThursday, 26 May 2011 11
  • 12 Step 4: BabyThursday, 26 May 2011 12
  • 13 Step 5: Cassandra Deployed Cassandra cluster on EC2 to handle buzz records Tested to > 10K writes/s All good! “So how many users did we have last night?”Thursday, 26 May 2011 13
  • 14 Where we ended... app.livetalkback.com cdn.livetalkback.com 10 Control plane HAProxy HAProxy CloudFront nodes Chef Webserver Webserver 100+ nodes Django Django Ubuntu Ubuntu Zabbix Memcached Cassandra Memcached Cassandra RDS Master S3Thursday, 26 May 2011 14
  • 15 Scaling up - and down Configuring 100+ servers by hand each week would have been a pain Used to Chef to automate Also builds the test swarm http://wiki.opscode.com/display/ chef/HomeThursday, 26 May 2011 15
  • 16 Now what? Still challenges with analytics & ad-hoc queries Looking at Brisk and Hadoop We’re sucking the Twitter firehose for Tellybug MySQL is coping so far, but only justThursday, 26 May 2011 16
  • 17 Questions? boxm@livetalkback.com @malcolmboxThursday, 26 May 2011 17