Scaling the Britain's Got Talent Buzzer

•

0 likes•1,046 views

Malcolm Box

How Live Talkback scaled the Britain's Got Talent buzzer to support 50,000 requests/second

Technology Entertainment & Humor

1

Powering the Britain’s Got Talent
buzzer*
*And Big Data

Big Data Meetup, London 25/5/2011

Thursday, 26 May 2011 1

3

Me

Malcolm Box, Co-founder & CTO

boxm@livetalkback.com

@malcolmbox

Thursday, 26 May 2011 3

4

The Buzzer

BIG DATA

Thursday, 26 May 2011 4

5

The challenge

10 Million+ viewers

Design goal of 50,000 requests/s, 10,000 buzzes/second

Equivalent to 130 Billion requests/month

But just on Saturday night

And four weeks to build

Thursday, 26 May 2011 5

6

The challenge

Where does 130
Billion requests ﬁt?

Source: http://www.google.com/adplanner/static/top1000/#

Thursday, 26 May 2011 6

7

Where we started....

app.livetalkback.com cdn.livetalkback.com

Control plane
ELB CloudFront

Zabbix
Webserver Webserver
Django Django
Ubuntu Ubuntu

MySQL S3

Thursday, 26 May 2011 7

8

Step 1: Testing

Started with a platform with a previous peak of 100 requests/s

No idea where it would break

Tsung! http://tsung.erlang-projects.org/

Thursday, 26 May 2011 8

9

Step 2: ELB

Amazon Elastic Load Balancer

“Inﬁnite capacity”

BUT very long impulse response and NO controls :(

HAProxy to the rescue

5K requests/s per node

Thursday, 26 May 2011 9

10

Step 3: Avoid the DB

MySQL was never going to be able to handle 10,000 writes/s, nor 50,000
reads

“Hey, Django does memcached. Problem solved”

Help, our memcached server I/O is maxed out :(

Two-layer cache: https://gist.github.com/953524

Write-behind data

Thursday, 26 May 2011 10

11

But we want analytics!

Now 10K things to write to disk every second

Logging? Database?

This is starting to look like BIG DATA

Thursday, 26 May 2011 11

12

Step 4: Baby

Thursday, 26 May 2011 12

13

Step 5: Cassandra

Deployed Cassandra cluster on EC2 to handle buzz records

Tested to > 10K writes/s

All good!

“So how many users did we have last night?”

Thursday, 26 May 2011 13

14

Where we ended...

app.livetalkback.com cdn.livetalkback.com
10
Control plane
HAProxy HAProxy CloudFront nodes

Chef

Webserver Webserver 100+
nodes
Django Django
Ubuntu Ubuntu
Zabbix

Memcached Cassandra
Memcached Cassandra RDS Master S3

Thursday, 26 May 2011 14

15

Scaling up - and down

Conﬁguring 100+ servers by
hand each week would have
been a pain

Used to Chef to automate

Also builds the test swarm

http://wiki.opscode.com/display/
chef/Home

Thursday, 26 May 2011 15

16

Now what?

Still challenges with analytics & ad-hoc queries

Looking at Brisk and Hadoop

We’re sucking the Twitter ﬁrehose for Tellybug

MySQL is coping so far, but only just

Thursday, 26 May 2011 16

17

Questions?
boxm@livetalkback.com

@malcolmbox

Thursday, 26 May 2011 17

Similar to Scaling the Britain's Got Talent Buzzer

Retrospective from a startup built in the cloud: top three big lessons learne...

Jeff Malek

Caridy patino - node-js

StarTech Conference

Conquistando el Servidor con Node.JS

Caridy Patino

http://wiki4.caucho.com/Java_EE_Tutorial_covering_JSP_2.2,_and_Servlets_3.0 This tutorial focuses on using Servlet's and JSP the right way. Servlet and JSP have evolved over the years, and now there is often more than one way to do things. For example, this tutorial uses EL and JSTL not JSP scriptlets, it uses JSPs in a Model 2/MVC style not in a model 1 style, etc. Consider it a tutorial that focuses only on the best practices and not the legacy ways to do things. There are other tutorials on this subject, but this tutorial is going to be different in that it is going to put all code into github, and you can follow along with Eclipse. Also instead of focusing on JSF, we are going to focus on JSP and Servlets as the main view technology. Java EE, JSP and Servlets have added a lot of features that are in other frameworks, making those other frameworks less relevant then they were before Java EE garnered these extra abilities. Even is you decide to use JSF, Struts, Stripes, Spring MVC, JSF, etc., this tutorial should help you have a better understanding of the JSP/Servlets core that they build on. We are going to start by building a simple bookstore. We will progressively add more features to the bookstore and as we do we will use more of Java EE/CDI, JSP and Servlets. For this tutorial, I am going to use Resin 4.0.x, but you could use any Java EE 6 Web Profile compliant application server.

Java EE Servlet JSP Tutorial- Cookbook 1

billdigman

Scaling Selenium

Noah Sussman

AppScale Talk at SBonRails

Chris Bunch

This was a talk given internally at BloomReach as well as a guest lecture to a grad level Data Structures and Algorithms class at the University of Texas at Arlington. A Deep dive on the history of containers, and how they work under the cover utilizing Linux Kernel features such as Process Namespaces and Control Groups. I also go over a bit of the history of Container technology, going from Chroot and Jails and Zones, to LXC and Docker

Corwin on Containers

Corwin Brown

What python can learn from java

jbellis

presentation-chaos-monkey

Matthew Campbell

2011 02-14-libre

Vu Hung Nguyen

WTF Is Messaging And Why You Should Use It?

James Russell

20110903 candycane

Yusuke Ando

Shake Hooves With BeEF - OWASP AppSec APAC 2012

Christian Frichot

Big Data Israel Meetup : Couchbase and Big Data

Tugdual Grall

We all feel alone sometimes. Akka got along well with the VM crew ever since it was born, but new friends and fresh ideas are always necessary. Which is why lately Akka loves spending time with Kubernetes! Maybe the reason why they like each other so much is their sharing of core values such as transparent scalability and resilience. How do these two technologies compare from a Reactive standpoint? Does one supersede the other? In fact, their powers can be combined to design distributed systems all the way from application code to cloud instance.

We all need friends and Akka just found Kubernetes

Fabio Tiriticco

Session presented at the 2nd IndicThreads.com Conference on Cloud Computing held in Pune, India on 3-4 June 2011. http://CloudComputing.IndicThreads.com Amazon Elastic Beanstalk provides developers an easy way to build and deploy highly scalable web applications. Elastic Beanstalk automatically handles the deployment details of capacity provisioning, load balancing, auto-scaling, and application health monitoring. Elastic Beanstalk leverages AWS services such as Amazon EC2, Amazon S3, Amazon Simple Notification Service, Elastic Load Balancing, and Auto-Scaling to deliver the same highly reliable, scalable, and cost-effective infrastructure that hundreds of thousands of businesses depend on today. This session will look at building and deploying application on Amazon’s Elastic Beanstalk. Speaker: Janakiram MSV is the Technology Evangelist at Amazon Web Services (AWS) in India. His responsibilities include enabling ISVs, SIs and enterprises with the skills to move their IT infrastructure and applications to the Cloud. In his previous role, he was the Technology Architect – Cloud at Microsoft Corporation where he was responsible for architecting enterprise solutions on the Microsoft Cloud Computing platform.

Amazon Elastic Beanstalk - Indicthreads.com cloud computing conference 2011

IndicThreads

HTTP cache @ PUG Rome 03-29-2011

Alessandro Nadalin

Architecting for the Cloud: demo and best practices, by Simone Brunozzi (2011...

Amazon Web Services

I will tell you war stories from Kubernetes implementations in two startups: a fashion ecommerce - Lykehq - and Fintech/Machine Learning - SMACC. Getting them to Continuous Deployment, mistakes I made, and how we solve them. Show why K8S is such a powerful tool and -- most important for me -- it gives you learn-as-you-go experience. The new linux, the new application server some say. Check: https://github.com/wojciech12/talk_cloudnative_and_kubernetes_waw

Effective Platform Building with Kubernetes. Is K8S new Linux?

Wojciech Barczyński

2 my sql overview technology etc[1]. etc

Prof. Erwin Globio

Similar to Scaling the Britain's Got Talent Buzzer (20)

Retrospective from a startup built in the cloud: top three big lessons learne...

Caridy patino - node-js

Conquistando el Servidor con Node.JS

Java EE Servlet JSP Tutorial- Cookbook 1

Scaling Selenium

AppScale Talk at SBonRails

Corwin on Containers

What python can learn from java

presentation-chaos-monkey

2011 02-14-libre

WTF Is Messaging And Why You Should Use It?

20110903 candycane

Shake Hooves With BeEF - OWASP AppSec APAC 2012

Big Data Israel Meetup : Couchbase and Big Data

We all need friends and Akka just found Kubernetes

Amazon Elastic Beanstalk - Indicthreads.com cloud computing conference 2011

HTTP cache @ PUG Rome 03-29-2011

Architecting for the Cloud: demo and best practices, by Simone Brunozzi (2011...

Effective Platform Building with Kubernetes. Is K8S new Linux?

2 my sql overview technology etc[1]. etc

Recently uploaded

Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf

FIDO Alliance

When you think of a highly secure meeting environment, do you instantly think 'Microsoft Teams'!? Or do you think about some unknown application, troublesome UI and daunting login process...? If you think the latter - let's change that! In this session Femke will show you how using Teams Premium features can create secure, but also good looking meetings! PRETTY. Make sure your company's brand is represented before, during and after the meeting with Customization policies in place. SECURE. Lets utilize Meeting templates and Sensitivity Labels to protect your meeting and data to prevent sensitive information from being leaked. After this session, you will have a clear understanding of the capabilities of Teams Premium features and how to set up the perfect meeting that suits your organizational requirements!

ECS 2024 Teams Premium - Pretty Secure

Femke de Vroome

PLAI - Acceleration Program for Generative A.I. Startups

Stefano

Discover the top Symfony development companies that excel in creating robust and scalable web applications. Our latest blog highlights the best firms specializing in Symfony, known for their expertise in delivering high-performance solutions. Whether you’re looking to start a new project or enhance an existing one, these companies offer the skills and experience needed to bring your vision to life. Read more to find your perfect Symfony development partner.

Top 10 Symfony Development Companies 2024

TopCSSGallery

Oauth 2.0 Introduction and Flows with MuleSoft

shyamraj55

AI presentation and introduction - Retrieval Augmented Generation RAG 101

vincent683379

Where to Learn More About FDO _ Richard at FIDO Alliance.pdf

FIDO Alliance

Extensible Python: Robustness through Addition - PyCon 2024

Patrick Viafore

Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...

FIDO Alliance

This talk offers actionable insights at an executive level for enhancing productivity and refining your portfolio management approach to propel your organization to greater heights. Key Points Covered: 1. Experience Transformation: - The core challenge remains consistent across organizations: converting budget into user-centric designs. - Strategies for deploying design resources effectively in both startups and large enterprises. 2. Strategic Frameworks: - Introduction to the "Ziggurat of Impact" model, detailing layers from basic system interactions to comprehensive customer experiences. - Practical insights on creating frameworks that scale with organizational complexity. 3. Organizational Impact: - Real-world examples of navigating design in large settings, focusing on the synthesis of consumer products and customer experiences. - Emphasis on the importance of designing systems that directly influence customer interactions. 4. Design Execution: - Detailed walkthrough of organizational layers affecting design execution, from touchpoints and customer activities to shared capabilities. - How to ensure design influences both the micro and macro aspects of customer interactions. 5. Measurement and Adaptation: - Techniques for measuring the impact of design decisions and adapting strategies based on data-driven insights. - The critical role of continuous improvement and feedback in refining customer experiences.

Structuring Teams and Portfolios for Success

UXDXConf

Join me in this session where I'll share our journey of building a fully serverless application that flawlessly managed check-ins for an event with a staggering 80 thousand registrations. We'll dive into three key strategies that made this possible. Firstly, by harnessing DynamoDB global tables, we ensured global service availability and data replication across regions, boosting performance and disaster recovery. Next, we'll explore how we seamlessly integrated real-time updates into the app using Appsync subscriptions, making the experience dynamic and engaging for users. Finally, I'll discuss how provisioned concurrency not only improved performance but also kept costs in check, highlighting the cost-effectiveness of serverless architectures. Through these strategies and the inherent scalability of serverless technology, our application effortlessly handled massive user loads without manual intervention. This session is a real world example to the power and efficiency of modern cloud-based solutions in enabling seamless scalability and robust performance with Serverless

How we scaled to 80K users by doing nothing!.pdf

Srushith Repakula

Designing inclusive products is not only a social responsibility but also a business imperative. This talk delves into the journey of creating accessible hardware products that cater to diverse user needs. Key Topics Covered: 1. Introduction to Inclusive Design - Importance of accessibility in product design - Overview of Comcast's commitment to making products accessible to a wide audience 2. Case Study: Xfinity Large Button Voice Remote - Initial challenges and the evolution of the product - User research and feedback that shaped the design - Key features of the final product and their benefits 3. Designing for Diverse Needs - Understanding human-centered design and its historical context - The impact of designing for people with disabilities on overall product quality - Examples from other industries, such as architecture and industrial design 4. Integrating Accessibility from the Beginning - The cost and efficiency benefits of designing for accessibility from the start - The process of embedding accessibility as a core trait rather than an optional feature 5. Real-World Impact and Continuous Improvement - Insights from in-home studies with users having assistive needs - How continuous feedback and iterative design lead to better products - The role of inclusive research and development practices

Designing for Hardware Accessibility at Comcast

UXDXConf

IESVE for Early Stage Design and Planning

IES VE

New customer? New industry? New cloud? New team? A lot to handle! How to ensure the success of the project? Start it well! I've created the 3 areas of focus at the beginning of the project that helped me in multiple roles (BA, PO, and Consultant). Learn from real-world experiences and discover how these insights can empower you to deliver unparalleled value to your customers right from the project's start.

Powerful Start- the Key to Project Success, Barbara Laskowska

CzechDreamin

This talk focuses on the practical aspects of integrating various telephony systems with Salesforce, drawing on examples from implementations in the Czech scene. It aims to inform attendees about the spectrum of telephony solutions available, from small to large scale, and their compatibility with Salesforce. The presentation will highlight key considerations for selecting a telephony provider that integrates smoothly with Salesforce, including important questions to support the decision-making process. It will also discuss methods for integrating existing telephony systems with Salesforce, aimed at companies contemplating or in the process of adopting this CRM platform. The discussion is designed to provide a straightforward overview of the steps and considerations involved in telephony and Salesforce integration, with an emphasis on functionality, compatibility, and the practical experiences of Czech companies.

Integrating Telephony Systems with Salesforce: Insights and Considerations, B...

CzechDreamin

Syngulon’s technology expands the capacity for selection of microorganisms. The ability to select individual microbes with a behavior of interest is essential, whether for simple cloning at the bench, or for industry-scale production. Synthetic biology uses the concept of “bioengineering” to improve or modify existing genetic systems to create microbes with desired behaviors, and Syngulon uses this approach to develop its selection technologies. This selection technology is based on bacteriocins, ribosomally-produced peptides naturally made by most bacteria to kill competitive microbial species. These bacteriocins can have a limited or wide target range against other microbial species. This technology offers advantageous over antibiotic selection for several reasons: it avoids the use of antibiotics in the first place, helping to reduce the spread of antibiotic resistant microbes. The technology also increases product yield; as bacteriocins are generally smaller peptides, they do not impose a heavy metabolic burden on the producing cell. They can have a wide target specificity, helping to avoid genetic drift. Finally, our system is 100% plasmid-based (e.g. without chromosomal mutations), making it applicable for use in any E. coli strains.

Syngulon - Selection technology May 2024.pdf

Syngulon

Screen flow is a powerful automation tool that is commonly designed for internal and external users. However, what about the guest users? We will dive into various methods of launching screen flows and understand how to make them publicly accessible, extending their usability to a broader audience. The presentation will also cover the implementation of security layers and highlight best practices for a smooth and protected user experience. Discover the potential of screen flows beyond conventional use and learn how to leverage them effectively.

Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade

CzechDreamin

Automation is transforming industries by streamlining operations and reducing costs. However, the success of automated systems hinges not only on their functionality but also on the user experience (UX). This talk, "The UX of Automated Processes," delves into the intricate relationship between automation and UX, providing a comprehensive guide to designing intuitive and effective automated systems. Key Topics Covered: 1. Introduction to Automated UX: * Understanding the importance of UX in automated systems * Real-world example: Robotic beer dispensers at a music festival 2. Step-by-Step Process for Enhancing Automated UX: * Step 0: Identifying main workflows and happy paths * Step 1: Conducting contextual inquiries to understand current user practices and edge cases * Step 2: Prioritizing issues and solutions through collaborative ideation sessions * Step 3: Prototyping and usability testing for both software and hardware 3. Prototyping and Testing: * Methods for simulating automated processes * Importance of user feedback in refining automated systems 4. Designing for Support Users: * Crafting the experience for support personnel who manage edge cases * Ensuring effective communication and issue prioritization for support users

The UX of Automation by AJ King, Senior UX Researcher, Ocado

UXDXConf

Speed Wins: From Kafka to APIs in Minutes

confluent

ScyllaDB has the potential to deliver impressive performance and scalability. The better you understand how it works, the more you can squeeze out of it. But before you squeeze, make sure you know what to monitor! Watch our experienced Postgres developer work through monitoring and performance strategies that help him understand what mistakes he’s made moving to NoSQL. And learn with him as our database performance expert offers friendly guidance on how to use monitoring and performance tuning to get his sample Rust application on the right track. This webinar focuses on using monitoring and performance tuning to discover and correct mistakes that commonly occur when developers move from SQL to NoSQL. For example: - Common issues getting up and running with the monitoring stack - Using the CQL optimizations dashboard - Common issues causing high latency in a node - Common issues causing replica imbalance - What a healthy system looks like in terms of memory - Key metrics to keep an eye on This isn’t “Death-by-Powerpoint.” We’ll walk through problems encountered while migrating a real application from Postgres to ScyllaDB – and try to fix them live as well.

Optimizing NoSQL Performance Through Observability

ScyllaDB

Recently uploaded (20)

Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf

ECS 2024 Teams Premium - Pretty Secure

PLAI - Acceleration Program for Generative A.I. Startups

Top 10 Symfony Development Companies 2024

Oauth 2.0 Introduction and Flows with MuleSoft

AI presentation and introduction - Retrieval Augmented Generation RAG 101

Where to Learn More About FDO _ Richard at FIDO Alliance.pdf

Extensible Python: Robustness through Addition - PyCon 2024

Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...

Structuring Teams and Portfolios for Success

How we scaled to 80K users by doing nothing!.pdf

Designing for Hardware Accessibility at Comcast

IESVE for Early Stage Design and Planning

Powerful Start- the Key to Project Success, Barbara Laskowska

Integrating Telephony Systems with Salesforce: Insights and Considerations, B...

Syngulon - Selection technology May 2024.pdf

Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade

The UX of Automation by AJ King, Senior UX Researcher, Ocado

Speed Wins: From Kafka to APIs in Minutes

Optimizing NoSQL Performance Through Observability

Scaling the Britain's Got Talent Buzzer

1. 1 Powering the Britain’s Got Talent buzzer* *And Big Data Big Data Meetup, London 25/5/2011 Thursday, 26 May 2011 1

2. 2 What we do Thursday, 26 May 2011 2

3. 3 Me Malcolm Box, Co-founder & CTO boxm@livetalkback.com @malcolmbox Thursday, 26 May 2011 3

4. 4 The Buzzer BIG DATA Thursday, 26 May 2011 4

5. 5 The challenge 10 Million+ viewers Design goal of 50,000 requests/s, 10,000 buzzes/second Equivalent to 130 Billion requests/month But just on Saturday night And four weeks to build Thursday, 26 May 2011 5

6. 6 The challenge Where does 130 Billion requests ﬁt? Source: http://www.google.com/adplanner/static/top1000/# Thursday, 26 May 2011 6

7. 7 Where we started.... app.livetalkback.com cdn.livetalkback.com Control plane ELB CloudFront Zabbix Webserver Webserver Django Django Ubuntu Ubuntu MySQL S3 Thursday, 26 May 2011 7

8. 8 Step 1: Testing Started with a platform with a previous peak of 100 requests/s No idea where it would break Tsung! http://tsung.erlang-projects.org/ Thursday, 26 May 2011 8

9. 9 Step 2: ELB Amazon Elastic Load Balancer “Inﬁnite capacity” BUT very long impulse response and NO controls :( HAProxy to the rescue 5K requests/s per node Thursday, 26 May 2011 9

10. 10 Step 3: Avoid the DB MySQL was never going to be able to handle 10,000 writes/s, nor 50,000 reads “Hey, Django does memcached. Problem solved” Help, our memcached server I/O is maxed out :( Two-layer cache: https://gist.github.com/953524 Write-behind data Thursday, 26 May 2011 10

11. 11 But we want analytics! Now 10K things to write to disk every second Logging? Database? This is starting to look like BIG DATA Thursday, 26 May 2011 11

12. 12 Step 4: Baby Thursday, 26 May 2011 12

13. 13 Step 5: Cassandra Deployed Cassandra cluster on EC2 to handle buzz records Tested to > 10K writes/s All good! “So how many users did we have last night?” Thursday, 26 May 2011 13

14. 14 Where we ended... app.livetalkback.com cdn.livetalkback.com 10 Control plane HAProxy HAProxy CloudFront nodes Chef Webserver Webserver 100+ nodes Django Django Ubuntu Ubuntu Zabbix Memcached Cassandra Memcached Cassandra RDS Master S3 Thursday, 26 May 2011 14

15. 15 Scaling up - and down Conﬁguring 100+ servers by hand each week would have been a pain Used to Chef to automate Also builds the test swarm http://wiki.opscode.com/display/ chef/Home Thursday, 26 May 2011 15

16. 16 Now what? Still challenges with analytics & ad-hoc queries Looking at Brisk and Hadoop We’re sucking the Twitter ﬁrehose for Tellybug MySQL is coping so far, but only just Thursday, 26 May 2011 16

17. 17 Questions? boxm@livetalkback.com @malcolmbox Thursday, 26 May 2011 17

Scaling the Britain's Got Talent Buzzer

Recommended

Recommended

More Related Content

Similar to Scaling the Britain's Got Talent Buzzer

Similar to Scaling the Britain's Got Talent Buzzer (20)

Recently uploaded

Recently uploaded (20)

Scaling the Britain's Got Talent Buzzer