SlideShare a Scribd company logo
1 of 17
Download to read offline
Scaling Twitter
      Alex & Britt
    (from Twitter)
Preconceived notions.
The story.
So, what happened?
What we did.
More boxes.
  Not so many then. But now...
‣
  Mongrels across 19 cores.
‣
  Message processing/delivery across 16 cores.
‣
  Jabber across 2 cores.
‣
  MySQL on one big 8 core box.
‣
  16GB+ for memcache across a bunch of
‣
  machines.
‣ 32 cores total.
That’s a lot.
Why?
    200 - 300 connections per second.
‣
    Spiking at 800 connections per second.
‣
    We’ve done 11,000 connections per second.
‣
    MySQL has spiked to 2,400 queries per second.
‣
    Alexa says we have a whole mess o’ traffic, and
‣
    they’re not even seeing the API traffic.
memcache.
memcache.
memcache.
Cache the hell out of everything.
Denormalize.
find(:all,
     :conditions => quot;substring(
        truncate(id,0),-2,1) =
        #{@fugly_dist_idx}quot;,
     :limit => 10)
Starling.
Community.
YOU CAN HAS QUESTIONS

More Related Content

Similar to Scaling Twitter - Railsconf 2007

Memory-Based Cloud Architectures
Memory-Based Cloud ArchitecturesMemory-Based Cloud Architectures
Memory-Based Cloud Architectures小新 制造
 
Introduction to Blockchains
Introduction to BlockchainsIntroduction to Blockchains
Introduction to BlockchainsRamesh Nair
 
Handling 20 billion requests a month
Handling 20 billion requests a monthHandling 20 billion requests a month
Handling 20 billion requests a monthDmitriy Dumanskiy
 
Deep Learning for Developers (January 2018)
Deep Learning for Developers (January 2018)Deep Learning for Developers (January 2018)
Deep Learning for Developers (January 2018)Julien SIMON
 
Optimizing Your Cluster with Coordinator Nodes (Eric Lubow, SimpleReach) | Ca...
Optimizing Your Cluster with Coordinator Nodes (Eric Lubow, SimpleReach) | Ca...Optimizing Your Cluster with Coordinator Nodes (Eric Lubow, SimpleReach) | Ca...
Optimizing Your Cluster with Coordinator Nodes (Eric Lubow, SimpleReach) | Ca...DataStax
 
Denis Radin - Mining crypto in browser_ GPU, WebAssembly and all the good thi...
Denis Radin - Mining crypto in browser_ GPU, WebAssembly and all the good thi...Denis Radin - Mining crypto in browser_ GPU, WebAssembly and all the good thi...
Denis Radin - Mining crypto in browser_ GPU, WebAssembly and all the good thi...Maria Kuneva
 
Denis Radin - Mining crypto in browser_ GPU, WebAssembly and all the good thi...
Denis Radin - Mining crypto in browser_ GPU, WebAssembly and all the good thi...Denis Radin - Mining crypto in browser_ GPU, WebAssembly and all the good thi...
Denis Radin - Mining crypto in browser_ GPU, WebAssembly and all the good thi...OdessaJS Conf
 
Mining crypto in browser as a bleeding edge performance challenge for the Web...
Mining crypto in browser as a bleeding edge performance challenge for the Web...Mining crypto in browser as a bleeding edge performance challenge for the Web...
Mining crypto in browser as a bleeding edge performance challenge for the Web...Denis Radin
 
Network Flow Analysis
Network Flow AnalysisNetwork Flow Analysis
Network Flow Analysisguest23ccda3
 
Network Flow Analysis
Network Flow AnalysisNetwork Flow Analysis
Network Flow Analysisguest23ccda3
 

Similar to Scaling Twitter - Railsconf 2007 (13)

Memory-Based Cloud Architectures
Memory-Based Cloud ArchitecturesMemory-Based Cloud Architectures
Memory-Based Cloud Architectures
 
Introduction to Blockchains
Introduction to BlockchainsIntroduction to Blockchains
Introduction to Blockchains
 
Handling 20 billion requests a month
Handling 20 billion requests a monthHandling 20 billion requests a month
Handling 20 billion requests a month
 
Deep Learning for Developers (January 2018)
Deep Learning for Developers (January 2018)Deep Learning for Developers (January 2018)
Deep Learning for Developers (January 2018)
 
Deep Learning for Developers
Deep Learning for DevelopersDeep Learning for Developers
Deep Learning for Developers
 
Optimizing Your Cluster with Coordinator Nodes (Eric Lubow, SimpleReach) | Ca...
Optimizing Your Cluster with Coordinator Nodes (Eric Lubow, SimpleReach) | Ca...Optimizing Your Cluster with Coordinator Nodes (Eric Lubow, SimpleReach) | Ca...
Optimizing Your Cluster with Coordinator Nodes (Eric Lubow, SimpleReach) | Ca...
 
Denis Radin - Mining crypto in browser_ GPU, WebAssembly and all the good thi...
Denis Radin - Mining crypto in browser_ GPU, WebAssembly and all the good thi...Denis Radin - Mining crypto in browser_ GPU, WebAssembly and all the good thi...
Denis Radin - Mining crypto in browser_ GPU, WebAssembly and all the good thi...
 
Denis Radin - Mining crypto in browser_ GPU, WebAssembly and all the good thi...
Denis Radin - Mining crypto in browser_ GPU, WebAssembly and all the good thi...Denis Radin - Mining crypto in browser_ GPU, WebAssembly and all the good thi...
Denis Radin - Mining crypto in browser_ GPU, WebAssembly and all the good thi...
 
Mining crypto in browser as a bleeding edge performance challenge for the Web...
Mining crypto in browser as a bleeding edge performance challenge for the Web...Mining crypto in browser as a bleeding edge performance challenge for the Web...
Mining crypto in browser as a bleeding edge performance challenge for the Web...
 
Network Flow Analysis
Network Flow AnalysisNetwork Flow Analysis
Network Flow Analysis
 
Network Flow Analysis
Network Flow AnalysisNetwork Flow Analysis
Network Flow Analysis
 
Deep Learning for Developers
Deep Learning for DevelopersDeep Learning for Developers
Deep Learning for Developers
 
The Smug Mug Tale
The Smug Mug TaleThe Smug Mug Tale
The Smug Mug Tale
 

More from Alex Payne

Splitting up your web app
Splitting up your web appSplitting up your web app
Splitting up your web appAlex Payne
 
The perils and rewards of working on stuff that matters
The perils and rewards of working on stuff that mattersThe perils and rewards of working on stuff that matters
The perils and rewards of working on stuff that mattersAlex Payne
 
Emerging Languages: A Tour of the Horizon
Emerging Languages: A Tour of the HorizonEmerging Languages: A Tour of the Horizon
Emerging Languages: A Tour of the HorizonAlex Payne
 
The Why and How of Scala at Twitter
The Why and How of Scala at TwitterThe Why and How of Scala at Twitter
The Why and How of Scala at TwitterAlex Payne
 
Building Distributed Systems in Scala
Building Distributed Systems in ScalaBuilding Distributed Systems in Scala
Building Distributed Systems in ScalaAlex Payne
 
Speedy, Stable, and Secure: Better Web Apps Through Functional Languages
Speedy, Stable, and Secure: Better Web Apps Through Functional LanguagesSpeedy, Stable, and Secure: Better Web Apps Through Functional Languages
Speedy, Stable, and Secure: Better Web Apps Through Functional LanguagesAlex Payne
 
Mind The Tools
Mind The ToolsMind The Tools
Mind The ToolsAlex Payne
 
Strange Loop 2009 Keynote: Minimalism in Computing
Strange Loop 2009 Keynote: Minimalism in ComputingStrange Loop 2009 Keynote: Minimalism in Computing
Strange Loop 2009 Keynote: Minimalism in ComputingAlex Payne
 
The Business Value of Twitter
The Business Value of TwitterThe Business Value of Twitter
The Business Value of TwitterAlex Payne
 
Twitter API 2.0
Twitter API 2.0Twitter API 2.0
Twitter API 2.0Alex Payne
 
The Interaction Design Of APIs
The Interaction Design Of APIsThe Interaction Design Of APIs
The Interaction Design Of APIsAlex Payne
 
Why Scala for Web 2.0?
Why Scala for Web 2.0?Why Scala for Web 2.0?
Why Scala for Web 2.0?Alex Payne
 
The Twitter API: A Presentation to Adobe
The Twitter API: A Presentation to AdobeThe Twitter API: A Presentation to Adobe
The Twitter API: A Presentation to AdobeAlex Payne
 
Protecting Public Hotspots
Protecting Public HotspotsProtecting Public Hotspots
Protecting Public HotspotsAlex Payne
 
Twitter at BarCamp 2008
Twitter at BarCamp 2008Twitter at BarCamp 2008
Twitter at BarCamp 2008Alex Payne
 
Securing Rails
Securing RailsSecuring Rails
Securing RailsAlex Payne
 
Designing Your API
Designing Your APIDesigning Your API
Designing Your APIAlex Payne
 

More from Alex Payne (18)

Splitting up your web app
Splitting up your web appSplitting up your web app
Splitting up your web app
 
The perils and rewards of working on stuff that matters
The perils and rewards of working on stuff that mattersThe perils and rewards of working on stuff that matters
The perils and rewards of working on stuff that matters
 
Emerging Languages: A Tour of the Horizon
Emerging Languages: A Tour of the HorizonEmerging Languages: A Tour of the Horizon
Emerging Languages: A Tour of the Horizon
 
The Why and How of Scala at Twitter
The Why and How of Scala at TwitterThe Why and How of Scala at Twitter
The Why and How of Scala at Twitter
 
Building Distributed Systems in Scala
Building Distributed Systems in ScalaBuilding Distributed Systems in Scala
Building Distributed Systems in Scala
 
Speedy, Stable, and Secure: Better Web Apps Through Functional Languages
Speedy, Stable, and Secure: Better Web Apps Through Functional LanguagesSpeedy, Stable, and Secure: Better Web Apps Through Functional Languages
Speedy, Stable, and Secure: Better Web Apps Through Functional Languages
 
Mind The Tools
Mind The ToolsMind The Tools
Mind The Tools
 
Strange Loop 2009 Keynote: Minimalism in Computing
Strange Loop 2009 Keynote: Minimalism in ComputingStrange Loop 2009 Keynote: Minimalism in Computing
Strange Loop 2009 Keynote: Minimalism in Computing
 
The Business Value of Twitter
The Business Value of TwitterThe Business Value of Twitter
The Business Value of Twitter
 
Twitter API 2.0
Twitter API 2.0Twitter API 2.0
Twitter API 2.0
 
The Interaction Design Of APIs
The Interaction Design Of APIsThe Interaction Design Of APIs
The Interaction Design Of APIs
 
Why Scala for Web 2.0?
Why Scala for Web 2.0?Why Scala for Web 2.0?
Why Scala for Web 2.0?
 
The Twitter API: A Presentation to Adobe
The Twitter API: A Presentation to AdobeThe Twitter API: A Presentation to Adobe
The Twitter API: A Presentation to Adobe
 
Protecting Public Hotspots
Protecting Public HotspotsProtecting Public Hotspots
Protecting Public Hotspots
 
Twitter at BarCamp 2008
Twitter at BarCamp 2008Twitter at BarCamp 2008
Twitter at BarCamp 2008
 
Securing Rails
Securing RailsSecuring Rails
Securing Rails
 
Why Scala?
Why Scala?Why Scala?
Why Scala?
 
Designing Your API
Designing Your APIDesigning Your API
Designing Your API
 

Recently uploaded

3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud DataEric D. Schabell
 
Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FESTBillieHyde
 
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox
 
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxEmil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxNeo4j
 
March Patch Tuesday
March Patch TuesdayMarch Patch Tuesday
March Patch TuesdayIvanti
 
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechWebinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechProduct School
 
AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024Brian Pichman
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameKapil Thakar
 
UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2DianaGray10
 
Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.IPLOOK Networks
 
Planetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl
 
EMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarEMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarThousandEyes
 
How to release an Open Source Dataweave Library
How to release an Open Source Dataweave LibraryHow to release an Open Source Dataweave Library
How to release an Open Source Dataweave Libraryshyamraj55
 
Extra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfExtra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfInfopole1
 
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxGraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxNeo4j
 
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInOutage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInThousandEyes
 
The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)IES VE
 
Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...DianaGray10
 
UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3DianaGray10
 

Recently uploaded (20)

3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data
 
Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FEST
 
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
 
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxEmil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
 
March Patch Tuesday
March Patch TuesdayMarch Patch Tuesday
March Patch Tuesday
 
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechWebinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
 
AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First Frame
 
UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2
 
Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.
 
Planetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile Brochure
 
SheDev 2024
SheDev 2024SheDev 2024
SheDev 2024
 
EMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarEMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? Webinar
 
How to release an Open Source Dataweave Library
How to release an Open Source Dataweave LibraryHow to release an Open Source Dataweave Library
How to release an Open Source Dataweave Library
 
Extra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfExtra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdf
 
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxGraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
 
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInOutage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
 
The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)
 
Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...
 
UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3
 

Scaling Twitter - Railsconf 2007