Ruby and Twitter at the Ruby drink-up of Sophia, January 2013
Ruby Drink-up / January 2013http://rivierarb.fr/2013/01/08/Drinkup/Karan Malkani
ABOUT THESE SLIDES…A Presentation of the Twitter / Ruby couple and why it did not lastMotivated by my own interest in Ruby, born 2 years ago when reading that Twitter was made using Ruby (or Ruby on Rails)An Internet-based desk researchBut…Not so much / easy available information due to the excellent Twitter communication
TWITTER HISTORY WITH (A BIG) SHORTCUT Born in the postcasting company Odeo Initial idea: an SMS service within a small group / Original code name: twttr / First prototype was used as an internal service for Odeo employees / Full version introduced publicly in July 2006 October 2006: Jack Dorsey, Biz Stonen, and Evan Williams formed Obvious Corporation, acquired Odeo (including twitter.com). Twitter, as a company, was launched in April 2007 Today: the 2nd social network in the world after Facebook: (Estimated Unique Monthly Visitors – January 2013) • 1 Facebook - 750,000,000 • 2 Twitter - 250,000,000 • 3 LinkedIn - 110,000,000 • 4 Pinterest - 85,500,000 • 5 MySpace - 70,500,000 • 6 Google Plus+ - 65,000,000 Source: How Twitter Was Born http://www.140characters.com/2009/01/30/how-twitter-was-born/ Twitter Development History in 3 minute and 53 seconds! http://www.shoutmeloud.com/twitter-development-history-in-3- minute-and-53-seconds.html Wikipedia: Twitter http://en.wikipedia.org/wiki/Twitter eBizMBA (http://www.ebizmba.com/articles/social-networking-websites)
A FANTASTIC GROWTH – (> 100M NEW USERS IN 2010) Source: http://blog.twitter.com/2010/02/measuring-tweets.html http://mashable.com/2010/12/16/twitter-stats-2010/
THE PROBLEM Source: http://www.whatisfailwhale.info/ http://en.wikipedia.org/wiki/Twitter#Outages http://www.wired.com/underwire/2008/07/twitter-fans-tu/
WHY SCALING?Number of usersNumber of messages (impacted by the Retweet feature realeased end 2009)Applications that can publish tweets / Applications that access tweetsSearch Engine (Twitter acquired Summize – July 2008)2009 : Some critical outages / Many FailWhales experiencesJune 12, 2009 – In what was called a potential "Twitpocalypse", the unique numerical identifier associated with each tweet exceeded the limit of 32-bit signedintegers (2,147,483,647 total messages). While Twitter itself was not affected, some third-party clients could no longer access recent tweets. Patcheswere quickly released, though some iPhone applications had to wait for approval from the App Store.June 25, 2009 – Twitter ran slowly for some time after over 50,000 tweets on Michael Jackson’s death were recorded in an hour.August 6, 2009 – Twitter and Facebook suffered from a denial-of-service attack, causing the Twitter website to go offline for several hours. It was later confirmed that the attacks were directed at one pro-Georgian user around the anniversary of the 2008 South Ossetia War, rather than the sites themselves.September 22, 2009 – The identifier exceeded the limit for 32-bit unsigned integers (4,294,967,296 total messages) again breaking some third-party clients. Source: http://blog.twitter.com/2009/11/retweet-limited-rollout.html http://techcrunch.com/2008/07/15/confirmed-twitter-acquires-summize-search- engine/ http://en.wikipedia.org/wiki/Twitter#Outages http://www.businessinsider.com/twitter-doesnt-have-product-geniuses-so-its- buying-them-2010-4
STARTING TWITTER WITH RUBYAn excellent choice to Quickly prototype Quickly launch a first real site Quickly deploy an API for third-party applications Tease and attract developersImproving performances with Ruby:Example - mid 2009: Evan Weaver (Twitter)We recently migrated Twitter from a custom Ruby 1.8.6 build to a Ruby Enterprise Edition release candidate, courtesy of Phusion. Our primary motivation was the integration of Brent’s MBARI patches, which increase memory stability.Some features of REE have no effect on our codebase, but we definitely benefit from the MBARI patchset, the Railsbench tunable GC, and the various leak fixes in 1.8.7p174. These are difficult to integrate and Phusion has done a fine job. Source: http://blog.evanweaver.com/2009/09/24/ree/ http://www.linkedin.com/in/evanweaver/
RUBY ISSUES KNOWN WHEN TWITTER WAS FOUNDEDFULL COPY/PASTE OF: HOW HAS RUBY ON RAILS BEEN HOLDING UP TO THE INCREASED LOAD? (5 QUESTION INTERVIEW WITH TWITTERDEVELOPER ALEX PAYNE – APRIL 2007)By various metrics Twitter is the biggest Rails site on the net right All the convenience methods and syntactical sugar that makesnow. Rails such a pleasure for coders ends up being absolutely punishing, performance-wise.Running on Rails has forced us to deal with scaling issues - issuesthat any growing site eventually contends with – far sooner than I Once you hit a certain threshold of traffic, either you need tothink we would on another framework. strip out all the costly neat stuff that Rails does for you (RJS, ActiveRecord, ActiveSupport, etc.) or move the slow parts ofThe common wisdom in the Rails community at this time is that your application out of Rails, or both.scaling Rails is a matter of cost: just throw more CPUs at it. It’s also worth mentioning that there shouldn’t be doubt inThe problem is that more instances of Rails (running as part of a anybody’s mind at this point that Ruby itself is slow.Mongrel cluster, in our case) means more requests to your database. It’s great that people are hard at work on faster implementationsAt this point in time there’s no facility in Rails to talk to more than one of the language, but right now, it’s tough. If you’re looking todatabase at a time. deploy a big web application and you’re language-agnostic,The solutions to this are caching the hell out of everything and realize that the same operation in Ruby will take less time insetting up multiple read-only slave databases, neither of which are Python.quick fixes to implement. All of us working on Twitter are big Ruby fans, but I think it’sSo it’s not just cost, it’s time, and time is that much more precious worth being frank that this isn’t one of those relativistic languagewhen people can[‘t] reach your site.None of these scaling issues. Ruby is slow.approaches are as fun and easy as developing for Rails. Source: http://yasulab.tumblr.com/post/10271634919/5-question-interview-with- twitter-developer-alex-payne/ The original article on Radical Behavior is no longer accessible
ABANDONNING RUBY ON RAILSApril 2008 - After 2 years of high scaling problems: decision is made to abandon RoRStart to change the infrastructure: Middleware moved to a mixture of C and Scala/JVM to run a messaging model, asynchronous process, 3 levels of cache (Twitter kept Ruby for the front-end only)April 2009 - Twitter on ScalaSpring 2010 – changing the back-end from MySQL to a real-time version of Lucene (Apache database)2011 – RoR front-end for search replaced with Blender, a Java server Source: http://techcrunch.com/2008/05/01/twitter-said-to-be-abandoning-ruby-on-rails/ http://www.artima.com/scalazine/articles/twitter_on_scala.html http://engineering.twitter.com/2011/04/twitter-search-is-now-3x-faster_1656.html
THE “GEEK” SLIDE (I DON’T UNDERSTAND THIS STUFF)> Ruby sideMongrelKiji (generational garbage collector)Starling (light-weight persistent queue server speaking MemCache protocol)> Java sideModified Lucene (real-time version of the java database)Optimized JSON fragment cache (to handle initial page load and every call from the client)Mustache (for the rendering – client and server side)oEmbed (standard to handle the inline media) Source: http://engineering.twitter.com/2010/09/tech-behind-new-twittercom.htm http://engineering.twitter.com/2010/10/twitters-new-search-architecture.html http://engineering.twitter.com/2011/04/twitter-search-is-now-3x-faster_1656.html http://www.infoq.com/news/2012/11/twitter-ruby-to-java
HOW TO TERMINATE THIS SLIDE SET?Sorry - This was not so much about RubyYellowpages.com did exactly the contrary in 2008, moving from Java on Rails to Ruby on RailsThe “New Twitter” is fantastic – and survived the US Election end 2012. Some impressive numbers: 327,452 tweets/minute, 31 M election-related tweets during the day, with a 15,107 tweets/second peak)Back to June 2009 (copy/paste from Scaling Twitter: Making Twitter 10000 Percent Faster) Twitter started as a side project and blew up fast, going from 0 to millions of page views within a few terrifying months. Early design decisions that worked well in the small melted under the crush of new users chirping tweets to all their friends. Web darling Ruby on Rails was fingered early for the scaling problems, but Blaine Cook, Twitters lead architect, held Ruby blameless: For us, it’s really about scaling horizontally - to that end, Rails and Ruby haven’t been stumbling blocks, compared to any other language or framework. The performance boosts associated with a “faster” language would give us a 10-20% improvement, but thanks to architectural changes that Ruby and Rails happily accommodated, Twitter is 10000% faster than it was in January. If Ruby on Rails wasnt to blame, how did Twitter learn to scale ever higher and higher? Source: http://techcrunch.com/2008/05/01/twitter-said-to-be-abandoning-ruby- on-rails/ http://highscalability.com/scaling-twitter-making-twitter-10000-percent- faster http://www.infoq.com/news/2012/11/twitter-ruby-to-java
A particular slide catching your eye?
Clipping is a handy way to collect important slides you want to go back to later.