Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Billions of Hits:
                         Scaling Twitter
                         John Adams
                         Tw...
John Adams                        @netik
                         •   Early Twitter employee (mid-2008)

                 ...
Wednesday, May 5, 2010
Growth.

Wednesday, May 5, 2010
752%
                         2008 Growth
  source: comscore.com - (based only on www traffic, not API)


Wednesday, May 5,...
1358%
                         2009 Growth
  source: comscore.com - (based only on www traffic, not API)


Wednesday, May 5...
12 th
                         most popular
 source: alexa.com (global ranking)


Wednesday, May 5, 2010
55M
                         Tweets per day
                         (640 TPS/sec, 1000 TPS/sec peak)
  source: twitter.co...
600M
                         Searches/Day
  source: twitter.com internal


Wednesday, May 5, 2010
Web      API




Wednesday, May 5, 2010
25%




                Web                  API
                               75%




Wednesday, May 5, 2010
Operations
                         •   What do we do?

                             •   Site Availability

              ...
What have we done?
                         •   Improved response time, reduced latency

                         •   Less...
Operations Mantra

                                          Move to
                  Find       Take
                   ...
Make an attack plan.
                 Symptom     Bottleneck   Vector     Solution

                                      ...
Make an attack plan.
                 Symptom     Bottleneck   Vector     Solution

                                      ...
Finding Weakness
                         •   Metrics + Graphs

                             •   Individual metrics are ir...
Monitoring
                         •   Twitter graphs and reports critical metrics in
                             as nea...
Analyze
                         •   Turn data into information

                             •   Where is the code base g...
Data Analysis
                         •   Instrumenting the world pays off.

                         •   “Data analysis,...
A New World for Admins
                         •   You’re not just a sysadmin anymore

                         •   Analy...
Forecasting                        Curve-fitting for capacity planning
                                        (R, fityk, ...
Internal Dashboard




Wednesday, May 5, 2010
External API Dashbord




                         http://dev.twitter.com/status
Wednesday, May 5, 2010
What’s a Robot ?
                         •   Actual error in the Rails stack (HTTP 500)

                         •   Unc...
What’s a Whale ?
                         •   HTTP Error 502, 503

                         •   Twitter has a hard and fas...
Whale Watcher
     •      Simple shell script,

           •      MASSIVE WIN by @ronpepsi

     •      Whale = HTTP 503 (...
Deploy Watcher
                  Sample window: 300.0 seconds

                  First start time:
                  Mon A...
Feature “Darkmode”
                         •   Specific site controls to enable and disable
                             ...
request flow
                                   Load Balancers

                                 Apache mod_proxy

       ...
unicorn
                         •   A single socket Rails application Server

                             •   Workers pu...
Rails
                         •   Mostly only for front-end and mobile

                         •   Back end mostly Java...
memcached
                         •   memcached isn’t perfect.

                             •   Memcached SEGVs hurt us ...
Loony
                         •   Central machine database (MySQL)

                             •   Python, Django, Para...
Murder
                         •   @lg rocks!

                         •   Bittorrent based replication for deploys

   ...
Kestrel
                   •     @robey

                   •     Works like memcache (same protocol)

                   ...
Asynchronous Requests
                         •   Inbound traffic consumes a unicorn worker

                         •  ...
Daemons
                         •   Daemons touch every tweet

                         •   Many different daemon types a...
Disk is the new Tape.
                         •   Social Networking application profile has
                             ...
Caching
                         •   We’re the real-time web, but lots of caching
                             opportunity...
MySQL
                         •   Sharding large volumes of data is hard

                         •   Replication delay ...
MySQL Challenges
                         •   Replication Delay

                             •   Single threaded. Slow.

...
Relational Databases
                         not a Panacea
                         •   Good for:

                      ...
Database Replication
                         •   Major issues around users and statuses tables

                         ...
Flock
                                                       Flock
          •      Scalable Social Graph Store

         ...
Cassandra
                         •   Originally written by Facebook

                         •   Distributed Data Store...
Lessons Learned
                         •   Instrument everything. Start graphing early.

                         •   Ca...
Join Us!
               @jointheflock




Wednesday, May 5, 2010
Q&A
Wednesday, May 5, 2010
Thanks!
                         •   @jointheflock

                         •   http://twitter.com/jobs

                ...
Upcoming SlideShare
Loading in …5
×
7,980 views

Published on

Finding Weakness
• Metrics + Graphs

• Individual metrics are irrelevant

• We aggregate metrics to find knowledge

• Logs

• SCIENCE!



Wednesday, May 5, 2010

Published in: Technology
  • Be the first to comment

×