Improving Running
  Components

  Evan Weaver
  Twitter, Inc.
  QCon London, 2009
Many tools:

  Rails
    C
  Scala
   Java
 MySQL
Rails front-end:

    rendering
cache composition
   db querying
Middleware:

  Memcached
Varnish (cache)
 Kestrel (MQ)
 comet server
Milestone 1: Cache policy

   Optimization plan:

    1. stop working
   2. share the work
     3. work faster
Old
Everything runs from
memory in Web 2.0.
First policy change:
       vector cache

Stores arrays of tweet pkeys
       Write-through
        99% hit rate
Second policy change:
       row cache

Store records from the db
    (Tweets and users)
      Write-through
       95% hi...
Third policy change:
     fragment cache

Stores rendered version of
    tweets for the API
      Read-through
       95% ...
Fourth policy change:
giving the page cache its own
          cache pool

     Generational keys
     Low hit rate (40%)
Visibility was lacking.

      Peep tool
Dumps a live memcached
         heap
Cache only was living
                         five hours
mysql> select round(round(log10(3576669 - last_read_time) * 5, 0)...
What does a timeline miss
        mean?

     Container union

/home rebuild reads through
  your followings’ profiles
New
Milestone 2:
     Message queue

A component with problems
Purpose in a web app:

Move operations out of the
synchronous request cycle

 Amortize load over time
Inauguration, 2009
Simplest MQ ever:
Gives up constraints for scalability
    No strict ordering of jobs
 No shared state among servers
     ...
First version was written in
            Ruby

  Ruby is “optimization-
        resistant”
  Mainly due to the GC
If the consumers could not
keep pace, the MQ would fill
         up and crash

 Ported it to Scala for this
          reason
Good tooling
for the Java GC:

   JConsole
    Yourkit
Poor tooling for the Ruby
          GC:

 Railsbench w/patches
 BleakHouse w/patches
  Valgrind/Memcheck
  MBARI 1.8.6 pat...
Our Railsbench GC tunings

      35% speed increase
RUBY_HEAP_MIN_SLOTS=500000
RUBY_HEAP_SLOTS_INCREMENT=250000
RUBY_HEAP_...
Situational decision:

Scala is a flexible language
(But libraries a bit lacking)
We have experienced JVM
         engineers
Big rewrites fail...?

      Small rewrite:
  No new features added
   Well-defined interface
Already went over the wire
Deployed to 1 MQ host

    Fixed regressions

Eventually deployed to all
          hosts
Milestone 3:
 the memcached client

Optimizing a critical path
Switched to libmemcached, a
 new C Memcached client

We are now the biggest user
  and biggest 3rd-party
       contributor
Uses a SWIG Ruby binding I
 started a year or so ago

  Compatibility among
memcached clients is critical
Twitter is big, and runs hot

Flushing the cache would be
        catastrophic
Spent endless time on
backwards compatibility

A/B tested the new client
     over 3 months
MQ also benefitted

Memcached can be a generic
lightweight service protocol

   We also use Thrift and
     HTTP internally
So many RPCs! Sometimes
 100s of Memcached round
     trips per request.

  “As a memory device gets
larger, it tends to g...
Performance hierarchy is
 supposed to look like:
At web scale, it looks more
           like:
End
Links:
  C tools:
 - Peep http://github.com/fauna/peep/
 - Libmemcached http://tangent.org/552/libmemcached.html
 - Valgri...
twitter.com/evan

blog.evanweaver.com

    cloudbur.st
Improving Running Components at Twitter
Improving Running Components at Twitter
Improving Running Components at Twitter
Improving Running Components at Twitter
Upcoming SlideShare
Loading in...5
×

Improving Running Components at Twitter

38,537

Published on

We will learn how a small team at Twitter keeps up with breakneck growth, by focusing on three performance milestones completed in the last year.

First, an upgraded caching strategy. Second, our new open source message queue. Third, optimizations made to the internal cache client. Real-life graphs will be viewed along the way, and we will discuss how the lessons learned can be applied to your own projects.

Published in: News & Politics, Technology
3 Comments
126 Likes
Statistics
Notes
  • twitter tools
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Great high-value content, good work. Very generously shared.

    Sam
    http://www.cross-country-running-shoes.net
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Very useful stuff. It would be much better if providing more details or samples

    [Comment posted from http://blog.evanweaver.com/articles/2009/03/13/qcon-presentation/]
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
38,537
On Slideshare
0
From Embeds
0
Number of Embeds
13
Actions
Shares
0
Downloads
1,134
Comments
3
Likes
126
Embeds 0
No embeds

No notes for slide

Improving Running Components at Twitter

  1. 1. Improving Running Components Evan Weaver Twitter, Inc. QCon London, 2009
  2. 2. Many tools: Rails C Scala Java MySQL
  3. 3. Rails front-end: rendering cache composition db querying
  4. 4. Middleware: Memcached Varnish (cache) Kestrel (MQ) comet server
  5. 5. Milestone 1: Cache policy Optimization plan: 1. stop working 2. share the work 3. work faster
  6. 6. Old
  7. 7. Everything runs from memory in Web 2.0.
  8. 8. First policy change: vector cache Stores arrays of tweet pkeys Write-through 99% hit rate
  9. 9. Second policy change: row cache Store records from the db (Tweets and users) Write-through 95% hit rate
  10. 10. Third policy change: fragment cache Stores rendered version of tweets for the API Read-through 95% hit rate
  11. 11. Fourth policy change: giving the page cache its own cache pool Generational keys Low hit rate (40%)
  12. 12. Visibility was lacking. Peep tool Dumps a live memcached heap
  13. 13. Cache only was living five hours mysql> select round(round(log10(3576669 - last_read_time) * 5, 0) / 5, 1) as log, round(avg(3576669 - last_read_time), -2) as freshness, count(*), rpad('', count(*) / 2000, '*') as bar from entries group by log order by log desc; +------+-----------+----------+------------------------------------------------------------------------------------------------------------------+ | log | freshness | count(*) | bar | +------+-----------+----------+------------------------------------------------------------------------------------------------------------------+ | NULL | 0| 13400 | ******* | | 6.6 | 3328300 | 940 | | | 6.2 | 1623200 | 1| | | 5.2 | 126200 | 1| | | 5.0 | 81100 | 343 | | | 4.8 | 64800 | 3200 | ** | | 4.6 | 34800 | 18064 | ********* | | 4.4 | 24200 | 96739 | ************************************************ | | 4.2 | 15700 | 212865 | ********************************************************************************************************** | | 4.0 | 10200 | 224703 | **************************************************************************************************************** | | 3.8 | 6500 | 158067 | ******************************************************************************* | | 3.6 | 4100 | 108034 | ****************************************************** | | 3.4 | 2600 | 82000 | ***************************************** | | 3.2 | 1600 | 65637 | ********************************* | | 3.0 | 1000 | 49267 | ************************* | | 2.8 | 600 | 34398 | ***************** | | 2.6 | 400 | 24322 | ************ | | 2.4 | 300 | 19865 | ********** | | 2.2 | 200 | 14810 | ******* | | 2.0 | 100 | 10108 | ***** | | 1.8 | 100 | 8002 | **** | | 1.6 | 0| 6479 | *** | | 1.4 | 0| 4014 | ** | | 1.2 | 0| 2297 | * | | 1.0 | 0| 1733 | * | | 0.8 | 0| 649 | | | 0.6 | 0| 710 | | | 0.4 | 0| 672 | | | 0.0 | 0| 319 | | +------+-----------+----------+------------------------------------------------------------------------------------------------------------------+
  14. 14. What does a timeline miss mean? Container union /home rebuild reads through your followings’ profiles
  15. 15. New
  16. 16. Milestone 2: Message queue A component with problems
  17. 17. Purpose in a web app: Move operations out of the synchronous request cycle Amortize load over time
  18. 18. Inauguration, 2009
  19. 19. Simplest MQ ever: Gives up constraints for scalability No strict ordering of jobs No shared state among servers Just like memcached Uses memcached protocol
  20. 20. First version was written in Ruby Ruby is “optimization- resistant” Mainly due to the GC
  21. 21. If the consumers could not keep pace, the MQ would fill up and crash Ported it to Scala for this reason
  22. 22. Good tooling for the Java GC: JConsole Yourkit
  23. 23. Poor tooling for the Ruby GC: Railsbench w/patches BleakHouse w/patches Valgrind/Memcheck MBARI 1.8.6 patches
  24. 24. Our Railsbench GC tunings 35% speed increase RUBY_HEAP_MIN_SLOTS=500000 RUBY_HEAP_SLOTS_INCREMENT=250000 RUBY_HEAP_SLOTS_GROWTH_FACTOR=1 RUBY_GC_MALLOC_LIMIT=50000000 RUBY_HEAP_FREE_MIN=4096
  25. 25. Situational decision: Scala is a flexible language (But libraries a bit lacking) We have experienced JVM engineers
  26. 26. Big rewrites fail...? Small rewrite: No new features added Well-defined interface Already went over the wire
  27. 27. Deployed to 1 MQ host Fixed regressions Eventually deployed to all hosts
  28. 28. Milestone 3: the memcached client Optimizing a critical path
  29. 29. Switched to libmemcached, a new C Memcached client We are now the biggest user and biggest 3rd-party contributor
  30. 30. Uses a SWIG Ruby binding I started a year or so ago Compatibility among memcached clients is critical
  31. 31. Twitter is big, and runs hot Flushing the cache would be catastrophic
  32. 32. Spent endless time on backwards compatibility A/B tested the new client over 3 months
  33. 33. MQ also benefitted Memcached can be a generic lightweight service protocol We also use Thrift and HTTP internally
  34. 34. So many RPCs! Sometimes 100s of Memcached round trips per request. “As a memory device gets larger, it tends to get slower.”
  35. 35. Performance hierarchy is supposed to look like:
  36. 36. At web scale, it looks more like:
  37. 37. End
  38. 38. Links: C tools: - Peep http://github.com/fauna/peep/ - Libmemcached http://tangent.org/552/libmemcached.html - Valgrind http://valgrind.org/ JVM tools: - Kestrel http://github.com/robey/kestrel/ - Smile http://github.com/robey/smile/ - Jconsole http://openjdk.java.net/tools/svc/jconsole/ - Yourkit http://www.yourkit.com/ Ruby tools: - BleakHouse http://github.com/fauna/bleak_house/ - Railsbench Ruby patches http://github.com/skaes/railsbench/ - MBARI Ruby patches http://github.com/brentr/matzruby/tree/v1_8_6_287-mbari General: - Danga stack http://www.danga.com/words/2005_oscon/oscon-2005.pdf - Seymour Cray quote http://books.google.com/books?client=safari&id=qM4Yzf8K9hwC&dq=rapid +development&q=cray&pgis=1 - Last.fm downtime http://blog.last.fm/2008/04/18/possible-lastfm-downtime
  39. 39. twitter.com/evan blog.evanweaver.com cloudbur.st
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×