0
From One to a Cluster
 Brian Moon, Senior Developer - dealnews.com
       2008 MySQL Conference & Expo
        http://deal...
The Early Years
• 1997 - 1999 Shared Account
   • Hand edited HTML
   • Perl, PHP, msql
• 1999 - 2000 Dedicated Servers
  ...
The first cluster

• 3-5 Web servers
• 1 NFS server
• 1 MySQL server
• 1 mail server
Bottlenecks
• Software load balancing
 • Wave effect
 • Closed, required specific OS version
• NFS did not scale
 • Disk Ca...
Solutions
• Hardware load balancing
 • Arrowpoint/Cisco (from eBay)
 • F5 BIG-IP (not cheap)
• Drop NFS
 • Memcached - dis...
Status in 2006
• 5 web nodes w/ hardware lb
• Using rsync to put code on
  servers
• 1 MySQL server
• memcached to cache d...
Yahoo! Effect
                                                    2nd Yahoo!

         Digg
Radio Tour      “Cyber Monday”...
First Yahoo!
Yahoo! Effect
                                                    2nd Yahoo!

         Digg
Radio Tour      “Cyber Monday”...
Second Yahoo!
New Bottlenecks

• Cache stampede
 • 1000 requests for the same thing
• Bandwidth
 • Image bandwidth alone hit 60Mb/s
• Hu...
Solutions
• Offload CSS, Javascript and images
  bandwidth to CDN
• Cache content in memory at the forward
  facing servers...
Using a CDN
Pros
• Offloads bandwidth
• Many locations, hopefully near your users
• Bandwidth is cheaper than you can buy i...
Caching Proxy
• Custom PHP script
 • Researched Squid and wrote Perl and
    Python versions too.
• Uses memcached for cac...
Pushed Cache
• User requests can never cause database load
 • No cache stampede
• Data can be prepared when we are ready
 ...
Straight from the DB?

• Use EXPLAIN a lot
• Avoid filesort and temporary
• Avoid complex joins (or any)
• Use InnoDB (row ...
Current Architecture Overview


                 Internet
                             Main DB




Proxy    App    Replica...
Load balancing
We cheat a lot with F5 BIG-IP
• Balances incoming traffic
• Balances internal services
• Not cheap, but wort...
From One to a Cluster
               Not MySQL Cluster, sorry =(

 Brian Moon, Senior Developer - dealnews.com
       2008...
Upcoming SlideShare
Loading in...5
×

From One to a Cluster

1,954

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,954
On Slideshare
0
From Embeds
0
Number of Embeds
22
Actions
Shares
0
Downloads
28
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "From One to a Cluster"

  1. 1. From One to a Cluster Brian Moon, Senior Developer - dealnews.com 2008 MySQL Conference & Expo http://dealnews.com/developers/
  2. 2. The Early Years • 1997 - 1999 Shared Account • Hand edited HTML • Perl, PHP, msql • 1999 - 2000 Dedicated Servers • Developed custom CMS • Dynamic content with PHP and MySQL
  3. 3. The first cluster • 3-5 Web servers • 1 NFS server • 1 MySQL server • 1 mail server
  4. 4. Bottlenecks • Software load balancing • Wave effect • Closed, required specific OS version • NFS did not scale • Disk Cache • Code on NFS
  5. 5. Solutions • Hardware load balancing • Arrowpoint/Cisco (from eBay) • F5 BIG-IP (not cheap) • Drop NFS • Memcached - distributed memory cache • Use rsync to push changes to production
  6. 6. Status in 2006 • 5 web nodes w/ hardware lb • Using rsync to put code on servers • 1 MySQL server • memcached to cache data from database • All pages built “on the fly” from cache (hopefully)
  7. 7. Yahoo! Effect 2nd Yahoo! Digg Radio Tour “Cyber Monday” Yahoo Front Page
  8. 8. First Yahoo!
  9. 9. Yahoo! Effect 2nd Yahoo! Digg Radio Tour “Cyber Monday” Yahoo Front Page
  10. 10. Second Yahoo!
  11. 11. New Bottlenecks • Cache stampede • 1000 requests for the same thing • Bandwidth • Image bandwidth alone hit 60Mb/s • Hundreds of lines of code =(
  12. 12. Solutions • Offload CSS, Javascript and images bandwidth to CDN • Cache content in memory at the forward facing servers • Use a “Pushed Cache” • Refactor all the code
  13. 13. Using a CDN Pros • Offloads bandwidth • Many locations, hopefully near your users • Bandwidth is cheaper than you can buy it Cons • Out of your control • More complicated to invalidate objects
  14. 14. Caching Proxy • Custom PHP script • Researched Squid and wrote Perl and Python versions too. • Uses memcached for cache storage • One copy of an item in cache, not several • Apache 2 worker MPM (Yes, it does work!) • Tried lighttpd with FastCGI as well
  15. 15. Pushed Cache • User requests can never cause database load • No cache stampede • Data can be prepared when we are ready • De-normalized to be ready for the site • Data can come straight from MySQL • Scales out with MySQL Replication
  16. 16. Straight from the DB? • Use EXPLAIN a lot • Avoid filesort and temporary • Avoid complex joins (or any) • Use InnoDB (row locking / transactions) • Write a library/object for code to use
  17. 17. Current Architecture Overview Internet Main DB Proxy App Replicated Process DB
  18. 18. Load balancing We cheat a lot with F5 BIG-IP • Balances incoming traffic • Balances internal services • Not cheap, but worth it for us Linux Virtual Server • Open source • Up/Down monitoring is not built in
  19. 19. From One to a Cluster Not MySQL Cluster, sorry =( Brian Moon, Senior Developer - dealnews.com 2008 MySQL Conference & Expo http://dealnews.com/developers/
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×