Building Scalable Websites with Perl          Perrin Harkins          Plus Three (http://plusthree.com/)
Who is doing it?• Yahoo  – Overture• Amazon  – IMDB• InterActiveCorp  – Ticketmaster  – CitySearch
How are they doing it?• Not just hardware• Basic techniques
Things we wont be covering• mod_perl tuning  – http://perl.apache.org/• DBI tuning  – http://search.cpan.org/~timb/• hardw...
Caching
Page-Level Caching• Best performance• Pre-generate from batch job                      page                               ...
wgetwget ­­mirror ­­convert­links    ­­html­extension ­­reject gif,jpg,png    ­­no­parent    http://app­server/dynamic/pag...
Generate-On-Demand                                         data            web     mod_proxy    appclient     server     c...
mod_proxyProxyRequests OffProxyPass /dynamic/stuff http://app­  server/ProxyPassReverse /dynamic/stuff   http://app­server...
Intercepting 404 ErrorsErrorDocument 404 /page/generator              web               pageclient             server     ...
Partial-Page Caching                       2 hours                       5 minutes                       24 hours
Mason cache  my $result = $m­>cache­>get(                         $search_term                             );  if (!define...
Cache::FastMmap  our $Cache = Cache::FastMmap­>new(                    cache_size  => 500m,                    expire_time...
Memcachedour $Memd = Cache::Memcached­>new({    servers => [      "10.0.0.15:11211","10.0.0.15:11212",      "10.0.0.17:112...
Job Queuing
Get in Line
“search in progress”                       web client                      server fork     worker          redirect       ...
Spread::Queue clientuse Spread::Queue::Sender;my $sender = Spread::Queue::Sender­>new(                                "myq...
Spread::Queue workeruse Spread::Queue::Worker;my $worker = Spread::Queue::Worker­>new("myqueue");$worker­>callbacks(      ...
Spread::Queue workersub myfunc {    my ($worker, $originator, $input) = @_;          my $result = {        response => "I ...
Resources• “Computer Science and Perl Programming”• “Mastering Algorithms with Perl”• “Patterns of Enterprise Application ...
Thanks!• Craig McLane and Adam Sussman of  Ticketmaster Online• Zack Steinkamp of Yahoo
Upcoming SlideShare
Loading in …5
×

Building Scalable Websites with Perl

2,256 views

Published on

This talk was presented at OSCON 2004, ApacheCon 2004, and YAPC::NA 2004.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,256
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
11
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Building Scalable Websites with Perl

  1. 1. Building Scalable Websites with Perl Perrin Harkins Plus Three (http://plusthree.com/)
  2. 2. Who is doing it?• Yahoo – Overture• Amazon – IMDB• InterActiveCorp – Ticketmaster – CitySearch
  3. 3. How are they doing it?• Not just hardware• Basic techniques
  4. 4. Things we wont be covering• mod_perl tuning – http://perl.apache.org/• DBI tuning – http://search.cpan.org/~timb/• hardware
  5. 5. Caching
  6. 6. Page-Level Caching• Best performance• Pre-generate from batch job page batch cache data job webclient server
  7. 7. wgetwget ­­mirror ­­convert­links    ­­html­extension ­­reject gif,jpg,png    ­­no­parent    http://app­server/dynamic/pages/ wget page cache web appclient data server server
  8. 8. Generate-On-Demand data web mod_proxy appclient server cache server
  9. 9. mod_proxyProxyRequests OffProxyPass /dynamic/stuff http://app­ server/ProxyPassReverse /dynamic/stuff  http://app­server/CacheRoot "/mnt/proxy­cache"CacheSize 500000CacheGcInterval 12CacheMaxExpire 36CacheDefaultExpire 2
  10. 10. Intercepting 404 ErrorsErrorDocument 404 /page/generator web pageclient server cache 404 handler data
  11. 11. Partial-Page Caching 2 hours 5 minutes 24 hours
  12. 12. Mason cache  my $result = $m­>cache­>get(                         $search_term                             );  if (!defined($result)) {      $result = run_search($search_term);      $m­>cache­>set($search_term,                     $result,                                   30 min);  }
  13. 13. Cache::FastMmap  our $Cache = Cache::FastMmap­>new(                    cache_size  => 500m,                    expire_time => 30m,                                   );  $Cache­>set($key, $value);  my $value = $Cache­>get($key);
  14. 14. Memcachedour $Memd = Cache::Memcached­>new({    servers => [      "10.0.0.15:11211","10.0.0.15:11212",      "10.0.0.17:11211",       [ "10.0.0.17:11211", 3 ]    ],    debug   => 0,    compress_threshold => 10_000,                                 });$Memd­>set($key, $value, 5*60 );my $value = $Memd­>get($key);
  15. 15. Job Queuing
  16. 16. Get in Line
  17. 17. “search in progress” web client server fork worker redirect web done client reload worker server yet? resulthttp://www.stonehenge.com/merlyn/WebTechniques/col20.html
  18. 18. Spread::Queue clientuse Spread::Queue::Sender;my $sender = Spread::Queue::Sender­>new(                                "myqueue"                                       );$sender­>submit(    "myfunc", { name => "value" }               );my $response = $sender­>receive();
  19. 19. Spread::Queue workeruse Spread::Queue::Worker;my $worker = Spread::Queue::Worker­>new("myqueue");$worker­>callbacks(           myfunc => &myfunc,                  );$SIG{INT} = &signal_handler;$worker­>run;
  20. 20. Spread::Queue workersub myfunc {    my ($worker, $originator, $input) = @_;          my $result = {        response => "I heard you!",                 };    $worker­>respond($originator, $result);}
  21. 21. Resources• “Computer Science and Perl Programming”• “Mastering Algorithms with Perl”• “Patterns of Enterprise Application Architecture” by Martin Fowler• http://oreillynet.com/
  22. 22. Thanks!• Craig McLane and Adam Sussman of Ticketmaster Online• Zack Steinkamp of Yahoo

×