NPW2009 - my.opera.com scalability v2.0

my.opera.com scalability v2 Nordic Perl Workshop ~ Oslo 2009 cosimo streppone <cosimo@cpan.org>

Users (k) 2.500 1.640 887 430 257 205 50 10 1 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

Users (k) Servers Dyn req/s 2.500 1.640 887 257 205 430 50 10 1 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

espenao> proxys all over. MOC is dying? hkmoen> did MOC just die? ... fred joined #myopera fred> lots of proxys... fred left #myopera ... cosimo> WTF? ok, looking into it

static avatars (or “put your http servers at work”)

new storage subsystem pools, servers fault tolerance, redundancy webdav, http, ftp, scp, mogilefs?, ...

user uploads use case # Create resource object for avatar my $res = MyOpera::Storage::Resource::Avatar->new( owner => '{userid}', content => '{binary data}', ); # Main storage subsystem handle my $storage = MyOpera::Storage->new(); # Upload on pools of servers all at once my $ok = $storage->upload($res);

resources (user uploads, binary blobs, ...) pools, servers URLs http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_o.png http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_t.jpg http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_m.jpg http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_l.jpg

package HTTP::DAV; ... # # Retrieves info about a DAV url # # depth=0 avoids descending into dirs # sub propfind { my ($self, $url, $depth) = @_; $depth ||= 1; # ... }

new cool static URLs http://static.myopera.com/ /avatars/pool1/ /dL/rLA/vdUxYn14kfe342QEM/ /<user-id>_s.jpg crappy CGI-style URLs http://my.opera.com/<user>/avatar.pl

/* results */ saved ~500k backend req/day browser cache used!

shared-nothing architecture (“zero tolerance for sharing”)

lvs + lw httpd mod_perl ..... backends nfs server disk cache users store

dogpile effect (or the cache “storms”)

nfs-based file cache /var/cache/myopera/ /var/cache/myopera/feeds/ /var/cache/myopera/blogs/ ... /var/cache/myopera/user/ /var/cache/myopera/user/1 /var/cache/myopera/user/10 /var/cache/myopera/user/11 /var/cache/myopera/user/2 ... (another 2 million entries) ... /var/cache/myopera/user/1999999 (on ext3)

soft counters use MyOpera::Counter; my $counter = MyOpera::Counter->new({ key => “blog.access-$entry”, sync => sub { my ($value) = @_; # Real table update code ... } }); $counter->inc();

dml statistics [DML] time=1237308152, user=, url=/tinh_yeu_cua_anh_b88/blog/index.dml/tag/..., name=XWA::User, variable=active, type=module, elapsed=0.068473, host=my.opera.com [DML] time=1237308152, user=, url=/community/, name=XWA::User, variable=, type=module, elapsed=0.015935, host=my.opera.com [DML] ...

top time-intensive modules XWA::XMLProd::User::Sidebar 2024.919s (27.2%, 0.28 s/call) XWA::User 1778.445s (23.9%, 0.09 s/call) XWA::User::Journal 1121.224s (15.1%, 0.24 s/call) XWA::User::Album 321.522s ( 4.3%, 0.17 s/call) XWA::User::Journal::Search 223.477s ( 3.0%, 20.32 s/call) XWA::User::Comments 188.011s ( 2.5%, 0.05 s/call) XWA::Skins 180.486s ( 2.4%, 0.49 s/call) XWA::User::JournalArchive 159.525s ( 2.1%, 4.43 s/call) XWA::User::Posts 146.644s ( 2.0%, 0.45 s/call) XWA::User::Picture 141.324s ( 1.9%, 0.10 s/call) XWA::Albums 93.740s ( 1.3%, 2.04 s/call) XWA::Journals 92.390s ( 1.2%, 2.37 s/call)

most called modules XWA::User 18784 calls 28.9% XWA::XMLProd::User::Sidebar 7186 calls 11.1% XWA::User::Spotlight 6439 calls 9.9% XWA::Util::OneTimeKey 5576 calls 8.6% XWA::User::Journal 4681 calls 7.2% XWA::OIVWrapper 3974 calls 6.1% XWA::User::Comments 3890 calls 6.0% XWA::User::Album 1881 calls 2.9% XWA::User::Ratings 1500 calls 2.3% XWA::User::Picture 1485 calls 2.3% XWA::XMLProd::MemberCount 611 calls 0.9% XWA::XMLProd::Queue 589 calls 0.9% XWA::Spotlight 569 calls 0.9% MyOpera::DML::FastFacts 567 calls 0.9% XWA::XMLProd::Variables 562 calls 0.9%

slowest modules XWA::User::Journal::Search 20.32s (11 calls) XWA::User::JournalArchive 4.43s (36 calls) XWA::User::GeneratePassword 3.76s (3 calls) XWA::XMLProd::Link::Bookmarks 3.45s (12 calls) XWA::Fans 3.37s (10 calls) XWA::User::ForumLatestFriends 3.36s (11 calls) XWA::User::ForumNew 2.42s (3 calls) XWA::Journals 2.37s (39 calls) XWA::Albums 2.04s (46 calls) XWA::XMLProd::UserSearch 1.87s (2 calls) XWA::Jahoo 1.83s (46 calls) XWA::Forum 1.64s (49 calls) XWA::XMLProd::Link::Trash 0.86s (2 calls) XWA::XMLProd::LocationSearch 0.72s (43 calls) XWA::User::ForumSearch 0.67s (78 calls)

?questions?

low level TCP/IP “throttling” # iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT all -- x.x.x.x/x anywhere limit: avg 20/sec burst 5 ACCEPT all -- y.y.y.y/x anywhere limit: avg 20/sec burst 5 DROP all -- x.x.x.x/x anywhere DROP all -- y.y.y.y/y anywhere

full-page caching - limiting “barrier” - url hotlist

mod_cband

Web site monitoring, in case the live demo fails :-)

NPW2009 - my.opera.com scalability v2.0

by Cosimo Streppone on Apr 20, 2009

Accessibility

Categories

Upload Details

Usage Rights

Statistics

NPW2009 - my.opera.com scalability v2.0 Presentation Transcript