my.opera.com scalability v2
Nordic Perl Workshop ~ Oslo 2009
cosimo streppone <cosimo@cpan.org>

Users (k)
2.500
1.640
887
430
257 205
50
10
1
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

Users (k)
Servers
Dyn req/s
2.500
1.640
887
257 205 430
50
10
1
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

espenao> proxys all over. MOC is dying?
hkmoen> did MOC just die?
...
fred joined #myopera
fred> lots of proxys...
fred left #myopera
...
cosimo> WTF? ok, looking into it

static avatars
(or “put your http servers at work”)

new storage subsystem
pools, servers
fault tolerance, redundancy
webdav, http, ftp,
scp, mogilefs?, ...

user uploads use case
# Create resource object for avatar
my $res = MyOpera::Storage::Resource::Avatar->new(
owner => '{userid}',
content => '{binary data}',
);
# Main storage subsystem handle
my $storage = MyOpera::Storage->new();
# Upload on pools of servers all at once
my $ok = $storage->upload($res);

resources
(user uploads, binary blobs, ...)
pools, servers
URLs
http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_o.png
http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_t.jpg
http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_m.jpg
http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_l.jpg

package HTTP::DAV;
...
#
# Retrieves info about a DAV url
#
# depth=0 avoids descending into dirs
#
sub propfind {
my ($self, $url, $depth) = @_;
$depth ||= 1;
# ...
}

new cool static URLs
http://static.myopera.com/
/avatars/pool1/
/dL/rLA/vdUxYn14kfe342QEM/
/<user-id>_s.jpg
crappy CGI-style URLs
http://my.opera.com/<user>/avatar.pl

/* results */
saved ~500k backend req/day
browser cache used!

shared-nothing
architecture
(“zero tolerance for sharing”)

lvs +
lw httpd
mod_perl
.....
backends
nfs server
disk cache
users store

dogpile effect
(or the cache “storms”)

nfs-based file cache
/var/cache/myopera/
/var/cache/myopera/feeds/
/var/cache/myopera/blogs/
...
/var/cache/myopera/user/
/var/cache/myopera/user/1
/var/cache/myopera/user/10
/var/cache/myopera/user/11
/var/cache/myopera/user/2
...
(another 2 million entries)
...
/var/cache/myopera/user/1999999
(on ext3)

soft counters
use MyOpera::Counter;
my $counter = MyOpera::Counter->new({
key => “blog.access-$entry”,
sync => sub {
my ($value) = @_;
# Real table update code
...
}
});
$counter->inc();

dml statistics
[DML] time=1237308152, user=,
url=/tinh_yeu_cua_anh_b88/blog/index.dml/tag/...,
name=XWA::User, variable=active, type=module,
elapsed=0.068473, host=my.opera.com
[DML] time=1237308152, user=, url=/community/,
name=XWA::User, variable=, type=module, elapsed=0.015935,
host=my.opera.com
[DML] ...

top time-intensive modules
XWA::XMLProd::User::Sidebar 2024.919s (27.2%, 0.28 s/call)
XWA::User 1778.445s (23.9%, 0.09 s/call)
XWA::User::Journal 1121.224s (15.1%, 0.24 s/call)
XWA::User::Album 321.522s ( 4.3%, 0.17 s/call)
XWA::User::Journal::Search 223.477s ( 3.0%, 20.32 s/call)
XWA::User::Comments 188.011s ( 2.5%, 0.05 s/call)
XWA::Skins 180.486s ( 2.4%, 0.49 s/call)
XWA::User::JournalArchive 159.525s ( 2.1%, 4.43 s/call)
XWA::User::Posts 146.644s ( 2.0%, 0.45 s/call)
XWA::User::Picture 141.324s ( 1.9%, 0.10 s/call)
XWA::Albums 93.740s ( 1.3%, 2.04 s/call)
XWA::Journals 92.390s ( 1.2%, 2.37 s/call)

most called modules
XWA::User 18784 calls 28.9%
XWA::XMLProd::User::Sidebar 7186 calls 11.1%
XWA::User::Spotlight 6439 calls 9.9%
XWA::Util::OneTimeKey 5576 calls 8.6%
XWA::User::Journal 4681 calls 7.2%
XWA::OIVWrapper 3974 calls 6.1%
XWA::User::Comments 3890 calls 6.0%
XWA::User::Album 1881 calls 2.9%
XWA::User::Ratings 1500 calls 2.3%
XWA::User::Picture 1485 calls 2.3%
XWA::XMLProd::MemberCount 611 calls 0.9%
XWA::XMLProd::Queue 589 calls 0.9%
XWA::Spotlight 569 calls 0.9%
MyOpera::DML::FastFacts 567 calls 0.9%
XWA::XMLProd::Variables 562 calls 0.9%

slowest modules
XWA::User::Journal::Search 20.32s (11 calls)
XWA::User::JournalArchive 4.43s (36 calls)
XWA::User::GeneratePassword 3.76s (3 calls)
XWA::XMLProd::Link::Bookmarks 3.45s (12 calls)
XWA::Fans 3.37s (10 calls)
XWA::User::ForumLatestFriends 3.36s (11 calls)
XWA::User::ForumNew 2.42s (3 calls)
XWA::Journals 2.37s (39 calls)
XWA::Albums 2.04s (46 calls)
XWA::XMLProd::UserSearch 1.87s (2 calls)
XWA::Jahoo 1.83s (46 calls)
XWA::Forum 1.64s (49 calls)
XWA::XMLProd::Link::Trash 0.86s (2 calls)
XWA::XMLProd::LocationSearch 0.72s (43 calls)
XWA::User::ForumSearch 0.67s (78 calls)

?questions?

low level TCP/IP “throttling”
# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- x.x.x.x/x anywhere
limit: avg 20/sec burst 5
ACCEPT all -- y.y.y.y/x anywhere
limit: avg 20/sec burst 5
DROP all -- x.x.x.x/x anywhere
DROP all -- y.y.y.y/y anywhere

full-page caching
- limiting “barrier”
- url hotlist

mod_cband

Web site monitoring,
in case the live demo fails
:-)