my.opera.com scalability v1
         Italian Perl Workshop ~ Pisa 2008
               cosimo streppone   <cosimo@cpan.org>
Scalability
   ==
Performance
Scalability
    !=
Performance
Scalability
    ==
memcached?
Scalability
   !=
memcached
My Opera Community
             http://my.opera.com


        blogs                          shoutbox
Opera browser promot...
2001
2003
2005
2006
2008
my.opera.com

small (un)known web sites
Users (k)
                                                                2.500 *




                                    ...
Users (k)
                              Servers

                                                               2.500 *

 ...
Users (k)
                              Servers
                              Dyn req/s




                              ...
proxy errors
the zen way to
indefinite scaling
Start



                   Y
 Does it run?

                       Does it   N
         N             scale?
 No problem....
/* recommendations */


  server-side statistics

  good stress testing tools
shared-nothing
  architecture
  (“zero tolerance for sharing”)
lvs +
        lw httpd



        mod_perl
.....
        backends



        nfs server
        disk cache
        users s...
lvs +
        lw httpd



        mod_perl
.....
        backends



        nfs server
        disk cache
        users s...
lvs +
        lw httpd



        mod_perl
.....
        backends



        nfs server
        disk cache
        users s...
... not so easy ...
dev.opera.com
articles cache
(or “don't guess, profile!”)
/* problem */


       1 function
(XWA::Quote2Dev::Quote)

   ~95%   req time
/* solution */
     Start



                   Y   Great. Serve
  In Cache?            from cache.

          N

Query DB...
/* solution */

      Yes, that simple!

      Profiling is key.

           try:
m{^ Devel:: .* Prof .*}x
static avatars
(or “put your http servers at work”)
/* problem */

   a single forum page
            ~~
        50 avatars

(crappy CGI backend requests)
new storage subsystem
          pools, servers
  fault tolerance, redundancy

      webdav, http, ftp,
      scp, mogilefs...
user uploads use case
# Create resource object for avatar
my $res = MyOpera::Storage::Resource::Avatar->new(
    owner => ...
resources
                   (user uploads, binary blobs, ...)




              pools, servers




                      ...
/* results */


saved ~500k backend req/day

    browser cache used!
dogpile effect
(or the cache “storms”)
/* problem */


very high load spikes
  on DB, backends
/* cause */


many concurrent clients
build same cached items
/* solution */


cache contention algorithm

      NFS-resistant
  optimistic file locking
/* solution */

    ideas taken from
HTML::Mason, Django, ...

 patched File::NFSLock
/* further ideas */

 LOCK_SPIN_TIME


distributed cache
  on backends
Opera releases
/* problem */


 10-30x normal load

handling of peak load?
/* try 1 */


 load-balanced
database profiles
/* try 2 */



url hot-list
/* solution? */


no solution this time,
        sorry.
/* further ideas */


   Amazon EC2 ?
Akamai CDN services?
        ... ?
soft counters
     (didn't work as expected, in progress...)




sacrifice speed for concurrency
         (mysql's myisam ...
/* future directions */

         database   partitioning?
   modperl backend optimization?
threading sucks in mp2. debian...
take-aways


find your PKIs
take-aways


always compare
before and after
take-aways


stress testing in your
    release cycle
take-aways


have fun!
   ;-)
?questions?
thanks!
IPW2008 - my.opera.com scalability
IPW2008 - my.opera.com scalability
IPW2008 - my.opera.com scalability
IPW2008 - my.opera.com scalability
IPW2008 - my.opera.com scalability
IPW2008 - my.opera.com scalability
IPW2008 - my.opera.com scalability
IPW2008 - my.opera.com scalability
Upcoming SlideShare
Loading in …5
×

IPW2008 - my.opera.com scalability

9,890 views
9,497 views

Published on

Opera Software is mainly known for its web browser, but Opera also created and develops a growing set of web applications like its own community dedicated to the browser, my.opera.com, started in 2001.

Since then, the community grew bigger and bigger, and had the "usual" scalability problems. The application has been rewritten 3 times and the current developers team is at a turning point now.

How to make the application and systems scalable with the growing amount of traffic and users?

Published in: Technology
40 Comments
5 Likes
Statistics
Notes
No Downloads
Views
Total views
9,890
On SlideShare
0
From Embeds
0
Number of Embeds
23
Actions
Shares
0
Downloads
1
Comments
40
Likes
5
Embeds 0
No embeds

No notes for slide

IPW2008 - my.opera.com scalability

  1. my.opera.com scalability v1 Italian Perl Workshop ~ Pisa 2008 cosimo streppone <cosimo@cpan.org>
  2. Scalability == Performance
  3. Scalability != Performance
  4. Scalability == memcached?
  5. Scalability != memcached
  6. My Opera Community http://my.opera.com blogs shoutbox Opera browser promotion photo albums RSS-Atom feeds slideshows custom designs Public APIs xml/json forums
  7. 2001
  8. 2003
  9. 2005
  10. 2006
  11. 2008
  12. my.opera.com small (un)known web sites
  13. Users (k) 2.500 * 1.640 IPW2008 887 430 257 205 1 10 50 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
  14. Users (k) Servers 2.500 * IPW2008 1.640 887 430 257 205 1 10 50 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
  15. Users (k) Servers Dyn req/s 2.500* 1.640 887 257 205 430 1 10 50 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
  16. proxy errors
  17. the zen way to indefinite scaling
  18. Start Y Does it run? Does it N N scale? No problem... Identify and fix Wait and restart Y bottlenecks End
  19. /* recommendations */ server-side statistics good stress testing tools
  20. shared-nothing architecture (“zero tolerance for sharing”)
  21. lvs + lw httpd mod_perl ..... backends nfs server disk cache users store
  22. lvs + lw httpd mod_perl ..... backends nfs server disk cache users store
  23. lvs + lw httpd mod_perl ..... backends nfs server disk cache users store
  24. ... not so easy ...
  25. dev.opera.com articles cache (or “don't guess, profile!”)
  26. /* problem */ 1 function (XWA::Quote2Dev::Quote) ~95% req time
  27. /* solution */ Start Y Great. Serve In Cache? from cache. N Query DB, etc... Done! Put it in cache this time!
  28. /* solution */ Yes, that simple! Profiling is key. try: m{^ Devel:: .* Prof .*}x
  29. static avatars (or “put your http servers at work”)
  30. /* problem */ a single forum page ~~ 50 avatars (crappy CGI backend requests)
  31. new storage subsystem pools, servers fault tolerance, redundancy webdav, http, ftp, scp, mogilefs?, ...
  32. user uploads use case # Create resource object for avatar my $res = MyOpera::Storage::Resource::Avatar->new( owner => '{userid}', content => '{binary data}', ); # Main storage subsystem handle my $storage = MyOpera::Storage->new(); # Upload on pools of servers all at once my $ok = $storage->upload($res);
  33. resources (user uploads, binary blobs, ...) pools, servers URLs http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_o.png http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_t.jpg http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_m.jpg http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_l.jpg
  34. /* results */ saved ~500k backend req/day browser cache used!
  35. dogpile effect (or the cache “storms”)
  36. /* problem */ very high load spikes on DB, backends
  37. /* cause */ many concurrent clients build same cached items
  38. /* solution */ cache contention algorithm NFS-resistant optimistic file locking
  39. /* solution */ ideas taken from HTML::Mason, Django, ... patched File::NFSLock
  40. /* further ideas */ LOCK_SPIN_TIME distributed cache on backends
  41. Opera releases
  42. /* problem */ 10-30x normal load handling of peak load?
  43. /* try 1 */ load-balanced database profiles
  44. /* try 2 */ url hot-list
  45. /* solution? */ no solution this time, sorry.
  46. /* further ideas */ Amazon EC2 ? Akamai CDN services? ... ?
  47. soft counters (didn't work as expected, in progress...) sacrifice speed for concurrency (mysql's myisam locks all table on update)
  48. /* future directions */ database partitioning? modperl backend optimization? threading sucks in mp2. debian anyone? improve memory sharing? sysadmin-level optimizations?
  49. take-aways find your PKIs
  50. take-aways always compare before and after
  51. take-aways stress testing in your release cycle
  52. take-aways have fun! ;-)
  53. ?questions?
  54. thanks!

×