Search

IPW2008 - my.opera.com scalability

Loading...

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

41 comments

Comments 1 - 10 of 41 previous next Post a comment

  • + cstrep Cosimo Streppone 2 years ago
    http://defectivebydesign.org
  • + cstrep Cosimo Streppone 2 years ago
    Anyone experience with mod_perl and threading?
  • + cstrep Cosimo Streppone 2 years ago
    Generic counter classes to avoid going to MySQL every time just to update 'visits = visits + 1'. That kills you, if you use MyISAM and the table is 90% read and 10% write.
  • + cstrep Cosimo Streppone 2 years ago
    Amazon EC2 sounds interesting. Akamai is a bit beyond our traffic for now.
  • + cstrep Cosimo Streppone 2 years ago
    Our goal would be to gracefully survive another major Opera release. Let’s see... :-)
  • + cstrep Cosimo Streppone 2 years ago
    This was probably the idea taken from Django and Catalyst’s pagecache. The fact is that I didn’t know about this at the time. :-)
  • + cstrep Cosimo Streppone 2 years ago
    The basic idea is: you don’t want to specify your hostnames when you connect to your database. You specify an LVS hostname, that routes your SQL connections to different (or even one, if you don’t need performances) SQL servers.



    This allows transparent failover and, if you have additional hardware, load splitting over many machines.
  • + cstrep Cosimo Streppone 2 years ago
    When there’s a new Opera browser release, we experience even 30x the normal load.

    Are you supposed to be dimensioned to handle 30x peak load?
  • + cstrep Cosimo Streppone 2 years ago
    We defined as LOCK_SPIN_TIME the time other processes should wait before returning a cache miss when the cache entry is already locked by someone else.



    If at the end of the spinning wait, the cached entry was generated (by the process that owned the lock), that’s like a full cache hit for every other process.
  • + cstrep Cosimo Streppone 2 years ago
    Basically, File::NFSLock was subclassed (with some functions overridden, sadly), to provide *persistent* locks, that don’t go away when the objects go out of scope.

    Another change I made was to add a new method to be able to detect whether the lock was acquired by our machine/process or not, basically parsing the lock file and returning the 'owner' information. If someone is interested in the actual code, speak up.



    From Mason I stole a really clever trick that uses utime() to set the cached item timestamp in the future. This allows the process that first locked the cache entry to elegantly and cheaply serve all other waiting processes a 'stale' copy of the cached item.

Comments 1 - 10 of 41 previous next

Post a comment
Embed Video
Edit your comment Cancel

Notes on slide 1

4 Favorites & 1 Event

IPW2008 - my.opera.com scalability - Presentation Transcript

  1. my.opera.com scalability v1 Italian Perl Workshop ~ Pisa 2008 cosimo streppone <cosimo@cpan.org>
  2. Scalability == Performance
  3. Scalability != Performance
  4. Scalability == memcached?
  5. Scalability != memcached
  6. My Opera Community http://my.opera.com blogs shoutbox Opera browser promotion photo albums RSS-Atom feeds slideshows custom designs Public APIs xml/json forums
  7. 2001
  8. 2003
  9. 2005
  10. 2006
  11. 2008
  12. my.opera.com small (un)known web sites
  13. Users (k) 2.500 * 1.640 IPW2008 887 430 257 205 1 10 50 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
  14. Users (k) Servers 2.500 * IPW2008 1.640 887 430 257 205 1 10 50 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
  15. Users (k) Servers Dyn req/s 2.500* 1.640 887 257 205 430 1 10 50 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
  16. proxy errors
  17. the zen way to indefinite scaling
  18. Start Y Does it run? Does it N N scale? No problem... Identify and fix Wait and restart Y bottlenecks End
  19. /* recommendations */ server-side statistics good stress testing tools
  20. shared-nothing architecture (“zero tolerance for sharing”)
  21. lvs + lw httpd mod_perl ..... backends nfs server disk cache users store
  22. lvs + lw httpd mod_perl ..... backends nfs server disk cache users store
  23. lvs + lw httpd mod_perl ..... backends nfs server disk cache users store
  24. ... not so easy ...
  25. dev.opera.com articles cache (or “don't guess, profile!”)
  26. /* problem */ 1 function (XWA::Quote2Dev::Quote) ~95% req time
  27. /* solution */ Start Y Great. Serve In Cache? from cache. N Query DB, etc... Done! Put it in cache this time!
  28. /* solution */ Yes, that simple! Profiling is key. try: m{^ Devel:: .* Prof .*}x
  29. static avatars (or “put your http servers at work”)
  30. /* problem */ a single forum page ~~ 50 avatars (crappy CGI backend requests)
  31. new storage subsystem pools, servers fault tolerance, redundancy webdav, http, ftp, scp, mogilefs?, ...
  32. user uploads use case # Create resource object for avatar my $res = MyOpera::Storage::Resource::Avatar->new( owner => '{userid}', content => '{binary data}', ); # Main storage subsystem handle my $storage = MyOpera::Storage->new(); # Upload on pools of servers all at once my $ok = $storage->upload($res);
  33. resources (user uploads, binary blobs, ...) pools, servers URLs http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_o.png http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_t.jpg http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_m.jpg http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_l.jpg
  34. /* results */ saved ~500k backend req/day browser cache used!
  35. dogpile effect (or the cache “storms”)
  36. /* problem */ very high load spikes on DB, backends
  37. /* cause */ many concurrent clients build same cached items
  38. /* solution */ cache contention algorithm NFS-resistant optimistic file locking
  39. /* solution */ ideas taken from HTML::Mason, Django, ... patched File::NFSLock
  40. /* further ideas */ LOCK_SPIN_TIME distributed cache on backends
  41. Opera releases
  42. /* problem */ 10-30x normal load handling of peak load?
  43. /* try 1 */ load-balanced database profiles
  44. /* try 2 */ url hot-list
  45. /* solution? */ no solution this time, sorry.
  46. /* further ideas */ Amazon EC2 ? Akamai CDN services? ... ?
  47. soft counters (didn't work as expected, in progress...) sacrifice speed for concurrency (mysql's myisam locks all table on update)
  48. /* future directions */ database partitioning? modperl backend optimization? threading sucks in mp2. debian anyone? improve memory sharing? sysadmin-level optimizations?
  49. take-aways find your PKIs
  50. take-aways always compare before and after
  51. take-aways stress testing in your release cycle
  52. take-aways have fun! ;-)
  53. ?questions?
  54. thanks!

+ Cosimo StrepponeCosimo Streppone, 2 years ago

custom

3866 views, 4 favs, 1 embeds more stats

Opera Software is mainly known for its web browser, more

More info about this document

© All Rights Reserved

Go to text version

  • Total Views 3866
    • 3853 on SlideShare
    • 13 from embeds
  • Comments 41
  • Favorites 4
  • Downloads 1
Most viewed embeds
  • 13 views on http://www.perl.it

more

All embeds
  • 13 views on http://www.perl.it

less

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate. If needed, use the feedback form to let us know more details.

Cancel
File a copyright complaint
Having problems? Go to our helpdesk?

Categories

Tags

Groups / Events

-->
Search
RSS Feed
What's new?

© 2009 SlideShare Inc. All Rights Reserved