Building scalable and reliable websites

2,769 views

Published on

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,769
On SlideShare
0
From Embeds
0
Number of Embeds
328
Actions
Shares
0
Downloads
0
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide
  • - rozdać kartki publiczności
  • Building scalable and reliable websites

    1. 1. Building scalable and reliable websites Studencki Festiwal Informatyczny Kraków, 12-14.03.2009 Tomasz Napierała
    2. 2. Tomasz Napierała Systems Architecture Engineer JID/email: tomasz.mapierala@allegro.pl
    3. 3. feed chain Naspers/MIH Tradus GaduGadu mail.ru Tencent QQ QXL Ricardo Allegro.pl QXL Poland
    4. 4. we are the borg 800 employees 650 in poland 125 in IT 13 services 10 on platform
    5. 5. . cz . ro . ua . bg . sk auctions ads payments shops inventory
    6. 6. meat the team
    7. 7. technical dept Application Unit 1 Application Unit 2 R&D BI & DWH PMO P&L Infrastructure Infrastructure: Systems Networks DBA Help Desk NOC
    8. 8. history Źródło: aukcjostat.pl Oracle DC2
    9. 9. ranks Site in Alexa: - 134th/206th worldwide - 4th in Poland Auctions: - 7th worldwide (counting eBay divisions) - biggest in central/eastern Europe
    10. 10. childhood
    11. 11. some numbers <ul><li>>500 servers </li></ul><ul><li>350TB of storage </li></ul><ul><li>270 milions images (last 2 months) </li></ul><ul><li>6.5TB of images </li></ul>
    12. 12. more numbers <ul><li>show_item: ca 30%, roughly 100.000 per minute </li></ul><ul><li>6 milions email notifications per day </li></ul><ul><li>100.000 http requests per sec </li></ul><ul><li>20.000 new http requests per sec </li></ul>
    13. 13. and even more <ul><li>5.5TB data in DB </li></ul><ul><li>6GB daily growth </li></ul><ul><li>2k – 10k queries/s, 200 milions queries per day </li></ul><ul><li>1000 tables in database </li></ul>
    14. 14. service lifecycle Internet www DB www DB Internet Internet ? Internet Internet www DB www DB cache
    15. 15. <ul><li>room for growth </li></ul><ul><li>easy maintenance </li></ul><ul><li>easy development </li></ul><ul><li>availability </li></ul><ul><li>linear scalability </li></ul><ul><li>simple </li></ul><ul><li>best value for money </li></ul><ul><li>redundancy </li></ul>problem
    16. 16. recipes do you have any? we've got some...
    17. 17. cache everything browser client proxy reverse proxy memcached eaccelerator/xcache db cache fs cache
    18. 18. frontend reduce DNS response optimize images use ETags flush buffer get rid of cookies minimize requests control cache gzip JS at the bottom CSS at the top
    19. 19. frontend – the award let's see, how much we can really earn
    20. 20. frontend – the award empty cache vs primed cache 93% gain empty cache primed cache size filetype size filetype 6.5K 1 html/text 6.5K 1 html/text 7.3K 4 javascript 0.0K 1 css 8.3K 12 css images 71.4K 19 images 93.6K 37 6.5K 1
    21. 21. frontend – the award sweets get without cookie: 360B get with cookie: 1003B 75% gain
    22. 22. frontend – numbers
    23. 23. load balancing … or buy LACP/bonding LVS/IPVS HAProxy perlbal varnish/nginx ...
    24. 24. distribute load Internet cache LB originals LB Internet cache LB originals
    25. 25. storage <ul><li>security </li></ul><ul><li>block size </li></ul><ul><li>data structure </li></ul><ul><li>hardware </li></ul><ul><li>filesystem size </li></ul><ul><li>copies </li></ul><ul><li>backups </li></ul><ul><li>I/O constraints </li></ul>
    26. 26. storage <ul><li>MogileFS </li></ul><ul><li>get your own </li></ul><ul><li>Lustre </li></ul><ul><li>Hadoop </li></ul><ul><li>Amazon S3 </li></ul>
    27. 27. database tune replicate, replicate more sharding
    28. 28. sharding + denormalization scale out availability small dataset - growing can be pain expensive joins plannig nightmare still replicate SPOF lookup table
    29. 29. eee, managebility easy to manage easy to deploy extensible easy to troubleshoot
    30. 30. mines sometimes better is worse
    31. 31. when time is an issue
    32. 32. sysinternals <ul><li>98% Linux shop </li></ul><ul><li>CentOS / Ubuntu / OpenSolaris / AIX </li></ul><ul><li>HP / IBM / 3PAR / Onstor / Isilon / Brocade </li></ul><ul><li>apache, lighttpd, squid, varnish, memcached, sphinx </li></ul><ul><li>Oracle, MySQL, PostgreSQL </li></ul>
    33. 33. sysinternals – storage <ul><li>Isilon: </li></ul><ul><li>low impact node failure </li></ul><ul><li>infiniband intraconnect </li></ul><ul><li>R: 480MB/s, W: 250MB/s </li></ul>OnStor: 32k IO/s on-line reconfiguration online fix/resize 3PAR: all online remote copy thin provisioning unique architecture
    34. 34. netinternals <ul><li>CISCO </li></ul><ul><li>Ironport </li></ul><ul><li>F5 – load balancer and much more </li></ul><ul><li>Juniper </li></ul>
    35. 35. monitor and manage <ul><li>svn </li></ul><ul><li>monarch </li></ul><ul><li>sauron </li></ul><ul><li>altiris </li></ul><ul><li>request tracker </li></ul><ul><li>otrs </li></ul><ul><li>jira </li></ul>nagios gomez cacti rancid collectd cflowd SolarWinds
    36. 36. application <ul><li>it's the code that doesn't scale, not the language </li></ul><ul><li>PHP, C, C++, Java </li></ul><ul><li>redefinig to more SOA </li></ul><ul><li>specialized daemons </li></ul>
    37. 37. application architecture Internet static www images LB cache originals Storage DB cache/pool cache originals upload
    38. 38. final thougths <ul><li>KISS </li></ul><ul><li>research </li></ul><ul><li>long term perspective </li></ul><ul><li>scale horizontally </li></ul><ul><li>be ready for growth </li></ul><ul><li>focus on important </li></ul><ul><li>check your graphs </li></ul><ul><li>listen to wiser </li></ul>while (true) { identify_and_fix_bottlenecks(); drink(); sleep(); notice_new_bottlenecks(); }
    39. 39. thank you
    40. 40. questions?

    ×