Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building scalable and reliable websites


Published on

Published in: Technology
  • Be the first to comment

Building scalable and reliable websites

  1. 1. Building scalable and reliable websites Studencki Festiwal Informatyczny Kraków, 12-14.03.2009 Tomasz Napierała
  2. 2. Tomasz Napierała Systems Architecture Engineer JID/email:
  3. 3. feed chain Naspers/MIH Tradus GaduGadu Tencent QQ QXL Ricardo QXL Poland
  4. 4. we are the borg 800 employees 650 in poland 125 in IT 13 services 10 on platform
  5. 5. . cz . ro . ua . bg . sk auctions ads payments shops inventory
  6. 6. meat the team
  7. 7. technical dept Application Unit 1 Application Unit 2 R&D BI & DWH PMO P&L Infrastructure Infrastructure: Systems Networks DBA Help Desk NOC
  8. 8. history Źródło: Oracle DC2
  9. 9. ranks Site in Alexa: - 134th/206th worldwide - 4th in Poland Auctions: - 7th worldwide (counting eBay divisions) - biggest in central/eastern Europe
  10. 10. childhood
  11. 11. some numbers <ul><li>>500 servers </li></ul><ul><li>350TB of storage </li></ul><ul><li>270 milions images (last 2 months) </li></ul><ul><li>6.5TB of images </li></ul>
  12. 12. more numbers <ul><li>show_item: ca 30%, roughly 100.000 per minute </li></ul><ul><li>6 milions email notifications per day </li></ul><ul><li>100.000 http requests per sec </li></ul><ul><li>20.000 new http requests per sec </li></ul>
  13. 13. and even more <ul><li>5.5TB data in DB </li></ul><ul><li>6GB daily growth </li></ul><ul><li>2k – 10k queries/s, 200 milions queries per day </li></ul><ul><li>1000 tables in database </li></ul>
  14. 14. service lifecycle Internet www DB www DB Internet Internet ? Internet Internet www DB www DB cache
  15. 15. <ul><li>room for growth </li></ul><ul><li>easy maintenance </li></ul><ul><li>easy development </li></ul><ul><li>availability </li></ul><ul><li>linear scalability </li></ul><ul><li>simple </li></ul><ul><li>best value for money </li></ul><ul><li>redundancy </li></ul>problem
  16. 16. recipes do you have any? we've got some...
  17. 17. cache everything browser client proxy reverse proxy memcached eaccelerator/xcache db cache fs cache
  18. 18. frontend reduce DNS response optimize images use ETags flush buffer get rid of cookies minimize requests control cache gzip JS at the bottom CSS at the top
  19. 19. frontend – the award let's see, how much we can really earn
  20. 20. frontend – the award empty cache vs primed cache 93% gain empty cache primed cache size filetype size filetype 6.5K 1 html/text 6.5K 1 html/text 7.3K 4 javascript 0.0K 1 css 8.3K 12 css images 71.4K 19 images 93.6K 37 6.5K 1
  21. 21. frontend – the award sweets get without cookie: 360B get with cookie: 1003B 75% gain
  22. 22. frontend – numbers
  23. 23. load balancing … or buy LACP/bonding LVS/IPVS HAProxy perlbal varnish/nginx ...
  24. 24. distribute load Internet cache LB originals LB Internet cache LB originals
  25. 25. storage <ul><li>security </li></ul><ul><li>block size </li></ul><ul><li>data structure </li></ul><ul><li>hardware </li></ul><ul><li>filesystem size </li></ul><ul><li>copies </li></ul><ul><li>backups </li></ul><ul><li>I/O constraints </li></ul>
  26. 26. storage <ul><li>MogileFS </li></ul><ul><li>get your own </li></ul><ul><li>Lustre </li></ul><ul><li>Hadoop </li></ul><ul><li>Amazon S3 </li></ul>
  27. 27. database tune replicate, replicate more sharding
  28. 28. sharding + denormalization scale out availability small dataset - growing can be pain expensive joins plannig nightmare still replicate SPOF lookup table
  29. 29. eee, managebility easy to manage easy to deploy extensible easy to troubleshoot
  30. 30. mines sometimes better is worse
  31. 31. when time is an issue
  32. 32. sysinternals <ul><li>98% Linux shop </li></ul><ul><li>CentOS / Ubuntu / OpenSolaris / AIX </li></ul><ul><li>HP / IBM / 3PAR / Onstor / Isilon / Brocade </li></ul><ul><li>apache, lighttpd, squid, varnish, memcached, sphinx </li></ul><ul><li>Oracle, MySQL, PostgreSQL </li></ul>
  33. 33. sysinternals – storage <ul><li>Isilon: </li></ul><ul><li>low impact node failure </li></ul><ul><li>infiniband intraconnect </li></ul><ul><li>R: 480MB/s, W: 250MB/s </li></ul>OnStor: 32k IO/s on-line reconfiguration online fix/resize 3PAR: all online remote copy thin provisioning unique architecture
  34. 34. netinternals <ul><li>CISCO </li></ul><ul><li>Ironport </li></ul><ul><li>F5 – load balancer and much more </li></ul><ul><li>Juniper </li></ul>
  35. 35. monitor and manage <ul><li>svn </li></ul><ul><li>monarch </li></ul><ul><li>sauron </li></ul><ul><li>altiris </li></ul><ul><li>request tracker </li></ul><ul><li>otrs </li></ul><ul><li>jira </li></ul>nagios gomez cacti rancid collectd cflowd SolarWinds
  36. 36. application <ul><li>it's the code that doesn't scale, not the language </li></ul><ul><li>PHP, C, C++, Java </li></ul><ul><li>redefinig to more SOA </li></ul><ul><li>specialized daemons </li></ul>
  37. 37. application architecture Internet static www images LB cache originals Storage DB cache/pool cache originals upload
  38. 38. final thougths <ul><li>KISS </li></ul><ul><li>research </li></ul><ul><li>long term perspective </li></ul><ul><li>scale horizontally </li></ul><ul><li>be ready for growth </li></ul><ul><li>focus on important </li></ul><ul><li>check your graphs </li></ul><ul><li>listen to wiser </li></ul>while (true) { identify_and_fix_bottlenecks(); drink(); sleep(); notice_new_bottlenecks(); }
  39. 39. thank you
  40. 40. questions?