Your SlideShare is downloading. ×
0
Building a Website To Scale    Target: 200 Million page views per day and beyond!                                         ...
Contents           1. The Context           2. The Requirements           3. The Architecture           4. The Good and th...
What are we talking about?  Aug         Apr        Dec       Feb                Apr  2006       2007        2007      2008...
Traffic In Perspective                                   Source: Alexa.com                  Alexa global rank 95  100 Gb/s...
The Context Written in PERL with a very complex architecture First few months dedicated to learning the site,  maintain ...
The requirements     1   Support 200 million+ daily requests     2   100% transparent to users     3   Six years of legacy...
The Architecture                   www.ManwinJobs.com
The Architecture                   www.ManwinJobs.com
The Architecture Fast and reliable load-balancing. Intelligent load distribution. Performs health-checks               ...
The Architecture                   www.ManwinJobs.com
The Architecture Reverse proxy optimized for better speed Reduces web and database server load Very rich and flexible c...
The Architecture Cache management (what, for how long) Edge Side Includes (ESIs) Health check on Web servers           ...
The Architecture                   www.ManwinJobs.com
The Architecture Custom logging of page views Used for tasks like view counters or related videos Between 8GB and 15GB ...
The Architecture                   www.ManwinJobs.com
The Architecture High-performance HTTP server. PHP-FPM External CDNs for Static files like CSS, images and JS          ...
The Architecture                   www.ManwinJobs.com
The Architecture FPM hosts our framework of choice: Symfony2. Fast and feature rich. A wealth of bundles already availa...
The Architecture                   www.ManwinJobs.com
The Architecture A messaging component Designed for large scale deployments ActiveMQ to do writes (MySQL and Redis)    ...
The Architecture Partially implemented with mitigated results. Too rigid for a site requiring constant changes. Gains n...
The Architecture                   www.ManwinJobs.com
The Architecture Ability to manage pools of servers with health checks. We maintain 2 pools:    Write pool with fail-ov...
The Architecture                   www.ManwinJobs.com
The Architecture Open source, advanced key-value store Read operations on Redis are FAST Primary data source           ...
The Architecture Updated in real time as with MySQL. Redis Sorted Sets for all lists. Pipelining is VERY important for ...
The Architecture Persistence needs tuning. RDB does a snapshot but is very IO extensive. AOF does incremental backups a...
The Architecture                   www.ManwinJobs.com
The Architecture Very normalized database since not used directly for site. Some tables have over 100 million rows. Use...
The good and the bad Main reasons for the delays:    Decisions concerning some of the technologies to use.    Learning ...
The good and the bad Was it a success?    Launch without any downtime    New site about 10% faster    Valuable experti...
Eric Pickup      eric.pickup@manwin.com Twitter: EricPickupYPwww.ManwinJobs.com
Upcoming SlideShare
Loading in...5
×

Building a Website to Scale to 100 Million Page Views Per Day and Beyond

4,256

Published on

Published in: Technology

Transcript of "Building a Website to Scale to 100 Million Page Views Per Day and Beyond "

  1. 1. Building a Website To Scale Target: 200 Million page views per day and beyond! By Eric Pickup eric.pickup@manwin.com Twitter: EricPickupYPwww.ManwinJobs.com
  2. 2. Contents 1. The Context 2. The Requirements 3. The Architecture 4. The Good and the Bad www.ManwinJobs.com
  3. 3. What are we talking about? Aug Apr Dec Feb Apr 2006 2007 2007 2008 2011 YP First 1 Million 100,000 100 million Acquired Launched daily Uploads daily by Manwin visitors page views www.ManwinJobs.com
  4. 4. Traffic In Perspective Source: Alexa.com Alexa global rank 95 100 Gb/s – 3 full DVDs streamed every single second www.ManwinJobs.com
  5. 5. The Context Written in PERL with a very complex architecture First few months dedicated to learning the site, maintain it, and plan the re-write. Re-write started in August 2011 and was originally planned for a delivery in mid-November. Actually launched at the end of January. www.ManwinJobs.com
  6. 6. The requirements 1 Support 200 million+ daily requests 2 100% transparent to users 3 Six years of legacy data 4 Even faster site www.ManwinJobs.com
  7. 7. The Architecture www.ManwinJobs.com
  8. 8. The Architecture www.ManwinJobs.com
  9. 9. The Architecture Fast and reliable load-balancing. Intelligent load distribution. Performs health-checks www.ManwinJobs.com
  10. 10. The Architecture www.ManwinJobs.com
  11. 11. The Architecture Reverse proxy optimized for better speed Reduces web and database server load Very rich and flexible configuration www.ManwinJobs.com
  12. 12. The Architecture Cache management (what, for how long) Edge Side Includes (ESIs) Health check on Web servers www.ManwinJobs.com
  13. 13. The Architecture www.ManwinJobs.com
  14. 14. The Architecture Custom logging of page views Used for tasks like view counters or related videos Between 8GB and 15GB of logs per hour! www.ManwinJobs.com
  15. 15. The Architecture www.ManwinJobs.com
  16. 16. The Architecture High-performance HTTP server. PHP-FPM External CDNs for Static files like CSS, images and JS www.ManwinJobs.com
  17. 17. The Architecture www.ManwinJobs.com
  18. 18. The Architecture FPM hosts our framework of choice: Symfony2. Fast and feature rich. A wealth of bundles already available. www.ManwinJobs.com
  19. 19. The Architecture www.ManwinJobs.com
  20. 20. The Architecture A messaging component Designed for large scale deployments ActiveMQ to do writes (MySQL and Redis) www.ManwinJobs.com
  21. 21. The Architecture Partially implemented with mitigated results. Too rigid for a site requiring constant changes. Gains not justifying Java and a separate infrastructure. www.ManwinJobs.com
  22. 22. The Architecture www.ManwinJobs.com
  23. 23. The Architecture Ability to manage pools of servers with health checks. We maintain 2 pools:  Write pool with fail-over to backup-Master.  Read pool with all servers except Master. www.ManwinJobs.com
  24. 24. The Architecture www.ManwinJobs.com
  25. 25. The Architecture Open source, advanced key-value store Read operations on Redis are FAST Primary data source www.ManwinJobs.com
  26. 26. The Architecture Updated in real time as with MySQL. Redis Sorted Sets for all lists. Pipelining is VERY important for performance. www.ManwinJobs.com
  27. 27. The Architecture Persistence needs tuning. RDB does a snapshot but is very IO extensive. AOF does incremental backups and is IO pain-free. www.ManwinJobs.com
  28. 28. The Architecture www.ManwinJobs.com
  29. 29. The Architecture Very normalized database since not used directly for site. Some tables have over 100 million rows. Used to populate Redis lists for new features www.ManwinJobs.com
  30. 30. The good and the bad Main reasons for the delays:  Decisions concerning some of the technologies to use.  Learning curve for new technologies longer than expected.  Data transfer and restructuring in MySQL and Redis  Staffing issues. www.ManwinJobs.com
  31. 31. The good and the bad Was it a success?  Launch without any downtime  New site about 10% faster  Valuable expertise gained A GOOD SUCCESS STORY WITH LESSONS LEARNED www.ManwinJobs.com
  32. 32. Eric Pickup eric.pickup@manwin.com Twitter: EricPickupYPwww.ManwinJobs.com
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×