SlideShare a Scribd company logo
Evolution of
large-scale
websites
WHAT IS
LARGE-SCALE
WEBSITE?
3 billions PVGoogle
24 x 7 - 100%Usability
1 billionimagesFacebook
They all
begin as ...
SYSTEM ARCHITECTURE
SPLIT APPLICATION AND DATABASE
“For many events, roughly 80% of
the effects come from 20% of the
causes.
▸Vilfredo Pareto, Italian economist
CACHE
APPLICATION CLUSTER AND CACHE
DATABASE READ AND WRITE SEPARATION
Place your screenshot here
But,
it is not
done yet ...
CDN and REVERSE PROXY
THERE ARE MORE
NoSQL and Search
Engine
Techniques to speed
up data storage and
data search
Business split
For examples,
facebook’s home
page and message
page are developed
and maintained by
different teams
Database split
Based on business
transaction.
Single table
horizontal split.
Single table vertical
split.
WHAT I HAVE LEARNT
▸Evolution
▸Business achieves technology
▸Core value of the system architecture
THANKS!

More Related Content

Similar to Evolution of large scale websites

The re architecting the firm - sharing session guspri
The re architecting the firm - sharing session guspriThe re architecting the firm - sharing session guspri
The re architecting the firm - sharing session guspri
Agus Prianto
 
SeattleUniv-IntroductionToCloudComputing-WinsowsAzure101
SeattleUniv-IntroductionToCloudComputing-WinsowsAzure101SeattleUniv-IntroductionToCloudComputing-WinsowsAzure101
SeattleUniv-IntroductionToCloudComputing-WinsowsAzure101
Mithun T. Dhar
 
Primend Pilvekonverents - Azure Infrastruktuur
Primend Pilvekonverents - Azure InfrastruktuurPrimend Pilvekonverents - Azure Infrastruktuur
Primend Pilvekonverents - Azure Infrastruktuur
Primend
 
#DataOnCloud New York Event
#DataOnCloud New York Event#DataOnCloud New York Event
#DataOnCloud New York Event
HARMAN Services
 
Cloud Computing Impact On Small Business
Cloud Computing Impact On Small BusinessCloud Computing Impact On Small Business
Cloud Computing Impact On Small Business
David Linthicum
 
Big Data Presentation - Data Center Dynamics Sydney 2014 - Dez Blanchfield
Big Data Presentation - Data Center Dynamics Sydney 2014 - Dez BlanchfieldBig Data Presentation - Data Center Dynamics Sydney 2014 - Dez Blanchfield
Big Data Presentation - Data Center Dynamics Sydney 2014 - Dez Blanchfield
Dez Blanchfield
 
Analyst View of Data Virtualization: Conversations with Boulder Business Inte...
Analyst View of Data Virtualization: Conversations with Boulder Business Inte...Analyst View of Data Virtualization: Conversations with Boulder Business Inte...
Analyst View of Data Virtualization: Conversations with Boulder Business Inte...
Denodo
 
Juan Jose Juarez. Simplivity. Cómo optimizar su infraestructura virtual. Sema...
Juan Jose Juarez. Simplivity. Cómo optimizar su infraestructura virtual. Sema...Juan Jose Juarez. Simplivity. Cómo optimizar su infraestructura virtual. Sema...
Juan Jose Juarez. Simplivity. Cómo optimizar su infraestructura virtual. Sema...
COIICV
 
OpenSistemas Corporate Presentation
OpenSistemas Corporate PresentationOpenSistemas Corporate Presentation
OpenSistemas Corporate Presentation
OpenSistemas
 
A Study on the Application of Web-Scale IT in Enterprises in IoT Era
A Study on the Application of Web-Scale IT in Enterprises in IoT EraA Study on the Application of Web-Scale IT in Enterprises in IoT Era
A Study on the Application of Web-Scale IT in Enterprises in IoT Era
Hassan Keshavarz
 
IBM Cloud pak for data brochure
IBM Cloud pak for data   brochureIBM Cloud pak for data   brochure
IBM Cloud pak for data brochure
Simon Harrison ACMA CGMA
 
Cloud Data Integration Best Practices
Cloud Data Integration Best PracticesCloud Data Integration Best Practices
Cloud Data Integration Best Practices
Darren Cunningham
 
Rearchitecting for Innovation.pdf
Rearchitecting for Innovation.pdfRearchitecting for Innovation.pdf
Rearchitecting for Innovation.pdf
Amazon Web Services
 
Jet Reports es la herramienta para construir el mejor BI y de forma mas rapida
Jet Reports es la herramienta para construir el mejor BI y de forma mas rapida  Jet Reports es la herramienta para construir el mejor BI y de forma mas rapida
Jet Reports es la herramienta para construir el mejor BI y de forma mas rapida
CLARA CAMPROVIN
 
Mdm
MdmMdm
Media And Entertainment Whitepaper 090308
Media And Entertainment Whitepaper 090308Media And Entertainment Whitepaper 090308
Media And Entertainment Whitepaper 090308
sydcarr
 
¿Cómo modernizar una arquitectura de TI con la virtualización de datos?
¿Cómo modernizar una arquitectura de TI con la virtualización de datos?¿Cómo modernizar una arquitectura de TI con la virtualización de datos?
¿Cómo modernizar una arquitectura de TI con la virtualización de datos?
Denodo
 
DataAquitaine February 2022
DataAquitaine February 2022DataAquitaine February 2022
DataAquitaine February 2022
Yves Caseau
 
Dix conseils pour supporter la croissance de votre Startup de 0 à 10 millions...
Dix conseils pour supporter la croissance de votre Startup de 0 à 10 millions...Dix conseils pour supporter la croissance de votre Startup de 0 à 10 millions...
Dix conseils pour supporter la croissance de votre Startup de 0 à 10 millions...
Amazon Web Services
 
Roman Pavlyuk, Yaroslav Ravlinko, Intellias. Enterprise IT Transformation and...
Roman Pavlyuk, Yaroslav Ravlinko, Intellias. Enterprise IT Transformation and...Roman Pavlyuk, Yaroslav Ravlinko, Intellias. Enterprise IT Transformation and...
Roman Pavlyuk, Yaroslav Ravlinko, Intellias. Enterprise IT Transformation and...
IT Arena
 

Similar to Evolution of large scale websites (20)

The re architecting the firm - sharing session guspri
The re architecting the firm - sharing session guspriThe re architecting the firm - sharing session guspri
The re architecting the firm - sharing session guspri
 
SeattleUniv-IntroductionToCloudComputing-WinsowsAzure101
SeattleUniv-IntroductionToCloudComputing-WinsowsAzure101SeattleUniv-IntroductionToCloudComputing-WinsowsAzure101
SeattleUniv-IntroductionToCloudComputing-WinsowsAzure101
 
Primend Pilvekonverents - Azure Infrastruktuur
Primend Pilvekonverents - Azure InfrastruktuurPrimend Pilvekonverents - Azure Infrastruktuur
Primend Pilvekonverents - Azure Infrastruktuur
 
#DataOnCloud New York Event
#DataOnCloud New York Event#DataOnCloud New York Event
#DataOnCloud New York Event
 
Cloud Computing Impact On Small Business
Cloud Computing Impact On Small BusinessCloud Computing Impact On Small Business
Cloud Computing Impact On Small Business
 
Big Data Presentation - Data Center Dynamics Sydney 2014 - Dez Blanchfield
Big Data Presentation - Data Center Dynamics Sydney 2014 - Dez BlanchfieldBig Data Presentation - Data Center Dynamics Sydney 2014 - Dez Blanchfield
Big Data Presentation - Data Center Dynamics Sydney 2014 - Dez Blanchfield
 
Analyst View of Data Virtualization: Conversations with Boulder Business Inte...
Analyst View of Data Virtualization: Conversations with Boulder Business Inte...Analyst View of Data Virtualization: Conversations with Boulder Business Inte...
Analyst View of Data Virtualization: Conversations with Boulder Business Inte...
 
Juan Jose Juarez. Simplivity. Cómo optimizar su infraestructura virtual. Sema...
Juan Jose Juarez. Simplivity. Cómo optimizar su infraestructura virtual. Sema...Juan Jose Juarez. Simplivity. Cómo optimizar su infraestructura virtual. Sema...
Juan Jose Juarez. Simplivity. Cómo optimizar su infraestructura virtual. Sema...
 
OpenSistemas Corporate Presentation
OpenSistemas Corporate PresentationOpenSistemas Corporate Presentation
OpenSistemas Corporate Presentation
 
A Study on the Application of Web-Scale IT in Enterprises in IoT Era
A Study on the Application of Web-Scale IT in Enterprises in IoT EraA Study on the Application of Web-Scale IT in Enterprises in IoT Era
A Study on the Application of Web-Scale IT in Enterprises in IoT Era
 
IBM Cloud pak for data brochure
IBM Cloud pak for data   brochureIBM Cloud pak for data   brochure
IBM Cloud pak for data brochure
 
Cloud Data Integration Best Practices
Cloud Data Integration Best PracticesCloud Data Integration Best Practices
Cloud Data Integration Best Practices
 
Rearchitecting for Innovation.pdf
Rearchitecting for Innovation.pdfRearchitecting for Innovation.pdf
Rearchitecting for Innovation.pdf
 
Jet Reports es la herramienta para construir el mejor BI y de forma mas rapida
Jet Reports es la herramienta para construir el mejor BI y de forma mas rapida  Jet Reports es la herramienta para construir el mejor BI y de forma mas rapida
Jet Reports es la herramienta para construir el mejor BI y de forma mas rapida
 
Mdm
MdmMdm
Mdm
 
Media And Entertainment Whitepaper 090308
Media And Entertainment Whitepaper 090308Media And Entertainment Whitepaper 090308
Media And Entertainment Whitepaper 090308
 
¿Cómo modernizar una arquitectura de TI con la virtualización de datos?
¿Cómo modernizar una arquitectura de TI con la virtualización de datos?¿Cómo modernizar una arquitectura de TI con la virtualización de datos?
¿Cómo modernizar una arquitectura de TI con la virtualización de datos?
 
DataAquitaine February 2022
DataAquitaine February 2022DataAquitaine February 2022
DataAquitaine February 2022
 
Dix conseils pour supporter la croissance de votre Startup de 0 à 10 millions...
Dix conseils pour supporter la croissance de votre Startup de 0 à 10 millions...Dix conseils pour supporter la croissance de votre Startup de 0 à 10 millions...
Dix conseils pour supporter la croissance de votre Startup de 0 à 10 millions...
 
Roman Pavlyuk, Yaroslav Ravlinko, Intellias. Enterprise IT Transformation and...
Roman Pavlyuk, Yaroslav Ravlinko, Intellias. Enterprise IT Transformation and...Roman Pavlyuk, Yaroslav Ravlinko, Intellias. Enterprise IT Transformation and...
Roman Pavlyuk, Yaroslav Ravlinko, Intellias. Enterprise IT Transformation and...
 

Recently uploaded

Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Safe Software
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Malak Abu Hammad
 
Video Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the FutureVideo Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the Future
Alpen-Adria-Universität
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
Tomaz Bratanic
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
Octavian Nadolu
 
Infrastructure Challenges in Scaling RAG with Custom AI models
Infrastructure Challenges in Scaling RAG with Custom AI modelsInfrastructure Challenges in Scaling RAG with Custom AI models
Infrastructure Challenges in Scaling RAG with Custom AI models
Zilliz
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems S.M.S.A.
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Speck&Tech
 
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
Edge AI and Vision Alliance
 
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and MilvusBuilding Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
Zilliz
 
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practicesNational Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
Quotidiano Piemontese
 
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERPBest 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
Pixlogix Infotech
 
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
Neo4j
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
Zilliz
 
Pushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 daysPushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 days
Adtran
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
DianaGray10
 
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
DianaGray10
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
shyamraj55
 
How to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For FlutterHow to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For Flutter
Daiki Mogmet Ito
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
Safe Software
 

Recently uploaded (20)

Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
 
Video Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the FutureVideo Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the Future
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
 
Infrastructure Challenges in Scaling RAG with Custom AI models
Infrastructure Challenges in Scaling RAG with Custom AI modelsInfrastructure Challenges in Scaling RAG with Custom AI models
Infrastructure Challenges in Scaling RAG with Custom AI models
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
 
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
 
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and MilvusBuilding Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
 
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practicesNational Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
 
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERPBest 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
 
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
 
Pushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 daysPushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 days
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
 
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
 
How to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For FlutterHow to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For Flutter
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
 

Evolution of large scale websites

Editor's Notes

  1. I thought it’d be good to do a talk about large scale websites/applications What do they do when developing from small to large
  2. Instinct is website like Google, Facebook, Twitter. Convert concept to data ...
  3. High Concurrency, high volume. Google PageViews is 3.5 billions per day Massive data High usability, 24 X 7 - Poor security environment, almost everyday it’s attacked
  4. But they all begin as a small project, in other word, small website. Facebook began in Harvard’s accommodation Google was first deployed in Stanford University So, imagine that maybe 10 years ago, Steve and Bill wanted to start an e-commerce website
  5. At the beginning Application, database, file are all in one server, simple Steve and Bill started to promote the website, asked the users the right question to improve user experiences, marketing They have more and more users. Soon they realised that they ran out of storage and website became slow
  6. Application server handles all the business logic so it needs powerful CPU Database server stores and searches data, so it needs fast hard disk and big RAM File server stores all documents, so it needs bigger hard disk So the website is technically ready for further development, in which Steve and Bill did The website now needs further optimisation
  7. Apply to the e-commerce, it means that 80% of the transactions are related to 20% of all products. Customers prefer to purchase few good review products Google, Twitter searches are focusing on a few hot hashtags
  8. Based on 80/20 rule, Steve and Bill had the idea to cache 20% data to memory Local cache is faster but limited In theory, remote distributed cache servers are unlimited Peak time, single application server is the bottleneck
  9. Use cluster, increase the number of application servers Apply load balance server, distributed requests evenly Application load will no longer be the bottleneck But … How about the database
  10. Most of the website activities are READ SPLIT database Write to the master database then replicate the slaves database Read from slaves database Perfect! System should be fast enough!
  11. As Steve and Bill learn from research, website delay and user churn rate have positive correlation To keep the users, they decided to add more JS, pictures to give better visualisation
  12. In fact, they are all caches CDN cache data in the closest ISP data centre to user Reverse proxy caches data in company’s data centre The purposes are to return data to user as quickly as possible, and to reduce backend servers’ load
  13. No website/application was borned as large-scale “This is how Google/Facebook did it”, may not work for your own company. Core value of the system architecture is not to build a large scale website right at the beginning. It’s to develop with the business.