SlideShare a Scribd company logo
1 of 48
Database Scalability
                 {Patterns}

                           / Robert Treat



Wednesday, July 21, 2010
robert treat
                             omniti




Wednesday, July 21, 2010
robert treat
                              omniti
                             postgres




Wednesday, July 21, 2010
robert treat
                               omniti
                              postgres
                           oracle - mysql




Wednesday, July 21, 2010
robert treat
                                  omniti
                                 postgres
                             oracle - mysql
                           mssql - sqlite - nosql




Wednesday, July 21, 2010
What are “Database Scalability Patterns?”




Wednesday, July 21, 2010
Part Design Patterns



                                 Part Application Life-Cycle




Wednesday, July 21, 2010
Part Design Patterns



                                 Part Application Life-Cycle




Wednesday, July 21, 2010
Part Design Patterns



                                 Part Application Life-Cycle




Wednesday, July 21, 2010
Part Design Patterns



                                 Part Application Life-Cycle




Wednesday, July 21, 2010
MyFirstDatabase




Wednesday, July 21, 2010
Vertical Partitioning




Wednesday, July 21, 2010
Vertical Scaling




Wednesday, July 21, 2010
Vertical Scaling




Wednesday, July 21, 2010
Vertical Scaling




Wednesday, July 21, 2010
Federated Data Storage


              “sharding”                 Horizontal Partitioning

                           Read Slaves

                                                 Multi-Master

                 Horizontal Scaling



Wednesday, July 21, 2010
Read Slaves / Master - Slave
                                Scale Read Load




Wednesday, July 21, 2010
Read Slaves / Master - Slave
                                Scale Read Load


                                                      writes go here!




Wednesday, July 21, 2010
Read Slaves / Master - Slave
                                   Scale Read Load


                                                         writes go here!




                           reads go here (or here) (or here)
Wednesday, July 21, 2010
Read Slaves / Master - Slave
                                      Scale Read Load

          database writes
           data to slaves                                writes go here!



                           slave db      slave db   slave db


                           reads go here (or here) (or here)
Wednesday, July 21, 2010
Read Slaves / Master - Slave
                                     Scale Read Load

          app writes data
            everywhere                                     writes go here!



                           memcached memcached memcached


                             reads go here (or here) (or here)
Wednesday, July 21, 2010
Read Slaves / Master - Slave
                                Scale Read Load

       • Typically
         • Full Copy of Data
            On Each Node
            • Asynchronous




Wednesday, July 21, 2010
Read Slaves / Master - Slave
                                Scale Read Load

       • Typically                         • Consider
         • Full Copy of Data                 • Partial Copy
            On Each Node                     • Synchronous
            • Asynchronous                   • Don’t use a RDBMS?




Wednesday, July 21, 2010
Read Slaves / Master - Slave
                                 Scale Read Load

       • Typically                          • Consider
         • Full Copy of Data                  • Partial Copy
            On Each Node                      • Synchronous
            • Asynchronous                    • Don’t use a RDBMS?
                           Requires Application Changes



Wednesday, July 21, 2010
Read Slaves / Master - Slave
                                 Scale Read Load

       • Typically                          • Consider
         • Full Copy of Data                  • Partial Copy
            On Each Node                      • Synchronous
            • Asynchronous                    • Don’t use a RDBMS?
                           Requires Application Changes
                                      “easy”

Wednesday, July 21, 2010
Scaling Writes




Wednesday, July 21, 2010
Scaling Writes




                            “not easy”




Wednesday, July 21, 2010
Multi-Master

                           many different ways to implement this,
                           few that actually work in production




Wednesday, July 21, 2010
Multi-Master

                           many different ways to implement this,
                           few that actually work in production


                           write to any node, database syncs data




Wednesday, July 21, 2010
Multi-Master

                           many different ways to implement this,
                           few that actually work in production


                           write to any node, database syncs data
                             can reduce cpu, doesn’t reduce i/o




Wednesday, July 21, 2010
Multi-Master

                           many different ways to implement this,
                           few that actually work in production


                           write to any node, database syncs data
                             can reduce cpu, doesn’t reduce i/o


                                      failover solution
                                  not a scalability solution

Wednesday, July 21, 2010
Horizontal Partitioning


                           “sharding”                 “SOA”




Wednesday, July 21, 2010
Horizontal Partitioning


                           “sharding”                 “SOA”




Wednesday, July 21, 2010
Horizontal Partitioning

    • Divide schema by job operations
    • Move each piece to own server
    • Duplicate some data as needed




Wednesday, July 21, 2010
Horizontal Partitioning

                                                     items
    • Divide schema by job operations
    • Move each piece to own server
    • Duplicate some data as needed

                                                               forums



                                                       users



Wednesday, July 21, 2010
Horizontal Partitioning

                                                     items
    • Divide schema by job operations
    • Move each piece to own server
    • Duplicate some data as needed

                                                               forums
    • You must separate dependencies
    in the app code first!
                                                       users



Wednesday, July 21, 2010
Horizontal Partitioning

                                                        items
    • Divide schema by job operations
    • Move each piece to own server
    • Duplicate some data as needed

                                                                    forums
    • You must separate dependencies
    in the app code first!
                                                          users
                  Each node is a new instance of vertical scaling

Wednesday, July 21, 2010
Horizontal Scaling
             • data split across servers based on algorithm
             • data dropped into buckets (multiple?)
                                   app


   magic hash algorithm




Wednesday, July 21, 2010
Horizontal Scaling
             • data split across servers based on algorithm
             • data dropped into buckets (multiple?)


   magic hash algorithm            app




Wednesday, July 21, 2010
Horizontal Scaling
             • data split across servers based on algorithm
             • data dropped into buckets (multiple?)
             •someone must keep track of data, and provide
             lookup services

   magic hash algorithm              app




Wednesday, July 21, 2010
Universal Truths of Scaling Databases

                   Vertical Scalability is Helpful for Every Pattern


                   Even in a horizontally scaled, fully distributed
                   database, the number of nodes needed is
                   affected by vertical scalability




Wednesday, July 21, 2010
Universal Truths of Scaling Databases

                               New Nodes Are Never Free


                           • Add points of failure
                           • Add management costs
                           • Add complexity to architecture
                           • Add complexity to your app code


Wednesday, July 21, 2010
MyFirstDB
                                      V
                            Vertical Partitioning
                                      V
                              Vertical Scaling
                                      V
                                Read Slaves
                                      V
                           Horizontal Partitioning



Wednesday, July 21, 2010
tips




Wednesday, July 21, 2010
tips

                           plan for layered data sources




Wednesday, July 21, 2010
tips

                            plan for layered data sources

                           read / write connections in code




Wednesday, July 21, 2010
tips

                             plan for layered data sources

                           read / write connections in code

                           use schemas to separate services




Wednesday, July 21, 2010
THANKS!

                                  more:
                                 xzilla.net
                                @robtreat2



                              omniti.com/surge
                           (scalability conference)


Wednesday, July 21, 2010

More Related Content

Viewers also liked

Db performance optimization with indexing
Db performance optimization with indexingDb performance optimization with indexing
Db performance optimization with indexingRajeev Kumar
 
Database Architecture & Scaling Strategies, in the Cloud & on the Rack
Database Architecture & Scaling Strategies, in the Cloud & on the Rack Database Architecture & Scaling Strategies, in the Cloud & on the Rack
Database Architecture & Scaling Strategies, in the Cloud & on the Rack Clustrix
 
Database index
Database indexDatabase index
Database indexRiteshkiit
 
Data indexing presentation
Data indexing presentationData indexing presentation
Data indexing presentationgmbmanikandan
 
What Every Developer Should Know About Database Scalability
What Every Developer Should Know About Database ScalabilityWhat Every Developer Should Know About Database Scalability
What Every Developer Should Know About Database Scalabilityjbellis
 
Banco de Dados Não Relacionais vs Banco de Dados Relacionais
Banco de Dados Não Relacionais vs Banco de Dados RelacionaisBanco de Dados Não Relacionais vs Banco de Dados Relacionais
Banco de Dados Não Relacionais vs Banco de Dados Relacionaisalexculpado
 
Scaling Pinterest
Scaling PinterestScaling Pinterest
Scaling PinterestC4Media
 
Relational Database Design
Relational Database DesignRelational Database Design
Relational Database DesignArchit Saxena
 
Sql vs NoSQL
Sql vs NoSQLSql vs NoSQL
Sql vs NoSQLRTigger
 

Viewers also liked (9)

Db performance optimization with indexing
Db performance optimization with indexingDb performance optimization with indexing
Db performance optimization with indexing
 
Database Architecture & Scaling Strategies, in the Cloud & on the Rack
Database Architecture & Scaling Strategies, in the Cloud & on the Rack Database Architecture & Scaling Strategies, in the Cloud & on the Rack
Database Architecture & Scaling Strategies, in the Cloud & on the Rack
 
Database index
Database indexDatabase index
Database index
 
Data indexing presentation
Data indexing presentationData indexing presentation
Data indexing presentation
 
What Every Developer Should Know About Database Scalability
What Every Developer Should Know About Database ScalabilityWhat Every Developer Should Know About Database Scalability
What Every Developer Should Know About Database Scalability
 
Banco de Dados Não Relacionais vs Banco de Dados Relacionais
Banco de Dados Não Relacionais vs Banco de Dados RelacionaisBanco de Dados Não Relacionais vs Banco de Dados Relacionais
Banco de Dados Não Relacionais vs Banco de Dados Relacionais
 
Scaling Pinterest
Scaling PinterestScaling Pinterest
Scaling Pinterest
 
Relational Database Design
Relational Database DesignRelational Database Design
Relational Database Design
 
Sql vs NoSQL
Sql vs NoSQLSql vs NoSQL
Sql vs NoSQL
 

More from Robert Treat

Advanced Int->Bigint Conversions
Advanced Int->Bigint ConversionsAdvanced Int->Bigint Conversions
Advanced Int->Bigint ConversionsRobert Treat
 
Explaining Explain
Explaining ExplainExplaining Explain
Explaining ExplainRobert Treat
 
the-lost-art-of-plpgsql
the-lost-art-of-plpgsqlthe-lost-art-of-plpgsql
the-lost-art-of-plpgsqlRobert Treat
 
Managing Chaos In Production: Testing vs Monitoring
Managing Chaos In Production: Testing vs MonitoringManaging Chaos In Production: Testing vs Monitoring
Managing Chaos In Production: Testing vs MonitoringRobert Treat
 
Managing Databases In A DevOps Environment 2016
Managing Databases In A DevOps Environment 2016Managing Databases In A DevOps Environment 2016
Managing Databases In A DevOps Environment 2016Robert Treat
 
Less Alarming Alerts - SRECon 2016
Less Alarming Alerts - SRECon 2016 Less Alarming Alerts - SRECon 2016
Less Alarming Alerts - SRECon 2016 Robert Treat
 
What Ops Can Learn From Design
What Ops Can Learn From DesignWhat Ops Can Learn From Design
What Ops Can Learn From DesignRobert Treat
 
Postgres 9.4 First Look
Postgres 9.4 First LookPostgres 9.4 First Look
Postgres 9.4 First LookRobert Treat
 
Less Alarming Alerts!
Less Alarming Alerts!Less Alarming Alerts!
Less Alarming Alerts!Robert Treat
 
Past, Present, and Pachyderm - All Things Open - 2013
Past, Present, and Pachyderm - All Things Open - 2013Past, Present, and Pachyderm - All Things Open - 2013
Past, Present, and Pachyderm - All Things Open - 2013Robert Treat
 
Big Bad "Upgraded" Postgres
Big Bad "Upgraded" PostgresBig Bad "Upgraded" Postgres
Big Bad "Upgraded" PostgresRobert Treat
 
Managing Databases In A DevOps Environment
Managing Databases In A DevOps EnvironmentManaging Databases In A DevOps Environment
Managing Databases In A DevOps EnvironmentRobert Treat
 
The Essential PostgreSQL.conf
The Essential PostgreSQL.confThe Essential PostgreSQL.conf
The Essential PostgreSQL.confRobert Treat
 
Advanced WAL File Management With OmniPITR
Advanced WAL File Management With OmniPITRAdvanced WAL File Management With OmniPITR
Advanced WAL File Management With OmniPITRRobert Treat
 
Scaling with Postgres (Highload++ 2010)
Scaling with Postgres (Highload++ 2010)Scaling with Postgres (Highload++ 2010)
Scaling with Postgres (Highload++ 2010)Robert Treat
 
Intro to Postgres 9 Tutorial
Intro to Postgres 9 TutorialIntro to Postgres 9 Tutorial
Intro to Postgres 9 TutorialRobert Treat
 
Database Scalability Patterns
Database Scalability PatternsDatabase Scalability Patterns
Database Scalability PatternsRobert Treat
 
A Guide To PostgreSQL 9.0
A Guide To PostgreSQL 9.0A Guide To PostgreSQL 9.0
A Guide To PostgreSQL 9.0Robert Treat
 

More from Robert Treat (20)

Advanced Int->Bigint Conversions
Advanced Int->Bigint ConversionsAdvanced Int->Bigint Conversions
Advanced Int->Bigint Conversions
 
Explaining Explain
Explaining ExplainExplaining Explain
Explaining Explain
 
the-lost-art-of-plpgsql
the-lost-art-of-plpgsqlthe-lost-art-of-plpgsql
the-lost-art-of-plpgsql
 
Managing Chaos In Production: Testing vs Monitoring
Managing Chaos In Production: Testing vs MonitoringManaging Chaos In Production: Testing vs Monitoring
Managing Chaos In Production: Testing vs Monitoring
 
Managing Databases In A DevOps Environment 2016
Managing Databases In A DevOps Environment 2016Managing Databases In A DevOps Environment 2016
Managing Databases In A DevOps Environment 2016
 
Less Alarming Alerts - SRECon 2016
Less Alarming Alerts - SRECon 2016 Less Alarming Alerts - SRECon 2016
Less Alarming Alerts - SRECon 2016
 
What Ops Can Learn From Design
What Ops Can Learn From DesignWhat Ops Can Learn From Design
What Ops Can Learn From Design
 
Postgres 9.4 First Look
Postgres 9.4 First LookPostgres 9.4 First Look
Postgres 9.4 First Look
 
Less Alarming Alerts!
Less Alarming Alerts!Less Alarming Alerts!
Less Alarming Alerts!
 
Past, Present, and Pachyderm - All Things Open - 2013
Past, Present, and Pachyderm - All Things Open - 2013Past, Present, and Pachyderm - All Things Open - 2013
Past, Present, and Pachyderm - All Things Open - 2013
 
Big Bad "Upgraded" Postgres
Big Bad "Upgraded" PostgresBig Bad "Upgraded" Postgres
Big Bad "Upgraded" Postgres
 
Managing Databases In A DevOps Environment
Managing Databases In A DevOps EnvironmentManaging Databases In A DevOps Environment
Managing Databases In A DevOps Environment
 
The Essential PostgreSQL.conf
The Essential PostgreSQL.confThe Essential PostgreSQL.conf
The Essential PostgreSQL.conf
 
Pro Postgres 9
Pro Postgres 9Pro Postgres 9
Pro Postgres 9
 
Advanced WAL File Management With OmniPITR
Advanced WAL File Management With OmniPITRAdvanced WAL File Management With OmniPITR
Advanced WAL File Management With OmniPITR
 
Scaling with Postgres (Highload++ 2010)
Scaling with Postgres (Highload++ 2010)Scaling with Postgres (Highload++ 2010)
Scaling with Postgres (Highload++ 2010)
 
Intro to Postgres 9 Tutorial
Intro to Postgres 9 TutorialIntro to Postgres 9 Tutorial
Intro to Postgres 9 Tutorial
 
Check Please!
Check Please!Check Please!
Check Please!
 
Database Scalability Patterns
Database Scalability PatternsDatabase Scalability Patterns
Database Scalability Patterns
 
A Guide To PostgreSQL 9.0
A Guide To PostgreSQL 9.0A Guide To PostgreSQL 9.0
A Guide To PostgreSQL 9.0
 

Recently uploaded

AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 

Recently uploaded (20)

AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 

Database Scalability Patterns

  • 1. Database Scalability {Patterns} / Robert Treat Wednesday, July 21, 2010
  • 2. robert treat omniti Wednesday, July 21, 2010
  • 3. robert treat omniti postgres Wednesday, July 21, 2010
  • 4. robert treat omniti postgres oracle - mysql Wednesday, July 21, 2010
  • 5. robert treat omniti postgres oracle - mysql mssql - sqlite - nosql Wednesday, July 21, 2010
  • 6. What are “Database Scalability Patterns?” Wednesday, July 21, 2010
  • 7. Part Design Patterns Part Application Life-Cycle Wednesday, July 21, 2010
  • 8. Part Design Patterns Part Application Life-Cycle Wednesday, July 21, 2010
  • 9. Part Design Patterns Part Application Life-Cycle Wednesday, July 21, 2010
  • 10. Part Design Patterns Part Application Life-Cycle Wednesday, July 21, 2010
  • 16. Federated Data Storage “sharding” Horizontal Partitioning Read Slaves Multi-Master Horizontal Scaling Wednesday, July 21, 2010
  • 17. Read Slaves / Master - Slave Scale Read Load Wednesday, July 21, 2010
  • 18. Read Slaves / Master - Slave Scale Read Load writes go here! Wednesday, July 21, 2010
  • 19. Read Slaves / Master - Slave Scale Read Load writes go here! reads go here (or here) (or here) Wednesday, July 21, 2010
  • 20. Read Slaves / Master - Slave Scale Read Load database writes data to slaves writes go here! slave db slave db slave db reads go here (or here) (or here) Wednesday, July 21, 2010
  • 21. Read Slaves / Master - Slave Scale Read Load app writes data everywhere writes go here! memcached memcached memcached reads go here (or here) (or here) Wednesday, July 21, 2010
  • 22. Read Slaves / Master - Slave Scale Read Load • Typically • Full Copy of Data On Each Node • Asynchronous Wednesday, July 21, 2010
  • 23. Read Slaves / Master - Slave Scale Read Load • Typically • Consider • Full Copy of Data • Partial Copy On Each Node • Synchronous • Asynchronous • Don’t use a RDBMS? Wednesday, July 21, 2010
  • 24. Read Slaves / Master - Slave Scale Read Load • Typically • Consider • Full Copy of Data • Partial Copy On Each Node • Synchronous • Asynchronous • Don’t use a RDBMS? Requires Application Changes Wednesday, July 21, 2010
  • 25. Read Slaves / Master - Slave Scale Read Load • Typically • Consider • Full Copy of Data • Partial Copy On Each Node • Synchronous • Asynchronous • Don’t use a RDBMS? Requires Application Changes “easy” Wednesday, July 21, 2010
  • 27. Scaling Writes “not easy” Wednesday, July 21, 2010
  • 28. Multi-Master many different ways to implement this, few that actually work in production Wednesday, July 21, 2010
  • 29. Multi-Master many different ways to implement this, few that actually work in production write to any node, database syncs data Wednesday, July 21, 2010
  • 30. Multi-Master many different ways to implement this, few that actually work in production write to any node, database syncs data can reduce cpu, doesn’t reduce i/o Wednesday, July 21, 2010
  • 31. Multi-Master many different ways to implement this, few that actually work in production write to any node, database syncs data can reduce cpu, doesn’t reduce i/o failover solution not a scalability solution Wednesday, July 21, 2010
  • 32. Horizontal Partitioning “sharding” “SOA” Wednesday, July 21, 2010
  • 33. Horizontal Partitioning “sharding” “SOA” Wednesday, July 21, 2010
  • 34. Horizontal Partitioning • Divide schema by job operations • Move each piece to own server • Duplicate some data as needed Wednesday, July 21, 2010
  • 35. Horizontal Partitioning items • Divide schema by job operations • Move each piece to own server • Duplicate some data as needed forums users Wednesday, July 21, 2010
  • 36. Horizontal Partitioning items • Divide schema by job operations • Move each piece to own server • Duplicate some data as needed forums • You must separate dependencies in the app code first! users Wednesday, July 21, 2010
  • 37. Horizontal Partitioning items • Divide schema by job operations • Move each piece to own server • Duplicate some data as needed forums • You must separate dependencies in the app code first! users Each node is a new instance of vertical scaling Wednesday, July 21, 2010
  • 38. Horizontal Scaling • data split across servers based on algorithm • data dropped into buckets (multiple?) app magic hash algorithm Wednesday, July 21, 2010
  • 39. Horizontal Scaling • data split across servers based on algorithm • data dropped into buckets (multiple?) magic hash algorithm app Wednesday, July 21, 2010
  • 40. Horizontal Scaling • data split across servers based on algorithm • data dropped into buckets (multiple?) •someone must keep track of data, and provide lookup services magic hash algorithm app Wednesday, July 21, 2010
  • 41. Universal Truths of Scaling Databases Vertical Scalability is Helpful for Every Pattern Even in a horizontally scaled, fully distributed database, the number of nodes needed is affected by vertical scalability Wednesday, July 21, 2010
  • 42. Universal Truths of Scaling Databases New Nodes Are Never Free • Add points of failure • Add management costs • Add complexity to architecture • Add complexity to your app code Wednesday, July 21, 2010
  • 43. MyFirstDB V Vertical Partitioning V Vertical Scaling V Read Slaves V Horizontal Partitioning Wednesday, July 21, 2010
  • 45. tips plan for layered data sources Wednesday, July 21, 2010
  • 46. tips plan for layered data sources read / write connections in code Wednesday, July 21, 2010
  • 47. tips plan for layered data sources read / write connections in code use schemas to separate services Wednesday, July 21, 2010
  • 48. THANKS! more: xzilla.net @robtreat2 omniti.com/surge (scalability conference) Wednesday, July 21, 2010