Database Scalability
                 {Patterns}

                           / Robert Treat



Wednesday, July 21, 2010
robert treat
                             omniti




Wednesday, July 21, 2010
robert treat
                              omniti
                             postgres




Wednesday, July 21, 2010
robert treat
                               omniti
                              postgres
                           oracl...
robert treat
                                  omniti
                                 postgres
                          ...
What are “Database Scalability Patterns?”




Wednesday, July 21, 2010
Part Design Patterns



                                 Part Application Life-Cycle




Wednesday, July 21, 2010
Part Design Patterns



                                 Part Application Life-Cycle




Wednesday, July 21, 2010
Part Design Patterns



                                 Part Application Life-Cycle




Wednesday, July 21, 2010
Part Design Patterns



                                 Part Application Life-Cycle




Wednesday, July 21, 2010
MyFirstDatabase




Wednesday, July 21, 2010
Vertical Partitioning




Wednesday, July 21, 2010
Vertical Scaling




Wednesday, July 21, 2010
Vertical Scaling




Wednesday, July 21, 2010
Vertical Scaling




Wednesday, July 21, 2010
Federated Data Storage


              “sharding”                 Horizontal Partitioning

                           Read...
Read Slaves / Master - Slave
                                Scale Read Load




Wednesday, July 21, 2010
Read Slaves / Master - Slave
                                Scale Read Load


                                           ...
Read Slaves / Master - Slave
                                   Scale Read Load


                                        ...
Read Slaves / Master - Slave
                                      Scale Read Load

          database writes
           d...
Read Slaves / Master - Slave
                                     Scale Read Load

          app writes data
            e...
Read Slaves / Master - Slave
                                Scale Read Load

       • Typically
         • Full Copy of D...
Read Slaves / Master - Slave
                                Scale Read Load

       • Typically                         •...
Read Slaves / Master - Slave
                                 Scale Read Load

       • Typically                         ...
Read Slaves / Master - Slave
                                 Scale Read Load

       • Typically                         ...
Scaling Writes




Wednesday, July 21, 2010
Scaling Writes




                            “not easy”




Wednesday, July 21, 2010
Multi-Master

                           many different ways to implement this,
                           few that actual...
Multi-Master

                           many different ways to implement this,
                           few that actual...
Multi-Master

                           many different ways to implement this,
                           few that actual...
Multi-Master

                           many different ways to implement this,
                           few that actual...
Horizontal Partitioning


                           “sharding”                 “SOA”




Wednesday, July 21, 2010
Horizontal Partitioning


                           “sharding”                 “SOA”




Wednesday, July 21, 2010
Horizontal Partitioning

    • Divide schema by job operations
    • Move each piece to own server
    • Duplicate some da...
Horizontal Partitioning

                                                     items
    • Divide schema by job operations
...
Horizontal Partitioning

                                                     items
    • Divide schema by job operations
...
Horizontal Partitioning

                                                        items
    • Divide schema by job operatio...
Horizontal Scaling
             • data split across servers based on algorithm
             • data dropped into buckets (m...
Horizontal Scaling
             • data split across servers based on algorithm
             • data dropped into buckets (m...
Horizontal Scaling
             • data split across servers based on algorithm
             • data dropped into buckets (m...
Universal Truths of Scaling Databases

                   Vertical Scalability is Helpful for Every Pattern


            ...
Universal Truths of Scaling Databases

                               New Nodes Are Never Free


                         ...
MyFirstDB
                                      V
                            Vertical Partitioning
                      ...
tips




Wednesday, July 21, 2010
tips

                           plan for layered data sources




Wednesday, July 21, 2010
tips

                            plan for layered data sources

                           read / write connections in co...
tips

                             plan for layered data sources

                           read / write connections in c...
THANKS!

                                  more:
                                 xzilla.net
                             ...
Upcoming SlideShare
Loading in...5
×

Database Scalability Patterns

8,263

Published on

** Talk Given at OSCon 2010 **

We often have clients approach us looking for help in scaling their systems, and all too often their long term vision is a mixed reality based on the approaches read about on popular blogs trying to solve very different problems. Hey, scaling your database can be difficult enough by itself, you don't want to get tripped up by not understanding where you're going. In Database Scalability Patterns we will attempt to distill all of the information/hype/discussions around scaling databases, and break down the common patterns we've seen dealing with scaling databases. "Buzzwords" we'll cover (and hopefully debuzz) include:

Vertical Scaling
Horizontal Partitioning
Horizontal Scaling
Read Slaves
Multi-Master
Vertical Partitioning
Federated Data Storage

More important than just describing what these things are (although that's a good first step), we'll also discuss along the way different points in the life-cycle of your database when you need to be thinking about the different options in front of you. We'll factor in the types of application that your working on (think OLTP vs OLAP, or Social Networking vs. Corporate Application), the environment you'll be working on (Scaling "in the cloud" is very different from DIY in the datacenter), and we will talk about the types of tools you'll need to accomplish these goals (All replication systems are not the same, and some won't help at all).

Published in: Technology
0 Comments
16 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
8,263
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
16
Embeds 0
No embeds

No notes for slide

Database Scalability Patterns

  1. 1. Database Scalability {Patterns} / Robert Treat Wednesday, July 21, 2010
  2. 2. robert treat omniti Wednesday, July 21, 2010
  3. 3. robert treat omniti postgres Wednesday, July 21, 2010
  4. 4. robert treat omniti postgres oracle - mysql Wednesday, July 21, 2010
  5. 5. robert treat omniti postgres oracle - mysql mssql - sqlite - nosql Wednesday, July 21, 2010
  6. 6. What are “Database Scalability Patterns?” Wednesday, July 21, 2010
  7. 7. Part Design Patterns Part Application Life-Cycle Wednesday, July 21, 2010
  8. 8. Part Design Patterns Part Application Life-Cycle Wednesday, July 21, 2010
  9. 9. Part Design Patterns Part Application Life-Cycle Wednesday, July 21, 2010
  10. 10. Part Design Patterns Part Application Life-Cycle Wednesday, July 21, 2010
  11. 11. MyFirstDatabase Wednesday, July 21, 2010
  12. 12. Vertical Partitioning Wednesday, July 21, 2010
  13. 13. Vertical Scaling Wednesday, July 21, 2010
  14. 14. Vertical Scaling Wednesday, July 21, 2010
  15. 15. Vertical Scaling Wednesday, July 21, 2010
  16. 16. Federated Data Storage “sharding” Horizontal Partitioning Read Slaves Multi-Master Horizontal Scaling Wednesday, July 21, 2010
  17. 17. Read Slaves / Master - Slave Scale Read Load Wednesday, July 21, 2010
  18. 18. Read Slaves / Master - Slave Scale Read Load writes go here! Wednesday, July 21, 2010
  19. 19. Read Slaves / Master - Slave Scale Read Load writes go here! reads go here (or here) (or here) Wednesday, July 21, 2010
  20. 20. Read Slaves / Master - Slave Scale Read Load database writes data to slaves writes go here! slave db slave db slave db reads go here (or here) (or here) Wednesday, July 21, 2010
  21. 21. Read Slaves / Master - Slave Scale Read Load app writes data everywhere writes go here! memcached memcached memcached reads go here (or here) (or here) Wednesday, July 21, 2010
  22. 22. Read Slaves / Master - Slave Scale Read Load • Typically • Full Copy of Data On Each Node • Asynchronous Wednesday, July 21, 2010
  23. 23. Read Slaves / Master - Slave Scale Read Load • Typically • Consider • Full Copy of Data • Partial Copy On Each Node • Synchronous • Asynchronous • Don’t use a RDBMS? Wednesday, July 21, 2010
  24. 24. Read Slaves / Master - Slave Scale Read Load • Typically • Consider • Full Copy of Data • Partial Copy On Each Node • Synchronous • Asynchronous • Don’t use a RDBMS? Requires Application Changes Wednesday, July 21, 2010
  25. 25. Read Slaves / Master - Slave Scale Read Load • Typically • Consider • Full Copy of Data • Partial Copy On Each Node • Synchronous • Asynchronous • Don’t use a RDBMS? Requires Application Changes “easy” Wednesday, July 21, 2010
  26. 26. Scaling Writes Wednesday, July 21, 2010
  27. 27. Scaling Writes “not easy” Wednesday, July 21, 2010
  28. 28. Multi-Master many different ways to implement this, few that actually work in production Wednesday, July 21, 2010
  29. 29. Multi-Master many different ways to implement this, few that actually work in production write to any node, database syncs data Wednesday, July 21, 2010
  30. 30. Multi-Master many different ways to implement this, few that actually work in production write to any node, database syncs data can reduce cpu, doesn’t reduce i/o Wednesday, July 21, 2010
  31. 31. Multi-Master many different ways to implement this, few that actually work in production write to any node, database syncs data can reduce cpu, doesn’t reduce i/o failover solution not a scalability solution Wednesday, July 21, 2010
  32. 32. Horizontal Partitioning “sharding” “SOA” Wednesday, July 21, 2010
  33. 33. Horizontal Partitioning “sharding” “SOA” Wednesday, July 21, 2010
  34. 34. Horizontal Partitioning • Divide schema by job operations • Move each piece to own server • Duplicate some data as needed Wednesday, July 21, 2010
  35. 35. Horizontal Partitioning items • Divide schema by job operations • Move each piece to own server • Duplicate some data as needed forums users Wednesday, July 21, 2010
  36. 36. Horizontal Partitioning items • Divide schema by job operations • Move each piece to own server • Duplicate some data as needed forums • You must separate dependencies in the app code first! users Wednesday, July 21, 2010
  37. 37. Horizontal Partitioning items • Divide schema by job operations • Move each piece to own server • Duplicate some data as needed forums • You must separate dependencies in the app code first! users Each node is a new instance of vertical scaling Wednesday, July 21, 2010
  38. 38. Horizontal Scaling • data split across servers based on algorithm • data dropped into buckets (multiple?) app magic hash algorithm Wednesday, July 21, 2010
  39. 39. Horizontal Scaling • data split across servers based on algorithm • data dropped into buckets (multiple?) magic hash algorithm app Wednesday, July 21, 2010
  40. 40. Horizontal Scaling • data split across servers based on algorithm • data dropped into buckets (multiple?) •someone must keep track of data, and provide lookup services magic hash algorithm app Wednesday, July 21, 2010
  41. 41. Universal Truths of Scaling Databases Vertical Scalability is Helpful for Every Pattern Even in a horizontally scaled, fully distributed database, the number of nodes needed is affected by vertical scalability Wednesday, July 21, 2010
  42. 42. Universal Truths of Scaling Databases New Nodes Are Never Free • Add points of failure • Add management costs • Add complexity to architecture • Add complexity to your app code Wednesday, July 21, 2010
  43. 43. MyFirstDB V Vertical Partitioning V Vertical Scaling V Read Slaves V Horizontal Partitioning Wednesday, July 21, 2010
  44. 44. tips Wednesday, July 21, 2010
  45. 45. tips plan for layered data sources Wednesday, July 21, 2010
  46. 46. tips plan for layered data sources read / write connections in code Wednesday, July 21, 2010
  47. 47. tips plan for layered data sources read / write connections in code use schemas to separate services Wednesday, July 21, 2010
  48. 48. THANKS! more: xzilla.net @robtreat2 omniti.com/surge (scalability conference) Wednesday, July 21, 2010

×