Database Scalability
{Patterns}
/ Robert Treat
Wednesday, July 21, 2010

robert treat
omniti
Wednesday, July 21, 2010

robert treat
omniti
postgres
Wednesday, July 21, 2010

robert treat
omniti
postgres
oracle - mysql
Wednesday, July 21, 2010

robert treat
omniti
postgres
oracle - mysql
mssql - sqlite - nosql
Wednesday, July 21, 2010

What are “Database Scalability Patterns?”
Wednesday, July 21, 2010

Part Design Patterns
Part Application Life-Cycle
Wednesday, July 21, 2010

Part Design Patterns
Part Application Life-Cycle
Wednesday, July 21, 2010

Part Design Patterns
Part Application Life-Cycle
Wednesday, July 21, 2010

Part Design Patterns
Part Application Life-Cycle
Wednesday, July 21, 2010

MyFirstDatabase
Wednesday, July 21, 2010

Vertical Partitioning
Wednesday, July 21, 2010

Vertical Scaling
Wednesday, July 21, 2010

Vertical Scaling
Wednesday, July 21, 2010

Vertical Scaling
Wednesday, July 21, 2010

Federated Data Storage
“sharding” Horizontal Partitioning
Read Slaves
Multi-Master
Horizontal Scaling
Wednesday, July 21, 2010

Read Slaves / Master - Slave
Scale Read Load
Wednesday, July 21, 2010

Read Slaves / Master - Slave
Scale Read Load
writes go here!
Wednesday, July 21, 2010

Read Slaves / Master - Slave
Scale Read Load
writes go here!
reads go here (or here) (or here)
Wednesday, July 21, 2010

Read Slaves / Master - Slave
Scale Read Load
database writes
data to slaves writes go here!
slave db slave db slave db
reads go here (or here) (or here)
Wednesday, July 21, 2010

Read Slaves / Master - Slave
Scale Read Load
app writes data
everywhere writes go here!
memcached memcached memcached
reads go here (or here) (or here)
Wednesday, July 21, 2010

Read Slaves / Master - Slave
Scale Read Load
• Typically
• Full Copy of Data
On Each Node
• Asynchronous
Wednesday, July 21, 2010

Read Slaves / Master - Slave
Scale Read Load
• Typically • Consider
• Full Copy of Data • Partial Copy
On Each Node • Synchronous
• Asynchronous • Don’t use a RDBMS?
Wednesday, July 21, 2010

Read Slaves / Master - Slave
Scale Read Load
• Typically • Consider
• Full Copy of Data • Partial Copy
On Each Node • Synchronous
• Asynchronous • Don’t use a RDBMS?
Requires Application Changes
Wednesday, July 21, 2010

Read Slaves / Master - Slave
Scale Read Load
• Typically • Consider
• Full Copy of Data • Partial Copy
On Each Node • Synchronous
• Asynchronous • Don’t use a RDBMS?
Requires Application Changes
“easy”
Wednesday, July 21, 2010

Scaling Writes
Wednesday, July 21, 2010

Scaling Writes
“not easy”
Wednesday, July 21, 2010

Multi-Master
many different ways to implement this,
few that actually work in production
Wednesday, July 21, 2010

Multi-Master
many different ways to implement this,
few that actually work in production
write to any node, database syncs data
Wednesday, July 21, 2010

Multi-Master
many different ways to implement this,
few that actually work in production
write to any node, database syncs data
can reduce cpu, doesn’t reduce i/o
Wednesday, July 21, 2010

Multi-Master
many different ways to implement this,
few that actually work in production
write to any node, database syncs data
can reduce cpu, doesn’t reduce i/o
failover solution
not a scalability solution
Wednesday, July 21, 2010

Horizontal Partitioning
“sharding” “SOA”
Wednesday, July 21, 2010

Horizontal Partitioning
“sharding” “SOA”
Wednesday, July 21, 2010

Horizontal Partitioning
• Divide schema by job operations
• Move each piece to own server
• Duplicate some data as needed
Wednesday, July 21, 2010

Horizontal Partitioning
items
• Divide schema by job operations
• Move each piece to own server
• Duplicate some data as needed
forums
users
Wednesday, July 21, 2010

Horizontal Partitioning
items
• Divide schema by job operations
• Move each piece to own server
• Duplicate some data as needed
forums
• You must separate dependencies
in the app code first!
users
Wednesday, July 21, 2010

Horizontal Partitioning
items
• Divide schema by job operations
• Move each piece to own server
• Duplicate some data as needed
forums
• You must separate dependencies
in the app code first!
users
Each node is a new instance of vertical scaling
Wednesday, July 21, 2010

Horizontal Scaling
• data split across servers based on algorithm
• data dropped into buckets (multiple?)
app
magic hash algorithm
Wednesday, July 21, 2010

Horizontal Scaling
• data split across servers based on algorithm
• data dropped into buckets (multiple?)
magic hash algorithm app
Wednesday, July 21, 2010

Horizontal Scaling
• data split across servers based on algorithm
• data dropped into buckets (multiple?)
•someone must keep track of data, and provide
lookup services
magic hash algorithm app
Wednesday, July 21, 2010

Universal Truths of Scaling Databases
Vertical Scalability is Helpful for Every Pattern
Even in a horizontally scaled, fully distributed
database, the number of nodes needed is
affected by vertical scalability
Wednesday, July 21, 2010

Universal Truths of Scaling Databases
New Nodes Are Never Free
• Add points of failure
• Add management costs
• Add complexity to architecture
• Add complexity to your app code
Wednesday, July 21, 2010

MyFirstDB
V
Vertical Partitioning
V
Vertical Scaling
V
Read Slaves
V
Horizontal Partitioning
Wednesday, July 21, 2010

tips
Wednesday, July 21, 2010

tips
plan for layered data sources
Wednesday, July 21, 2010

tips
plan for layered data sources
read / write connections in code
Wednesday, July 21, 2010

tips
plan for layered data sources
read / write connections in code
use schemas to separate services
Wednesday, July 21, 2010

THANKS!
more:
xzilla.net
@robtreat2
omniti.com/surge
(scalability conference)
Wednesday, July 21, 2010