Scaling with Postgres
Robert Treat
Percona Performance Conference
2009
Wednesday, April 22, 2009

Disclaimer
Every “really big” site uses custom solutions.
Don’t expect to get there using out of the box
software. You don’t have to reinvent the wheel.
But you may have to build the car.
Wednesday, April 22, 2009

Who am I ?
Wednesday, April 22, 2009

Who am I ?
OmniTI
TB+ databases
top 100 websites
Oracle/MySQL/Postgres
Not necessarily in that order :-)
Wednesday, April 22, 2009

Scaling Databases 101
Wednesday, April 22, 2009

Scaling Databases 101
Implementation
Wednesday, April 22, 2009

Scaling Databases 101
Implementation
Tuning
Optimize Queries
Optimize Schema
Wednesday, April 22, 2009

Scaling Databases 101
Implementation
Tuning
Optimize Queries
Optimize Schema
Throw Hardware At It
Caching
Wednesday, April 22, 2009

Scaling Databases 101
Implementation
Tuning
Optimize Queries
Optimize Schema
Throw Hardware At It
Caching
Replication
Wednesday, April 22, 2009

Scaling Databases 101
Implementation
Tuning
Optimize Queries
Optimize Schema
Throw Hardware At It
Caching
Replication
Federation
Wednesday, April 22, 2009

Implementation
Normalization
Foreign Keys
Relational Modeling
Wednesday, April 22, 2009

Implementation
Normalization
Foreign Keys
Relational Modeling
Mostly Irrelevant
Wednesday, April 22, 2009

Implementation
Wednesday, April 22, 2009

Implementation
Monitoring
nagios, munin, etc..
check_postgres
Wednesday, April 22, 2009

Implementation
Monitoring
nagios, munin, etc..
check_postgres
Trending
cacti, noit
watch your hardware!
Wednesday, April 22, 2009

Tuning
Wednesday, April 22, 2009

Tuning
Get the Basics Right
http://wiki.postgresql.org/wiki/
Tuning_Your_PostgreSQL_Server
Wednesday, April 22, 2009

Tuning
Get the Basics Right
http://wiki.postgresql.org/wiki/
Tuning_Your_PostgreSQL_Server
Someday
Xlogs, Checkpoints
Vacuum Strategy
Costs and other Minutia
Wednesday, April 22, 2009

Optimize Queries
explain
explain analyze
explain analyze verbose
Wednesday, April 22, 2009

Optimize Queries
writing queries tuned to your
databases planner
Wednesday, April 22, 2009

Optimize Schema
Wednesday, April 22, 2009

Optimize Schema
indexing
partial, functional
fix data types
add/remove tables
de-normalize (maybe)
Wednesday, April 22, 2009

Why is this important?
Postgres is good at scaling vertically
Flawed database schema hides the
real problems
Wednesday, April 22, 2009

Why is this unimportant?
Optimizing is often a trade off of time
vs. money
It is not an ultimate solution
Wednesday, April 22, 2009

Throw Hardware At It
Find your bottlenecks
Memory / RAM
CPU
I/O, DISKS
usually the biggest problem
Wednesday, April 22, 2009

Can you take me higher?
32 Cores
128 GB+ RAM
Spindles out the wazoo
tablespaces
Wednesday, April 22, 2009

Caching
Wednesday, April 22, 2009

Caching
Internal
materialized views
Wednesday, April 22, 2009

Caching
Internal
materialized views
External
memcached, et al.
static files
Wednesday, April 22, 2009

Oh, the places you’ll go
Nothing Fancy or Exciting
Solid Engineering
Understanding needs / limits
Wednesday, April 22, 2009

Oh, the places you’ll go
Nothing Fancy or Exciting
Solid Engineering
Understanding needs / limits
Top 1000+ Website
Wednesday, April 22, 2009

Replication
Ignore the hype
Triggers won’t destroy your site
Scripting solves a lot of config
issues
Telling your DBA to do his !@#$ job
solves config issues too
Add-on is FUD for Modular
Wednesday, April 22, 2009

Replication
Truth in advertising
Oracle does it better
MySQL does it simpler
Wednesday, April 22, 2009

Replication
Wednesday, April 22, 2009

Replication
Scaling Reads
Master, many slaves
Slony, Londiste, Bucardo, PGPoolII
Wednesday, April 22, 2009

Replication
Scaling Reads
Master, many slaves
Slony, Londiste, Bucardo, PGPoolII
Scaling Writes
Mostly impossible
Bucardo, PGPoolII
Wednesday, April 22, 2009

What is Federation?
A process for eliminating performance
and/or scaleability as a requirement for
your database
Wednesday, April 22, 2009

Federation
Wednesday, April 22, 2009

Federation
MySQL software has no federation
advantage
Wednesday, April 22, 2009

Federation
MySQL software has no federation
advantage
MySQL community has large federation
advantage
Wednesday, April 22, 2009

Federation
Things to consider
Horizontal Partitioning
Application Based Partitioning
BASE
golconde, gearman
Wednesday, April 22, 2009

El Fin
www.xzilla.net
www.omniti.com
www.postgresql.us
Wednesday, April 22, 2009

Postgres BOF
Tonight @ 7:30 PM
Location @ Camino Real
Wednesday, April 22, 2009