Scaling with Postgres Robert Treat Percona Performance Conference 2009

Robert Treat

Percona Performance Conference

2009

Disclaimer Every “really big” site uses custom solutions. Don’t expect to get there using out of the box software. You don’t have to reinvent the wheel. But you may have to build the car.

Who am I ? OmniTI TB+ databases top 100 websites Oracle/MySQL/Postgres Not necessarily in that order :-) P.S. We’re Hiring

OmniTI

TB+ databases

top 100 websites

Oracle/MySQL/Postgres

Not necessarily in that order :-)

P.S. We’re Hiring

Scaling Databases 101 Implementation Tuning Optimize Queries Optimize Schema Throw Hardware At It Caching Replication Federation

Implementation

Tuning

Optimize Queries

Optimize Schema

Throw Hardware At It

Caching

Replication

Federation

Implementation Normalization Foreign Keys Relational Modeling Mostly Irrelevant

Normalization

Foreign Keys

Relational Modeling

Implementation Monitoring nagios, munin, etc.. check_postgres Trending cacti, noit watch your hardware!

Monitoring

nagios, munin, etc..

check_postgres

Trending

cacti, noit

watch your hardware!

Tuning Get the Basics Right http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server Someday Xlogs, Checkpoints Vacuum Strategy Costs and other Minutia

Get the Basics Right

http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server

Someday

Xlogs, Checkpoints

Vacuum Strategy

Costs and other Minutia

Optimize Queries explain explain analyze explain analyze verbose

explain

explain analyze

explain analyze verbose

Optimize Queries writing queries tuned to your databases planner

writing queries tuned to your databases planner

Optimize Schema indexing partial, functional fix data types add/remove tables de-normalize (maybe)

indexing

partial, functional

fix data types

add/remove tables

de-normalize (maybe)

Why is this important? Postgres is good at scaling vertically Flawed database schema hides the real problems

Postgres is good at scaling vertically

Flawed database schema hides the real problems

Why is this unimportant? Optimizing is often a trade off of time vs. money It is not an ultimate solution

Optimizing is often a trade off of time vs. money

It is not an ultimate solution

Throw Hardware At It Find your bottlenecks Memory / RAM CPU I/O, DISKS usually the biggest problem

Find your bottlenecks

Memory / RAM

CPU

I/O, DISKS

usually the biggest problem

Can you take me higher? 32 Cores 128 GB+ RAM Spindles out the wazoo tablespaces

32 Cores

128 GB+ RAM

Spindles out the wazoo

tablespaces

Caching Internal materialized views External memcached, et al. static files

Internal

materialized views

External

memcached, et al.

static files

Oh, the places you’ll go Nothing Fancy or Exciting Solid Engineering Understanding needs / limits Top 1000+ Website

Nothing Fancy or Exciting

Solid Engineering

Understanding needs / limits

Replication Ignore the hype Triggers won’t destroy your site Scripting solves a lot of config issues Telling your DBA to do his !@#$ job solves config issues too Add-on is FUD for Modular

Ignore the hype

Triggers won’t destroy your site

Scripting solves a lot of config issues

Telling your DBA to do his !@#$ job solves config issues too

Add-on is FUD for Modular

Replication Truth in advertising Oracle does it better MySQL does it simpler

Truth in advertising

Oracle does it better

MySQL does it simpler

Replication Scaling Reads Master, many slaves Slony, Londiste, Bucardo, PGPoolII Scaling Writes Mostly impossible Bucardo, PGPoolII

Scaling Reads

Master, many slaves

Slony, Londiste, Bucardo, PGPoolII

Scaling Writes

Mostly impossible

Bucardo, PGPoolII

What is Federation? A process for eliminating performance and/or scaleability as a requirement for your database

A process for eliminating performance and/or scaleability as a requirement for your database

Federation MySQL software has no federation advantage MySQL community has large federation advantage

MySQL software has no federation advantage

MySQL community has large federation advantage

Federation Things to consider Horizontal Partitioning Application Based Partitioning BASE golconde, gearman

Things to consider

Horizontal Partitioning

Application Based Partitioning

BASE

golconde, gearman

El Fin www.xzilla.net www.omniti.com www.postgresql.us

www.xzilla.net

www.omniti.com

www.postgresql.us

Postgres BOF Tonight @ 7:30 PM Location @ Camino Real

Tonight @ 7:30 PM

Location @ Camino Real