Pro Postgres 9

Pro Postgres 9 Robert Treat, JDCon East 2011Thursday, March 31, 2011

Who Am I? (Why Listen To Me) PostgreSQL User Since 6.5.x DBA of High Trafﬁc / Large PostgreSQL Instances Long Time Contributor to PostgreSQL Project Contribute / Maintain Several Open Source Projects Co-Author Beginning PHP & PostgreSQL 8 (Apress) I like to call it “Postgres” Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Outline What you need to know about the project Backups Getting Started Replication Upgrading Query Tuning Conﬁguring Your Server Other stuff you should know about Hardware Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Know Your Way Around The ProjectThursday, March 31, 2011

Know Your Way Around The Project www.postgresql.org downloads security alerts documentation wiki bug reports support companies rss -> news - events - version Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Know Your Way Around The Project planet.postgresql.org project news helpful tips / examples community news Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Know Your Way Around The Project www.pgfoundry.org URI Type SkyTools modules Npgsql programs pg_bulkload resources plpgsql-debugger pagila, sample database Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Know Your Way Around The Project git.postgresql.org check_postgres modules edb-installers programs website code resources json type works in progress pg_top user repo’s see also github.com Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Know Your Way Around The Project archives.postgresql.org mailing list archives back to 1997 full text search via postgresql FTS keyword search suggestions lists for users, developers, regional, user groups Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Know Your Way Around The Project #postgresql irc.freenode.net real time help pg_docbot - ??help Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Know Your Way Around The Project project management core team committers -hackers road map web team Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Get Off To A Good StartThursday, March 31, 2011

Get Off To A Good Start Use Package Management Consistent Standardized Simple Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Get Off To A Good Start Use Package Management Different across systems Upgrades can be a problem Trust your packager? Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Get Off To A Good Start Use Package Management Different across systems Upgrades can be a problem Trust your packager? Don’t Be Afraid To Roll Your Own Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Get Off To A Good Start A Note About Versions Just playing? Default packages probably OK, (min 8.2) Any production level work should start on 9.0 Any project not due to launch for 6 months, 9.1 Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Get Off To A Good Start So What Do You Get? pg_control version number: 903 Catalog version number: 201008051 Database system identifier: 5546136951888795324 Database cluster state: in archive recovery pg_control last modified: Tue Mar 22 11:50:24 2011 Latest checkpoint location: 7C/1D000058 Prior checkpoint location: 7C/1C000058 Latest checkpoints REDO location: 7C/1D000020 Latest checkpoints TimeLineID: 1 Latest checkpoints NextXID: 0/4924 Latest checkpoints NextOID: 16608 Latest checkpoints NextMultiXactId: 1 Latest checkpoints NextMultiOffset: 0 Latest checkpoints oldestXID: 654 Latest checkpoints oldestXIDs DB: 1 pg_controldata Latest checkpoints oldestActiveXID: Time of latest checkpoint: 4924 Tue Mar 22 11:46:19 2011 Minimum recovery ending location: 7C/16000000 Backup start location: 0/0 Current wal_level setting: hot_standby Current max_connections setting: 500 Current max_prepared_xacts setting: 0 Current max_locks_per_xact setting: 64 Maximum data alignment: 8 Database block size: 8192 Blocks per segment of large relation: 131072 WAL block size: 8192 Bytes per WAL segment: 16777216 Maximum length of identifiers: 64 Maximum columns in an index: 32 Maximum size of a TOAST chunk: 1996 Date/time type storage: 64-bit integers Float4 argument passing: by value Float8 argument passing: by value Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Get Off To A Good Start So What Do You Get?-rw------- 1 postgres postgres 4 Dec 2 12:23 PG_VERSIONdrwxr-xr-x 6 postgres postgres 6 Dec 2 13:41 basedrwxr-xr-x 2 postgres postgres 42 Jan 28 17:49 globaldrwxr-xr-x 2 postgres postgres 3 Dec 2 12:23 pg_clog-rw------- 1 postgres postgres 3938 Dec 19 11:25 pg_hba.conf-rw------- 1 postgres postgres 1636 Dec 19 11:25 pg_ident.confdrwxr-xr-x 3 postgres postgres 113 Mar 22 00:00 pg_logdrwxr-xr-x 4 postgres postgres 4 Dec 2 12:23 pg_multixactdrwxr-xr-x 2 postgres postgres 3 Jan 12 14:23 pg_notifydrwxr-xr-x 2 postgres postgres 3 Jan 12 14:23 pg_stat_tmpdrwxr-xr-x 2 postgres postgres 3 Dec 2 12:23 pg_subtransdrwxr-xr-x 2 postgres postgres 2 Dec 2 12:23 pg_tblspcdrwxr-xr-x 2 postgres postgres 2 Dec 2 12:23 pg_twophaselrwxrwxrwx 1 postgres postgres 30 Dec 19 11:13 pg_xlog-rw------- 1 postgres postgres 17915 Mar 3 21:52 postgresql.conf-rw------- 1 postgres postgres 63 Jan 12 14:23 postmaster.opts-rw------- 1 postgres postgres 54 Jan 12 14:23 postmaster.pid-rw-r--r-- 1 postgres postgres 336 Dec 20 16:14 recovery.conf Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Get Off To A Good Start Conﬁgure Your Logging $PGDATA/pg_log /var/log/postgresql when in doubt... (postgresql.conf) separate disk Logging is often overlooked, but it is the ﬁrst step toward troubleshooting! Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Get Off To A Good Start Conﬁgure Authentication most systems have different defaults ﬁrewalls / selinux (FATAL?) rtfm (pg_hba.conf, grant, revoke) Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Get Off To A Good Start Authentication Methods TRUST md5 IDENT Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Get Off To A Good Start /contrib Trust these more than your own code package dependent use different schemas (when able) pgcrypto Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Get Off To A Good Start Additional Supplied Modules “contrib” Trust these more than your own code package dependent use different schemas (when able) pgcrypto, pg_upgrade http://www.postgresql.org/docs/current/interactive/contrib.html Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Let’s Talk About UpgradesThursday, March 31, 2011

Let’s Talk About Upgrades Versioning First Digit (7.4.16 -> 8.2.0) Second Digit (8.3.6 -> 8.4.0) Third Digit (9.0.0 -> 9.0.2) Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Let’s Talk About Upgrades Achtung!! Make Backups! Read the Release Notes! Read the Commit Logs! Test! Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Let’s Talk About Upgrades pg_dump/pg_restore simple -Fc is your friend dump with new version of pg_dump pitfalls: (time, hdd) Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Let’s Talk About Upgrades the “slony” method not simple create slave on new version switchover (switch back?) pitfalls: (initial synch, compatibility) Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Let’s Talk About Upgrades pg_upgrade in place upgrades rewrites system catalog info no way to go back (fs snapshots) first official release was 9.0 Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Let’s Talk About Upgrades upgrading legacy databases < 8.2 is no longer supported (upgrade now!) pg_upgrade only works on 8.3 or greater older slony can work back to 7.3ish pre-7.3 ? Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Let’s Talk About Upgrades is it worth it? any database with significant transaction workload (read or write) >= 8.3 (vxid) most systems will want at least 8.4 (fsm) replication is nice too though... Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Let’s Talk About Upgrades Versioning First Digit (7.4.16 -> 8.2.0) Second Digit (8.3.6 -> 8.4.0) Third Digit (9.0.0 -> 9.0.2) Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Let’s Talk About Upgrades minor release versioning policy drop in replacement no feature changes bug fixes only (includes security fixes) sometimes involves extra steps also referred to as service packs, hot fixes, fix packs, etc... It is considered more dangerous to not upgrade! Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Figure Your ConﬁgureThursday, March 31, 2011

Figure Your Conﬁgure the basics: performance shared_buffers effective_cache_size work_mem checkpoint_segments Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Figure Your Conﬁgure the basics: logging stderr/pg_log vs. syslog/eventlog log_min_duration_statement log_line_prefix (%t, %p, %d, %u, %r) Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Figure Your Conﬁgure other stuff worth looking at maintenance_work_mem max_prepared_transactions wal_buffers checkpoint_completion_target autovacuum_freeze_max_age Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Figure Your Conﬁgure the basics: replication primary: wal_level = ‘hot_standby’ primary: max_wal_senders = n slave: hot_standby = on Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Hardware For SoftwareThursday, March 31, 2011

Hardware for Software People cpu and you connections, autovacuum, bgwriter, stats collector, replication Multiple CPU’s work wonders, up to 32 processors http://tweakers.net Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Hardware for Software People disk does and don’ts Put WAL on it’s own disk (RAID 1) Put DATA directory on it’s own disk (RAID 10) More Spindles is Good More Controllers Even Gooder (tablespaces) Battery Backed Controllers, Write Cache Enabled Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Hardware for Software People disk don’ts NFS == Not Fully Safe! RAID 5 Beware disks / controllers that lie Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Can You Back That Up?Thursday, March 31, 2011

Can You Back That Up? types of backups logical (sql export) physical (files on disk) Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Can You Back That Up? logical backups slow to create slow to restore “pure”, no system level corruption susceptible to database level corruption pg_dump is your friend pg_dumpall for globals Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Can You Back That Up? physical backups replication / failover machine “tarball” (pitr) filesystem snapshots (pitr) Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Can You Back That Up? replication / failover node good idea, but... offsite? fragile? we’ll talk more in replication section :-) Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Can You Back That Up? “tarball” (pitr) basic idea is to copy all database files and relevant xlogs use multiple machines if able use rsync if able copy the slave if able!! Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Can You Back That Up? ﬁlesystem snapshots (pitr) basic idea is to copy all database files and relevant xlogs use multiple machines if able use rsync if able copy the slave if able!! Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Replication, ReplicationThursday, March 31, 2011

Replication, Replication 9.0 “Introduced” Replication In Postgres Built In Solution Based on WAL replay True Binary Replication 1st Cut, Still A Little Rough Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Replication, Replication However, There Are Lots Of Alternatives! Most Are Trigger Based: Slony, Bucardo, Londiste, RubyRep, etc... Statement Based (PGPoolII) Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Replication, Replication Quick Rundown: Slony Trigger Based Asynchronous Master -> (Multiple, Cascading) Slave Switchover “A Pain To Work With” Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Replication, Replication Quick Rundown: Londiste Trigger Based Asynchronous Master -> (Multiple) Slave Switchover “I Don’t Like Slony” Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Replication, Replication Quick Rundown: Bucardo Trigger Based Asynchronous Master -> (Multiple) Slave Master -> Master “Small Community” Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Replication, Replication Quick Rundown: Ruby Rep Trigger Based Asynchronous Master -> Slave Master -> Master “Huh?” Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Replication, Replication Determine Your Needs: Failover Simple 2 Server Setup? 9.0 Streaming Replication Multiple Slaves? Slony Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Replication, Replication Determine Your Needs: Switchover Master -> Slave Slony Londiste Master -> Master Bucardo Ruby Rep Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Replication, Replication Determine Your Needs: Single Read Slave Postgres Streaming Replication no changes to the slave Any of the Trigger Based Solutions Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Replication, Replication Determine Your Needs: Multiple Read Slaves Slony (Cascading) Bucardo (Split) Londiste (meh) Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Replication, Replication A Note On Scaling Writes Scaling Writes Is A Hard Problem Bucardo/RubyRep Help IF CPU Bound If you are bound on write i/o, you probably need to federate your database Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Query Your QueriesThursday, March 31, 2011

Query Your Queries ﬁnding slow queries: log_min_duration_statement -1, 0, n superuser only alter user LOG: duration: 5005.273ms statement: select pg_sleep(5); Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Query Your Queries ﬁnding slow queries: log analyzers command line tools i/o load generate reports cpu load archive reports Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Query Your Queries ﬁnding slow queries: log analyzers pqa pgfouine maatkit (percona) pgsi (endpoint) Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Query Your Queries ﬁxing slow queries: explain analyze universal tool good for specific queries “explain” for large queries could be it’s own talk Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Query Your Queries ﬁxing slow queries: explain analyze universal tool good for specific queries “explain” for large queries could be it’s own talk http://wiki.postgresql.org/Using_EXPLAIN Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Query Your Queries ﬁxing slow queries: indexing (basic) use explain to find large sequential reads use pg_stat_* tables to find numerous reads use btree (gist/gin) dual column vs. single column Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Query Your Queries ﬁxing slow queries: indexing (partial) restrain index to rows that matter can give significant speed improvements where clause of index should match where clause of query create index address_ba_part_idx on address (district) where district = ‘Buenos Aires’; Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Query Your Queries ﬁxing slow queries: indexing (partial) restrain index to rows that matter can give significant speed improvements where clause of index should match where clause of query create index customer_active_part_idx on customer (customer_id) where active is true; Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Query Your Queries ﬁxing slow queries: indexing (functional) some people prefer to call these expressional indexes Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Query Your Queries ﬁxing slow queries: indexing (expressional) push expensive functions into your index system sees just WHERE indexedcolumn = ‘constant’ expression of index should match expression of queries narrow scope, but nice gains create unique index one_true_email_xidx on customer (lower(email)); Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Query Your Queries ﬁxing slow queries: indexing (expressional) push expensive functions into your index system sees just WHERE indexedcolumn = ‘constant’ expression of index should match expression of queries narrow scope, but nice gains create index fullname_xidx on customer ((firstname||’ ‘||lastname)); Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Other Stuff I Should MentionThursday, March 31, 2011

Other Stuff I Should Mention pgcrypto cryptography /contrib (export issues) md5, sha1, blowfish, many more Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Other Stuff I Should Mention dblink pgsql -> pgsql connections /contrib (still under development) can have performance issues on large queries make it live in it’s own schema may be replaced by foreign data wrappers Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Other Stuff I Should Mention *-link heterogenous connections for postgresql db specific and db independent options any pl/u language can implement this similar performance issue as dblink dblink-tds, dbi-link, oralink, odbclink Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Other Stuff I Should Mention autonomous logging tool persistent logging for postgresql functions built on top of dblink make it live in it’s own schema https://labs.omniti.com/trac/pgtreats Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Other Stuff I Should Mention check_postgres nagios oriented monitoring script comprehensive list of items for alterting adapted for other systems, some trending http://bucardo.org/check_postgres Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Other Stuff I Should Mention phpPgAdmin web based gui for postgresql remote administration of multiple servers implements much of postgresql functionality http://phppgadmin.sourceforge.net/ Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Other Stuff I Should Mention omnipitr advanced WAL file management scripts pitr archiving, pitr restore complete filesystem based backup can create backups from a slave server https://labs.omniti.com/trac/pgtreats Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Other Stuff I Should Mention books PostgreSQL 9.0 High Performance, by Greg Smith PostgreSQL 9.0 Admin Cookbook, by Simon Riggs Robert Treat | Pro Postgres 9Thursday, March 31, 2011

Other Stuff I Should Mention we’re hiring :-) Database Operations Big Systems High Volume Workloads Keggerator! Robert Treat | Pro Postgres 9Thursday, March 31, 2011

El Fin Thanks For Coming!Thursday, March 31, 2011

http://debop.blogspot.kr	333
http://debop.blogspot.com	15
https://pri.skarven.net	8
http://debop.blogspot.jp	2
http://www.hanrss.com	1

Pro Postgres 9

by Robert Treat

Accessibility

Categories

Upload Details

Usage Rights

5 Embeds 359

Statistics

Pro Postgres 9 Presentation Transcript