Postgres backup-and-recovery2.pptx

PostgreSQL Backup and
Recovery
www.consistentstate.com
kevink@consistentstate.com
PG West, 2009
by Kevin Kempter
Except where otherwise noted, this work is licensed under
http://creativecommons.org/licenses/by/3.0/

Session Topics
PG West, 2009
by Kevin Kempter
●
●
●
●
●
Overview
Backup Options
Restore Options
Point in Time Recovery (PITR)
Warm Standby

Backup Options
PG West, 2009
by Kevin Kempter
●
●
pg_dump
pg_dumpall

pg_dump
PG West, 2009
by Kevin Kempter
●
●
●
●
●
A PostgreSQL database backup/dump utility
Creates consistent backups (even if the db is in use)
Non-blocking
Multiple Output File Formats
Highly Flexible Options

pg_dump
PG West, 2009
by Kevin Kempter
● Syntax:
pg_dump [options] dbname
● Connection Options:
pg_dump -h <host> -p <port> -U <user> -W
pg_dump --host=<host> --port=<port> --username=<user> --password
● Environment Variables:
PGDATABASE PGHOST PGPORT PGUSER

pg_dump
PG West, 2009
by Kevin Kempter
●
●
●
●
●
-s (--schema-only)
-t (--table=<tablename>)
-T (--exclude-table=<tablename>)
-v (--verbose)
-F <format> (--format=<format>)
● Common Options:
●
●
●
●
●
●
-a (--data-only)
-c (--clean)
-C (--create)
-d (--inserts)
-N (--exclude-schema=<schema>)
-n (--schema=<schema>)

pg_dump
PG West, 2009
by Kevin Kempter
● File Format Options:
●
●
●
●
●
●
Plain
Creates a plain text SQL file (default)
-F p (--format=plain)
Custom
Create a custom compressed format (compatible with pg_restore)
-F c (--format=custom)
Tar
Creates a tar format file (also compatible with pg_restore)
-F t (--format=tar)

pg_dump Examples
PG West, 2009
by Kevin Kempter
● $ pg_dump -C --inserts prod1_db > prod1_db.sql
Creates a dump of insert statements including a create database statement
● $ pg_dump --data-only --table=customer -F c prod1_db > prod1_db.fc.dmp
Dump the customer table in a custom format from the prod1_db database
● $ pg_dump -S prod1_db > prod1_db.ddl_only.sql
Creates a DDL only dump of the prod1_db database
● $ pg_dump --schema=gold -F t prod1_db > prod1_db.gold_schema.dmp
Creates a dump of the gold schema in the prod1_db database in a tar format

pg_dumpall
PG West, 2009
by Kevin Kempter
●
●
●
●
●
A PostgreSQL cluster (instance) backup/dump utility
Creates consistent backups (even if the db is in use)
Non-blocking
Multiple Output File Formats
Highly Flexible Options

pg_dumpall
PG West, 2009
by Kevin Kempter
● Syntax:
pg_dumpall [options]

pg_dumpall
PG West, 2009
by Kevin Kempter
●
●
●
●
●
●
-r (--roles-only)
-s (--schema-only)
-S (--superuser=<username>)
-v (--verbose)
-t (--tablespaces-only)
--disable-triggers
● Common Options:
●
●
●
●
●
●
-a (--data-only)
-c (--clean)
-D (--column-inserts) (--attribute-inserts)
-d (--inserts)
-g (--globals-only)
-o (--oids)

pg_dumpall Examples
PG West, 2009
by Kevin Kempter
● $ pg_dumpall -g > prod1_db.global_structures.sql
Creates a cluster dump containing only the cluster global structures
● $ pg_dumpall --tablespaces-only > prod1_db.tablespaces.sql
Dump the cluster tablespaces
● $ pg_dumpall -r > prod1_db.roles_only.sql
Creates a dump of only the cluster roles
● $ pg_dumpall -o -S gold_user > prod1_db.oids.dmp
Creates a dump of the cluster including oid's as the superuser 'gold_user'

Restore Options
PG West, 2009
by Kevin Kempter
●
●
pg_restore
psql

pg_restore
PG West, 2009
by Kevin Kempter
● Syntax:
pg_restore [options] [filename]

pg_restore
PG West, 2009
by Kevin Kempter
●
●
●
●
●
●
●
●
--disable-triggers
-P function-name (args)
--function=function-name(args)
-s (--schema-only)
-S (--superuser=<username>)
-v (--verbose)
-t <table> (--table=<table>)
-T <trigger> (--trigger=<trigger>)
-F <format> (--format=<format>)
(c or t only)
●
●
●
●
●
●
●
●
●
●
Common Options:
-d <dbname> (--dbname=<dbname>)
-a (--data-only)
-c (--clean)
-C (--create)
-i (--ignore-version)
-I (--index=<index>)
-l (--list)
-L <filename> (--use-list=<list-file>)
-n (--schema=<schema-name>)

pg_restore Examples
PG West, 2009
by Kevin Kempter
● $ pg_restore -a -F c -d prod2_db prod1_db.fc.dmp
Restores data only from a custom formatted file into database prod2_db
● $ pg_restore -c --schema=gold_partners -v -F t -d prod2_db prod_dump.tar.dmp
Cleans (removes data & structures first) then restores the gold_partners schema
From a tar formatted file into the prod2_db database
● $ pg_restore --schema-only -d qa1_db -F c prod1_db.fc.dmp
Restores the schema only (DDL) from a custom formatted file
into the qa1_db database

Create a TOC file from a dump
PG West, 2009
by Kevin Kempter
●
●
$ pg_dump -Fc db1 > db1.fc.dmp
$ pg_restore -Fc -l db1.dmp > db1.lst

Examining the TOC File
The TOC Header
PG West, 2009
by Kevin Kempter
; Archive created at Sat Mar 28 15:02:28 2009
; dbname: dbmon
; TOC Entries: 16
; Compression: 0
; Dump Version: 1.10-0
; Format: TAR
; Integer: 4 bytes
; Offset: 8 bytes
; Dumped from database version: 8.3.5
; Dumped by pg_dump version: 8.3.5
;

Examining the TOC File
The TOC File Contents
PG West, 2009
by Kevin Kempter
; Selected TOC Entries:
;
3; 2615 2200 SCHEMA - public postgres
1745; 0 0 COMMENT - SCHEMA public postgres
1746; 0 0 ACL - public postgres
1469; 1259 352554 TABLE public dbmon_thresh postgres
1467; 1259 352545 TABLE public vac_density_hist postgres
1468; 1259 352552 SEQUENCE public dbmon_thresh_dbmon_thresh_id_seq postgres
1747; 0 0 SEQUENCE OWNED BY public dbmon_thresh_dbmon_thresh_id_seq postgres
1748; 0 0 SEQUENCE SET public dbmon_thresh_dbmon_thresh_id_seq postgres
1736; 2604 352557 DEFAULT public dbmon_thresh_id postgres
1741; 0 352554 TABLE DATA public dbmon_thresh postgres
1740; 0 352545 TABLE DATA public vac_density_hist

Restore via a TOC File
PG West, 2009
by Kevin Kempter
●
●
$ createdb dbmon2
$ pg_restore -L db.lst -Ft dbmon-dg.dmp -d dbmon2

Restoring with psql
PG West, 2009
by Kevin Kempter
●
●
●
●
●
$ pg_dump prod1_db > prod1_db.sql
$ psql -ef prod1_db.sql > load_db.log 2>&1
$ pg_dump prod1_db | psql -h qa_server
$ pg_dumpall -g | psql -h dev_server -p 5433 > load_dev.log 2>&1
$ pg_dump prod1_db | psql -e test_db > load_test_db.log 2>&1

Backup/Restore
PG West, 2009
by Kevin Kempter
Summary

PITR
PG West, 2009
by Kevin Kempter
●
●
PostgreSQL has built-in facilities for Point in Time
Recovery
PITR Backups
–
–
Archiving the WAL segments
Making Base Backups
● PITR Recovery
–
–
–
–
Restore the last Base Backup
Prepare the recovered system data directory
Create a recovery.conf file
Start the postmaster

PITR Backups
Enable WAL Archiving
PG West, 2009
by Kevin Kempter
● Enable / set the following parameters in the postgresql.conf file:
–
–
archive_mode = on
archive_command = 'cp %p /stage/wal/%f'
Can be any valid shell command (including scripts)
– archive_timeout = 0
● Special tags
–
–
– %p = full path (absolute path) and the filename of the WAL
segment to be archived
%f = only the filename of the WAL segment
%% = insert a % character in the command string.

PITR Backups
Enable WAL Archiving - Example
PG West, 2009
by Kevin Kempter
●
●
mkdir /stage/wal
chown postgres:postgres /stage/wal
(or other postgres cluster owner)
● Re-start the Server

PITR Backups
Create Transactions - Example
PG West, 2009
by Kevin Kempter
● Execute SQL commands / transactions
– Enable access, turn on applications, etc
●
This should force the creation of multiple archived WAL files in
the /stage/wal directory
● WAL segments are copied when:
–
–
The WAL segment is full (see checkpoint_segments)
Number of seconds specified in archive_timeout has passed

PITR Backups
Create Base Backup - Example
PG West, 2009
by Kevin Kempter
● Execute pg_start_backup
–
–
$ psql pitr_test
# select pg_start_backup ('tag')
● Archive the cluster data directory (and any related tablespaces)
–
–
–
$ tar -czvf /backups/pitr/<date>.data.tar.gz ./data
$ rsync
$ other copy methods
● Execute pg_stop_backup
–
–
$ psql pitr_test
# select pg_stop_backup ()

PITR Backups
Create More Transactions - Example
PG West, 2009
by Kevin Kempter
● Execute SQL commands / transactions
– The application, user connections, etc will continue to
generate transactions (and archived WAL segments)
● Verify the creation of additional archived WAL files in
the /stage/wal directory

PITR Recovery
PG West, 2009
by Kevin Kempter
(1)If available copy the original cluster data directory to
an alternate location
if space is an issue at least copy the old pg_xlog dir
it may contain additional unarchived WAL segments
(2) Ensure the postmaster is not running

PITR Recovery
PG West, 2009
by Kevin Kempter
If your backup was an rsync to a second server
then skip steps 3 & 4
(3)Remove the cluster data directory and any tablespace
directories
(4) Restore your last system backup
– make sure permissions are retained
– If you're using tablespaces then verify that the
symbolic links in pg_tblspc/ were restored

PITR Recovery
PG West, 2009
by Kevin Kempter
(5)Remove any wal segments from the pg_xlog dir that
were restored from the backup
– If you didn't backup pg_xlog then create it, make
sure you re-establish it as a symbolic link if needed
(if you had moved the tx logs to another disk)
– If needed also re-create the
pg_xlog/archive_status directory
(6)Copy the files from the original pg_xlog dir (if
available) into the new pg_slog dir (do a copy as
opposed to a move in case you need to start over).

PITR Recovery
PG West, 2009
by Kevin Kempter
(7)Create a recovery command file named recovery.conf
in the cluster data directory.
(8)[Optional] Temporarily modify pg_hba.conf to prevent
ordinary users from connecting until the recovery is
complete
(9) Start the server.
The server will go into recovery mode via the recovery.conf file.
Once the recovery is complete then the server will become
available and rename the recovery.conf file to recovery.done
If an error interrupts the recovery (or stops the server) then simply
re-starting the server will restart the recovery

PITR Recovery
PG West, 2009
by Kevin Kempter
(10) Verify the recovery.
If the database was not recovered properly (or to a state that you
desire) then go back to step 1
(11)restore the pg_hba.conf to its original state and run a
pg_ctl reload (if it was modified for the recovery)

PITR Recovery
recovery.conf
PG West, 2009
by Kevin Kempter
● Recovery settings are placed in the file 'recovery.conf'
● restore_command (string)
must return nonzero
–
–
restore_command = 'cp /stage/wal/%f %p'
restore_command = '/usr/local/bin/restore_shell.sh %p %f'

PITR Recovery
recovery.conf
PG West, 2009
by Kevin Kempter
● recovery_target_time (timestamp)
–
–
–
specifies the time stamp up to which recovery will proceed.
recovery_target_time and recovery_target_xid are mutually
exclusive
The default is to recover to the end of the WAL log.

PITR Recovery
recovery.conf
PG West, 2009
by Kevin Kempter
● recovery_target_xid (string)
–
–
–
specifies the transaction ID up to which recovery will proceed.
Although transaction IDs are assigned sequentially at
transaction start, transactions can complete in a different
numeric order.
All transactions committed before the specified XID will be
recovered

PITR Recovery
recovery.conf
PG West, 2009
by Kevin Kempter
● recovery_target_timeline (string)
– Specifies recovering into a particular timeline for complex re-
recovery scenarios. See the online docs for more info.
● log_restartpoints (boolean)
– If true, each restart point will be logged. This can be helpful to
track the progress of a long recovery. The default is false.

PITR Recovery
PG West, 2009
by Kevin Kempter
Summary

Warm Standby
PG West, 2009
by Kevin Kempter
●
●
The PostgreSQL built-in facilities for PITR can be
leveraged to create a 'warm standby' server
A Warm Standby server consists of the following:
– A server with a running postmaster (in
recovery mode)
– A recovery command that keeps the server
in recovery mode
– A continous stream of archived WAL
segments from the “master”
– A failover mechanism

Warm Standby
Overview
PG West, 2009
by Kevin Kempter
●
●
●
●
●
Configure WAL archiving on the master
Make the WAL segments available to the warm standby
server
Setup recovered cluster on the warm standby server
Setup the recovery command scripts
Setup failover mechanism

Warm Standby
Setup WAL Archiving
PG West, 2009
by Kevin Kempter
● Enable WAL archiving in the postgresql.conf file (on the
master)
–
–
–
archive_mode = on
archive_command = 'cp %p /stage/wal/%f'
archive_timeout = 0
● Create a directory for the archived WAL segments
– $ mkdir /stage/wal
– $ chmod postgres:postgres /stage/wal

Warm Standby
Make WAL segments available to the warm standby server
PG West, 2009
by Kevin Kempter
●
●
●
●
NFS (not the best choice)
rsync
scp
Other methods

Warm Standby
Setup recovered cluster on the warm standby server
PG West, 2009
by Kevin Kempter
Use the same method as described in the PITR
recovery steps

Warm Standby
Setup recovery command scripts
PG West, 2009
by Kevin Kempter
● Pseudocode for a suitable restore_command is:
triggered = false;
while (!NextWALFileReady() && !triggered)
{
sleep(100000L); /* wait for ~0.1 sec */
if (CheckForExternalTrigger())
triggered = true;
}
if (!triggered)
CopyWALFileForRecovery();
http://www.postgresql.org/docs/8.3/static/warm-standby.html

Warm Standby
Setup recovery command scripts – pg_standby
* pg_standby.c
* Production-ready example of how to create a Warm Standby
* database server using continuous archiving as a
* replication mechanism
*
* We separate the parameters for archive and next WAL file
* so that we can check the archive exists, even if the
* WAL file doesn't (yet).
*
* This program will be executed once in full for each file
* requested by the warm standby server.
*
* It is designed to cater to a variety of needs, as well
* providing a customizable section.
*
* Original author:
Simon Riggs simon@2ndquadrant.com
* Current maintainer: Simon Riggs
●
PG West, 2009
by Kevin Kempter
PostgreSQL contrib pg_standby

Warm Standby
Setup recovery command scripts – recovery.conf file
PG West, 2009
by Kevin Kempter
● Using a shell script
– restore_command = '/home/postgres/wst/bin/shell_script.sh %p %f'
● Using the pg_standby contrib
– restore_command =
pg_standby [options] archivelocation %f %p %r [RESTARTWALFILE]
%r = “The size of the WAL archive can be minimized by using the %r option of the
restore_command. This option specifies the last archive file name that needs
to be kept to allow the recovery to restart correctly. This can be used to truncate
the archive once files are no longer required, if the archive is writable from the
standby server.” **
** http://www.postgresql.org/docs/8.3/static/warm-standby.html

Warm Standby
** http://www.postgresql.org/docs/8.3/static/pgstandby.html
PG West, 2009
by Kevin Kempter
● pg_standby [options] **
Option Default Use cp or copy command to restore WAL files from archive.
-c yes Remove files from archivelocation so that no more than this many WAL files
before the current one are kept in the archive. Zero (the default) means not to
remove any files from archivelocation.
-d no Use ln command to restore WAL files from archive. Link is more efficient than
copy, but the default is copy since link will not work in all scenarios
-k <numfiles> 0 Remove files from archivelocation so that no more than this many WAL
files before the current one are kept in the archive. Zero (the default)
means not to remove any files from archivelocation. This parameter will
be silently ignored if restartwalfile is specified
-l no Use ln command to restore WAL files from archive. Link is more
efficient than copy, but the default is copy since link will not work in all
scenarios.

Warm Standby
** http://www.postgresql.org/docs/8.3/static/pgstandby.html
PG West, 2009
by Kevin Kempter
● pg_standby [options] **
Option Default Use cp or copy command to restore WAL files from archive.
-r <max retries> 3 Set the maximum number of times to retry the copy or link command if it
fails. After each failure, we wait for sleeptime * num_retries so that the
wait time increases progressively.
-s <sleep time> 5 Set the number of seconds (up to 60) to sleep between tests to see if
the WAL file to be restored is available in the archive yet.
-t <trigger file> none Specify a trigger file whose presence should cause recovery to end
whether or not the next WAL file is available.
-w <max wait time> 0 Set the maximum number of seconds to wait for the next WAL file, after
which recovery will end and the standby will come up. A setting of zero
(the default) means wait forever.

Warm Standby
PG West, 2009
by Kevin Kempter
Summary

End
PG West, 2009
by Kevin Kempter

Postgres backup-and-recovery2.pptx

More Related Content

What's hot

Similar to Postgres backup-and-recovery2.pptx

Recently uploaded

Postgres backup-and-recovery2.pptx