Your SlideShare is downloading. ×
Kevin Kempter PostgreSQL Backup and Recovery Methods @ Postgres Open
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Kevin Kempter PostgreSQL Backup and Recovery Methods @ Postgres Open

994
views

Published on

Published in: Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
994
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
23
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Consistent State www.consistentstate.com kevink@consistentstate.com PostgreSQL Backup and Recovery Methods Kevin Kempter Chief Data Architect Wednesday, September 18, 13
  • 2. -or- How NOT to be this guy 2 Wednesday, September 18, 13
  • 3. Session Topics ✓Overview ✓Backup Options ✓Restore Options ✓Point in Time Recovery 3 Wednesday, September 18, 13
  • 4. Overview • Multiple backup methods • Multiple backup file formats • Many recovery choices / options if a pg_restore compatible format is used • PITR • PITR is the base construct for WAL shipping (Warm Standby) 4 Wednesday, September 18, 13
  • 5. PostgreSQL Backup Options ✓pg_dump ✓pg_dumpall 5 Wednesday, September 18, 13
  • 6. pg_dump ✓ Utility to dump a snapshot of a single database ✓ Multiple output options ✓ Non-blocking ✓ Creates a “Consistent” backup - even if the database is in use 6 Wednesday, September 18, 13
  • 7. pg_dump • Syntax • pg_dump [connection-options] [dump-options] [dbname] • Connection Options • -h, --host=HOSTNAME • -p, --port=PORT • -U, --username=NAME • -w, --no-password • -W, --password (should happen automatically) • --role=ROLENAME (do SET ROLE before dump) • Environment Variables • PGDATABASE • PGHOST • PGOPTIONS • PGPORT • PGUSER 7 Wednesday, September 18, 13
  • 8. pg_dump - Common Options • -s [--schema-only] • Dump schema (DDL) only, no data • -a [--data-only] • Dump data only - no DDL • -c [--clean] • Generate drop statements for all created objects • -C [--create] • Generate a “CREATE DATABASE” statement • -n schema [--schema=schema] • Only dump the specified schema, wildcard characters are allowed, also multiple -n’s are allowed 8 Wednesday, September 18, 13
  • 9. pg_dump - Common Options (continued) • -N schema [--exclude-schema=schema] • Exclude specified schema • -F format [--format=format] • Output format • p (Plain) plain sql file #default • c (Custom) custom binary format • t (tar) tar format • d (directory) Creates a directory with one file per table/blob, plus a TOC file in a binary format that pg_restore can read • -o [--oids] • Dump table OID’s • -O [--no-owner] • Do not generate ownership commands 9 Wednesday, September 18, 13
  • 10. pg_dump - Common Options (continued) • -t table [--table=table] • Only dump the specified table, wildcard characters are allowed, also multiple -t’s are allowed, over-rides -n and -N options • -x [--no-privileges] [--no-acl] • Do not dump access privileges • --inserts • Generate INSERT statements • --disable-triggers • disable triggers during restore (for a data only restore) when doing a data-only dump • --lock-wait-timeout=timeout • fail if shared lock on an object cannot be acquired within timeout time 10 Wednesday, September 18, 13
  • 11. pg_dump - Common Options (continued) • -Z 0..9 [--compress=0..9] • Specify compression level for custom format or plain format (not supported for tar format) • -v [--verbose] • -V [--version] 11 Wednesday, September 18, 13
  • 12. pg_dump - Examples $ pg_dump -C --inserts prod1_db > prod1_db.sql Creates a dump of insert statements including a create database statement $ pg_dump --data-only --table=customer -Fc prod1_db > prod1_db.cust.fc.dmp Dump the customer table data (data only) in a custom format from the prod1_db database $ pg_dump -s prod1_db > prod1_db.ddl_only.sql Creates a DDL only dump of the prod1_db database $ pg_dump --schema=gold -Ft prod1_db > prod1_db.gold_schema.dmp Creates a dump of the gold schema in the prod1_db database in a tar format 12 Wednesday, September 18, 13
  • 13. pg_dump Summary 13 Wednesday, September 18, 13
  • 14. pg_dumpall ✓ Utility to dump a snapshot of a full database cluster (or cluster-wide constructs) ✓ Dumps only to plain sql format ✓ Non-blocking ✓ Creates a “Consistent” backup - even if the database is in use 14 Wednesday, September 18, 13
  • 15. pg_dumpall • Syntax • pg_dump [connection-options] [dump-options] • Connection Options • -h, --host=HOSTNAME • -p, --port=PORT • -U, --username=NAME • -w, --no-password • -W, --password (should happen automatically) • --role=ROLENAME (do SET ROLE before dump) • Environment Variables • PGDATABASE • PGHOST • PGOPTIONS • PGPORT • PGUSER 15 Wednesday, September 18, 13
  • 16. pg_dumpall - Common Options • -s [--schema-only] • Dump schema (DDL) only, no data • -a [--data-only] • Dump data only - no DDL • -c [--clean] • Generate drop statements for all created objects • -o [--oids] • Dump table OID’s • -O [--no-owner] • Do not generate ownership commands 16 Wednesday, September 18, 13
  • 17. pg_dumpall - Common Options (continued) • -r [--roles-only] • Dump only CREATE ROLE data • -t [--tablespaces-only] • Dump only CREATE TABLESPACE data • -g [--globals-only] • Dump Global Structures (Roles and Tablespaces) • --no-tablespaces • Do NOT dump CREATE TABLESPACE Data • --inserts • Generate INSERT statements 17 Wednesday, September 18, 13
  • 18. pg_dumpall - Common Options (continued) • --disable-triggers • disable triggers during restore (for a data only restore) when doing a data-only dump • --lock-wait-timeout=timeout • fail if shared lock on an object cannot be acquired within timeout time • -v [--verbose] • -V [--version] 18 Wednesday, September 18, 13
  • 19. pg_dumpall - Examples $ pg_dumpall -g > prod1_db_cluster.global_structures.sql Creates a cluster dump containing only the cluster global structures $ pg_dumpall --tablespaces-only > prod1_db_cluster.tablespaces.sql Dump the cluster tablespaces $ pg_dumpall --no-tablespaces > prod1_db_cluster.no_tablespaces.sql Creates a dump of the cluster without any tablespace references $ pg_dumpall -a > prod1_db_cluster.data_only.sql Creates a dump of the cluster - data only 19 Wednesday, September 18, 13
  • 20. pg_dumpall summary 20 Wednesday, September 18, 13
  • 21. PostgreSQL Restore Options ✓psql ✓pg_restore 21 Wednesday, September 18, 13
  • 22. Restoring with psql $ psql -ef prod1_db.sql > load_db.log 2>&1 $ pg_dump prod1_db | psql -h qa_server $ pg_dumpall -g | psql -h dev_server -p 5433 > load_dev.log 2>&1 $ pg_dump prod1_db | psql -e test_db > load_test_db.log 2>&1 22 Wednesday, September 18, 13
  • 23. pg_restore ✓ Utility to restore a data file created by pg_dump • Works only with non plain text file formats 23 Wednesday, September 18, 13
  • 24. pg_restore • Syntax • pg_restore [connection-option...] [option...] [filename] • Connection Options • -h, --host=HOSTNAME • -p, --port=PORT • -U, --username=NAME • -w, --no-password • -W, --password (should happen automatically) • --role=ROLENAME (do SET ROLE before dump) • Environment Variables • PGDATABASE • PGHOST • PGOPTIONS • PGPORT • PGUSER 24 Wednesday, September 18, 13
  • 25. pg_restore - Common Options • -d dbname [--dbname=dbname] • -C [--create] • Create the specified database before restore • -c [--clean] • Generate drop statements for all created objects • -s [--schema-only] • Dump schema (DDL) only, no data • -a [--data-only] • Dump data only - no DDL 25 Wednesday, September 18, 13
  • 26. pg_restore - Common Options (continued) • -n namespace [--schema=schema] • Restore only objects in the specified schema • -O [--no-owner] • Do not restore ownership of objects • -I index [--index=index] • Restore specified index only • -P function-name(argtype [, ...]) [ --function=function-name(argtype [, ...]) ] • Restore specified function only • -T trigger [--trigger=trigger] • Restore specified trigger only 26 Wednesday, September 18, 13
  • 27. pg_restore - Common Options (continued) • -t table [--table=table] • Restore specified table only • --no-tablespaces • Do not restore any TABLESPACES • -F format [--format=format] • Output format • c (Custom) custom binary format • t (tar) tar format • d (directory) 27 Wednesday, September 18, 13
  • 28. pg_restore - Common Options (continued) • -t table [--table=table] • Restore specified table only • --no-tablespaces • Do not restore any TABLESPACES • -j number-of-jobs [--jobs=number-of-jobs] • Use parallel jobs to perform the restore • --disable-triggers • Disable triggers during the restore (for a data only restore) • -e [--exit-on-error] • Exits upon any error 28 Wednesday, September 18, 13
  • 29. pg_restore - Common Options (continued) • -l [--list] • Create a list (TOC) file • -L list-file [--use-list=list-file] • Restore based on the specified list file • -V [--version] • -v [--verbose] 29 Wednesday, September 18, 13
  • 30. pg_restore - Examples $ pg_restore -a -Fc -d prod2_db prod1_db.fc.dmp Restores data only from a custom formatted file into database prod2_db $ pg_restore -c --schema=gold_partners -v -Ft -d prod2_db prod.tar.dmp Cleans (removes data & structures first) then restores the gold_partners schema from a tar formatted file into the prod2_db database (with verbose output) $ pg_restore --schema-only -d qa1_db -Fc -j 10 prod1_db.fc.dmp Restores the schema only (DDL) from a custom formatted file into the qa1_db database using 10 parallel streams to do the restore 30 Wednesday, September 18, 13
  • 31. Restoring via a list file • pg_restore can create a list file from a pg_dump file • List file will contain one line per needed operation such as: • CREATE TABLE • COPY • CREATE INDEX • List file can be modified as desired to create a custom restore 31 Wednesday, September 18, 13
  • 32. Create a list file from the pg_dump file $ pg_dump -Ft db1 > db1.fc.dmp $ pg_restore -Ft -l db1.dmp > db1.lst 32 Wednesday, September 18, 13
  • 33. Sample list file header ; ; Archive created at Tue Sep 10 09:42:24 2013 ; dbname: testdb ; TOC Entries: 34 ; Compression: -1 ; Dump Version: 1.12-0 ; Format: CUSTOM ; Integer: 4 bytes ; Offset: 8 bytes ; Dumped from database version: 9.2.4 ; Dumped by pg_dump version: 9.2.4 ; ; 33 Wednesday, September 18, 13
  • 34. Sample list file contents ; Selected TOC Entries: ; 1981; 1262 16386 DATABASE - testdb_old postgres 6; 2615 2200 SCHEMA - public postgres 1982; 0 0 COMMENT - SCHEMA public postgres 1983; 0 0 ACL - public postgres 181; 3079 11730 EXTENSION - plpgsql 1984; 0 0 COMMENT - EXTENSION plpgsql 168; 1259 16411 TABLE public testdb_jasper_metrics_tables postgres 169; 1259 16414 TABLE public testdb_jasper_metrics_tables_tmp1 postgres 170; 1259 16417 TABLE public testdb_metrics_activity postgres 171; 1259 16423 TABLE public testdb_metrics_database postgres 172; 1259 16426 TABLE public testdb_postgres_metrics_bgwriter postgres 173; 1259 16429 TABLE public testdb_postgres_metricsio_user_tables postgres 174; 1259 16432 TABLE public testdb_testdb_gf_metrics_tables postgres 175; 1259 16435 TABLE public testdb_testdb_transition_metrics_tables postgres 176; 1259 16438 TABLE public testdb_testdb_transition_metrics_tables_tmp1 postgres 177; 1259 16441 TABLE public testdb_testdb_transition_metrics_tables_tmp2 postgres 34 Wednesday, September 18, 13
  • 35. Sample list file contents (cont) 178; 1259 16444 TABLE public idle_conn_metrics postgres 179; 1259 16447 TABLE public total_conn_metrics postgres 180; 1259 16450 TABLE public waiting_conn_metrics postgres 1964; 0 16411 TABLE DATA public testdb_jasper_metrics_tables postgres 1965; 0 16414 TABLE DATA public testdb_jasper_metrics_tables_tmp1 postgres 1966; 0 16417 TABLE DATA public testdb_metrics_activity postgres 1967; 0 16423 TABLE DATA public testdb_metrics_database postgres 1968; 0 16426 TABLE DATA public testdb_postgres_metrics_bgwriter postgres 1969; 0 16429 TABLE DATA public testdb_postgres_metricsio_user_tables postgres 1970; 0 16432 TABLE DATA public testdb_testdb_gf_metrics_tables postgres 1971; 0 16435 TABLE DATA public testdb_testdb_transition_metrics_tables postgres 1972; 0 16438 TABLE DATA public testdb_testdb_transition_metrics_tables_tmp1 postgres 1973; 0 16441 TABLE DATA public testdb_testdb_transition_metrics_tables_tmp2 postgres 1974; 0 16444 TABLE DATA public idle_conn_metrics postgres 1975; 0 16447 TABLE DATA public total_conn_metrics postgres 1976; 0 16450 TABLE DATA public waiting_conn_metrics postgres 35 Wednesday, September 18, 13
  • 36. Restore via list file - example $ pg_dump -Ft prod_db > prod_db.fc.dmp $ pg_restore -Ft -l prod_db.dmp > prod_db.lst $ createdb qadb3 Edit prod_db.lst as needed / desired $ pg_restore -L prod_db.lst -Ft -d qadb3 prod_db.dmp 36 Wednesday, September 18, 13
  • 37. Restore Options Summary 37 Wednesday, September 18, 13
  • 38. Point In Time Recovery (PITR) 38 Wednesday, September 18, 13
  • 39. PITR Overview • PITR Backups • Archiving the WAL segments • Making Base Backups • PITR Recovery • Restore the last Base Backup • Prepare the recovered system data directory • Create a recovery.conf file • Start the postmaster 39 Wednesday, September 18, 13
  • 40. PITR Setup • Enable / set the following parameters in the postgresql.conf file: • wal_level = archive (or hot_standby) • archive_mode = on • archive_command = 'valid archive command' Can be any valid shell command (including scripts) • archive_timeout = [timeout] • Special archive_command (and recovery.conf file) tags • %p = full path (absolute path) and the file name of the WAL segment to be archived • %f = only the file name of the WAL segment • %% = insert a % character in the command string. 40 Wednesday, September 18, 13
  • 41. PITR Example • Enable / set the following parameters in the postgresql.conf file: • wal_level = archive • archive_mode = on • archive_command = 'cp %p /stage/wal/%f' Can be any valid shell command (including scripts) • archive_timeout = 0 • mkdir /stage/wal • chown postgres:postgres /stage/wal • Re-start the Server 41 Wednesday, September 18, 13
  • 42. PITR Example - create transactions • Execute SQL commands / transactions • Enable access, turn on applications, etc • This should force the creation of multiple archived WAL files in the /stage/wal directory • WAL segments are copied when: • The WAL segment is full (see checkpoint_segments) • Number of seconds specified in archive_timeout has passed 42 Wednesday, September 18, 13
  • 43. PITR Example - create base backup • Execute pg_start_backup $ psql pitr_test # select pg_start_backup ('tag') ; • Archive the cluster data directory (and any related tablespaces) $ tar -czvf /backups/pitr/<date>.data.tar.gz ./data rsync other copy methods • Execute pg_stop_backup $ psql pitr_test # select pg_stop_backup () ; 43 Wednesday, September 18, 13
  • 44. PITR Example - create more transactions • Execute SQL commands / transactions • The application, user connections, etc will continue to generate transactions (and archived WAL segments) • Verify the creation of additional archived WAL files in the /stage/wal directory 44 Wednesday, September 18, 13
  • 45. PITR - recovery.conf file (common options) Recovery settings are placed in the file 'recovery.conf' • restore_command (string) must return nonzero • restore_command = 'cp /stage/wal/%f %p' • restore_command = '/usr/local/bin/restore_shell.sh %p %f' 45 Wednesday, September 18, 13
  • 46. PITR - recovery.conf file (common options) recovery_target_time (timestamp) • specifies the time stamp up to which recovery will proceed. • recovery_target_time and recovery_target_xid are mutually exclusive • The default is to recover to the end of the WAL log. 46 Wednesday, September 18, 13
  • 47. PITR Recovery (1) If available copy the original cluster data directory to an alternate location if space is an issue at least copy the old pg_xlog dir it may contain additional unarchived WAL segments (2) Ensure the postmaster is not running 47 Wednesday, September 18, 13
  • 48. PITR Recovery If your backup was an rsync to a second server then skip steps 3 & 4 (3) Remove the cluster data directory and any tablespace directories (4) Restore your last system backup • make sure permissions are retained • If you're using tablespaces then verify that the symbolic links in pg_tblspc/ were restored 48 Wednesday, September 18, 13
  • 49. PITR Recovery (5) Remove any wal segments from the pg_xlog dir that were restored from the backup If you didn't backup pg_xlog then create it, make sure you re-establish it as a symbolic link if needed If needed also re-create the pg_xlog/archive_status directory (6) Copy the files from the original pg_xlog dir (if available) into the new pg_xlog dir do a copy as opposed to a move in case you need to start over 49 Wednesday, September 18, 13
  • 50. PITR Recovery (7) Create a recovery command (recovery.conf) in the cluster data directory. (8) [Optional] Temporarily modify pg_hba.conf to prevent ordinary users from connecting until the recovery is complete (9) Start the server. The server will go into recovery mode via the recovery.conf file. Once the recovery is complete then the server will become available and rename the recovery.conf file to recovery.done If an error interrupts the recovery (or stops the server) then simply re-starting the server will restart the recovery 50 Wednesday, September 18, 13
  • 51. PITR Recovery (10) Verify the recovery. If the database was not recovered properly (or to a state that you desire) then go back to step 1 (11) restore the pg_hba.conf to its original state and run a pg_ctl reload (if it was modified for the recovery) 51 Wednesday, September 18, 13
  • 52. Questions? Thank You! 52 Wednesday, September 18, 13