SlideShare a Scribd company logo
1 of 34
Download to read offline
Save your data with
pgBackRest
Stefan Fercot
12 July 2018
1
Who Am I?
Stefan Fercot
aka. pgstef
PostgreSQL user since 2010
involved in the community since 2016
@dalibo since 2017
2
Dalibo
Services
Support Training Advice
Based in France
Contributing to PostgreSQL community
We’re hiring!
3 . 1
Introduction
4
Write Ahead Log (WAL)
transactions written sequentially
considered committed when flushed to disk
WAL replay a er a crash
make the database consistent
5
Point-In-Time Recovery (PITR)
combine
file-system-level backup
continuous archiving of the WAL files
restore the file-system-level backup and replay the
archived WAL files
not mandatory to replay the WAL entries all the way to the
end
6
How To Do It? (1)
postgresql.conf
archive_mode
archive_command
archive_timeout
7
How To Do It? (2)
pg_start_backup()
file-system-level backup : tar, rsync,…
pg_stop_backup()
8
How To Do It? (3)
so many ways to get that wrong
what about backup retention?
…
9
What Is pgBackRest?
aims to be a simple, reliable backup and restore system
developed by David Steele & Stephen Frost (CrunchyData)
Perl & C
MIT license
10
Main Features
custom protocol
local or remote operation (via SSH)
multi-process
full/differential/incremental backup
backup rotation and archive expiration
parallel, asynchronous WAL push and get
Amazon S3 support
encryption
…
11
Installation
Use the PGDG repository, Luke!
yum / apt-get install pgbackrest
1 package with some (~ 40) dependencies
12
Configuration
/etc/pgbackrest.conf , example :
main configuration in the [global] part
each PostgreSQL cluster to backup has its own
configuration, called stanza
[global]
repo1-path=/var/lib/pgsql/10/backups
log-level-console=info
[my_stanza]
pg1-path=/var/lib/pgsql/10/data
13
Global Section - some examples
process-max: max processes to use for compress/transfer
repo1-path: path where backups and archive are stored
repo1-cipher-pass: passphrase used to encrypt/decrypt
files of the repository
repo1-retention-full: number of full backups to retain
repo1-retention-diff: number of differential backups to
retain
repo1-host: repository host when operating remotely via
SSH
repo1-host-user: repository host user when repo1-host is
set
…
14
Stanza Section
pg1-path: PostgreSQL data directory
pg1-port
any global configuration can be overridden
…
15
PostgreSQL Configuration
archive_mode = on
wal_level = replica
archive_command = 'pgbackrest --stanza=my_stanza archive-push %p'
16
Initialize The Stanza
create the stanza
check the configuration and if archiving is working
# sudo -u postgres pgbackrest --stanza=my_stanza stanza-create
# sudo -u postgres pgbackrest --stanza=my_stanza check
17
Perform a Backup
supported types: incr, diff, full
# sudo -u postgres pgbackrest --stanza=my_stanza --type=full backup
18
Backup Information
# sudo -u postgres pgbackrest --stanza=my_stanza info
stanza: my_stanza
status: ok
db (current)
wal archive min/max (10-1):
000000010000000000000001 / 000000010000000000000003
full backup: 20180620-180524F
timestamp start/stop: 2018-06-20 18:05:24 / 2018-06-20 18:05:39
wal start/stop: 000000010000000000000003 / 000000010000000000000003
database size: 23.2MB, backup size: 23.2MB
repository size: 2.7MB, repository backup size: 2.7MB
19
Restore a Backup
options
--delta
--target
…
# sudo -u postgres pgbackrest --stanza=ma_stanza restore
20
Demo - Point-in-Time
Recovery
21
Step 1: Backup
# sudo -u postgres pgbackrest --stanza=my_stanza --type=full backup
P00 INFO: backup command begin 2.03:
--pg1-path=/var/lib/pgsql/10/data --repo1-path=/var/lib/pgsql/10/backups
--stanza=my_stanza --type=full
P00 INFO: execute non-exclusive pg_start_backup()
with label "pgBackRest backup started at 2018-06-28 16:53:20":
backup begins after the next regular checkpoint completes
P00 INFO: backup start archive = 000000020000000000000019, lsn = 0/19000060
P00 INFO: full backup size = 23.2MB
P00 INFO: execute non-exclusive pg_stop_backup() and
wait for all WAL segments to archive
P00 INFO: backup stop archive = 000000020000000000000019, lsn = 0/19000130
P00 INFO: new backup label = 20180628-165320F
P00 INFO: backup command end: completed successfully
P00 INFO: expire command begin
P00 INFO: expire command end: completed successfully
22
Step 2: Create Important Data
# sudo -iu postgres psql -c " 
begin; 
create table important_table (message text); 
insert into important_table values ('Important Data'); 
commit; 
select * from important_table;"
message
----------------
Important Data
(1 row)
23
Step 3: Get Current Time
# sudo -iu postgres psql -Atc "select current_timestamp"
2018-06-28 16:54:22.221035+02
24
Step 4: Ooops…
# sudo -iu postgres psql -c " 
begin; 
drop table important_table; 
commit; 
select * from important_table;"
ERROR: relation "important_table" does not exist
LINE 1: ...drop table important_table; commit; select * from important_...
^
25
Step 5: Stop PostgreSQL
# sudo -iu postgres psql -c "select pg_switch_wal()"
# systemctl stop postgresql-10.service
26
Step 6: Restore
# sudo -u postgres pgbackrest --stanza=my_stanza --delta
--type=time --target="2018-06-28 16:54:22.221035+02" --target-action=promote
restore
P00 INFO: restore command begin 2.03: --delta
--pg1-path=/var/lib/pgsql/10/data --repo1-path=/var/lib/pgsql/10/backups
--stanza=my_stanza --target="2018-06-28 16:54:22.221035+02"
--target-action=promote --type=time
P00 INFO: restore backup set 20180628-165320F
P00 INFO: remove invalid files/paths/links from /var/lib/pgsql/10/data
P00 INFO: cleanup removed 22 files
P00 INFO: write /var/lib/pgsql/10/data/recovery.conf
P00 INFO: restore global/pg_control
(performed last to ensure aborted restores cannot be started)
P00 INFO: restore command end: completed successfully
27
Step 7: Check recovery.conf
# cat /var/lib/pgsql/10/data/recovery.conf
restore_command = 'pgbackrest --stanza=my_stanza archive-get %f "%p"'
recovery_target_time = '2018-06-28 16:54:22.221035+02'
recovery_target_action = 'promote'
28
Step 8: Start PostgreSQL
# systemctl start postgresql-10.service
# cat /var/lib/pgsql/10/data/log/*
LOG: recovery stopping before commit of transaction 561,
time 2018-06-28 16:54:59.481247+02
LOG: redo done at 0/1A01E590
LOG: last completed transaction was at log time 2018-06-28 16:54:13.370025+02
29
Step 9: Check the data
# sudo -iu postgres psql -c "select * from important_table"
message
----------------
Important Data
(1 row)
30
Streaming Replication
/etc/pgbackrest.conf
recovery-option=primary_conninfo=db.mydomain.com
recovery-option=standby_mode=on
…
31
Conclusion
test it,
use it!
32
Where
official website:
code:
rpm and deb: in the PGDG repositories!
user guide:
support:
https://pgbackrest.org
https://github.com/pgbackrest/pgbackrest
https://pgbackrest.org/user-guide.html
https://github.com/pgbackrest/pgbackrest/issues
33
Thank you for your
attention!
34

More Related Content

What's hot

Percona XtraDB Cluster ( Ensure high Availability )
Percona XtraDB Cluster ( Ensure high Availability )Percona XtraDB Cluster ( Ensure high Availability )
Percona XtraDB Cluster ( Ensure high Availability )Mydbops
 
PGConf.ASIA 2017 Logical Replication Internals (English)
PGConf.ASIA 2017 Logical Replication Internals (English)PGConf.ASIA 2017 Logical Replication Internals (English)
PGConf.ASIA 2017 Logical Replication Internals (English)Noriyoshi Shinoda
 
Troubleshooting PostgreSQL Streaming Replication
Troubleshooting PostgreSQL Streaming ReplicationTroubleshooting PostgreSQL Streaming Replication
Troubleshooting PostgreSQL Streaming ReplicationAlexey Lesovsky
 
PostgreSQL Administration for System Administrators
PostgreSQL Administration for System AdministratorsPostgreSQL Administration for System Administrators
PostgreSQL Administration for System AdministratorsCommand Prompt., Inc
 
Linux tuning to improve PostgreSQL performance
Linux tuning to improve PostgreSQL performanceLinux tuning to improve PostgreSQL performance
Linux tuning to improve PostgreSQL performancePostgreSQL-Consulting
 
How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015
How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015
How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015PostgreSQL-Consulting
 
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...ScaleGrid.io
 
Postgresql Database Administration Basic - Day1
Postgresql  Database Administration Basic  - Day1Postgresql  Database Administration Basic  - Day1
Postgresql Database Administration Basic - Day1PoguttuezhiniVP
 
PostgreSQL Streaming Replication Cheatsheet
PostgreSQL Streaming Replication CheatsheetPostgreSQL Streaming Replication Cheatsheet
PostgreSQL Streaming Replication CheatsheetAlexey Lesovsky
 
Why we love pgpool-II and why we hate it!
Why we love pgpool-II and why we hate it!Why we love pgpool-II and why we hate it!
Why we love pgpool-II and why we hate it!PGConf APAC
 
Patroni: Kubernetes-native PostgreSQL companion
Patroni: Kubernetes-native PostgreSQL companionPatroni: Kubernetes-native PostgreSQL companion
Patroni: Kubernetes-native PostgreSQL companionAlexander Kukushkin
 
Patroni - HA PostgreSQL made easy
Patroni - HA PostgreSQL made easyPatroni - HA PostgreSQL made easy
Patroni - HA PostgreSQL made easyAlexander Kukushkin
 
OpenGurukul : Database : PostgreSQL
OpenGurukul : Database : PostgreSQLOpenGurukul : Database : PostgreSQL
OpenGurukul : Database : PostgreSQLOpen Gurukul
 
Postgres connections at scale
Postgres connections at scalePostgres connections at scale
Postgres connections at scaleMydbops
 
High Availability and Disaster Recovery in PostgreSQL - EQUNIX
High Availability and Disaster Recovery in PostgreSQL - EQUNIXHigh Availability and Disaster Recovery in PostgreSQL - EQUNIX
High Availability and Disaster Recovery in PostgreSQL - EQUNIXJulyanto SUTANDANG
 
PGConf.ASIA 2019 Bali - Setup a High-Availability and Load Balancing PostgreS...
PGConf.ASIA 2019 Bali - Setup a High-Availability and Load Balancing PostgreS...PGConf.ASIA 2019 Bali - Setup a High-Availability and Load Balancing PostgreS...
PGConf.ASIA 2019 Bali - Setup a High-Availability and Load Balancing PostgreS...Equnix Business Solutions
 
PostgreSQL continuous backup and PITR with Barman
 PostgreSQL continuous backup and PITR with Barman PostgreSQL continuous backup and PITR with Barman
PostgreSQL continuous backup and PITR with BarmanEDB
 
Wars of MySQL Cluster ( InnoDB Cluster VS Galera )
Wars of MySQL Cluster ( InnoDB Cluster VS Galera ) Wars of MySQL Cluster ( InnoDB Cluster VS Galera )
Wars of MySQL Cluster ( InnoDB Cluster VS Galera ) Mydbops
 
Deep dive into PostgreSQL statistics.
Deep dive into PostgreSQL statistics.Deep dive into PostgreSQL statistics.
Deep dive into PostgreSQL statistics.Alexey Lesovsky
 
Logical Replication in PostgreSQL
Logical Replication in PostgreSQLLogical Replication in PostgreSQL
Logical Replication in PostgreSQLEDB
 

What's hot (20)

Percona XtraDB Cluster ( Ensure high Availability )
Percona XtraDB Cluster ( Ensure high Availability )Percona XtraDB Cluster ( Ensure high Availability )
Percona XtraDB Cluster ( Ensure high Availability )
 
PGConf.ASIA 2017 Logical Replication Internals (English)
PGConf.ASIA 2017 Logical Replication Internals (English)PGConf.ASIA 2017 Logical Replication Internals (English)
PGConf.ASIA 2017 Logical Replication Internals (English)
 
Troubleshooting PostgreSQL Streaming Replication
Troubleshooting PostgreSQL Streaming ReplicationTroubleshooting PostgreSQL Streaming Replication
Troubleshooting PostgreSQL Streaming Replication
 
PostgreSQL Administration for System Administrators
PostgreSQL Administration for System AdministratorsPostgreSQL Administration for System Administrators
PostgreSQL Administration for System Administrators
 
Linux tuning to improve PostgreSQL performance
Linux tuning to improve PostgreSQL performanceLinux tuning to improve PostgreSQL performance
Linux tuning to improve PostgreSQL performance
 
How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015
How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015
How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015
 
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...
 
Postgresql Database Administration Basic - Day1
Postgresql  Database Administration Basic  - Day1Postgresql  Database Administration Basic  - Day1
Postgresql Database Administration Basic - Day1
 
PostgreSQL Streaming Replication Cheatsheet
PostgreSQL Streaming Replication CheatsheetPostgreSQL Streaming Replication Cheatsheet
PostgreSQL Streaming Replication Cheatsheet
 
Why we love pgpool-II and why we hate it!
Why we love pgpool-II and why we hate it!Why we love pgpool-II and why we hate it!
Why we love pgpool-II and why we hate it!
 
Patroni: Kubernetes-native PostgreSQL companion
Patroni: Kubernetes-native PostgreSQL companionPatroni: Kubernetes-native PostgreSQL companion
Patroni: Kubernetes-native PostgreSQL companion
 
Patroni - HA PostgreSQL made easy
Patroni - HA PostgreSQL made easyPatroni - HA PostgreSQL made easy
Patroni - HA PostgreSQL made easy
 
OpenGurukul : Database : PostgreSQL
OpenGurukul : Database : PostgreSQLOpenGurukul : Database : PostgreSQL
OpenGurukul : Database : PostgreSQL
 
Postgres connections at scale
Postgres connections at scalePostgres connections at scale
Postgres connections at scale
 
High Availability and Disaster Recovery in PostgreSQL - EQUNIX
High Availability and Disaster Recovery in PostgreSQL - EQUNIXHigh Availability and Disaster Recovery in PostgreSQL - EQUNIX
High Availability and Disaster Recovery in PostgreSQL - EQUNIX
 
PGConf.ASIA 2019 Bali - Setup a High-Availability and Load Balancing PostgreS...
PGConf.ASIA 2019 Bali - Setup a High-Availability and Load Balancing PostgreS...PGConf.ASIA 2019 Bali - Setup a High-Availability and Load Balancing PostgreS...
PGConf.ASIA 2019 Bali - Setup a High-Availability and Load Balancing PostgreS...
 
PostgreSQL continuous backup and PITR with Barman
 PostgreSQL continuous backup and PITR with Barman PostgreSQL continuous backup and PITR with Barman
PostgreSQL continuous backup and PITR with Barman
 
Wars of MySQL Cluster ( InnoDB Cluster VS Galera )
Wars of MySQL Cluster ( InnoDB Cluster VS Galera ) Wars of MySQL Cluster ( InnoDB Cluster VS Galera )
Wars of MySQL Cluster ( InnoDB Cluster VS Galera )
 
Deep dive into PostgreSQL statistics.
Deep dive into PostgreSQL statistics.Deep dive into PostgreSQL statistics.
Deep dive into PostgreSQL statistics.
 
Logical Replication in PostgreSQL
Logical Replication in PostgreSQLLogical Replication in PostgreSQL
Logical Replication in PostgreSQL
 

Similar to PGDay.Amsterdam 2018 - Stefan Fercot - Save your data with pgBackRest

pg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLpg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLCommand Prompt., Inc
 
pg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLpg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLMark Wong
 
Oracle to Postgres Migration - part 2
Oracle to Postgres Migration - part 2Oracle to Postgres Migration - part 2
Oracle to Postgres Migration - part 2PgTraining
 
Out of the Box Replication in Postgres 9.4(pgconfsf)
Out of the Box Replication in Postgres 9.4(pgconfsf)Out of the Box Replication in Postgres 9.4(pgconfsf)
Out of the Box Replication in Postgres 9.4(pgconfsf)Denish Patel
 
Out of the Box Replication in Postgres 9.4(PgCon)
Out of the Box Replication in Postgres 9.4(PgCon)Out of the Box Replication in Postgres 9.4(PgCon)
Out of the Box Replication in Postgres 9.4(PgCon)Denish Patel
 
Out of the Box Replication in Postgres 9.4(PgCon)
Out of the Box Replication in Postgres 9.4(PgCon)Out of the Box Replication in Postgres 9.4(PgCon)
Out of the Box Replication in Postgres 9.4(PgCon)Denish Patel
 
Out of the box replication in postgres 9.4
Out of the box replication in postgres 9.4Out of the box replication in postgres 9.4
Out of the box replication in postgres 9.4Denish Patel
 
Streaming replication in practice
Streaming replication in practiceStreaming replication in practice
Streaming replication in practiceAlexey Lesovsky
 
Automating Disaster Recovery PostgreSQL
Automating Disaster Recovery PostgreSQLAutomating Disaster Recovery PostgreSQL
Automating Disaster Recovery PostgreSQLNina Kaufman
 
What’s new in 9.6, by PostgreSQL contributor
What’s new in 9.6, by PostgreSQL contributorWhat’s new in 9.6, by PostgreSQL contributor
What’s new in 9.6, by PostgreSQL contributorMasahiko Sawada
 
configuring a warm standby, the easy way
configuring a warm standby, the easy wayconfiguring a warm standby, the easy way
configuring a warm standby, the easy wayCommand Prompt., Inc
 
Out of the Box Replication in Postgres 9.4(PgConfUS)
Out of the Box Replication in Postgres 9.4(PgConfUS)Out of the Box Replication in Postgres 9.4(PgConfUS)
Out of the Box Replication in Postgres 9.4(PgConfUS)Denish Patel
 
Out of the box replication in postgres 9.4(pg confus)
Out of the box replication in postgres 9.4(pg confus)Out of the box replication in postgres 9.4(pg confus)
Out of the box replication in postgres 9.4(pg confus)Denish Patel
 
Postgres 12 Cluster Database operations.
Postgres 12 Cluster Database operations.Postgres 12 Cluster Database operations.
Postgres 12 Cluster Database operations.Vijay Kumar N
 
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...Command Prompt., Inc
 
pg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLpg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLMark Wong
 
Postgresql Database Administration Basic - Day2
Postgresql  Database Administration Basic  - Day2Postgresql  Database Administration Basic  - Day2
Postgresql Database Administration Basic - Day2PoguttuezhiniVP
 
pg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLpg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLCommand Prompt., Inc
 

Similar to PGDay.Amsterdam 2018 - Stefan Fercot - Save your data with pgBackRest (20)

pg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLpg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQL
 
pg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLpg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQL
 
Oracle to Postgres Migration - part 2
Oracle to Postgres Migration - part 2Oracle to Postgres Migration - part 2
Oracle to Postgres Migration - part 2
 
Out of the Box Replication in Postgres 9.4(pgconfsf)
Out of the Box Replication in Postgres 9.4(pgconfsf)Out of the Box Replication in Postgres 9.4(pgconfsf)
Out of the Box Replication in Postgres 9.4(pgconfsf)
 
Out of the Box Replication in Postgres 9.4(PgCon)
Out of the Box Replication in Postgres 9.4(PgCon)Out of the Box Replication in Postgres 9.4(PgCon)
Out of the Box Replication in Postgres 9.4(PgCon)
 
Out of the Box Replication in Postgres 9.4(PgCon)
Out of the Box Replication in Postgres 9.4(PgCon)Out of the Box Replication in Postgres 9.4(PgCon)
Out of the Box Replication in Postgres 9.4(PgCon)
 
Out of the box replication in postgres 9.4
Out of the box replication in postgres 9.4Out of the box replication in postgres 9.4
Out of the box replication in postgres 9.4
 
Streaming replication in practice
Streaming replication in practiceStreaming replication in practice
Streaming replication in practice
 
Automating Disaster Recovery PostgreSQL
Automating Disaster Recovery PostgreSQLAutomating Disaster Recovery PostgreSQL
Automating Disaster Recovery PostgreSQL
 
What’s new in 9.6, by PostgreSQL contributor
What’s new in 9.6, by PostgreSQL contributorWhat’s new in 9.6, by PostgreSQL contributor
What’s new in 9.6, by PostgreSQL contributor
 
configuring a warm standby, the easy way
configuring a warm standby, the easy wayconfiguring a warm standby, the easy way
configuring a warm standby, the easy way
 
Gg steps
Gg stepsGg steps
Gg steps
 
Out of the Box Replication in Postgres 9.4(PgConfUS)
Out of the Box Replication in Postgres 9.4(PgConfUS)Out of the Box Replication in Postgres 9.4(PgConfUS)
Out of the Box Replication in Postgres 9.4(PgConfUS)
 
Out of the box replication in postgres 9.4(pg confus)
Out of the box replication in postgres 9.4(pg confus)Out of the box replication in postgres 9.4(pg confus)
Out of the box replication in postgres 9.4(pg confus)
 
Postgres 12 Cluster Database operations.
Postgres 12 Cluster Database operations.Postgres 12 Cluster Database operations.
Postgres 12 Cluster Database operations.
 
Pitr Made Easy
Pitr Made EasyPitr Made Easy
Pitr Made Easy
 
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...
 
pg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLpg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQL
 
Postgresql Database Administration Basic - Day2
Postgresql  Database Administration Basic  - Day2Postgresql  Database Administration Basic  - Day2
Postgresql Database Administration Basic - Day2
 
pg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLpg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQL
 

Recently uploaded

The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxLoriGlavin3
 
Visualising and forecasting stocks using Dash
Visualising and forecasting stocks using DashVisualising and forecasting stocks using Dash
Visualising and forecasting stocks using Dashnarutouzumaki53779
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr BaganFwdays
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxLoriGlavin3
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024Lonnie McRorey
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 3652toLead Limited
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsSergiu Bodiu
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek SchlawackFwdays
 
Ryan Mahoney - Will Artificial Intelligence Replace Real Estate Agents
Ryan Mahoney - Will Artificial Intelligence Replace Real Estate AgentsRyan Mahoney - Will Artificial Intelligence Replace Real Estate Agents
Ryan Mahoney - Will Artificial Intelligence Replace Real Estate AgentsRyan Mahoney
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterScale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterMydbops
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsPixlogix Infotech
 
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESmohitsingh558521
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity PlanDatabarracks
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfAlex Barbosa Coqueiro
 
Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLDeveloper Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLScyllaDB
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxLoriGlavin3
 
Sample pptx for embedding into website for demo
Sample pptx for embedding into website for demoSample pptx for embedding into website for demo
Sample pptx for embedding into website for demoHarshalMandlekar2
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfAddepto
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxLoriGlavin3
 

Recently uploaded (20)

The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptx
 
Visualising and forecasting stocks using Dash
Visualising and forecasting stocks using DashVisualising and forecasting stocks using Dash
Visualising and forecasting stocks using Dash
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platforms
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
 
Ryan Mahoney - Will Artificial Intelligence Replace Real Estate Agents
Ryan Mahoney - Will Artificial Intelligence Replace Real Estate AgentsRyan Mahoney - Will Artificial Intelligence Replace Real Estate Agents
Ryan Mahoney - Will Artificial Intelligence Replace Real Estate Agents
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterScale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL Router
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and Cons
 
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity Plan
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdf
 
Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLDeveloper Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQL
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
 
Sample pptx for embedding into website for demo
Sample pptx for embedding into website for demoSample pptx for embedding into website for demo
Sample pptx for embedding into website for demo
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdf
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
 

PGDay.Amsterdam 2018 - Stefan Fercot - Save your data with pgBackRest

  • 1. Save your data with pgBackRest Stefan Fercot 12 July 2018 1
  • 2. Who Am I? Stefan Fercot aka. pgstef PostgreSQL user since 2010 involved in the community since 2016 @dalibo since 2017 2
  • 3. Dalibo Services Support Training Advice Based in France Contributing to PostgreSQL community We’re hiring! 3 . 1
  • 5. Write Ahead Log (WAL) transactions written sequentially considered committed when flushed to disk WAL replay a er a crash make the database consistent 5
  • 6. Point-In-Time Recovery (PITR) combine file-system-level backup continuous archiving of the WAL files restore the file-system-level backup and replay the archived WAL files not mandatory to replay the WAL entries all the way to the end 6
  • 7. How To Do It? (1) postgresql.conf archive_mode archive_command archive_timeout 7
  • 8. How To Do It? (2) pg_start_backup() file-system-level backup : tar, rsync,… pg_stop_backup() 8
  • 9. How To Do It? (3) so many ways to get that wrong what about backup retention? … 9
  • 10. What Is pgBackRest? aims to be a simple, reliable backup and restore system developed by David Steele & Stephen Frost (CrunchyData) Perl & C MIT license 10
  • 11. Main Features custom protocol local or remote operation (via SSH) multi-process full/differential/incremental backup backup rotation and archive expiration parallel, asynchronous WAL push and get Amazon S3 support encryption … 11
  • 12. Installation Use the PGDG repository, Luke! yum / apt-get install pgbackrest 1 package with some (~ 40) dependencies 12
  • 13. Configuration /etc/pgbackrest.conf , example : main configuration in the [global] part each PostgreSQL cluster to backup has its own configuration, called stanza [global] repo1-path=/var/lib/pgsql/10/backups log-level-console=info [my_stanza] pg1-path=/var/lib/pgsql/10/data 13
  • 14. Global Section - some examples process-max: max processes to use for compress/transfer repo1-path: path where backups and archive are stored repo1-cipher-pass: passphrase used to encrypt/decrypt files of the repository repo1-retention-full: number of full backups to retain repo1-retention-diff: number of differential backups to retain repo1-host: repository host when operating remotely via SSH repo1-host-user: repository host user when repo1-host is set … 14
  • 15. Stanza Section pg1-path: PostgreSQL data directory pg1-port any global configuration can be overridden … 15
  • 16. PostgreSQL Configuration archive_mode = on wal_level = replica archive_command = 'pgbackrest --stanza=my_stanza archive-push %p' 16
  • 17. Initialize The Stanza create the stanza check the configuration and if archiving is working # sudo -u postgres pgbackrest --stanza=my_stanza stanza-create # sudo -u postgres pgbackrest --stanza=my_stanza check 17
  • 18. Perform a Backup supported types: incr, diff, full # sudo -u postgres pgbackrest --stanza=my_stanza --type=full backup 18
  • 19. Backup Information # sudo -u postgres pgbackrest --stanza=my_stanza info stanza: my_stanza status: ok db (current) wal archive min/max (10-1): 000000010000000000000001 / 000000010000000000000003 full backup: 20180620-180524F timestamp start/stop: 2018-06-20 18:05:24 / 2018-06-20 18:05:39 wal start/stop: 000000010000000000000003 / 000000010000000000000003 database size: 23.2MB, backup size: 23.2MB repository size: 2.7MB, repository backup size: 2.7MB 19
  • 20. Restore a Backup options --delta --target … # sudo -u postgres pgbackrest --stanza=ma_stanza restore 20
  • 22. Step 1: Backup # sudo -u postgres pgbackrest --stanza=my_stanza --type=full backup P00 INFO: backup command begin 2.03: --pg1-path=/var/lib/pgsql/10/data --repo1-path=/var/lib/pgsql/10/backups --stanza=my_stanza --type=full P00 INFO: execute non-exclusive pg_start_backup() with label "pgBackRest backup started at 2018-06-28 16:53:20": backup begins after the next regular checkpoint completes P00 INFO: backup start archive = 000000020000000000000019, lsn = 0/19000060 P00 INFO: full backup size = 23.2MB P00 INFO: execute non-exclusive pg_stop_backup() and wait for all WAL segments to archive P00 INFO: backup stop archive = 000000020000000000000019, lsn = 0/19000130 P00 INFO: new backup label = 20180628-165320F P00 INFO: backup command end: completed successfully P00 INFO: expire command begin P00 INFO: expire command end: completed successfully 22
  • 23. Step 2: Create Important Data # sudo -iu postgres psql -c " begin; create table important_table (message text); insert into important_table values ('Important Data'); commit; select * from important_table;" message ---------------- Important Data (1 row) 23
  • 24. Step 3: Get Current Time # sudo -iu postgres psql -Atc "select current_timestamp" 2018-06-28 16:54:22.221035+02 24
  • 25. Step 4: Ooops… # sudo -iu postgres psql -c " begin; drop table important_table; commit; select * from important_table;" ERROR: relation "important_table" does not exist LINE 1: ...drop table important_table; commit; select * from important_... ^ 25
  • 26. Step 5: Stop PostgreSQL # sudo -iu postgres psql -c "select pg_switch_wal()" # systemctl stop postgresql-10.service 26
  • 27. Step 6: Restore # sudo -u postgres pgbackrest --stanza=my_stanza --delta --type=time --target="2018-06-28 16:54:22.221035+02" --target-action=promote restore P00 INFO: restore command begin 2.03: --delta --pg1-path=/var/lib/pgsql/10/data --repo1-path=/var/lib/pgsql/10/backups --stanza=my_stanza --target="2018-06-28 16:54:22.221035+02" --target-action=promote --type=time P00 INFO: restore backup set 20180628-165320F P00 INFO: remove invalid files/paths/links from /var/lib/pgsql/10/data P00 INFO: cleanup removed 22 files P00 INFO: write /var/lib/pgsql/10/data/recovery.conf P00 INFO: restore global/pg_control (performed last to ensure aborted restores cannot be started) P00 INFO: restore command end: completed successfully 27
  • 28. Step 7: Check recovery.conf # cat /var/lib/pgsql/10/data/recovery.conf restore_command = 'pgbackrest --stanza=my_stanza archive-get %f "%p"' recovery_target_time = '2018-06-28 16:54:22.221035+02' recovery_target_action = 'promote' 28
  • 29. Step 8: Start PostgreSQL # systemctl start postgresql-10.service # cat /var/lib/pgsql/10/data/log/* LOG: recovery stopping before commit of transaction 561, time 2018-06-28 16:54:59.481247+02 LOG: redo done at 0/1A01E590 LOG: last completed transaction was at log time 2018-06-28 16:54:13.370025+02 29
  • 30. Step 9: Check the data # sudo -iu postgres psql -c "select * from important_table" message ---------------- Important Data (1 row) 30
  • 33. Where official website: code: rpm and deb: in the PGDG repositories! user guide: support: https://pgbackrest.org https://github.com/pgbackrest/pgbackrest https://pgbackrest.org/user-guide.html https://github.com/pgbackrest/pgbackrest/issues 33
  • 34. Thank you for your attention! 34