This document provides an overview of PostgreSQL backup and recovery methods, including pg_dump, pg_dumpall, psql, pg_restore, and point-in-time recovery (PITR). It discusses the options and usage of each tool and provides examples.
This presentation covers all aspects of PostgreSQL administration, including installation, security, file structure, configuration, reporting, backup, daily maintenance, monitoring activity, disk space computations, and disaster recovery. It shows how to control host connectivity, configure the server, find the query being run by each session, and find the disk space used by each database.
The paperback version is available on lulu.com there http://goo.gl/fraa8o
This is the first volume of the postgresql database administration book. The book covers the steps for installing, configuring and administering a PostgreSQL 9.3 on Linux debian. The book covers the logical and physical aspect of PostgreSQL. Two chapters are dedicated to the backup/restore topic.
Spencer Christensen
There are many aspects to managing an RDBMS. Some of these are handled by an experienced DBA, but there are a good many things that any sys admin should be able to take care of if they know what to look for.
This presentation will cover basics of managing Postgres, including creating database clusters, overview of configuration, and logging. We will also look at tools to help monitor Postgres and keep an eye on what is going on. Some of the tools we will review are:
* pgtop
* pg_top
* pgfouine
* check_postgres.pl.
Check_postgres.pl is a great tool that can plug into your Nagios or Cacti monitoring systems, giving you even better visibility into your databases.
En savoir plus sur www.opensourceschool.fr
Ce support est diffusé sous licence Creative Commons (CC BY-SA 3.0 FR) Attribution - Partage dans les Mêmes Conditions 3.0 France
Plan :
1. Introduction
2. Installation
3. The psql client
4. Authentication and privileges
5. Backup and restoration
6. Internal Architecture
7. Performance optimization
8. Stats and monitoring
9. Logs
10. Replication
This ppt was used by Devrim at pgDay Asia 2017. He talked about some important facts about WAL - Transaction Logs or xlogs in PostgreSQL. Some of these can really come handy on a bad day
This presentation covers all aspects of PostgreSQL administration, including installation, security, file structure, configuration, reporting, backup, daily maintenance, monitoring activity, disk space computations, and disaster recovery. It shows how to control host connectivity, configure the server, find the query being run by each session, and find the disk space used by each database.
The paperback version is available on lulu.com there http://goo.gl/fraa8o
This is the first volume of the postgresql database administration book. The book covers the steps for installing, configuring and administering a PostgreSQL 9.3 on Linux debian. The book covers the logical and physical aspect of PostgreSQL. Two chapters are dedicated to the backup/restore topic.
Spencer Christensen
There are many aspects to managing an RDBMS. Some of these are handled by an experienced DBA, but there are a good many things that any sys admin should be able to take care of if they know what to look for.
This presentation will cover basics of managing Postgres, including creating database clusters, overview of configuration, and logging. We will also look at tools to help monitor Postgres and keep an eye on what is going on. Some of the tools we will review are:
* pgtop
* pg_top
* pgfouine
* check_postgres.pl.
Check_postgres.pl is a great tool that can plug into your Nagios or Cacti monitoring systems, giving you even better visibility into your databases.
En savoir plus sur www.opensourceschool.fr
Ce support est diffusé sous licence Creative Commons (CC BY-SA 3.0 FR) Attribution - Partage dans les Mêmes Conditions 3.0 France
Plan :
1. Introduction
2. Installation
3. The psql client
4. Authentication and privileges
5. Backup and restoration
6. Internal Architecture
7. Performance optimization
8. Stats and monitoring
9. Logs
10. Replication
This ppt was used by Devrim at pgDay Asia 2017. He talked about some important facts about WAL - Transaction Logs or xlogs in PostgreSQL. Some of these can really come handy on a bad day
Connection Pooling in PostgreSQL using pgbouncer Sameer Kumar
The presentation was presented at 5th Postgres User Group, Singapore.
It explain how to setup pgbouncer and also shows a few demonstration graphs comparing the advantages/gains in performance when using pgbouncer instead of direct connections to PostgreSQL database.
This technical presentation shows you the best practices with EDB Postgres tools, that are designed to make database administration easier and more efficient:
● Tune a new database using Postgres Expert
● Set up streaming replication in EDB Postgres Enterprise Manager (PEM)
● Create a backup schedule in EDB Postgres Backup and Recovery
● Automatically failover with EDB Postgres Failover Manager
● Use SQL Profiler and Index Advisor to add indexes
The presentation also included a demonstration. To access the recording visit www.enterprisedb.com and access the webcast recordings section or email info@enterprisedb.com.
MySQL Administrator
Basic course
- MySQL 개요
- MySQL 설치 / 설정
- MySQL 아키텍처 - MySQL 스토리지 엔진
- MySQL 관리
- MySQL 백업 / 복구
- MySQL 모니터링
Advanced course
- MySQL Optimization
- MariaDB / Percona
- MySQL HA (High Availability)
- MySQL troubleshooting
네오클로바
http://neoclova.co.kr/
Introduction to the Mysteries of ClickHouse Replication, By Robert Hodges and...Altinity Ltd
Presented at the webinar, July 31, 2019
Built-in replication is a powerful ClickHouse feature that helps scale data warehouse performance as well as ensure high availability. This webinar will introduce how replication works internally, explain configuration of clusters with replicas, and show you how to set up and manage ZooKeeper, which is necessary for replication to function. We'll finish off by showing useful replication tricks, such as utilizing replication to migrate data between hosts. Join us to become an expert in this important subject!
All about Zookeeper and ClickHouse Keeper.pdfAltinity Ltd
ClickHouse clusters depend on ZooKeeper to handle replication and distributed DDL commands. In this Altinity webinar, we’ll explain why ZooKeeper is necessary, how it works, and introduce the new built-in replacement named ClickHouse Keeper. You’ll learn practical tips to care for ZooKeeper in sickness and health. You’ll also learn how/when to use ClickHouse Keeper. We will share our recommendations for keeping that happy as well.
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...ScaleGrid.io
Compare top PostgreSQL high availability frameworks - PostgreSQL Automatic Failover (PAF), Replication Manager (repmgr) and Patroni to improve your app uptime. ScaleGrid blog - https://scalegrid.io/blog/whats-the-best-postgresql-high-availability-framework-paf-vs-repmgr-vs-patroni-infographic/
Robert Haas
Why does my query need a plan? Sequential scan vs. index scan. Join strategies. Join reordering. Joins you can't reorder. Join removal. Aggregates and DISTINCT. Using EXPLAIN. Row count and cost estimation. Things the query planner doesn't understand. Other ways the planner can fail. Parameters you can tune. Things that are nearly always slow. Redesigning your schema. Upcoming features and future work.
This is the presentation delivered by Karthik.P.R at MySQL User Camp Bangalore on 09th June 2017. ProxySQL is a high performance MySQL Load Balancer Designed to scale database servers.
PostgreSQL - backup and recovery with large databasesFederico Campoli
Life on a rollercoaster, backup and recovery with large databases
Dealing with large databases is always a challenge.
The backups and the HA procedures evolve meanwhile the database installation grow up over the time.
The talk will cover the problems solved by the DBA in four years of working with large databases, which size increased from 1.7 TB single cluster, up to 40 TB in a multi shard environment.
The talk will cover either the disaster recovery with pg_dump and the high availability with the log shipping/streaming replication.
The presentation is based on a real story. The names are changed in order to protect the innocents.
Josh Berkus
You've heard that PostgreSQL is the highest-performance transactional open source database, but you're not seeing it on YOUR server. In fact, your PostgreSQL application is kind of poky. What should you do? While doing advanced performance engineering for really high-end systems takes years to learn, you can learn the basics to solve performance issues for 80% of PostgreSQL installations in less than an hour. In this session, you will learn: -- The parts of database application performance -- The performance setup procedure -- Basic troubleshooting tools -- The 13 postgresql.conf settings you need to know -- Where to look for more information.
This talk will cover experiences from writing a FDW for Informix and will discuss differences between 9.1 and 9.2, as well as the new writable API with the upcoming 9.3 release, additionally data type mapping and conversion, optimizer support and performance related topics.
The talk tries to give the attendees an overall idea behind the techniques and pitfalls they may experience when they want to write their own.
Connection Pooling in PostgreSQL using pgbouncer Sameer Kumar
The presentation was presented at 5th Postgres User Group, Singapore.
It explain how to setup pgbouncer and also shows a few demonstration graphs comparing the advantages/gains in performance when using pgbouncer instead of direct connections to PostgreSQL database.
This technical presentation shows you the best practices with EDB Postgres tools, that are designed to make database administration easier and more efficient:
● Tune a new database using Postgres Expert
● Set up streaming replication in EDB Postgres Enterprise Manager (PEM)
● Create a backup schedule in EDB Postgres Backup and Recovery
● Automatically failover with EDB Postgres Failover Manager
● Use SQL Profiler and Index Advisor to add indexes
The presentation also included a demonstration. To access the recording visit www.enterprisedb.com and access the webcast recordings section or email info@enterprisedb.com.
MySQL Administrator
Basic course
- MySQL 개요
- MySQL 설치 / 설정
- MySQL 아키텍처 - MySQL 스토리지 엔진
- MySQL 관리
- MySQL 백업 / 복구
- MySQL 모니터링
Advanced course
- MySQL Optimization
- MariaDB / Percona
- MySQL HA (High Availability)
- MySQL troubleshooting
네오클로바
http://neoclova.co.kr/
Introduction to the Mysteries of ClickHouse Replication, By Robert Hodges and...Altinity Ltd
Presented at the webinar, July 31, 2019
Built-in replication is a powerful ClickHouse feature that helps scale data warehouse performance as well as ensure high availability. This webinar will introduce how replication works internally, explain configuration of clusters with replicas, and show you how to set up and manage ZooKeeper, which is necessary for replication to function. We'll finish off by showing useful replication tricks, such as utilizing replication to migrate data between hosts. Join us to become an expert in this important subject!
All about Zookeeper and ClickHouse Keeper.pdfAltinity Ltd
ClickHouse clusters depend on ZooKeeper to handle replication and distributed DDL commands. In this Altinity webinar, we’ll explain why ZooKeeper is necessary, how it works, and introduce the new built-in replacement named ClickHouse Keeper. You’ll learn practical tips to care for ZooKeeper in sickness and health. You’ll also learn how/when to use ClickHouse Keeper. We will share our recommendations for keeping that happy as well.
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...ScaleGrid.io
Compare top PostgreSQL high availability frameworks - PostgreSQL Automatic Failover (PAF), Replication Manager (repmgr) and Patroni to improve your app uptime. ScaleGrid blog - https://scalegrid.io/blog/whats-the-best-postgresql-high-availability-framework-paf-vs-repmgr-vs-patroni-infographic/
Robert Haas
Why does my query need a plan? Sequential scan vs. index scan. Join strategies. Join reordering. Joins you can't reorder. Join removal. Aggregates and DISTINCT. Using EXPLAIN. Row count and cost estimation. Things the query planner doesn't understand. Other ways the planner can fail. Parameters you can tune. Things that are nearly always slow. Redesigning your schema. Upcoming features and future work.
This is the presentation delivered by Karthik.P.R at MySQL User Camp Bangalore on 09th June 2017. ProxySQL is a high performance MySQL Load Balancer Designed to scale database servers.
PostgreSQL - backup and recovery with large databasesFederico Campoli
Life on a rollercoaster, backup and recovery with large databases
Dealing with large databases is always a challenge.
The backups and the HA procedures evolve meanwhile the database installation grow up over the time.
The talk will cover the problems solved by the DBA in four years of working with large databases, which size increased from 1.7 TB single cluster, up to 40 TB in a multi shard environment.
The talk will cover either the disaster recovery with pg_dump and the high availability with the log shipping/streaming replication.
The presentation is based on a real story. The names are changed in order to protect the innocents.
Josh Berkus
You've heard that PostgreSQL is the highest-performance transactional open source database, but you're not seeing it on YOUR server. In fact, your PostgreSQL application is kind of poky. What should you do? While doing advanced performance engineering for really high-end systems takes years to learn, you can learn the basics to solve performance issues for 80% of PostgreSQL installations in less than an hour. In this session, you will learn: -- The parts of database application performance -- The performance setup procedure -- Basic troubleshooting tools -- The 13 postgresql.conf settings you need to know -- Where to look for more information.
This talk will cover experiences from writing a FDW for Informix and will discuss differences between 9.1 and 9.2, as well as the new writable API with the upcoming 9.3 release, additionally data type mapping and conversion, optimizer support and performance related topics.
The talk tries to give the attendees an overall idea behind the techniques and pitfalls they may experience when they want to write their own.
PostgresOpen 2013 A Comparison of PostgreSQL Encryption OptionsFaisal Akber
Are you looking to encrypt your data within PostgreSQL? We will review the various options available for encrypting data with PostgreSQL. We will also look at various options available to employ encryption and review various configuration and performance for using encryption.
There are a number of options available when encrypting data with PostgreSQL. When determining the mechanisms to use, it is important to understand the data, the application and how it is being used. We will compare different methods of encrypting data in their feature-sets and performance.
We will try to answer the following questions: Where do I enable the encryption? Where is my data safe and where is it exposed? Why should I use the various encryption modules available?
Security Best Practices for your Postgres DeploymentPGConf APAC
These slides were used by Sameer Kumar of Ashnik for presenting his topic at pgDay Asia 2016. He took audience through some of the security best practices for deploying and hardening PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLMark Wong
pc_proctab is a collection of PostgreSQL stored functions that allow you to access the operating system process table using SQL. See examples on how to use these stored functions to collect processor and I/O statistics on SQL statements run against the database.
Right now postgres can't compress its data in many situations and that leads sometimes to increased storage overhead by the order of magnitude comparing with commercial DBMS. Common viewpoint that this task can be accomplished by file system level compression but most popular and well tested Linux file system can't do that. I will talk about our patches that implements page compression on disk or on disk + in memory; in what situation it is better to use what kind of compression; and also discuss experience of using compression in production.
pg_proctab: Accessing System Stats in PostgreSQLMark Wong
pg_proctab is a collection of PostgreSQL stored functions that provide access to the operating system process table using SQL. We'll show you which functions are available and where they collect the data, and give examples of their use to collect processor and I/O statistics on SQL queries.
Mark Wong
pg_proctab is a collection of PostgreSQL stored functions that provide access to the operating system process table using SQL. We'll show you which functions are available and where they collect the data, and give examples of their use to collect processor and I/O statistics on SQL queries.
This short text will get you up to speed in no time on creating visualizations using R's ggplot2 package. It was developed as part of a training to those who had no prior experience in R and had limited knowledge on general programming concepts. It's a must have initial guide for those exploring the field of Data Science
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
Le nuove frontiere dell'AI nell'RPA con UiPath Autopilot™UiPathCommunity
In questo evento online gratuito, organizzato dalla Community Italiana di UiPath, potrai esplorare le nuove funzionalità di Autopilot, il tool che integra l'Intelligenza Artificiale nei processi di sviluppo e utilizzo delle Automazioni.
📕 Vedremo insieme alcuni esempi dell'utilizzo di Autopilot in diversi tool della Suite UiPath:
Autopilot per Studio Web
Autopilot per Studio
Autopilot per Apps
Clipboard AI
GenAI applicata alla Document Understanding
👨🏫👨💻 Speakers:
Stefano Negro, UiPath MVPx3, RPA Tech Lead @ BSP Consultant
Flavio Martinelli, UiPath MVP 2023, Technical Account Manager @UiPath
Andrei Tasca, RPA Solutions Team Lead @NTT Data
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Assure Contact Center Experiences for Your Customers With ThousandEyes
Kevin Kempter PostgreSQL Backup and Recovery Methods @ Postgres Open
1. Consistent State www.consistentstate.com kevink@consistentstate.com
PostgreSQL Backup and Recovery Methods
Kevin Kempter Chief Data Architect
Wednesday, September 18, 13
4. Overview
• Multiple backup methods
• Multiple backup file formats
• Many recovery choices / options if a
pg_restore compatible format is used
• PITR
• PITR is the base construct for WAL
shipping (Warm Standby)
4
Wednesday, September 18, 13
6. pg_dump
✓ Utility to dump a snapshot of a single
database
✓ Multiple output options
✓ Non-blocking
✓ Creates a “Consistent” backup - even if the
database is in use
6
Wednesday, September 18, 13
8. pg_dump - Common Options
• -s [--schema-only]
• Dump schema (DDL) only, no data
• -a [--data-only]
• Dump data only - no DDL
• -c [--clean]
• Generate drop statements for all created objects
• -C [--create]
• Generate a “CREATE DATABASE” statement
• -n schema [--schema=schema]
• Only dump the specified schema, wildcard characters are allowed, also multiple -n’s are
allowed
8
Wednesday, September 18, 13
9. pg_dump - Common Options (continued)
• -N schema [--exclude-schema=schema]
• Exclude specified schema
• -F format [--format=format]
• Output format
• p (Plain) plain sql file #default
• c (Custom) custom binary format
• t (tar) tar format
• d (directory) Creates a directory with one file per table/blob, plus a TOC file in a binary
format that pg_restore can read
• -o [--oids]
• Dump table OID’s
• -O [--no-owner]
• Do not generate ownership commands
9
Wednesday, September 18, 13
10. pg_dump - Common Options (continued)
• -t table [--table=table]
• Only dump the specified table, wildcard characters are allowed, also multiple -t’s are
allowed, over-rides -n and -N options
• -x [--no-privileges] [--no-acl]
• Do not dump access privileges
• --inserts
• Generate INSERT statements
• --disable-triggers
• disable triggers during restore (for a data only restore) when doing a data-only dump
• --lock-wait-timeout=timeout
• fail if shared lock on an object cannot be acquired within timeout time
10
Wednesday, September 18, 13
11. pg_dump - Common Options (continued)
• -Z 0..9 [--compress=0..9]
• Specify compression level for custom format or plain format (not supported for tar
format)
• -v [--verbose]
• -V [--version]
11
Wednesday, September 18, 13
12. pg_dump - Examples
$ pg_dump -C --inserts prod1_db > prod1_db.sql
Creates a dump of insert statements including a create database statement
$ pg_dump --data-only --table=customer -Fc prod1_db > prod1_db.cust.fc.dmp
Dump the customer table data (data only) in a custom format from the prod1_db database
$ pg_dump -s prod1_db > prod1_db.ddl_only.sql
Creates a DDL only dump of the prod1_db database
$ pg_dump --schema=gold -Ft prod1_db > prod1_db.gold_schema.dmp
Creates a dump of the gold schema in the prod1_db database in a tar format
12
Wednesday, September 18, 13
14. pg_dumpall
✓ Utility to dump a snapshot of a full
database cluster (or cluster-wide
constructs)
✓ Dumps only to plain sql format
✓ Non-blocking
✓ Creates a “Consistent” backup - even
if the database is in use
14
Wednesday, September 18, 13
16. pg_dumpall - Common Options
• -s [--schema-only]
• Dump schema (DDL) only, no data
• -a [--data-only]
• Dump data only - no DDL
• -c [--clean]
• Generate drop statements for all created objects
• -o [--oids]
• Dump table OID’s
• -O [--no-owner]
• Do not generate ownership commands
16
Wednesday, September 18, 13
17. pg_dumpall - Common Options (continued)
• -r [--roles-only]
• Dump only CREATE ROLE data
• -t [--tablespaces-only]
• Dump only CREATE TABLESPACE data
• -g [--globals-only]
• Dump Global Structures (Roles and Tablespaces)
• --no-tablespaces
• Do NOT dump CREATE TABLESPACE Data
• --inserts
• Generate INSERT statements
17
Wednesday, September 18, 13
18. pg_dumpall - Common Options (continued)
• --disable-triggers
• disable triggers during restore (for a data only restore) when doing a data-only dump
• --lock-wait-timeout=timeout
• fail if shared lock on an object cannot be acquired within timeout time
• -v [--verbose]
• -V [--version]
18
Wednesday, September 18, 13
19. pg_dumpall - Examples
$ pg_dumpall -g > prod1_db_cluster.global_structures.sql
Creates a cluster dump containing only the cluster global structures
$ pg_dumpall --tablespaces-only > prod1_db_cluster.tablespaces.sql
Dump the cluster tablespaces
$ pg_dumpall --no-tablespaces > prod1_db_cluster.no_tablespaces.sql
Creates a dump of the cluster without any tablespace references
$ pg_dumpall -a > prod1_db_cluster.data_only.sql
Creates a dump of the cluster - data only
19
Wednesday, September 18, 13
25. pg_restore - Common Options
• -d dbname [--dbname=dbname]
• -C [--create]
• Create the specified database before restore
• -c [--clean]
• Generate drop statements for all created objects
• -s [--schema-only]
• Dump schema (DDL) only, no data
• -a [--data-only]
• Dump data only - no DDL
25
Wednesday, September 18, 13
26. pg_restore - Common Options (continued)
• -n namespace [--schema=schema]
• Restore only objects in the specified schema
• -O [--no-owner]
• Do not restore ownership of objects
• -I index [--index=index]
• Restore specified index only
• -P function-name(argtype [, ...]) [ --function=function-name(argtype [, ...]) ]
• Restore specified function only
• -T trigger [--trigger=trigger]
• Restore specified trigger only
26
Wednesday, September 18, 13
27. pg_restore - Common Options (continued)
• -t table [--table=table]
• Restore specified table only
• --no-tablespaces
• Do not restore any TABLESPACES
• -F format [--format=format]
• Output format
• c (Custom) custom binary format
• t (tar) tar format
• d (directory)
27
Wednesday, September 18, 13
28. pg_restore - Common Options (continued)
• -t table [--table=table]
• Restore specified table only
• --no-tablespaces
• Do not restore any TABLESPACES
• -j number-of-jobs [--jobs=number-of-jobs]
• Use parallel jobs to perform the restore
• --disable-triggers
• Disable triggers during the restore (for a data only restore)
• -e [--exit-on-error]
• Exits upon any error
28
Wednesday, September 18, 13
29. pg_restore - Common Options (continued)
• -l [--list]
• Create a list (TOC) file
• -L list-file [--use-list=list-file]
• Restore based on the specified list file
• -V [--version]
• -v [--verbose]
29
Wednesday, September 18, 13
30. pg_restore - Examples
$ pg_restore -a -Fc -d prod2_db prod1_db.fc.dmp
Restores data only from a custom formatted file into database prod2_db
$ pg_restore -c --schema=gold_partners -v -Ft -d prod2_db prod.tar.dmp
Cleans (removes data & structures first) then restores the gold_partners
schema from a tar formatted file into the prod2_db database (with verbose
output)
$ pg_restore --schema-only -d qa1_db -Fc -j 10 prod1_db.fc.dmp
Restores the schema only (DDL) from a custom formatted file
into the qa1_db database using 10 parallel streams to do the restore
30
Wednesday, September 18, 13
31. Restoring
via a list file
• pg_restore can create a list file
from a pg_dump file
• List file will contain one line per
needed operation such as:
• CREATE TABLE
• COPY
• CREATE INDEX
• List file can be modified as desired
to create a custom restore
31
Wednesday, September 18, 13
32. Create a list file from the pg_dump file
$ pg_dump -Ft db1 > db1.fc.dmp
$ pg_restore -Ft -l db1.dmp > db1.lst
32
Wednesday, September 18, 13
33. Sample list file header
;
; Archive created at Tue Sep 10 09:42:24 2013
; dbname: testdb
; TOC Entries: 34
; Compression: -1
; Dump Version: 1.12-0
; Format: CUSTOM
; Integer: 4 bytes
; Offset: 8 bytes
; Dumped from database version: 9.2.4
; Dumped by pg_dump version: 9.2.4
;
;
33
Wednesday, September 18, 13
34. Sample list file contents
; Selected TOC Entries:
;
1981; 1262 16386 DATABASE - testdb_old postgres
6; 2615 2200 SCHEMA - public postgres
1982; 0 0 COMMENT - SCHEMA public postgres
1983; 0 0 ACL - public postgres
181; 3079 11730 EXTENSION - plpgsql
1984; 0 0 COMMENT - EXTENSION plpgsql
168; 1259 16411 TABLE public testdb_jasper_metrics_tables postgres
169; 1259 16414 TABLE public testdb_jasper_metrics_tables_tmp1 postgres
170; 1259 16417 TABLE public testdb_metrics_activity postgres
171; 1259 16423 TABLE public testdb_metrics_database postgres
172; 1259 16426 TABLE public testdb_postgres_metrics_bgwriter postgres
173; 1259 16429 TABLE public testdb_postgres_metricsio_user_tables postgres
174; 1259 16432 TABLE public testdb_testdb_gf_metrics_tables postgres
175; 1259 16435 TABLE public testdb_testdb_transition_metrics_tables postgres
176; 1259 16438 TABLE public testdb_testdb_transition_metrics_tables_tmp1 postgres
177; 1259 16441 TABLE public testdb_testdb_transition_metrics_tables_tmp2 postgres
34
Wednesday, September 18, 13
35. Sample list file contents (cont)
178; 1259 16444 TABLE public idle_conn_metrics postgres
179; 1259 16447 TABLE public total_conn_metrics postgres
180; 1259 16450 TABLE public waiting_conn_metrics postgres
1964; 0 16411 TABLE DATA public testdb_jasper_metrics_tables postgres
1965; 0 16414 TABLE DATA public testdb_jasper_metrics_tables_tmp1 postgres
1966; 0 16417 TABLE DATA public testdb_metrics_activity postgres
1967; 0 16423 TABLE DATA public testdb_metrics_database postgres
1968; 0 16426 TABLE DATA public testdb_postgres_metrics_bgwriter postgres
1969; 0 16429 TABLE DATA public testdb_postgres_metricsio_user_tables postgres
1970; 0 16432 TABLE DATA public testdb_testdb_gf_metrics_tables postgres
1971; 0 16435 TABLE DATA public testdb_testdb_transition_metrics_tables postgres
1972; 0 16438 TABLE DATA public testdb_testdb_transition_metrics_tables_tmp1 postgres
1973; 0 16441 TABLE DATA public testdb_testdb_transition_metrics_tables_tmp2 postgres
1974; 0 16444 TABLE DATA public idle_conn_metrics postgres
1975; 0 16447 TABLE DATA public total_conn_metrics postgres
1976; 0 16450 TABLE DATA public waiting_conn_metrics postgres
35
Wednesday, September 18, 13
36. Restore via list file - example
$ pg_dump -Ft prod_db > prod_db.fc.dmp
$ pg_restore -Ft -l prod_db.dmp > prod_db.lst
$ createdb qadb3
Edit prod_db.lst as needed / desired
$ pg_restore -L prod_db.lst -Ft -d qadb3 prod_db.dmp
36
Wednesday, September 18, 13
39. PITR Overview
• PITR Backups
• Archiving the WAL segments
• Making Base Backups
• PITR Recovery
• Restore the last Base Backup
• Prepare the recovered system data directory
• Create a recovery.conf file
• Start the postmaster
39
Wednesday, September 18, 13
40. PITR Setup
• Enable / set the following parameters in the
postgresql.conf file:
• wal_level = archive (or hot_standby)
• archive_mode = on
• archive_command = 'valid archive command'
Can be any valid shell command (including scripts)
• archive_timeout = [timeout]
• Special archive_command (and recovery.conf file) tags
• %p = full path (absolute path) and the file name of the WAL
segment to be archived
• %f = only the file name of the WAL segment
• %% = insert a % character in the command string.
40
Wednesday, September 18, 13
41. PITR Example
• Enable / set the following parameters in the postgresql.conf file:
• wal_level = archive
• archive_mode = on
• archive_command = 'cp %p /stage/wal/%f'
Can be any valid shell command (including scripts)
• archive_timeout = 0
• mkdir /stage/wal
• chown postgres:postgres /stage/wal
• Re-start the Server
41
Wednesday, September 18, 13
42. PITR Example - create transactions
• Execute SQL commands / transactions
• Enable access, turn on applications, etc
• This should force the creation of multiple archived WAL
files in the /stage/wal directory
• WAL segments are copied when:
• The WAL segment is full (see checkpoint_segments)
• Number of seconds specified in archive_timeout has passed
42
Wednesday, September 18, 13
43. PITR Example - create base backup
• Execute pg_start_backup
$ psql pitr_test
# select pg_start_backup ('tag') ;
• Archive the cluster data directory (and any related
tablespaces)
$ tar -czvf /backups/pitr/<date>.data.tar.gz ./data
rsync
other copy methods
• Execute pg_stop_backup
$ psql pitr_test
# select pg_stop_backup () ;
43
Wednesday, September 18, 13
44. PITR Example - create more transactions
• Execute SQL commands / transactions
• The application, user connections, etc will continue to
generate transactions (and archived WAL segments)
• Verify the creation of additional archived WAL files in
the /stage/wal directory
44
Wednesday, September 18, 13
45. PITR - recovery.conf file (common options)
Recovery settings are placed in the file 'recovery.conf'
• restore_command (string)
must return nonzero
• restore_command = 'cp /stage/wal/%f %p'
• restore_command = '/usr/local/bin/restore_shell.sh %p %f'
45
Wednesday, September 18, 13
46. PITR - recovery.conf file (common options)
recovery_target_time (timestamp)
• specifies the time stamp up to which recovery will proceed.
• recovery_target_time and recovery_target_xid are mutually
exclusive
• The default is to recover to the end of the WAL log.
46
Wednesday, September 18, 13
47. PITR Recovery
(1) If available copy the original cluster data directory to an
alternate location
if space is an issue at least copy the old pg_xlog dir it may contain
additional unarchived WAL segments
(2) Ensure the postmaster is not running
47
Wednesday, September 18, 13
48. PITR Recovery
If your backup was an rsync to a second server then skip steps 3 & 4
(3) Remove the cluster data directory and any tablespace
directories
(4) Restore your last system backup
• make sure permissions are retained
• If you're using tablespaces then verify that the symbolic links in
pg_tblspc/ were restored
48
Wednesday, September 18, 13
49. PITR Recovery
(5) Remove any wal segments from the pg_xlog dir that were
restored from the backup
If you didn't backup pg_xlog then create it, make sure you
re-establish it as a symbolic link if needed
If needed also re-create the pg_xlog/archive_status directory
(6) Copy the files from the original pg_xlog dir (if available) into
the new pg_xlog dir
do a copy as opposed to a move in case you need to start over
49
Wednesday, September 18, 13
50. PITR Recovery
(7) Create a recovery command (recovery.conf) in the cluster
data directory.
(8) [Optional] Temporarily modify pg_hba.conf to prevent
ordinary users from connecting until the recovery is complete
(9) Start the server.
The server will go into recovery mode via the recovery.conf file.
Once the recovery is complete then the server will become available and
rename the recovery.conf file to recovery.done
If an error interrupts the recovery (or stops the server) then simply
re-starting the server will restart the recovery
50
Wednesday, September 18, 13
51. PITR Recovery
(10) Verify the recovery.
If the database was not recovered properly (or to a state that you desire) then go
back to step 1
(11) restore the pg_hba.conf to its original state and run a
pg_ctl reload (if it was modified for the recovery)
51
Wednesday, September 18, 13