The document provides an overview of out-of-the-box replication in PostgreSQL 9.4. It discusses PostgreSQL write-ahead logging (WAL), setting up basic streaming replication between a primary and standby server, taking base backups with pg_basebackup, and using replication slots and pg_receivexlog to archive WAL files without external tools. The presentation includes steps to set up a demo of this replication method on a virtual machine.
Out of the Box Replication in Postgres 9.4(pgconfsf)Denish Patel
Denish Patel gave a presentation on PostgreSQL replication. He began by introducing himself and his background. He then discussed PostgreSQL write-ahead logging (WAL), replication history, and how replication is currently setup. The presentation covered replication slots, demoing replication without external tools using pg_basebackup, streaming replication with slots, and pg_receivexlog. Patel also discussed monitoring replication and answered questions from the audience.
Out of the box replication in postgres 9.4Denish Patel
This document provides an overview of setting up out of the box replication in PostgreSQL 9.4 without third party tools. It discusses write-ahead logs (WAL), replication slots, pg_basebackup, and pg_receivexlog. The document then demonstrates setting up replication on VMs with pg_basebackup to initialize a standby server, configuration of primary and standby servers, and monitoring of replication.
This document provides instructions for installing a single-node Hadoop cluster on Ubuntu. It outlines downloading and configuring Java, installing Hadoop, configuring SSH access to localhost, editing Hadoop configuration files, and formatting the HDFS filesystem via the namenode. Key steps include adding a dedicated Hadoop user, generating SSH keys, setting properties in core-site.xml, hdfs-site.xml and mapred-site.xml, and running 'hadoop namenode -format' to initialize the filesystem.
To know more, Register for Online Hadoop Training at WizIQ.
Click here : http://www.wiziq.com/course/21308-hadoop-big-data-training
A complete guide to Hadoop Installation that will help you when ever you face problems while installing Hadoop !!
The document provides requirements and sample exam questions for the Red Hat Certified Engineer (RHCE) EX294 exam. It outlines 18 exam questions to test Ansible skills. Key requirements include setting up 5 virtual machines, one as the Ansible control node and 4 managed nodes. The questions cover tasks like Ansible installation, ad-hoc commands, playbooks, roles, vaults and more. Detailed solutions are provided for each question/task.
This document provides instructions for setting up Hadoop in single node mode on Ubuntu. It describes adding a Hadoop user, installing Java and SSH, downloading and extracting Hadoop, configuring environment variables and Hadoop configuration files, and formatting the NameNode.
Out of the Box Replication in Postgres 9.4(pgconfsf)Denish Patel
Denish Patel gave a presentation on PostgreSQL replication. He began by introducing himself and his background. He then discussed PostgreSQL write-ahead logging (WAL), replication history, and how replication is currently setup. The presentation covered replication slots, demoing replication without external tools using pg_basebackup, streaming replication with slots, and pg_receivexlog. Patel also discussed monitoring replication and answered questions from the audience.
Out of the box replication in postgres 9.4Denish Patel
This document provides an overview of setting up out of the box replication in PostgreSQL 9.4 without third party tools. It discusses write-ahead logs (WAL), replication slots, pg_basebackup, and pg_receivexlog. The document then demonstrates setting up replication on VMs with pg_basebackup to initialize a standby server, configuration of primary and standby servers, and monitoring of replication.
This document provides instructions for installing a single-node Hadoop cluster on Ubuntu. It outlines downloading and configuring Java, installing Hadoop, configuring SSH access to localhost, editing Hadoop configuration files, and formatting the HDFS filesystem via the namenode. Key steps include adding a dedicated Hadoop user, generating SSH keys, setting properties in core-site.xml, hdfs-site.xml and mapred-site.xml, and running 'hadoop namenode -format' to initialize the filesystem.
To know more, Register for Online Hadoop Training at WizIQ.
Click here : http://www.wiziq.com/course/21308-hadoop-big-data-training
A complete guide to Hadoop Installation that will help you when ever you face problems while installing Hadoop !!
The document provides requirements and sample exam questions for the Red Hat Certified Engineer (RHCE) EX294 exam. It outlines 18 exam questions to test Ansible skills. Key requirements include setting up 5 virtual machines, one as the Ansible control node and 4 managed nodes. The questions cover tasks like Ansible installation, ad-hoc commands, playbooks, roles, vaults and more. Detailed solutions are provided for each question/task.
This document provides instructions for setting up Hadoop in single node mode on Ubuntu. It describes adding a Hadoop user, installing Java and SSH, downloading and extracting Hadoop, configuring environment variables and Hadoop configuration files, and formatting the NameNode.
The document provides step-by-step instructions for installing a single-node Hadoop cluster on Ubuntu Linux using VMware. It details downloading and configuring required software like Java, SSH, and Hadoop. Configuration files are edited to set properties for core Hadoop functions and enable HDFS. Finally, sample data is copied to HDFS and a word count MapReduce job is run to test the installation.
This document provides an introduction to using Ansible in a top-down approach. It discusses using Ansible to provision infrastructure including load balancers, application servers, and databases. It covers using ad-hoc commands and playbooks to configure systems. Playbooks can target groups of hosts, apply roles to automate common tasks, and allow variables to customize configurations. Selective execution allows running only certain parts of a playbook. Overall the document demonstrates how Ansible can be used to deploy and manage infrastructure and applications in a centralized, automated way.
The document discusses Hadoop and HDFS. It provides an overview of HDFS architecture and how it is designed to be highly fault tolerant and provide high throughput access to large datasets. It also discusses setting up single node and multi-node Hadoop clusters on Ubuntu Linux, including configuration, formatting, starting and stopping the clusters, and running MapReduce jobs.
Docker and friends at Linux Days 2014 in Praguetomasbart
Docker allows deploying applications easily across various environments by packaging them along with their dependencies into standardized units called containers. It provides isolation and security while allowing higher density and lower overhead than virtual machines. Core OS and Mesos both integrate with Docker to deploy containers on clusters of machines for scalability and high availability.
Software Packaging for Cross OS DistributionJian-Hong Pan
The document discusses software packaging for cross-platform distribution by building Debian packages inside a container. It provides examples of building a simple "Hello World" package and modifying an existing "nvidia-graphics-drivers" package. It demonstrates setting up a Debian environment in a Docker container, creating packages without root privileges, and implementing packaging as infrastructure code using a Dockerfile. The document promotes open source and discusses using Podman instead of Docker for containerization.
Red Hat Certified Engineer (RHCE) EX294 Exam QuestionsStudy Material
Do you want to succeed in attaining Red Hat Certified Engineer (RHCE) in one shot? Dumpspedia can do that for you. It’s no joke! We have fantastic set of several RedHat Practice Test Questions Answers to choose from. All of them extracted directly from Red Hat Certified Engineer (RHCE) exam for Red Hat Enterprise Linux 8 Test Questions. EX294 Test Questions are verified and authentic with possibilities highest as they come to be on your actual exam. We put your satisfaction on top while making a perfect collection of valid EX294 Practice Questions. Join us on our website to have a better insight.
https://www.dumpspedia.com/EX294-dumps-questions.html
A Journey to Boot Linux on Raspberry PiJian-Hong Pan
Each processor/chip architecture has its own procedure to boot the kernel. It works with desgined partition layout and vendor specific firmwares/bootloaders in the boot partition. We can learn the related knowledge from the Raspbian image for Raspberry Pi, which is the board we can obtain easily. However, the diversity between the special booting procedures with specific firmwares/bootloaders increases the complexity for distribution maintainers. It will be great if there is a way to make it more generic that can be applied to most of the chip architectures/boards to boot up the system.
After referring to some Linux distributions, we learned U-Boot may play a role in the solution. It splits the booting procedure into hardware specific and generic system parts. This helps distribution maintainers deploy the generic system with OSTree, including device trees.
Let’s deep dive into this magic booting procedure!
Practical advices how to achieve persistence in Redis. Detailed overview of all cons and pros of RDB snapshots and AOF logging. Tips and tricks for proper persistence configuration with Redis pools and master/slave replication.
Out of the box replication in postgres 9.4(pg confus)Denish Patel
This document contains notes from a presentation on PostgreSQL replication. It discusses write-ahead logs (WAL), replication history in PostgreSQL from versions 7.0 to 9.4, how to set up basic replication, tools for backups and monitoring replication, and demonstrates setting up replication without third party tools using pg_basebackup, replication slots, and pg_receivexlog. It also includes contact information for the presenter and an invitation to join the PostgreSQL Slack channel.
Vagrant, Ansible, and OpenStack on your laptopLorin Hochstein
The document discusses using Ansible and Vagrant together to easily test and deploy OpenStack. Ansible allows writing idempotent infrastructure scripts, while Vagrant allows testing them by booting reproducible virtual machines. The document provides an example of using Ansible plays to install NTP and using Vagrant to define VMs for an OpenStack controller and compute node.
The document outlines an 90 minute introduction to Ansible using Docker. It discusses setting up the environment with Docker, using ad-hoc commands and playbooks to automate tasks like installing Apache and configuring variables. Exercises demonstrate inventory management, templating configurations with Jinja2, and other core Ansible concepts. The document provides an overview but does not cover more advanced topics like dynamic inventory, roles, writing custom modules, or Ansible Tower.
This document summarizes Marian Marinov's testing and experience with various distributed filesystems including CephFS, GlusterFS, MooseFS, OrangeFS, and BeeGFS. Some key findings are:
- CephFS requires significant resources but lacks redundancy for small clusters. GlusterFS offers redundancy but can have high CPU usage.
- MooseFS and OrangeFS were easy to setup but MooseFS offered better reliability and stats.
- Performance testing found MooseFS and NFS+Ceph to have better small file creation times than GlusterFS and OrangeFS. Network latency was identified as a major factor impacting distributed filesystem performance.
- Tuning efforts focused on NFS
The document discusses various strategies for backing up and recovering PostgreSQL databases. It begins by introducing the speaker and their background. It then covers the objectives of business continuity planning. The main types of backups discussed are logical backups using pg_dump and physical/file-system level backups. Advantages and disadvantages of each approach are provided. Validation of backups through continuous testing of restores is emphasized. Automating the backup process through scripting and configuration management tools is presented as a best practice. Specific tools discussed include pg_basebackup, ZFS snapshots for file-system level backups, Barman and WAL-E for third party managed backups, and examples of custom in-house solutions.
Cumulus Linux supports great networking, what’s next? Matt Peterson (@dorkmatt) our resident expert from the office of the CTO shares his previous experience, his views on devops, and how Cumulus Networks makes it easier to manage networks with ONIE, ZTP and no CLI! “Devops is a lifestyle, shared responsibility”. With Linux as the networks OS, “it’s all just one apt-get away!”
This document provides an overview of Hadoop HDFS/MapReduce architecture, hardware requirements, installation and configuration process, monitoring options, and key components like the Namenode. It discusses configuring the Namenode, JobTracker, DataNodes, and TaskTrackers. Hardware requirements for the NameNode/JobTracker and DataNode/TaskTracker are specified. Installation can be done via tar file download or using a prebuilt rpm. Configuration involves editing XML configuration files and starting required services. Monitoring can be done via web scraping, Ganglia, or Cacti. The Namenode writes edits to RAM and write-ahead log, with checkpoints to the filesystem. High availability is experimental in YARN and Hadoop
Out of the Box Replication in Postgres 9.4(PgConfUS)Denish Patel
This document contains notes from a presentation on PostgreSQL replication. It discusses write-ahead logs (WAL), replication history in PostgreSQL from versions 7.0 to 9.4, how to set up basic replication, tools for backups and monitoring replication, and demonstrates setting up replication without third party tools using pg_basebackup, replication slots, and pg_receivexlog. Contact information is provided for the presenter and information on their employer, Medallia, is included at the end.
This document provides a summary of a presentation on becoming an accidental PostgreSQL database administrator (DBA). It covers topics like installation, configuration, connections, backups, monitoring, slow queries, and getting help. The presentation aims to help those suddenly tasked with DBA responsibilities to not panic and provides practical advice on managing a PostgreSQL database.
Joshua D. Drake
Are you tired of not having a real solution for PITR? Enter PITRTools, a single and secure solution for using Point In Time Recovery for PostgreSQL.
The document provides step-by-step instructions for installing a single-node Hadoop cluster on Ubuntu Linux using VMware. It details downloading and configuring required software like Java, SSH, and Hadoop. Configuration files are edited to set properties for core Hadoop functions and enable HDFS. Finally, sample data is copied to HDFS and a word count MapReduce job is run to test the installation.
This document provides an introduction to using Ansible in a top-down approach. It discusses using Ansible to provision infrastructure including load balancers, application servers, and databases. It covers using ad-hoc commands and playbooks to configure systems. Playbooks can target groups of hosts, apply roles to automate common tasks, and allow variables to customize configurations. Selective execution allows running only certain parts of a playbook. Overall the document demonstrates how Ansible can be used to deploy and manage infrastructure and applications in a centralized, automated way.
The document discusses Hadoop and HDFS. It provides an overview of HDFS architecture and how it is designed to be highly fault tolerant and provide high throughput access to large datasets. It also discusses setting up single node and multi-node Hadoop clusters on Ubuntu Linux, including configuration, formatting, starting and stopping the clusters, and running MapReduce jobs.
Docker and friends at Linux Days 2014 in Praguetomasbart
Docker allows deploying applications easily across various environments by packaging them along with their dependencies into standardized units called containers. It provides isolation and security while allowing higher density and lower overhead than virtual machines. Core OS and Mesos both integrate with Docker to deploy containers on clusters of machines for scalability and high availability.
Software Packaging for Cross OS DistributionJian-Hong Pan
The document discusses software packaging for cross-platform distribution by building Debian packages inside a container. It provides examples of building a simple "Hello World" package and modifying an existing "nvidia-graphics-drivers" package. It demonstrates setting up a Debian environment in a Docker container, creating packages without root privileges, and implementing packaging as infrastructure code using a Dockerfile. The document promotes open source and discusses using Podman instead of Docker for containerization.
Red Hat Certified Engineer (RHCE) EX294 Exam QuestionsStudy Material
Do you want to succeed in attaining Red Hat Certified Engineer (RHCE) in one shot? Dumpspedia can do that for you. It’s no joke! We have fantastic set of several RedHat Practice Test Questions Answers to choose from. All of them extracted directly from Red Hat Certified Engineer (RHCE) exam for Red Hat Enterprise Linux 8 Test Questions. EX294 Test Questions are verified and authentic with possibilities highest as they come to be on your actual exam. We put your satisfaction on top while making a perfect collection of valid EX294 Practice Questions. Join us on our website to have a better insight.
https://www.dumpspedia.com/EX294-dumps-questions.html
A Journey to Boot Linux on Raspberry PiJian-Hong Pan
Each processor/chip architecture has its own procedure to boot the kernel. It works with desgined partition layout and vendor specific firmwares/bootloaders in the boot partition. We can learn the related knowledge from the Raspbian image for Raspberry Pi, which is the board we can obtain easily. However, the diversity between the special booting procedures with specific firmwares/bootloaders increases the complexity for distribution maintainers. It will be great if there is a way to make it more generic that can be applied to most of the chip architectures/boards to boot up the system.
After referring to some Linux distributions, we learned U-Boot may play a role in the solution. It splits the booting procedure into hardware specific and generic system parts. This helps distribution maintainers deploy the generic system with OSTree, including device trees.
Let’s deep dive into this magic booting procedure!
Practical advices how to achieve persistence in Redis. Detailed overview of all cons and pros of RDB snapshots and AOF logging. Tips and tricks for proper persistence configuration with Redis pools and master/slave replication.
Out of the box replication in postgres 9.4(pg confus)Denish Patel
This document contains notes from a presentation on PostgreSQL replication. It discusses write-ahead logs (WAL), replication history in PostgreSQL from versions 7.0 to 9.4, how to set up basic replication, tools for backups and monitoring replication, and demonstrates setting up replication without third party tools using pg_basebackup, replication slots, and pg_receivexlog. It also includes contact information for the presenter and an invitation to join the PostgreSQL Slack channel.
Vagrant, Ansible, and OpenStack on your laptopLorin Hochstein
The document discusses using Ansible and Vagrant together to easily test and deploy OpenStack. Ansible allows writing idempotent infrastructure scripts, while Vagrant allows testing them by booting reproducible virtual machines. The document provides an example of using Ansible plays to install NTP and using Vagrant to define VMs for an OpenStack controller and compute node.
The document outlines an 90 minute introduction to Ansible using Docker. It discusses setting up the environment with Docker, using ad-hoc commands and playbooks to automate tasks like installing Apache and configuring variables. Exercises demonstrate inventory management, templating configurations with Jinja2, and other core Ansible concepts. The document provides an overview but does not cover more advanced topics like dynamic inventory, roles, writing custom modules, or Ansible Tower.
This document summarizes Marian Marinov's testing and experience with various distributed filesystems including CephFS, GlusterFS, MooseFS, OrangeFS, and BeeGFS. Some key findings are:
- CephFS requires significant resources but lacks redundancy for small clusters. GlusterFS offers redundancy but can have high CPU usage.
- MooseFS and OrangeFS were easy to setup but MooseFS offered better reliability and stats.
- Performance testing found MooseFS and NFS+Ceph to have better small file creation times than GlusterFS and OrangeFS. Network latency was identified as a major factor impacting distributed filesystem performance.
- Tuning efforts focused on NFS
The document discusses various strategies for backing up and recovering PostgreSQL databases. It begins by introducing the speaker and their background. It then covers the objectives of business continuity planning. The main types of backups discussed are logical backups using pg_dump and physical/file-system level backups. Advantages and disadvantages of each approach are provided. Validation of backups through continuous testing of restores is emphasized. Automating the backup process through scripting and configuration management tools is presented as a best practice. Specific tools discussed include pg_basebackup, ZFS snapshots for file-system level backups, Barman and WAL-E for third party managed backups, and examples of custom in-house solutions.
Cumulus Linux supports great networking, what’s next? Matt Peterson (@dorkmatt) our resident expert from the office of the CTO shares his previous experience, his views on devops, and how Cumulus Networks makes it easier to manage networks with ONIE, ZTP and no CLI! “Devops is a lifestyle, shared responsibility”. With Linux as the networks OS, “it’s all just one apt-get away!”
This document provides an overview of Hadoop HDFS/MapReduce architecture, hardware requirements, installation and configuration process, monitoring options, and key components like the Namenode. It discusses configuring the Namenode, JobTracker, DataNodes, and TaskTrackers. Hardware requirements for the NameNode/JobTracker and DataNode/TaskTracker are specified. Installation can be done via tar file download or using a prebuilt rpm. Configuration involves editing XML configuration files and starting required services. Monitoring can be done via web scraping, Ganglia, or Cacti. The Namenode writes edits to RAM and write-ahead log, with checkpoints to the filesystem. High availability is experimental in YARN and Hadoop
Out of the Box Replication in Postgres 9.4(PgConfUS)Denish Patel
This document contains notes from a presentation on PostgreSQL replication. It discusses write-ahead logs (WAL), replication history in PostgreSQL from versions 7.0 to 9.4, how to set up basic replication, tools for backups and monitoring replication, and demonstrates setting up replication without third party tools using pg_basebackup, replication slots, and pg_receivexlog. Contact information is provided for the presenter and information on their employer, Medallia, is included at the end.
This document provides a summary of a presentation on becoming an accidental PostgreSQL database administrator (DBA). It covers topics like installation, configuration, connections, backups, monitoring, slow queries, and getting help. The presentation aims to help those suddenly tasked with DBA responsibilities to not panic and provides practical advice on managing a PostgreSQL database.
Joshua D. Drake
Are you tired of not having a real solution for PITR? Enter PITRTools, a single and secure solution for using Point In Time Recovery for PostgreSQL.
This document discusses streaming replication in PostgreSQL. It covers how streaming replication works, including the write-ahead log and replication processes. It also discusses setting up replication between a primary and standby server, including configuring the servers and verifying replication is working properly. Monitoring replication is discussed along with views and functions for checking replication status. Maintenance tasks like adding or removing standbys and pausing replication are also mentioned.
Docker provides containerization capabilities while Ansible provides automation and configuration capabilities. Together they are useful DevOps tools. Docker allows building and sharing application environments while Ansible automates configuration and deployment. Key points covered include Docker concepts like images and containers, building images with Dockerfiles, and using Docker Compose to run multi-container apps. Ansible is described as a remote execution and configuration tool using YAML playbooks and roles to deploy applications. Their complementary nature makes them good DevOps partners.
PGDay.Amsterdam 2018 - Stefan Fercot - Save your data with pgBackRestPGDay.Amsterdam
Ever heard of Point-in-time recovery? pgBackRest is an awsome tool to handle backups, restores and even helps you build streaming replication ! This talk will introduce the tool, its basic features and how to use it.
This document provides an overview of PITR (Point-in-Time Recovery) and introduces PITRTools, an open source tool that simplifies implementing PITR for PostgreSQL databases. PITRTools acts as a wrapper around common utilities like rsync, ssh, and pg_standby to enable features like warm standbys, backups, failover, and monitoring alerts. It works by pushing transaction log files from a master database to a slave, and includes configuration files and scripts to initialize, start, and manage the master archiver and slave standby processes.
This document discusses puppetizing complex applications like sipXecs, an open source voice over IP telephony server. It provides an overview of Puppet and how it can be used to deploy and configure sipXecs in a repeatable, automated way. Challenges with the existing sipXecs installation and configuration are discussed. The document explores potential approaches like using test frameworks and APIs but concludes there is no perfect solution yet and engagement with upstream suppliers may be needed.
This document provides a summary of PostgreSQL 9.1 features presented at the Postgres Open conference in September 2011. It discusses synchronous and asynchronous replication, per-column collations, writable common table expressions, serialized snapshot isolation, unlogged tables, extensions, and other features. Sessions at the conference focused on unlogged tables, accelerating local search, PostgreSQL in data management, and serializable transactions.
9 steps to install and configure postgre sql from source on linuxchinkshady
The document outlines 9 steps to install and configure PostgreSQL from source on Linux:
1. Download the PostgreSQL source code
2. Install PostgreSQL by running configure, make, and make install commands
3. The configure command allows specifying various options like installation location, enabling/disabling features
4. After installation, PostgreSQL is configured and ready to use
This document discusses Docker and provides an introduction and overview. It introduces Docker concepts like Dockerfiles, commands, linking containers, volumes, port mapping and registries. It also discusses tools that can be used with Docker like Fig, Baseimage, Boot2Docker and Flynn. The document provides examples of Dockerfiles, commands and how to build, run, link and manage containers.
PostgreSQL Replication describes PostgreSQL Replicator, an open source solution for replicating PostgreSQL databases. Key features include asynchronous replication from a master to multiple slaves, supporting various replication types like role, grant, and large object replication without triggers. Replicator uses a Master Control Process to manage replication between nodes. It allows unlimited slaves without impacting the master and operates on any server.
PostgreSQL Replication describes PostgreSQL Replicator, an open source solution for replicating PostgreSQL databases. Key features include asynchronous replication from a master to multiple slaves, supporting various replication types like role, grant, and large object replication without triggers. Replicator uses a Master Control Process to manage replication between nodes. It allows unlimited slaves without impacting the master and operates on any server.
This document outlines the steps to restore a production Oracle database to a development platform for testing purposes, including restoring file system directories, shutting down the development instance, restoring the server parameter file, control file, database files, and opening the database while resetting redo logs. It also discusses modifying memory parameters, building a new Oracle Enterprise Manager repository, dropping the copied database, and the importance of backup and recovery.
Slides presented at Percona Live Europe Open Source Database Conference 2019, Amsterdam, 2019-10-01.
Imagine a world where all Wikipedia articles disappear due to a human error or software bug. Sounds unreal? According to some estimations, it would take an excess of hundreds of million person-hours to be written again. To prevent that scenario from ever happening, our SRE team at Wikimedia recently refactored the relational database recovery system.
In this session, we will discuss how we backup 550TB of MariaDB data without impacting the 15 billion page views per month we get. We will cover what were our initial plans to replace the old infrastructure, how we achieved recovering 2TB databases in less than 30 minutes while maintaining per-table granularity, as well as the different types of backups we implemented. Lastly, we will talk about lessons learned, what went well, how our original plans changed and future work.
This document introduces Docker and provides an overview of its key concepts and capabilities. It explains that Docker allows deploying applications into lightweight Linux containers that are isolated but share resources and run at native speeds. It describes how Docker uses namespaces and cgroups for isolation and copy-on-write storage for efficiency. The document also outlines common Docker workflows for building, testing, and deploying containerized applications both locally and in production environments at scale.
This document provides an overview and introduction to PostgreSQL for new users. It covers getting started with PostgreSQL, including installing it, configuring authentication and logging, upgrading to new versions, routine maintenance tasks, hardware recommendations, availability and scalability options, and query tuning and optimization. The document is presented as a slide deck with different sections labeled by letters (e.g. K-0, S-0, U-0).
This document provides an overview of PostgreSQL and instructions for installing and configuring it. It discusses using the initdb command to initialize a PostgreSQL database cluster and create the template1 and postgres databases. It also explains that the template1 database serves as a template that is copied whenever new databases are created.
The document discusses PostgreSQL replication using the Replicator tool. It provides an overview of Replicator's features, how to install, configure, and test Replicator for basic replication between a master and multiple slave databases. Instructions are given for initializing the master database, enabling replication on tables, and checking replication status.
Similar to Out of the Box Replication in Postgres 9.4(PgCon) (20)
This document discusses advanced Postgres monitoring. It begins with an introduction of the speaker and an agenda for the discussion. It then covers selection criteria for monitoring solutions, compares open source and SAAS monitoring options, and provides examples of collecting specific Postgres metrics using CollectD. It also discusses alerting, handling monitoring changes, and being prepared to respond to incidents outside of normal hours.
Out of the Box Replication in Postgres 9.4(PgCon)Denish Patel
This document provides an overview of setting up out of the box replication in PostgreSQL 9.4 without third party tools. It discusses write-ahead logs (WAL), replication slots, pg_basebackup, pg_receivexlog and monitoring replication. The presenter then demonstrates setting up replication on VMs with pg_basebackup to initialize a standby, configuration of primary and standby servers, and monitoring replication status.
This document discusses using PostgreSQL with Amazon RDS. It begins with an introduction to Amazon RDS and then discusses setting up a PostgreSQL RDS instance, available features like backups and monitoring, limitations, pricing, and references for further reading. The document is intended to provide an overview of deploying and managing PostgreSQL on Amazon RDS.
The document discusses top 10 factors to consider when choosing a database: 1) Credibility - assess reputation through independent reviews. 2) Proximity - consider location of available resources. 3) Affordability - evaluate total cost of ownership over time. 4) Maturity - review case studies and customer base. 5) Customer service - assess quality of product documentation and community support.
The document discusses scaling Postgres databases. It covers vertical and horizontal scaling techniques. Vertical scaling involves upgrading hardware resources like CPU, RAM and storage, while horizontal scaling involves adding multiple servers. The document provides tips for optimizing Postgres configuration, monitoring performance, and tuning queries.
Denish Patel deployed PostgreSQL on Amazon EC2 for a startup that had its entire IT architecture on the Amazon cloud. The initial deployment involved setting up master-slave configurations across two EC2 environments with issues like weekly instance failures and lack of monitoring. Patel then consolidated the environments, configured high availability using replication across availability zones and regions, implemented automation using Puppet, and added monitoring and backups to improve the stability and management of the PostgreSQL deployment.
The document discusses the challenges of big data including volume, velocity, variety, and veracity. It proposes using PostgreSQL and Hadoop together as a solution, with PostgreSQL serving as a multi-model database server that can handle structured, semi-structured, and unstructured data in relational, object-relational, nested-relational, array, key-value, document, and range formats. Hadoop is described as a distributed file structure that can perform ETL to convert unstructured to structured data for analytics, easily handle unstructured data like logs and streaming data, and enable parallel processing via MapReduce at the petabyte scale. The takeaway is that together, PostgreSQL and Hadoop can solve "most" big
Deploying Maximum HA Architecture With PostgreSQLDenish Patel
This document proposes a "Maximum HA architecture" for PostgreSQL that aims to provide 99.99% application uptime and reduce mean time to recovery (MTTR) for both planned and unplanned outages. It discusses using techniques like streaming replication, failover, hot backups, log shipping, PITR, and pg_reorg to achieve high availability and minimize downtime from system failures, data failures, planned maintenance, and data growth.
Deploying Maximum HA Architecture With PostgreSQLDenish Patel
This document proposes a "Maximum HA architecture" for PostgreSQL that aims to provide 99.99% application uptime and reduce mean time to recovery (MTTR) for planned and unplanned outages. It discusses using PostgreSQL features like streaming replication, failover, hot backups, WAL archiving and point-in-time recovery (PITR) to achieve high availability, prevent data loss from failures, and allow fast recovery from outages through redundancy and automation. The architecture builds upon traditional high availability techniques and aims to handle different types of failures including system failures, site failures, data failures, and planned changes through comprehensive disaster recovery planning.
This document provides information about database bloat and performance tuning. It introduces Denish Patel, a database architect with expertise in heterogeneous databases including PostgreSQL, Oracle and MySQL. The document discusses what causes database bloat, issues it can create, and tools for identifying, measuring and removing bloat. These include vacuum, vacuum full, cluster, pg_bloat_report, check_postgres_bloat, compact_table and pg_reorg. Monitoring and prevention techniques are also covered.
The document discusses achieving PCI compliance when using PostgreSQL for databases. It provides an overview of PCI requirements, how they apply to databases, and how PostgreSQL features like encryption, access control, and logging can help fulfill the requirements. Specific examples are given for how to implement encryption of cardholder data, restrict access according to the principle of least privilege, and maintain regularly updated software in PostgreSQL.
1) Oracle 10g introduces flashback query which allows users to query past states of data within a specified time period by accessing the undo logs.
2) Flashback table allows users to recover accidentally dropped tables from the recycle bin.
3) Rollback monitoring provides estimated time to complete long running transactions such as rollbacks.
Denish Patel, a database architect at OmniTi, gave a presentation on the RubyRep database replication tool. RubyRep is an asynchronous master-master and master-slave replication tool that is easy to install, platform independent, and supports large objects and tables without primary keys. It was developed as an alternative to Slony because Slony has limitations such as requiring tables to have primary keys and not supporting large objects or master-master replication. The presentation covered how to install, configure, scan, sync, and replicate databases using RubyRep.
1. /
Out of the Box Replication
In
Postgres 9.4
Denish Patel
Lead Database Architect
2. Who am I ?
• Database Architect with OmniTI for last 8+ years
• Expertise in PostgreSQL , Oracle, MySQL, NoSQL
• Contact : denish@omniti.com
• Twitter: @DenishPatel
• Blog: http://www.pateldenish.com
• Providing Solutions for business problems to deliver
• Scalability
• Reliability
• High Availability
• Security
We are hiring!!
Apply @ l42.org/lg
1
3. Agenda
! What is WAL?
! Postgres Replication History
! How you setup replication now?
! What’s missing ?
! Why replication slots?
! Demo
I’m NOT going to discuss …
! SLONY or other 3rd party replication tools
! 3rd party replication management tools
2
4. WAL - Write Ahead Log
• Roll-forward recovery aka REDO
• flushed to disk to guarantee commit durability
• sequential writes
• lower cost than flushing page cache
• Allows us to do cool things
• Crash recovery
• Binary backups
• Point-In Time Recovery
• Replication
3
5. WAL Internals (Basic)
• Automatically enabled; no action required
• Make sure to have enough space on server
• Stored under pg_xlog directory
• Normally, 16MB in size
" --with-wal-segsize config option at build
• Each segment is divided into pages (8 kB page)
" --with-wal-blocksize config option at build
• Segment name starts with..
000000010000000000000000
4
6. What is logged?
select * from pg_settings where name='wal_level';
name | wal_level
setting | hot_standby
unit |
category | Write-Ahead Log / Settings
short_desc | Set the level of information written to the WAL.
extra_desc |
context | postmaster
vartype | enum
source | configuration file
min_val |
max_val |
enumvals | { minimal, archive, hot_standby, logical }
boot_val | minimal
reset_val | hot_standby
sourcefile | /var/lib/pgsql/9.4/data/postgresql.auto.conf
sourceline | 4
5
7. What’s NOT logged?
Almost everything is replicated, but...
! unlogged tables (As name suggests)
! temporary tables
! hash indexes? (generally don’t use?)
6
30. Replication slots benefits
! Keep necessary WAL files
! Each standby can have different WAL apply status
! Single access control setup
! fsync on receiving side
29
32. How about archiving?
! Meet pg_receivexlog
! (Available since Postgers 9.1!)
pg_receivexlog
-D archivedir
--slot archivingslot
-h primaryhost -U replication
! -- synchronous option in 9.5
31
33. Postgres 9.4 – Out of the Box Replication
Are you ready to setup replication without any external
tools?
1. pg_basebackup
2. streaming with Replication Slots
3. pg_receivexlog
32
34. Tutorial – Let’s bring up VM!
! Google Drive: https://drive.google.com/open?
id=0BxnXwkT5PRBdeVByQVYySkhIemc
! Login: pgtraining/pgcon
! pgtraining user has sudo access
! You can access terminal on the desktop
! Internet should be working within VM
! Copy/paste should work between VM and Host
! Take snapshot along the process so you can rollback
easily
33
35. Installing Postgres
! Remove Postgres 8.4 version
sudo yum erase postgresql.*
! Setup yum repo for your desired version
sudo yum install http://yum.postgresql.org/9.4/redhat/rhel-6-
x86_64/pgdg-redhat94-9.4-1.noarch.rpm
! Install PostgreSQL 9.4 & Contrib modules
sudo yum install postgresql94-server postgresql94-contrib
! Create postgres cluster & initial automatic startup
sudo service postgresql-9.4 initdb
sudo chkconfig postgresql-9.4 on
sudo service postgresql-9.4 start
34
36. Create Role and Database
! Become postgres system user
sudo su - postgres
! Log into database using psql (? to see all available commands)
! Create a role & database for yourself
CREATE ROLE pgtraining WITH LOGIN SUPERUSER;
CREATE DATABASE pgtraining;
! You can login as pgtraining user (psql –U pgtraining –d pgtraining)
! Create replication role for later
CREATE ROLE replication WITH LOGIN REPLICATION;
! Set password (”replication”)
password replication
35
38. Prepare Primary DB server
alter system set wal_level = hot_standby;
alter system set archive_mode=on;
alter system set max_replication_slots=8;
alter system set archive_timeout = 60;
alter system set max_wal_senders = 8;
alter system set wal_keep_segments=100;
alter system set logging_collector=on;
37
39. Restart Primary DB server
! Restart database
sudo service postgresql-9.4 restart
! Verify settings
psql=# show max_wal_senders;
38
45. Status : pg_replication_slots
pgtraining=# select * from pg_replication_slots;
-[ RECORD 1 ]+-----------
slot_name | standby1
plugin |
slot_type | physical
datoid |
database |
active | t
xmin |
catalog_xmin |
restart_lsn | 0/270000EC
44
46. What about archiving?
! Create slot for archiving:
SELECT * FROM
pg_create_physical_replication_slot('archiver1');
! Create archive directory
mkdir /var/lib/pgsql/9.4/archive
! Start archiving process in background
/usr/pgsql-9.4/bin/pg_receivexlog -h 127.0.0.1 -p 5432 -U
replication –S 'archiver1' -n -v -D /var/lib/pgsql/9.4/
archive
! Put under init.d for continuous run
! Switch xlog : primary_db_sever# select pg_switch_xlog();
45
47. Monitoring
! Monitor Disk space
! Monitor slave lag
select pg_xlog_location_diff(sent_location, write_location) AS
byte_lag
from pg_stat_replication
where application_name='pg_receivexlog';
! Monitor WAL archive process
select pg_xlog_location_diff(sent_location, write_location) AS
byte_lag
from pg_stat_replication
where application_name='walreceiver';
46
48. Monitoring
• Number of pg_xlogs on master
• Monitor pg_recievexlog process
• Make sure archive location has new files
• pg_basebackup log files for successful backup… look
for “pg_basebackup: base backup completed”
47
49. Thanks to….
! OmniTI for travel sponsorships
! Pgcon conference committee
! Peter Eisentraut
! You!!
48