This document discusses setting up high availability for a PostgreSQL 9.0 database cluster. It describes migrating from PostgreSQL 8.3.7, which did not have native replication support, to PostgreSQL 9.0, which introduces built-in streaming replication. The author outlines configuring a master-slave replication setup with PostgreSQL 9.0 and Corosync for failover capability. Topics covered include streaming replication specifications and configuration, clustering with Corosync, backups, monitoring, and automation with Puppet.
SymfonyCon Madrid 2014 - Rock Solid Deployment of Symfony AppsPablo Godel
Web applications are becoming increasingly more complex, so deployment is not just transferring files with FTP anymore. We will go over the different challenges and how to deploy our PHP applications effectively, safely and consistently with the latest tools and techniques. We will also look at tools that complement deployment with management, configuration and monitoring.
Target audience: intermediate, beginner
a quick slide for users who work with "git", but not familiar with it.
Outline
- What is "git" ?
- Life Cycle
- How does git work?
- Better practice?
- Working with other?
- Pull, Push, Merge, behind the scene
An introduction to git, assuming very little. I introduce some core concepts, the commands used to work with them, and briefly touch on Github flow (interpreted in quite a specific way) and recap the commands used for that.
The examples could be used as exercises for a class learning git live, with a bit of fleshing out.
Two days git training with labs
First day covers git basis and essential commands
Second day covers git additional command with a big lab using a git workflow
Github - Git Training Slides: FoundationsLee Hanxue
Slide deck with detailed step breakdown that explains how git works, together with simple examples that you can try out yourself. Slides originated from http://teach.github.com/articles/course-slides/
Author: https://twitter.com/matthewmccull
A story of how we went about packaging perl and all of the dependencies that our project has.
Where we were before, the chosen path, and the end result.
The pitfalls and a view on the pros and cons of the previous state of affairs versus the pros/cons of the end result.
The slide was prepared for GIT introduction and workshop at International Islamic University Chittagong.
It was a 2 day (15th-16th Nov, 2019) Seminar and Workshop for IIUC Students by Devnet Limited.
Symfony Live NYC 2014 - Rock Solid Deployment of Symfony AppsPablo Godel
Web applications are becoming increasingly more complex, so deployment is not just transferring files with FTP anymore. We will go over the different challenges and how to deploy our PHP applications effectively, safely and consistently with the latest tools and techniques. We will also look at tools that complement deployment with management, configuration and monitoring.
This talk aims to cover a breadth of topics about package management and Chef, starting with some fundamentals and continuing on to more advanced techniques and tips.
This talk will begin by explaining why packages and package management are fundamental tenants to managing infrastructure. We'll examine why the common practice of simply running "make install" in a Chef recipe is a bad idea and what users can do when they see recipes like this in the wild.
An extremely common problem with package management is misconfiguration of package repositories and client software. Most of the existing documentation available does not cover all of the configuration required to correctly setup and access package repositories securely and lots of configurations are simply copy-and-pasted from unreliable sources.
In order to combat some of this, the talk will continue by examining some common Chef resources for controlling package repositories with care to carefully go over commonly misunderstood and misused options. We'll examine how to generate secure package repositories, what options must be set in Chef recipes to access repositories securely, and what bugs you may bump into in your infrastructure that may prevent you from securely accessing package repositories.
Finally, this talk will wrap up with some concluding tips, tricks, and thoughts about packaging and how to use it to carefully manage infrastructure.
https://youtu.be/-HJ7EZ85THU
In recent years there has been a tremendous amount of progress and innovation around tools and applications available to web developers that improve the quality, efficiency and speed of our applications, and it is hard to keep up with all of it.
A Basic Git intro presentation for SVN long timers doing their first steps in Git.
This presentation is meant to clear up most of the basic concepts which cause confusion with developers using Git as if it was an SVN.
Git 101 - Crash Course in Version Control using GitGeoff Hoffman
Find out why more and more developers are switching to Git - distributed version control. This intro to Git covers the basics, from cloning to pushing for beginners.
SymfonyCon Madrid 2014 - Rock Solid Deployment of Symfony AppsPablo Godel
Web applications are becoming increasingly more complex, so deployment is not just transferring files with FTP anymore. We will go over the different challenges and how to deploy our PHP applications effectively, safely and consistently with the latest tools and techniques. We will also look at tools that complement deployment with management, configuration and monitoring.
Target audience: intermediate, beginner
a quick slide for users who work with "git", but not familiar with it.
Outline
- What is "git" ?
- Life Cycle
- How does git work?
- Better practice?
- Working with other?
- Pull, Push, Merge, behind the scene
An introduction to git, assuming very little. I introduce some core concepts, the commands used to work with them, and briefly touch on Github flow (interpreted in quite a specific way) and recap the commands used for that.
The examples could be used as exercises for a class learning git live, with a bit of fleshing out.
Two days git training with labs
First day covers git basis and essential commands
Second day covers git additional command with a big lab using a git workflow
Github - Git Training Slides: FoundationsLee Hanxue
Slide deck with detailed step breakdown that explains how git works, together with simple examples that you can try out yourself. Slides originated from http://teach.github.com/articles/course-slides/
Author: https://twitter.com/matthewmccull
A story of how we went about packaging perl and all of the dependencies that our project has.
Where we were before, the chosen path, and the end result.
The pitfalls and a view on the pros and cons of the previous state of affairs versus the pros/cons of the end result.
The slide was prepared for GIT introduction and workshop at International Islamic University Chittagong.
It was a 2 day (15th-16th Nov, 2019) Seminar and Workshop for IIUC Students by Devnet Limited.
Symfony Live NYC 2014 - Rock Solid Deployment of Symfony AppsPablo Godel
Web applications are becoming increasingly more complex, so deployment is not just transferring files with FTP anymore. We will go over the different challenges and how to deploy our PHP applications effectively, safely and consistently with the latest tools and techniques. We will also look at tools that complement deployment with management, configuration and monitoring.
This talk aims to cover a breadth of topics about package management and Chef, starting with some fundamentals and continuing on to more advanced techniques and tips.
This talk will begin by explaining why packages and package management are fundamental tenants to managing infrastructure. We'll examine why the common practice of simply running "make install" in a Chef recipe is a bad idea and what users can do when they see recipes like this in the wild.
An extremely common problem with package management is misconfiguration of package repositories and client software. Most of the existing documentation available does not cover all of the configuration required to correctly setup and access package repositories securely and lots of configurations are simply copy-and-pasted from unreliable sources.
In order to combat some of this, the talk will continue by examining some common Chef resources for controlling package repositories with care to carefully go over commonly misunderstood and misused options. We'll examine how to generate secure package repositories, what options must be set in Chef recipes to access repositories securely, and what bugs you may bump into in your infrastructure that may prevent you from securely accessing package repositories.
Finally, this talk will wrap up with some concluding tips, tricks, and thoughts about packaging and how to use it to carefully manage infrastructure.
https://youtu.be/-HJ7EZ85THU
In recent years there has been a tremendous amount of progress and innovation around tools and applications available to web developers that improve the quality, efficiency and speed of our applications, and it is hard to keep up with all of it.
A Basic Git intro presentation for SVN long timers doing their first steps in Git.
This presentation is meant to clear up most of the basic concepts which cause confusion with developers using Git as if it was an SVN.
Git 101 - Crash Course in Version Control using GitGeoff Hoffman
Find out why more and more developers are switching to Git - distributed version control. This intro to Git covers the basics, from cloning to pushing for beginners.
PaaSTA, Yelp's platform as a service (PaaS) built on top of open source tools, provides tooling for developers to quickly turn their microservice into a monitored, highly available application spanning multiple data centers and cloud regions. Nathan Handler outlines the technologies that power PaaSTA and discusses how Yelp uses PaaSTA to empower developers and solve key problems.
Video: https://youtu.be/vISUXKeoqXM
Based on user feedback, I discuss the most requested features for PostgreSQL, their implementation status, difficulties, blockers, and future plans. Items include replication, materialized views, parallel queries, in-place upgrade.
Spring Boot to Quarkus: A real app migration experience | DevNation Tech TalkRed Hat Developers
Running a Spring Boot application but still want to benefit from Quarkus and its supersonic, subatomic Java capabilities? Me too! With a “hello world” everything looks simple, but what about a real app? Will it be easy? Or fun? In this session we’ll show our experience migrating a Spring Boot app to Quarkus. Technologies involved in the app include Hibernate, Prometheus, REST endpoints, and more. Be prepared to listen to a journey of reality, failure, and wins in the Quarkus universe.
Flux Terraform Controller is a controller for Flux to reconcile Terraform configurations in the GitOps way with the power of Flux and Terraform, Terraform Controller allows you to GitOps-ify your infrastructure, and your application resources, in the Kubernetes and Terraform universe.
Flux Terraform Controller ensures what you’ve defined in your Terraform configurations is what’s always running and available. Flux continuously looks for changes and reconciles with the desired state. Take advantage of all the benefits of GitOps: streamlined and secure deployments, quicker time to market, and more time to concentrate on app development!
Priyanka "Pinky" Ravi will walk you through how to manage Terraform configurations, the GitOps way using Flux Terraform Controller.
Liquibase få kontroll på dina databasförändringarSqueed
You never develop code without version control, why do you develop your database without it? With Liquibase, database changes are stored in human XML-files and committed to the source control system. Changes are applied to the developers local databases. As changes are committed they are distributed to all other environments including all developers local databases, test databases, staging databases, and even to production databases. This presentation will introduce you to Liquibase and the topic database change management. We will also present some advanced topics based on real life experience and a few tips and tricks as well
Rikard Thulin, Squeed and Roger Nilsson, Altran
GeoServer is an amazing project, and an amazing project to work on!
Please attend this workshop to:
* Get Started with the GeoServer codebase
* Orientation with a Tour of the GeoServer architecture
* Introduction the service dispatch framework, includin creating your own service
* Built chain and test facilities
* Create a custom function for use with map styling
* Create a custom process for use with style transformations and web processing service
* Anatomy of a successful pull request
Attendees will build their own GeoServer, learn a bit about how our community operates, and enjoy extending the base application.
If you are a developer looking to support GeoServer, or join us for a sprint or bug-stomp, this workshop is great introduction.
This course features hands-on development. We encourage and expect you to bring your favourite Java development environment.
For a good time with open source join GeoServer today!
The devops approach to monitoring, Open Source and Infrastructure as Code StyleJulien Pivotto
Monitoring is critical for every decent application that runs on production. Many of the monitoring tools widely used show their limits at the age of Infrastructure as Code and Cloud computing. Let's investigate how monitoring can face the new challenges: scalability, reproducability and automation
Built in physical and logical replication in postgresql-Firat GulecFIRAT GULEC
What is Replication?
Why do we need Replication?
How many replication layers do we have?
Understanding milestones of built-in Database Physical Replication.
What is the purpose of replication? and How to rescue system in case of failover?
What is Streaming Replication and what is its advantages? Async vs Sync, Hot standby etc.
How to configurate Master and Standby Servers? And What is the most important parameters? Example of topoloji.
What is Cascading Replication and how to configurate it? Live Demo on Terminal.
What is Logical Replication coming with PostgreSQL 10? And What is its advantages?
Logical Replication vs Physical Replication
Limitations of Logical Replication
Quorum Commit for Sync Replication etc.
What is coming up with PostgreSQL 11 about replication?
10 Questions quiz and giving some gifts to participants according to their success.
We'll give an update on how Facebook manages CentOS at scale on our fleet, how working with the community helps us solve problems at scale and touch upon some of the tooling and processes we've developed. We'll specifically focus on the challenges around upgrading the fleet to a new major release and discuss how we plan to leverage CentOS Stream in our environment.
What's New in Prometheus and Its EcosystemJulien Pivotto
Let's have a look at all the recent features and changes in the Prometheus server and the community. We will introduce the new features and see how you can integrate them in your workflows to get a better Prometheus experience.
Prometheus: What is is, what is new, what is comingJulien Pivotto
Prometheus is a metrics-based monitoring and alerting system and also the project with the second longest tenure within the CNCF. As such you have probably heard about it by now. We will give you a short introduction to Prometheus, what it is and why it was such a big deal when it was initially released. In all those years since then, the project has only gained speed, which provides us with the opportunity to tell you about all the exciting new features that have just been released or are in the pipeline, including opportunities to contribute to the project and its wider ecosystem.
Talk at kubecon 2021
Monitoring in a fast-changing world with PrometheusJulien Pivotto
Prometheus is an open source monitoring project used to gather metrics.
It as many capabilities built-in, such as service discovery, which makes it very suitable for an automated environment.
This talk will give a brief introduction of Prometheus, what are the latest developments, and then give practical tips and examples about how you can use it in an automated world.
Graphs can represent many different things. Across the years I have learned how to display different situations in Grafana effectively. I share how to visualize different kinds of situations and make them easy to read by using advanced features of Grafana.
HAProxy is often used to route ingress traffic, but we use it the other way around. We use it for egress. Our applications talk to the outside world through HAProxy. We get a lot of benefits from this unique approach: throttling, guaranteed response times, unified monitoring, and path rewriting. I will highlight how we use HAProxy at Inuits and how we achieve observability via Prometheus and Grafana.
Improved alerting with Prometheus and AlertmanagerJulien Pivotto
One of the reasons we collect metrics is to be able to alert on them. This presentation will introduce you some concepts of PromQL, prometheus and alertmanager to highly improve the quality and reliability of your alerts. This talk will cover different topic, including: - Reducing flapping alerts - Hysteresis - "Time of the day" based alerting - Computed thresholds with data history
Monitoring as an entry point for collaborationJulien Pivotto
In the last years, we have been building complex stacks, made from lots of components. All of this backed by multiple teams. This talk will present how you can use monitoring to look at the business side and have everyone looking at the same dashboards, making cooperation a reality.
his talk will introduce you to the Prometheus monitoring solution and how you can use it to monitor yous CentOS servers, and the applications that run on top of them. It will provide tips about the setup and show some great, real life example.
A small demo involving OpenShift will also be produced, to demonstrate how Prometheus can work with dynamic environments.
Automation is at the heart of modern infrastructure. Ansible is a great tool to automate your routing workflows and your infrastructure.
This talk will present you the best of Ansible - how you can quickly get started and start automating your infrastructure with it.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
2. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
The mission
Before the migration
Table of content
1 Overview
The mission
Before the migration
2 PostgreSQL 9.0
Intro
Streaming replication
Master configuration
Slave configuration
3 Clustering
Set up of corosync
OCF resource
Synthesis
4 Backups and monitoring
Backups with cron jobs
BackupPC
Nagios
Munin
5 Automation
Puppet module
The node file
6 The end
Julien Pivotto PostgreSQL 9.0 HA
3. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
The mission
Before the migration
Who am I
• Julien Pivotto
• @roidelapluie
• roidelapluie@inuits.eu
• Consultant at Inuits (Belgium) since May 2011
• FOSS defender since 2005
Julien Pivotto PostgreSQL 9.0 HA
4. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
The mission
Before the migration
A.R.S.I.A.
• Association Régionale de Santé et d’Identification Animales
(Belgique)
Julien Pivotto PostgreSQL 9.0 HA
5. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
The mission
Before the migration
A.R.S.I.A.
• Association Régionale de Santé et d’Identification Animales
(Belgique)
• 30 linux servers in several locations
Julien Pivotto PostgreSQL 9.0 HA
6. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
The mission
Before the migration
A.R.S.I.A.
• Association Régionale de Santé et d’Identification Animales
(Belgique)
• 30 linux servers in several locations
• A lot of Open Source:
• CentOS, Samba, Open-xchange, mailscanner, Cyrus,
• . . . Puppet, jenkins, foreman, OpenVPN, GLPI, rabbitmq,
• . . . BackupPC, CUPS, icinga, trac, zope, plone,
• . . . solr, pentaho, funambol, munin, squid, asterisk,
• . . . and PostgreSQL, . . .
Julien Pivotto PostgreSQL 9.0 HA
7. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
The mission
Before the migration
C.E.R.I.S.E
• A web application:
• Plone (python)
• 15k+ visits, 500k+ pages and 2.000.000+ hits each month
• Developped by another open-source company, Affinitic
Julien Pivotto PostgreSQL 9.0 HA
8. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
The mission
Before the migration
C.E.R.I.S.E
• A web application:
• Plone (python)
• 15k+ visits, 500k+ pages and 2.000.000+ hits each month
• Developped by another open-source company, Affinitic
• Several databases:
• PostgreSQL 9.0
• Oracle database
Julien Pivotto PostgreSQL 9.0 HA
9. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
The mission
Before the migration
C.E.R.I.S.E
• A web application:
• Plone (python)
• 15k+ visits, 500k+ pages and 2.000.000+ hits each month
• Developped by another open-source company, Affinitic
• Several databases:
• PostgreSQL 9.0
• Oracle database
• Several servers/services:
• Reverse proxies
• Application servers
• PostgreSQL servers
• Oracledb server
• Development server
• Pentaho server
Julien Pivotto PostgreSQL 9.0 HA
10. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
The mission
Before the migration
PostgreSQL before the migration
• PostgreSQL 8.3.7
Julien Pivotto PostgreSQL 9.0 HA
11. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
The mission
Before the migration
PostgreSQL before the migration
• PostgreSQL 8.3.7
• No native support of HA
Julien Pivotto PostgreSQL 9.0 HA
12. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
The mission
Before the migration
PostgreSQL before the migration
• PostgreSQL 8.3.7
• No native support of HA
• High availability with heartbeat 2 and DRBD
Julien Pivotto PostgreSQL 9.0 HA
13. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
The mission
Before the migration
PostgreSQL before the migration
• PostgreSQL 8.3.7
• No native support of HA
• High availability with heartbeat 2 and DRBD
• Installed on the application servers
Julien Pivotto PostgreSQL 9.0 HA
14. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
The mission
Before the migration
PostgreSQL before the migration
• PostgreSQL 8.3.7
• No native support of HA
• High availability with heartbeat 2 and DRBD
• Installed on the application servers
• Nothing automated
Julien Pivotto PostgreSQL 9.0 HA
15. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
The mission
Before the migration
PostgreSQL before the migration
• PostgreSQL 8.3.7
• No native support of HA
• High availability with heartbeat 2 and DRBD
• Installed on the application servers
• Nothing automated
• Failover: Passive node is not even read only
Julien Pivotto PostgreSQL 9.0 HA
16. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
The mission
Before the migration
Monitoring before the installation
• Icinga
• Check of the DRBD
• Simple connection check to PostgreSQL
Julien Pivotto PostgreSQL 9.0 HA
17. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
The mission
Before the migration
Monitoring before the installation
• Icinga
• Check of the DRBD
• Simple connection check to PostgreSQL
• Graphing with Cacti:
• Size of the databases
• Connexions to the database
• Checkpoints
Julien Pivotto PostgreSQL 9.0 HA
18. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
The mission
Before the migration
Backups before the installation
• Hourly local backups
• External backups (on disk and on tape)
• Made with pg_dump command
• External backups tool: BackupPC
Julien Pivotto PostgreSQL 9.0 HA
19. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
PostgreSQL 9.0
• Released in september 2010
Julien Pivotto PostgreSQL 9.0 HA
20. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
PostgreSQL 9.0
• Released in september 2010
• First version with native replication
Julien Pivotto PostgreSQL 9.0 HA
21. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
PostgreSQL 9.0
• Released in september 2010
• First version with native replication
• No native failover tool
Julien Pivotto PostgreSQL 9.0 HA
22. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
PostgreSQL 9.0
• Released in september 2010
• First version with native replication
• No native failover tool
• We need to use PostgreSQL + Corosync
Julien Pivotto PostgreSQL 9.0 HA
23. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Write-Ahead Logging
• Every change to datafile is first written into a log file
Julien Pivotto PostgreSQL 9.0 HA
24. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Write-Ahead Logging
• Every change to datafile is first written into a log file
• Less disk writes
Julien Pivotto PostgreSQL 9.0 HA
25. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Write-Ahead Logging
• Every change to datafile is first written into a log file
• Less disk writes
• More cohesion
Julien Pivotto PostgreSQL 9.0 HA
26. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Write-Ahead Logging
• Every change to datafile is first written into a log file
• Less disk writes
• More cohesion
• Logs can be replayed
Julien Pivotto PostgreSQL 9.0 HA
27. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
What is streaming replication
• Ship and apply WAL XLOGS to standby servers
Julien Pivotto PostgreSQL 9.0 HA
28. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
What is streaming replication
• Ship and apply WAL XLOGS to standby servers
• Possibility to have multiple standby servers
Julien Pivotto PostgreSQL 9.0 HA
29. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
What is streaming replication
• Ship and apply WAL XLOGS to standby servers
• Possibility to have multiple standby servers
• Standby servers can be read-only ("Hot standby")
Julien Pivotto PostgreSQL 9.0 HA
30. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Disadvantages of streaming replication
Julien Pivotto PostgreSQL 9.0 HA
31. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Disadvantages of streaming replication
• Asynchronous log-shipping
Julien Pivotto PostgreSQL 9.0 HA
32. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Disadvantages of streaming replication
• Asynchronous log-shipping
• No native HA features in PostgreSQL
Julien Pivotto PostgreSQL 9.0 HA
33. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Disadvantages of streaming replication
• Asynchronous log-shipping
• No native HA features in PostgreSQL
• Single-threaded replication
Julien Pivotto PostgreSQL 9.0 HA
34. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Specifications of streaming replication
• Asynchronous log-shipping
• Delay is often very close to 0
• No native HA features in PostgreSQL
• Single-threaded replication
Julien Pivotto PostgreSQL 9.0 HA
35. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Specifications of streaming replication
• Asynchronous log-shipping
• Delay is often very close to 0
• No native HA features in PostgreSQL
• We will use corosync
• Single-threaded replication
Julien Pivotto PostgreSQL 9.0 HA
36. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Specifications of streaming replication
• Asynchronous log-shipping
• Delay is often very close to 0
• No native HA features in PostgreSQL
• We will use corosync
• Single-threaded replication
• Single-threaded replication. . .
Julien Pivotto PostgreSQL 9.0 HA
37. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Master configuration
Only one specific configuration file.
Julien Pivotto PostgreSQL 9.0 HA
38. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Master configuration
Only one specific configuration file.
Configuration related to SR
wal_level = hot_standby
max_wal_senders = 2
wal_keep_segments = 128
Julien Pivotto PostgreSQL 9.0 HA
39. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Master configuration
• wal_level = hot_standby
Julien Pivotto PostgreSQL 9.0 HA
40. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Master configuration
• wal_level = hot_standby
Allows stanby server to be readable
Julien Pivotto PostgreSQL 9.0 HA
41. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Master configuration
• wal_level = hot_standby
Allows stanby server to be readable
• max_wal_senders = 2
Julien Pivotto PostgreSQL 9.0 HA
42. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Master configuration
• wal_level = hot_standby
Allows stanby server to be readable
• max_wal_senders = 2
We allow up to 2 standby nodes
Julien Pivotto PostgreSQL 9.0 HA
43. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Master configuration
• wal_level = hot_standby
Allows stanby server to be readable
• max_wal_senders = 2
We allow up to 2 standby nodes
• wal_keep_segments = 128
Julien Pivotto PostgreSQL 9.0 HA
44. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Master configuration
• wal_level = hot_standby
Allows stanby server to be readable
• max_wal_senders = 2
We allow up to 2 standby nodes
• wal_keep_segments = 128
The minimum wal segments to keep
Julien Pivotto PostgreSQL 9.0 HA
45. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Slave configuration
• Two configuration files
Julien Pivotto PostgreSQL 9.0 HA
46. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Slave configuration
• Two configuration files
• PostgreSQL.conf file
Julien Pivotto PostgreSQL 9.0 HA
47. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Slave configuration
• Two configuration files
• PostgreSQL.conf file
• Recovery.conf file
Julien Pivotto PostgreSQL 9.0 HA
48. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Slave configuration
• Two configuration files
• PostgreSQL.conf file
• Recovery.conf file
• A trigger file to stop replication can be specified
Julien Pivotto PostgreSQL 9.0 HA
49. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Slave configuration
• Two configuration files
• PostgreSQL.conf file
• Recovery.conf file
• A trigger file to stop replication can be specified
PostgreSQL.conf - Configuration related to SR
wal_level = hot_standby
hot_standby = on
Julien Pivotto PostgreSQL 9.0 HA
50. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Slave configuration
recovery.conf
standby_mode = ‘on’
primary_conninfo = ‘host=192.168.142.31 user=replicuser’
Julien Pivotto PostgreSQL 9.0 HA
51. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Slave configuration
recovery.conf
standby_mode = ‘on’
primary_conninfo = ‘host=192.168.142.31 user=replicuser’
• standby_mode = standby server
Julien Pivotto PostgreSQL 9.0 HA
52. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Slave configuration
recovery.conf
standby_mode = ‘on’
primary_conninfo = ‘host=192.168.142.31 user=replicuser’
• standby_mode = standby server
• primary_conninfo = connection to the master
Julien Pivotto PostgreSQL 9.0 HA
53. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Replication user
• Super user called replication is created
Julien Pivotto PostgreSQL 9.0 HA
54. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Replication user
• Super user called replication is created
SQL query to create super user
CREATE USER replication SUPERUSER LOGIN CONNECTION LIMIT 1 ENCRYPTED PASSWORD ‘foobar’;
Julien Pivotto PostgreSQL 9.0 HA
55. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
pg_hba.conf
• pg_hba.conf contains some kind of ACLs for the PostgreSQL
connections
Julien Pivotto PostgreSQL 9.0 HA
56. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
pg_hba.conf
• pg_hba.conf contains some kind of ACLs for the PostgreSQL
connections
• Need to add the nodes as trusted.
Julien Pivotto PostgreSQL 9.0 HA
57. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
pg_hba.conf
• pg_hba.conf contains some kind of ACLs for the PostgreSQL
connections
• Need to add the nodes as trusted.
pg_hba.conf
hostnossl all all 192.168.142.31/32 trust
hostnossl all all 192.168.142.31/32 trust
hostnossl replication replicuser 192.168.142.32/32 trust
Julien Pivotto PostgreSQL 9.0 HA
58. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Setting up a slave
• 3 commands required to add a slave
Julien Pivotto PostgreSQL 9.0 HA
59. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
Setting up a slave
• 3 commands required to add a slave
Adding a standby server
psql -c "SELECT pg_start_backup(’label’, true)"
rsync -a ${PGDATA}/ standby:/srv/pgsql/standby/ --exclude postmaster.pid --exclude ‘*-master’
--exclude ‘*-slave’
psql -c "SELECT pg_stop_backup()"
Julien Pivotto PostgreSQL 9.0 HA
60. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Set up of corosync
OCF resource
Synthesis
Corosync configuration
• Goal: switching between master/slave
Julien Pivotto PostgreSQL 9.0 HA
61. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Set up of corosync
OCF resource
Synthesis
Corosync configuration
• Goal: switching between master/slave
• One and only one node should be master
Julien Pivotto PostgreSQL 9.0 HA
62. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Set up of corosync
OCF resource
Synthesis
Corosync configuration
• Goal: switching between master/slave
• One and only one node should be master
• Master need to be connected to the router
Julien Pivotto PostgreSQL 9.0 HA
63. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Set up of corosync
OCF resource
Synthesis
Corosync configuration
• Goal: switching between master/slave
• One and only one node should be master
• Master need to be connected to the router
• Custom master/slave ocf resource
Julien Pivotto PostgreSQL 9.0 HA
64. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Set up of corosync
OCF resource
Synthesis
crm.conf
• Main configuration file of corosync is
/etc/corosync/crm.conf.
• Contains all the resources/nodes/etc. . .
Julien Pivotto PostgreSQL 9.0 HA
65. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Set up of corosync
OCF resource
Synthesis
crm.conf
• Main configuration file of corosync is
/etc/corosync/crm.conf.
• Contains all the resources/nodes/etc. . .
Defining the nodes
node dupont.demo
attributes standby="off"
node dupond.demo
attributes standby="on"
Julien Pivotto PostgreSQL 9.0 HA
66. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Set up of corosync
OCF resource
Synthesis
crm.conf
Primitives
primitive pgsql ocf:inuits:pgsql-ms
primitive virt_ip ocf:heartbeat:IPaddr2
params nic="eth1" iflabel="0" ip="192.168.142.30" cidr_netmask="24" broadcast="192.168.142.255"
meta target-role="Started" is-managed="true"
primitive ping ocf:pacemaker:ping
params host_list="192.168.142.1"
op monitor interval="10s" timeout="10s"
op start interval="0" timeout="45s"
op stop interval="0" timeout="50s"
Julien Pivotto PostgreSQL 9.0 HA
67. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Set up of corosync
OCF resource
Synthesis
crm.conf
Primitives
primitive pgsql ocf:inuits:pgsql-ms
primitive virt_ip ocf:heartbeat:IPaddr2
params nic="eth1" iflabel="0" ip="192.168.142.30" cidr_netmask="24" broadcast="192.168.142.255"
meta target-role="Started" is-managed="true"
primitive ping ocf:pacemaker:ping
params host_list="192.168.142.1"
op monitor interval="10s" timeout="10s"
op start interval="0" timeout="45s"
op stop interval="0" timeout="50s"
• 3 primitives:
Julien Pivotto PostgreSQL 9.0 HA
68. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Set up of corosync
OCF resource
Synthesis
crm.conf
Primitives
primitive pgsql ocf:inuits:pgsql-ms
primitive virt_ip ocf:heartbeat:IPaddr2
params nic="eth1" iflabel="0" ip="192.168.142.30" cidr_netmask="24" broadcast="192.168.142.255"
meta target-role="Started" is-managed="true"
primitive ping ocf:pacemaker:ping
params host_list="192.168.142.1"
op monitor interval="10s" timeout="10s"
op start interval="0" timeout="45s"
op stop interval="0" timeout="50s"
• 3 primitives:
• pgsql, the PostgreSQL primitive
Julien Pivotto PostgreSQL 9.0 HA
69. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Set up of corosync
OCF resource
Synthesis
crm.conf
Primitives
primitive pgsql ocf:inuits:pgsql-ms
primitive virt_ip ocf:heartbeat:IPaddr2
params nic="eth1" iflabel="0" ip="192.168.142.30" cidr_netmask="24" broadcast="192.168.142.255"
meta target-role="Started" is-managed="true"
primitive ping ocf:pacemaker:ping
params host_list="192.168.142.1"
op monitor interval="10s" timeout="10s"
op start interval="0" timeout="45s"
op stop interval="0" timeout="50s"
• 3 primitives:
• pgsql, the PostgreSQL primitive
• virt_ip, the floating IP address
Julien Pivotto PostgreSQL 9.0 HA
70. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Set up of corosync
OCF resource
Synthesis
crm.conf
Primitives
primitive pgsql ocf:inuits:pgsql-ms
primitive virt_ip ocf:heartbeat:IPaddr2
params nic="eth1" iflabel="0" ip="192.168.142.30" cidr_netmask="24" broadcast="192.168.142.255"
meta target-role="Started" is-managed="true"
primitive ping ocf:pacemaker:ping
params host_list="192.168.142.1"
op monitor interval="10s" timeout="10s"
op start interval="0" timeout="45s"
op stop interval="0" timeout="50s"
• 3 primitives:
• pgsql, the PostgreSQL primitive
• virt_ip, the floating IP address
• ping, the primitive that will check that the servers are
connected to the router
Julien Pivotto PostgreSQL 9.0 HA
71. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Set up of corosync
OCF resource
Synthesis
crm.conf
Configuring the primitives
ms pgsql-ms pgsql
params pgsqlconfig="/var/lib/pgsql/data/postgresql.conf"
lsb_script="/etc/init.d/postgresql-9.0"
pgsqlrecovery="/var/lib/pgsql/data/recovery.conf"
meta clone-max="2" clone-node-max="1" master-max="1" master-node-max="1" notify="false"
clone clone-ping ping
meta globally-unique="false"
Julien Pivotto PostgreSQL 9.0 HA
72. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Set up of corosync
OCF resource
Synthesis
crm.conf
Configuring the primitives
ms pgsql-ms pgsql
params pgsqlconfig="/var/lib/pgsql/data/postgresql.conf"
lsb_script="/etc/init.d/postgresql-9.0"
pgsqlrecovery="/var/lib/pgsql/data/recovery.conf"
meta clone-max="2" clone-node-max="1" master-max="1" master-node-max="1" notify="false"
clone clone-ping ping
meta globally-unique="false"
• PostgreSQL M/S configuration: the init script, the
configuration files. . .
Julien Pivotto PostgreSQL 9.0 HA
73. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Set up of corosync
OCF resource
Synthesis
crm.conf
Configuring the primitives
ms pgsql-ms pgsql
params pgsqlconfig="/var/lib/pgsql/data/postgresql.conf"
lsb_script="/etc/init.d/postgresql-9.0"
pgsqlrecovery="/var/lib/pgsql/data/recovery.conf"
meta clone-max="2" clone-node-max="1" master-max="1" master-node-max="1" notify="false"
clone clone-ping ping
meta globally-unique="false"
• PostgreSQL M/S configuration: the init script, the
configuration files. . .
• Ping resource as a clone (launched on both servers)
Julien Pivotto PostgreSQL 9.0 HA
74. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Set up of corosync
OCF resource
Synthesis
crm.conf
group and colocation
group PSQL virt_ip
location connected PSQL
rule $id="connected-rule" -inf: not_defined pingd or pingd lte 0
colocation ip_psql inf: PSQL pgsql-ms:Master
property $id="cib-bootstrap-options"
cluster-infrastructure="openais"
expected-quorum-votes="2"
stonith-enabled="false"
no-quorum-policy="ignore"
default-resource-stickiness="INFINITY"
rsc_defaults $id="rsc_defaults-options"
migration-threshold="INFINITY"
failure-timeout="10"
resource-stickiness="INFINITY"
• Master should always be on the same node as the floating IP
address
Julien Pivotto PostgreSQL 9.0 HA
75. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Set up of corosync
OCF resource
Synthesis
crm.conf
group and colocation
group PSQL virt_ip
location connected PSQL
rule $id="connected-rule" -inf: not_defined pingd or pingd lte 0
colocation ip_psql inf: PSQL pgsql-ms:Master
property $id="cib-bootstrap-options"
cluster-infrastructure="openais"
expected-quorum-votes="2"
stonith-enabled="false"
no-quorum-policy="ignore"
default-resource-stickiness="INFINITY"
rsc_defaults $id="rsc_defaults-options"
migration-threshold="INFINITY"
failure-timeout="10"
resource-stickiness="INFINITY"
• Master should always be on the same node as the floating IP
address
• Master should be connected to the router
Julien Pivotto PostgreSQL 9.0 HA
76. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Set up of corosync
OCF resource
Synthesis
OCF resource
• Custom OCF resource to manage the master/slave
PostgreSQL
Julien Pivotto PostgreSQL 9.0 HA
77. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Set up of corosync
OCF resource
Synthesis
OCF resource
• Custom OCF resource to manage the master/slave
PostgreSQL
• Based on an example of resource written by Andrew Beekhof
from Clusterlabs
Julien Pivotto PostgreSQL 9.0 HA
78. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Set up of corosync
OCF resource
Synthesis
OCF resource
• Custom OCF resource to manage the master/slave
PostgreSQL
• Based on an example of resource written by Andrew Beekhof
from Clusterlabs
• The file is in /usr/lib/ocf/resource.d/inuits/pgsql-ms
• Available on github!
https://github.com/roidelapluie/puppet-cluster
Julien Pivotto PostgreSQL 9.0 HA
79. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Set up of corosync
OCF resource
Synthesis
OCF resource
• The script does the following:
Julien Pivotto PostgreSQL 9.0 HA
80. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Set up of corosync
OCF resource
Synthesis
OCF resource
• The script does the following:
• Move the postgresql.conf-master to postgresql.conf
when a node is promoted/master
Julien Pivotto PostgreSQL 9.0 HA
81. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Set up of corosync
OCF resource
Synthesis
OCF resource
• The script does the following:
• Move the postgresql.conf-master to postgresql.conf
when a node is promoted/master
• Move the postgresql.conf-slave to postgresql.conf
when a node is depromoted/slave
Julien Pivotto PostgreSQL 9.0 HA
82. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Set up of corosync
OCF resource
Synthesis
OCF resource
• The script does the following:
• Move the postgresql.conf-master to postgresql.conf
when a node is promoted/master
• Move the postgresql.conf-slave to postgresql.conf
when a node is depromoted/slave
• Ensure that recovery.conf-slave is on recovery.conf on
slave and absent on master
Julien Pivotto PostgreSQL 9.0 HA
83. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Set up of corosync
OCF resource
Synthesis
OCF resource
• The script does the following:
• Move the postgresql.conf-master to postgresql.conf
when a node is promoted/master
• Move the postgresql.conf-slave to postgresql.conf
when a node is depromoted/slave
• Ensure that recovery.conf-slave is on recovery.conf on
slave and absent on master
• Starts/restarts PostgreSQL when needed.
Julien Pivotto PostgreSQL 9.0 HA
87. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Backups with cron jobs
BackupPC
Nagios
Munin
Backups of the databases
• The databases are quite small (a few gigabytes)
• Hourly on each node (one at minute 0 and one at minute 30)
• Daily external backup of each node
• We keep 24 hourly backups and 7 daily backups on disk
• BackupPC: we keep months of backups
Julien Pivotto PostgreSQL 9.0 HA
88. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Backups with cron jobs
BackupPC
Nagios
Munin
Hourly backup script
/usr/local/bin/backup_hourly.sh
#!/bin/bash
DATE=$(date +%H)
BACKUP_PATH=/var/lib/backups/hourly
for db in foobar_db foobar2_db
do
/usr/bin/pg_dump $db | gzip > $BACKUP_PATH/${db}_$DATE.pgsql.gz
ln -fs $BACKUP_PATH/${db}_$DATE.pgsql.gz $BACKUP_PATH/${db}_current.pgsql.gz
done
Julien Pivotto PostgreSQL 9.0 HA
89. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Backups with cron jobs
BackupPC
Nagios
Munin
Hourly backup script
/usr/local/bin/backup_hourly.sh
#!/bin/bash
DATE=$(date +%H)
BACKUP_PATH=/var/lib/backups/hourly
for db in foobar_db foobar2_db
do
/usr/bin/pg_dump $db | gzip > $BACKUP_PATH/${db}_$DATE.pgsql.gz
ln -fs $BACKUP_PATH/${db}_$DATE.pgsql.gz $BACKUP_PATH/${db}_current.pgsql.gz
done
The daily script is almost the same.
Julien Pivotto PostgreSQL 9.0 HA
90. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Backups with cron jobs
BackupPC
Nagios
Munin
BackupPC script
/usr/local/bin/backup_backuppc.sh
#!/bin/bash
DATE=$(date +%u)
BACKUP_PATH=/var/lib/backups/backuppc
for db in cerise trackitquality trackit zodb_cerise
do
/usr/bin/pg_dump -U postgres $db | gzip > $BACKUP_PATH/${db}_$DATE.pgsql.gz
ln -fs $BACKUP_PATH/${db}_$DATE.pgsql.gz $BACKUP_PATH/${db}_current.pgsql.gz
done
Julien Pivotto PostgreSQL 9.0 HA
91. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Backups with cron jobs
BackupPC
Nagios
Munin
BackupPC script
/usr/local/bin/backup_backuppc.sh
#!/bin/bash
DATE=$(date +%u)
BACKUP_PATH=/var/lib/backups/backuppc
for db in cerise trackitquality trackit zodb_cerise
do
/usr/bin/pg_dump -U postgres $db | gzip > $BACKUP_PATH/${db}_$DATE.pgsql.gz
ln -fs $BACKUP_PATH/${db}_$DATE.pgsql.gz $BACKUP_PATH/${db}_current.pgsql.gz
done
In the backupPC config, I added the following:
BackupPC config
$Conf{DumpPreUserCmd} = ‘$sshPath -t -q -x -l backuppc $host /usr/local/bin/backup_backuppc.sh’;
Julien Pivotto PostgreSQL 9.0 HA
92. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Backups with cron jobs
BackupPC
Nagios
Munin
check_postgres script
• check_postgres.pl is a nagios-compatible perl script
Julien Pivotto PostgreSQL 9.0 HA
93. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Backups with cron jobs
BackupPC
Nagios
Munin
check_postgres script
• check_postgres.pl is a nagios-compatible perl script
• Available on http://www.bucardo.org/check_postgres/
and on Github
Julien Pivotto PostgreSQL 9.0 HA
94. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Backups with cron jobs
BackupPC
Nagios
Munin
check_postgres script
• check_postgres.pl is a nagios-compatible perl script
• Available on http://www.bucardo.org/check_postgres/
and on Github
• Checks:
Julien Pivotto PostgreSQL 9.0 HA
95. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Backups with cron jobs
BackupPC
Nagios
Munin
check_postgres script
• check_postgres.pl is a nagios-compatible perl script
• Available on http://www.bucardo.org/check_postgres/
and on Github
• Checks:
• Current connections number
Julien Pivotto PostgreSQL 9.0 HA
96. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Backups with cron jobs
BackupPC
Nagios
Munin
check_postgres script
• check_postgres.pl is a nagios-compatible perl script
• Available on http://www.bucardo.org/check_postgres/
and on Github
• Checks:
• Current connections number
• Replication delay
Julien Pivotto PostgreSQL 9.0 HA
97. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Backups with cron jobs
BackupPC
Nagios
Munin
Check hot_standby latency
• The check_postgres.pl script has a check for hot_standby
delay
Julien Pivotto PostgreSQL 9.0 HA
98. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Backups with cron jobs
BackupPC
Nagios
Munin
Check hot_standby latency
• The check_postgres.pl script has a check for hot_standby
delay
• We do not know who is the master and the slave
Julien Pivotto PostgreSQL 9.0 HA
99. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Backups with cron jobs
BackupPC
Nagios
Munin
Check hot_standby latency
• The check_postgres.pl script has a check for hot_standby
delay
• We do not know who is the master and the slave
• Here is a bash script I wrote to know the M/S order
Julien Pivotto PostgreSQL 9.0 HA
100. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Backups with cron jobs
BackupPC
Nagios
Munin
Check hot_standby latency
• The check_postgres.pl script has a check for hot_standby
delay
• We do not know who is the master and the slave
• Here is a bash script I wrote to know the M/S order
Master/slave replication check
#!/bin/bash
/usr/lib64/nagios/plugins/check_postgres.pl --db="$1"
--action hot_standby_delay -w 300 -c 600 --host=$(
crm_resource --resource pgsql-ms --locate|
awk ‘/Master/ {master=$6} / $/ {slave=$6} END {print master","slave}’
)
Julien Pivotto PostgreSQL 9.0 HA
101. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Backups with cron jobs
BackupPC
Nagios
Munin
Munin postgres scripts
• Munin is shipped with perl plugins for postgresql
Julien Pivotto PostgreSQL 9.0 HA
102. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Backups with cron jobs
BackupPC
Nagios
Munin
Munin postgres scripts
• Munin is shipped with perl plugins for postgresql
• We use four of them:
Julien Pivotto PostgreSQL 9.0 HA
103. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Backups with cron jobs
BackupPC
Nagios
Munin
Munin postgres scripts
• Munin is shipped with perl plugins for postgresql
• We use four of them:
• postgres_size,
• postgres_checkpoints,
• postgres_connections_db,
• postgres_cache
Julien Pivotto PostgreSQL 9.0 HA
105. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Puppet module
The node file
Puppet module
• Puppet-postgres module forked from Kris Buytaert’s github
page
Julien Pivotto PostgreSQL 9.0 HA
106. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Puppet module
The node file
Puppet module
• Puppet-postgres module forked from Kris Buytaert’s github
page
• Forked to remove all references to services
Julien Pivotto PostgreSQL 9.0 HA
107. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Puppet module
The node file
Puppet module
• Puppet-postgres module forked from Kris Buytaert’s github
page
• Forked to remove all references to services
• Creates the users, the super users, the databases
Julien Pivotto PostgreSQL 9.0 HA
108. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Puppet module
The node file
Puppet module
• Puppet-postgres module forked from Kris Buytaert’s github
page
• Forked to remove all references to services
• Creates the users, the super users, the databases
• Parameterized classes
Julien Pivotto PostgreSQL 9.0 HA
109. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Puppet module
The node file
Puppet module
• Puppet-postgres module forked from Kris Buytaert’s github
page
• Forked to remove all references to services
• Creates the users, the super users, the databases
• Parameterized classes
• The cache sizes are parameterized too (useful for Vagrant
boxes)
Julien Pivotto PostgreSQL 9.0 HA
111. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Puppet module
The node file
Example in the node file
Here is the result in the node file:
Julien Pivotto PostgreSQL 9.0 HA
112. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Puppet module
The node file
Example in the node file
Here is the result in the node file:
site.pp
node postgres-server {
include postgres
include site
class {
’cluster’:
cluster_bind_interface => ’eth1’,
cluster_mcastaddr => ’226.94.1.1’,
}
postgres::hba {
$::fqdn:
password => ’yes’,
allowedrules => [
"host all all $::ipaddress/32 password",
’hostnossl replication replicuser 192.168.142.31/24 trust’,
’hostnossl replication replicuser 192.168.142.32/24 trust’,
’hostnossl all all 192.168.142.0/24 password’,
]
}
include site::sshkeys
Postgres::Cluster[$::fqdn] -> Class[’cluster’]
}
Julien Pivotto PostgreSQL 9.0 HA
113. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Puppet module
The node file
Example in the node file
dupont.pp
node dupont inherits postgres-server {
postgres::cluster {
$::fqdn:
other_node_ip => ’192.168.142.31’,
listen => ’*’,
max_connections => ’100’,
shared_buffers => ’120MB’,
work_mem => ’3MB’,
effective_cache_size => ’352MB’,
}
class {
’postgres::firstsync’:
remotehost => ’dupont’,
password => ’rmll2012’,
}
Class[’postgres’] -> Postgres::Cluster[$::fqdn]
Julien Pivotto PostgreSQL 9.0 HA
114. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Puppet module
The node file
Example in the node file
dupont.pp
exec {
’/bin/sleep 10’:
refreshonly => true,
subscribe => Exec[’load_crm_config’],
}
exec {
’crm node online dupond.demo’:
refreshonly => true,
path => $::path,
subscribe => Exec[’load_crm_config’],
require => Exec[’/bin/sleep 10’, ’crm node online dupond.demo’],
}
Class[’site’] -> Class[’cluster’] -> Class[’postgres::firstsync’]
Class[’postgres::firstsync’] -> Postgres::Hba[$::fqdn]
Class[’cluster’] -> Exec[’/bin/sleep 10’] -> Class[’postgres::firstsync’]
Julien Pivotto PostgreSQL 9.0 HA
115. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Puppet module
The node file
Example in the node file
dupont.pp
postgres::createuser {
’tintin’:
password => ’rmll2012’,
host => ’192.168.142.30’,
passwd => ’milou’,
require => Exec[’/bin/sleep 10’],
}
postgres::createdb {
’moulinsart’:
password => ’rmll2012’,
host => ’192.168.142.30’,
owner => ’tintin’,
require => [Exec[’/bin/sleep 10’],Postgres::Createuser[’tintin’]],
}
postgres::createsuperuser {
’replicuser’:
password => ’rmll2012’,
host => ’192.168.142.30’,
passwd => ’icanhazapassword’,
require => Exec[’/bin/sleep 10’],
}
}
Julien Pivotto PostgreSQL 9.0 HA
116. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
Test it yourself
You will need vagrant (http://vagrantup.com).
Test-it-yourself howto
$ git clone
git://github.com/roidelapluie/postgres9-demo-vagrant.git
$ cd postgres9-demo-vagrant
$ git submodule update --init
$ vagrant up
Julien Pivotto PostgreSQL 9.0 HA
118. ;
Overview
PostgreSQL 9.0
Clustering
Backups and monitoring
Automation
The end
• Presentation made with latex-beamer
• PostgreSQL logo is used following the policy
http://wiki.postgresql.org/wiki/Trademark_Policy
• Tango icons have been released into the public domain
• Database icon is under CC BY-SA 3.0
http://dracos.deviantart.com/#/d2y5ele
• RMLL logo is under CC BY-SA 3.0
http://2012.rmll.info/kit-media
Julien Pivotto PostgreSQL 9.0 HA