With use of modern deployment tools like Ansible, a high availability stack is built with only open-source products. Automatic fail-over, Backup- recovery, Monitoring, SSL, Virtual IP and an environment to manage the stack is part of the setup. Complete setup with industry-standard tools like Barman, Repmgr, Prometheus/Grafana, HAProxy and Keepalived. This talk will deliver a live demo of building the environment, and testing a demo application while doing a switch over, and prove that no transactions are lost.

1. True Postgres High Availability Architecture
with industry standard open-source tooling
Mike Sijmons (Nibble-IT)
maandag 21 december 2020

2. Michel Sijmons
• Co-Founder of Nibble-IT (1990)
• Postgres user since 2002
• 1st EDB Partner in Europe (2010)
• Oracle/DB2/SQL Server to Postgres Migrations
Monday, 21 December 2020

3. Agenda
• Requirements
• The Design
• The Tools
• Demo
• Wrap-up
• Questions
maandag 21 december 2020

4. Requirements
• No application reconfigure after fail-over/switch-over
• RPO = 0 after fail-over/switch-over
• Fail-over should be automatic
• Back-up en recovery
• Automatic deployment
• Monitoring
• Encryption
• Auditing
• Automatic maintenance
• Performance insights in queries
maandag 21 december 2020

5. The Design
maandag 21 december 2020

6. Design
maandag 21 december 2020

7. Application design
maandag 21 december 2020
• Detect connection problems
• Retry reconnect for a period of time
• Resubmit failed transaction
• This as extra loop around normal transaction handling

8. The Tools
maandag 21 december 2020

9. No application reconfigure
• IP Address should not change
• Virtual ip address
• Keepalived (https://www.keepalived.org)
• HAProxy (https://www.haproxy.org)
• Routing to primary server
• Xinetd daemon
maandag 21 december 2020

10. RPO = 0
• No committed transaction may be lost
• Postgres ACID properties
• Uncommitted/Inflight transactions could be retryed
• Application design
maandag 21 december 2020

11. Automatic fail-over
• Repmgr (https://repmgr.org)
• Easy management of HA stack
• Repmgrd
• Fencing of lost primary
• Possibility of witness
maandag 21 december 2020

12. Back-up and recovery
• Barman (https://www.pgbarman.org)
• Back-up catalogs
• Full/Incremental backups
• Back-up server
• Integration of pg_receivewal
maandag 21 december 2020

13. Automatic Deployment
• Ansible (https://www.ansible.com)
• Agentless architecture
• Manages Unix/Linux and Windows systems
• Can manage large number of hosts in parallel
• Extendable through modules
• Playbooks are machine and human readable
maandag 21 december 2020

14. Monitoring
• Prometheus (https://prometheus.io)
• Grafana (https://grafana.com)
• Scalable
• Collectors for Postgres, OS, Keepalived, HAProxy
• (https://github.com/wrouesnel/postgres_exporter)
(https://github.com/prometheus/node_exporter)
(https://github.com/cafebazaar/keepalived-exporter)
(* no separate exporter is needed for HAProxy Metrics)
maandag 21 december 2020

15. Encryption
• Storage layer can take care of data in rest
• Using SSL protocol for data in transit
• Possibility to inject certificates or use self-signed
• Close DBMS servers for interactive access
maandag 21 december 2020

16. Auditing
• pgAudit (https://www.pgaudit.org)
• comply with government, financial and ISO
certifications
• Easy to use
maandag 21 december 2020

17. Automatic Maintenance
• Cron job on DBMS Servers
• Clean-up logfiles
• Analyzes and vacuums tables missed by AV
• Reporting of work done
maandag 21 december 2020

18. Performance insights
• pgBadger (http://pgbadger.darold.net)
• Insights for developers
• Complements with OS/Database Metrics
maandag 21 december 2020

19. Demo
maandag 21 december 2020

20. Wrap-up
maandag 21 december 2020

21. Wrap-up
maandag 21 december 2020
• Proven reliable
• Use the strength of each tool
• The chain is only as strong as its weakest link
• Avoid manual work, script tasks
• Testing is important
• Future steps

22. maandag 21 december 2020
CentOS https://www.centos.org/
Postgresql https://www.postgresql.org/
Repmgr https://repmgr.org/
Barman https://www.pgbarman.org/
HAProxy https://www.haproxy.org/
Keepalived https://www.keepalived.org/
Prometheus https://prometheus.io/
https://github.com/wrouesnel/postgres_exporter/
https://github.com/prometheus/node_exporter/
https://github.com/cafebazaar/keepalived-exporter/
Grafana https://grafana.com/
pgAudit https://www.pgaudit.org/
pgBadger http://pgbadger.darold.net/
Ansible https://www.ansible.com/
Questions
If you want more information about the implementation, you can always contact me at
m.sijmons@nibble-it.nl