Webinar slides: How to Automate & Manage PostgreSQL with ClusterControl

1. January 2019 Automation & Management of PostgreSQL with ClusterControl Sebastian Insausti Presenter sebastian@severalnines.com

2. Copyright 2018 Severalnines AB I'm Jean-Jérôme from the Severalnines Team and I'm your host for today's webinar! Feel free to ask any questions in the Questions section of this application or via the Chat box. You can also contact me directly via the chat box or via email: info@severalnines.com during or after the webinar. Your host & some logistics

5. Copyright 2018 Severalnines AB Automation & Management Deployment (Free Community) ● Deploy a Cluster in Minutes ○ On-Prem ○ Cloud (AWS/Azure/Google) - paid Monitoring (Free Community) ● Systems View with 1 sec Resolution ● DB / OS stats & Performance Advisors ● Configurable Dashboards ● Query Analyzer ● Agentless & agent-based Management (Paid Features) ● Backup Management ● Upgrades & Patching ● Security & Compliance ● Operational Reports ● Automatic Recovery & Repair ● Performance Management ● Automatic Performance Advisors

8. Poll - multiple choice Copyright 2018 Severalnines AB Which method do you use to perform your DBA tasks? ● Manual actions ● Custom scripts ● External tools and utilities ● ClusterControl ● Others (please specify in the chat box)

9. January 2019 Automation & Management of PostgreSQL with ClusterControl Sebastian Insausti Presenter sebastian@severalnines.com

10. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB Managing PostgreSQL ● Requirements for managing production databases “The Old Way” ● Labour intensive, manual operations ● Scripts, tools and utilities ● What happens when the DBA leaves? Agenda Automation and Management with ClusterControl ● Deployment ● Monitoring & Alerting ● Performance Management ● Backup management ● High Availability setups ● Single VIP endpoint with Load Balancer ● Automatic Failover Live Demo

12. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB Deployment ● Install ○ The first step to begin the game ● Secure ○ Keep your data safe and secure ● Configure ○ Environment requirements

13. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB Monitoring & Alerting ● Define metrics ○ Database ○ Operating System ● Configure monitoring ○ Custom Monitoring ○ System Monitoring ● Configure alerts ○ Email alerts ○ Tool Integrations

14. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB ● Important but complex task ● PostgreSQL is configured with compatibility and stability by default ● Depends on the hardware and on the system itself ● Performance queries Performance Management

15. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB ● Backups Methods: ○ Logical backup: Plain text files with SQL statements ○ Physical backup: A copy of the datafiles ● Backup tasks: ○ Schedule ○ Rotation / Retention ○ Secure Backup ○ Verify Backup ○ Point-in-time recovery backup ○ Upload to external datacenter, e.g., cloud storage ○ Reporting Backup Management

16. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB ● Define topology ○ Master-Slave ○ Master-Master ● Define replication type ○ Synchronous: There is no loss of data even if our master node is lost. The commits in the master must wait for a confirmation from the slave. ○ Asynchronous: There is a possibility of data loss in case we lose our master node. If the replica is not updated at the time of the incident, the information that has not been copied may be lost. ○ Logical Replication: Performed via SQL statements. ○ Physical Replication: Performed via internal data structure. ● Failover ● Single endpoint for application High Availability setups

17. Poll - multiple choice Copyright 2018 Severalnines AB According to you, what are your most challenging tasks in production? ● A well tuned and secure configuration ● Monitoring uptime & Alerting ● Managing performance / queries ● Backup management ● Failover & High Availability

19. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB Deployment ● Installation: ○ Repositories or packages ○ Some packages dependencies could be required ● Secure installation: ○ Configuration files (pg_hba.conf or postgresql.conf) ○ Authentication ● Basic configuration: ○ For example: Listen address or wal_level ○ Require DBA or Sysadmin skills

20. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB What to monitor ● Database ○ Queries ○ Sessions ○ Locks ○ IO ○ Replication ○ Backups ● System ○ CPU ○ Memory ○ Load Average ○ Network Monitoring & Alerting

21. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB Performance Management ● Time consuming task ● There is not a standard configuration for this task ● Check charts ○ e.g: max_connections ● Database commands ○ EXPLAIN: Check the query plan ○ VACUUM: For several maintenance tasks ● Parameters ○ shared_buffers: Amount of memory that the database uses for shared memory ○ work_mem: Amount of memory used by internal operations like ORDER BY, DISTINCT, JOIN, among others ● Require deep DBA knowledge

22. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB Backup Management ● Typically done using custom scripts ● Some just copy the datafiles (!) ● Schedule using cron ● Manual rotation ● No backup verification or PITR backup

23. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB ● Master-Slave Topology ● Asynchronous replication ● Manual setup of slave ○ Install a new server ○ Copy data from master ○ Configure replication High Availability Setups

24. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB Automatic Failover ● We need to know when we need the failover ● PostgreSQL itself does not include an automatic failover mechanism ● After failover, the applications need to know about the new master

25. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB ● Load Balancer: ○ Balancing traffic ○ Redirect traffic to available nodes ○ Specify ports for different roles ○ It's an external component that requires installation, configuration and monitoring ● Virtual IP: ○ Keepalived is a linux service that allows to configure a virtual IP within an active/passive group of servers ○ Another external component that requires installation, configuration and monitoring Single VIP endpoint with Load Balancer

26. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB ● Where is the documentation? ● How do all these scripts hang together? ● It is often hard to know how the database is managed ● There is nobody to take over what you did ● It can be a mess What happens when the DBA leaves?

37. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB ClusterControl CLI $ s9s cluster --list --long ID STATE TYPE OWNER GROUP NAME COMMENT 1 FAILURE galera system admins Galera1 Cluster failed to start. 2 STARTED postgresql_single system admins PostgreSQL1 All nodes are operational. 5 FAILURE replication system admins MySQL1 Cluster failed to start. Total: 3 $ s9s node --list --cluster-id=2 --long STAT VERSION CID CLUSTER HOST PORT COMMENT coC- 1.7.1.2965 2 PostgreSQL1 192.168.100.110 9500 Up and running. ?o-- 2.3.2 2 PostgreSQL1 192.168.100.110 9090 Process 'prometheus' is running. ho-- 1.5.18 2 PostgreSQL1 192.168.100.114 9600 Process 'haproxy' is running. koM- 1.3 2 PostgreSQL1 192.168.100.114 112 Process 'keepalived' is running. ho-- 1.5.18 2 PostgreSQL1 192.168.100.115 9600 Process 'haproxy' is running. ko-- 1.3 2 PostgreSQL1 192.168.100.115 112 Process 'keepalived' is running. poM- 10.5 2 PostgreSQL1 192.168.100.123 5432 Up and running. poS- 10.5 2 PostgreSQL1 192.168.100.124 5432 Up and running. poS- 10.5 2 PostgreSQL1 192.168.100.125 5432 Up and running. Total: 9

39. ● Deployment ● Add replication slave ● Add load balancer ● Deploy keepalived ● Monitoring ● Performance ● Backup ● Topology view ● CLI Cluster Control PostgreSQL Management Demo Copyright 2018 Severalnines AB

40. Poll - multiple choice (choose one of below) Copyright 2018 Severalnines AB Where do you need the most improvements in your environment? ● Monitoring & Alerting ● Performance Management ● Backups ● High Availability & Automatic Failover ● Other (please specify in the chat box)

42. Links & Resources ● ClusterControl for PostgreSQL ● Download / install ClusterControl ● ClusterControl Community Edition Page ● Contact us: info@severalnines.com