Postgresql 9.0 HA at LOADAYS 2012

Julien Pivotto
Julien PivottoOpen Source Consultant at Inuits
;
PostgreSQL 9.0 HA
Julien Pivotto
April, 1 2012 @ Loadays
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
The mission
Before the migration
Table of content
1 Overview
The mission
Before the migration
2 PostgreSQL 9.0
Intro
Streaming replication
Master configuration
Slave configuration
PostgreSQL specific tricks
Setting up a slave
3 Clustering
Set up of corosync
OCF resource
4 Backups
Cron jobs
BackupPC
5 Monitoring
Nagios
Munin
6 Automation
Puppet module
The node file
#TODO
7 The end
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
The mission
Before the migration
Who am I
• Julien Pivotto
• Consultant at Inuits since May 2011
• FOSS defender since 2005
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
The mission
Before the migration
A.R.S.I.A.
• Association Régionale de Santé et d’Identification Animales
• 30 linux servers in several locations
• A lot of Open Source
• CentOS, Samba, Open-xchange, mailscanner, Cyrus,
• . . . Puppet, jenkins, foreman, OpenVPN, GLPI, rabbitmq,
• . . . BackupPC, CUPS, icinga, trac, zope, plone,
• . . . solr, pentaho, funambol, munin, squid, asterisk,
• . . . and PostgreSQL, . . .
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
The mission
Before the migration
C.E.R.I.S.E
• A web application
• Plone (python)
• 15k+ visits, 500k+ pages and 2.000.000+ hits each month
• Developped by Affinitic
• Several databases
• PostgreSQL 9.0
• Oracle database
• Several servers/services
• Two reverse proxies in failover HA
• Two application servers in load balancing HA
• Two PostgreSQL servers in failover HA
• An oracledb server
• A development server
• A pentaho server
• Being integrated in jenkins (to be continued. . . )
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
The mission
Before the migration
PostgreSQL before the migration
• PostgreSQL 8.3.7
• No native support of HA
• High availability with heartbeat 2 and DRBD
• Installed on the application servers
• Nothing automated
• Failover: Passive node is not even read only
• Installed in November 2008
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
The mission
Before the migration
Monitoring before the installation
• Icinga
• Check of the DRBD
• Simple connection check to PostgreSQL
• Graphing with Cacti
• Size of the databases
• Connexions to the database
• Checkpoints
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
The mission
Before the migration
Backups before the installation
• Backups were done every hour one the same machine
• External backups once a day on disk and on tape
• Backups are made with pg_dump command
• BackupPC get those files
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
PostgreSQL specific tricks
Setting up a slave
PostgreSQL 9.0
• PostgreSQL 9.0 was out in september 2010
• It brings to the world native replication in PostgresSQL
• There is not any native failover tool
• So we need to use PostgreSQL + Corosync
• The setup of PostgreSQL is managed by Puppet
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
PostgreSQL specific tricks
Setting up a slave
Write-Ahead Logging
• It means that every change to datafile must first be written
into a log file
• Less disk writes: only the log file needs to be flushed to disk to
guarantee that a transaction is committed, rather than every
data file changed by the transaction
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
PostgreSQL specific tricks
Setting up a slave
What is streaming replication
• Streaming replication provides the capability to ship and apply
WAL XLOGS to standby servers
• It’s possible to have multiple standby servers
• Standby servers can be read-only ("Hot standby")
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
PostgreSQL specific tricks
Setting up a slave
DisadvantagesSpecifications of streaming replication
• Streaming replication supports only asynchronous log-shipping
• But when the database is used, the delay is close to
synchronous log-shipping
• Adding a standby server requires manual action
• But in our case we will only have one standby server
• PostgreSQL does not provide HA feature
• But Corosync does
• It is a single-threaded replication
• It is a single-threaded replication
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
PostgreSQL specific tricks
Setting up a slave
Master configuration
The master only needs one configuration file.
Configuration non-related to SR
#Postgresql configuration
#http://www.postgresql.org/docs/9.0/interactive/index.html
listen_addresses = ’*’
max_connections = 200
shared_buffers = 4096MB
work_mem = 4096MB
effective_cache_size = 10024MB
commit_delay = 100000
effective_cache_size = 2560
log_destination = ‘stderr’
log_directory = ‘pg_log’
logging_collector = on
log_filename = ‘postgresql-%Y-%m-%d_%H%M%S.log’
log_truncate_on_rotation = on
log_rotation_age = 1d
log_rotation_size = 0
log_min_messages = notice
log_min_duration_statement = 1000
log_line_prefix = ‘%t %u ’
log_statement = ‘none’
datestyle = ‘iso, dmy’
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
PostgreSQL specific tricks
Setting up a slave
Master configuration
Configuration related to SR
wal_level = hot_standby
max_wal_senders = 2
wal_keep_segments = 128
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
PostgreSQL specific tricks
Setting up a slave
Master configuration
• wal_level = hot_standby
Allows stanby server to be readable
• max_wal_senders = 2
We allow up to 2 standby nodes
• wal_keep_segments = 128
The minimum wal segments to keep
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
PostgreSQL specific tricks
Setting up a slave
Slave configuration
• The slave requires at least two configuration files
• A postgreSQL.conf file
• A recovery.conf file, used to apply the WAL XLOGS shipped by
the master
• A trigger file to stop replication can be specified
PostgreSQL.conf - Configuration related to SR
wal_level = hot_standby
hot_standby = on
Note that the file also have the same first part of the config file
than the master configuration.
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
PostgreSQL specific tricks
Setting up a slave
Slave configuration
recovery.conf
standby_mode = ‘on’
primary_conninfo = ‘host=192.168.177.2 user=replicuser’
• standby_mode means that this is a standby server
• primary_conninfo is the connection to the master
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
PostgreSQL specific tricks
Setting up a slave
Replication user
• A super user called replication has to be created
• The SQL command to create it is
CREATE USER replication SUPERUSER LOGIN CONNECTION
LIMIT 1 ENCRYPTED PASSWORD ‘foobar’;
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
PostgreSQL specific tricks
Setting up a slave
pg_hba.conf
• pg_hba.conf is the file that contains some kind of ACLs for
the PostgreSQL connections
• In that file we will add both nodes as ‘trusted’ and the
replication user as trusted too
pg_hba.conf
hostnossl all all 10.0.10.8/32 trust
hostnossl all all 10.0.10.9/32 trust
hostnossl replication replicuser 192.168.177.2/24 trust
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Intro
Streaming replication
Master configuration
Slave configuration
PostgreSQL specific tricks
Setting up a slave
Setting up a slave
• You have to type a bunch of commands on the master when
you add a new standby server
Adding a standby server
psql -c "SELECT pg_start_backup(’label’, true)"
rsync -a ${PGDATA}/ standby:/srv/pgsql/standby/ --exclude postmaster.pid --exclude ‘*-master’
--exclude ‘*-slave’
psql -c "SELECT pg_stop_backup()"
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Set up of corosync
OCF resource
Corosync configuration
• The goal of corosync is to make the switch between
master/slave when needed
• It will ensure that a master is online and connected to the
router
• The two servers are connected to each other on eth1
• Corosync is installed by Puppet
• We take it from the clusterlabs repositories
• We use a personalized master/slave ocf resource to manage
the PostgreSQL M/S
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Set up of corosync
OCF resource
crm.conf
The main configuration file of corosync is
/etc/corosync/crm.conf. It contains all the
resources/nodes/etc. . .
Defining the nodes
node babar.interne.arsia.be 
attributes standby="off"
node dumbo.interne.arsia.be 
attributes standby="off"
In this code, the two nodes are defined, and we tell corosync that
they should be started at launch.
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Set up of corosync
OCF resource
crm.conf
Defining the primitives
primitive pgsql ocf:inuits:pgsql-ms
primitive virt_ip ocf:heartbeat:IPaddr2 
params nic="eth0" iflabel="0" ip="10.0.10.10" cidr_netmask="24" broadcast="10.0.10.255" 
meta target-role="Started" is-managed="true"
primitive ping ocf:pacemaker:ping 
params host_list="10.0.10.1" 
op monitor interval="10s" timeout="10s" 
op start interval="0" timeout="45s" 
op stop interval="0" timeout="50s"
• We define 3 primitives:
• pgsql, the PostgreSQL primitive
• virt_ip, the floating IP address
• ping, the primitive that will check that the servers are
connected to the router
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Set up of corosync
OCF resource
crm.conf
Configuring the primitives
ms pgsql-ms pgsql 
params pgsqlconfig="/var/lib/pgsql/data/postgresql.conf" 
lsb_script="/etc/init.d/postgresql-9.0" 
pgsqlrecovery="/var/lib/pgsql/data/recovery.conf" 
meta clone-max="2" clone-node-max="1" master-max="1" master-node-max="1" notify="false"
clone clone-ping ping 
meta globally-unique="false"
• We configure the PostgreSQL M/S: the init script, the
configuration files. . .
• We also configure the ping resource as a clone (it will be
launched on both servers)
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Set up of corosync
OCF resource
crm.conf
Defining the nodes
group PSQL virt_ip
location connected PSQL 
rule $id="connected-rule" -inf: not_defined pingd or pingd lte 0
colocation ip_psql inf: PSQL pgsql-ms:Master
property $id="cib-bootstrap-options" 
cluster-infrastructure="openais" 
expected-quorum-votes="2" 
stonith-enabled="false" 
no-quorum-policy="ignore" 
default-resource-stickiness="INFINITY"
rsc_defaults $id="rsc_defaults-options" 
migration-threshold="INFINITY" 
failure-timeout="10" 
resource-stickiness="INFINITY"
• These lines will ensure that the master is always on the same
node as the floating IP address
• And also that the master is connected to the router
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Set up of corosync
OCF resource
OCF resource
• There is a custom OCF resource to manage the master/slave
PostgreSQL
• It is based on an example of resource written by Andrew
Beekhof from Clusterlabs
• The file has to be in
/usr/lib/ocf/resource.d/inuits/pgsql-ms
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Set up of corosync
OCF resource
OCF resource
• The script does the following:
• It moves the postgresql.conf-master to
postgresql.conf when a node is promoted/master
• It moves the postgresql.conf-slave to postgresql.conf
when a node is depromoted/slave
• It ensure that recovery.conf-slave is on recovery.conf
on slave and absent on master
• It starts/restarts PostgreSQL when needed.
• I will post that file on Github soon
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Cron jobs
BackupPC
Backups of the databases
• Sometimes, you need backups (especially when you don’t have
backups. . . )
• We do a backup per hour on each node (one at minute 0 and
one at minute 30)
• We do a backup per day on each node
• We do a backup per day on before BackupPC backup on each
node.
• We keep 24 hourly backups and 7 daily backups on disk
• With BackupPC we keep months of backups
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Cron jobs
BackupPC
Hourly backup script
/usr/local/bin/backup_hourly.sh
#!/bin/bash
DATE=$(date +%H)
BACKUP_PATH=/var/lib/backups/hourly
for db in foobar_db foobar2_db
do
/usr/bin/pg_dump $db | gzip > $BACKUP_PATH/${db}_$DATE.pgsql.gz
ln -fs $BACKUP_PATH/${db}_$DATE.pgsql.gz $BACKUP_PATH/${db}_current.pgsql.gz
done
The daily script is almost the same.
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Cron jobs
BackupPC
BackupPC script
/usr/local/bin/backup_backuppc.sh
#!/bin/bash
DATE=$(date +%u)
BACKUP_PATH=/var/lib/backups/backuppc
for db in cerise trackitquality trackit zodb_cerise
do
/usr/bin/pg_dump -U postgres $db | gzip > $BACKUP_PATH/${db}_$DATE.pgsql.gz
ln -fs $BACKUP_PATH/${db}_$DATE.pgsql.gz $BACKUP_PATH/${db}_current.pgsql.gz
done
In the backupPC config, I added the following:
BackupPC config
$Conf{DumpPreUserCmd} = ‘$sshPath -t -q -x -l backuppc $host /usr/local/bin/backup_backuppc.sh’;
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Nagios
Munin
check_postgres script
• The check_postgres.pl is a nagios-compatible perl script
• Available on http://www.bucardo.org/check_postgres/
and on Github
• What we check with it:
• The current connections
• The status of the replication (the delay)
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Nagios
Munin
Check hot_standby latency
• The check_postgres.pl script has a check for hot_standby
delay
• But we do not know who is the master and the slave, and it is
required to launch the script
• So, here is a bash script I wrote to know the M/S order
Master/slave replication check
#!/bin/bash
/usr/lib64/nagios/plugins/check_postgres.pl --db="$1" 
--action hot_standby_delay -w 300 -c 600 --host=$(
crm_resource --resource pgsql-ms --locate|
awk ‘/Master/ {master=$6} / $/ {slave=$6} END {print master","slave}’
)
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Nagios
Munin
Munin postgres scripts
• Munin is shipped with perl plugins for postgresql
• We use four of them:
• postgres_size,
• postgres_checkpoints,
• postgres_connections_db,
• postgres_cache
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Nagios
Munin
Munin postgres scripts
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Puppet module
The node file
#TODO
Puppet module
• The puppet postgres module is forked from Kris Buytaert’s
github page
• It is modified to remove all references to services, because we
want corosync to manage them
• It creates the users, the super users, the databases
• It is a parameterized class, with a "cluster" parameter. So we
can also install simple PostgreSQL
• The cache sizes are parameterized too, so we can also use that
in Vagrant boxes
• Here are some examples from the module I will upload on
Github ASAP
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Puppet module
The node file
#TODO
Class postgres
The postgres class installs the packages and makes the initdb stuff.
init.pp
class postgres (
$cluster = ‘no’,
$running_ip = ‘127.0.0.1’
){ ...
• The cluster parameter indicates if we want or not clustering
• running_ip is used for the SQL commands. In case of a
cluster, you have to put cluste’s IP address here.
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Puppet module
The node file
#TODO
Sqlexec definition
sqlexec.pp
define postgres::sqlexec($username, $database, $sql, $sqlcheck) {
exec{ "psql -h $postgres::running_ip –username=${username} $database
-c ¨${sql}¨>> /var/log/puppet-postgresql.sql.log 2>&1 && /bin/sleep 5":
environment => "PGPASSWORD=${postgres_password}",
path => $::path,
timeout => 600,
unless => "psql -h $postgres::running_ip -U $username $database -c $sqlcheck",
require => Service[’postgresql-9.0’],
}
}
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Puppet module
The node file
#TODO
Example in the node file
Here is the result in the node file:
dumbo.pp
node babar {
class {
’postgres’:
cluster => ’yes’,
running_ip => ’10.0.10.10’,
}
include postgres::munin
include postgres::backup
include cluster::node
postgres::config{
$::fqdn: listen => ’*’,
}
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Puppet module
The node file
#TODO
Example in the node file
dumbo.pp
postgres::hba {
$::fqdn:
allowedrules => [
"host all all $::ipaddress/32 trust",
’hostnossl all all 10.0.10.8/32 trust’,
’hostnossl all all 10.0.10.9/32 trust’,
’hostnossl all all 10.0.10.10/32 trust’,
’hostnossl replication replicuser 192.168.177.2/24 trust’,
],
}
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Puppet module
The node file
#TODO
Example in the node file
dumbo.pp
postgres::createsuperuser{
’replicuser’:
passwd => ’foobar’,
}
postgres::createuser{
’cerise’:
passwd => ’foobar’;
}
postgres::createdb{
’zodb_cerise’:
owner => ’cerise’,
require => Postgres::Createuser[’cerise’],
}
}
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Puppet module
The node file
#TODO
#TODO
• The first synchronisation is not puppetized
• More advanced checks on the database #monitoringsucks
(e.g. slow queries)
• A disaster recovery
• Improve the ocf script
• Check the content of the backups
• . . .
Julien Pivotto PostgreSQL 9.0 HA
;
Overview
PostgreSQL 9.0
Clustering
Backups
Monitoring
Automation
The end
Any questions?
Julien Pivotto PostgreSQL 9.0 HA
1 of 42

Recommended

Steve Singer - Managing PostgreSQL with Puppet @ Postgres Open by
Steve Singer - Managing PostgreSQL with Puppet @ Postgres OpenSteve Singer - Managing PostgreSQL with Puppet @ Postgres Open
Steve Singer - Managing PostgreSQL with Puppet @ Postgres OpenPostgresOpen
5.1K views38 slides
Logstash and friends by
Logstash and friendsLogstash and friends
Logstash and friendsJulien Pivotto
13.1K views41 slides
Vagrant and CentOS 7 by
Vagrant and CentOS 7Vagrant and CentOS 7
Vagrant and CentOS 7Julien Pivotto
15.4K views67 slides
Linux conna kpatch-without-stopmachine-fixed by
Linux conna kpatch-without-stopmachine-fixedLinux conna kpatch-without-stopmachine-fixed
Linux conna kpatch-without-stopmachine-fixedTommy Lee
654 views43 slides
Spot Trading - A case study in continuous delivery for mission critical finan... by
Spot Trading - A case study in continuous delivery for mission critical finan...Spot Trading - A case study in continuous delivery for mission critical finan...
Spot Trading - A case study in continuous delivery for mission critical finan...SaltStack
3.9K views34 slides
Puppetconf 2015 - Puppet Reporting with Elasticsearch Logstash and Kibana by
Puppetconf 2015 - Puppet Reporting with Elasticsearch Logstash and KibanaPuppetconf 2015 - Puppet Reporting with Elasticsearch Logstash and Kibana
Puppetconf 2015 - Puppet Reporting with Elasticsearch Logstash and Kibanapkill
3.8K views100 slides

More Related Content

What's hot

Hacking on WildFly 9 by
Hacking on WildFly 9Hacking on WildFly 9
Hacking on WildFly 9Virtual JBoss User Group
4.9K views20 slides
Continuous Infrastructure: Modern Puppet for the Jenkins Project - PuppetConf... by
Continuous Infrastructure: Modern Puppet for the Jenkins Project - PuppetConf...Continuous Infrastructure: Modern Puppet for the Jenkins Project - PuppetConf...
Continuous Infrastructure: Modern Puppet for the Jenkins Project - PuppetConf...Puppet
5K views51 slides
Puppet evolutions by
Puppet evolutionsPuppet evolutions
Puppet evolutionsAlessandro Franceschi
10.3K views27 slides
Serverspec and Sensu - Testing and Monitoring collide by
Serverspec and Sensu - Testing and Monitoring collideServerspec and Sensu - Testing and Monitoring collide
Serverspec and Sensu - Testing and Monitoring collidem_richardson
22K views23 slides
Portland PUG April 2014: Beaker 101: Acceptance Test Everything by
Portland PUG April 2014: Beaker 101: Acceptance Test EverythingPortland PUG April 2014: Beaker 101: Acceptance Test Everything
Portland PUG April 2014: Beaker 101: Acceptance Test EverythingPuppet
2.1K views19 slides
Transforming the Ceph Integration Tests with OpenStack by
Transforming the Ceph Integration Tests with OpenStack Transforming the Ceph Integration Tests with OpenStack
Transforming the Ceph Integration Tests with OpenStack Ceph Community
200 views27 slides

What's hot(20)

Continuous Infrastructure: Modern Puppet for the Jenkins Project - PuppetConf... by Puppet
Continuous Infrastructure: Modern Puppet for the Jenkins Project - PuppetConf...Continuous Infrastructure: Modern Puppet for the Jenkins Project - PuppetConf...
Continuous Infrastructure: Modern Puppet for the Jenkins Project - PuppetConf...
Puppet5K views
Serverspec and Sensu - Testing and Monitoring collide by m_richardson
Serverspec and Sensu - Testing and Monitoring collideServerspec and Sensu - Testing and Monitoring collide
Serverspec and Sensu - Testing and Monitoring collide
m_richardson22K views
Portland PUG April 2014: Beaker 101: Acceptance Test Everything by Puppet
Portland PUG April 2014: Beaker 101: Acceptance Test EverythingPortland PUG April 2014: Beaker 101: Acceptance Test Everything
Portland PUG April 2014: Beaker 101: Acceptance Test Everything
Puppet2.1K views
Transforming the Ceph Integration Tests with OpenStack by Ceph Community
Transforming the Ceph Integration Tests with OpenStack Transforming the Ceph Integration Tests with OpenStack
Transforming the Ceph Integration Tests with OpenStack
Ceph Community 200 views
Puppet for dummies - ZendCon 2011 Edition by Joshua Thijssen
Puppet for dummies - ZendCon 2011 EditionPuppet for dummies - ZendCon 2011 Edition
Puppet for dummies - ZendCon 2011 Edition
Joshua Thijssen13K views
SaltConf14 - Oz Akan, Rackspace - Deploying OpenStack Marconi with SaltStack by SaltStack
SaltConf14 - Oz Akan, Rackspace - Deploying OpenStack Marconi with SaltStackSaltConf14 - Oz Akan, Rackspace - Deploying OpenStack Marconi with SaltStack
SaltConf14 - Oz Akan, Rackspace - Deploying OpenStack Marconi with SaltStack
SaltStack1.2K views
Verifying your Ansible Roles using Docker, Test Kitchen and Serverspec by Edmund Dipple
Verifying your Ansible Roles using Docker, Test Kitchen and ServerspecVerifying your Ansible Roles using Docker, Test Kitchen and Serverspec
Verifying your Ansible Roles using Docker, Test Kitchen and Serverspec
Edmund Dipple773 views
SCALE12X: Chef for OpenStack by Matt Ray
SCALE12X: Chef for OpenStackSCALE12X: Chef for OpenStack
SCALE12X: Chef for OpenStack
Matt Ray2.3K views
SaltConf 2015: Salt stack at web scale: Better, Stronger, Faster by Thomas Jackson
SaltConf 2015: Salt stack at web scale: Better, Stronger, FasterSaltConf 2015: Salt stack at web scale: Better, Stronger, Faster
SaltConf 2015: Salt stack at web scale: Better, Stronger, Faster
Thomas Jackson5.1K views
Puppet Availability and Performance at 100K Nodes - PuppetConf 2014 by Puppet
Puppet Availability and Performance at 100K Nodes - PuppetConf 2014Puppet Availability and Performance at 100K Nodes - PuppetConf 2014
Puppet Availability and Performance at 100K Nodes - PuppetConf 2014
Puppet8.8K views
Modern Infrastructure from Scratch with Puppet by Puppet
Modern Infrastructure from Scratch with PuppetModern Infrastructure from Scratch with Puppet
Modern Infrastructure from Scratch with Puppet
Puppet1.9K views
SaltConf14 - Saurabh Surana, HP Cloud - Automating operations and support wit... by SaltStack
SaltConf14 - Saurabh Surana, HP Cloud - Automating operations and support wit...SaltConf14 - Saurabh Surana, HP Cloud - Automating operations and support wit...
SaltConf14 - Saurabh Surana, HP Cloud - Automating operations and support wit...
SaltStack1.2K views
Package Management and Chef - ChefConf 2015 by Chef
Package Management and Chef - ChefConf 2015Package Management and Chef - ChefConf 2015
Package Management and Chef - ChefConf 2015
Chef999 views
OSDC2014: Testing Server Infrastructure with #serverspec by Andreas Schmidt
OSDC2014: Testing Server Infrastructure with #serverspecOSDC2014: Testing Server Infrastructure with #serverspec
OSDC2014: Testing Server Infrastructure with #serverspec
Andreas Schmidt2.3K views
Steamlining your puppet development workflow by Tomas Doran
Steamlining your puppet development workflowSteamlining your puppet development workflow
Steamlining your puppet development workflow
Tomas Doran4.5K views
Sensu and Sensibility - Puppetconf 2014 by Tomas Doran
Sensu and Sensibility - Puppetconf 2014Sensu and Sensibility - Puppetconf 2014
Sensu and Sensibility - Puppetconf 2014
Tomas Doran5.2K views
London devops logging by Tomas Doran
London devops loggingLondon devops logging
London devops logging
Tomas Doran17.3K views
Arnold Bechtoldt, Inovex GmbH Linux systems engineer - Configuration Manageme... by SaltStack
Arnold Bechtoldt, Inovex GmbH Linux systems engineer - Configuration Manageme...Arnold Bechtoldt, Inovex GmbH Linux systems engineer - Configuration Manageme...
Arnold Bechtoldt, Inovex GmbH Linux systems engineer - Configuration Manageme...
SaltStack2.2K views

Similar to Postgresql 9.0 HA at LOADAYS 2012

Postgresql 9.0 HA at RMLL 2012 by
Postgresql 9.0 HA at RMLL 2012Postgresql 9.0 HA at RMLL 2012
Postgresql 9.0 HA at RMLL 2012Julien Pivotto
4K views119 slides
Como creamos QuestDB Cloud, un SaaS basado en Kubernetes alrededor de QuestDB... by
Como creamos QuestDB Cloud, un SaaS basado en Kubernetes alrededor de QuestDB...Como creamos QuestDB Cloud, un SaaS basado en Kubernetes alrededor de QuestDB...
Como creamos QuestDB Cloud, un SaaS basado en Kubernetes alrededor de QuestDB...javier ramirez
20 views43 slides
SUSE Container as a Service Platform by
SUSE Container as a Service PlatformSUSE Container as a Service Platform
SUSE Container as a Service PlatformSUSE
650 views14 slides
Built in physical and logical replication in postgresql-Firat Gulec by
Built in physical and logical replication in postgresql-Firat GulecBuilt in physical and logical replication in postgresql-Firat Gulec
Built in physical and logical replication in postgresql-Firat GulecFIRAT GULEC
404 views49 slides
Switch as a Server - PuppetConf 2014 - Leslie Carr by
Switch as a Server - PuppetConf 2014 - Leslie CarrSwitch as a Server - PuppetConf 2014 - Leslie Carr
Switch as a Server - PuppetConf 2014 - Leslie CarrCumulus Networks
912 views31 slides
2009-01-20 RHEL 5.3 for System z by
2009-01-20 RHEL 5.3 for System z2009-01-20 RHEL 5.3 for System z
2009-01-20 RHEL 5.3 for System zShawn Wells
268 views33 slides

Similar to Postgresql 9.0 HA at LOADAYS 2012(20)

Como creamos QuestDB Cloud, un SaaS basado en Kubernetes alrededor de QuestDB... by javier ramirez
Como creamos QuestDB Cloud, un SaaS basado en Kubernetes alrededor de QuestDB...Como creamos QuestDB Cloud, un SaaS basado en Kubernetes alrededor de QuestDB...
Como creamos QuestDB Cloud, un SaaS basado en Kubernetes alrededor de QuestDB...
javier ramirez20 views
SUSE Container as a Service Platform by SUSE
SUSE Container as a Service PlatformSUSE Container as a Service Platform
SUSE Container as a Service Platform
SUSE650 views
Built in physical and logical replication in postgresql-Firat Gulec by FIRAT GULEC
Built in physical and logical replication in postgresql-Firat GulecBuilt in physical and logical replication in postgresql-Firat Gulec
Built in physical and logical replication in postgresql-Firat Gulec
FIRAT GULEC404 views
Switch as a Server - PuppetConf 2014 - Leslie Carr by Cumulus Networks
Switch as a Server - PuppetConf 2014 - Leslie CarrSwitch as a Server - PuppetConf 2014 - Leslie Carr
Switch as a Server - PuppetConf 2014 - Leslie Carr
Cumulus Networks912 views
2009-01-20 RHEL 5.3 for System z by Shawn Wells
2009-01-20 RHEL 5.3 for System z2009-01-20 RHEL 5.3 for System z
2009-01-20 RHEL 5.3 for System z
Shawn Wells268 views
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica... by Command Prompt., Inc
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...
PaaSTA: Running applications at Yelp by Nathan Handler
PaaSTA: Running applications at YelpPaaSTA: Running applications at Yelp
PaaSTA: Running applications at Yelp
Nathan Handler1.1K views
Sanger OpenStack presentation March 2017 by Dave Holland
Sanger OpenStack presentation March 2017Sanger OpenStack presentation March 2017
Sanger OpenStack presentation March 2017
Dave Holland655 views
Cloud Foundry on OpenStack - An Experience Report | anynines by anynines GmbH
Cloud Foundry on OpenStack - An Experience Report | anynines Cloud Foundry on OpenStack - An Experience Report | anynines
Cloud Foundry on OpenStack - An Experience Report | anynines
anynines GmbH1.6K views
Automating Software Development Life Cycle - A DevOps Approach by Akshaya Mahapatra
Automating Software Development Life Cycle - A DevOps ApproachAutomating Software Development Life Cycle - A DevOps Approach
Automating Software Development Life Cycle - A DevOps Approach
Akshaya Mahapatra3.7K views
Making Spinnaker Go @ Stitch Fix by Diana Tkachenko
Making Spinnaker Go @ Stitch FixMaking Spinnaker Go @ Stitch Fix
Making Spinnaker Go @ Stitch Fix
Diana Tkachenko2.3K views
Introduction to PostgreSQL for System Administrators by Jignesh Shah
Introduction to PostgreSQL for System AdministratorsIntroduction to PostgreSQL for System Administrators
Introduction to PostgreSQL for System Administrators
Jignesh Shah4.4K views
A3Sec Advanced Deployment System by a3sec
A3Sec Advanced Deployment SystemA3Sec Advanced Deployment System
A3Sec Advanced Deployment System
a3sec1.4K views
Ceph Day Shanghai - CeTune - Benchmarking and tuning your Ceph cluster by Ceph Community
Ceph Day Shanghai - CeTune - Benchmarking and tuning your Ceph cluster Ceph Day Shanghai - CeTune - Benchmarking and tuning your Ceph cluster
Ceph Day Shanghai - CeTune - Benchmarking and tuning your Ceph cluster
Ceph Community 167 views
Streaming Processing with a Distributed Commit Log by Joe Stein
Streaming Processing with a Distributed Commit LogStreaming Processing with a Distributed Commit Log
Streaming Processing with a Distributed Commit Log
Joe Stein3.1K views
Sanger, upcoming Openstack for Bio-informaticians by Peter Clapham
Sanger, upcoming Openstack for Bio-informaticiansSanger, upcoming Openstack for Bio-informaticians
Sanger, upcoming Openstack for Bio-informaticians
Peter Clapham271 views

More from Julien Pivotto

The O11y Toolkit by
The O11y ToolkitThe O11y Toolkit
The O11y ToolkitJulien Pivotto
38 views24 slides
What's New in Prometheus and Its Ecosystem by
What's New in Prometheus and Its EcosystemWhat's New in Prometheus and Its Ecosystem
What's New in Prometheus and Its EcosystemJulien Pivotto
12 views42 slides
Prometheus: What is is, what is new, what is coming by
Prometheus: What is is, what is new, what is comingPrometheus: What is is, what is new, what is coming
Prometheus: What is is, what is new, what is comingJulien Pivotto
43 views27 slides
What's new in Prometheus? by
What's new in Prometheus?What's new in Prometheus?
What's new in Prometheus?Julien Pivotto
15 views10 slides
Introduction to Grafana Loki by
Introduction to Grafana LokiIntroduction to Grafana Loki
Introduction to Grafana LokiJulien Pivotto
199 views11 slides
Why you should revisit mgmt by
Why you should revisit mgmtWhy you should revisit mgmt
Why you should revisit mgmtJulien Pivotto
10 views46 slides

More from Julien Pivotto(20)

What's New in Prometheus and Its Ecosystem by Julien Pivotto
What's New in Prometheus and Its EcosystemWhat's New in Prometheus and Its Ecosystem
What's New in Prometheus and Its Ecosystem
Julien Pivotto12 views
Prometheus: What is is, what is new, what is coming by Julien Pivotto
Prometheus: What is is, what is new, what is comingPrometheus: What is is, what is new, what is coming
Prometheus: What is is, what is new, what is coming
Julien Pivotto43 views
Introduction to Grafana Loki by Julien Pivotto
Introduction to Grafana LokiIntroduction to Grafana Loki
Introduction to Grafana Loki
Julien Pivotto199 views
Observing the HashiCorp Ecosystem From Prometheus by Julien Pivotto
Observing the HashiCorp Ecosystem From PrometheusObserving the HashiCorp Ecosystem From Prometheus
Observing the HashiCorp Ecosystem From Prometheus
Julien Pivotto37 views
Monitoring in a fast-changing world with Prometheus by Julien Pivotto
Monitoring in a fast-changing world with PrometheusMonitoring in a fast-changing world with Prometheus
Monitoring in a fast-changing world with Prometheus
Julien Pivotto33 views
5 tips for Prometheus Service Discovery by Julien Pivotto
5 tips for Prometheus Service Discovery5 tips for Prometheus Service Discovery
5 tips for Prometheus Service Discovery
Julien Pivotto38 views
Prometheus and TLS - an Introduction by Julien Pivotto
Prometheus and TLS - an IntroductionPrometheus and TLS - an Introduction
Prometheus and TLS - an Introduction
Julien Pivotto15 views
HAProxy as Egress Controller by Julien Pivotto
HAProxy as Egress ControllerHAProxy as Egress Controller
HAProxy as Egress Controller
Julien Pivotto2.9K views
Improved alerting with Prometheus and Alertmanager by Julien Pivotto
Improved alerting with Prometheus and AlertmanagerImproved alerting with Prometheus and Alertmanager
Improved alerting with Prometheus and Alertmanager
Julien Pivotto4.5K views
SIngle Sign On with Keycloak by Julien Pivotto
SIngle Sign On with KeycloakSIngle Sign On with Keycloak
SIngle Sign On with Keycloak
Julien Pivotto10K views
Monitoring as an entry point for collaboration by Julien Pivotto
Monitoring as an entry point for collaborationMonitoring as an entry point for collaboration
Monitoring as an entry point for collaboration
Julien Pivotto1.3K views
Monitor your CentOS stack with Prometheus by Julien Pivotto
Monitor your CentOS stack with PrometheusMonitor your CentOS stack with Prometheus
Monitor your CentOS stack with Prometheus
Julien Pivotto712 views
Monitor your CentOS stack with Prometheus by Julien Pivotto
Monitor your CentOS stack with PrometheusMonitor your CentOS stack with Prometheus
Monitor your CentOS stack with Prometheus
Julien Pivotto704 views

Recently uploaded

Cencora Executive Symposium by
Cencora Executive SymposiumCencora Executive Symposium
Cencora Executive Symposiummarketingcommunicati21
139 views14 slides
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ... by
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...ShapeBlue
123 views28 slides
Business Analyst Series 2023 - Week 4 Session 8 by
Business Analyst Series 2023 -  Week 4 Session 8Business Analyst Series 2023 -  Week 4 Session 8
Business Analyst Series 2023 - Week 4 Session 8DianaGray10
86 views13 slides
Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And... by
Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And...Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And...
Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And...ShapeBlue
63 views12 slides
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ... by
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...ShapeBlue
79 views17 slides
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas... by
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...Bernd Ruecker
50 views69 slides

Recently uploaded(20)

How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ... by ShapeBlue
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...
ShapeBlue123 views
Business Analyst Series 2023 - Week 4 Session 8 by DianaGray10
Business Analyst Series 2023 -  Week 4 Session 8Business Analyst Series 2023 -  Week 4 Session 8
Business Analyst Series 2023 - Week 4 Session 8
DianaGray1086 views
Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And... by ShapeBlue
Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And...Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And...
Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And...
ShapeBlue63 views
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ... by ShapeBlue
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...
ShapeBlue79 views
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas... by Bernd Ruecker
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...
Bernd Ruecker50 views
Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ... by ShapeBlue
Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ...Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ...
Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ...
ShapeBlue85 views
Declarative Kubernetes Cluster Deployment with Cloudstack and Cluster API - O... by ShapeBlue
Declarative Kubernetes Cluster Deployment with Cloudstack and Cluster API - O...Declarative Kubernetes Cluster Deployment with Cloudstack and Cluster API - O...
Declarative Kubernetes Cluster Deployment with Cloudstack and Cluster API - O...
ShapeBlue88 views
KVM Security Groups Under the Hood - Wido den Hollander - Your.Online by ShapeBlue
KVM Security Groups Under the Hood - Wido den Hollander - Your.OnlineKVM Security Groups Under the Hood - Wido den Hollander - Your.Online
KVM Security Groups Under the Hood - Wido den Hollander - Your.Online
ShapeBlue181 views
Confidence in CloudStack - Aron Wagner, Nathan Gleason - Americ by ShapeBlue
Confidence in CloudStack - Aron Wagner, Nathan Gleason - AmericConfidence in CloudStack - Aron Wagner, Nathan Gleason - Americ
Confidence in CloudStack - Aron Wagner, Nathan Gleason - Americ
ShapeBlue88 views
CloudStack and GitOps at Enterprise Scale - Alex Dometrius, Rene Glover - AT&T by ShapeBlue
CloudStack and GitOps at Enterprise Scale - Alex Dometrius, Rene Glover - AT&TCloudStack and GitOps at Enterprise Scale - Alex Dometrius, Rene Glover - AT&T
CloudStack and GitOps at Enterprise Scale - Alex Dometrius, Rene Glover - AT&T
ShapeBlue112 views
The Power of Heat Decarbonisation Plans in the Built Environment by IES VE
The Power of Heat Decarbonisation Plans in the Built EnvironmentThe Power of Heat Decarbonisation Plans in the Built Environment
The Power of Heat Decarbonisation Plans in the Built Environment
IES VE69 views
Mitigating Common CloudStack Instance Deployment Failures - Jithin Raju - Sha... by ShapeBlue
Mitigating Common CloudStack Instance Deployment Failures - Jithin Raju - Sha...Mitigating Common CloudStack Instance Deployment Failures - Jithin Raju - Sha...
Mitigating Common CloudStack Instance Deployment Failures - Jithin Raju - Sha...
ShapeBlue138 views
Future of AR - Facebook Presentation by Rob McCarty
Future of AR - Facebook PresentationFuture of AR - Facebook Presentation
Future of AR - Facebook Presentation
Rob McCarty62 views
Elevating Privacy and Security in CloudStack - Boris Stoyanov - ShapeBlue by ShapeBlue
Elevating Privacy and Security in CloudStack - Boris Stoyanov - ShapeBlueElevating Privacy and Security in CloudStack - Boris Stoyanov - ShapeBlue
Elevating Privacy and Security in CloudStack - Boris Stoyanov - ShapeBlue
ShapeBlue179 views
NTGapps NTG LowCode Platform by Mustafa Kuğu
NTGapps NTG LowCode Platform NTGapps NTG LowCode Platform
NTGapps NTG LowCode Platform
Mustafa Kuğu365 views
Digital Personal Data Protection (DPDP) Practical Approach For CISOs by Priyanka Aash
Digital Personal Data Protection (DPDP) Practical Approach For CISOsDigital Personal Data Protection (DPDP) Practical Approach For CISOs
Digital Personal Data Protection (DPDP) Practical Approach For CISOs
Priyanka Aash153 views
Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ... by ShapeBlue
Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ...Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ...
Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ...
ShapeBlue144 views

Postgresql 9.0 HA at LOADAYS 2012

  • 1. ; PostgreSQL 9.0 HA Julien Pivotto April, 1 2012 @ Loadays
  • 2. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end The mission Before the migration Table of content 1 Overview The mission Before the migration 2 PostgreSQL 9.0 Intro Streaming replication Master configuration Slave configuration PostgreSQL specific tricks Setting up a slave 3 Clustering Set up of corosync OCF resource 4 Backups Cron jobs BackupPC 5 Monitoring Nagios Munin 6 Automation Puppet module The node file #TODO 7 The end Julien Pivotto PostgreSQL 9.0 HA
  • 3. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end The mission Before the migration Who am I • Julien Pivotto • Consultant at Inuits since May 2011 • FOSS defender since 2005 Julien Pivotto PostgreSQL 9.0 HA
  • 4. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end The mission Before the migration A.R.S.I.A. • Association Régionale de Santé et d’Identification Animales • 30 linux servers in several locations • A lot of Open Source • CentOS, Samba, Open-xchange, mailscanner, Cyrus, • . . . Puppet, jenkins, foreman, OpenVPN, GLPI, rabbitmq, • . . . BackupPC, CUPS, icinga, trac, zope, plone, • . . . solr, pentaho, funambol, munin, squid, asterisk, • . . . and PostgreSQL, . . . Julien Pivotto PostgreSQL 9.0 HA
  • 5. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end The mission Before the migration C.E.R.I.S.E • A web application • Plone (python) • 15k+ visits, 500k+ pages and 2.000.000+ hits each month • Developped by Affinitic • Several databases • PostgreSQL 9.0 • Oracle database • Several servers/services • Two reverse proxies in failover HA • Two application servers in load balancing HA • Two PostgreSQL servers in failover HA • An oracledb server • A development server • A pentaho server • Being integrated in jenkins (to be continued. . . ) Julien Pivotto PostgreSQL 9.0 HA
  • 6. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end The mission Before the migration PostgreSQL before the migration • PostgreSQL 8.3.7 • No native support of HA • High availability with heartbeat 2 and DRBD • Installed on the application servers • Nothing automated • Failover: Passive node is not even read only • Installed in November 2008 Julien Pivotto PostgreSQL 9.0 HA
  • 7. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end The mission Before the migration Monitoring before the installation • Icinga • Check of the DRBD • Simple connection check to PostgreSQL • Graphing with Cacti • Size of the databases • Connexions to the database • Checkpoints Julien Pivotto PostgreSQL 9.0 HA
  • 8. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end The mission Before the migration Backups before the installation • Backups were done every hour one the same machine • External backups once a day on disk and on tape • Backups are made with pg_dump command • BackupPC get those files Julien Pivotto PostgreSQL 9.0 HA
  • 9. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Intro Streaming replication Master configuration Slave configuration PostgreSQL specific tricks Setting up a slave PostgreSQL 9.0 • PostgreSQL 9.0 was out in september 2010 • It brings to the world native replication in PostgresSQL • There is not any native failover tool • So we need to use PostgreSQL + Corosync • The setup of PostgreSQL is managed by Puppet Julien Pivotto PostgreSQL 9.0 HA
  • 10. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Intro Streaming replication Master configuration Slave configuration PostgreSQL specific tricks Setting up a slave Write-Ahead Logging • It means that every change to datafile must first be written into a log file • Less disk writes: only the log file needs to be flushed to disk to guarantee that a transaction is committed, rather than every data file changed by the transaction Julien Pivotto PostgreSQL 9.0 HA
  • 11. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Intro Streaming replication Master configuration Slave configuration PostgreSQL specific tricks Setting up a slave What is streaming replication • Streaming replication provides the capability to ship and apply WAL XLOGS to standby servers • It’s possible to have multiple standby servers • Standby servers can be read-only ("Hot standby") Julien Pivotto PostgreSQL 9.0 HA
  • 12. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Intro Streaming replication Master configuration Slave configuration PostgreSQL specific tricks Setting up a slave DisadvantagesSpecifications of streaming replication • Streaming replication supports only asynchronous log-shipping • But when the database is used, the delay is close to synchronous log-shipping • Adding a standby server requires manual action • But in our case we will only have one standby server • PostgreSQL does not provide HA feature • But Corosync does • It is a single-threaded replication • It is a single-threaded replication Julien Pivotto PostgreSQL 9.0 HA
  • 13. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Intro Streaming replication Master configuration Slave configuration PostgreSQL specific tricks Setting up a slave Master configuration The master only needs one configuration file. Configuration non-related to SR #Postgresql configuration #http://www.postgresql.org/docs/9.0/interactive/index.html listen_addresses = ’*’ max_connections = 200 shared_buffers = 4096MB work_mem = 4096MB effective_cache_size = 10024MB commit_delay = 100000 effective_cache_size = 2560 log_destination = ‘stderr’ log_directory = ‘pg_log’ logging_collector = on log_filename = ‘postgresql-%Y-%m-%d_%H%M%S.log’ log_truncate_on_rotation = on log_rotation_age = 1d log_rotation_size = 0 log_min_messages = notice log_min_duration_statement = 1000 log_line_prefix = ‘%t %u ’ log_statement = ‘none’ datestyle = ‘iso, dmy’ Julien Pivotto PostgreSQL 9.0 HA
  • 14. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Intro Streaming replication Master configuration Slave configuration PostgreSQL specific tricks Setting up a slave Master configuration Configuration related to SR wal_level = hot_standby max_wal_senders = 2 wal_keep_segments = 128 Julien Pivotto PostgreSQL 9.0 HA
  • 15. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Intro Streaming replication Master configuration Slave configuration PostgreSQL specific tricks Setting up a slave Master configuration • wal_level = hot_standby Allows stanby server to be readable • max_wal_senders = 2 We allow up to 2 standby nodes • wal_keep_segments = 128 The minimum wal segments to keep Julien Pivotto PostgreSQL 9.0 HA
  • 16. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Intro Streaming replication Master configuration Slave configuration PostgreSQL specific tricks Setting up a slave Slave configuration • The slave requires at least two configuration files • A postgreSQL.conf file • A recovery.conf file, used to apply the WAL XLOGS shipped by the master • A trigger file to stop replication can be specified PostgreSQL.conf - Configuration related to SR wal_level = hot_standby hot_standby = on Note that the file also have the same first part of the config file than the master configuration. Julien Pivotto PostgreSQL 9.0 HA
  • 17. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Intro Streaming replication Master configuration Slave configuration PostgreSQL specific tricks Setting up a slave Slave configuration recovery.conf standby_mode = ‘on’ primary_conninfo = ‘host=192.168.177.2 user=replicuser’ • standby_mode means that this is a standby server • primary_conninfo is the connection to the master Julien Pivotto PostgreSQL 9.0 HA
  • 18. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Intro Streaming replication Master configuration Slave configuration PostgreSQL specific tricks Setting up a slave Replication user • A super user called replication has to be created • The SQL command to create it is CREATE USER replication SUPERUSER LOGIN CONNECTION LIMIT 1 ENCRYPTED PASSWORD ‘foobar’; Julien Pivotto PostgreSQL 9.0 HA
  • 19. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Intro Streaming replication Master configuration Slave configuration PostgreSQL specific tricks Setting up a slave pg_hba.conf • pg_hba.conf is the file that contains some kind of ACLs for the PostgreSQL connections • In that file we will add both nodes as ‘trusted’ and the replication user as trusted too pg_hba.conf hostnossl all all 10.0.10.8/32 trust hostnossl all all 10.0.10.9/32 trust hostnossl replication replicuser 192.168.177.2/24 trust Julien Pivotto PostgreSQL 9.0 HA
  • 20. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Intro Streaming replication Master configuration Slave configuration PostgreSQL specific tricks Setting up a slave Setting up a slave • You have to type a bunch of commands on the master when you add a new standby server Adding a standby server psql -c "SELECT pg_start_backup(’label’, true)" rsync -a ${PGDATA}/ standby:/srv/pgsql/standby/ --exclude postmaster.pid --exclude ‘*-master’ --exclude ‘*-slave’ psql -c "SELECT pg_stop_backup()" Julien Pivotto PostgreSQL 9.0 HA
  • 21. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Set up of corosync OCF resource Corosync configuration • The goal of corosync is to make the switch between master/slave when needed • It will ensure that a master is online and connected to the router • The two servers are connected to each other on eth1 • Corosync is installed by Puppet • We take it from the clusterlabs repositories • We use a personalized master/slave ocf resource to manage the PostgreSQL M/S Julien Pivotto PostgreSQL 9.0 HA
  • 22. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Set up of corosync OCF resource crm.conf The main configuration file of corosync is /etc/corosync/crm.conf. It contains all the resources/nodes/etc. . . Defining the nodes node babar.interne.arsia.be attributes standby="off" node dumbo.interne.arsia.be attributes standby="off" In this code, the two nodes are defined, and we tell corosync that they should be started at launch. Julien Pivotto PostgreSQL 9.0 HA
  • 23. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Set up of corosync OCF resource crm.conf Defining the primitives primitive pgsql ocf:inuits:pgsql-ms primitive virt_ip ocf:heartbeat:IPaddr2 params nic="eth0" iflabel="0" ip="10.0.10.10" cidr_netmask="24" broadcast="10.0.10.255" meta target-role="Started" is-managed="true" primitive ping ocf:pacemaker:ping params host_list="10.0.10.1" op monitor interval="10s" timeout="10s" op start interval="0" timeout="45s" op stop interval="0" timeout="50s" • We define 3 primitives: • pgsql, the PostgreSQL primitive • virt_ip, the floating IP address • ping, the primitive that will check that the servers are connected to the router Julien Pivotto PostgreSQL 9.0 HA
  • 24. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Set up of corosync OCF resource crm.conf Configuring the primitives ms pgsql-ms pgsql params pgsqlconfig="/var/lib/pgsql/data/postgresql.conf" lsb_script="/etc/init.d/postgresql-9.0" pgsqlrecovery="/var/lib/pgsql/data/recovery.conf" meta clone-max="2" clone-node-max="1" master-max="1" master-node-max="1" notify="false" clone clone-ping ping meta globally-unique="false" • We configure the PostgreSQL M/S: the init script, the configuration files. . . • We also configure the ping resource as a clone (it will be launched on both servers) Julien Pivotto PostgreSQL 9.0 HA
  • 25. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Set up of corosync OCF resource crm.conf Defining the nodes group PSQL virt_ip location connected PSQL rule $id="connected-rule" -inf: not_defined pingd or pingd lte 0 colocation ip_psql inf: PSQL pgsql-ms:Master property $id="cib-bootstrap-options" cluster-infrastructure="openais" expected-quorum-votes="2" stonith-enabled="false" no-quorum-policy="ignore" default-resource-stickiness="INFINITY" rsc_defaults $id="rsc_defaults-options" migration-threshold="INFINITY" failure-timeout="10" resource-stickiness="INFINITY" • These lines will ensure that the master is always on the same node as the floating IP address • And also that the master is connected to the router Julien Pivotto PostgreSQL 9.0 HA
  • 26. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Set up of corosync OCF resource OCF resource • There is a custom OCF resource to manage the master/slave PostgreSQL • It is based on an example of resource written by Andrew Beekhof from Clusterlabs • The file has to be in /usr/lib/ocf/resource.d/inuits/pgsql-ms Julien Pivotto PostgreSQL 9.0 HA
  • 27. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Set up of corosync OCF resource OCF resource • The script does the following: • It moves the postgresql.conf-master to postgresql.conf when a node is promoted/master • It moves the postgresql.conf-slave to postgresql.conf when a node is depromoted/slave • It ensure that recovery.conf-slave is on recovery.conf on slave and absent on master • It starts/restarts PostgreSQL when needed. • I will post that file on Github soon Julien Pivotto PostgreSQL 9.0 HA
  • 28. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Cron jobs BackupPC Backups of the databases • Sometimes, you need backups (especially when you don’t have backups. . . ) • We do a backup per hour on each node (one at minute 0 and one at minute 30) • We do a backup per day on each node • We do a backup per day on before BackupPC backup on each node. • We keep 24 hourly backups and 7 daily backups on disk • With BackupPC we keep months of backups Julien Pivotto PostgreSQL 9.0 HA
  • 29. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Cron jobs BackupPC Hourly backup script /usr/local/bin/backup_hourly.sh #!/bin/bash DATE=$(date +%H) BACKUP_PATH=/var/lib/backups/hourly for db in foobar_db foobar2_db do /usr/bin/pg_dump $db | gzip > $BACKUP_PATH/${db}_$DATE.pgsql.gz ln -fs $BACKUP_PATH/${db}_$DATE.pgsql.gz $BACKUP_PATH/${db}_current.pgsql.gz done The daily script is almost the same. Julien Pivotto PostgreSQL 9.0 HA
  • 30. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Cron jobs BackupPC BackupPC script /usr/local/bin/backup_backuppc.sh #!/bin/bash DATE=$(date +%u) BACKUP_PATH=/var/lib/backups/backuppc for db in cerise trackitquality trackit zodb_cerise do /usr/bin/pg_dump -U postgres $db | gzip > $BACKUP_PATH/${db}_$DATE.pgsql.gz ln -fs $BACKUP_PATH/${db}_$DATE.pgsql.gz $BACKUP_PATH/${db}_current.pgsql.gz done In the backupPC config, I added the following: BackupPC config $Conf{DumpPreUserCmd} = ‘$sshPath -t -q -x -l backuppc $host /usr/local/bin/backup_backuppc.sh’; Julien Pivotto PostgreSQL 9.0 HA
  • 31. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Nagios Munin check_postgres script • The check_postgres.pl is a nagios-compatible perl script • Available on http://www.bucardo.org/check_postgres/ and on Github • What we check with it: • The current connections • The status of the replication (the delay) Julien Pivotto PostgreSQL 9.0 HA
  • 32. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Nagios Munin Check hot_standby latency • The check_postgres.pl script has a check for hot_standby delay • But we do not know who is the master and the slave, and it is required to launch the script • So, here is a bash script I wrote to know the M/S order Master/slave replication check #!/bin/bash /usr/lib64/nagios/plugins/check_postgres.pl --db="$1" --action hot_standby_delay -w 300 -c 600 --host=$( crm_resource --resource pgsql-ms --locate| awk ‘/Master/ {master=$6} / $/ {slave=$6} END {print master","slave}’ ) Julien Pivotto PostgreSQL 9.0 HA
  • 33. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Nagios Munin Munin postgres scripts • Munin is shipped with perl plugins for postgresql • We use four of them: • postgres_size, • postgres_checkpoints, • postgres_connections_db, • postgres_cache Julien Pivotto PostgreSQL 9.0 HA
  • 35. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Puppet module The node file #TODO Puppet module • The puppet postgres module is forked from Kris Buytaert’s github page • It is modified to remove all references to services, because we want corosync to manage them • It creates the users, the super users, the databases • It is a parameterized class, with a "cluster" parameter. So we can also install simple PostgreSQL • The cache sizes are parameterized too, so we can also use that in Vagrant boxes • Here are some examples from the module I will upload on Github ASAP Julien Pivotto PostgreSQL 9.0 HA
  • 36. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Puppet module The node file #TODO Class postgres The postgres class installs the packages and makes the initdb stuff. init.pp class postgres ( $cluster = ‘no’, $running_ip = ‘127.0.0.1’ ){ ... • The cluster parameter indicates if we want or not clustering • running_ip is used for the SQL commands. In case of a cluster, you have to put cluste’s IP address here. Julien Pivotto PostgreSQL 9.0 HA
  • 37. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Puppet module The node file #TODO Sqlexec definition sqlexec.pp define postgres::sqlexec($username, $database, $sql, $sqlcheck) { exec{ "psql -h $postgres::running_ip –username=${username} $database -c ¨${sql}¨>> /var/log/puppet-postgresql.sql.log 2>&1 && /bin/sleep 5": environment => "PGPASSWORD=${postgres_password}", path => $::path, timeout => 600, unless => "psql -h $postgres::running_ip -U $username $database -c $sqlcheck", require => Service[’postgresql-9.0’], } } Julien Pivotto PostgreSQL 9.0 HA
  • 38. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Puppet module The node file #TODO Example in the node file Here is the result in the node file: dumbo.pp node babar { class { ’postgres’: cluster => ’yes’, running_ip => ’10.0.10.10’, } include postgres::munin include postgres::backup include cluster::node postgres::config{ $::fqdn: listen => ’*’, } Julien Pivotto PostgreSQL 9.0 HA
  • 39. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Puppet module The node file #TODO Example in the node file dumbo.pp postgres::hba { $::fqdn: allowedrules => [ "host all all $::ipaddress/32 trust", ’hostnossl all all 10.0.10.8/32 trust’, ’hostnossl all all 10.0.10.9/32 trust’, ’hostnossl all all 10.0.10.10/32 trust’, ’hostnossl replication replicuser 192.168.177.2/24 trust’, ], } Julien Pivotto PostgreSQL 9.0 HA
  • 40. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Puppet module The node file #TODO Example in the node file dumbo.pp postgres::createsuperuser{ ’replicuser’: passwd => ’foobar’, } postgres::createuser{ ’cerise’: passwd => ’foobar’; } postgres::createdb{ ’zodb_cerise’: owner => ’cerise’, require => Postgres::Createuser[’cerise’], } } Julien Pivotto PostgreSQL 9.0 HA
  • 41. ; Overview PostgreSQL 9.0 Clustering Backups Monitoring Automation The end Puppet module The node file #TODO #TODO • The first synchronisation is not puppetized • More advanced checks on the database #monitoringsucks (e.g. slow queries) • A disaster recovery • Improve the ocf script • Check the content of the backups • . . . Julien Pivotto PostgreSQL 9.0 HA