● I used to be a Dev, Then Became an
● Senior Linux and Open Source
● „Infrastructure Architect“
● Building Clouds since before the
● Surviving the 10th floor test
● Co-Author of some books
● Guest Editor at some sites
What is HA Clustering ?
● One service goes down
=> others take over its work
● IP address takeover, service takeover,
● Not designed for high-performance
● Not designed for high troughput (load balancing)
Does it Matter ?
● Downtime is expensive
● You mis out on $$$
● Your boss complains
● New users don't return
Lies, Damn Lies, and
Statistics Counting nines
(slide by Alan R)
99.9999% 30 sec
99.999% 5 min
99.99% 52 min
99.9% 9 hr
99% 3.5 day
The Rules of HA
● Keep it Simple
● Keep it Simple
● Prepare for Failure
● Complexity is the enemy of
● Test your HA setup
● Virtualization will solve your HA Needs
● Live migration is the solution to all your problems
● HA will make your platform more stable
You care about ?
● Your data ?
• Eventual Consistent
● Your Connection
• Most of the time
Eliminating the SPOF
● Find out what Will Fail
• Power (Supplies)
● Find out what Can Fail
• Going Out Of Memory
● Communications failures can lead to separated partitions of
● If those partitions each try and take control of the cluster,
then it's called a split-brain condition
● If this happens, then bad things will happen
● Issues :
• Who Writes ?
• Who Reads ?
• What if 2 Active application want to write ?
• What if an active server crashes during writing ?
• Can we accept delays ?
• Can we accept readonly data ?
● Hardware Requirements
● Filesystem Requirements (GFS, GpFS, ...)
● Distributed Replicated Block Device
● In the Linux Kernel (as of very recent)
● Usually only 1 mount
• Multi mount as of 8.X
• Requires GFS / OCFS2
● Regular FS ext3 ...
● Only 1 application instance Active accessing data
● Upon Failover application needs to be started on other node
● What happens when you pull the plug of a Physical
• Minimal Timeout
• Why did the crash happen ?
• Is my data still correct ?
Alternatives to DRBD
● GlusterFS looked promising
• “Friends don't let Friends use Gluster”
• Consistency problems
• Stability Problems
• Maybe later
• Not posix
• App needs to implement the API
● Linux HA Project
● Red Hat Cluster Suite
● Application Specific Clustering Software
• e.g Terracotta, MySQL NDBD
● No shared storage
● Serial Connections to UPS
Replication or no shared
● e.g Static Website,
● Heartbeat v1
• Max 2 nodes
• No finegrained resources
• Monitoring using “mon”
● Heartbeat v2
• XML usage was a consulting opportunity
• Stability issues
• Forking ?
● Stonithd : The Heartbeat fencing
● Lrmd : Local Resource Management
Daemon. Interacts directly with resource
● pengine Policy Engine. Computes the next
state of the cluster based on the current
state and the configuration.
● cib Cluster Information Base. Contains
definitions of all cluster options, nodes,
resources, their relationships to one
another and current status. Synchronizes
updates to all cluster nodes.
● crmd Cluster Resource Management
Daemon. Largely a message broker for the
PEngine and LRM, it also elects a leader to
co-ordinate the activities of the cluster.
● openais messaging and membership
● heartbeat messaging layer, an alternative
● ccm Short for Consensus Cluster
Membership. The Heartbeat membership
● Cluster Resource Manager property $id="cibbootstrapoptions"
● Keeps Nodes in Sync noquorumpolicy=ignore
● XML Based rsc_defaults $id="rsc_defaultsoptions"
● cibadm primitive d_mysql ocf:local:mysql
op monitor interval="30s"
● Cli manageable params test_user="sure"
● Crm primitive ip_db ocf:heartbeat:IPaddr2
params ip="172.17.4.202" nic="bond0"
op monitor interval="10s"
group svc_db d_mysql ip_db
● Heartbeat resource (+status)
● OCF (Open Cluster FrameWork) (+monitor)
● Clones (don't use in HAv2)
● Multi State Resources
LSB Resource Agents
● LSB == Linux Standards Base
● LSB resource agents are standard System V-style init
scripts commonly used on Linux and other UNIX-like OSes
● LSB init scripts are stored under /etc/init.d/
● This enables Linux-HA to immediately support nearly every
service that comes with your system, and most packages
which come with their own init script
● It's straightforward to change an LSB script to an OCF
● OCF == Open Cluster Framework
● OCF Resource agents are the most powerful
type of resource agent we support
● OCF RAs are extended init scripts
• They have additional actions:
• monitor – for monitoring resource health
• meta-data – for providing information
about the RA
● OCF RAs are located in
● Defined in the OCF Resource script
● Configured in the parameters
● Tomcat :
• Checks a configurable health page
● MySQL :
• Checks query from a configurable table
● Others :
• Basic proces state
Anatomy of a Cluster
• Cluster properties
• Resource Defaults
• Primitive Definitions
• Resource Groups and Constraints
No-quorum-policy = We'll ignore the loss of quorum as this is a 2 node cluster
pe-* = restricg logging
Start-failure : When set to FALSE, the cluster will instead use the resource's failcount and
value for resource-failure-stickiness
failure-timeout means that after a failure there will be a 60 second timeout before the
resource can come back to the node on which it failed.
Migration-treshold=1 means that after 1 failure the resource will try to start on the other node
Resource-stickiness=INFINITY means that the resource really wants to stay where it is now.
Parsing a config
● Isn't always done correctly
● Even a verify won't find all issues
● Unexpected behaviour might occur
Where a resource runs
• multi state resources
• Master – Slave ,
• e.g mysql master-slave, drbd
• Resources that can run on multiple nodes
• Multimaster mysql servers
• Mysql slaves
• Stateless applications
• Preferred location to run resource, eg. Based on hostname
• Resources that have to live together
• e.g ip address + service
Define what resource has to start first, or wait for another
• Colocation + order
A Tomcat app on DRBD
● DRBD can only be active on 1 node
● The filesystem needs to be mounted on that active DRBD
Resource Groups and
group svc_mine d_mine ip_mine
ms ms_drbd_storage drbd_storage
meta master_max="1" master_node_max="1" clone_max="2"
location cli-prefer-svc_db svc_db
rule $id="cli-prefer-rule-svc_db" inf: #uname eq db-a
colocation fs_on_drbd inf: svc_mine ms_drbd_storage:Master
order fs_after_drbd inf: ms_drbd_storage:promote svc_mine:start
Start the cluster resource manager
Change in to resource mode
Change into configure mode
Crm configure show
Show the current resource config
Crm resource show
Show the current resource state
Dump the full Cluster Information Base in XML
Pacemaker and Puppet
● Plenty of non usable modules around
• Strict set of ops / parameters
● Make sure your modules don't enable resources
● I've been using templates till to populate
● Cibadm to configure
● Crm is complex , even crm doesn't parse correctly yet
● Plenty of work ahead !
● #linux-ha on irc.freenode.org
't Hemeltje Kheops Business
Gemeentepark 2 Center
2930 Brasschaat Avenque Georges
891.514.231 Lemaître 54
+32 473 441 636 889.780.406