This is about an overview of High Availability software stack provided by and for Linux landscape, for mission critical services/applications.

1. 1
High Availability
on Linux
Roger Zhou
zzhou@suse.com
openSUSE.Asia
Summit 2015
SUSE
way

2. © SUSE, All rights reserved.2
CURIOSITY in the land
of Linux High Availability

3. 3
Agenda
• HA architectural components
• Use case examples
• Future outlook & Demo

4. 4
What is Cluster?
• HPC (super computing)
• Load Balancer (Very high capacity)
• High Availability
‒ 99.999% = 5 m/year MTTR
‒ SPOF(single point of failure)
‒ Murphy's Law
"Everything that can go wrong will
go wrong"

5. 5
"HA", widely used, often confusing
• VMWare vSphere HA
‒ hypervisor and hardware level. Close-source.
‒ Agnostic on Guest OS inside the VM.
• SUSE HA
‒ Inside Linux OS.
‒ That said, Windows need Windows HA solution.
• Different Industries
‒ We are Enterprise.
‒ HADOOP HA (dot com)
‒ OpenStack HA ( paas )
‒ OpenSAF (telecom)

6. 6
History of HA in Linux OS
• 1990s, Heartbeat project. Simply two nodes.
• Early 2000s, Heartbeat 2.0 too complex.
‒ Industry demands to split.
1) one for cluster membership
2) one for resource management
• Today, ClusterLabs.org
‒ A completely different solution in early days,
pacemaker + corosnc
‒ While merged Heartbeat project.
‒ 2015 HA Summit

7. 7
Typical HA Problem - Split Brain
• Clustering
‒ multiple nodes share the same resources.
• Split partitions run the same service
‒ It just breaks data integrity !!!
• Two key concepts as the solution:
‒ Fencing
Cluster doesn't accept any confusing state.
STONITH - "shoot the other node in the head".
‒ Quorum
It stands for "majority". No quorum, then no actions,
no resource management, no fencing.

8. 8
HA Hardware Components
• Multiple networks
‒ A user network for end user access.
‒ A dedicated network for cluster communication/heartbeat.
‒ A dedicated storage network infrastructure.
• Network Bonding
‒ aka. Link Aggregation
• Fencing/STONITH devices
‒ remote “powerswitch”
• Shared storage
‒ NAS(nfs/cifs), SAN(fc/iscsi)

9. 9
Architectural Software Components
"clusterlabs.org"
‒ Corosync
‒ Pacemaker
‒ Resource Agents
‒ Fencing/STONITH Devices
‒ UI(crmsh and Hawk2)
‒ Booth for GEO Cluster
Outside of "clusterlabs.org"
‒ LVS: Layer 4, ip+port, kernel space.
‒ HAproxy: Layer 7/ HTTP, user space.
‒ Shared filesystem: OCFS2 / GFS2
‒ Block device replication:
DRBD, cLVM mirroring, cluster-md
‒ Shared storage:
SAN (FC / FCoE / iSCSI)
‒ Multipathing

10. Software Components in details

11. 11
Corosync: messaging and membership
• Consensus algorithm
‒ "Totem Single Ring Ordering and Membership protocol"
• Closed Process Group
‒ Analogue “TCP/IP 3-way hand shaking”
‒ Membership handling.
‒ Message ordering.
• A quorum system
‒ notifies apps when quorum is achieved or lost.
• In-memory object database
‒ for Configuration engines and Service engines.
‒ Shared-nothing cluster.

12. 12
Pacemaker: the resources manager
• The brain of the cluster.
• Policy engine for decision making.
‒ To start/stop resources on a node according to the score.
‒ To monitor resources according to interval.
‒ To restart resources if monitor fails.
‒ To fence/STONITH a node if stop operation fails.

13. 13
Shoot The Other Node In The Head
• Data integrity does not
tolerate any confusing state.
Before migrating resources
to another node in the
cluster, the cluster must
confirm the suspicious node
really is down.
• STONITH is mandatory for
*enterprise* Linux HA
clusters.

14. 14
Popular STONITH devices
• APC PDU
‒ network based powerswitch
• Standard Protocols Integrated with Servers
‒ Intel AMT, HP iLO, Dell DRAC, IBM IMM, IPMI Alliance
• Software libraries
‒ to deal with KVM, Xen and VMware Vms.
• Software based
‒ SBD (STONITH Block Device) to do self termination.
The last implicit option in the fencing topology.
• NOTE: Fencing devices can be chained.

15. 15
Resources Agents (RAs)
• Write RA for your applications
• LSB shell scripts:
start / stop / monitor
• More than hundred contributors in upstream github.

16. 16
Cluster Filesystem
• OCFS2 / GFS2
‒ On the shared storage.
‒ Multiple nodes concurrently access the same filesystem.
http://clusterlabs.org/doc/fr/Pacemaker/1.1/html/Clusters_from_Scratch/_pacemaker_architecture.html

17. 17
Cluster Block Device
• DRBD
‒ network based raid1.
‒ high performance data replication over network.
• cLVM2 + cmirrord
‒ Clustered lvm2 mirroring.
‒ Multiple nodes can manipulate volumes on the shared disk.
‒ clvmd distributes LVM metadata updates in the cluster.
‒ Data replication speed is way too slow.
• Cluster md raid1
‒ multiple nodes use the shared disks as md-raid1.
‒ High performance raid1 solution in cluster.

18. Cluster Examples

19. 19
NFS Server ( High Available NAS )
Cluster Example in Diagram
VM1
DRBD MasterDRBD Master
LVMLVM
Filesystem(ext4)Filesystem(ext4)
NFS exportsNFS exports
Virtual IPVirtual IP
VM2
DRBD SlaveDRBD Slave
LVMLVM
Filesystem(ext4)Filesystem(ext4)
NFS exportsNFS exports
Virtual IPVirtual IP
Failover
Pacemaker + CorosyncPacemaker + Corosync
KernelKernel KernelKernel
network raid1

20. 20
HA iSCSI Server ( Active/Passive )
Cluster Example in Diagram
VM1
DRBD MasterDRBD Master
LVMLVM
iSCSITargetiSCSITarget
iSCSILogicalUnitiSCSILogicalUnit
Virtual IPVirtual IP
VM2
DRBD SlaveDRBD Slave
LVMLVM
iSCSITargetiSCSITarget
iSCSILogicalUnitiSCSILogicalUnit
Virtual IPVirtual IP
Failover
Pacemaker + CorosyncPacemaker + Corosync
network raid1
KernelKernel KernelKernel

21. 21
Cluster FS - OCFS2 on shared disk
Cluster Example in Diagram
Host3Host1
OCFS2OCFS2
Host2
OCFS2OCFS2
KernelKernel KernelKernel
Virtual Machine Images and Configuration ( /mnt/images/ )Virtual Machine Images and Configuration ( /mnt/images/ )
Cluster MD-RAID1Cluster MD-RAID1 Cluster MD-RAID1Cluster MD-RAID1
Replication / Backup
/dev/md127 /dev/md127
VMVM VMmigration
Pacemaker + CorosyncPacemaker + Corosync

22. 22
Future outlook
• Upstream activities
‒ OpenStack: from the control plane into the compute domain.
‒ Scalability of corosync/pacemaker
‒ Docker adoption
‒ “Zero” Downtime HA VM
‒ ...

23. 23
Join us ( all *open-source* )
• Play with Leap 42.1: http://www.opensuse.org
Doc: https://www.suse.com/documentation/sle-ha-12/
• Report and Fix Bugs: http://bugzilla.opensuse.org
• Discussion: opensuse-ha@opensuse.org
• HA ClusterLabs: http://clusterlabs.org/
• General HA Users: users@clusterlabs.org

24. Demo + Q&A + Have fun

25. 25