This document discusses quorum nodes in Spectrum Scale clusters and recovery from failures. It describes how quorum nodes determine the active cluster and prevent partitioning. The document outlines best practices for quorum nodes and provides steps to recover from loss of a quorum node majority or failure of the primary and secondary configuration servers.

1. Spectrum Scale 4.1 System Administration
Quorums, Quorum Loss
and recovery
© Copyright IBM Corporation 2015

2. Unit objectives
After completing this unit, you should be able to:
• Review Quorums
• Review some Quorum best practices
• Describe recovery from quorum and cluster configuration node
failures
© Copyright IBM Corporation 2015

3. Spectrum Scale Quorum Nodes
Why do we need Quorum nodes?
• Quorum Nodes prevent multiple nodes from assuming the role
of the file system manager
• A majority of the quorum nodes MUST remain active in order
for the cluster to sustain normal file system activity.
• This prevents the cluster from becoming partitioned off where
two different nodes think they are each in control and corrupt
the file system
© Copyright IBM Corporation 2015

4. Best Practices: Quorum
• Quorum – set of 3-7 nodes use to determine the state of the
cluster
• Small subset of nodes designated as quorum nodes
• If there are only two nodes tiebreaker disks can be used to
determine quorum
• Recommendations
– Use an odd number of nodes
– Choose your most reliable nodes
– Choose nodes in separate: cabinets, power systems
– Use tiebreaker disks only on small clusters
– For small clusters, a simple VM can serve as a quorum node
– Quorum nodes should not be placed on extremely busy servers
© Copyright IBM Corporation 2015

5. RECOVERY SCENARIOS
© Copyright IBM Corporation 2015

6. How to recover: Node Quorum
How to recover from loss of majority for quorum nodes
• Scenario: You have 3 quorum nodes and 2 quorum nodes die.
– Nodes go into Arbitrating state
– Add 2 new quorum nodes
– mmchnode -–quorum –N node4,node5
– Now you have 5 quorum nodes and 3 are running
– Running nodes return to Active state
– Delete missing quorum nodes
– mmdelnode –N badnode1,badnode2
– or
– mmchnode –-nonquorum –N badnode1,badnode2
– Now you have 3 quorum nodes and 3 are running
– Running nodes remain in Active state
• You may have to clean up original quorum nodes once they
recover.
© Copyright IBM Corporation 2015

7. Recover from loss of primary configuration server
How to recover from loss of a primary configuration server
• Scenario: Primary cluster configuration server fails
– Secondary steps in
– Reassign primary
– mmchcluster –p newprimarynode
– To sync up the new primary
– mmchcluster –p LATEST
• Use the same process for failure of secondary cluster
configuration server.
© Copyright IBM Corporation 2015

8. How to recover: Loss of both configuration servers
How to recover from permanent loss of the primary configuration
and secondary servers
• Scenario: Primary and secondary cluster configuration servers
fail
– Cluster status
• Data is online
• mmgetstate -a no longer works
– Reassign both node configuration servers at the same time
– mmchcluster -p nodenewprimary –s nodenewsecondary
– Old servers still think they are in charge until
– mmchcluster –p Latest
© Copyright IBM Corporation 2015

9. I/O Concepts: Node config parameters (1 of 2)
• Spectrum Scale configuration parameters
• File system configuration
• Storage configuration
• Interconnect tuning
• Application design
© Copyright IBM Corporation 2015

10. I/O Concepts: Node config parameters (2 of 2)
• Spectrum Scale configuration parameters
• pagepool
• maxFilesToCache
• maxStatCache
• prefetchThreads
• socketRcvBufferSize
• socketSndBufferSize
• seqDiscardThreshhold
• Worker1threads
• generalWorkerThreads <<< Verify Role
• tokenMem <<< Verify Spelling
© Copyright IBM Corporation 2015

11. Performance tuning: File system parameters
• File system configuration
• -B Blocksize
• -n numnodes
• -l LogFileSize
• Storage Consideration
– RAID level
– RAID Strip Size (Segment Size)
– Storage Type: Flash, 10k RPM, 7,200 RPM
© Copyright IBM Corporation 2015

12. Performance tuning: Storage
• Storage configuration
– IO balance
• Even number of LUNS per path
• Even number of LUNS per NSD server
– RAID level
• Right level for IO pattern
– Cache configuration
• Read ahead not recommended for –j scatter
© Copyright IBM Corporation 2015

13. Performance tuning: Network configuration
• Interconnect selection and tuning
– TCP/IP tuning
– InfiniBand configuration
• Test Network performance with nsdperf
– In /usr/lpp/mmfs/samples/net/*
– README is documentation
– Some tools on Spectrum Scale Wiki
© Copyright IBM Corporation 2015

14. Performance tuning: Application design
• Make sure your applications are cluster aware
• Avoid and/or optimize for heavy operations
– Multiple nodes write locking a common entire file
• Use byte range locking
• Separate files
– Reduce use of IO heavy operations
• fsync()
– Some local node optimizations not applicable on cluster
• mmap
© Copyright IBM Corporation 2015

15. Direct I/O
• What is it?
– Bypass Spectrum Scale cache and transfer data directly from disk into
the user space buffer versus placing data in kernel memory.
• Why do I want it?
– Application with low file system cache hit rates
– Application with very large I/O
– When the application knows how to optimize IO
– Example: Oracle Database environment.
• How do I enable it?
– mmchattr –D yes <file_name>, or
– Specify O_DIRECT file access mode on the open() of the file.
© Copyright IBM Corporation 2015

16. RECOVERY SCENARIOS
© Copyright IBM Corporation 2015

17. How to recover: Node Quorum
How to recover from loss of majority for quorum nodes
• Scenario: You have 3 quorum nodes and 2 quorum nodes die.
– Nodes go into Arbitrating state
– Add 2 new quorum nodes
– mmchnode -–quorum –N node4,node5
– Now you have 5 quorum nodes and 3 are running
– Running nodes return to Active state
– Delete missing quorum nodes
– mmdelnode –N badnode1,badnode2
– or
– mmchnode –-nonquorum –N badnode1,badnode2
– Now you have 3 quorum nodes and 3 are running
– Running nodes remain in Active state
• You may have to clean up original quorum nodes once they
recover.
© Copyright IBM Corporation 2015

18. Recover from loss of primary configuration server
How to recover from loss of a primary configuration server
• Scenario: Primary cluster configuration server fails
– Secondary steps in
– Reassign primary
– mmchcluster –p newprimarynode
– To sync up the new primary
– mmchcluster –p LATEST
• Use the same process for failure of secondary cluster
configuration server.
© Copyright IBM Corporation 2015

19. How to recovery: Lose both configuration servers
How to recover from permanent loss of the primary configuration
and secondary servers
• Scenario: Primary and secondary cluster configuration servers
fail
– Cluster status
• Data is online
• mmgetstate -a no longer works
– Reassign both node configuration servers at the same time
– mmchcluster -p nodenewprimary –s nodenewsecondary
– Old servers still think they are in charge until
– mmchcluster –p Latest
© Copyright IBM Corporation 2015

20. Unit summary
Having completed this unit, you should be able to:
• Review some Spectrum Scale best practices
• Discuss approaches to Spectrum Scale performance tuning
• Describe recovery and recovery from quorum and cluster
configuration node failures
© Copyright IBM Corporation 2015