Disaster recovery usingVeritas Storage Foundation Enterprise HAwith IBM Storwize V7000 Metro Mirror and              Globa...
Table of contentsAbstract....................................................................................................
Adding the VCS SAN Volume Controller copy services resource ......................................................... 26  ...
Abstract     This paper describes how Symantec and IBM have installed, configured, and validated high     availability (HA...
About IBMIBM is one of the worlds largest information technology company, with over 80 years of leadership inhelping busin...
The IBM Storwize V7000 system was introduced in the fall of 2010 to provide a cost-effective, flexiblestorage platform des...
Find the IBM Storwize V7000 on the web at: ibm.com/systems/storage/disk/storwize_v7000/index.htmlOverviewThis whitepaper i...
IBM Storwize V7000                                                                                                        ...
Site A                                                                Site B                                              ...
Application host VCS Cluster Servers / IBM POWER7 based LPARsCluster sites                           Site A ( Primary )   ...
Table 3 lists the product documentation set that is required for installation, configuration, andtroubleshooting the setup...
Storwize V7000 Software installationYou can refer to the IBM Storwize V7000 Information Center athttp://publib.boulder.ibm...
Storwize V7000 storage configurationThis section assumes that the Storwize V7000 storage subsystems are up and running. Pr...
IBM_2076:ISV7K4:admin> svctask mkvdiskhostmap –host <hostname 2> –scsi 0 -force 2                                Master cl...
IBM_2076:ISV7K4:admin>To establish fully-functional remote copy partnerships on the Storwize V7000 systems, issue thesecom...
data files and log files are held on separate volumes and consequently are being managed by separaterelationships. In the ...
them to one of the FlashCopy consistency groups created in the “Creating Storwize V7000 Metro Mirror,Global Mirror, and Fl...
Supported AIX operating systemsThis release of Veritas Storage Foundation (5.1, SP1, PR1) operates on:IBM POWER7 processor...
You need to add the following directories to your PATH environment variable:       If you are using Bourne or Korn shell ...
2. Place the Veritas software disk into a DVD drive connected to your system.    3. Mount the disk by determining the devi...
Note: Each system requires a product license before installation. License keys for additional productfeatures should also ...
If the information is correct, press Enter. If the information is not correct, enter n. The installer    prompts you to en...
22. Enter n to enable Storage Foundation Management Server Management. You are prompted to start    Storage Foundation.   ...
# installp -ac -d VRTSvcsor.rte.bff VRTSvcsorInstalling VCS agent for IBM SAN Volume Controller copyservicesThe VCS agent ...
.Configuring applications for disaster recoveryHaving installed and configured the base system software and applications, ...
#cp main.cf.siteA main.cf        #rcp main.cf isvp13:/etc/VRTSvcs/conf/config/main.cf        On Site B cluster node 1 (isp...
Linking clustersRefer to the section Linking clusters in the Veritas Cluster Server User’s Guide. The Remote ClusterConfig...
Configuring VCS application service groupThe VCS service groups can be setup once the VCS agents have been installed. In t...
Figure 3: Sample VCS dependency tree for service group resourcesAdding the VCS SAN Volume Controller copy services resourc...
    Verify the hardware setup for the agent.    Make sure that the cluster has an effective heartbeat mechanism in place...
TsDisaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror:...
TsDisaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror:...
TsDisaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror:...
TsDisaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror:...
TsDisaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror:...
TsDisaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror:...
TsDisaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror:...
TsDisaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror:...
TsDisaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror:...
TsDisaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror:...
TsDisaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror:...
TsDisaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror:...
TsDisaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror:...
TsDisaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror:...
TsDisaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror:...
TsDisaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror:...
TsDisaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror:...
TsDisaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror:...
TsDisaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror:...
TsDisaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror:...
TsDisaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror:...
TsDisaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror:...
TsDisaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror:...
TsDisaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror:...
TsDisaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror:...
Upcoming SlideShare
Loading in …5
×

TsDisaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror: Solution installation and configuration

540 views
426 views

Published on

This paper describes how Symantec and IBM have installed, configured, and validated high availability and disaster recovery configurations for Oracle with IBM Storwize V7000 systems. To know more about the IBM Storwize V7000, visit http://ibm.co/TaLb6Q.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
540
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
11
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

TsDisaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror: Solution installation and configuration

  1. 1. Disaster recovery usingVeritas Storage Foundation Enterprise HAwith IBM Storwize V7000 Metro Mirror and Global Mirror Solution installation and configuration. . . . . . . . IBM Systems and Technology Group ISV Enablement April 2011 © Copyright IBM Corporation, 2011.
  2. 2. Table of contentsAbstract..................................................................................................................................... 1Introduction .............................................................................................................................. 1 About high availability .............................................................................................................................. 1 About disaster recovery ........................................................................................................................... 1 About IBM ................................................................................................................................................ 2 About Symantec ....................................................................................................................................... 2 About Veritas Storage Foundation HA ..................................................................................................... 2 About IBM Storwize V7000 system ......................................................................................................... 2 Storwize V7000 product highlights ................................................................................................... 3Overview ................................................................................................................................... 4Test system configuration ....................................................................................................... 4Installing and configuring Storwize V7000 ............................................................................. 8 Fabric zoning ........................................................................................................................................... 8 Storwize V7000 Software installation ...................................................................................................... 9 Storwize V7000 storage configuration ................................................................................................... 10 Setting up the Storwize V7000 remote copy partnership ...................................................................... 11 Creating Storwize V7000 Metro Mirror, Global Mirror, and FlashCopy consistency groups ................. 12 Creating Storwize V7000 Metro Mirror, Storwize V7000 Global Mirror relationship, and FlashCopy mappings ............................................................................................................................................... 13Installing Veritas Storage Foundation .................................................................................. 14 Symantec product licensing ................................................................................................................... 14 Supported AIX operating systems ......................................................................................................... 15 Database requirements.......................................................................................................................... 15 Disk space ............................................................................................................................................. 15 Environment variables ........................................................................................................................... 15 Virtual IP address ................................................................................................................................... 16 Prerequisites for local and remote cluster installation ........................................................................... 16 Mounting a software disk ....................................................................................................................... 16 Installing SFHA 5.1 SP1PR1 using the Veritas product installers ......................................................... 17 Installing Veritas Storage Foundation HA using webinstaller interface ................................................. 20 Installing VCS agents............................................................................................................................. 20Installing VCS agent for Oracle ............................................................................................. 20Installing VCS agent for IBM SAN Volume Controller copy services ................................. 21Installing and configuring Oracle .......................................................................................... 21Configuring applications for disaster recovery.................................................................... 22 Quick setup ............................................................................................................................................ 22 General configuration steps ................................................................................................................... 23 Setting up Storwize V7000 cluster partnership ...................................................................................... 23 Configuring Storwize V7000 Metro Mirror, GlobalMirror, and FlashCopy relationship .......................... 23 Setting up VCS Global Custer ............................................................................................................... 23 Configuring VCS application service group ........................................................................................... 25 © Copyright IBM Corporation, 2011 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror
  3. 3. Adding the VCS SAN Volume Controller copy services resource ......................................................... 26 Before you configure the agent for SVCCopyServices .................................................................. 26 Adding the agent resource type ..................................................................................................... 27 SVCCopyServices resource type definition ................................................................................... 27 Attribute definitions for the SVCCopyServices agent ..................................................................... 28 Required attributes ......................................................................................................................... 28 Setting up Fire Drill ................................................................................................................................ 29 About Fire Drill resource................................................................................................................. 29 SVCCopyServicesSnap resource type definition ........................................................................... 30 Attribute definitions for the SVCCopyServicesSnap agent ............................................................ 30 Required attributes ......................................................................................................................... 30 Configure the Fire Drill service group ............................................................................................. 31 Enable the Fire Drill attribute .......................................................................................................... 32Failover scenarios .................................................................................................................. 32 Application host failover ......................................................................................................................... 33 Disaster recovery in a Global Cluster configuration .............................................................................. 34 Checking failover readiness with Fire Drill ............................................................................................. 35Summary ................................................................................................................................. 36Appendix A: OpenSSH client configuration between AIX and Storwize V7000 ................. 37Appendix B: Setting up the database applications .............................................................. 41 Setting up the Oracle database application ........................................................................................... 41Appendix C: Sample main.cf file - isvp_sfha_clusterA ........................................................ 46Appendix D: Veritas Software file sets listing ...................................................................... 50Trademarks and Special Notices........................................................................................... 51 © Copyright IBM Corporation, 2011 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror
  4. 4. Abstract This paper describes how Symantec and IBM have installed, configured, and validated high availability (HA) and disaster recovery (DR) configurations for Oracle with IBM Storwize V7000 systems. These validations include local HA configurations using Veritas Storage Foundation and Veritas Cluster Server (VCS). The configuration was extended to a DR configuration using IBM Storwize V7000 Metro Mirror for synchronous replication and Storwize V7000 Global Mirror for asynchronous replication using the VCS agent for IBM SVCCopyServices and VCS Global Cluster Option (GCO) for alternate site failover and failback capability.IntroductionInfrastructure for mission-critical applications must be able to meet the organizations recovery timeobjective (RTO) and recovery point objective (RPO) for resuming operation in the event of a site disaster.This solution addresses environments where the RPOs and RTOs are in the range of minutes to a fewhours. While backup is the foundation for any DR plan, typical RTOs for tape-based backup are wellbeyond these objectives. Also, replication alone is not enough as having the application data at a DR siteis of limited use without also having the ability to start the correct sequence of database managementsystems, application servers, and business applications.Symantec‟s DR solutions, Metro Clustering and Global Clustering, are extensions of local HA clusteringusing Veritas Storage Foundation and Veritas Cluster Server. This validated and documented solution isan example of Global Clustering, which is a collection of two or more VCS clusters at separate locationslinked together with VCS Global Cluster option to enable wide-area failover and disaster recovery. Eachlocal cluster within the global cluster is connected to its own shared storage. Local clustering provideslocal failover for each site. IBM® Storwize® V7000 storage system Metro Mirror replicates data betweensites to maintain synchronized copies of storage at the two sites. For a disaster that affects an entire site,the customer makes a decision on whether or not to move operations to the disaster recovery site. Whenthat decision to move the operations is made, the application is automatically migrated to a system at theDR site. IBM Storwize V7000 Global Mirror replicates data asynchronously between sites and appliesrecovery at the DR sites.About high availabilityThe term high availability or HA refers to a state where data and applications are highly available becausesoftware or hardware is in place to maintain the continued functioning in the event of computer failure. HAcan refer to any software or hardware that provides fault tolerance, but generally the term has becomeassociated with clustering. Local clustering provides HA through database and application failover.Veritas Storage Foundation Enterprise HA (SF/HA) includes Veritas Storage Foundation and VeritasCluster Server and provides the capability for local clustering. The Storwize V7000 disk system includes awide range of HA features as well.About disaster recoveryWide-area disaster recovery provides the ultimate protection for data and applications in the event of adisaster. With an appropriate disaster recovery solution in place, if a disaster affects a local ormetropolitan area, data and critical services can be failed over to a site hundreds or even thousands ofmiles away. IBM Storwize V7000 Metro Mirror and Global Mirror, combined with Veritas StorageFoundation Enterprise HA/DR provide the capability for implementing disaster recovery. © Copyright IBM Corporation, 2011 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror 1
  5. 5. About IBMIBM is one of the worlds largest information technology company, with over 80 years of leadership inhelping businesses innovate by delivering a wide range of solutions and technologies that enablecustomers, large and small, to deliver more-efficient and effective services. IBMs comprehensive server,storage, software, and services portfolio are designed to help you create new business insight byintegrating, analyzing, and optimizing information on demand. From its foundations of virtualization,openness, and innovation through collaboration, IBM can optimize management of information throughtechnology innovations and infrastructure simplification to help achieve maximum business productivity.You can visit IBM at ibm.com.About SymantecSymantec is a global leader in infrastructure software, enabling businesses and consumers to haveconfidence in a connected world. The company helps customers protect their infrastructure, information,and interactions by delivering software and services that address risks to security, availability,compliance, and performance. Headquartered in Cupertino, California, Symantec has operations in morethan 40 countries. You can visit Symantec at symantec.comAbout Veritas Storage Foundation HAVeritas Storage Foundation HA is a comprehensive solution that delivers data and application availabilityby bringing together two industry-leading products: Veritas Storage Foundation and Veritas ClusterServer. Veritas Storage Foundation provides a complete solution for heterogeneous online storage management.Based on the industry-leading Veritas Volume Manager and Veritas File System, it provides a standardset of integrated tools to centrally manage explosive data growth, maximize storage hardwareinvestments, provide data protection, and adapt to changing business requirements. Unlike pointsolutions, Storage Foundation enables IT organizations to manage their storage infrastructure with onetool. With advanced features, such as centralized storage management, nondisruptive configuration andadministration, dynamic storage tiering, dynamic multipathing, data migration, and local and remotereplication, Storage Foundation enables organizations to reduce operational costs and capitalexpenditures across the data center. Veritas Cluster Server is an industry leading clustering solution for reducing both planned and unplanneddowntime. By monitoring the status of applications and automatically moving them to another server inthe event of a fault, Veritas Cluster Server can dramatically increase the availability of an application ordatabase. Veritas Cluster Server can detect faults in an application and all its dependent components,including the associated database, operating system, network, and storage resources. When a failure isdetected, Veritas Cluster Server gracefully shuts down the application, restarts it on an available server,connects it to the appropriate storage device, and resumes normal operations. Veritas Cluster Server cantemporarily move applications to a standby server when routine maintenance, such as upgrades orpatches, requires that the primary server be taken offline.About IBM Storwize V7000 systemIBM Storwize V7000 system is a large midrange storage platform, which in addition to new functionality,leverages proven virtualization technologies available previously in other popular IBM storage products. © Copyright IBM Corporation, 2011 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror 2
  6. 6. The IBM Storwize V7000 system was introduced in the fall of 2010 to provide a cost-effective, flexiblestorage platform designed to help users reduce the costs and complexity associated with storage, andimprove efficiency. Similar to the earlier IBM System Storage® SAN Volume Controller product, the IBMStorwize V7000 system is capable of pooling storage volumes from other IBM and non-IBM storagesystems into reservoirs of capacity for centralized management. One of thekey differentiators between Storwize V7000 system and the IBM SANVolume Controller is that the Storwize V7000 ships with serial-attachedSCSI (SAS) solid-state drives (SSDs), or a combination of each in the primary storage engine, and canbe scaled further in size using expansion enclosures containing additional drives. Storwize V7000 product highlights The Storwize V7000 system is designed to:  Provide mid-range customers with a cost-effective, scalable storage platform that can provide advanced features available typically only with more-expensive enterprise class products  Deliver a new, easy to use graphical interface which makes exploiting the features of this platform simple  Introduce new automated tiering capability called IBM Easy Tier™ technology, which when enabled, can reallocate the busiest disk extents to faster SSDs, maximizing the utilization of storage resources  Provide multiprotocol support for Fibre Channel (FC) and Internet Small Computer Systems Interface (iSCSI) attachment.   Deliver the same key virtualization features made popular with IBM SAN Volume Controller. The Storwize V7000 system can also: − Combine storage capacity from multiple disk systems into a single reservoir of capacity that can be better managed as a business resource and not as separate boxes − Help increase storage utilization by providing host applications with more-flexible access to capacity − Help improve productivity of storage administrators by enabling management of heterogeneous storage systems using a common interface − Support improved application availability by insulating host applications from changes to the physical storage infrastructure − Enable a tiered storage environment in which the cost of storage can be better matched to the value of data − Support advanced copy services from higher to lower-cost devices and across storage systems from multiple vendors © Copyright IBM Corporation, 2011 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror 3
  7. 7. Find the IBM Storwize V7000 on the web at: ibm.com/systems/storage/disk/storwize_v7000/index.htmlOverviewThis whitepaper illustrates the steps involved in installing and configuring Veritas Storage Foundation HAand IBM Storwize V7000 Metro Mirror (synchronous copy) or IBM Storwize V7000 Global Mirror(asynchronous copy) for a host failover and disaster recovery readiness scenario.The host failover scenario simulates a fault in a node of a two-node Veritas Cluster Server (VCS) clusterat Site A. When an application service group fails over to the other node, a fault is injected in to that nodecausing the application to failover to a node of a two-node cluster at Site B.The disaster recovery readiness scenario simulates a disaster by introducing a fault in affecting both thenodes of the cluster and storage at Site A, causing the application to failover to a node of a two-nodecluster at Site B. VCS configured with Global Cluster Option (GCO) brings up the application globalservice group on that node at Site B. As soon as Site A is restored and becomes available, theapplication is failed back to Site A. The VCS IBM Storwize V7000 Copy Services agent manages thestate of the Storwize V7000 remote copy relationship, required for failover and failback. The “Failoverscenarios” section provides procedural details.Two sets of volume groups are used for the configuration described for the scenario in this section. Oneset is used for configuring Metro Mirror and another set is used for Global Mirror. Optionally, to test for DRreadiness the VCS Fire Drill set is configured. A fire drill procedure verifies the fault-readiness of a ®disaster recovery configuration. VCS Fire Drill uses IBM Tivoli® Storage FlashCopy Manager to createcopies of the Metro Mirror or Global Mirror target volumes at Site B. A third volume group set is used forFlashCopy.This paper provides instructions for installing and configuring a test setup and then provides proceduresfor the scenarios.Test system configurationA cluster configuration can be configured as shown in Figure 1. This configuration includes:  A cluster at Site A consisting of two IBM POWER7® processor-based servers running on IBM AIX®, which are virtual I/O (VIO) client logical partitions (LPARs), configured as a two-node VCS cluster, a Storwize V7000 system providing FC-attached storage volumes to the AIX VIO server, then through to the client LPAR.  A cluster at Site B consisting of two IBM POWER7® processor-based servers running on IBM AIX, which are VIO client LPARs configured as a two-node VCS cluster, with a Storwize V7000 system providing FC-attached storage volumes to the AIX VIO server, then through to the client LPAR.  GCO in VCS on each cluster was enabled to allow failover of resource groups between the two clusters. © Copyright IBM Corporation, 2011 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror 4
  8. 8. IBM Storwize V7000 IBM Storwize V7000 6.1.0.0 6.1.0.0 2498-B24 0 4 1 5 2 6 3 7 8 12 9 13 10 14 11 15 16 20 17 21 18 22 19 23 2498-B24 0 4 1 5 2 6 3 7 8 12 9 13 10 14 11 15 16 20 17 21 18 22 19 23 2498-B24 0 4 1 5 2 6 3 7 8 12 9 13 10 14 11 15 16 20 17 21 18 22 19 23 2498-B24 0 4 1 5 2 6 3 7 8 12 9 13 10 14 11 15 16 20 17 21 18 22 19 23 SAN24B-4 SAN24B-4 SAN24B-4 SAN24B-4 isvp13 / AIX 7.1 570 isvp12 / AIX 7.1 isvp14 / AIX 7.1 570 isvp15 / AIX 7.1 SFHA SFHA SFHA SFHA 5.1SP1,PR1 5.1SP1,PR1 5.1SP1,PR1 5.1SP1,PR1 p790 p790 Isv_sfha_clusterA Isv_sfha_clusterB Site A Site BFigure 1: VCS cluster and IBM Storwize V7000 Metro Mirror and Global Mirror for disaster recoveryThe high-level configuration setup built for this white paper (refer to Figure 1) consists of four applicationhosts. The configuration and hardware and software components are listed in Table 1 and Table 2. Thehosts are VIO client LPARs to a single VIO server on each of the two physical IBM Power® 790 servers.The four LPARs are configured to form two 2-node Storage Foundation for High Availability (SFHA)clusters. Figure 2 shows a detailed test system configuration.The two clusters represent two sites, Site A and Site B. The cluster at Site A is the active cluster and thecluster at Site B is the standby cluster. Table 5 shows the Storwize V7000 logical unit number (LUN)mapping. As this is a VIO server configuration, all volumes are mapped to the VIO server on thePower 790 server at each site, and then passed through to a virtual Small Computer System Interface(VSCSI) adapter on the VIO client LPAR (the cluster nodes).In the test configuration, there is only one interswitch link (ISL) shown between the remote and localcluster. This is adequate for demonstration but the design of a production configuration should ensureadequate provisioning of intercluster bandwidth. More than two ISLs might be required and they mightneed to be trunked. In this case, 900 Mbps of bandwidth has been allocated for the interclusterconnectivity. © Copyright IBM Corporation, 2011 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror 5
  9. 9. Site A Site B mdisk3 mdisk6 mdisk6 mdisk7 Storage Pool mdisk8 „mdiskgrp 0 ‟ - mdisk9 8. 1 TB mdisk10 mdisk11 Storage Pool „isvp‟- 2 TB sfha_ gmdisk001 sfha_ dr_ gmdisk001 sfha_ gmdisk002 sfha_ dr_ gmdisk002 sfha_ mmdisk001 sfha_dr_ mmdisk001 isv7k4 sfha_ mmdisk002 sfha_dr_ mmdisk002 isv7 kd10 sfha_FD_ dr_ gmdisk001 sfha_FD_ dr_ gmdisk002 sfha_ FD_ dr_ mmdisk001 sfha_ FD_ dr_ mmdisk002 2498 B24 - 04 1 5 26 3 7 8 12 9 13 10 14 11 15 16 20 17 21 18 22 19 23 2498 B24 - 04 1 5 26 3 7 8 12 9 13 10 14 11 15 16 20 17 21 18 22 19 23 2498 B24 - 04 1 5 26 3 7 8 12 9 13 10 14 11 15 16 20 17 21 18 22 19 23 2498 B24 - 04 1 5 26 3 7 8 12 9 13 10 14 11 15 16 20 17 21 18 22 19 23 SAN24B-4 SAN24B-4 SAN24B-4 SAN24B-4 SAN02 SAN04 SAN02 01 SAN03 isv_ sfha_ clust isv_ sfha_ clust isv_ sfha_ clust isv_ sfha_ clust erA erA erB erB isvp12 isvp13 isvp 14 isvp 15 IBM p 790 IBM p790 IBM p790 IBM p 790 AIX 7 .1 AIX 7 .1 AIX 7. 1 AIX 7. 1 Symantec Symantec Symantec Symantec SFHA SFHA SFHA SFHA 5. 1 SP1 PR1 5 .1 SP1 PR1 5. 1 SP1 PR1 5. 1 SP1 PR1 Site A Site B GM MM Firedrill / FlashCopyFigure 2: Test cluster storage configuration with Storwize V7000 © Copyright IBM Corporation, 2011 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror 6
  10. 10. Application host VCS Cluster Servers / IBM POWER7 based LPARsCluster sites Site A ( Primary ) Site B ( Secondary )VCS cluster names isvp_sfha_clusterA isvp_sfha_clusterBSystem / Cluster node isvp12 isvp13 isvp14 isvp15namesNumber of processors 2 2 2 2Processor clock 3000 MHz 3000 MHz 3300 MHz 3300 MHzspeedProcessor type 64-bit 64-bit 64-bit 64-bitKernel type 64-bit 64-bit 64-bit 64-bitMemory size 8192 MB 8192 MB 8192 MB 8192 MBPhysical machine 8233-E8B 8233-E8B 8233-E8B 8233-E8Btype / modelActivated platform 86 86 86 86level Storwize V7000 configurationCluster sites Site A ( Master ) Site B ( Auxiliary )Hardware model 2076-124Basic I/O System American Megatrends v.28(BIOS) versionCode level 6.1.0.0 (build 49.0.1010130001)Storwize V7000 isv7k4.storage.tucson.ibm.com isv7kd10.storage.tucson.ibm.comSystem / Cluster node node1 node2 node1 node2namesConfiguration node Yes No Yes NoHost bus adapter 4 4(HBA) portsCapacity 15.5 TB / 3.5 TB assigned 8.1 TB / 1.8 TB assignedI/O group io_grp0 io_grp0Table 1: Hardware configuration SAN Switch model IBM 2498-B24 x 4 Firmware version V6.4.0a Ports 24 each Vendor Software VersionIBM AIX 7.1 7.1.0.0Oracle Corporation Oracle 11gR2Symantec Veritas Storage Foundation Enterprise 5.1 SP1 PR1Symantec Veritas High Availability 5.1 SP1 PR1 5.1SP1,PR1 Agent for Oracle by SymantecSymantec Veritas Clustering Support for IBM 5.0.6.0 SVCCopyServices (VRTSvcssvc.rte)Table 2: Software configuration © Copyright IBM Corporation, 2011 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror 7
  11. 11. Table 3 lists the product documentation set that is required for installation, configuration, andtroubleshooting the setup. Product Use this guideStorwize V7000 Storwize V7000 Infocenter http://publib.boulder.ibm.com/infocenter/storwize/ic/index.jspVeritas Storage Veritas Storage Foundation and High Availability Installation GuideFoundation https://sort.symantec.com/public/documents/sfha/5.1sp1pr1/aix/productguides/pdEnterprise HA f/sf_install_51sp1pr1_aix.pdfVeritas Cluster Veritas Cluster Server Installation GuideServer https://sort.symantec.com/public/documents/sfha/5.1sp1pr1/aix/productguides/pd f/vcs_install_51sp1pr1_aix.pdf Veritas Cluster Server Administrator‟s Guide https://sort.symantec.com/public/documents/sfha/5.1sp1pr1/aix/productguides/pd f/vcs_admin_51sp1pr1_aix.pdfVeritas Volume Veritas Volume Manager Administrators GuideManager https://sort.symantec.com/public/documents/sfha/5.1sp1pr1/aix/productguides/pd f/vxvm_admin_51sp1pr1_aix.pdfVeritas Cluster Veritas Cluster Server Agent for Oracle Installation and Configuration GuideServer Agents https://sort.symantec.com/public/documents/sfha/5.1sp1pr1/aix/productguides/pd f/vcs_oracle_agent_51sp1pr1_aix.pdf Veritas Cluster Server Agent for IBM SVCCopyServices Installation and Configuration Guide https://sort.symantec.com/agents/download_docs/2192/vcs_svc_installTable 3: Required documentsInstalling and configuring Storwize V7000Included here are some general guidelines for configuration of the Storwize V7000 in preparation for theStorage Foundation for High Availability installation.Fabric zoningFor more details on zoning requirements for Storwize V7000, review the sections on Zoning details andZoning examples available in the Storwize V7000 Information Center athttp://publib.boulder.ibm.com/infocenter/storwize/ic/index.jsp. Table 4 shows the zones created for thisconfiguration. Zone configuration – Fabric 1 and 2 (SAN01, SAN03 / Zone SAN02, SAN04) ISV7K4 includes aliases for two Storwize V7000 ports and a port each from ISVP12 / ISVP13 includes aliases for two Storwize V7000 ports and a port each ISV7KD10 from ISVP14 / ISVP15 includes one Storwize V7000 port from each disk system toISV7KD10_ISV7K4_Mirror allow formation of intercluster link, for remote copy (Global Mirror and Metro Mirror)Table 4: Fabric zoning © Copyright IBM Corporation, 2011 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror 8
  12. 12. Storwize V7000 Software installationYou can refer to the IBM Storwize V7000 Information Center athttp://publib.boulder.ibm.com/infocenter/storwize/ic/index.jsp to obtain detailed planning, installation, andsoftware configuration instructions.From the Storwize V7000 GUI interface or command-line interface (CLI), ensure FlashCopy, Metro Mirror,and Global Mirror licenses are activated. © Copyright IBM Corporation, 2011 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror 9
  13. 13. Storwize V7000 storage configurationThis section assumes that the Storwize V7000 storage subsystems are up and running. Provided youhave configured the Secure Shell (SSH) client on each participating node in your cluster, you canSSH/login to the Storwize V7000 CLI from any VCS cluster node or you can connect to the StorwizeV7000 integrated GUI console using the IP address of the controller canister. You can find detailedinstruction on configuring SSH between AIX hosts and the Storwize V7000 disk system in “Appendix A:OpenSSH client configuration between AIX and Storwize V7000”.The method shown in this section demonstrates the Storwize V7000 CLI interface that is used toconfigure volumes from managed disks (MDisks) available to the Storwize V7000 storage system. WithStorwize V7000, MDisks can be either LUNS from an external SAN-attached storage device oralternatively (as in this case) from SAS drives or SSDs available inside the Storwize V7000 disk system.In this configuration, the storage is Storwize V7000 physical disk. The volumes were created from astorage pool containing seven MDisks, each of which represents an individual RAID5 array spanning 7 or8 x 300 GB SAS drives – delivering a usable capacity of 1.6 TB or 1.9 TB, respectively.Typical disk configuration consists of initial disk discovery (in the case of externally-attached SAN disks),or configuration of existing physical disks in the Storwize V7000 disk enclosures into MDisks/arrays of aparticular RAID type, then adding these to one or more storage pools. Volumes are then created from theavailable capacity in the storage pools and finally are mapped to the hosts.The volume and host mapping required for this setup is shown in this section. Table 5 shows the volumesconfigured for Metro and Global Mirror used in this configuration. The following commands given providean example of creating storage pools (formerly manged disk groups), volumes, and mapping volumes tohosts.Creating a storage pool (formerly known as managed disk group): IBM_2076:ISV7K4:admin> svctask detectmdisk IBM_2076:ISV7K4:admin> svcinfo lsmdiskcandidate IBM_2076:ISV7K4:admin> svcinfo lsmdisk -delim : -filtervalue mode=unmanaged IBM_2076:ISV7K4:admin> svctask mkmdiskgrp -name isvp -ext 256 -mdisk mdisk3 IBM_2076:ISV7K4:admin> svctask addmdisk -mdisk mdisk6 mdisk7 mdisk8 mdisk9 mdisk10 mdisk11 isvp IBM_2076:ISV7K4:admin> svcinfo lsmdiskgrpCreating volumes (formerly known as virtual disks or VDisks - this sample shows creation ofMetro Mirror volumes): IBM_2076:ISV7K4:admin> svctask mkvdisk -name sfha_mmdisk001 -iogrp io_grp0 - mdiskgrp ispv –vtype striped -size 100 -unit gb IBM_2076:ISV7K4:admin> svctask mkvdisk -name sfha_mmdisk002 -iogrp io_grp0 - mdiskgrp isvp -vtype striped -size 100 -unit gb IBM_2076:ISV7K4:admin> svcinfo lsvdiskMapping VDisks to hosts: IBM_2076:ISV7K4:admin> svctask lsvdiskhostmap –host <hostname1> –scsi 0 –force 1 © Copyright IBM Corporation, 2011 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror 10
  14. 14. IBM_2076:ISV7K4:admin> svctask mkvdiskhostmap –host <hostname 2> –scsi 0 -force 2 Master cluster (Site A) Auxiliary cluster (Site B) Volumes for Metro Mirror sfha_mmdisk001-002 sfha_DR_mmdisk001-002 Volumes for Global Mirror sfha_gmdisk001-002 sfha_DR_gmdisk001-002 sfha_FD_dr_mmdisk001-002 Volumes for Fire - sfha_FD_dr_gmdisk001-002 Drill Relationship Name Master volume Auxiliary volume Metro Mirror sfha_mmdisk001- relationship v7k_mm_DR 002 sfha_dr_mmdisk001-002 Global Mirror sfha_gmdisk001- relationship v7k_gm_DR 002 sfha_dr_gmdisk001-002 FlashCopy Name Fire Drill (FD)volume Auxiliary volume mappings FC_mm_fcmap0001- sfha_FD_dr_mmdisk001- sfha_dr_mmdisk001- Metro Mirror 0002 002 002 FD_gm_fcmap0001- sfha_FD_dr_gmdisk001- sfha_dr_mmdisk001- Global Mirror 0002 002 002 FlashCopy consistency group FD_mm_fcconsistgrp (Metro Mirror volumes) FlashCopy consistency group (Global FD_gm_fcconsistgrp Mirror volumes) Metro Mirror and Global Mirror consistency groups Name State Copy Master Auxiliary Primary Relationship type cluster cluster count v7k_mm_D Consistent Metro isv7k4 isv7kd10 Master 2 R synchronized v7k_gm_D Consistent Global isv7k4 isv7kd10 Master 2 R synchronized Cluster Location State Bandwidth Link Inter-cluster Intra-cluster Name tolerance delay delay (s) simulation simulation (ms) (ms) Metro Mirror cluster partnership isv7k4 Remote Fully 900 300 20 0 configured Global Mirror cluster partnership isv7kd10 Remote Fully 900 300 20 0 configuredTable 5: Storwize V7000 disk configurationSetting up the Storwize V7000 remote copy partnershipLogin to the Storwize V7000 through SSH from any VCS cluster node or alternatively connect to theStorwize V7000 GUI console.#ssh -l admin -i /.ssh/id_dsa isv7k4.storage.tucson.ibm.com © Copyright IBM Corporation, 2011 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror 11
  15. 15. IBM_2076:ISV7K4:admin>To establish fully-functional remote copy partnerships on the Storwize V7000 systems, issue thesecommands on both the disk systems. This step is a prerequisite for creating Metro Mirror or GlobalMirrors between volumes on the Storwize V7000 systems.IBM_2076:ISV7K4:admin> svctask mkpartnership -bandwidth 900 ISV7KD10IBM_2076:ISV7KD10:admin> svctask mkpartnership -bandwidth 900 ISV7K4Update intercluster delay for the Global Mirror configuration. To configure the parameters, you can referto chapter 15 of IBM System Storage SAN Volume Controller and Storwize V7000 Command-LineInterface User’s Guide at:ftp://ftp.software.ibm.com/storage/san/sanvc/V6.1.0/SVC_and_V7000_Command_Line_Interface_Users_Guide.pdf.Available intercluster delay parameters include:-gmlinktolerance link_tolerance:  Is the length of time, in seconds, for which an inadequate inter-cluster link will be tolerated for Global Mirror operation.  Accepts values from 60 to 86400 seconds in steps of 10 seconds. Default is 300 seconds. The link tolerance can be disabled by entering the value 0 for this parameter.-gminterdelaysimulation inter_cluster_delay_simulation:  Is the Inter-cluster delay simulation that simulates Global Mirror round-trip delay between two clusters, in milliseconds. Default is 0, valid range is 0 to 100 milliseconds.-gmintradelaysimulation intra_cluster_delay_simulation:  Is the Intra-cluster delay simulation that simulates Global Mirror round trip-delay in milliseconds. Default is 0, valid range is 0 to 100 milliseconds.You can get the cluster alias using the following commands.IBM_2076:ISV7K4:admin> svcinfo lscluster ISV7K4IBM_2076:ISV7K4:admin> svctask chcluster -alias 00000200A0C000A0 -name ISV7K4 -gmlinktolerance 300 -gminterdelaysimulation 20 -gmintradelaysimulation 0Creating Storwize V7000 Metro Mirror, Global Mirror, and FlashCopy consistencygroupsYou can create Metro Mirror and Global Mirror consistency groups by specifying a name and the remoteStorwize V7000 name as shown below. Make sure the two Storwize V7000 systems are up and incommunication throughout the create process. The new consistency group does not contain anyrelationships and will be in an empty state. A consistency group is used to ensure that a number ofrelationships are managed so that in the event of a disconnection of the relationships, the data on allvolumes within the group is in a consistent state. This can be important in a database application where © Copyright IBM Corporation, 2011 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror 12
  16. 16. data files and log files are held on separate volumes and consequently are being managed by separaterelationships. In the event of a disaster, the primary and secondary sites might become disconnected. Ifthe relationships associated with the volumes are not in a consistency group, then as the disconnectionhappens, and the Mirror relationships stop copying data from the primary to the secondary site, there isno assurance that updates to the two separate secondary volumes stops in a consistent manner.IBM_2076:ISV7K4:admin> svctask mkrcconsistgrp -name v7k_mm_DR -cluster ISV7K4IBM_2076:ISV7K4:admin> svctask mkrcconsistgrp -name v7k_gm_DR -cluster ISV7K4The VCS Fire Drill function utilizes FlashCopy to create and maintain the snapshot of the applicationresource. Therefore create a FlashCopy consistency group from the Storwize V7000 at Site B.You need to ensure that the Storwize V7000 FlashCopy license is enabled. In this case, the test team iscreating a FlashCopy consistency group for the Metro Mirror volumes and another for Global Mirrorvolumes.IBM_2076: ISV7KD10:admin> svctask chlicense -flash onIBM_2076:ISV7KD10:admin>svctask mkfcconsistgrp -name FD_mm_fcconsistgrpFlashCopy Consistency Group, id [1], successfully createdIBM_2076:ISV7KD10:admin>svctask mkfcconsistgrp -name FD_gm_fcconsistgrpFlashCopy Consistency Group, id [2], successfully createdCreating Storwize V7000 Metro Mirror, Storwize V7000 Global Mirror relationship,and FlashCopy mappingsNow create Metro Mirror and Global Mirror relationships using the following command. This relationshippersists until it is deleted. The auxiliary virtual disk needs to be identical in size to the master virtual disk.The master and auxiliary cannot be in an existing relationship. Neither of the disks can be the target of aFlashCopy mapping. Copy type defaults to metro if -global is not specified.IBM_2076:ISV7K4:admin>svctask mkrcrelationship -master 6 -aux 11 -sync -cluster ISV7KD10 -name v7k_mmrel001 -consistgrp v7k_mm_DRIBM_2076:ISV7K4:admin>svctask mkrcrelationship -master 7 -aux 13 -sync -cluster ISV7KD10 -name v7k_mmrel002 -consistgrp v7k_mm_DRIBM_2076:ISV7K4:admin>>svctask mkrcrelationship -master 8 -aux 13 -sync -cluster ISV7KD10 -name v7k_gmrel001 -consistgrp v7k_gm_DR -globalIBM_2076:ISV7K4:admin>>svctask mkrcrelationship -master 9 -aux 14 -sync -cluster ISV7KD10 -name v7k_gmrel002 -consistgrp v7k_gm_DR -globalNext, you need to create FlashCopy mappings. The mkfcmap command creates a new FlashCopymapping that maps a source volume to a target volume for subsequent copying. Here, the test team hasa FlashCopy target volume on the Stowize V7000 for every Metro Mirror and Global Mirror volume to beused to support Fire Drill testing. The test team is creating a total of four FlashCopy mappings and adding © Copyright IBM Corporation, 2011 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror 13
  17. 17. them to one of the FlashCopy consistency groups created in the “Creating Storwize V7000 Metro Mirror,Global Mirror, and FlashCopy consistency groups” sectionIBM_2076:ISV7KD10:admin>svctask mkfcmap -source 13 -target 17 -consistgrp 2 -nameFC_gm_fcmap0001 -copyrate 100 -incremental -cleanrate 100FlashCopy Mapping, id [1], successfully createdIBM_2076:ISV7KD10:admin>svctask mkfcmap -source 14 -target 18 -consistgrp 2 -nameFC_gm_fcmap0002 -copyrate 100 -incremental -cleanrate 100FlashCopy Mapping, id [2], successfully createdIBM_2076:ISV7KD10:admin>svctask mkfcmap -source 11 -target 15 -consistgrp 1 -nameFC_mm_fcmap0001 -copyrate 100 -incremental -cleanrate 100FlashCopy Mapping, id [3], successfully createdIBM_2076:ISV7KD10:admin>svctask mkfcmap -source 12 -target 16 -consistgrp 1 -nameFC_mm_fcmap0002 -copyrate 100 -incremental -cleanrate 100FlashCopy Mapping, id [4], successfully createdInstalling Veritas Storage FoundationThis section provides a walk through of the installation prerequisites, as well as detailed command lineoutput resulting from the installation of our first SFHA cluster.Symantec product licensingSymantec SFHA licensing is typically done during product installation. During the installation process,both of the Symantec installer options provide opportunity to enter the product license key.The VRTSvlic package enables product licensing. After VRTSvlic is installed, the following commandsand their manual pages are available on the system:  vxlicinst installs a license key for a Symantec product  vxlicrep displays currently installed licenses  vxlictest retrieves features and their descriptions encoded in a license keyYou need to make sure that you have activated the Veritas Storage Foundation Enterprise HA/DRAIX 5.1 license key. © Copyright IBM Corporation, 2011 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror 14
  18. 18. Supported AIX operating systemsThis release of Veritas Storage Foundation (5.1, SP1, PR1) operates on:IBM POWER7 processor running  AIX 7.1 TL0 or higher  AIX 6.1 TL5 with SP1 or higher  AIX 5.3 in POWER6 or IBM Power6+™ compatibility at TL11, with Service Pack 2 or higher, or TL 10 with Service Pack 4 or later.IBM POWER6® processors, at one of the following levels:  AIX 7.1, TL0 or higher  AIX 6.1, TL2 or higher  AIX 5.3 at one of the following levels: − TL7 with Service Pack 6 or higher − TL8 with Service Pack 4 or higherBoth of the Symantec product installer interfaces verify the required update levels. The installationprocess terminates if the target systems do not meet maintenance-level requirements.For any Veritas cluster product, all nodes in the cluster need to have the same operating system versionand update level.Database requirementsThe Oracle database version supported by Veritas high availability release 5.1SP1, PR1 is Oracle 11gR2,but generally the VCS agent supports a particular Oracle release as long as Oracle already supports therelease on that particular AIX level.Disk spaceUse the Perform a Preinstallation Check (P) option from the product installer to determine whetherthere is sufficient space.Environment variablesMost of the commands used in the installation are in the /sbin or /usr/sbin directory. However, there areadditional variables that are required to use a Veritas Storage Foundation product after installation. © Copyright IBM Corporation, 2011 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror 15
  19. 19. You need to add the following directories to your PATH environment variable:  If you are using Bourne or Korn shell (sh or ksh), use the commands:$ PATH=$PATH:/usr/sbin:/opt/VRTSvxfs/sbin:/opt/VRTSob/bin:/opt/VRTSvcs/bin:/opt/VRTS/bin$MANPATH=/usr/share/man:/opt/VRTS/man:$MANPATH$ export PATH MANPATH  If you are using a C shell (csh or tcsh), use the commands:% set path = ( $path /usr/sbin /opt/VRTSvxfs/sbin /opt/VRTSvcs/bin /opt/VRTSob/bin/opt/VRTS/bin )% setenv MANPATH /usr/share/man:/opt/VRTS/man:$MANPATHNote: The nroff versions of the online manual pages are not readable using the man command if thebos.txt.tfs file set is not installed; however, the VRTSvxvm and VRTSvxfs packages install ASCII versionsin the /opt/VRTS/ man/catman* directories that are readable without the bos.txt.tfs file set.Virtual IP addressThis configuration needs several IP addresses depending on the products you are enabling. You need tohave at least six virtual IPs allocated for the two clusters. The following list shows virtual IPs required forthis configuration, in this case a virtual address on each cluster for Oracle and Global Cluster. Purpose isvp_sfha_clusterA isvp_sfha_clusterB Oracle failover 10.10.0.152 10.10.0.153 GCO 10.10.0.150 10.10.0.151Prerequisites for local and remote cluster installationEstablishing communication between nodes is required to install Veritas software from a remote systemor to install and configure a cluster.The node from which the installation utility is running must have permissions to run remote shell (rsh) orSSH utilities as root on all cluster nodes or remote systems. For both of the cluster installations, the SFHAproduct code was unpacked on a single node and deployed to the remaining hosts using the Symantecinstaller invoked on that initial host (isvp12).You need to make sure that the hosts to be configured as cluster nodes have two or more networkinterface cards (NICs) and are connected for heartbeat links. Refer to the Veritas Cluster Serverinstallation Guide at:https://sort.symantec.com/public/documents/sfha/5.1sp1pr1/aix/productguides/pdf/vcs_install_51sp1pr1_aix.pdf for more details.Mounting a software diskIf you are using DVD as an installation media, you must have superuser (root) privileges to mount theVeritas disks.To mount the Veritas software disk: 1. Log in as superuser. © Copyright IBM Corporation, 2011 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror 16
  20. 20. 2. Place the Veritas software disk into a DVD drive connected to your system. 3. Mount the disk by determining the device access name of the DVD drive. The format for the device access name is cdX where X is the device number. After inserting the disk, type the following commands: # mkdir -p /cdrom # mount -V cdrfs -o ro /dev/cdX /cdromInstalling SFHA 5.1 SP1PR1 using the Veritas product installersNote: Veritas products are installed under the /opt directory on the specified host systems. Ensure thatthe directory /opt exists and has write permissions for root before starting an installation procedure.The Veritas product installer is the recommended method to license and install the product. There arecommand-line-driven and web-based versions of the product installer available. Both the versions enableyou to verify preinstallation requirements, configure the product, and view the product‟s description.You can use the product installer to install Veritas Storage Foundation and Veritas Storage FoundationEnterprise HA. Usually, during an installation, you can type b (back) to return to a previous section of theinstallation procedure. The back feature of the installation scripts is context-sensitive, so it returns to thebeginning of a grouped section of questions. If an installation procedure hangs, press Ctrl+C to stop andexit the program. There is a short delay before the script exits.To install a Storage Foundation product, run the following steps from one node in each cluster.1. If the installation file sets are on a DVD media, make sure the disk is mounted. Refer to the “Mounting a software disk” section. This installation scenario uses downloaded compressed tarfiles that have been unpacked into the /tmp directory on one of the cluster nodes.2. To invoke the common installer, run the installer command on the disk as shown in this example: # /tmp/installer (to invoke the menu-based installation interface) or, alternatively # /tmp/webinstaller (to invoke the web-based installation interface)3. Enter I to install a product and press Enter to begin.4. When the list of available products is displayed, select the product you want to install and enter the corresponding number and press Enter. The product installation begins automatically.5. Enter the Storage Foundation Enterprise HA/DR product license information. Enter a product_name license key for isvp_sfha_clusterA: [?] XXXX-XXXX-XXXX-XXXX-XXXX- XXXX-X XXXX-XXXX-XXXX-XXXX-XXXX-XXXX-X successfully registered on isvp_sfha_clusterA Do you want to enter another license key for isvp_sfha_clusterA? [y,n,q,?] (n) Enter a product_name license key for isvp_sfha_clusterB: [?] XXXX-XXXX-XXXX-XXXX-XXXX- XXXX-X XXXX-XXXX-XXXX-XXXX-XXXX-XXXX-X successfully registered on isvp_sfha_clusterB Do you want to enter another license key for isvp_sfha_clusterB? [y,n,q,?] (n) Enter n if you have no further license keys to add for a system. You are then prompted to enter the keys for the next system. © Copyright IBM Corporation, 2011 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror 17
  21. 21. Note: Each system requires a product license before installation. License keys for additional productfeatures should also be added at this time.6. Choose to install all file sets. SF can be installed without optional filesets to conserve disk space. Additional filesets are typically installed to simplify future upgrades. 1) Required Veritas Storage Foundation filesets - 928 MB required 2) All Veritas Storage Foundation filesets - 1063 MB required Select the filesets to be installed on all systems? [1-2,q,?] (2)7. At the installed product list page, enter y or press Enter to configure the Storage Foundation and VCS products. It is possible to install SF filesets without performing configuration. It is optional to configure SF now. If you choose to configure SF later, you can do so manually or run the installsf -configure command. Are you ready to configure SF? [y,n,q] (y) Do you want to configure VCS on these systems at this time? [y,n,q] (y)8. The installer prompts for details for configuring the VCS cluster for Storage Foundation (SF) Enter the unique cluster name and cluster ID number. Enter the unique cluster name: [?]isvp_sfha_clusterA Enter the unique Cluster ID number between 0-65535: [b,?] 0 The installer discovers the NICs available on the first system and reports them, Discovering NICs on isvp_sfha_clusterA...discovered en0 en1 en29. Enter private heartbeat NIC information for each host. Enter the NIC for the first private heartbeat link on isvp12:[b,?] en1 Would you like to configure a second private heartbeat link?[y,n,q,b,?] (y) y Enter the NIC for the second private heartbeat link on isvp12:[b,?] en2 Would you like to configure a third private heartbeat link?[y,n,q,b,?] (n) n Do you want to configure an additional low priority heartbeat link? [y,n,q,b,?] (n) n Are you using the same NICs for private heartbeat links on all systems? [y,n,q,b,?] (y) y Note: When answering as y, be sure that the same NICs are available on each system; the installer does not verify this. Notice that in this example, en0 is not selected for use as a private heartbeat NIC because it is already in use as the public network interface.10. A summary of the information you entered is given. When prompted, confirm that the information is correct. Is this information correct? [y,n,q] (y) © Copyright IBM Corporation, 2011 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror 18
  22. 22. If the information is correct, press Enter. If the information is not correct, enter n. The installer prompts you to enter the information again.11. When prompted to configure the product to use Veritas Security Services, enter y or n to configure. Note: Before configuring a cluster to operate using Veritas Security Services, another system must already have Veritas Security Services installed and be operating as a Root Broker. Refer to the Veritas Cluster Server Installation Guide for more information on configuring a VxSS Root Broker. Would you like to configure product_name to use Veritas Security Services? [y,n,q] (n) n12. A message displays notifying you of the information required to add users. When prompted, set the user name and /or password for the administrator. Do you want to set the username and/or password for the Admin user (default username = admin, password=password)? [y,n,q] (n)13. Enter n if you want to decline. If you enter y, you are prompted to change the password. You are prompted to add another user to the cluster. Do you want to add another user to the cluster? [y,n,q] (n) Enter n if you want to decline, enter y if you want to add another user. You are prompted to verify the user. Is this information correct? [y,n,q] (y) Enter y or n to verify if this information is correct.14. You are prompted to configure the cluster management console. Enter y or n to configure the cluster management console. Do you want to configure the Cluster Management Console [y,n,q] (n) n15. You are prompted to configure the cluster connector. Enter y or n to configure the cluster connector. Do you want to configure the cluster connector [y,n,q] (n)16. When prompted to configure SMTP notification, enter y to not configure SMTP. Do you want to configure SMTP notification? [y,n,q] (n)17. When prompted to configure SNMP notification, enter y to not configure SMTP notification. Do you want to configure SNMP notification? [y,n,q] (n)18. When prompted to set up the default disk group for each system, enter n to set up the disk group for each system. Do you want to set up a default disk group for each system? [y,n,q,?] (n)19. You are prompted to enter the fully qualified hostname of system isvp12. Enter y for the isvp12.domain_name. Is the fully qualified hostname of system "isvp12" ="isvp12.domain_name"? [y,n,q] (y)20. 23 You are prompted to enter the fully qualified hostname of system isvp13. Enter y for the ispv13 domain_name. Is the fully qualified hostname of system "isvp13" ="isvp13.domain_name"? [y,n,q] (y)21. You are prompted to enable Storage Foundation Management Server Management. Enable Storage Foundation Management Server Management? [y,n,q] (n) © Copyright IBM Corporation, 2011 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror 19
  23. 23. 22. Enter n to enable Storage Foundation Management Server Management. You are prompted to start Storage Foundation. Do you want to start Veritas Storage Foundation processes now? [y,n,q] (y)...Startup completed successfully on all systems You declined to set up the name of the default disk group for isvp12. You declined to set up the name of the default disk group for isvp13. Installation log files, summary file, and response file are saved at: /opt/VRTS/install/logs/installsf-7ai12iWhen installsf installs the software, some software might be applied rather than committed. It is theresponsibility of the system administrator to commit the software, which can be performed later with the -coption of the installp command.Installing Veritas Storage Foundation HA using webinstaller interfaceIn addition to the command-line installer, Symantec provides a simple-to-use web-based interface. Theinstaller can be run from root directory of the installation media or the directory resulting from havingunpacked the SFHA installation bundle.In this test case, the installer is run from the directory /tmp/dvd1-aix:isvp12> ./webinstaller &Installing VCS agentsThis section is intended to provide some configuration guidance for the VCS bundled agents for Oracleand Storwize V7000 replication (the SVCCopyServices agent)Installing VCS agent for OracleFor complete details refer to the Veritas Cluster Server Agent for Oracle Installation and ConfigurationGuide at:https://sort.symantec.com/public/documents/sfha/5.1sp1pr1/aix/productguides/pdf/vcs_oracle_agent_51sp1pr1_aix.pdfYou must install the Oracle agent on each node in the cluster. In global cluster environments, install theagent on each node in each cluster. These instructions assume that you have already installed ClusterServer. Peform the following steps to install the agent. 1. Make sure the disk is mounted. Refer to the “Mounting a software disk” section for more information on it. 2. Run the following command to navigate to the location of the agent packages. # cd /cdrom/cluster_server_agents/oracle_agent/pkgs 3. Run the following command to add the file sets for the software © Copyright IBM Corporation, 2011 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror 20
  24. 24. # installp -ac -d VRTSvcsor.rte.bff VRTSvcsorInstalling VCS agent for IBM SAN Volume Controller copyservicesThe VCS agent used to support remote copy services on IBM Storwize V7000 system is the same agentthat has been used in previous VCS configurations with IBM SAN Volume Controller. Despite the agentname (IBM SVCCopyServices) the required resource attributes are applicable to both SAN VolumeController and Storwize V7000 disk systems.For complete details refer to the Veritas Cluster Server Agent for IBM SVCCopyServices Installation andConfiguration Guide at:https://sort.symantec.com/agents/download_docs/2192/vcs_svc_installYou must install the IBM SAN Volume Controller copy services agent on each node in the cluster. Inglobal cluster environments, install the agent on each node in each cluster. These instructions assumethat the Veritas Cluster Server is already installed. Perform the following steps to install the agent. 1. Make sure the disk is mounted. See “Mounting a Software Disk”. 2. Run the following command to navigate to the location of the agent packages: # cd /cdrom/aix/replication/svccopyservices_agent /version/pkgs The variable version represents the version of the agent. In this scenario, version 5.0.3.0 has been installed. 3. Run the following command to add the file sets for the software. # installp -ac -d VRTSvcssvc.rte.bff VRTSvcssvcAll the required software components have now been installed. You should be able to list out the filesetsthat are mentioned in the “Appendix D: Veritas Software file sets listing” on each application host.Installing and configuring OracleInstalling and configuring Oracle involves the following tasks:  Installation of Oracle software  Creation of an Oracle instance  Creation of the databaseFirst, you need to install Oracle on all nodes in both clusters. Make sure that the installation prerequisitesare met and are identical on all nodes, especially the user and group ID, passwords, owner and grouppermissions and listener port ID. Refer to the appropriate “Appendix B: Setting up the databaseapplications” section for instructions to setup the database.In this configuration, a database representing testmm schema is built. A database workload application isused to populate and simulate a Transaction Processing Performance Council OLTP (TPC-C) workload.You will need a workload application to exercise the database load. © Copyright IBM Corporation, 2011 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror 21
  25. 25. .Configuring applications for disaster recoveryHaving installed and configured the base system software and applications, the test team is now ready forconfiguring the required applications for high availability and disaster recovery. Most clusteredapplications can be adapted to a disaster recovery environment by:  Setting up the Storwize V7000 remote copy partnership  Creating Storwize V7000 Metro Mirror, Storwize V7000 Global Mirror relationship, and FlashCopy mappings  Setting up VCS Global Custer  Configuring VCS application service group  Adding the VCS SAN Volume Controller copy services resource  Setting upTo quickly setup a similar test configuration, follow the steps in the “Quick setup” section. You can alsofollow the procedure in the “General configuration steps” section and refer to the documents mentioned inthose sections for detailed configuration steps.Quick setupThe following section lists the steps to configure a new SFHA cluster using the provided sample VCSconfiguration file from “Appendix C: Sample main.cf file - isvp_sfha_clusterA”. 1. Make sure that all of objects mentioned in VCS configuration file are created and available. 2. Halt the cluster server from any node in the clusters in Site A and Site B #/opt/VRTSvcs/hastop –all 3. Cut and paste the VCS configuration file (main.cf) from Appendix C: Sample main.cf file - isvp_sfha_clusterA to /etc/VRTSvcs/conf/config directory as shown here. On cluster nodes isvp12, isvp13 in Site A as: main.cf.siteA On cluster nodes isvp14, isvp15 in Site B as: main.cf.siteB 4. Modify the values of hostnames, IP addresses, mount points, volumes and disk group resources, cluster names, passwords, and so on to match your site specific configuration. 5. Run the following commands to copy the VCS agent type definition files, if they are not present in the destination directory. #cp /etc/VRTSagents/ha/conf/Oracle/OracleTypes.cf /etc/VRTSvcs/conf/config/ #cp /etc/VRTSvcs/conf/SVCCopyServicesTypes.cf /etc/VRTSvcs/conf/config/ 6. Run the following commands to overwrite the existing main.cf file on the primary node on each of the respective clusters, then use remote shell or SSH to copy that file to its local partner. For example main.cf on Site A cluster node 1 (isvp12) #cd /etc/VRTSvcs/conf/config © Copyright IBM Corporation, 2011 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror 22
  26. 26. #cp main.cf.siteA main.cf #rcp main.cf isvp13:/etc/VRTSvcs/conf/config/main.cf On Site B cluster node 1 (ispv14) #cd /etc/VRTSvcs/conf/config #cp main.cf.siteB main.cf #rcp main.cf isvp15:/etc/VRTSvcs/conf/config/main.cf 7. Run the following command to verify that the main.cf does not have any errors and fix it if there are any issues. #/opt/VRTSvcs/bin/hacf –verify If there are no errors, command returns with exit code 0. 8. Run the following commands to start the cluster on each node in the clusters in Site A and Site B. #/opt/VRTSvcs/hastart –all #/opt/VRTSvcs/hastart –all 9. Start the cluster manager from any node in the cluster Site A. Login to one of the nodes with the administrator userID as admin and password as password. 10. On the first node at Site A, login to the cluster manager and bring the app_grp1 service group online if it is not already online. 11. Now you are ready to manage the clusters from the cluster manager GUI. To test HA and DR scenarios, proceed with the instructions in the “Failover scenarios” section.General configuration stepsIn this section, the steps mentioned in the setup are explained in detail. Refer to the documents listed inTable 3 for additional details of configuration procedures. You will need the following guides:  Veritas Cluster Server User’s Guide  Veritas Cluster Server Bundled Agents Reference Guide  Veritas Cluster Server Agent for Oracle Installation and Configuration Guide  Veritas Cluster Server Agent for IBM SVCCopyServices installation and Configuration GuideSetting up Storwize V7000 cluster partnershipRefer to “Setting up the Storwize V7000 remote copy partnership” section for more information.Configuring Storwize V7000 Metro Mirror, GlobalMirror, and FlashCopyrelationshipFor more details, refer to the following sections: “Creating Storwize V7000 Metro Mirror, Global Mirror,and FlashCopy consistency groups” and “Creating Storwize V7000 Metro Mirror, Storwize V7000 GlobalMirror relationship, and FlashCopy mappings”.Setting up VCS Global Custer © Copyright IBM Corporation, 2011 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror 23
  27. 27. Linking clustersRefer to the section Linking clusters in the Veritas Cluster Server User’s Guide. The Remote ClusterConfiguration wizard provides an easy interface to link clusters. Before linking clusters, verify that thevirtual IP address for the ClusterAddress attribute for each cluster is set. Use the same IP address as theone assigned to the IP resource in the ClusterService group. Have the following information ready: Theactive host name or IP address of each cluster in the global configuration and of the cluster being addedto the configuration, the administrator login name and password for each cluster in the configuration.The VCS cluster management GUI (Java™ Console) is no longer packaged in the latest 5.1 release ofVCS. If you attempt to issue the command used to manage previous installations, you will see thefollowing message: isvp12> /opt/VRTSvcs/bin/hagui & [1] 13369510 isvp12> VCS Single Cluster Manager (Java Console) is no longer packaged with VCS. Symantec recommends use of the Veritas Operations Manager (VOM) to manage, monitor and report on multi-cluster environments. You can download VOM at http://go.symantec.com/vom . If you wish to continue using the VCS Single Cluster Manager, you can get it at no charge at the http://go.symantec.com/vcsm_download website.If you elect to download and use the VCS single-cluster manager, you can still click Edit  Add/DeleteRemote Cluster or alternatively use the Veritas Operations Manager (VOM)Configuring global clusterFrom any node in the clusters in Site A and Site B, run the GCO Configuration wizard to create or updatethe ClusterService group. The wizard verifies your configuration and validates it for a global cluster setup. #/opt/VRTSvcs/bin/gcoconfigThe wizard discovers the NIC devices on the local system and prompts you to enter the device to be usedfor the global cluster.Specify the name of the device and press Enter. If you do not have NIC resources in your configuration,the wizard asks you whether the specified NIC will be the public NIC used by all systems. Enter y if it isthe public NIC; otherwise enter n. If you entered n, the wizard prompts you to enter the names of NICs onall systems.Enter the virtual IP to be used for the global cluster, which you have already identified. If you do not haveIP resources in your configuration, the wizard prompts you for the netmask associated with the virtual IP.The wizard detects the netmask; you can accept the suggested value or enter another value.The wizard starts running the commands to create or update the ClusterService group. Variousmessages indicate the status of these commands. After running these commands, the wizard brings theClusterService group online. © Copyright IBM Corporation, 2011 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror 24
  28. 28. Configuring VCS application service groupThe VCS service groups can be setup once the VCS agents have been installed. In this setup Oracle isinstalled on all the cluster nodes.Figure 3 shows the dependency graph of the VCS resources within each service group. There are twoservice groups. The application service group test_shared_DG and the cluster service group which iscreated when you enable Global Cluster Option. The application group has been configured as a globalservice group, which means it will be able to fail over to local and remote cluster nodes, if required. Youalso need to observe that based on the resource dependency view, the Oracle database resourcedepends on the mount, volume, diskgroup, and SVCCopyServices resources being available.Refer to the Veritas Cluster Server Agent for Oracle Installation and Configuration Guide for adding andconfiguring Oracle resource types.Configure the application service group on the primary and secondary clusters. It involves the followingsteps: 1. Start the VCS Single Cluster Manager (Java Console), or alternatively use the Veritas Operations Manager (VOM) and log on to the cluster. 2. If the agent resource type (Oracle / SVCCopyServices ) is not added to your configuration, add it. From the Cluster Manager File menu, click Import Types and select the appropriate file types for the agent in the following directory path : /etc/VRTSagents/ha/conf/Oracle/OracleTypes.cf, /etc/VRTSvcs/conf/SVCCopyServicesTypes.cf 3. Click Import. 4. Save the configuration. 5. Create the service groups. In this case, you can create the test_shared_DG service group. Refer to chapter 4 of the Veritas Cluster Server Agent for Oracle Installation and Configuration guide to create the service group. 6. Add the resources mentioned in the test_shared_DG service group in main.cf for the respective cluster. 7. Configure the service group as a global group using the Global Group Configuration wizard. Refer to the Veritas Cluster Server User’s Guide for more information. © Copyright IBM Corporation, 2011 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror 25
  29. 29. Figure 3: Sample VCS dependency tree for service group resourcesAdding the VCS SAN Volume Controller copy services resourceRefer to chapter 3 of Veritas Cluster Server Agent for IBM SVCCopyServices installation andConfiguration Guide for detailed information on the configuration of Fire Drill resource.The VCS agent for IBM SVCCopyServices manages replication relationships and consistency groups thatare defined on SAN Volume Controller clusters or IBM Storwize V7000 subsystems. Before you configure the agent for SVCCopyServices Before configuring the agent, review the following information:  Set up the SSH identity file on the VCS hosts prior to configuring the service group. Use the SSH keygen, if required. Generate a public and private key pair using the ssh-keygen utility. Copy the public key on the Storwize V7000 system. For detailed instruction on generating the keypair for AIX, refer to the “Appendix A: OpenSSH client configuration between AIX and Storwize V7000” section.  Review the configuration concepts, which describe the agent‟s type definition and attributes.  Verify that you have installed the agent on all the nodes in the cluster. © Copyright IBM Corporation, 2011 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror 26
  30. 30.  Verify the hardware setup for the agent. Make sure that the cluster has an effective heartbeat mechanism in place.Adding the agent resource typePerform the following steps on every node in the cluster. Adding and configuring the agent manuallyin a global cluster involves the following tasks. Next section describes details of the resource typedefinition and attributes.1. Start the VCS Single Cluster Manager (Java Console), or alternatively use the VOM and log on to the cluster.2. If the agent resource type is not added to your configuration, add it. From the Cluster Manager File menu, click Import Types and select /etc/VRTSvcs/conf/SVCCopyServicesTypes.cf3. Click Import.4. Save the configuration.Now you can configure the SVCCopyServices resource by:  Creating service groups for SVCCopyServices, which you have already done in the previous section by creating test_shared_DG service group.  Adding a resource of type SVCCopyServices at the bottom of the service group. If you are adding the resource, use the Java Console GUI, VOM, or the command line and follow the resource names from the example and create the v7k_agent resource #hares –add v7k_agent SVCCopyServices test_shared_DG  Configuring the attributes of the SVCCopyServices resource. In this example add the GroupName and Storwize V7000 IP attribute values. #hares –modify app_svccp GroupName v7k_mm_DR #hares –modify app_svccp SVCClusterIP XXX.XXX.XXXSVCCopyServices resource type definitionThe IBM SVCCopyServices agent is represented by the SVCCopyServices type in VCS. type SVC CopyServices ( str GroupName int IsConsistencyGroup = 1 str SSHBinary = "/usr/bin/ssh" str SSHPathToIDFile str SVCClusterIP str SVCUserName = admin int StopTakeover = 0 int DisconnectTakeover = 0 static str ArgList[] = {GroupName, IsConsistencyGroup, SSHBinary, SSHPathToIDFile, SVCClusterIP, SVCUserName, StopTakeover, DisconnectTakeover} ) © Copyright IBM Corporation, 2011Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror 27

×